Ian Bicking: the old part of his blog

Functional testing in Paste

I was just looking at Rails' functional testing. Here's their first example:

require File.dirname(__FILE__) + '/../test_helper'

# grab our HomeController because we're going to test it
require 'home_controller'

# Raise errors beyond the default web-based presentation
class HomeController; def rescue_action(e) raise e end; end

class HomeControllerTest < Test::Unit::TestCase
  def setup
    @controller = HomeController.new
    @request = ActionController::TestRequest.new
    @response = ActionController::TestResponse.new
  end

  # let's test our main index page
  def test_index
    get :index
    assert_response :success
  end
end

Here's how you'd do it in Paste:

from paste.tests.fixture import *

def test_index():
    res = app.get('/')

(Note: 100% framework neutral!)

Why is it so much shorter? Well, first it uses py.test instead of an XUnit-based system, which gets rid of the stupid cruft of XUnit. This isn't a criticism of Rails or Ruby, per se -- we have the same module in Python and it's no better. py.test is much better, though.

Second, setup is done in paste.tests.fixture.setup_module, which is a special function that py.test calls (and you imported with import *). This finds your configuration file and creates your WSGI application. This part (finding a configuration file and using that to construct an application) is Paste-specific, but everything else sticks to WSGI. This also puts app (an object intended for testing) in the module's namespace.

Lastly, this object assumes when you say app.get('/') that you expect everything to work. That means a 200 OK response (as well as a couple other checks -- redirects are also okay). You have to specifically indicate that you expect another response; since this is an object intended for testing, why not build test-friendly assumptions into it?

Looking through the Rails docs on this, there's some other useful features I'd like to add. Because this only uses opaque WSGI applications, you can't look at the variables used to render the document; the only communication you get is the actual textual response. Here's what I frequently do:

def test_index():
    res = app.get('/')
    for item in db.TodoItem.select():
        res.mustcontain(item.title)

I'm not actually that interested in how the view got the items, so looking at the communication between view and controller isn't that interesting. But I can imagine adding some convention so that frameworks (when run in testing mode) could stuff objects into the WSGI environment for later inspection.

One of the things I like about Paste's tests compared to tests that are more opaque (e.g., actually speak to the app over HTTP) is that I have access to the backend objects (like db.TodoItem), and exceptions propogate (which gives me py.test's fancy tracebacks). As a result I've become less excited about the functional testing provided by Twill and others.

The whole thing is fairly young, but I think it is pleasantly simple and yet quite useful. If you are interested look at the docs.

Created 01 Jun '05

Comments:

I'd be nice to see examples of the test failures. As you say, the fancy tracebacks. These are often one of the hidden beauties of testing frameworks.

# michael

Where does app come from?

From looking at the test I can't tell where app comes from.

I suppose it could be a good thing to write generic tests for multiple applications.

Perhaps explicitly tell it where the config file is?

However what is in this config file? If I was writing this test from scratch I'd need to know the config file format, and what needs to go in it.

Importing your app module directly might be the go to make it more explicit?

from paste.tests.fixture import *

import yourapp
app = yourapp.app

def test_index():
    res = app.get('/')
# Rene Dudfield

The configuration is a Paste configuration file, which describes an application. The configuration file is found in server.conf, and the fixture searches parent directories for such a file until it finds one.

The app object is actually paste.tests.fixture.TestApp(paste_app).

Generally this should work easily if you have your application set up to run in Paste. But not so easy otherwise. Hrm... and I don't think I really have good documentation for the configuration file at this time :(

# Ian Bicking