What are the best web application test frameworks people have seen? Any language, though I'm thinking more about unit tests, not as much about through-HTTP acceptance testing.
I'm curious mostly because I want to see other useful features for me to borrow. Right now the test framework in Paste is pretty simple, but IMHO in a good way.
There's some other features I've seen that I'm unsure of. For instance, the ibofobi framework for Django uses Beautiful Soup for extracting response strings for testing. Do people think this is useful? Personally I'm happy enough testing for the simple existence of strings, without caring about where in the structure of the document they appear. Also it works better for negative assertions, i.e., asserting that a string doesn't appear anywhere in the document; I do a lot of these in my tests.
Also, what do people think about doctest in this context? I started out using that, but ended up reverting to py.test because my tests quickly became more complicated than doctest could really handle elegantly. Doctest still worked, I just wasn't using any of the features that make it convenient (and I was missing the py.test features). Note that paste.fixture doesn't really require py.test, since py.test is a kind of null framework; I just don't go out of my way to add any special unittest hooks (like, say, a custom TestCase subclass).
I haven't looked closely at Twill since shortly after it split from PBP. I like in-process tests a lot, so Twill doesn't really test in the way I'd want it to. Though I bet I could tweak it to test in-process too... anyway, a feature it has is how it fills in forms nicely. I just added form filling to Paste, but it's very rough still. And I feel bad that I'm using regexes and not Mechanize or something similar. One thing Paste can do is require reasonably well-formed markup, because it's not intended for testng applications you can't control (but I'm not willing to require XHTML, just reasonably unambiguous HTML 4).
Zope 3 emphasizes testing, but I have to admit that I am unimpressed by the functional testing (and doctest functional testing) there. It seems long-winded. There might also be better systems people have set up there. Paste creates a wrapper around the "application," which seems much better than the kind of manual request setup I see for many frameworks (like in TurboGears).
One thing I've done in paste.fixture is to expect success unless told otherwise. So you don't confirm a request returned a 200 status code -- it's an error if it doesn't, unless you explicitly say you expect something else. I think explicitly checking for a 200 status code is crazy.
What about patterns for matching markup? I test strings against a normalized form of the page (mostly whitespace normalized), but sometimes you do want to test the structure, and I want to avoid fragile tests that test for more than they should (and will fail, for example, due to small design changes). Regular expressions aren't good here; instead it's good to have something that knows about the structure of markup, knows to ignore attribute order, knows when to ignore whitespace (and where not to), etc. I've played with this before, but I'd be very happy if there was someone else's code I could use here.
Do you tried Selenium? I have not tried it but read some reviews etc about it and tried the online demo. Looked really useful for Browser tests which I still think a decent test is 8or maybe was) lacking...
http://agiletesting.blogspot.com/2005/02/articles-and-tutorials.html
# chris
meant "did you try" of course but was just typing away...
I have, and we'll probably use it on some level, but it doesn't really apply to unit-style testing, or even what I would classify as functional testing. It's much more about acceptance and regression testing. I find the tests too verbose to write by hand; there's a nice Firefox extension for recording, but you can only record after the application is written.
# Ian Bicking
i also have a cherrypy filter that i wrote that lets me write Selenium tests in a much less verbose syntax (no ugly html table code). i could share it if anyone's interested.
i still use py.test for most of my testing but for acceptance testing of web apps, Selenium has proven itself extremely useful. there's something extremely comforting about seeing that everything works together from end to end all the way to the browser.
Ian-
You might want to check out TestGen4Web (http://developer.spikesource.com/wiki/index.php?title=Projects:TestGen4Web) which is a firefox extension for capturing browser activity. You can save your script as xml and then use adapters to change that xml into testcases for testframeworks. There are currently translaters for simpletest (php) and httpunit (java).
matt
That project has an awesome schematic! It seems like an interesting project, and after playing around with several recorders the translation seems very reasonable -- capturing higher-level information than just HTTP requests is useful, and best done at a browser level, but it doesn't need to be that closely attached to the target test framework.
However, like I mentioned in the Selenium comment, this doesn't really feel right for unit testing IMHO, and certainly doesn't work well for TDD.
# Ian Bicking
From the article:
Zope 3 emphasizes testing, but I have to admit that I am unimpressed by the functional testingZope 3.2 will include zope.testbrowser which is a much nicer way to do functional tests. See http://svn.zope.org/Zope3/trunk/src/zope/testbrowser/README.txt?rev=40035&view=markup for example usage.
It is also usable outside of Zope, and I hope on packaging up a stand-alone version around the time 3.2 is released (December). If I get time I'd also like to do a WSGI version.
zope.testbrowser looks like it has a nicer API than mechanize, at least for testing (speaking as the author of mechanize), and I see Stephan Richter is hoping to implement the same API using Selenium -- cool.
BTW, mechanize should support
easy_setup mechanize==dev
properly as soon as the latest setuptools is released... would be nice to fetch zope.testbrowser the same way
# John Lee
Yes, that's much better indeed. Now I can't remember quite why I wrapped my application in an object, instead of using a browser object as the central metaphor. In the end it's not that much different, except that some things like "back" don't exist for an application, and it's not as stateful (which has pluses and minuses).
It looks like zope.browser actually uses HTTP requests? WSGI would definitely be nice, as it would allow in-process requests, something that I have found very convenient. Seeing the framework's nice-formatted HTTP exception reports is not particularly useful when you are running automated tests. I also like that I can send out-of-band information; for instance, I use REMOTE_USER to set the username (or sometimes put it in test-specific configuration), instead of simulating a full login. Then I don't need fake logins, or to put my password in the tests, etc.
Another feature that I added to paste.fixture -- in part after looking at Rails' and CherryPy's tests -- is the framework hooks which allow frameworks to record internal information into the response object. This means you can simultaneously test the full-stack rendering and response, and internal details like what variables were passed to the template. Obviously you can only do this in-process.
# Ian Bicking
It looks like zope.[test]browser actually uses HTTP requests?For Zope 3 it directly communicates with the in-process publisher. Not only is it much faster that way, you can have non-testbrowser code do other types of tests (as you mention below). There's also a generic browser object you can use that makes connections over the wire for testing "remote" applications or doing non-testing tasks.
I intend on integrating the same way with WSGI; zope.testbrowser would actually be a WSGI server into which you would plug the application(s) under test.
Of course this fits into my desire to make Zope 3 paste.deploy compatible.
Twill exports functions named the same as the twill commands, so I guess that's your "in process" requirement ticked...
In-process would mean that it calls an application in the same process, instead of generating an HTTP request. So an exception in the application would go all the way up to the command runner (py.test, unittest, etc).
# Ian Bicking
Oh, I see, seems you're right.
You'd need a urllib2.HTTPHandler that does what you want instead of real HTTP, then subclass mechanize.Browser to override the handler_classes attribute, and persuade twill to use your Browser subclass.
It would be a clever hack to monkeypatch urllib2.HTTPHandler to override requests to certain hosts and instead send them to in-process WSGI applications. That would mean that any urllib2-using Python app should be able to work this way, including Twill or whatever else.
# Ian Bicking
That's what I meant, but no "monkeypatch" is required: you just use a different handler instead of urllib2.HTTPHandler.
...oh, see what you mean -- it's the "persuade twill (or whatever) to use it" part that's troublesome. Sorry for spamming your blog so much ;-)
Form-filling in mechanize, twill (assuming Titus starts using the new code), and zope.testbrowser will all get nicer when the latest ClientForm is released in a stable version (soon I guess, since it's been released for some time labelled 'alpha'), thanks to Benji York & Gary Posters improvements for zope.testbrowser.
Correction: zope.testbrowser already has these improvements.
I think doctest + (twill or zope.testbrowser) look like a nice combination.
One feature that I really like, and that I haven't seen elsewhere is the fixture support found in RubyOnRails. This allows you to have a YAML file that loads data into your db for each test, and then removes it when the test is complete. Very helpful for testing models.
I haven't looked closely enough at paste to see if unit test stubs are autogenerated for models, controllers etc when the respective code is generated. That might be a nice touch too if you don't have it already.
I did add something recently for loading data into SQLObject classes with a CSV file. However, actually populating a database for a test is a somewhat separate concern. Right now I'm doing that in some ad hoc setup_module() functions (which py.test invokes), but a complete stack should handle pulling all these things together.
I like having paster create templates build space for tests, but that this time I'm -0 on actually creating things for each controller and model. It seems excessive; I'd rather just add another function to a file. And I'd rather keep the boilerplate down, so that adding a test for a new controller just means:
def test_view(): res = app.get('/view')Which isn't something that needs generating, IMHO. (I do generate a test for app.get('/'), but that's more of an example than anything).
# Ian Bicking
Re the firefox Selenium recorder extension: httpunit is noteworthy for JS support, IIRC. Not sure if it's actually useful, though.
Other firefox Selenium recorder extensions:
http://www.mail-archive.com/selenium-devel@lists.public.thoughtworks.org/msg00099.html
http://www.augure.com/dev/SeleniumEditor.xpi
and:
http://seleniumrecorder.mozdev.org/
Related: twill has maxq support for recording tests from a browser. Not extension-based, though -- it just uses a proxy, so there's less info there to work from.
A couple quick comments re testing & twill.
I'm starting to use 'nose' for unit testing, and it's great.
In conjunction with my 'nose' unit tests, I've also just added a fairly simple interface for specifying (a) an app to run and (b) a twill script to use to test that app. Watch planetpython for my blog post on that.
The maxq recorder for twill sucks. Or should that be Sucks? I'm hoping to get some work done on it soon, but right now it's not terribly useful. (Ironic, given that one of the reasons I wrote twill was so that I could play back recorded tests.)
I'm intrigued by John's opinion of zope.testbrowser. I liked the look of it too, but it'd be a big change for twill and I'm not feeling that energetic right now ;).
Why not define a new URI scheme, e.g. 'wsgi://', and then put in a urllib2-like WsgiHandler or some such? That could potentially be a simpler way to do "in process" tests than monkeypatching urllib... either way, I like the idea of getting twill/mechanize/* to talk to things without going through HTTP.
John -- I promise I'll start using the new mechanize code soon ;). Right now I'm a bit overwhelmed, but it's on my TODO list.
--titus
Why not define a new URI scheme, e.g. 'wsgi://', and then put in a urllib2-like WsgiHandler or some such?
Lots of code expects a constrained set of URI schemes. http and https, in particular; I know I don't generally write code with extensible schemes in mind. And either way, I think a WSGI app would have to be mounted into place... though I perhaps automatic mounting would be possible, where you did wsgi://path.to.module/object or something like that. And I suppose Paste Deploy has URIs, though the specific schemes it uses are a bit ambiguous in this case.
I don't think the monkeypatching would be hard. I guess it's just a question of whether it is wise. But it seems as good as anything to me.
# Ian Bicking
At the risk of repeating myself, no monkeypatching is required to add wsgi: URL support -- urllib2 is designed so that adding new URL schemes and behaviour is easy iff you have access to the OpenerDirector instance (the thing you call .open(urlOrRequest) on). Titus, as the author of twill, clearly does have such access, though I don't know if it's also exposed as part of the twill API.
Point taken -- rather than breakage with wsgi:// (which I thought might be clean) I will look at subverting urllib2. The goal will be to produce a WSGI-compliant server interface into which any WSGI app can be plugged & played.
This is a good direction for twill, IMO, so I'm moderately enthusiastic about implementing it, too.
A few additional comments:
zope.testbrowser looks nice, but it's unclear to me what the compelling advantage is. I've got to look at it more before I make my life more complicated by trying to switch to it... ;)
the twill API doesn't provide direct access to anything mechanize-tic, but the full mechanize interface is of course accessible to extension functions, which are just Python (of course!).
Finally, even if twill becomes WSGI-aware, writing the tests is going to be a huge pain. (Hmm, maybe if I get the BickingBot entangled in twill he'll write a WSGI recorder!)
--titus
Simple (but working) implementation of in-process testing with twill: <a href="http://www.advogato.org/person/titus/diary.html?start=119">discussed here</a>.
I don't imagine there'd be much purpose in re-writing twill for zope.testbrowser, would there? Would just introduce an unneccesary extra layer of complexity.
# anonymous
oops, that last comment about twill / zope.testbrowser was me.
Why invent the 'wsgi:' scheme, when you can just reimplement urllib2.HTTPHandler?
# anonymous
When I wrote my webapp framework last year, I too found that Zope testing was inadequate, and I still do. I ended up writing my own app end-to-end (I could not bear without an autoreload feature for the source code, and all Zope3 facilities for fast development were broken, couldn't use webdav to upload files, had to restart server on every change, there was no way I was going to work like that), and using mechanize for testing it. IMO best testing for the web should be done from a browser's perspective to really be effective. I have found so many problems that would have slipped by without my tests.
One useful idea that I introduced in my app was to return custom meta fields in the head, that reflect some of the internal state of the server app, for example, the currently logged in user, and a return status string. These only show up when the app is in debug mode, and the tests assert that the return statii are what is expected.
Another idea was to add extra test resources/pages in debug mode. For example, when my app sends email to other users, well, in debug mode, it doesn't really send, rather, it puts the emails in a DB table (only the last email for each user) and then the tests can recuperate the last email sent for a particular user, whose contents it then validates. I can parse the email to get some links with tokens, etc. Just trying to do as close to the real processes as possible.
cheers,
paste.fixture does both of these:
- There's a special key for putting values for use in tests (paste.testing_variables), which takes the place of headers. It's out-of-band with respect to any HTTP information, and values needn't be serialized to strings.
- There's also ways to detect when you are in a debugging environment (paste.testing) -- some people think this is really bad, but I think those people are just scarred from having someone abuse that feature to put workarounds in the code for the tests. The rule of consenting adults say, to me, that the variable belongs there. Though, I suppose, those people want to set up all the mock objects ahead of time using a complicated setup process (e.g., dependency injection). A boolean is easier.
# Ian Bicking
I have been playing around with Turbo Gears and the first thing I wanted to do from the begining was write web tests. TurboGears testing simillar to Paste. However the one thing I missed alot was testing forms, this is usually what I want to test in my webpages.
Also, if you give some semantics to your html, like use classes you can extract more data out of it and, with the aid of BeautifulSoup, match the number of, let's say divs with the class 'product_entry' exist in the page and compare it with your model representation.
Here are the posts I wrote about the functions I mentioned:
- Post 1: http://s1x.homelinux.net/node/150 talks about form_fill, form_to_dict and browse_url
- Post 2: http://s1x.homelinux.net/node/149 talks about form_encode
Have you checked out PyMeld and family? Although they require XHTML, I think.
have you tried Funkload ?
Web surfing is fun, but many tasks are repetitious: Checking on the same sites everyday, remembering passwords, submitting to search engines or testing web sites over and over again. With iMacros you record these tasks once and then let the iMacros software execute them whenever you need them. Any combination of browsing, form filling, clicking and information gathering can be recorded into a macro and the iMacros software assists you during the recording with visual feedback. Do you need to extract price lists, stock information or any other data from websites?
iOpus Internet Macros can help you: http://www.yaodownload.com/internet-tools/browsers/iopusinternetmacros/
# joe
Check out WatiN. It's an open source project which enables (unit) testing through internet exploer in any .Net langauge. For more info visit http://watin.sourceforge.net