Ian Bicking: the old part of his blog

Unittest Rant

I don't like the unittest module. I'm just starting to really do test-driven development (takes a while to catch on to), and I find unittest to be obnoxious and highly under-featured. Am I missing something?

For instance, I often want to be able to test data sets, i.e., a bunch of inputs with expected outputs (or expected errors). This is a pain in the butt. It's easy enough to use a for loop and iterate over the data, but that's bad for test driven development -- I want to see progress right away, I don't want my test to bomb on the first failure.

So, I make a subclass of TestCase, maybe like:

class DecodeTest(unittest.TestCase):
    def __init__(self, input, output):
        self.input = input
        self.output = output
    def test(self):
        self.assertEqual(decode(input), output)
OK, so it's annoying to create this class since it seems like an obvious need in testing, but once I've done it I can move on. Except I have to collect these all into suites and add them to the system. And once I've done that all my tests are essentially anonymous -- I can run a particular suite, but I can't run a particular input/output test.

Probably the issue here is that unittest.main is stupid. You can't add options (e.g., your own verbosity hooks), you can't easily indicate what tests you want to do... it's just dumb and not extensible.

The features unittest does have are things like pluggable loggers, so you can have graphical output. Who cares? Maybe other people are more motivated by pretty lights, but personally I am more motivated by reducing the tedium of writing tests. I'm okay with writing the tests -- I like TDD, really -- but I've yet to hear anyone say they really like writing tests.

But you can't get around it -- tests are tedious, because you have to think of the corner cases, the possible combinations of usage, etc. At least let us get rid of the boilerplate! unittest just feels like a lot of boilerplate, and methods like assertEqual are useful but not compelling.

I gotta find a new way to do tests. doctest is too minimal -- I do need setup and teardown (but not setUp and tearDown -- I'm sorry, but this is English, "setup" and "teardown" are two words, not four). I would just write my own, but someone else out there must have cared enough to make something Pythonic and convenient...

Created 10 Oct '03
Modified 14 Dec '04

Comments:

I wrote something Pythonic and minimalistic, but I'm not sure if it does what you want... then again it can always be extended. It's called Testipy and can be found in my download section. (http://zephyrfalcon.org/download/) If you want, drop me an email so we can discuss what features you would find useful. If I agree, I might add them. :-)

I wrote Testipy because unittest got in my way a few times, so I decided to make my own little testing framework. I'm using it now for some of my personal projects.
# Hans Nowak

There's no credibility when you complain even about the method names' case.
#

I'm agreed about the cruft in these unit testing frameworks. When it's just easier to write a test "main program" and to use shell scripts to make sense of the results, these frameworks don't really seem to be hitting the spot.

And I'm totally in agreement about the hypothetical graphical progress bar cruft.
# Paul Boddie

I use the Sancho testing framework, http://www.mems-exchange.org/software/sancho/,
instead of unittest. A useful feature is that it can optionally do code coverage, so you can quickly see which branches or functions aren't being exercised by the tests.
# A.M. Kuchling

Agree.
# Tim

In general I agree that the feature set is a little thin in the stock module, but there is enough to get started writing tests quickly. I think it depends on your perspective too. I use Python for functional/system testing of other sytems (not written in Python), and all my previous test harnesses have been centered around data-driven test drivers (along the lines of your DecodeTest example). If you look at the 'common' examples of unit testing today, with JUnit for example, you almost never see it used that way.

I did a couple of things. I made a base TestCase class for data-driven tests. The parameters are passed in via __init__ (exactly like your DecodeTest). There's a matching SuiteBuilder that can read various table formats (ODBC db tables, excel spreadsheets, csv files, etc) that is used to instantiate each test. Each column in the table is a different param (input,output for your example), and you just add any additional columns as needed. For example, I generally have a 'description' column that gets passed to all tests. That get used to change the the result of shortDescription(), so the tests are no longer 'anonymous'. At that point, adding new tests for a particular case is just a matter of adding another line to a spreadsheet (or record in a db table, whatever). Your suite builder can use the extra columns to pick out a single test, or filter out some number of them. Maybe unittest could have provided some of that out-of-the-box, but it really wasn't that much work to tack on.
# Chris Prinos

1. You don't need suites. Just add:

if __name__ == '__main__': unittest.main()

at the end of your file, and you can run the script and it'll find all the tests and automatically run them.

2. assertEquals is useful. The reason is, if it fails, you get to see what two values were, whereas "assert x == y" won't display the difference. Also you should be writing multiple test cases per class, not one, and using setUp and tearDown. Passing info for tests in __init__ shouldn't be done. Each test should set it up itself, or if its for all tests in the class, use setUp.

3. It is kinda limited. Twisted ended up writing an API compatible system that is more powerful in many ways (and customized to our needs), and has a command line test running tool, so you can say "trial twisted.test" and it runs all tests in the modules in that package. Zope3 has written a similar commandline tool for python's unittest.
# Itamar Shtull-Trauring

The correct URL for Sancho is http://www.mems-exchange.org/software/sancho/ .
# Neil Schemenauer

Another option for a test runner (with additional functionality, including coverage) is:

http://www.nullcube.com/software/pylid.html
# Phil

Zope's test.py driver addresses some of the issues you raised. I agree that unittest's main() isn't very useful and that unittest as a whole is hard to extend or customize.

I wrote a
weblog entry
about the test driver.
# Jeremy Hylton

Itamar: I know you can loop over the input, use setUp and tearDown, etc. But it's not very convenient. Take the example from Dive Into Python -- it tests a bunch of known integer to roman numeral conversions. The unit test isn't much of a unit, because the first conversion that fails causes the entire conversion test to fail. So you don't see that ten of the conversions worked, but five didn't. If one fails, you don't get to see if any of the later ones failed. Not a big deal in roman numerals, but that's a trivial example. When you are working through corner cases one by one, it's nice to be able to see and rerun each failure individually, and unittest doesn't give that sort of granularity. (I don't want to define 100 methods to test 100 possible input/output combinations either)

Chris: yes, I end up adding this functionality in subclasses. Unfortunately it doesn't work well with unittest.main. As Jeremy notes (and I tried to imply) unittests that take arguments to their __init__ don't have proper names (even arbitrary names), and collecting them for main() is unintuitive. They work, and maybe that means that unittest.TestCase is a fine class (if not perfect), but these instances don't fit into the system well.

Anyway, thanks all for the suggestions. I'm downloading them and it looks like there's some good stuff in these.
# Ian Bicking

#

I've had some success with dynamic tests, for example in my self-testing test framework.

The trick is to define self-stocking descendants of unittest.TestSuite which hook __call__ to fill themselves with dynamically created instances of unittest.FunctionTestCase (or descendants with appropriate setUp methods). I defined the test functions on the fly but you could use any callable or, for extra evil points, lambda.
# Garth T Kidd

2 points

> There's no credibility when you complain even about the method names' case.

Why not? what would you say if the method was called sEtuP?
setup is one word in english. So the complaint is very legitimate.

Another big complaint is its OOP only interface:

In many testing scenarios I simply do not want classes/methods/inheritance/etc

what I want is this:

chk("foo of 1 correct", foo(1), value)
chk("foo of 2 correct", foo(2), value)

In 99% cases that's the only thing which is needed but unittest doesnot
provide it


#

I agree the OOP-only interface is a pain. Many times, I'll have a large set of basic tests I'd want to include. Something like


for i in range(100):
for j in range(100):
self.assertEquals(my_add(i,j), i+j)


Having all of these in a single test that either fails or succeeds as one, is often not what I want.

I can hack this, like so:


class AddTests(unittest.TestCase):
pass

for i in range(100):
for j in range(100):
def t(self):
self.assertEqual(myadd(i,j), i+j)
setattr(AddTests, 'test_%s_%s' % (i, j), t)


but boy does that feel like a hack!

(I hope this comes out formatted OK...)
# Paul Moore

Nope. Sorry - I hope it made sense. Is there any way of putting properly indented code in a comment here?
# Paul Moore

Doubt it.

I've put up an article about self-stocking TestSuites on my alternate site. The technique is pretty useful. David Goodger put something similar in the docutils tests...
# Garth T Kidd

Hi Ian,

Thanks for the rant! I'm always keen to hear what people like and dislike about unittest.

I'm not inclined to defend the spelling habits of Beck or Gamma, from whom the spelling of the method names was appropriated, but I can at least point out that Merriam Webster lists 'tear down' and 'set up' as perfectly valid alongside their single word counterparts. Were I to write the module again, I would choose a more pythonic naming convention for the method names, but it's sadly rather late for that now.

The inconvenience of testing large data sets with PyUnit is often cited, and I am sympathetic to this. This style of testing diverges from that of true unit testing, and is not conveniently supported by JUnit either. True test-driven development typically specifies behaviour incrementally in a relatively small number of pertinently named and short unit test methods; unittest supports this well, I believe.

Of course there is also a place for larger-scale testing with extended sets of data. It seems reasonable to expect that unittest could be turned to this purpose, and I am investigating what can be done to make life easier in this regard. It could also be argued that a different kind of testing framework might work better for such tests or, perhaps, for your stripped-down individual needs.

Whatever the case, if you have concrete suggestions or further comments I'd be very happy indeed to hear from you and to do what I can to make your life easier.
# Steve Purcell