Lately I’ve been doing most of my testing with doctest, primarily using stand-alone text files. I generally like it (otherwise I wouldn’t be using it), but it does make me frustrated with doctest sometimes. On my wishlist (roughly in order):
-
I wish output was always displayed, even when there’s an exception. I see no reason for the current behavior. Really exceptions could be treated like any other output (if ELLIPSIS was on by default).
-
I wish you could turn on options like ELLIPSIS from within a doctest, for all expressions. (# doctest: +ELLIPSIS on every line is beyond ugly.)
-
<BLANKLINE> is terribly ugly.
-
There’s no way of saying "I don’t care what this prints". You can’t do:
>>> some_function()
...
because the ... is treated like a continuation.
-
Plugging in an alternate output checker is kind of tedious, and can’t be done from within a doctest (without horrible hacks).
-
I’d like to be able to easily jump into an interactive state from doctest. Maybe pdb can do this, but I’ve never figured that out exactly.
-
Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.
-
There’s no way to abort the doctest. Sometimes I’d like to run some environment checks early on, and be able to stop the test if they fail.
-
I wish it was easier to apply to non-Python code. (I’ve adapted it via subclassing for Logo but I wouldn’t do that often.)
-
I wish I could copy and paste from doctests to consoles. But I don’t see any solution to this problem.
-
The integration with unittest is pretty hacky. Not that I’ve used unittest in years. But some other test frameworks build off this integration.
-
python -m doctest sometest.txt doesn’t do what it should do. Instead it runs doctest’s self-tests.
Automatically generated list of related posts:
- Doctest for Ruby Finally, someone wrote a version of doctest for Ruby. Recently...
- Doctest.js & Callbacks Many years ago I wrote a fairly straight-forward port of...
“Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.”
Yes that is a problem, the nose plugin is not really handy. We ended up faking regular test cases to be able to run our .txt doctests easily with Nose.
see http://www.gawel.org/weblog/2008/07/nose-doctest-plugin-sucks for an example of such a script.
I think Nose could be enhanced there.
“There’s no way to abort the doctest. Sometimes I’d like to run some environment checks early on, and be able to stop the test if they fail.”
You could do a simple assert on your test, with the
REPORT_ONLY_FIRST_FAILUREflag on. The assert can be wrapped in a global function you provide in the test environment.On dropping into the interpreter, Duncan Booth set me strait [here](http://groups.google.com/group/comp.lang.python/browse_frm/thread/9c9274bdc43a17d2/0d941af24f2057a2?lnk=gst&q=doctest+pdb#0d941af24f2057a2 “(In answer to) Ruby doctest”).
You can drop into
pdbwith the usualimport pdb; pdb.set_trace(), but, in my experience, you land in the middle ofpdb‘s code. Justnextto get back to the doctest scope.The zope.testing [1] testrunner launched with the -D option drops pdb as postmortem shell when a test fails. This test runner can be used easily for non-zope related project. This is really a huge time saving feature when doing TDD.
[1] https://launchpad.net/zope.testing
Some notes:
Stuff like
pdb.set_trace()is good, but it’s better when you can do it when the test fails (what zope.testing and nose can do with options). Interaction isn’t part of the test, it’s part of the testing process.Something I forgot to mention:
REPORT_UDIFFalso has this problem. You have to put the option into the test to see the diff’d output, but it’s really part of the testing process, something you apply when you can’t quickly see the differences in output.REPORT_ONLY_FIRSTdoesn’t work: it still runs all the other examples, it just doesn’t report them. This is often fine, but sometimes I really want to abort the test. A common example would be a database-oriented set of tests, when I can’t open the database connection. Sometimes when testing destructive file operations it’s actually dangerous to continue the test if you can’t confirm that you are in a safe scratch area.Regarding pdb, a tip I learned from Rob Miller is that you want to put your set_trace() on the same line as the code you want to step into:
>>> import pdb; pdb.set_trace(); foo()
And once you step out of foo(), you can still step through the doctest, but stupidly you can’t see what line you’re on. This sucks.
“Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.”
Me too, which is why I put them in setup.cfg and/or ~/.noserc:
“see http://www.gawel.org/weblog/2008/07/nose-doctest-plugin-sucks … I think Nose could be enhanced there.”
Unfortunately, I don’t speak French, so I’m only guessing that the post there is complaining about the lack of support in the doctest plugin for fixtures. If that’s the case, some support is scheduled for the next nose release:
http://code.google.com/p/python-nose/issues/detail?id=60
You can find the current state of fixture support in the ticket-93 branch in nose’s svn. If you find it still too sucky, patches are always welcome.
” … I wish I could copy and paste from doctests to consoles. But I don’t see any solution to this problem. … “
I use IPython. It solves quite some problems:
allows for pasting doctests and evaluating them (IPython.dtutils.idoctest), optionally
raising a exception on the first failure
doing a post-mortem pdb automatically
executing the doctest in a separate namespace
switching to a “normal” prompt (ipython normally has a fancy, colored, history-enabled prompt), so you can paste code from the interpreter to a doctest. One might consider this not a new feature, because the standard interpreter has this too :P
Btw, you can invoke a IPython-enabled pdb by using http://pypi.python.org/pypi/ipdb and:
from ipdb import settrace; settrace()
I like IPython :)
Stefan.
I will second Stefan’s comments about IPython, it has changed the way I write Python code, and as such doctest is even more important. Fortunately IPython has a form of support for doctest. I think there are some improvements that could be made, but I find it very useful to go into doctest mode in IPython to try something out, then when I am done, paste that code into a test.
@Jason Pellerin
Thanks for the info, I’ll give a shot
I agree with your points. It would be good to be able to set global fixtures for doctest (not only with nose, but in doctest in general). It is very silly when you write functions that open and read files, and you have to repeat a StringIO or TemporaryFile import everytime. In general, I don’t understand why there is no way to declare some setup and teardown methods for doctests, while it would be the most useful.
You can enable ELLIPSIS = True for every test automatically by passing optionflags=doctest.ELLIPSIS to testmod(), but it is ugly.
ad: “I’d like to be able to easily jump into an interactive state from doctest.”:
I recently released interlude which is available at [pypi](http://pypi.python.org/pypi/interlude). See also my [blog post](http://bluedynamics.com/articles/jens/interlude-write-python-doctests-interactive).
In short: it allows to write…
… in your doctest and it jump into interactive mode. It feels like youre in the doctest and different from pdb. You can just copy paste from interactive mode into the test.