Ian Bicking: the old part of his blog

Python, metaprogramming, and macros

I was looking at Phillip's CLOS-style generic functions recently, and it got me thinking about metaprogramming in Python. If you haven't looked at those generic functions, it's a very neat idea and implementation, and can be expressed quite elegantly in Python (at least after adding 2.4's @ decorator syntax). (This post is another recent item that got me thinking about metaprogramming.)

Python doesn't have much of anything like macros (which CLOS generic methods are implemented with, I believe), but it doesn't seem like a big impediment. And there's a lot of metaprogramming that is done in Python, without a macro in sight. By metaprogramming I mean programming structures that operate on other programming structures. Every decorator is a case of metaprogramming; the decorator is a function that takes a function as an argument, and returns a function (or function-like) object.

There's a lot of ways you can do metaprogramming in Python. Decorators are one way, and metaclasses are another. But it can be easier than that; you can modify classes on the fly, you can you can do all sorts of things with higher-order functions (i.e., functions that take functions as arguments), and even do funny sorts of things with import hooks, the new module, etc.

What makes this possible is the fact Python is a scripting language. I know some people hate that term; but I think it means something very specific, not merely derogatory. A Python module is a script; when Python loads a module, it reads the file, evaluates the first line, the second line, etc., until it's done. Functions aren't declared; rather def creates a function on the spot; you can redefine the function later, or define it inside another scope, or delete it, or whatever. This causes problems (like the problem of changing code in long-running processes, or circular references); but it also has some advantages.

When we use a module, we can forget that we aren't using the source we see in its .py file. We're using the objects that are created by that source. When loading a module, we're running a bunch of statements; some statements are rather benign with few side effects (mostly def); others are somewhere in between (like class); and others are executed immediately, and we only have access to their effects. Decorators and metaclasses are two instances; but you can put any statements in a module, and any of them can modify what's in the module. You can even modify modules after they are loaded (a monkey patch).

There are some problems. You can't look inside code programmitically (or at least it isn't easy), and lazy evaluation is fairly limited. The generics package is forced to use strings for its expressions, then compile and execute them in a closed environment. SQLObject uses SQLBuilder which uses Python's magic methods, but is limited; Numeric uses something similar. Still, there's a huge amount you can do, and all without any macros, all simply due to the highly imperative nature of Python source.

Created 21 Nov '04
Modified 14 Dec '04