So, decorators are neat (maybe check out a new tutorial on them). Descriptors are neat but may seem hard (though they hardly take a long time to describe). Sometimes these two things intersect, and this post describes how.
Here’s an example decorator where this comes up. First, we want something that looks like a WSGI application:
def application(environ, start_response):
start_response('200 OK', [('content-type', 'text/html')])
return ['hi!']
But we want to use WebOb, like:
from webob import Request, Response
def application(environ, start_response):
req = Request(environ)
resp = Response('hi!')
return resp(environ, start_response)
(We don’t use req in this example, but of course you probably would in a real WSGI application)
Now req = Request(environ) is boilerplate, and it’d be nicer to just do return resp instead of return resp(environ, start_response). So let’s make a decorator to do that:
class wsgiapp(object):
def __init__(self, func):
self.func = func
def __call__(self, environ, start_response):
resp = self.func(Request(environ))
return resp(environ, start_response)
@wsgiapp
def application(req):
return Response('hi!')
If you don’t understand what happened
there, go read up on decorators.
Now, what if you want to decorate a method?
For instance:
class Application(object):
def __init__(self, text):
self.text = text
@wsgiapp
def __call__(self, req):
return Response(self.text)
application = Application('hi!')
This won’t quite work, because @wsgiapp will call Application.__call__(req) — with no self argument. This is generally a problem with any decorator that changes the signature, because the signature for methods has this extra self argument. Descriptors can handle this. First we’ll have the same wsgiapp definition as we had before, but we’ll add the magic descriptor method __get__:
class wsgiapp(object):
def __init__(self, func):
self.func = func
def __call__(self, environ, start_response):
resp = self.func(Request(environ))
return resp(environ, start_response)
def __get__(self, obj, type=None):
if obj is None:
return self
new_func = self.func.__get__(obj, type)
return self.__class__(new_func)
So, to explain:
When you get an attribute from an instance, like Application().__call__, Python will check if the object that was fetched has a __get__ method. If it does, it will call that method and use the result of that method.
This part:
if obj is None:
return self
is what happens when you do Application.__call__ — in other words, when get a class attribute. In that case obj (self) will be None, and it will just return the descriptor (it could do something else, like in this example, but usually it doesn’t).
Functions already have a __get__ method. You can try it yourself:
>>> def example(*args):
... print 'got', args
>>> example_bound = example.__get__(1)
>>> example_bound('test')
got (1, 'test')
So in the example with wsgiapp we are just changing the decorator to wrap the new bound function instead of the old unbound function. This allows wsgiapp to be compatible with both plain functions and methods. In fact, it would probably be preferable to always call func.__get__(obj, type) (even if obj is None), as then we could also wrap class methods or other kinds of descriptors.
Automatically generated list of related posts:
- WebOb decorator Lately I’ve been writing a few applications (e.g., PickyWiki and...
- Defaults & Inheritance I thought I’d note a way I try to make...
- What Does A WebOb App Look Like? Lately I’ve been writing code using WebOb and just a...
- Inverted Partials I was talking with a coworker some time ago about...
- Opening Python Classes So, I was reading through comments to despam my old...
Fascinating post Ian, thanks! Descriptors and decorators can be so helpful make code cleaner and more decoupled, it’s a shame they are perceived as so “difficult”. They do take a bit of effort to learn I guess, but it’s WELL worth it! [ed. note: correction applied, thanks]
Cheers, Ben
I’ve just had a play with some of this… It never occured to me but you can use this to make new methods after instantiation:
t.times2(4)
now acts like a normal method and gives8
.I don’t quite understand it all yet, but this solves an old problem I had in TurboGears where I was trying to abstract controllers. I would make a class Foo, and ran up against this:
I came up with an ugly hack making them all static methods, but with Ben’s deco class, I can now do this and it all works:
I think it should be
application = Application("some text")
instead of
application = Application
[fixed, thanks!]
In the first decorator (class wsgiapp), call takes two arguments other than self (environ, response). But the function it’s decorating, application, takes just one parameter. Is this right?
Very nice. It does seem like the
__get__
method of the decorator would get called on every call of the wrapped method though. Would there be any way of instantiating the bound method on first call and on all subsequent calls just using the previously instantiated method?Ale, I’ve tried everything to accomplish that and have not succeeded due to __get__ living on the class and not the instance. For example, you could do this:
This will re-bind the inner function to the obj instance and then remove the __get__ method, however you just monkey-patched your decorator — meaning you can only decorate one function. Not very useful, obviously. The reason this works is the reason you have to instantiate on every call: Descriptors operate on the class level.
To be honest, I think it makes more sense to decorate instance methods with functional rather than class-based decorators. For one, it does not have the problem of the unbound method failing to pass implicit self. Second of all, decorating is inherently a functional operation. Wrapping the whole thing in a class with __call__ defined is just ugly and confusing IMO. Then on top of that, overriding __get__ to re-bind the function is a hack; one that could have significant performance implications if called in a tight loop.
Hmm.. thought about this some more. You could always just use __get__ to stash the self object, and pass it explicitly in the wrapped function. I’m not sure if this performs better or not, but it’s at least easier to understand:
Your solution is a bit of a hack but I like it better. The problem with the __get __ method originally suggested is that it does “return self.__class __(new_func)”. This will cause a new instance of the class to be created again and again…
Indeed, but any solution is going to be a hack because it’s working around what is (IMO) a bug in Python class-based decorators.
I got around to benchmarking this and, surprisingly, creating a new class on each call is actually less expensive than the extra attribute access, even with __slots__ defined. I’m guessing this has to do with behind-the-scenes MRO stuff going on. The difference is very slight and probably isn’t a practical consideration, but just throwing that out there. Here’s the code and results (CPython 2.6 on single-core i386):
Chris says
From my point of view, very useful, since that means that if you transform it from a class to a function that returns a class, you are set.
ie
Seems to work well, even though it’s an uglyish syntax.
Using a class decorator makes it a little more succinct:
And further, increasing the magic quotient by creating a new class on the fly, we can make it shorter:
which can be used as following:
This last one looks fragile.
Yeah, I guess my point is you could just use a function instead of a class in the first place and not bother with any of this. This problem comes up every so often and it seems like people forget you can do that, because they get so invested in solving the bound/unbound method problem. When you think about it, adding __call__ to the class instance is just pretending it’s a function. Here’s how I would write the original decorator:
Does the same thing, no magic.
I suppose class-based deco’s are useful if your decorator needs to save state between calls or share it with other decorated functions, but there are other ways to do that, such as having the inner function close over a dict. IMO, decorators are a functional idiom and trying to force them to work as an object by introducing all this magic might not be the best practical solution, although it’s a fun exercise in python internal hacking.
Using the function you suggest will work for method decorating but not for function decorating. That was the point of using a descriptor and the reason to go with a class; it can handle both cases
You can kind of do this, but it’s difficult. Your descriptor would have to know the attribute it was bound to. So you might do:
Then you could do something like:
Or maybe separately:
I don’t cache these values in WebOb, but I do use a pattern of nested descriptors (
webob.converter
).If the decorated objects are functions (or methods), they have a
__name__
attribute, so you don’t need to pass explicitly the name to the descriptor. The cached_property class in Armin Ronacher’s Werkzeug uses it:If anyone is interested, here’s a set of decorators I’m using all the time: http://svn.pythonpaste.org/Paste/WebOb/contrib/decorators.py
@Schuyler: here’s why I think your use case for @deco is working.
You originally tried:
class Foo: def bar(self,x): return x*3
class Baz: #what some TG decorators do: foo = Foo() foo.bar.expose = True
which gave you an AttributeError: ‘instancemethod’ object has no attribute ‘expose’.
What’s going on here is that foo.bar is a bound method (isinstance(foo.bar, types.MethodType)) and neither they (nor unbound methods like Foo.bar) have dict‘s. Hence you can’t assign new attributes to them.
When you use the @deco technique instead:
class Foo: @deco def bar(self,x): return x*3
class Baz: foo = Foo() foo.bar.expose = True
now foo.bar is no longer a method. Instead, it’s an attribute of foo which has a deco object as its value. This object does have a dict and can be assigned arbitrary new attributes, like expose.