Lately I’ve been writing code using WebOb and just a few other small libraries. It’s not entirely obvious what this looks like, so I thought I’d give a simple example.
I make each application a class. Instances of the class are "configured applications". So it looks a little like this (for an application that takes one configuration parameter, file_path):
class Application(object):
def __init__(self, file_path):
self.file_path = file_path
Then the app needs to be a WSGI app, because that’s how I roll. I use webob.dec:
from webob.dec import wsgify
from webob import exc
from webob import Response
class Application(object):
def __init__(self, file_path):
self.file_path = file_path
@wsgify
def __call__(self, req):
return Response('Hi!')
Somewhere separate from the application you actually instantiate Application. You can use Paste Deploy for that, configure it yourself, or just do something ad hoc (a lot of mod_wsgi .wsgi files are like this, basically).
I use webob.exc for things like exc.HTTPNotFound(). You can raise that as an exception, but I mostly just return the object (to the same effect).
Now you have Hello World. I then sometimes do something terrible, I start handling URLs like this:
@wsgify
def __call__(self, req):
if req.path_info == '/':
return self.index(req)
elif req.path_info.startswith('/view/'):
return self.view(req)
return exc.HTTPNotFound()
This is lazy and a very bad idea. So you want a dispatcher. There are several (e.g., selector). I’ll use Routes here… the latest release makes it a bit easier (though it could still be streamlined a bit). Here’s a pattern I think makes sense:
from routes import Mapper
class Application(object):
map = Mapper()
map.connect('index', '/', method='index')
map.connect('view', '/view/{item}', method='view')
def __init__(self, file_path):
self.file_path = file_path
@wsgify
def __call__(self, req):
results = self.map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(self.map, req.environ)
req.urlvars = ((), match)
kwargs = match.copy()
method = kwargs.pop('method')
req.link = link
return getattr(self, method)(req, **kwargs)
def index(self, req):
...
def view(self, req, item):
...
Another way you might do it is to skip the class, which means skipping a clear place for configuration. I don’t like that, but if you don’t care about that, then it looks like this:
def index(self, req):
...
def view(self, req, item):
...
map = Mapper()
map.connect('index', '/', view=index)
map.connect('view', '/view/{item}', view=view)
@wsgify
def application(req):
results = map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(map, req.environ)
req.urlvars = ((), match)
kwargs = match.copy()
view = kwargs.pop('view')
req.link = link
return view(req, **kwargs)
Then application is pretty much boilerplate. You could put configuration in the request if you wanted, or use some other technique (like Contextual).
I talked some with Ben Bangert about what he’s trying with these patterns, and he’s doing something reminiscent of Pylons controllers (but without the rest of Pylons) and it looks more like this (with my own adaptations):
class BaseController(object):
special_vars = ['controller', 'action']
def __init__(self, request, link, **config):
self.request = request
self.link = link
for name, value in config.items():
setattr(self, name, value)
def __call__(self):
action = self.request.urlvars.get('action', 'index')
if hasattr(self, '__before__'):
self.__before__()
kwargs = req.urlsvars.copy()
for attr in self.special_vars
if attr in kwargs:
del kwargs[attr]
return getattr(self, action)(**kwargs)
class Index(BaseController):
def index(self):
...
def view(self, item):
...
class Application(object):
map = Mapper()
map.connect('index', '/', controller=Index)
map.connect('view', '/view/{item}', controller=Index, action='view')
def __init__(self, **config):
self.config = config
@wsgify
def __call__(self, req):
results = self.map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(self.map, req.environ)
req.urlvars = ((), match)
controller = match['controller'](req, link, **self.config)
return controller()
That’s a lot of code blocks, but they all really say the same thing ;) I think writing apps with almost-no-framework like this is pretty doable, so if you have something small you should give it a go. I think it’s especially appropriate for applications that are an API (not a "web site").
Automatically generated list of related posts:
- WebOb decorator Lately I’ve been writing a few applications (e.g., PickyWiki and...
- WebOb I’ve have it in my head to extract/rewrite parts of...
- JSON-RPC WebOb Example I just saw this json-rpc recipe go by as a...
- App Engine and Pylons So I promised some more technical discussion of App Engine...
- App Engine and Open Source This is about Google App Engine which probably everyone has...
See also: pasteob.Autodispatch
This seems like the start of CherryPy dispatching, which was later know as Object Dispatch and popularized by TurboGears. I both love and hate it. It’s awesome to give structure to your code and app eliminating that pesky routes configuration, however it’s bad when you need to get outside of the norm and you have to add several workarounds for the edge cases, take a look at dispatch and other * methods in TurboGears2.1 codebase.
It is somewhat similar to CherryPy, but there are important differences. First of all everything is WSGI, no query-as-arguments and stuff. Second, the names are picked in such a way that there’s no need for @expose. It also handles edge cases very well.
I know the other solutions and their shortcomings, so you might find that Autodispatch finally gets it right.
I’ve never quite been able to get my head around these “almost-no-framework” things. Why not go all the way and just be pretty much no framework? What are WebOb and things like it providing here that isn’t obfuscation above a simple set of callables doing
(environ, start_response)
dispatched by selector or Routes? To me, doing it that way keeps the mechanics of the “web” stuff transparent and highly visible, which I’ve found quite critical in maintaining good HTTP hygiene. I want to be able to see.My own particular accretion of bits of code started from what I learned from Joe Gregorio’s [Robaccia](http://bitworking.org/news/WhysomanyPythonweb_frameworks) and has evolved into a bunch of pieces of WSGI I use here and there as needed, like lego.
I think the best case for the “almost no framework” is when you’re clear what the small helper abstractions are giving you and doing for you. I kind of envision it as satisfying the initial impulse that leads most people to build a web framework in the first place (getting rid of the same repetitive, error-prone tasks you have to do on every request) but avoids the “It does everything for you poof it’s MAGIC” behavior that almost all frameworks seem to evolve into. You’re right it takes some of the web “stuff” out of view, but as long as you know what that “stuff” is, you can always hack it when you need to.
WebOb request / response objects represent HTTP requests / responses much closer than raw WSGI does.
Two things
1- removing repetition for example in this case you will get tired of creating Response objects on every return so you will either create a default response you clone and return a string (this is what pylons did) or make up some marker to determine how to create it (this is what TurboGears did with the @expose decorator + returning a dict)
2- Raw WSGI is hard to maintain not to mention very unfriendly to object oriented programming, reading from the environ and checking if the key is valid gets old quick which is why webob was born.
In other words as others have pointed out the framework grows out of the necessity of eliminating repetition which pulls in constraints on design and style. IMO that is the trade off and I’ll salute everyone that will be able to abstract repetition without adding constraints.
Seems like an interesting approach for quick, one-off applications (or as you say, APIs).
I’ve been writing some web applications with a very thin custom web framework I put together. I was inspired by (and used many of the ideas in) your “Do-It-Yourself Framework” articles. It ended up using WSGI, WebOb, Routes, Mako, and my own non-orm (not a fan of ORMs) db layer as the foundation pieces. It’s lighter weight than Pylons and requires a lot less configuration with the obvious trade off being that it’s less flexible (only works with MySQL or the filesystem for the storage etc…).
I wanted to say thanks for those articles and all of your contributions!
I’ve written a few mini-apps (e.g. restview, which you can find on PyPI) like this but using only Python’s stdlib. It was rather painful, but then “only depends on Python’s stdlib” is a compelling feature sometimes.
I’ll go the other way from Chris Dent on this and ask if you’re going to do this, why not just use a framework? Above, you’re just typing code that someone has already typed up in a module in some existing framework. And without tests.
This is a rhetorical question. I know the right answers to this from first-person experience: “because if I type it, I’ll understand it” and “I wont need anybody’s permission to change it later”. There are the counterarguments: understanding other people’s code is a valuable skill to have; working with others is also valuable.
But in any case, I don’t think “no framework” is ever the right goal. “Best fit for my brain” is almost always the right goal, though, so if they are one in the same, great.
This speaks my mind. In all aspects.
I’d generally agree. Though there’s some other points:
By using the least amount of infrastructure that you need, the code is more navigable for everyone. This is actually a substantial part of why I don’t use Pylons for these small projects, I just find the file layout unnecessarily deep (
paster create -t pylons_minimal
helps, but I like going even smaller).Components have different levels of stability. There’s simple code stability (a lack of bugs), but even more for me is the issue of conceptual stability: is this a good factoring of the domain, does it address the important problems, does it fully encapsulate its complexity, is it sufficiently adaptable? There are components I feel fairly confident about (e.g., WebOb, Routes or another simple dispatching library, most template libraries, etc). Most frameworks incorporate things that make me uncomfortable.
Some problems just have uncomfortable aspects, but I’m a lot more comfortable if those parts are ad hoc and per-application.
This is not completely defeatist, as I also believe this library-of-things-I-am-comfortable-with increases over time.
I believe there is a genuine benefit to avoiding embedding control structures in frameworks and libraries. For instance, I think there is a benefit in doing a route match with a library but doing the actual request routing in my application code.
I don’t think I’d say “best fit for my brain” is how I’d qualify it. Maybe one general caveat is that I want to code for me-right-now and me-of-the-future; I’ve made many mistakes by forgetting me-of-the-future. But I would also use this pattern in a collaborative environment, if I thought those other advantages were compelling; i.e., I’d use this pattern (sometimes) even when the code is intended for unknown future developers.
The counterargument is kind of [Greenspun's Rule](http://en.wikipedia.org/wiki/Greenspun'sTenthRule): “Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.” Or, “Any sufficiently complicated no-framework application contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a web framework.” Except… the scales aren’t really that large; half of Pylons just isn’t an intimidating size. I think that’s because we’ve been moving up to higher levels, both at the language level and the tool level.
I’m not surprised that I’m sort of alone on this “no-framework” notion. It’s not that I’m anti-DRY or something like that, but rather that I think of code as human communication more than computer communication, and beyond it being human communication I think of it as an educational tool. When I look at a web application and all the web “stuff” has been abstracted away (including, in response to Sergey, how the server is delivering my application information) the learning opportunity is constrained to a rather small domain. My future me is incredibly stupid, and I’m pretty sure it’s neither offensive nor incorrect to claim that everyone else’s is too.
I suppose in many cases this limited learning domain ought to be fine, it’s that small domain where the action is supposed to be. However, this leaves armies of developers with no idea how the web works, and importantly how it could work if they could see some of the workings. Obviously there’s a limit. I’m not saying we should all be pushing things into registers, but where we choose to draw the boundary is an indicator of what we think matters, what we think needs to be seen.
FWIW, the
__str__
methods of webob.Request and Response give the HTTP request and response pretty much how it goes over the wire.I’m honestly kind of confused by this. There may well be “armies of developers with no idea how the web works,” but I don’t see how it’s due to WebOb, or Pylons, or Django. When I write, say, Django views, I work with a Request object and a Response object. They have a status code, and headers I can inspect and manipulate, and body content. Each time a request comes in, I send back a response. If that isn’t pretty much “how the web works,” I’m missing something. The exact details of the raw WSGI API might be useful to know in some circumstances, but describing them as “how the web works” seems over the top.
This vast collection of comments has helped me to work out part of what my issues are. I’m not attempting to make any claim to correctness on my part, more just exploring. Firstly I think I have an unfortunate tendency to have a knee jerk negative reaction to seeing objects where I wouldn’t want them to be. They make grand sense in all kinds of places, but I do not like them as the first visible manifestation of the handler of a web request: too much tendency towards action at a distance and various buried magic. I prefer something much closer to properly functional because in my mental model (what “fits my brain”) a web request is functional (not strictly so, there are side effects which show up in the persistence layer but for the most part I want them as state free and idempotent as possible). In that sense, a bare WSGI callable is really quite nice. That leads nicely to the second point: In Ian’s second code block (prefixed with “Then the app needs to be a WSGI app”) at first glance I see no WSGI app. I see something which under the covers is a wsgi app, being made so by “other stuff”.
I have no objection to there being an object which encapsulates an HTTP request or response nor extensive use of libraries, what is the problem is the WSGI stack being obscured. Establish the request object further up the stack and hork it into
environ
and let me pull it out later if I want, but don’t change my application signatures. Those signatures help to ensure that I’m creating what amounts to a stackable series of independent tiny little web apps (even if to the outside world they look like just one). Creating using that model helps to ensure that my codebase is deliciously extensible and scalable.@Chris, The wsgify-decorated functions/methods have dual-signatures and they are actually WSGI apps, but the fact it’s not obvious is bad enough. I voiced this concern some time ago, so you might want to check out the discussion: * http://groups.google.com/group/paste-users/browsethread/thread/7346e75940413f46 * http://groups.google.com/group/paste-users/browsethread/thread/aae70301addad062
Yeah as you intuited via the Greenspun comment I’m pretty sure that any particular brand of no-framework environment ultimately ends up as a framework. It’s the no-framework-framework created by the whomever wrote its first hundred lines of code. Even the 30-50 lines of code you entered above has contracts hidden in it: to generate a URL from within a view, you need to know enough to call route_url; views must be defined with the kwargs defined in the route, and so forth. Frameworks are all about contracts like this. I think the real differences between the code you’ve created above and any other framework are these:
You’ve not committed to any contracts (there are no documented contracts, and you call always call “implementation detail” if someone complains due to a change you made). This is the moral equivalent of “I dont need to ask anyone if I can change this”.
The notional contracts that do exist are maybe fewer, so you can keep them in your head all at once. This is the moral equivalent of “I typed it all so I understand it all”.
So yeah. I agree with Greenspun. If you’re a lone developer on a web project, and you will always be the lone developer on that project, maybe creating-the-mainloop-from-scratch on that project makes sense if you’re the most comfortable with that solution. That’s cool. But if you’re trying to collaborate with one or more people on a project, though, you always need some contracts, because collaboration at some level just requires them as a replacement for a personal mind-meld. And once there’s a contract or two, I think you become the proud new owner of a framework.
Not to discount legitimate technical arguments one way or the other, but it’s striking to me how many of these preferences seem explainable by which abstraction level you cut your teeth on. When I got into Python web dev, Django was well-recommended and had good docs, so I tried it first. I had success, it fit my brain, and I learned its internals to the point where not much that it does feels magical; I know where to find the relevant code, and how to bend it to my needs. Sure, there are chunks of it (and even parts of the driving philosophy) that I don’t like very much, but the value outweighs the discomfort; the idea of reinventing in my own code pieces of Django that are well-tested, widely used, and have worked well for me in the past doesn’t hold a lot of appeal. I cut my teeth working on top of Django, not on alternatives to it.
And then there’s Pinax, another frameworky thing one level higher on the abstraction stack, which also addresses problems I face. And I find I’m quite resistant to using it (still haven’t for anything serious) because I already worked on those problems myself, and built solutions that I like. I don’t want to abandon my solutions in favor of someone else’s conception. I’m willing to use other people’s code at Pinax’ level of abstraction, but I’m much more picky: it has to be tightly focussed, clearly correctly factored, library-like rather than framework-like (so I can fit it into my personal no-framework framework); all the things that sound pretty similar to Ian’s comments about using Routes or WebOb.
Grizzled veterans of C don’t like having Python manage their memory for them, either…
On the other hand, maybe this just means that I need to take a couple months and dive head-long into writing my own web framework, to broaden my awareness of the design choices made. Probably so. Python hackers ought to learn C memory management, too.
I hope we all agree on at least one thing: everyone should make their own framework! Whether even that person should use their framework, eh…
Sigh. Well, you can at least be sure it’ll use WebOb.
(doesn’t want to get called out on Twitter by Ian for writing another set of non-WebOb request/response objects)
Thanks for the clear exposition of the techniques I’ve been copping off of you since FlatAtomPub!
Two thoughts:
You mention Contextual. If you feel so inclined it would be illuminating to see how you would handle the boilerplate using Contextual.
I like the examination of the least possible or “goldilocks” framework approach as it inevitably leads to matching the purpose of the codebase to the product (the application) rather than fitting an application to an existing framework. And that thing mcdonc said about fitting your head.
Though chris R will curse me for mentioning it, his “nothing” framework is a really nice example of the nok framework principle: http://www.bitbucket.org/chrisrossi/happy/overview/ . As he describes it, it’s all about removing inversion of control. The most obvious effect of this is the increase amount of code in the area of application boilerplate. OTOH, auditing what happens is extremely easy: http://www.bitbucket.org/chrisrossi/edwin/src/tip/src/edwin/edwin/application.py .
The use of Contextual is more speculative, I haven’t really tried it. I think it might look like this:
Then you could configure an application like this:
Though I find Contextual perhaps more clever than is necessary. I wish it just looked like this:
With all the other pieces the same, and with no other fanciness (like Contextual’s Services, or the fancy
<<=
assignment). Also perhaps the ability to associate values with strings instead of only concrete objects. But it deserves some exploration before settling on anything.I think both Happy and PasteOb (and a few other things I’ve heard about) represent a kind of continuation of a big-bag-of-tricks which Paste core represented for a long time (but which is long in the tooth, so I’m glad to see other people looking at these tasks again with a fresh eye).
Here’s a real example based on contextual-fork and some proprietary modules:
PasteOb has similar hooks for webauth, which is something I plan to describe in [paste-users](http://groups.google.com/group/paste-users) sometime later.
What’s your opinion on the microframeworks, like [bottle](http://bottle.paws.de/)? It’s seems these fit the bill for very quick prototyping for an api or backend interface. I have used bottle a few times now, with great success. Yes, its a framework, but it doesn’t really have any dependencies and its easy to modify for the specific application if needed. On top of that, its smoking fast.
Bottle fits into the “why don’t you use webob” problem. IMO it’s a great tool but why implement all the parsing yet one more time.
For a website I generally would not promote this style of microframework. Bottle seems to be focused on smaller things. I am not anti-library… Bottle eschews all libraries, which to me makes it uninteresting. I might steal some ideas from it if I thought there were ideas worth stealing. I’m not aware of any. I don’t like the decorators for routes technique as it relies on import side-effects (the routes get registered when the decorator is executed). The thread-local request object is bad stuff.
Sorry to jump late into the party but I have to say I both agree and disagree with this approach. First of all the moment I discovered webob.wsgify I was like ohhh that is what I have been wanting for quite some time and I was stuck at the problem regarding routing (btw why is the if/else approach naive? is it because of routes optimization or I’m missing something?) in fact I find python based routing to be a good thing! because then you are not forced to route on URL, and you can do interesting things like A/B testing or just use completely different code paths depending on the type of user. So I guess in this particular case this fits my brain. I also agree that it could become bad fast for “big applications” but then the problem could be framed in “do we really need big applications”?
That said this leads us down the route of having to reinvent everything as it is not possible to reuse anything for this “framework” as there is nothing written in it except for what you wrote and IMO that is very bad, mainly because there is always someone out there that could and in fact already wrote a better solution to this particular problem.
That brings me to another plus point which is code reuse, in this no-framework environment reusing code is actually the best part of it as it is all wsgi and since you are using the least amount of code it means you will be compatible with others and (at least in theory) you should be able to plug in code written in other frameworks into your own system.
Last but not least thank you for putting words to what I have been playing around with. I have been doing something like this for two small apps playing around with this idea, and so far I like what I have. I have been slowly getting converted into the philosophy of libraries > frameworks and the ability to plug-n-play with component to create that dream of a “Best of” framework.
As for the argument regarding “developers that don’t know better” I think we have to be pragmatic here, you should (at different points in time) both be the super wise guy that dives into the HTTP headers while at others you just want to have a method be called and return a string.
Anyway sorry for the long comment and keep up the good work recently your post have been enlightening filling peaces of the puzzle
Re: routing: I’ve done if/else routing and found it very hard to maintain. It’s like when I’ve generated HTML without a template: in some circumstances it might seem okay or even a good idea at first, but I always am annoyed with my choice later. The one exception in my experience is something like a CMS where the routing itself is very application-specific and data-driven. In my example it would still be easy to do A/B testing, as the routing just returns a match but doesn’t invoke it, so it’s clear where you’d put different logic in.
I’d be curious what people think: if you were making an application, what’s the next thing you’d look to a library for? Templates are a no-brainer. I’d add an optional constraint: barring things that abstract out logic and conditionals (e.g.,, what
Application
does), what library (or kind of library) would you want to use?I’d love to see a routing system w/ a similar API to that described here (but Python, not PHP):
http://www.google.com/buzz/pjkeane/CWyFG8ydaoT/URL-Dispatching-made-easy-Ive-not-yet-found-a-URL