Ian Bicking: a blog

My old do-it-yourself framework tutorial was getting a bit long in the tooth, so I rewrote it to use WebOb. Now: the new do-it-yourself framework.

So I promised some more technical discussion of App Engine than my last two posts. Here it is:

Google App Engine uses a somewhat CGI-like model. That is, a script is run, and it uses stdin/stdout/environ to handle the requests. To avoid the overhead of CGI a process can be reused by defining __main__.main(). But while a process can be reused, it might not be, and of course it might get run on an entirely separate server. So in many ways it’s like the CGI model, with a small optimization so that, particularly under load, your requests can run with less latency.

This part is all well and good. I’ve already come to terms with servers going up and down without warning. But the environment itself has a number of other restrictions. It seems that App Engine is providing security in the language itself. The interpreter has been modified so that code is sandboxed, with no ability to write to the disk, open sockets, import C extensions, and see quite a few things in its environment. It’s these things that are a bit harder to come to terms with.

While they claim it supports any Python framework, these restrictions don’t actually make it easy. So for the last few days quite a few of us have been hacking various things to get stuff working.

The first thing people noticed is that Mako and Genshi didn’t work, because they use the ast (via the parser module) to handle the templating, and that module has been restricted. Apparently arbitrary bytecode is not safe in this environment, and so anything that can produce bytecode is considered dangerous. From what I understand Philip Jenvy has been working on Mako and the trunk is currently working. He’d already been doing work to get Mako working on Jython, which had similar issues. Genshi is also in progress and fairly close to working, though with some missing features. Genshi has the harder task as Mako was primarily reading the ast, while Genshi was writing it.

The first thing I noticed is that Setuptools doesn’t work. I’m flattered that one of the only 3 libraries included with App Engine is WebOb, but of course I am more enamored of a rich set of reusable libraries. Setuptools didn’t work because several modules and functions have been removed — this like os.open, os.uname, imp.acquire_lock, etc. Some of these are kind of reasonable, while others are not. The removal of many functions from imp doesn’t really make sense, for instance (I think the motivation was the difficulty of auditing the implementation of those functions, not that the functionality itself is dangerous). And while some functions can’t be used in the environment, the fact you can’t import those functions is more problematic. For instance, The Setuptools’ pkg_resources module has support to unzip eggs when they are imported. App Engine doesn’t support importing from zip files at all, and you certainly can’t unzip to a temporary location. But withoutthe necessary modules and objects pkg_resources won’t even import.

To work around this I started a new project: appengine-monkey, which adds several monkeypatches and replacement dummy modules to the environment to simulate a more typical environment. It’s just a small list so far (mostly in this module), but I expect as people experiment with other libraries the list will increase. For example, I would welcome implementations of things like httplib on top of urlfetch in this library. (Implementing httplib and stubbing out parts of socket would probably make urllib run.)

But the good news is that Pylons is pretty much working on App Engine, as is Setuptools and you can manage your libraries using virtualenv.

The instructions are all located in the appengine-monkey Pylons wiki page. Please leave comments if you have improvements or problems with that process. I also welcome contributors and developers to the project itself — this is a project for expediting App Engine development, it is not a project I care to champion or control. Or support to any large degree.

One ticket which is rather important is the apparent maximum number of files and blobs: 1000. Libraries involve lots of files, and the base Pylons install is only barely under this limit. Now I just wish I could use lxml, but that’s probably going to be a long time coming.

Update: As of April 2009 these issues were fixed; it took a year, but at least it’s done. The 1000 file limit has been relaxed (1000 code plus 1000 static) but still exists. lxml remains unlikely.

So, I was at a university the other day, talking with some people about a sprint project, and there was a student there. He was somewhat eager to write "algorithms". I’m not sure what that means exactly, but I was reminded of him because I was just about to write a function to make an absolute path relative and this somehow felt like an "algorithm". That’s not how I’d describe most of the coding I do.

But I digress… this student seemed to be excited about his ability. Well, okay, young programmers tend to be very eager, and often overestimate their ability (or the difficulty of the task). For students in this position I think it’s good to let them take on large projects, so they can get a better concept of their ability. This student seemed to take to heart the idea that a typical programmer produces 10 lines of working code a day. People who hear this are inclined to think "damn, I can write way more than 10 lines of code a day!"

I suspect a more fair way of thinking about this is that on average a programmer gets 50 lines of code into production a week, or 200 lines a month. Sure, lots of people are more productive than this, but the actual lines of code that go into production are usually way smaller than it feels like during the process that goes into getting that code there. That student had probably never gotten a single line of code into production. I’m pretty sure he wouldn’t have appreciated what that getting code into production is different than just hacking out some code.

The kid didn’t end up hanging around. I don’t think we had any algorithms for him. That’s too bad, maybe he could have seen what programming really looks like.

Another what-is-programming-really-like story… Emily was working on her blog and editing some templates to change the feed URL. It didn’t work, and I took a look at it and showed her that somehow a space had gotten into the URL, so she had to go back and correct them all. She found this frustrating. I pointed out to her that this is exactly what I spend my days doing.

Some people seem angered when people call writing HTML "programming". Personally I do not. Most programming is a lot more like writing HTML than it’s like writing a compiler. I bet writing a compiler is a lot more like writing HTML than what most programmers think it’s like to write a compiler.

I like this phrasing of the debate about App Engine’s role, from Doug Cutting: Cloud: commodity or proprietary? (via). (Well, I like it more than the sharecropping phrasing referenced in my last post.) I guess I’m excited about this because like Doug I do want a "cloud" of sorts, and this is a move towards that in a way that makes sense to me. Maybe to state my motivations more clearly: I hate computers. I really hate them a lot. I dream of some world of Platonic ideals where software just exists, and existence that state it works. App Engine feels like a strong move in the direction of computers-not-mattering. What does App Engine run on? I don’t care! Where is the server located? I don’t care! What is BigTable? I am comfortable thinking of it only in its abstract sense, an API that works, and I don’t know how, nor do I need to know how. I don’t need to know these things if they just work. Always. Totally reliably. I’m not shy about digging into code. I tend to be light on my reading of documentation, because I’d much rather open up source code and poke around. But when something can work so reliably that I can treat it as completely opaque then it’s a blessing, because I can start to forget about it and think about bigger goals.

There was a time when people were concerned about Big Endian vs. Little Endian in computers. You had to think about this sort of little detail when programming. People formed actual opinions on which way was best. To think! Similarly XML has removed a large number of fairly pointless format decisions people might make. There is progress. Commodity hosting (reliable, consistent hosting, better than what we have now) feels like similar progress.

Unlike Doug I’m optimistic that App Engine is a move in the direction of a commodity cloud. The APIs seem to lack the stench of proprietary APIs. They are based on Google services, but they reasonably abstract and reasonably minimal. This does not seem like some kind of "play" (and the developers’ seem to be reassuring about their intentions). There’s a tendency to be cynical about any company’s work, that it has underlying intentions that are at odds with any competitor (present or future), that anything good is just a loss leader meant to hook you in so they can squeeze you later. Some companies deserve such cynicism. I don’t know that this company, or this team, deserves that.

Mind you, I don’t say this from a Best-Tool-For-The-Job perspective. I believe in the moral foundations of Free Software, not just the technical advantages of its development process. But I’m a Utilitarian, and it doesn’t make me uncomfortable that not everything is Free if I think it’s a step forward for overall freedom. I think App Engine has the potential to be a very powerful tool for enabling people to create and use web applications. If it was great, but still a dead end, then maybe that wouldn’t be good enough. But I don’t think this is a dead end.

Update: Indeed people are reimplementing the interface: see the appdrop.com announcement

This is about Google App Engine which probably everyone has read about already.

I’m quite excited about it. Hosting has been the bane of the Python web world for a long time. This provides a very compelling hosting situation for Python applications.

I’m not as interested in this from a competitive perspective as I am from a simple this-is-awesome perspective. Regardless of how this positions Python relative to other languages, this is something Python needs. But even looking beyond that, I think this is something the open source world needs. Open source web development is in a funny place. There’s a lot of reasons why web programming is a good domain for open source. The barrier to entry for web development is extremely low. Developers have choice in their tools, as browsers don’t really care what software you use so long as you serve up HTML. This leads to experimentation and excitement and the kind of self-direction that is very motivating to developers. It leads to the kind of personal excitement that underlies most open source development.

Despite this, open source web application development doesn’t seem sustainable. There’s some applications, sure. WordPress, Trac, MediaWiki, MoinMoin. But most wiki software doesn’t have a vibrant community. Many a bug tracker has fallen by the wayside. Blog software projects have a horrible time building a viable community. Other website software hardly gets anywhere at all. A lot of the development that might appear to be application development really is more like a framework when you look closely (e.g., Plone, Drupal).

I think deployment concerns are a huge part of this. And, given its better deployment story, it’s no surprise PHP is the basis of most viable open source web applications. Being interested in a project requires that you be able to use the product (and usually use it casually, as that’s the point of entry for many developers). Right now most people can’t use open source web applications.

But people can use hosted applications, and that’s where all the effort has gone in the past few years. I am comfortable saying that Trac is a better issue tracker than Google Code’s issue tracker. But I’d probably recommend Google Code to someone starting a new project, because it’s so much less work. Similarly I’d try to dissuade most people from installing their own blog software. I still don’t know what to tell people about a CMS.

Many people are excited about how far up you might be able to scale something based on App Engine, but (like Dave) I’m excited about how far it could be scaled down. For the majority of sites the free quota will be more than enough. But that alone isn’t the point: there’s lots of free services people can use. The difference here is that the free services can be modified and controlled by the anyone who signs up and installs an application.

From the perspective of open source it’s a bit awkward that the platform itself is proprietary. Questions about sharecropping are a valid concern, but I’m optimistic about the ultimate outcome. The SDK is under a permissive open source license, and the APIs are all reasonable enough that they could be reimplemented with open source backends (maybe without the same scalability, but that’s not the aspect I care about anyway). Perhaps the BigTable APIs will serve as the basis for future storage APIs.

But even if other people make compatible implementations of these APIs, would it matter? If Google offers free hosting, is someone else really going to be able to provide a better hosting option? Or would these other implementations just be strawmen, a way to show that It Could Be Done? If the libraries are just written to prove a point, I can’t see them gaining much traction. But I think these could be viable as there are other constraints to the App Engine environment that people may want to escape at some point in their application development.

As to the details of App Engine? Can you run Pylons or Paste on it? Well, that’s a topic for another post.

Update: I wrote up some more thoughts

I just saw this json-rpc recipe go by as a popular link on del.icio.us. It’s yet-another-*Server based recipe (BaseHTTPServer, XMLRPCServer, etc). I don’t know why people keep writing these. WSGI is in all ways easier, clearer, and more useful.

So I figured I’d give it a go myself, using WebOb. Then I figured it might make a good document, and I annotated it and turned it into a tutorial. It’s also an example of using WebOb as a client library.

I’ve added several tutorials in the past months, which I thought I should also point out. The wiki example is fairly pedestrian, but shows how to do typical application-style development with WebOb. A more interesting example is the comment middleware example, which shows how much easier it can be to write middleware using WebOb than traditional WSGI middleware.

I think WebOb’s particular strong points are with middleware (where it makes middleware vastly easier to write) and web services of various kinds (RESTful or not).