We can answer on question what is VPS? and what is cheap dedicated servers?

Why toppcloud (Silver Lining) will not be agnostic

I haven’t received a great deal of specific feedback on toppcloud (update: renamed Silver Lining), only a few people (Ben Bangert, Jorge Vargas) seem to have really dived deeply into it. But — and this is not unexpected — I have already gotten several requests about making it more agnostic with respect to… stuff. Maybe that it not (at least forever) require Ubuntu. Or maybe that it should support different process models (e.g., threaded and multiple processes). Or other versions of Python.

The more I think about it, and the more I work with the tool, the more confident I am that toppcloud should not be agnostic on these issues. This is not so much about an "opinionated" tool; toppcloud is not actually very opinionated. It’s about a well-understood system.

For instance, Ben noticed a problem recently with weird import errors. I don’t know quite why mod_wsgi has this particular problem (when other WSGI servers that I’ve used haven’t), but the fix isn’t that hard. So Ben committed a fix and the problem went away.

Personally I think this is a bug with mod_wsgi. Maybe it’s also a Python bug. But it doesn’t really matter. When a bug exists it "belongs" to everyone who encounters it.

toppcloud is not intended to be a transparent system. When it’s working correctly, you should be able to ignore most of the system and concentrate on the relatively simple abstractions given to your application. So if the configuration reveals this particular bug in Python/mod_wsgi, then the bug is essentially a toppcloud bug, and toppcloud should (and can) fix it.

A more flexible system can ignore such problems as being "somewhere else" in the system. Or, if you don’t define these problems as someone else’s problem, then a more flexible system is essentially always broken somewhere; there is always some untested combination, some new component, or some old component that might get pushed into the mix. Fixes for one person’s problem may introduce a new problem in someone else’s stack. Some fixes aren’t even clear. toppcloud has Varnish in place, so it’s quite clear where a fix related to Django and Varnish configuration goes. If these were each components developed by different people at different times (like with buildout recipes) then fixing something like this could get complicated.

So I feel very resolved: toppcloud will hardcode everything it possibly can. Python 2.6 and only 2.6! (Until 2.7, but then only 2.7!). Only Varnish/Apache/mod_wsgi. I haven’t figured out threads/processes exactly, but once I do, there will be only one way! And if I get it wrong, then everyone (everyone) will have to switch when it is corrected! Because I’d much rather have a system that is inflexible than one that doesn’t work. With a clear and solid design I think it is feasible to get this to work, and that is no small feat.

Relatedly, I think I’m going to change the name of toppcloud, so ideas are welcome!

2010 02 10

Programming
Python
Silver Lining
Web

Comments (23)

Permalink

Leaving TOPP

After three and a half years at The Open Planning Project, my time there is done.

For a while TOPP has been trying to find itself, to determine what it is that it can do best, and how to do that. I think TOPP has finally started really figuring that out, focusing on civic participation, revisiting "planning", and though it’s not fully developed there seems to be a strong potential for TOPP to serve as a point of collaboration for other more ad hoc open source government efforts: government workers and volunteers can provide substantial and well-informed development efforts, but the long-term shepherding of a project is difficult, and TOPP has the potential to provide that kind of long-term neutral guidance. At the same time, communities of people have been developing around these issues; people have gotten past simply calling for government to be inclusive or transparent and have started to do the real work of making that happen.

But… unfortunately I won’t be able to figure out their next steps with them. Mark Gorton has been very generous in his support of TOPP, and helped us get started. But while he has been patient, and even at times seemingly immune from the economic trends… well, he’s not immune, and he has had to cut back his support for TOPP before we were able to become self-sufficient. And so there have been layoffs, myself among them.

I suspect what I’ll do next will have a very different focus. This feels a bit weird, a kind of lost identity. Overall I’m feeling pretty optimistic about finding something new and interesting to do, but I’m still a bit melancholy about leaving things behind. That TOPP and the people I’ve worked with are far away in New York makes it feel like more of a loss because I don’t know when I’ll be back next. But onward and upward! Now to find out what is next…

2010 02 09

Non-technical

Comments (21)

Permalink

Weave: valuable client-side data

I’ve been looking at Weave some lately. The large-print summary on the page is Synchronize Your Firefox Experience Across Desktop and Mobile. Straight forward enough.

Years and years ago I stopped really using bookmarks. You lose them moving from machine to machine (which Weave could help), but mostly I stopped using them because it was too hard to (a) identify interesting content and place it into a taxonomy and (b) predict what I would later be interested in. If I wanted to refer to something I’d seen before there’s a good chance I wouldn’t have saved it, while my bookmarks would be flooded with things that time would show were of transient interest.

So… synchronizing bookmarks, eh. Saved form data and logins? Sure, that’s handy. It would make browsing on multiple machines nicer. But it feels more like a handy tweak.

All my really useful data is kept on servers, categorized and protected by a user account. Why is that? Well, of course, where else would you keep it? In cookies? Ha!

Why not in cookies? So many reasons… because cookies are opaque and can’t hold much data, can’t be exchanged, and probably worst of all they just disappear randomly.

What if cookies weren’t so impossibly lame for holding important data? Suddenly sync seems much more interesting. Instead of storing documents and data on a website, the website could put all that data right into your browser. And conveniently HTML 5 has an API for that. Everyone thinks about that API as a way of handling off-line caching, because while it handles many problems with cookies it doesn’t handle the problem of data disappearing as you move between computers and browser. That’s where Weave synchronization could change things. I don’t think this technique is something appropriate for every app (maybe not most apps), but it could allow a new class of applications.

Advantages: web development and scaling becomes easy. If you store data in the browser scaling is almost free; serving static pages is a Solved Problem. Development is easier because development and deployment of HTML and Javascript is pretty easy. Forking is easy — just copy all the resources. So long as you don’t hardcode absolute links into your Javascript, you can even just save from the browser and get a working local copy of the application.

Disadvantages: another silo. You might complain about Facebook keeping everyone’s data, but the data in Facebook is still more transparent than data held in files or locally with a browser. Let’s say you create a word processor that uses local storage for all its documents. If you stored that document online sharing and collaboration would be really easy; but with it stored locally the act of sharing is not as automatic, and collaboration is very hard. Sure, the "user" is in "control" of their data, but that would be more true on paper than in practice. Building collaboration on top of local storage is hard, and without that… maybe it’s not that interesting?

Anyway, there is an interesting (but maybe Much Too Hard) problem in there. (DVCS in the browser?)

Update: in this video Aza talks about just what I talk about here. A few months ago. The Weave APIs also allude to things like this, including collaboration. So… they are on it!

2010 02 06

HTML
Web

Comments (2)

Permalink

toppcloud (Silver Lining) and Django

I wrote up instructions on using toppcloud (update: renamed Silver Lining) with Django. They are up on the site (where they will be updated in the future), but I’ll drop them here too…

Creating a Layout

First thing you have to do (after installing toppcloud of course) is create an environment for your new application. Do that like:


$ toppcloud init sampleapp
 

This creates a directory sampleapp/ with a basic layout. The first thing we’ll do is set up version control for our project. For the sake of documentation, imagine you go to bitbucket and create two new repositories, one called sampleapp and another called sampleapp-lib (and for the examples we’ll use the username USER).

We’ll go into our new environment and use these:


$ cd sampleapp
$ hg clone http://bitbucket.org/USER/sampleapp src/sampleapp
$ rm -r lib/python/
$ hg clone http://bitbucket.org/USER/sampleapp-lib lib/python
$ mkdir lib/python/bin/
$ echo "syntax: glob
bin/python*
bin/activate
bin/activate_this.py
bin/pip
bin/easy_install*
" > lib/python/.hgignore
$ mv bin/* lib/python/bin/
$ rmdir bin/
$ ln -s lib/python/bin bin
 

Now there is a basic layout setup, with all your libraries going into the sampleapp-lib repository, and your main application in the sampleapp repository.

Next we’ll install Django:


$ source bin/activate
$ pip install Django
 

Then we’ll set up a standard Django site:


$ cd src/sampleapp
$ django-admin.py sampleapp
 

Also we’d like to be able to import this file. It’d be nice if there was a setup.py file, and we could run pip -e src/sampleapp, but django-admin.py doesn’t create that itself. Instead we’ll get that on the import path more manually with a .pth file:


$ echo "../../src/sampleapp" > lib/python/sampleapp.pth
 

Also there’s the tricky $DJANGO_SETTINGS_MODULE that you might have had problems with before. We’ll use the file lib/python/toppcustomize.py (which is imported everytime Python is started) to make sure that is always set:


$ echo "import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'sampleapp.settings'
" > lib/python/toppcustomize.py
 

Also we have a file src/sampleapp/sampleapp/manage.py, and that file doesn’t work quite how we’d like. Instead we’ll put a file into bin/manage.py that does the same thing:


$ rm sampleapp/manage.py
$ cd ../..
$ echo '#!/usr/bin/env python
from django.core.management import execute_manager
from sampleapp import settings
if __name__ == "__main__":
    execute_manager(settings)
' > bin/manage.py
$ chmod +x bin/manage.py
 

Now, if you were just using plain Django you’d do something like run python manage.py runserver. But we’ll be using toppcloud serve instead, which means we have to set up the two other files toppcloud needs: app.ini and the runner. Here’s a simple app.ini:


$ echo '[production]
app_name = sampleapp
version = 1
runner = src/sampleapp/toppcloud-runner.py
' > src/sampleapp/toppcloud-app.ini
$ rm app.ini
$ ln -s src/sampleapp/toppcloud-app.ini app.ini
 

The file must be in the "root" of your application, and named app.ini, but it’s good to keep it in version control, so we set it up with a symlink.

It also refers to a "runner", which is the Python file that loads up the WSGI application. This looks about the same for any Django application, and we’ll put it in src/sampleapp/toppcloud-runner.py:


$ echo 'import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
' > src/sampleapp/toppcloud-runner.py
 

Now if you want to run the application, you can:


$ toppcloud serve .
 

This will load it up on http://localhost:8080, and serve up a boring page. To do something interesting we’ll want to use a database.

Setting Up A Database

At the moment the only good database to use is PostgreSQL with the PostGIS extensions. Add this line to app.ini:


service.postgis =
 

This makes the database "available" to the application. For development you still have to set it up yourself. You should create a database sampleapp on your computer.

Next, we’ll need to change settings.py to use the new database configuration. Here’s the lines that you’ll see:


DATABASE_ENGINE = ''           # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = ''             # Or path to database file if using sqlite3.
DATABASE_USER = ''             # Not used with sqlite3.
DATABASE_PASSWORD = ''         # Not used with sqlite3.
DATABASE_HOST = ''             # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = ''             # Set to empty string for default. Not used with sqlite3.
 

First add this to the top of the file:


import os
 

Then you’ll change those lines to:


DATABASE_ENGINE = 'postgresql_psycopg2'
DATABASE_NAME = os.environ['CONFIG_PG_DBNAME']
DATABASE_USER = os.environ['CONFIG_PG_USER']
DATABASE_PASSWORD = os.environ['CONFIG_PG_PASSWORD']
DATABASE_HOST = os.environ['CONFIG_PG_HOST']
DATABASE_PORT = ''
 

Now we can create all the default tables:


$ manage.py syncdb
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
...
 

Now we have an empty project that doesn’t do anything. Let’s make it do a little something (this is all really based on the Django tutorial).


$ manage.py startapp polls
 

Django magically knows to put the code in src/sampleapp/sampleapp/polls/ — we’ll setup the model in src/sampleapp/sampleapp/polls/models.py:


from django.db import models

class Poll(models.Model):
    question = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

class Choice(models.Model):
    poll = models.ForeignKey(Poll)
    choice = models.CharField(max_length=200)
    votes = models.IntegerField()
 

And activate the application by adding 'sampleapp.polls' to INSTALLED_APPS in src/sampleapp/sampleapp/settings.py. Also add 'django.contrib.admin' to get the admin app in place. Run manage.py syncdb to get the tables in place.

You can try toppcloud serve . and go to /admin/ to login and see your tables. You might notice all the CSS is broken.

toppcloud serves static files out of the static/ directory. You don’t actually put static in the URLs, these files are available at the top-level (unless you create a static/static/ directory). The best way to put files in there is generally symbolic links.

For Django admin, do this:


$ cd static
$ ln -s ../lib/python/django/contrib/admin/media admin-media
 

Now edit src/sampleapp/sampleapp/settings.py and change ADMIN_MEDIA_PREFIX to '/admin-media'.

(Probably some other links should be added.)

One last little thing you might want to do; replace this line in
settings:


SECRET_KEY = 'ASF#@$@#JFAS#@'
 

With this:


from tcsupport.secret import get_secret
SECRET_KEY = get_secret()
 

Then you don’t have to worry about checking a secret into version control.

You still don’t really have an application, but the rest is mere "programming" so have at it!

2010 02 05

Programming
Python
Silver Lining
Web

Comments (2)

Permalink

A new way to deploy web applications

Deployment is one of the things I like least about development, and yet without deployment the development doesn’t really matter.

I’ve tried a few things (e.g. fassembler), built a few things (virtualenv, pip), but deployment just sucked less as a result. Then I got excited about App Engine; everyone else was getting excited about "scaling", but really I was excited about an accessible deployment process. When it comes to deployment App Engine is the first thing that has really felt good to me.

But I can’t actually use App Engine. I was able to come to terms with the idea of writing an application to the platform, but there are limits… and with App Engine there were simply too many limits. Geo stuff on App Engine is at best a crippled hack, I miss lxml terribly, I never hated relational databases, and almost nothing large works without some degree of rewriting. Sometimes you can work around it, but you can never be sure you won’t hit some wall later. And frankly working around the platform is tiring and not very rewarding.

So… App Engine seemed neat, but I couldn’t use it, and deployment was still a problem.

What I like about App Engine: an application is just files. There’s no build process, no fancy copying of things in weird locations, nothing like that; you upload files, and uploading files just works. Also, you can check everything into version control. Not just your application code, but every library you use, the exact files that you installed. I really wanted a system like that.

At the same time, I started looking into "the cloud". It took me a while to get a handle on what "cloud computing" really means. What I learned: don’t overthink it. It’s not magic. It’s just virtual private servers that can be requisitioned automatically via an API, and are billed on a short time cycle. You can expand or change the definition a bit, but this definition is the one that matters to me. (I’ve also realized that I cannot get excited about complicated solutions; only once I realized how simple cloud computing is could I really get excited about the idea.)

Given the modest functionality of cloud computing, why does it matter? Because with a cloud computing system you can actually test the full deployment stack. You can create a brand-new server, identical to all servers you will create in the future; you can set this server up; you can deploy to it. You get it wrong, you throw away that virtual server and start over from the beginning, fixing things until you get it right. Billing is important here too; with hourly billing you pay cents for these tests, and you don’t need a pool of ready servers because the cloud service basically manages that pool of ready servers for you.

Without "cloud computing" we each too easily find ourselves in a situation where deployments are ad hoc, server installations develop over time, and servers and applications are inconsistent in their configuration. Cloud computing makes servers disposable, which means we can treat them in consistent ways, testing our work as we go. It makes it easy to treat operations with the same discipline as software.

Given the idea from App Engine, and the easy-to-use infrastructure of a cloud service, I started to script together something to manage the servers and start up applications. I didn’t know what exactly I wanted to do to start, and I’m not completely sure where I’m going with this. But on the whole this feels pretty right. So I present the provisionally-named: toppcloud (Update: this has been renamed Silver Cloud).

How it works: first you have a directory of files that defines your application. This probably includes a checkout of your "application" (let’s say in src/mynewapp/), and I find it also useful to use source control on the libraries (which are put in lib/python/). There’s a file in app.ini that defines some details of the application (very similar to app.yaml).

While app.ini is a (very minimal) description of the application, there is no description of the environment. You do not specify database connection details, for instance. Instead your application requests access to a database service. For instance, one of these services is a PostgreSQL/PostGIS database (which you get if you put service.postgis in your app.ini file). If you ask for that then there will be evironmental variables, CONFIG_PG_DBNAME etc., that will tell your application how to connect to the database. (For local development you can provide your own configuration, based on how you have PostgreSQL or some other service installed.)

The standard setup is also a virtualenv environment. It is setup so every time you start that virtualenv environment you’ll get those configuration-related environmental variables. This means your application configuration is always present, your services always available. It’s available in tests just like it is during a request. Django accomplishes something similar with the (much maligned) $DJANGO_SETTINGS_MODULE but toppcloud builds it into the virtualenv environment instead of the shell environment.

And how is the server setup? Much like with App Engine that is merely an implementation detail. Unlike App Engine that’s an implementation detail you can actually look at and change (by changing toppcloud), but it’s not something you are supposed to concern yourself with during regular application development.

The basic lifecycle using toppcloud looks like:

toppcloud create-node
Create a new virtual server; you can create any kind of supported server, but only Ubuntu Jaunty or Karmic are supported (and Jaunty should probably be dropped). This step is where the "cloud" part actually ends. If you want to install a bare Ubuntu onto an existing physical machine that’s fine too — after toppcloud create-node the "cloud" part of the process is pretty much done. Just don’t go using some old Ubuntu install; this tool is for clean systems that are used only for toppcloud.
toppcloud setup-node
Take that bare Ubuntu server and set it up (or update it) for use with toppcloud. This installs all the basic standard stuff (things like Apache, mod_wsgi, Varnish) and some management script that toppcloud runs. This is written to be safe to run over and over, so upgrading and setting up a machine are the same. It needs to be a bare server, but
toppcloud init path/to/app/
Setup a basic virtualenv environment with some toppcloud customizations.
toppcloud serve path/to/app
Serve up the application locally.
toppcloud update --host=test.example.com path/to/app/
This creates or updates an application at the given host. It edits /etc/hosts so that the domain is locally viewable.
toppcloud run test.example.com script.py
Run a script (from bin/) on a remote server. This allows you to run things like django-admin.py syncdb.

There’s a few other things — stuff to manage the servers and change around hostnames or the active version of applications. It’s growing to fit a variety of workflows, but I don’t think its growth is unbounded.

So… this is what toppcloud. From the outside it doen’t do a lot. From the inside it’s not actually that complicated either. I’ve included a lot of constraints in the tool but I think it offers an excellent balance. The constraints are workable for applications (insignificant for many applications), while still exposing a simple and consistent system that’s easier to reason about than a big-ball-of-server.

Some of the constraints:

  1. Binary packages are supported via Ubuntu packages; you only upload portable files. If you need a library like lxml, you need to request that package (python-lxml) to be installed in your app.ini. If you need a version of a binary library that is not yet packaged, I think creating a new deb is reasonable.
  2. There is no Linux distribution abstraction, but I don’t care.
  3. There is no option for the way your application is run — there’s one way applications are run, because I believe there is a best practice. I might have gotten the best practice wrong, but that should be resolved inside toppcloud, not inside applications. Is Varnish a terrible cache? Probably not, but if it is we should all be able to agree on that and replace it. If there are genuinely different needs then maybe additional application or deployment configuration will be called for — but we shouldn’t add configuration just because someone says there is a better practice (and a better practice that is not universally better); there must be justifications.
  4. By abstracting out services and persistence some additional code is required for each such service, and that code is centralized in toppcloud, but it means we can also start to add consistent tools usable across a wide set of applications and backends.
  5. All file paths have to be relative, because files get moved around. I know of some particularly problematic files (e.g., .pth files), and toppcloud fixes these automatically. Mostly this isn’t so hard to do.

These particular compromises are ones I have not seen in many systems (and I’ve started to look more). App Engine I think goes too far with its constraints. Heroku is close, but closed source.

This is different than a strict everything-must-be-a-package strategy. This deployment system is light and simple and takes into account reasonable web development workflows. The pieces of an application that move around a lot are all well-greased and agile. The parts of an application that are better to Do Right And Then Leave Alone (like Apache configuration) are static.

Unlike generalized systems like buildout this system avoids "building" entirely, making deployment a simpler and lower risk action, leaning on system packages for the things they do best. Other open source tools emphasize a greater degree of flexibility than I think is necessary, allowing people to encode exploratory service integration into what appears to be an encapsulated build (I’m looking at you buildout).

Unlike requirement sets and packaging and versioning libraries, this makes all the Python libraries (typically the most volatile libraries) explicit and controlled, and can better ensure that small updates really are small. It doesn’t invalidate installers and versioning, but it makes that process even more explicit and encourages greater thoughtfulness.

Unlike many production-oriented systems (what I’ve seen in a lot of "cloud" tools) this encorporates both the development environment and production environment; but unlike some developer-oriented systems this does not try to normalize everyone’s environment and instead relies on developers to set up their systems however is appropriate. And unlike platform-neutral systems this can ensure an amount of reliability and predictability through extremely hard requirements (it is deployed on Ubuntu Jaunty/Karmic only).

But it’s not all constraints. Toppcloud is solidly web framework neutral. It’s even slightly language neutral. Though it does require support code for each persistence technique, it is fairly easy to do, and there are no requirements for "scalable systems"; I think unscalable systems are a perfectly reasonable implementation choice for many problems. I believe a more scalable system could be built on this, but as a deployment and development option, not a starting requirement.

So far I’ve done some deployments using toppcloud; not a lot, but some. And I can say that it feels really good; lots of rough edges still, but the core concept feels really right. I’ve made a lot of sideways attacks on deployment, and a few direct attacks… sometimes I write things that I think are useful, and sometimes I write things that I think are right. Toppcloud is at the moment maybe more right than useful. But I genuinely believe this is (in theory) a universally appropriate deployment tool.

Alright, so now you think maybe you should look more at toppcloud…

Well, I can offer you a fair amount of documentation. A lot of that documentation refers to design, and a bit of it to examples. There’s also a couple projects you can look at; they are all small, but :

  • Frank (will be interactivesomerville.org) which is another similar Django/Pinax project (Pinax was a bit tricky). This is probably the largest project. It’s a Django/Pinax volunteer-written application for collecting community feedback the Boston Greenline project, if that sounds interesting to you might want to chip in on the development (if so check out the wiki).
  • Neighborly, with minimal functionality (we need to run more sprints) but an installation story.
  • bbdocs which is a very simple bitbucket document generator, that makes the toppcloud site.
  • geodns which is another simple no-framework PostGIS project.

Now, the letdown. One thing I cannot offer you is support. THERE IS NO SUPPORT. I cannot now, and I might never really be able to support this tool. This tool is appropriate for collaborators, for people who like the idea and are ready to build on it. If it grows well I hope that it can grow a community, I hope people can support each other. I’d like to help that happen. But I can’t do that by bootstrapping it through unending support, because I’m not good at it and I’m not consistent and it’s unrealistic and unsustainable. This is not a open source dead drop. But it’s also not My Future; I’m not going to build a company around it, and I’m not going to use all my free time supporting it. It’s a tool I want to exist. I very much want it to exist. But even very much wanting something is not the same as being an undying champion, and I am not an undying champion. If you want to tell me what my process should be, please do!

If you want to see me get philosophical about packaging and deployment and other stuff like that, see my upcoming talk at PyCon.

2010 01 29

Packaging
Programming
Python
Silver Lining
Web

Comments (13)

Permalink

Toward a new self-definition for open source

This is roughly the speech I gave as a keynote address at DjangoCon 2009 in Portland.

I’ve been programming Python web software for quite a while now. I considered coming here and talked about WSGI, standards, cross-framework integration, etc., but I decided I wanted to come up here and talk to you as compatriots, as fellow open source programmers.

Over the past year or so I have been thinking a lot about politics. Not electoral politics per se, or the geopolitical situation, but the question of the meaning of what we are doing. Maybe it is some sort of programmer midlife crisis: why am I doing what I’m doing, and why does it matter? And also I have been thinking a lot about open source — what this thing that we’re doing means in the larger context. Are we just engineers? Is there some sort of movement? If so, what is that movement? Especially as open source has become more popular, the sense of a movement seems to dwindle. It felt like a movement 10 years ago, but not as much today. Why should this happen? Why now, in the midst of success does open source seem less politically relevant than it did 10 years ago?

I’m also speaking somewhat to Django specifically, as I think it is one of the communities with a bit more resistance to the idea of the politics of code. The Django philosophy is more: the value of this code is the value of what you do with it. I’m not here to criticize this perspective, but to think about how we can find meaning without relying on the traditional free software ideas. To see if there’s something here that isn’t described yet.

I’d like to start with a quick history of free and open source software. My own start was in highschool where I was taking a class in which we all used Emacs. This was my first introduction to any real sort of programming environment, to Unix, to a text editor that was anything to talk about. At the time Emacs would say at startup "to learn more about the GNU project hit Control-H Control-P" — because of course you need a keyboard shortcut to get to a philosophy statement about an editor. So one day I hit Control-H Control-P. I was expecting to see some sort of About Message, or if you remember the software of the times maybe something about Shareware, or even "if you really like this software, consider giving to the Red Cross." But instead I came upon the GNU Manifesto.

GNU Manifesto

I’d like to read a couple quotes from the GNU Manifesto. There are more modern descriptions of GNU, but this is one of the first documents describing the project and its mission, written by Richard Stallman. Let me quote the section "Why I Must Write GNU":

"I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. [it continues...]

"So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free."

[later it goes on...]

"The fundamental act of friendship among programmers is the sharing of programs; marketing arrangements now typically used essentially forbid programmers to treat others as friends. The purchaser of software must choose between friendship and obeying the law. Naturally, many decide that friendship is more important. But those who believe in law often do not feel at ease with either choice. They become cynical and think that programming is just a way of making money."

When I read this statement I was immediate head-over-heels in love with this concept. As a teenager, thinking about programming, thinking about the world, having a statement that was so intellectually aggressive was exciting. It didn’t ask: "how wrong is piracy really", or "why are our kids doing this", but it asked "is piracy a moral imperative" — that’s the kind of aggressive question that feels revolutionary and passionate.

Let me go over one of the lines that I think exemplifies this:

"I consider that the golden rule requires that if I like a program I must share it with other people who like it."

It wasn’t saying: what are we not allowed to do, nor did it talk about some kind of societal injustice. It didn’t talk about the meaning of actions or their long-term effects. Instead it asked: what must we do, not as a society, not in service of some end, but what are we called upon to do as an individual, right now, in service of the people we call friends. It didn’t allude to any sociological explanation, natural selection, economics; there is just the golden rule, the most basic tenant of moral thought.

Free Software vs. Open Source

When I first encountered free software, I suppose about 15 years ago, this was during one of the more difficult periods of its evolution. It was past the initial excitement, the initial optimism that the project would take only a couple years to reach parity with closed-source Unix, it was before it was clear how the project would move forward. Linux was young and seemed to be largely off the radar of Richard Stallman and other GNU advocates, and they were struggling to fill in final key pieces, things like a kernel, widgets, and they hadn’t even thought about entirely new things like browsers.

The message that came through from that period is not the message I wish came through. The message I wish came through was that message from the GNU Manifesto, that spirit of a new sense of duty and ability. When people talk about Richard Stallman and the Free Software Foundation and the GNU Project, they’ll often point to the GNU General Public License as the most important contribution — the GPL. I’m sure you all know something about it. Of course the core concept there is the idea of copyleft. Not only will the software be open, but it’s takes the implied perspective that the principles of freedom are rights — that unfortunately the world is not wise enough to see that use, distribution, and modification are rights; but the GPL still asserts that these are your rights. When you become one of us, when you participate in the freedoms this license grants you, when you use the GPL, there is encoded in the license a support for this sense of the natural right of free software. We, GNU, can’t encode that for the world, but for the software that we write these rights are inalienable.

But as I said those were difficult times. There was a great deal of pressure. People were trying to understand what open source meant. People still struggle with questions: how would an economy function, how would a programmer get a job, if this is as successful as people hoped will we all just be out of jobs? Other questions were: who will write the software that no one wants to write? How can I, embedded in a situation where I can’t actually use only free software — remember at this time there was no way to use completely free software — how can I assert a duty to do something that is not possible? How can I be useful unless I interface with all these proprietary things? If I deal with companies which aren’t comfortable with open source software, then what? After all, open source seemed only barely plausible at this time. It was not an easy argument to make.

And all this was before the term "open source" really took hold as a distinct idea. That itself is an interesting story. There was a time during this marketing period when there arose a kind of terminology divide — free software vs. open source software. The terminology divide was that the "free" in free software implied you couldn’t charge anything, that made people think about price, might even imply cheapness. Open source directly refers to programming, uses the feel-good term "open", and doesn’t sound too revolutionary. But besides that there was also a substantial philosophical difference about the value of the software itself.

So there was a difference in how things were going to be pitched, but also a difference in what people thought the general value of this project was. From GNU and Richard Stallman there was the notion that this was right because it was right; it was a moral imperative. The virtue of what we build is in its freedom; if it is also technically superior then that’s great, but it is not how we should judge our success. We were giving people self-determination: programmer self-determination, user self-determination… on the open source side the argument was that this is a good way to create software. Programmers working together can do better work. With many eyes all bugs are shallow. All working together, we’ll work faster, you get the benefit of free contributions from all sorts of people. People were clamouring to get all these proprietary companies with failing software products to open source their software; miracles will occur! What you thought was useless will regain value! You’ll reattain relevance!

The open source and free software philosophical divide: on one side practical concerns, on the other moral. And this is what I want to talk about later: can we find a moral argument for these practical concerns?

The basic free/open disagreement continues in debates over licenses: the GPL vs. the BSD and other permissive licenses. If you read the GPL it talks a great deal about philosophy; if you read the BSD license it’s really just some disclaimers and basic instructions, and the one line: "Redistribution and use in source and binary forms, with or without modification, are permitted." It doesn’t say why you’ve received this software, or any philosophy about rights and freedoms, or even encourage you to use the same license on your own software. An engineer’s license.

So these two licenses in many ways became a focus and definition of free and open source software. If you look at the Open Source Initiative, which has served to define what "open source" means, it is basically just a list of approved licenses. If you use one of those licenses, the software is open source, if not then it is closed.

I think this is disappointing, because licenses are just law, and law is not very interesting. A law tells you what you shouldn’t do, it doesn’t tell you what you should do. When both sides are talking about freedom, the licenses just define freedom as the lack of certain restrictions. Take away those restrictions and voila, you are free… as though we are all just bundles of freedom waiting to be released.

Self-Definitions

With licenses we have a negative definition of our community. Either license you choose, the license feels like a reaction against closed source software. If you can imagine a world in which there was no copyright, where our platforms were all setup to distribute source code in modifiable forms, where everything was open and everything was free, then none of these licenses would matter. No one would be compelled to create the GPL in such a world; we wouldn’t advocate for copyright just so we can secure people’s freedoms. In that kind of world all this licensing disappears. And this isn’t even so weird a world. You can pretend there’s no copyright now. Maybe you have to reverse-engineer some stuff. There’s lots of places in the world where no one really gives a damn about copyright. But those places don’t feel open source to me, they don’t feel that more free. We aren’t made unfree just by legal restrictions; freedom is something we have to actively grasp.

I don’t think what we do is predicated on copyright. Indeed, many projects are comfortable with an entirely confused copyright ownership. This causes very few problems. A focus on licensing makes us into a reaction against proprietary software, where we allow proprietary software and its makers to define what it means to be us.

This concerns me because it isn’t just about formal definitions and terminology. When I say what do I do, I say I am an open source programmer. That’s not just an attribute, like saying that my hair is brown. Open source is a way in which I see myself, a way I think about my craft, my profession, and a way I justify the work I put out to the world: that it aligns with these values. So it’s very important to me what these values are. And it’s frustrating to see open source defined in reaction to closed source software, because personally I don’t care about closed source software that much.

I never really cared much about fighting Microsoft, and I certainly don’t care now. I see myself as a builder; this is what always drew me towards programming. The desire to build new things. This is our privilege as programmers, that we always have the opportunity to build new things. If we’re asked to do something again and again and again, you always have the power to do it in a more abstract way, to generalize it away, until you can start to ignore the requests and move on to another problem. This is something unique thing to computer programming. These are the kind of unique attributes that make us different as a profession and as a craft than just about anything I can think of.

So I’m frustrated. Here we are stuck in this notion of a license as a self-definition. I want to find a new self-definition, a new idea of what makes us us.

What Makes Us Us

So… what makes us us?

I was saying about Django, the community is not particularly enthusiastic about philosophy. Or maybe I should say, Django’s philosophy is: the value of the code is the thing you do with it. These abstract discussions about architecture, reuse, big plans… instead, Django as a community encourages you to keep your head in the code, think about what you want to do, and then do it. Don’t shave yaks.

But I’m not here to tell you to get philosophical about freedom, or to berate you for a functional definition of value. I’d like to look at this community for what it is, and ask: what is the value system here? Maybe it isn’t described, but I also don’t think it is therefore absent.

So… when I say I identify as an open source programmer, what is it that I am identifying as?

I don’t believe licensing makes something truly open source. There was this clamour in the past to get companies to open source their products. This has stopped, because all the software that got open source sucked. It’s just not very interesting to have a closed source program get open sourced. It doesn’t help anyone, because the way closed source software is created in a very different way than open source software. The result is a software base that just does not engage people in a way to make it a valid piece of software for further development. At least not unless you have something peculiar going on… an economic force like you had behind Mozilla that could push things forward even in the midst of all the problems that project had. One might even ask, is Mozilla still suffering from that proprietary background, when something like KHTML or WebKit which came from a truly open source background, and has been a more successful basis for collaboration and new work.

So whatever it is that makes something open source, it’s not just licensing. Given a codebase, we can’t necessarily expect that someone going to care about it and love it and make it successful. A lot of people have described what makes a successful open source codebase; I’d like to talk some about what the communities look like.

Open source works as a fairly loose federation of people together. Everyone involved is involved as an individual. Companies seldom participate directly in open source. Companies might use open source, they might sponsor people to work on open source projects, they might ask an employee to act as a liason. But it’s not cool to submit a bug as a company. You submit it as yourself. If someone asks a question, you answer as yourself. You don’t join a mailing list under the company’s name. And even when you put a company name on a product, it’s hard to relate to the product as a project without some sense of authorship, of the underlying individual.

There’s also very little money being moved about. There’s not a lot of commercial contracts. You might get software, you might get bug fixes, you might get reputation, but there’s seldom any formal way setup to introduce commerce into the community. How many projects let you pay to escalate a bug? Even if everyone involved might like that, it’s just not there.

But I want to get back to individuals. How things are created is not that someone determines a set of priorities, lays them out, then people work on implementation based on those priorities. That of course is how things typically work at a company, as an employee. But open source software and open source projects are created because an individual looks at the world and sees an opportunity to create something they think should exist. Maybe it resolves a tension they’ve felt in their work, maybe it allows that person to respond to the priorities from above better, but the decision to implement lies primarily with the implementor. When someone makes a decision to move a product from simply private code — regardless of the license — to being a real open source project, that decision is almost always driven by the programmer.

Underneath most open source work there is a passion for the craft itself. This is what leads to a certain kind of quality that is not the norm in closed source software. It’s not necessarily less bugs or more features, but a pride in the expression itself. A sense of aesthetic that applies to even the individual lines of software, not just to the functionality produced. This kind of aesthetic defies scheduling and relies on personal motivation.

As open source programmers we are not first concerned with how a task fits into institutions, how a task can be directed by a hierarchy or an authority, or even how the task can be directed by economics. The tasks that we take on are motivated by aesthetic, by personal excitement and drive.

We are also in a profession where there is little stasis. If you can create something once, you can create it a thousand times, through iteration or abstraction. You can constantly make your own effort obsolete. A good programmer is always trying to work themselves out of a job.

Djangocon didn’t exist a couple years ago. Django didn’t exist only a few years ago. And I don’t think there’s anyone here who thinks that, having found Django, they’ve reached some terminal point. It’s hardly even a point to pause. There’s a constant churn, a constant push forward that we’re all participating in.

As a result it’s demanded of us that we have a tight feedback cycle, that education is not a formal affair but a constant process in our own work. There’s a constant churn, and a professional sense we’re kind of like fruit flies. A generation of knowledge and practice is short enough that the evolution is rapid and visible. You don’t have to be particularly old or even thoughtful to see the changes. You can look back even on your own work and on communities to see changes over the course of a couple years, to see changes and shifts and a maturing of the work and the community.

Another attribute of open source: our communities are ad hoc and temporary. We do not overvalue these communities and institutions; we regularly migrate, split, recombine, and we constantly rewrite. There is both an arrogance and a humility to this. We are arrogant to think This Time Will Be Different. But we are humble enough to know that last time wasn’t different either. There will always be a next thing, another technique, another vision.

Because of the ad hoc nature of the communities, we don’t have long collective plans. The ad hoc community may be the intersection of different personal long range plans, a time when different visions somehow coincide in a similar implementation. Or perhaps it’s just serendipity, or leadership. But we make each decision anew. I believe this protects us from being misled by sunk costs. The idea of a sunk cost is that when you make an investment, you’ve put in effort, that effort is gone. Just because you’ve put in effort doesn’t mean you’ve received value, or that the path of investment remains valid. But as humans we are highly reluctant to let go of a plan that we’ve invested in. We have a hard time recognizing sunk costs.

I believe in our developer community we approach our work with sufficient humility that we can see our work as simply a sunk cost. Our effort does not entitle us to any particular success, so we can choose new directions with more rationality than an institution. Though it can also be argued that we are too quick to dismiss past investments; there is a youthfulness even to our oldest members.

We do not have hierachies with decision makers above implementors. Some people have veto power (a BDFL), but no one has executive power. A decision only is valid paired with an implementation. You cannot decide something based on information you wish was true; you cannot decide on something then blame the implementors for getting it wrong. We are in this sense vertically integrated, decision and implementation are combined. The result may be success or failure, commitment or abandonment, but the hierarchy is flat and the feedback loop is very tight. And if an individual feels stymied, there is always another community to join or create.

Though this is only a start, it’s these attributes that I would prefer define us, not licenses.

I also would like that this could be a model for how other work should be done.

Why Us?

Why would we, as programmers, be unique or worthy of emulation? I mentioned before that we constantly work ourselves out of our job. We also create the tools we use to do the work. We define the structure of our communities. We’re consistently finding novel ways to use the internet build those communities. It’s not that we as a group are somehow uniquely wise, or some Greatest Generation, but we have become distinctly self-empowered. There is a uniqueness to us. It might be a coincidence of history, but it is there.

A question I might then ask: is there a political meaning to this? This is the form our craft takes, but does that mean anything? We work with computers, someone else might work with their hands, an artist considers color, a salesperson learns how to put on a good smile.

I haven’t quite figured this out yet, but I think there’s something in this. Over the years I’ve found myself looking at politics in a increasingly technocratic lens; more so than as a liberal, conservative, or radical. That is, instead of looking at the world and seeing what’s wrong about it, and explaining it in terms of a class struggle, a cultural conflict, in terms of advocacy or invented enemies or allies, I see a system that just works how it works. It’s more like gears than like a war. The gears turn and sometimes we don’t like what results, but it’s not malice.

But I also don’t think we are slaves to the technical functioning of the system. None of us are inevitably caught up in some statistical outcome of markets, or condemned by money in politics or advertising. At any time we can say Here Is What I Believe, and it is as powerful as any of those other things; we’re too quick to look at the people who aren’t asserting a belief, who aren’t asserting their own potential for self-empowerment and direction, and we ignore everyone who is aware and concerned and attempting self-determination. We are at danger of ignoring the great potential around us.

It is in this sense that I wonder not just how we can spread the idea of freedom through licensing, which has inspired the free culture movement, but also how we can spread this idea and practice of individual action, of combining decision and implementation, and of constant ad hoc collaboration.

I’m not just thinking of politics directly, but of professional lives as well. Right now we’re talking about healthcare. It’s a very political issue, and yet healthcare is ultimately a profession, a job, an action. How we work on that, collaboratively or not, is as political as any aspect of the system.

One anecdote that made me think about this, is a task I had that involved putting authentication in front of a mailing list. The mailing list happened to be for wound, ostomy, and continence nurses, and in the process of the job I read a bunch of their emails from the archives. As wound nurses they spent a lot of time asking about specific questions — maybe a wound that wouldn’t heal, they kept draining the puss and it discharge kept reappearing, and did anyone have ideas of the next technique to try?

Reading a few of these I could tell this was a profession where you needed a strong stomach. But the whole interaction, the way they described problems, the way people came back with answers, it felt very familiar to me. It was the same kind of discussions I could imagine having about Linux administration or debugging. And the goals were similar. No one was making money, there wasn’t really reputation on the line, it was just people who wanted to help their patients and who wanted to help each other.

So that mailing list was great, but it’s unfortunately not that common. And if nurses were open to that kind of collaboration, doctors don’t seem nearly as ready. And there’s a lot of professions where there’s not even that thoughtfulness. I believe in any profession there’s the ability to do it well or not; there’s nothing so rote or well understood that there’s no room for improvement. It doesn’t have to be fancy technology, it can just be a technique, a way of managing work; all things worth doing have some way of improving, by bringing in this same sense of collaboration and individual action and thoughtfulness, all things can be implemented better than they are now. What I’m describing isn’t a fancy new website for professionals, but about people look at their own work differently; the technology is not the hard part.

The Political

Changing how people look at their work I think is political. It involves individual empowerment. It can mean economic change. I also think it deemphasizes competition. When I think about Pylons, or Django, or TurboGears, or WSGI, there’s competition, but it’s also collegial. There’s not really that much of a sense of survival. We aren’t carving out territories, we’re just finding paths to some unknown end. If something else wins out, well, we’re all just along for the ride. In the end it is inevitable that something else other than what any of us are working on will win out over what any of us are doing. Just like everyone eventually loses their job at least to death or retirement. There’s no permanency. But if we can be individually more productive, it doesn’t have to mean we’ve put someone else out. It could mean we all, all of society, all of humanity, just do more. Why do we have to set ourselves against the Chinese, or Europe against the U.S.? Why do we have to set ourselves one economy against another?

Or consider government itself: we’re obsessed with our elected officials, but of course government is far larger than just the elected officials. The U.S. Federal Government alone has 1.8 million employees. We constantly threaten to institute accountability, meaning that we’ll poke and prod government workers from the outside and expect better outcomes. That we expect anything to come of this is absurd, but somehow accountability has become an easy alternative to constructive suggestions for improvement.

But why shouldn’t we expect that government workers want to do better? I believe in fact those people doing the work are especially well equiped to figure out how to do better. But it’s not automatic. They aren’t empowered in a system that is so exceptionally hierarchical. Lately we’ve seen lots of efforts to ask the public how to do government work better, but we’ve seen nothing asking government how to do government work better.

These are the kinds of things I’d like to see us all think about more: open source has done incredible things, has inspired new ideas, about more than just software and engineering, but I think we have yet more things to give.

Thank you for your time.

2009 09 10

Licensing
Politics
Programming

Comments (22)

Permalink

WebOb decorator

Lately I’ve been writing a few applications (e.g., PickyWiki and a revisiting a request-tracking application VaingloriousEye), and I usually use no framework at all. Pylons would be a natural choice, but given that I am comfortable with all the components, I find myself inclined to assemble the pieces myself.

In the process I keep writing bits of code to make WSGI applications from simple WebOb -based request/response cycles. The simplest form looks like this:


from webob import Request, Response, exc

def wsgiwrap(func):
    def wsgi_app(environ, start_response):
        req = Request(environ)
        try:
            resp = func(req)
        except exc.HTTPException, e:
            resp = e
        return resp(environ, start_response)
    return wsgi_app

@wsgiwrap
def hello_world(req):
    return Response('Hi %s!' % (req.POST.get('name', 'You')))
 

But each time I’d write it, I change things slightly, implementing more or less features. For instance, handling methods, or coercing other responses, or handling middleware.

Having implemented several of these (and reading other people’s implementations) I decided I wanted WebOb to include a kind of reference implementation. But I don’t like to include anything in WebOb unless I’m sure I can get it right, so I’d really like feedback. (There’s been some less than positive feedback, but I trudge on.)

My implementation is in a WebOb branch, primarily in webob.dec (along with some doctests).

The most prominent way this is different from the example I gave is that it doesn’t change the function signature, instead it adds an attribute .wsgi_app which is WSGI application associated with the function. My goal with this is that the decorator isn’t intrusive. Here’s the case where I’ve been bothered:


class MyClass(object):
    @wsgiwrap
    def form(self, req):
        return Response(form_html...)

    @wsgiwrap
    def form_post(self, req):
        handle submission
 

OK, that’s fine, then I add validation:


@wsgiwrap
def form_post(self, req):
    if req not valid:
        return self.form
    handle submission
 

This still works, because the decorator allows you to return any WSGI application, not just a WebOb Response object. But that’s not helpful, because I need errors…


@wsgiwrap
def form_post(self, req):
    if req not valid:
        return self.form(req, errors)
    handle submission
 

That is, I want to have an option argument to the form method that passes in errors. But I can’t do this with the traditional wsgiwrap decorator, instead I have to refactor the code to have a third method that both form and form_post use. Of course, there’s more than one way to address this issue, but this is the technique I like.

The one other notable feature is that you can also make middleware:


@wsgify.middleware
def cap_middleware(req, app):
    resp = app(req)
    resp.body = resp.body.upper()
    return resp

capped_app = cap_middleware(some_wsgi_app)
 

Otherwise, for some reason I’ve found myself putting an inordinate amount of time into __repr__. Why I’ve done this I cannot say.

2009 05 22

Programming
Python
Web

Comments (11)

Permalink

Treating configuration values as templates

A while back I described fassembler and one of the things I liked in it is how the configuration works. It uses a conventional declarative INI-style but also allows arbitrary code, so that defaults can be based on each other.

Here’s a basic example of a default configuration:


[some_app]
port_offset = 10
port = {{int(section.DEFAULT['base_port'])+int(port_offset)}}
 

Then if another configuration file defines base_port then this will all resolve. You can do this in Python, but you don’t get sections, and you have to define everything in just the right order. So while base_port will probably be defined in a deployment-specific configuration, it has to be defined before these other derivative settings are defined. On the other hand, you want deployment-specific configuration to take precedence… so there’s really no good ordering.

Anyway, the implementation really isn’t that hard. I use Tempita as the templating language because, well, I wrote it, and because it’s simple and appropriate for small strings. For the configuration parsing, ConfigParser will do.

Here’s what the basic code looks like in ConfigParser:


from ConfigParser import ConfigParser
from tempita import Template

class TempitaConfigParser(ConfigParser):

    def _interpolate(self, section, option, rawval, vars):
        ns = _Namespace(self, section, vars)
        tmpl = Template(rawval, name='%s.%s' % (section, option))
        value = tmpl.substitute(ns)
        return value
 

Actually instead of using tempita.Template, we could just do eval(rawval, {}, ns), it would just require a lot more quoting (every value would have to be a valid Python expression). Either with that or Tempita the implementation of _Namespace will look the same.

Here’s a simple implementation:


from UserDict import DictMixin

class _Namespace(DictMixin):
    def __init__(self, config, section, vars):
        self.config = config
        self.section = section
        self.vars = vars

    def __getitem__(self, key):
        if key == 'section':
            return _Section(self)
        if self.config.has_option(self.section, key):
            return self.config.get(self.section, key)
        if vars and key in self.vars:
            return self.vars[key]
        raise KeyError(key)

   def __setitem__(self, key, value):
       if self.vars is None:
           self.vars = {key: value}
       else:
           self.vars[key] = value
 

We’ve introduced a magic variable section, which is used to refer to other sections. It looks like this:


class _Section(object):
    def __init__(self, namespace):
        self._namespace = namespace

    def __getattr__(self, attr):
        if attr.startswith('_'):
            raise AttributeError(attr)
        return _Namespace(self._namespace.config, attr,     self._namespace.vars)
 

With these I think you get many of the benefits of using Python code as your configuration format, while still having the benefits of a more declarative approach to configuration, one that allows for forward and backward references.

A full implementation has several more things than I show here, but you can see the full example in my recipes. It also has an example of using INITools instead of ConfigParser to give more accurate filenames and line numbers when there is an exception, while otherwise using the same interface.

2009 04 20

Programming

Comments (3)

Permalink

Woonerf and Python

At TOPP there’s a lot of traffic discussion, since a substantial portion of the organization is dedicated to Livable Streets initiatives. One of the traffic ideas people have gotten excited about is Woonerf. This is a Dutch traffic planning idea. In areas where there’s the intersection of lots of kinds of traffic (car, pedestrian, bike, destinations and through traffic) you have to deal with the contention for the streets. Traditionally this is approached as a complicated system of rules and right-of-ways. There’s spaces for each mode of transportation, lights to say which is allowed to go when (with lots of red and green arrows), crosswalk islands, concrete barriers, and so on.

A problem with this is that a person can only pay attention to so many things at a time. As the number of traffic controls increases, the controls themselves dominate your attention. It’s based on the ideal that so long as everyone pays attention tothe controls, they don’t have to pay attention to each other. Of course, if there’s a circumstance the controls don’t take into account then people will deviate (for instance, crossing somewhere other than the crosswalk, or getting in the wrong lane for a turn, or the simple existance of a bike is usually unaccounted for). If all attention is on the controls, and everyone trusts that the controls are being obeyed, these deviations can lead to accidents. This can create a negative feedback cycle where the controls become increasingly complex to try to take into account every possibility, with the addition of things like Jersey barriers to exclude deviant traffic. At least in the U.S., and especially in the suburbs or in complex intersections, this feeling of an overcontrolled and restricted traffic plan is common.

Copenhagen retail street

So: Woonerf. This is an extreme reaction to traffic controls. An intersection designed with the principles of Woonerf eschews all controls. This includes even things like curbs and signage. It removes most cues about behavior, and specifically of the concept of "right of way". Every person entering the intersection must view it as a negotiation. The use of eye contact, body language, and hand signals determines who takes the right of way. In this way all kinds of traffic are peers, regardless of destination or mode of transport. Also each person must focus on where they are right now, and not where they will be a minute from now; they must stay engaged.

Code as Jersey Barrier

So, I was reading a critique of Python where someone was saying how they missed public/private/protected distinctions on attributes and methods. And it occurred to me: Python’s object model is like Woonerf.

Python does not enforce rules about what you must and must not do. There are cues, like leading underscores, the __magic_method__ naming pattern, or at the module level there’s __all__. But there are no curbs, you won’t even feel the slightest bump when you access a "private" attribute on an instance.

This can lead to conflicts. For example, during discussions on installation, some people will argue for creating requirements like "SomeLibrary>=1.0,<2.0", with the expectation that while version 2.0 doesn’t exist, so long as you install something in the 1.x line it will maintain compatibility with your application. This is an unrealistic expectation. Do you and the library maintainer have the same idea about what compatibility means? What if you depend on something the maintainer considers a bug?

Practically, you can’t be sure that future versions of a library will work. You also can’t be sure they won’t work; there’s nothing that requires the maintainer of the library to break your application with version 2.0. This is where it becomes a negotiation. If you decide to cross without a crosswalk (use a non-public API) then okay. You just have to keep an eye out. And library authors, whether they like it or not, need to consider the API-as-it-is-used as much as the API-they-have-defined. In open source in particular, there are a lot of ways to achieve this communication. We don’t use some third party (e.g., a QA team or language features) to enforce rules on both sides (there are no traffic controls), instead the communication is more flat, and speaks as much to intentions as mechanisms. When someone asks "how do I do X?" a common response is: "what are you trying to accomplish?" Often an answer to the second question makes the first question irrelevant.

Woonerf is great for small towns, for creating a humane space. Is it right for big cities and streets, for busy people who want to get places fast, for trucking and industry? I’m not sure, but probably not. This is where a multi-paradigm approach is necessary. Over time libraries have to harden, become more static, innovation should happen on top of them and not in the library. Some times we create third party controls through interfaces (of one kind or another). I suppose in this case there is a kind of negotiation about how we negotiate — there’s no one process for how to build negotiation-free foundations in Python. But it’s best not to harden things you aren’t sure are right, and I’m pretty sure there’s no "right" at this very-human level of abstraction.

2009 01 16

Programming

Comments (9)

Permalink

Cultural Imperialism, Technology, and OLPC

A couple posts have got me thinking about cultural imperialism lately: a post by Guido van Rossum about "missionaries" and OLPC not about OLPC at all, a post by Chris Hardie and a speech by Wade Davis.

Some of the questions raised: are we destroying cultures? If so, what can we do about it? Must we be hands off? I will add these questions: is it patronizing to make these choices for other people, no matter how enlightened we try to be? How much change is inevitable? Can we help make the change positive instead of resisting change?

More specifically: what is the effect of OLPC on cultures where it is introduced? Especially small cultures, cultures that have been relatively isolated, cultures that are vulnerable. The internet Quechua community is pretty slim, for example. Introducing the internet into a community will lead the children to favor Spanish more strongly, and identify with that more dominant culture over their family and community culture.

Criticisms like Guido’s are common:

I’m not surprised that the pope is pleased by the OLPC program. The mentality from which it springs is the same mentality which in past centuries created the missionary programs. The idea is that we, the west, know what’s good for the rest of the world, and that we therefore must push our ideas onto the "third world" by means of the most advanced technology available. In past centuries, that was arguably the printing press, so we sent missionaries armed with stacks of bibles. These days, we have computers, so we send modern missionaries (of our western lifestyle, including consumerism, global warming, and credit default swaps) armed with computers.

This kind of criticism is easy, because it doesn’t have any counterproposal. It’s not saying much more than "you all suck" to the people involved.

Cultural imperialism is a genuine phenomena. In an attempt to subjugate or assimilate, the dominant culture may explicitly and cynically enforce its cultural norms, through its religion, requiring all schools to operate in the dominant language, even going as far as suggesting how we arrange ourselves during sex.

But it’s not clear to me that what’s happening now is cultural imperialism. It’s more market-oriented homogenization. Food manufacturers don’t use high-fructose corn syrup because they want to make us fat — they just give us what we want, and they are enabling our latent tendency to become obese. Similarly I think the way culture is spread currently encourages homogeneity, without explicit attempting to destroy culture.

This is where I think a protectionist stance — the idea we should just be hands-off — is patronizing. People aren’t abandoning their cultures because they are stupid and they are being manipulated. People make decisions, what they think is the best decision for themself and their families. These decisions lead them to leave rural areas, learn the dominant language, try to conform through education, and even just lead them to enjoy a dominant culture which is often far more entertaining than a smaller and more traditional culture.

The irony is that once they’ve done this they’ve traded their position for a place in the bottom rung of the dominant society. And it’s true that in many cases they’ve made these decisions because they’ve been forced out of their traditional life by political and legal systems they don’t understand. But to blame it all on oppression is to be blind to the many concrete benefits of our modern world. Corrugated metal roofs are simply superior to thatched roofs, and we can get all romantic about traditional building processes and material independence, but we do so from homes with roofs that don’t leak. Leaking roofs are just objectively unpleasant. And frankly people like TV, you don’t have to tell people to like TV, it just happens.

So I believe that assimilation pressure is natural and inevitable in our times.

What then of technology, of the internet and laptops?

I believe OLPC takes an important stance when it selects open source and open licensing for its content. It is valuing freedom, but more importantly encouraging self-determination, trying to build up a user base that can act as peers in this project, not as simply receivers of first-world largess. But it will be culturally disruptive. And I’m okay with that. In a patriarchal culture, giving girls access to this technology will be destructive to that power structure. Yay! I believe in the moral rightness of that one girl making her own choices, finding her own truths, more than I believe in the validity of the culture she was born into. If you believe people should be able to make their own choices (so long as they are aware of the real consequence of their choices), then you must allow for them to choose to abandon their own cultures for something they find more appealing. They might know better than you if that’s a good choice. I think we all hope that instead they transform their own cultures, but that’s not our choice to make.

What I find unpleasant is if they leave a true identity to find themselves in a place of cultural subservience. If they feel they can’t preserve the part of their culture they most value. Perhaps because of discrimination they feel they must hide their past, or they build up a sense of self-loathing. Perhaps they become isolated, unable to find peers that understand where they come from. And perhaps there is no higher culture at all that they can use to exalt their understanding of the world — do they have a literature? Do they have non-traditional music forms of their own? Do they have a forum where people who share their perspective can have serious discussions? Cultures aren’t destroyed so much as they are starved out of existence.

I think assimilation is inevitable, and can be positive. If we were all able to speak to each other, with some shared second or third language, I think the world would be a better place. I’m not a Christian, but I’m not afraid of anyone knowing The Bible. There’s no piece of culture that I would want to deny from anyone. Each new song, each new book, each new idea… I believe they will all make you a better person, if only in a small way.

And on the internet our culture is cumulative. There’s only so many hours of programming on TV or the radio, only so many pages in a newspaper. On the internet the presence of one kind of culture does not exclude any other. There’s room for a Quechua community as much of any other. But the online Quechua community won’t have exclusive rights to its members like a traditional culture claims — children will live between cultures.

Cumulative culture is not a promise that anyone will care. Languages can still die, cultures can still die, identities become forgotten. If these smaller cultures are going to be preserved, they must adapt to the partially-assimilated status of their members. There must be new art and new ideas and new identities. This is why I believe in the laptop project, because it can enable the creation and sharing of these new ideas. I think it will give smaller cultures a chance to survive — there’s no promises, literature doesn’t write itself, but maybe there is at least a chance.

This is also why I am more skeptical of mobile phones, audio devices, and any device that doesn’t actively enable content creation. Mobile phones are not how culture is made. It let’s people chat, consume information, communicate in a 12-key pidgin. But the mobile phone user is not a peer in a world wide web of information. The mobile phone user lives on a proprietary network, with a proprietary device, and while it perhaps it breaks down some hierarchies through disintermediation, it does so in a transient way. The uptake is certainly faster, but the potential seems so much lower.

I don’t know if OLPC will be successful. That’s as unclear now as ever. But it’s trying to do the right thing, and I think it’s a better chance than most for maintaining or improving the richness of the worlds’ culture.

2009 01 14

Non-technical
OLPC
Politics

Comments (22)

Permalink

« Older posts
Newer posts »