I posted this to distutils-sig as a kind of rambly brainstorm. I will subject you all to it too. If you don't care about Python installation, you most certainly won't care about this post; be warned!

So, I'm coming back to the idea of a working environment, an isolated and more-or-less self-contained environment for holding installed packages. We discussed this a fair amount at PyCon.

I'm assuming such an environment will be encapsulated in a single directory, looking something like:

env/
   bin/
   lib/python2.4/
   src/
   conf/

The conf/ directory doesn't really relate to much of this specifically, but in many situations it would be useful. Depending on the situation, other subdirectories may exist.

Each of the scripts in bin/ should know what their working environment is. This is slightly tricky, depending on what that means. If it is a totally isolated environment -- no site-packages/ on sys.path -- then I feel like the script wrappers have to be shell scripts, to invoke Python with -S (which is hard to do portably on the #! line). I don't know the details of doing the same thing on Windows, but I assume it is possible. The actual directory location should be portable -- all paths should be relative, and you should be able to move the directory around.

lib/python2.4/ is for packages. I'm almost inclined to say that --single-version-externally-managed makes sense on some level, with a record kept in some standard place (lib/python2.4/install-record.txt?) -- but I'm basically indifferent. I at least don't see a need for multiple simultaneous versions in this setup, and multiple versions do lead to confusion. Version metadata is still nice, of course.

src/ is for checkouts, where each package is installed with setup.py develop. These are naturally single-version, which is part of why I like the idea of only using single-version setups. I'm a little unsure of how src/ should be layed out. In practice I want all "my" packages to be installed in src/ as checkouts, either from tags or the trunk (or a branch or whatever). Even in the case of tags, a checkout can be turned into a branch fairly easily. So I'm not sure if I should name the subdirectories after the package, or maybe even the package plus a tag name.

One of the things SwitchTower (now "Cappucino", I think) does in Rails land is it makes a dated checkout, then activates that checkout (it does that with a symlink; we'd do it with setup.py develop). It then rolls back by switching to an existing checkout. Of course svn switch + svn up does this in place, and with less checkout trash laying around, even if rollbacks aren't as fast as a result. So, I'm thinking just src/PackageName/, or whatever convention you felt like (e.g., src/PackageName-branchname/)

There's an installation issue with checkouts -- it would be nice if I could say "these are the packages I want to install as editable" and easy_install would pick those up (maybe detecting based on what package index the package was found in) and install them in src/ as editable.

Anyway, sys.path would contain /usr/lib/python2.4 (or Windows equivalent), env/lib/python2.4/ and optionally /usr/lib/python2.4/site-packages, and all the similar directories. Unfortunately figuring out what "similar" directories there are is hard. sys.path on my machine now has 63 entries normally and 12 with python -S. I guess I'd really like to start with 12 and build up, instead of 63 and try to strip them down.

Installation as a whole is an open issue. Putting in env/setup.cfg with the setting specific to that working environment works to a degree -- easy_install will pick it up if invoked from the env/ directory. But that doesn't work with setup.py develop, or setup.py install, or some other scenarios. The system distutils.cfg doesn't really work, because the only expansion it knows how to do is of user directories, so there's little way to pass interesting information in (like a "this is my setup.cfg" environmental variable or something). Maybe with $PYTHONPATH to indicate the working environment, and a distutils monkeypatch put into lib/python2.4/distutils/__init__.py? I played around with putting the path setup in sitecustomize, but that runs after site.py, and doesn't run at all if python -S is used, so it seems like it brings in too much before it can remove stuff.

Another option is a completely new python interpreter bound to the environment. Basically the virtual-python.py option.

In this model using env/bin/python indicates the proper environment, and you'd have local installs of everything including easy_install. This fixes so many problems without crazy hacks that it strongly appeals to me, especially if we can make it somewhat lighter. I get this /usr/lib/python2.4.zip on my path, a file that doesn't usually exist; if we could create that zip on demand and use such a big-bundle-zip, that seems lighter and faster and nicer, especially if shared. If we just put .pyc files in the zip, and those .pyc files refer back to the actual module source (in /usr/lib/python2.4/), then tracebacks should also still work, I believe? No actual symlinks either, so it should work on Windows. I'm not entirely sure where I'm going with this, though.

So, by the end I'm thinking an improved virtual-python.py is what I'm looking for, as kind of the One True Way to do good Python installation and deployment.