working-env.py

Update: a newer version of this script is discussed here

After mulling over the stuff from my last distutils post I started playing around with ideas. In the end I came up with working-env.py. This is similar to virtual-python.py, but perhaps "lighter", because it doesn't create lots of symlinks or a new Python interpreter.

Basically it works like:

$ python working-env.py new-working-env
$ source new-working-env/bin/activate

That activate script just sets some environmental variables; mostly equivalent to export PYTHONPATH=/path/to/new-working-env/lib/python2.4

Setting $PYTHONPATH is what activates the working environment. Your path will be fixed up. Optionally (and by default) site-packages won't be picked up, so the only thing you bring in from external sources is the standard library. I'd like to add some support for copying over packages from site-packages explicitly, probably with Egg-style annotation and links. E.g., copy_package psycopg version=2.0 will create psycopg-2.0.egg-info/, fix up your .pth file, etc.

It monkeypatches distutils to do this. I think that's nice and effective. It also monkeypatches setuptools, which isn't sustainable -- distutils isn't a moving target, so the patch is pretty stable, but setuptools is a different story. For setuptools it is changing the way scripts are generated, so that the working environment is hardcoded into the script (i.e., using new-working-env/bin/some-script automatically activates that working environment).

I'm still trying to figure this out, but I feel fairly good about working-env.py; I'm very interested in feedback.

Created 15 Mar '06
Modified 26 Apr '06

Comments:

One thing I'd point out as well is that I really like being able to use easy_install and setup.py commands without having to remember any special options. It's amazingly comforting to know you can't mess it up unless you actively try to mess it up. That's something this (and virtual-python.py) handles that several other options don't handle.

# Ian Bicking

I moved my current virtual-python install I had in $HOME out of the way, restarted my shell so bash would see /usr/bin/python, installed setuptools globally, and then did this:

$ rm -rf lib
$ python bin/working-env.py $HOME
Making working environment in /home/gldnspud
Creating lib/python2.4
Creating lib/python2.4/distutils
Creating lib/python2.4/setuptools
Creating file lib/python2.4/distutils/distutils.cfg
Creating file lib/python2.4/distutils/__init__.py
Creating file lib/python2.4/setuptools.pth
Creating file bin/activate
Creating file lib/python2.4/setuptools/__init__.py
Creating file lib/python2.4/site.py

Now I can't import setuptools:

$ source bin/activate
$ python
>>> import setuptools
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/gldnspud/lib/python2.4/setuptools/__init__.py", line 10, in ?
    execfile(os.path.join(setuptools_path, '__init__.py'))
NameError: name 'setuptools_path' is not defined

I checked sys.path and there's no setuptools anywhere in it. When I went back to the global python, it does pick up all the egg files from the pth files.

Any ideas on where to go from there?

# Matthew Scott

The setuptools stuff is hacky, and now I can't remember why I did it the way I did. That lib/python2.4/setuptools.pth file should contain the real path to setuptools, which gets used by setuptools/__init__.py. I put in a slightly better error message in this case (at least not NameError). I've been trying to figure out if working-env.py should actually install setuptools locally as part of the setup; it kind of seems like it should.

# Ian Bicking
I think that would be a logical way to do it. I was admittedly surprised that system-wide setuptools was required by working-env, since I was used to putting it locally into a virtual-python. I think it would be less hassle if it were possible to use working-env as a pre-setuptools bootstrap. Perhaps working-env could optionally/automatically run ez_setup.py within the newly-created environment?

I've switched back to virtual-python for now :)

# Matthew Scott

I have to second this comment. The whole reason for me running something like working-env or virtual-python is so that I don't have to mess with my virgin Python installation. If I have to install setuptools first, it kind of defeats the purpose....

# mbogosian
Your activate script is also kind of tricky (i.e., it looks like "activating" twice really means deactivate). If I may make a suggestion to rename it "toggle-wenv" or something to make that clearer.

Also, if PYTHONPATH is unset when it is run, the new PYTHONPATH will have a trailing ':' which may result in including the current working directory in the PYTHONPATH which may not be the best idea. It also may be harmless, but I don't think it hurts to be careful, especially if other modifying scripts acticipate a well-formed PYTHONPATH variable.

In addition, I might suggest using more unique names for your saved PYTHONPATH and PATH variables (this is a common trick and one probably doesn't want to risk a name clash). Here's a patch for working-env.py which includes all my suggestions (it's also a little friendlier to those WORKING_ENV paths which contain spaces:
--- working-env.py      2006-04-20 23:21:49.000000000 -0700
+++ working-env.py.new  2006-04-21 10:09:50.000000000 -0700
@@ -203,22 +203,22 @@
 script_dir = __WORKING__/bin/
 """

-files_to_write['bin/activate'] = """\
-# This file must be used with "source bin/activate"
+files_to_write['bin/toggle-wenv'] = """\
+# This file must be used with "source bin/toggle-wenv"
 # you cannot run it directly
-export WORKING_ENV=%(working_env)s
-if [ -n "$_OLD_WORKING_PATH" ] ; then
-    PATH="$_OLD_WORKING_PATH"
+export WORKING_ENV='%(working_env)s'
+if [ -n "$_WORKING_ENV_OLD_PATH" ] ; then
+    PATH="$_WORKING_ENV_OLD_PATH"
 else
-    _OLD_WORKING_PATH="$PATH"
+    _WORKING_ENV_OLD_PATH="$PATH"
 fi
 export PATH=$WORKING_ENV/bin:$PATH
-if [ -n "$_OLD_PYTHONPATH" ] ; then
-    PYTHONPATH="$_OLD_PYTHONPATH"
+if [ -n "$_WORKING_ENV_OLD_PYTHONPATH" ] ; then
+    PYTHONPATH="$_WORKING_ENV_OLD_PYTHONPATH"
 else
-    _OLD_PYTHONPATH="$PYTHONPATH"
+    _WORKING_ENV_OLD_PYTHONPATH="$PYTHONPATH"
 fi
-export PYTHONPATH=$WORKING_ENV/lib/python%(python_version)s:$PYTHONPATH
+export PYTHONPATH="$WORKING_ENV/lib/python%(python_version)s${PYTHONPATH:+:}$PYTHONPATH"
 """

 def make_working_environment(
I hope this helps a little.
# mbogosian
Oops....

I forgot to also change this line:
export PATH=$WORKING_ENV/bin:$PATH
To this (note the quotes):
export PATH="$WORKING_ENV/bin:$PATH"
# mbogosian

Ian Bicking: the old part of his blog

working-env.py

Comments: