A lot of people have found workingenv useful, but it’s always been a bit fragile. If you’ve seen the .../site.py is not a setuptools-generated site.py; please remove it. message, you know what I mean.
For a while I tried to refactor and improve workingenv, but it didn’t go very well. So I decided to ditch it completely and revisit the ideas of virtual-python.py. That script works by copying the Python executable, and in doing so change sys.prefix — it’s pretty consistent that all other paths in Python derive from sys.prefix.
The result is virtualenv, which I think is now featureful enough for general use. It might still be buggy, but it’s worked well for some of us, and I expect the bugs to all be much easier than they were in workingenv.
Unlike virtual-python.py, virtualenv works on Windows and in the latest release also Mac framework builds. It also handles site-packages better, so you can manage some of your packages using packages from your OS distribution (e.g., debs or rpms), while also installing environment-local packages.
So, in summary: use virtualenv, don’t use workingenv. And if you were using the --requirements option to workingenv, the package PoachEggs lets you do a similar batch installation after you’ve created the environment.
Automatically generated list of related posts:
- pyinstall is dead, long live pip! I’ve finished renaming pyinstall to its new name: pip. The...
- Monkeypatching and dead ends Bill de hÓra and then Patrick Logan picked up on...
- 2 Python Environment Experiments two experiments in the Python environment. The first is virtualenv,...
To get easy_installed libraries in a working-env to show up in a mod_python deployment, I’ve been using the activate_workingenv.py script from the Pylons wiki (for example I run this in the settings.py file in Django apps, alright so it’s a little weird). Any other solutions, e.g. involving PythonImport, totally failed to work for me.
It sounds like this’ll be easier with virtual-env – what do you think it might look like to have, say, two mod_python apps using different virtual-envs?
Well, it’ll be a little tricky. It would be nice if there was a clear way to handle this in mod_python or mod_wsgi. Maybe we’ll figure something out.
Anyway, in mod_* you can’t really change the Python interpreter to change the environment, so the normal way of “activating” the virtual environment won’t work (i.e., you can’t change your PATH or your interpreter). But that really matters most for installation and management of packages, which you probably will be doing from the command-line anyway.
If you just want to get the libraries from the virtual environment, not install anything,
import site; site.addsitedir('/path/to/env/lib/pythonX.Y')
should do it.The best you can do with mod_python and mod_wsgi is point the whole Apache server at a different virtualenv directory. This must though be for the exact same Python version that mod_python and mod_wsgi was originally compiled for as it is still using the embedded Python interpreter that it was compiled with and is ignoring the python executable in the virtualenv directory.
This all means that you cannot have different applications running inside of mod_python and mod_wsgi using different virtualenv environments. This is the case even for mod_wsgi daemon mode, as the Python interpreter is only initialised once in the Apache parent process and the daemons all inherit that instance.
If using mod_python you can point it at a different virtualenv by setting and exporting PYTHONEXECUTABLE environment variable in the Apache envvars file. In mod_wsgi you can use the WSGIPythonExecutable directive in the Apache configuration file. The WSGIPythonExecutable directive only sets the reference point though so Python can find where the library directory is actually located.
I have some ideas for how to better support virtualenv with daemon mode of mod_wsgi, but it effectively involves creating a hybrid system where mod_wsgi would run up separate Python process which it would transparently manage and direct to load up and run mod_wsgi specific modules to handle the in Python process side of the daemon mechanism. Note sure when I will get around to it though as want to get transient daemon process support added first.
Other than that I really need to research what the perceived problems are with workingenv and see whether it is still an adequate solution for more constrained environment of mod_wsgi.
Graham: I don’t think it’s as complicated as all that. All the complexity of workingenv and virtualenv are around distutils and having scripts use the right libraries and paths. But just getting the libraries loaded on sys.path is pretty simple. As long as you aren’t installing packages from inside the mod_wsgi environment, all you have to do is get the libraries loaded. And all you have to do to load the libraries is that one line of code I gave (well, two lines).
That still means you have to have separate Python environments with different sys.path’s. From what I understand, you can have that with mod_wsgi, as there are separate interpreters. It’s just that all the interpreters are forks of a single original interpreter. And that’s fine.
Agree, and using daemon mode of mod_wgsi means you can isolate applications wanting to use incompatible versions of C extension modules for Python.
In respect of hybrid mode am actually looking a bit further in trying to address one of the other complaints about solutions such as mod_python and mod_wsgi for embedding Python in Apache. That is the restriction that one must use the version (major/minor) of Python that the Apache module was compiled for and you cant use anything else. Some people have been quite critical of this as they want to be able to use different versions of Python for different applications. Yes they could use mod_proxy with Python applications managed by separate supervisor, but am thinking about how one could support this whereby Apache was used as the supervisor, thereby avoiding the need to install a separate supervisor system with all its dependencies.
If support for this was ever added, then it becomes very simple to use virtualenv as in designating the mod_wsgi daemon process you define the location of the python executable in the virtualenv and instead of just a fork for daemon and using internal mod_wsgi server support, it does a subsequent exec and runs up mod_wsgi server support in the separate Python process.
So, I don’t discount that doing necessary sys.path fixups is easy, and that someone has written instructions on how to do it in mod_wsgi already indicates it does work, I just still haven’t had time to sit down and look at it all properly, but I will. :-)
Great work Ian.
Regarding activating a workingenv/virtualenv, activate_workingenv.py is really over engineered. I faced this problem two weeks ago and ended up with the very same solution you gave (site.addsitedir), it’s simple and I can confirm that it works really great (I’m using it inside postgres for some plpythonu functions).
Thanks for the tip about site.addsitedir, I guess I didn’t realize that active_workingenv.py even solved a different problem.
activate.bat doesn’t seem to work in windows much. I was going to diagnose the problem further but looking at the script I can see this has no hope in hell of working and just needs an overhaul. All the percent signs and backslashed are doubled, the _OLD_VIRTUAL_PROMPT is used before it is set, and it sets the path to %VIRTUAL_ENV%\bin, even though python and easy_install are being installed in Scripts … Aside from that, I would prefer to see it installing things into bin/ for cross-platform compatibility, too, instead of Scripts.
C:\work>test_env\Scripts\activate.bat
(test_env) %PROMPT%deactivate
‘deactivate’ is not recognized as an internal or external command, operable program or batch file.
(test_env) %PROMPT%python
‘python’ is not recognized as an internal or external command, operable program or batch file.
(test_env) %PROMPT%easy_install
‘easy_install’ is not recognized as an internal or external command, operable program or batch file.
[Note: fixed in trunk]
Has anyone managed to use this (or workingenv.py) for running a Windows service? I’m not sure how to get the service to use the proper python.exe.
I’ve not been able to use a ‘python setup.py develop’ with virtualenv on macosx, something that used to work with workingenv.py and a ‘user mac os x setup’ of easyinstall. It failes with:
[Errno 13] Permission denied: ‘/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/test-easy-install-27420.write-test’
Why does it want to write to the framework directory? Other than that, easy_install works like it should. Great work!
Just to be sure, I double-checked
python setup.py develop
on Linux, and it works fine there. Of course you must use the right Python executable for it to work at all. Otherwise it’s possible that there’s something funny there on Macs in particular, as there is some differences in the code paths there.Problem installing mercurial with virtualenv v0.9.1 on ubuntu 7.04/python2.5:
[I believe this is just a conflict between easyinstall and Mercurial, as Mercurial is subclassing an object that easyinstall monkeypatches]
> conflict between easyinstall and Mercurial
It is looking that way… I have a workingenv based setup with a successful egg install of mercurial, but I cannot successfully install it again w/o seeing the same problem – it is likely/possible that easy_install was upgraded at some point in the last week or two.
As an aside are there any issues with copying the installed packages from a workingenv-based ‘lib/python2.5′ to a virtualenv-based ‘lib/python2.5/site-packages’ and updating the easy_install.pth accordingly?
Problem installing virtualenv on MacOSX 10.3.9 (PPC), python 2.5.1 Framework build:
in
create_environment
the python interpreter is copied and installed. The necessaryinstall_name_tool
executable is available from the Apple developer tools package only (which is quite a big dependency of 1GB in size and you’ve also got to have some spare time and a CD/DVD around to install it).Do you think this dependency could be worked around ?
Tried to use virtualenv on MacOsX 10.4 and 10.5 (with preinstalled easyinstall. I had to run it with admin rights (In order to have write permissions for /usr/local/bin and /Library/Python). At first glance everything but one thing looks good: Seems like easyinstall is missing from the bin dir in the virtualenv. Am i missing something
I also am seeing the behavior that gkamp (in above post) is seeing. I’m running Leopard (os x 10.5). Also, before using virtualenv I had to install XCode, (as d2m mentions above) is this behavior required/expected? Thanks for any info.
FWIW, I have also had various issues on MacOS X with virtualenv and packages still wanting to write into system directories where root privileges would be required. It got frustrating enough that I gave up and went back to using what I call a poor man’s Python virtual environment. That is, use the inbuilt feature of Python to define the PYTHONHOME environment variable to be where the Python installation is installed. You can find more details on my [blog](http://blog.dscpl.com.au/2007/11/poor-mans-python-virtual-environment.html) about it.
These Mac problems should be fixed in the newest release.
Thanks Ian, looking forward to it.
Gave it a try yesterday evening. Looks good to me.
Version 0.9.2 still doesn’t work on MacOS X 10.4 with OS supplied Python 2.3.
This time it appears to be because ‘set’ was used in site.py when Python 2.3 doesn’t support ‘set’. Using ‘python -v’ with easy_install yields:
BTW, any chance you can document –no-site-packages option on PyPi site where documentation is for virtualenv. I see this as an important option when using mod_wsgi for shared hosting as in that sort of situation you want to base the underlying Python environment used by mod_wsgi off a virtual environment which has an empty site-packages. Specific applications would then add on top using their respective virtual environments using site.addsitedir() based mechanisms. If however the application owners didn’t use –no-site-packages option and a required module was in global site-packages, and thus easy_install didn’t add a copy to their own virtual environment, that module can then show up as missing when their application is run since mod_wsgi will not be using the globally installed Python environment.
On Win32, packages that distutils need to build C modules for as part of easy_install within virtualenv require (at least for me), the workaround of copying c:\Python25\libs and c:\Python25\include – since there is not an include symlink like there is on virtualenv on unix. This was true with both MinGW and VC++ 2003 environments; this may be something to note in documentation. To duplicate within the virtual environment ‘easy_install zope.interface’ – this will attempt to build one C module in zope.interface after fetching the package, which fails prior to the workaround, but succeeds fine after copying the libs and include directory.
I have a problem creating a virtual environment with virtualenv-0.9.2 on Cygwin. The problem occurs when the Python interpreter/executable should be copied.
The problem seems to be that Cygwin mostly hides the “.exe” file extension (or it is not considered for
sys.executable
in the Python port). But the file extension is needed forshutil.copyfile()
to work.QUICKFIX:
The other issue with Python 2.3 which had forgotten and only remembered when I tried to run in on a different MacOS X system, is that Python 2.3 doesn’t have the ‘subprocess’ module. Thus it is necessary to copy this from a new version of Python and install it by hand before running virtualenv.
I think the Windows deactivate.bat script still has a couple of bugs. It contains this snippet:
That’s three different spellings of
_OLD_VIRTUAL_PATH
:If I replace them all with
_OLD_VIRTUAL_PATH
, everything seems to work properly.I’m still having problems with certain packages on OS X 10.5, particularly Paste, but there may be others out there. I am trying to get TG2 into it’s own virtualenv so I don’t nuke my current development projects. Included is the result of easy_installing Paste (with the output from the end of an SQLAlchemy install to show what the paths should be):
It seems that Paste is still being used from the global site-packages dir, even if I create the environment with the
--no-site-packages
option.I encountered problems while testing virtualenv under cygwin for an application port to windows. Here, you ll find a patch to make it working (it seems to). http://distfiles.minitage.org/public/externals/minitage/patches/virtualenv.win.diff
To be honnest, i’m not a windows user and do no want to.
Please remove the ‘not’ on line 400 of virtualenv.py Maybe I’m too tired, but it sure seems assbackwards.
Should virtualenv override the PYTHONPATH environment variable as well in the activate script?
Hi,
Virualenv is great, and many thanks for this essential tool.
I have a Mac 0S 10.5 + XCode, Python 2.5 that ships with the MacOS bundle and Python 2.4 for Zope/plone works. When installing a virtualenv for Python 2.5 I need to :
So far, so good, but if I need after this a virtualenv for Python 2.4, I need to re-install virtualenv for Python 2.4.
So this would be cool if installing virtualenv creates the “virtualenv” and “virtualenv-2.x” scripts (just as done for easy_install), unless there’s a feature I didn’t get ;)
You can also do
virtualenv -p python2.5 ENV
Me really stupit. Should have RTFM. Sorry and thanks again.