two experiments in the Python environment. The first is virtualenv, which is a rethinking of virtual-python.py, and my attempt to move away from workingenv. It works mostly like virtual-python.py, and on systems where it works (not Windows, nor Framework Mac Python) I think it works considerably better than workingenv. No more not a setuptools' site.py, in particular. The basic technique is that it creates/copies a new python executable, and anything that uses that executable (including a script that references that executable with #!) will use that environment.
On the systems where it doesn’t work, I’m not quite sure what to do. The problem with the Mac is that sys.prefix is not determined by the location of the python executable, it’s hard-coded in some fashion. I asked about it on distutils-sig and got some response, but haven’t figured out any solution yet.
On Windows similarly sys.prefix is not determined by the executable location. What it’s determined by there I don’t know — the location of python25.dll, something in the registry? If I could figure it out, perhaps this could work there too — the existance of symlinks isn’t as important as it was with virtual-python.py.
If I can get these figured out, I think this will be a much happier experience than workingenv, and a somewhat friendlier experience than virtual-python.py. On non-Mac posix systems it works well right now.
The other experiment is in buildutils (downloadable with Mercurial): a new command python setup.py bundle, run in the application package you want to bundle. This creates a directory with all the dependencies of the application, and scripts that load up the appropriate dependencies. You can then ship the entire thing in a zip file as a runnable application that doesn’t require any installation except for unpacking.
Actually creating the bundle can be a little finicky, because easy_install has a tendency to prefer things on the local machine even though it shouldn’t. Probably it would be best to run this inside a virtualenv; when you are done you can also feel more confident that you’ve actually included all the dependencies (if you use --no-site-packages when creating the virtualenv).
Anyway, while both of these are a little incomplete I’m feeling optimistic about them, and I’m hoping intrepid souls can give feedback on how they work.
Update: virtualenv 0.8.2 is out, featuring Better Error Messages (and nothing else). Still doesn’t work on Mac Framework Pythons, or Windows. You’ll have to keep using workingenv there — but patches extremely welcome! Contact me if you are interested in supporting these platforms. It will involve some digging, but maybe we can just do the digging once for everyone.
Update 2: virtualenv 0.8.3 is out, featuring Windows!
Automatically generated list of related posts:
- Python Application Package I’ve been thinking some more about deployment of Python web...
- The Shrinking Python Web Framework World When I was writing the summary of differences between WebOb...
- Opening Python Classes So, I was reading through comments to despam my old...
- Python HTML Parser Performance In preparation for my PyCon talk on HTML I thought...
- A Python Web Application Package and Format (we should make one) At PyCon there was an open space about deployment, and...
I’m interested in the virtualenv script, as I’ve been using workingenv.py quite a lot, and I do keep getting bitten by the ‘not a setuptools site.py’ problem.
I think there is a typo in virtualenv.py – it tries to create the no-global-site-packages.txt file in the stdlib:
writefile(join(stdlib_dir, 'no-global-site-packages.txt'), '')This should probably just be:
writefile(join(lib_dir, 'no-global-site-packages.txt'), '')shouldn’t it?
Thanks for another great contribution.
[Ed note: updated in trunk]
It’s strange – virtualenv.py works fine with my from-python.org version of 2.5 – but you’re perfectly correct that it doesn’t play nice with the default-framework version. If I get a chance, I’ll see if I can unsnarl it – but I might also look into testing this out on Leopard instead (leopard had 2.5+ on it by default).
Curses, I take that back. Initial setup works fine, but as you point out on the mac, easy_setup still points stuff down into the frameworks library. I’ll look and see if there’s some tricky that can be done but for now I am back to using good old workingenv.py.
There is a minor problem in Windows
There are the steps
install Python 2.5 in c:\Python25
install setuptools (through ez_setup.py)
Where I tried to use the virtualenv.exe in Scripts, there is a problem
c:\Python25>c:\Python25\Scripts\virtualenv.exe ENV Creating ENV\Lib Creating ENV\Lib\site-packages Creating ENV\Scripts Copying C:\Python25\python.exe to ENV\Scripts ERROR: The executable ENV\Scripts\python.exe is not functioning ERROR: It thinks sys.prefix is ‘C:\Python25\ENV’ (should be ‘ENV’) ERROR: virtualenv is not compatible with this system or executable
c:\Python25>python lib\site-packages\virtualenv-0.8.3-py2.5.egg\virtualenv.py ENV Overwriting ENV\Lib\orig-prefix.txt with new content Copying c:\Python25\python.exe to ENV\Scripts Installing setuptools Processing setuptools-0.6c7-py2.5.egg Copying setuptools-0.6c7-py2.5.egg to c:\python25\env\lib\site-packages Adding setuptools 0.6c7 to easy-install.pth file Installing easyinstall-script.py script to c:\Python25\ENV\Scripts Installing easyinstall.exe script to c:\Python25\ENV\Scripts Installing easyinstall-2.5-script.py script to c:\Python25\ENV\Scripts Installing easyinstall-2.5.exe script to c:\Python25\ENV\Scripts
Installed c:\python25\env\lib\site-packages\setuptools-0.6c7-py2.5.egg Processing dependencies for setuptools==0.6c7 Finished processing dependencies for setuptools==0.6c7 Processing setuptools-0.6c7-py2.5.egg Removing c:\python25\env\lib\site-packages\setuptools-0.6c7-py2.5.egg Copying setuptools-0.6c7-py2.5.egg to c:\python25\env\lib\site-packages setuptools 0.6c7 is already the active version in easy-install.pth Installing easyinstall-script.py script to c:\Python25\ENV\Scripts Installing easyinstall.exe script to c:\Python25\ENV\Scripts Installing easyinstall-2.5-script.py script to c:\Python25\ENV\Scripts Installing easyinstall-2.5.exe script to c:\Python25\ENV\Scripts
Installed c:\python25\env\lib\site-packages\setuptools-0.6c7-py2.5.egg Processing dependencies for setuptools==0.6c7 Finished processing dependencies for setuptools==0.6c7
Thanks for yout great job
Agustin
[Ed. note: message fixed in 0.8.4]
Fun stuff. You should check out [how virtualenv is being used with zopeproject](http://baijum81.livejournal.com/22191.html).
One nit: the include directory doesn’t get copied by virtualenv. So eggs which build some C extensions (e.g., numpy) will not install with the easy_install copied over by virtualenv.
[Ed. note: the include directory is copied over in virtualenv trunk]
Good work, Ian, as always. Pylons is deciding which virtual Python environment to recommend/HOWTO in its tutorials. (Can HOWTO be a verb?) I have been recommending workingenv, and James G has started promoting zc.buildout. My main concern is the proliferation of approaches and the instability if we keep changing our minds. Kind of like the problem of too many web frameworks. I’d like to see a Python standard emerge but I guess it’s too soon for that.
I hope the incompatibility with Apple’s Python gets resolved soon, since portability to all platforms is an important factor for some people choosing a virtual environment.
Hi again,
(Where should virtualenv issues be reported, by the way? Is the distutils list a better place?)
I don’t think virtualenv (0.8.4) works if you don’t have an internet connection and you don’t already have setuptools installed. This is because it runs ez_setup.py, and ez_setup.py always downloads setuptools unless it finds the appropriate egg in os.curdir (ie. it ignores the egg mentioned on the command line)
A possible workaround would be to change directory to the support-files directory before running ez_setup.py. Otherwise ez_setup.py itself needs to be changed.
Thanks again,
Simon
activate.bat doesn’t seem to work in windows much. I was going to diagnose the problem further but looking at the script I can see this has no hope in hell of working and just needs an overhaul. All the percent signs and backslashed are doubled, the _OLD_VIRTUAL_PROMPT is used before it is set, and it sets the path to %VIRTUALENV%\bin, even though python and easyinstall are being installed in Scripts … need I go on? I would prefer to see it installing things into bin/ for cross-platform compatibility, too, instead of scripts.
C:\work>test_env\Scripts\activate.bat (test_env) %PROMPT%deactivate ‘deactivate’ is not recognized as an internal or external command, operable program or batch file.
(test_env) %PROMPT%python ‘pyth\on’ is not recognized as an internal or external command, operable program or batch file.
(test_env) %PROMPT%easy_install ‘easy_install’ is not recognized as an internal or external command, operable program or batch file.
[Ed. note: the include directory is copied over in virtualenv trunk]
What happened?
Python 2.5.1 straight out of Ubuntu Feisty repo; fresh Feisty; just testing. include is over in /usr/include/python2.5. Is there a command line option to point out the correct directory? I guess I can manually copy this over? Is this a Python 2.5 thing?
Chris: in the trunk I switched to a monkeypatch of distutils.sysconfig, which should point distutils at your site-wide include directory. This should make it unnecessary to copy over include.