lucumr

Deploying Python applications is not that hard, but there are some pitfalls. The main problem everyone stumbles upon sooner or later is that sys.modules is a singleton. Modules are cached there which the effect that there can be exaclty one version of a library loaded at the same time per interpreter. There is no way around that and it’s fine. However there is another problem: Everyone installs everything into a system wide singleton called site-packages. Sooner or later everything ends up there, and while pkg_sources supports switching between versions this is very unconfortable and requires extra hackery.

My plan to solve these problems always was creating a folder, putting all libraries into that folder and site.addsitedir()ing that folder in the file that starts up the application. But especially for web applications that’s a dull repetitive task and easy_install and other libraries don’t play well with this approach.

Fortunately Ian Bicking wrote a really cool module called “virtualenv” that creates virtual Python environments. And it does that in a very cool manner. The following example shows how we run LodgeIt on our server:

First you need to install virtualenv. This assumes you already have easy_install:

mitsuhiko@nausicaa:~$ sudo easy_install virtualenv
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Best match: virtualenv 1.1
Downloading virtualenv-1.1-py2.5.egg
Processing virtualenv-1.1-py2.5.egg
creating /opt/local/lib/python2.5/site-packages/virtualenv-1.1-py2.5.egg
Extracting virtualenv-1.1-py2.5.egg to /opt/local/lib/python2.5/site-packages
Adding virtualenv 1.1 to easy-install.pth file
Installing virtualenv script to /opt/local/bin

Installed /opt/local/lib/python2.5/site-packages/virtualenv-1.1-py2.5.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Now we can create a folder folder the application. In our case the folder is called after the domain where the web application runs:

mitsuhiko@nausicaa:~$ virtualenv paste.pocoo.org
New python executable in paste.pocoo.org/bin/python
Installing setuptools.......................done.

The virtual python is now available in that folder and we can switch into that environment by sourcing the activate file:

mitsuhiko@nausicaa:$ cd paste.pocoo.org/
mitsuhiko@nausicaa:~/paste.pocoo.org$ . bin/activate
(paste.pocoo.org)mitsuhiko@nausicaa:~/paste.pocoo.org$

From that point onwards “python” and “easy_install” point to the executables in the ~/paste.pocoo.org/bin folder. The prompt is prefixed with the name of the virtual environment so that we are reminded that “python” is our virtual Python. We can now install all the dependencies:

$ easy_install SQLAlchemy==0.4.4 Jinja2 Werkzeug Pygments

Half a minute later the virtual python interpreter will now have the libraries available and we’re ready to add out application:

$ hg clone http://dev.pocoo.org/hg/lodgeit-main

Lodgeit itself doesn’t have a setup.py file but we can just cd into the folder and run the testing server to test if it works:

$ cd lodgeit-main/
$ python runlodgeit.py runserver
 * Running on http://localhost:5000/
 * Restarting with reloader...

That works perfectly for scripts invoking “python” directly such as standalone servers like CherryPy but won’t work for mod_wsgi where the interpreter is created by mod_wsgi and doesn’t point to the virtual environment.

There are two ways to solve the problem. For mod_wsgi 2.0 you can run the web application in daemon mode and pass it the path to the site packages in the apache config:

WSGIDaemonProcess lodgeit python-path=/path/to/site-packages

Alternatively you can go into the .wsgi file and add the site-packages by hand:

import site
site.addsitedir("/path/to/site-packages")

The site-packages of the virtual environment are located at /path/to/virtual/env/lib/python2.X/site-packages.

Even though I was not interested in virtualenv in the beginning I changed my mind because it’s straightforward to use and easy to maintain. More information about virtualenv is available in the Cheeseshop and the announcement blog post in Ian’s blog.

This entry was posted on Saturday, July 5th, 2008 at 2:16 pm and tagged as python. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.