lucumr

The fiinal version of the Jinja2 Django-inspired template engine was just released. It comes with tons of improvements over the older Jinja1 engine and breaks backwards compatibility to some point over the old Jinja1 engine. It’s packaged as a separate package so that you can use both right next to each other for an easier transition.

Compared to Jinja1 it provides tons of new features:

• Dynamic inheritance. It’s now possible to use dynamic inheritance which means that the name of the master template expanded at render time. This makes it easy to switch between different designs.
• Improved macro and import system. Macros can be called with keyword arguments now, are much more lightweight and got their own import system which makes templates easier to understand. The import syntax follows the Python one with small adjustments.
• Heavily improved for-loops. Loops can be filtered now and the length is calculated lazily. This makes it possible to iterate over generators with an unknown length in a much more efficient way.
• Improved behavior of undefined values. Jinja1 had a very silent undefined behavior. If a variable was undefined you were able to call it without getting errors or access any attribute. Jinja2 ships three undefined types that make it easier to debug templates. The default undefined types allows you to print the undefined variables (which when printed outputs nothing) and loop over it (works like iterating over an empty list). However every other operatation raises an UndefinedError. Additionally there an undefined type with the same behavior but it prints the name of the variable or attribute missing if it’s printed. The third builtin undefined type is the strict type which doesn’t allow any operation except of testing if it’s undefined which is the closest you can get to the default Python behavior.
• Improved sandbox. The sandboxed environment if enabled is now easier to use and more secure. In the default configuration everything starting with an underscore is considered insecure so it’s not needed any longer to mark those attributes explicitly. It’s also a lot faster now and easily modifyable by subclassing.
• Line statements. Jinja2 allows to to specify a line statement prefix that marks a whole line as statement. This concept is inspired from Mako and Cheetah and allows very clean templates in many situations. If a line starts with the prefix character (for example “#”), everything up to the end of the line is handled as block.
• Easier API. The API is much easier to use now on nearly every level. The loaders got refactored so that you only have to provide a single methods, filters and tests are normal Python functions now.
• Automatic escaping. Jinja2 comes with optional automatic escaping compatible to Pylons and Genshi
• Improved i18n support. Jinja2 integrates into Babel now which makes internationalizing web applications a charm.
• Extension interface. Jinja2 provides a documented interface that can be used to extend the template engine. The interface could even be used to create Jinja inspired template engines on top of the existing compiler interface with a completely different syntax.

Get it while it’s hot from the Cheeseshop (or PyPI for those of you who prefer the new name).

Every once in a while I’m really impressed by a library I stumble upon. A while back that was virtualenv, now i stumbled upon fabric. I was using capistrano for a project I was working on which was kinda okay but somehow I wasn’t sold to it.

Yesterday however apollo13 stumbled upon fabric which is capistrano just in Python, with a working put command and less annoying in general.

In combination with a custom virtualenv bootstrapping script Python web application deployment is a charm. One “fab bootstrap” later the servers are creating a virtual python environment, compiling all dependencies, checking out all eggs and initializing the application environment. Updates are just one “fab production deploy” away.

And the best part is that fabric is not limited to Python. You can use it to deploy anything you can control over ssh.

Here an example fabfile (the file that controls the deployment)

set(
    fab_hosts = ['srv1.example.com', 'srv2.example.com']
)

def deploy():
    """Deploy the latest version."""
    # pull all changes from mercurial and touch the wsgi file to
    # tell the apache to reload the application.
    run("hg pull -u; touch application.wsgi")

def bootstrap():
    """Asks for a list of servers and bootstrapps the application there."""
    set(fab_hosts=[x.strip() for x in raw_input('Servers: ').split()])
    run("hg clone http://repository.example.com/application")
    local("./generate-wsgi-file.py > /tmp/application.wsgi")
    put("/tmp/application.wsgi", "application.wsgi")

Saved as fabfile.py “fab bootstrap” then asks for some servers and bootstraps the application there, after changes in the repository you can “fab deploy” the latest version. Of course that’s just a very basic made up example, but it shows how you can use fabric.

I’m using makefiles currently to execute common tasks for various Python projects (like releasing code, resting unittests and much more), I suppose fabric could also do that for me. And that would have the advantage that it works for windows users too.

Deploying Python applications is not that hard, but there are some pitfalls. The main problem everyone stumbles upon sooner or later is that sys.modules is a singleton. Modules are cached there which the effect that there can be exaclty one version of a library loaded at the same time per interpreter. There is no way around that and it’s fine. However there is another problem: Everyone installs everything into a system wide singleton called site-packages. Sooner or later everything ends up there, and while pkg_sources supports switching between versions this is very unconfortable and requires extra hackery.

My plan to solve these problems always was creating a folder, putting all libraries into that folder and site.addsitedir()ing that folder in the file that starts up the application. But especially for web applications that’s a dull repetitive task and easy_install and other libraries don’t play well with this approach.

Fortunately Ian Bicking wrote a really cool module called “virtualenv” that creates virtual Python environments. And it does that in a very cool manner. The following example shows how we run LodgeIt on our server:

First you need to install virtualenv. This assumes you already have easy_install:

mitsuhiko@nausicaa:~$ sudo easy_install virtualenv
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Best match: virtualenv 1.1
Downloading virtualenv-1.1-py2.5.egg
Processing virtualenv-1.1-py2.5.egg
creating /opt/local/lib/python2.5/site-packages/virtualenv-1.1-py2.5.egg
Extracting virtualenv-1.1-py2.5.egg to /opt/local/lib/python2.5/site-packages
Adding virtualenv 1.1 to easy-install.pth file
Installing virtualenv script to /opt/local/bin

Installed /opt/local/lib/python2.5/site-packages/virtualenv-1.1-py2.5.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Now we can create a folder folder the application. In our case the folder is called after the domain where the web application runs:

mitsuhiko@nausicaa:~$ virtualenv paste.pocoo.org
New python executable in paste.pocoo.org/bin/python
Installing setuptools.......................done.

The virtual python is now available in that folder and we can switch into that environment by sourcing the activate file:

mitsuhiko@nausicaa:$ cd paste.pocoo.org/
mitsuhiko@nausicaa:~/paste.pocoo.org$ . bin/activate
(paste.pocoo.org)mitsuhiko@nausicaa:~/paste.pocoo.org$

From that point onwards “python” and “easy_install” point to the executables in the ~/paste.pocoo.org/bin folder. The prompt is prefixed with the name of the virtual environment so that we are reminded that “python” is our virtual Python. We can now install all the dependencies:

$ easy_install SQLAlchemy==0.4.4 Jinja2 Werkzeug Pygments

Half a minute later the virtual python interpreter will now have the libraries available and we’re ready to add out application:

$ hg clone http://dev.pocoo.org/hg/lodgeit-main

Lodgeit itself doesn’t have a setup.py file but we can just cd into the folder and run the testing server to test if it works:

$ cd lodgeit-main/
$ python runlodgeit.py runserver
 * Running on http://localhost:5000/
 * Restarting with reloader...

That works perfectly for scripts invoking “python” directly such as standalone servers like CherryPy but won’t work for mod_wsgi where the interpreter is created by mod_wsgi and doesn’t point to the virtual environment.

There are two ways to solve the problem. For mod_wsgi 2.0 you can run the web application in daemon mode and pass it the path to the site packages in the apache config:

WSGIDaemonProcess lodgeit python-path=/path/to/site-packages

Alternatively you can go into the .wsgi file and add the site-packages by hand:

import site
site.addsitedir("/path/to/site-packages")

The site-packages of the virtual environment are located at /path/to/virtual/env/lib/python2.X/site-packages.

Even though I was not interested in virtualenv in the beginning I changed my mind because it’s straightforward to use and easy to maintain. More information about virtualenv is available in the Cheeseshop and the announcement blog post in Ian’s blog.

I was reading a thread on ruby-forum.com about Python that said that the whitespace-sensitivity of Python is from hell or something. There are people from every programming language that can rant about Whitespace sensitivity in Python but clearly not Ruby programmers. Why? Because Python doesn’t care about Whitespace at all. The only thing that somewhat has to do with whitespace is the indentation that the lexer convers into indent and outdent tokens. But after that, no whitespace any more, the parser doesn’t know anything about that.

That however is not true for Ruby! foo[42] does a completely different thing than foo [42]. The first calls foo without argument and calls the [] method of the return value with 42 as argument, the latter calls foo with [42] as Argument which happens to be an Array with one element. But there are more examples.

Take this example:

foo = 23
def bar
  42
end

puts bar/foo

That prints “1″.

However take this minor modification:

foo = 23
def bar
  42
end

puts bar /foo

Now this gives you an error that the regular Expression literal is unterminated. That’s what I call whitespace sensitivity :)

You’re wonderhing why I’m using a method for “bar” and not a locale variable? Because the parser keeps track of all assigned local variables or methods (Not sure what exactly it does) and the syntax ambiguities are resolved that way.