lucumr

I now uploaded the documentation for Jinja2 to the website for those of you who are eager and want to play with it :-) On jinja.pocoo.org you have now the choice to chose between Jinja1 and Jinja2.

The new docs are powered by Sphinx and Jinja2 with a custom templating bridge.

Read the documenation.

Some of you might have noticed that pocoo.org redirects to the development center now. No direct reference to the old pocoo project any more. The reason for this is that a part of the pocoo team and the ubuntuusers webteam is working on a new software that combines wiki, forum, planet, pastebin and blog into one big portal software. It’s not yet open sourced but it will become a pocoo project, most likely licensed under the GPL around June, with a semi stable release at the end of 2008. I’ll post some details the next week I think.

But why am I writing about this? Mainly because the work on inyoka (the name of the pocoo successor) showed some weaknesses in Jinja and we did some brainstorming to come up with really cool improvements for Jinja that resulted in a complete rewrite. Because we love backwards compatibility these changes will go into a package called “jinja2″ and both Jinja 1.x and the Jinja 2.x will get updates in the future. Also Jinja 2 is not yet released and you shouldn’t expect a release anytime soon, if you want to play with it you will need the current hg version.

But what are the changes to Jinja 1 and why is 2 better? What we noticed when working on inyoka is that quite a lot of display logic code goes into the templates. Quite a lot of logic is in those templates (not application logic though). This isn’t a bad thing actually but it showed that especially for large templates with many dynamic parts such as loops or even variable tags Jinja doesn’t have the performance we wished it would have. Especially if templates become more complex quite a lot of CPU time is spent there are there was room of improvement in Jinja. Unfortunately a design mistake made it impossible to do any optimization there and I was forced to change the scoping rules to something saner. I would say for 98% of the templates the changed scoping won’t make any difference as many have avoided the side effects of the old scoping anyways. But still, it’s a change that will break things, so Jinja 2.

But right after I decided to break backwards compatibility there was more room for improvement which resulted in some real kickass features. Finally filters are simple python functions again and not factories any longer. They now also support keyword arguments, a feature on the wishlist for nearly as long as Jinja exists. Dynamic inheritance is now something that works without pain too and the template inclusions where simplified. And that also means that templates can be included into namespaces now. So {% include macros = "helpers/macros.html" %} gives you an object with you can print or where you can access all the macros or defined variables etc.

What really makes me happy is that on the surface few things have changed. Most of the implementation details are hidden in the compiler and are not visible on the templates. For example that Jinja 2 knows five different ways to iterate over sequences and that stuff is not visible in templates at all. By changing the scoping rules everything of the user friendliness is still in place.

Other nice changes are that you can now filter for loops while iterating over them like list comprehensions in python and that the lexer knows line statements now. Line statements are lines prefixed with some optional whitespace and one or more marker characters. Everything after that until the end of the line is a single statement. That was shamelessly stolen from Mako and Cheetah and is something you have to enable explicitly, but I kinda like it for some template scenarios:

# for item in seq
  * {{ item }}
# endfor

Another very neat idea we implemented in Jinja 2 (wasn’t my idea unfortunately ^^) is that we broke the template scoping into globals and locals. Globals are known at compile time and render time, locals are known at render time only. That way Jinja can automatically evaluate parts of the template at compile time. The idea is that in many templates you have data that doesn’t change that often. A good example is a forum software. The list of forums changes seldom, at ubuntuusers about twice a year but every information that lives longer than fifteen minutes or so can be treated the same way. Anyways. The list of forums is information that doesn’t change, if we can give it Jinja as global variable for the template, Jinja can do quite a lot of compile time optimizations. In the best case the whole list of forums is one giant static block not processed at render time at all.

That keeps the templates designer friendly as the person who designs the template doesn’t have to know anything about that but gives the application developer a simple way to optimize performance. This approach however has some rough edges we have to polish.

And the last important change is that the sandbox is optional now and even disabled by default. Most users were not using it anyways and there is no need to put that into the system by default. It’s however a built in functionality and will get some improvements as well. The new sandbox will give a better control over what’s secure and what not.

How fast is it currently? I really don’t want to throw pointless numbers around but for the test table we’re using currently the speed without sandbox is more than comparable with mako and a lot, lot faster than django’s templates. But please take those numbers with a shovel of salt as we’re talking about an unreleased project here and a more than biased benchmark for one particular use case. Jinja tries to make templating as simple as possible and not as fast as possible.

Making a template engine that’s fast is incredible simple. But making a template engine that doesn’t suck and performs well is a lot harder.

Georg already blogged about the new ast to code compilation feature of Python so I won’t cover that any more. Basically the old compiler package is nor surpassed by python’s real internal AST which is freaking awesome :)

The only thing that is missing is some sort of high level interface to it which makes manipulating and debugging it easier. The motivation behind it is that template engines such as Genshi, Jinja or Mako which all operate on the AST don’t have to write that much boilerplate code just to modify small pieces of the AST.

I don’t know if this module will make it into the standard library but even if not it will surely be available via the cheeseshop. So what does it do? It provides classes and utility functions to manipulate or traverse the AST in a way that is actually useful for real world applications. For example with the transformer it’s not possible to replace and remove nodes. Additionally it’s possible to generate python sourcecode from an AST to simplify debugging.

Here a small example of a transformer that brings Genshi like Loop behavior to a piece of Python sourcecode:

class GenshiSemantics(ast.NodeTransformer):

    def context(self, method):
        return ast.Expr(ast.Call(
            ast.Attribute(ast.Name('data', ast.Load()), method, ast.Load()),
            [], [], None, None
        ))

    def visit_For(self, node):
        self.generic_visit(node)
        return [self.context('push'), node, self.context('pop')]

    def visit_Name(self, node):
        self.generic_visit(node)
        return ast.Subscript(
            ast.Name('data', ast.Load()),
            ast.Index(ast.Str(node.id)),
            node.ctx
        )

When applied this piece of code:

seq = [1, 2, 3, 4, 5]
item = 42
for item in seq:
    print 'inside', item
print 'outside', item

becomes

data['seq'] = [1, 2, 3, 4, 5]
data['item'] = 42
data.push()
for data['item'] in data['seq']:
    print 'inside', data['item']
data.pop()
print 'outside', data['item']

The sourcecode of the module is in the Pocoo sandbox as usual. Unfortunately there is a bug in the initialization function of the ASTs right now so you will need a patch for it. And obviously this requires Python 2.6.

The full sourcecode for the example above can be found here.

So Python 3 is getting ready and looking at all the libraries I maintain I was really afraid of the final release. Looking at all those language changes it sounded like a horrible job to do. But there is that nice 2to3 script which should transform sourcecode from python2 to python3 in a semiautomatic manner. And what should I say? It kick ass.

Getting the current hg head of Jinja running on top of Python 3 was a matter of roughly 15 minutes. The only things I had to change was adding a missing sys import that came from the intern -> sys.intern translation and fixing a couple of sort() calls. Sort now requires you to use a key function. But that was easy to solve and it was only there for python 2.3 backwards compatibility anyways. The other minor thing I had to change was unicode behavior. Previously Jinja has had a env.to_unicode function that coerced bytestrings and unicode strings into unicode strings depending on the encoding setting on the environment. With Python 3 everything will use unicode internally so no need to have an environment charset anways so I replaced env.to_unicode with str everywhere. Theo the other unicode change was replacing file(foo, 'r').read().decode(charset) (pseudocode) with file(foo, encoding=charset).read().

That’s it! Well. I must admit Jinja is by far the easiest templating language to port from the trio of templating languages I use (Jinja, Mako, Genshi) as it has it’s own parser and does not relay on the now removed compiler module which was superseded with the new _ast with a different structure. So it clearly depends on what you’re using in your library how the quality of the conversion will be. For most libraries 2to3 will do a very good job.

Oh. What it also doesn’t convert are C extensions :) I don’t know if the Jinja traceback tools or speedups module work in Python 3 too as they are optional and I was too lazy to compile them, but that’s another thing you have to keep in mind. Also I was unable to run the Jinja testsuite to run everything as py.test doesn’t work on Python 3 so far.

But I’m very happy. If the process stays that simple my forecast of not using Python 3 before 2012 may be wrong. So, go on, port your libraries now, at least for testing purposes and let’s make some of the best available right with the Python 3.0 release which is scheduled for September. If the initial number of working libraries is high enough for some applications it improves Python’s changes for a quicker adoption a lot. I know it’s very unlikely for big projects to switch in a short timeframe but at least smaller projects will be able to benefit from Python 3 earlier.

And even if Guido disagrees: Python 3 is *the* change to break *small pieces* of the API. Don’t break them in a way that everybody is confused and doesn’t know how things work any more. But if there are some design flaws in the library you want to change adapt them with the change to Python 3 rather than between two Python 2 versions. (And of course document them!)

Unfortunately I’m very busy lately so there are few updates on Werkzeug and all the other libraries I personally contributed (and there are even some patches in my Mail queue I have to apply after reviewing) but that doesn’t mean that there is no progress :-)

There is actually quite a lot of new stuff between the 0.1 release and now. Werkzeug tries to implement stupid stuff you reimplement in every second application in a way that you can use it with minor modifications in your application. Because many of those features only come up if you have implemented them often in your own applications they didn’t make it into 0.1. Thanks to all the early adopters we now have cool new stuff that was implemented because there was need for it.

For example Werkzeug now has RFC-compliant Etag and Cache-Control parsing. You can also generate etags automatically for responses and make them conditional for some requests. The utility module was extended with many new stuff that fix limitations in the standard library or implement long missed functions like finding modules in a package (very useful if you want to automatically register controllers), importing modules by a string (useful if your URL map endpoints are (partial) import paths), generating URLs like trac does by calling an Href object, functions to fix URLs similar to the way Firefox fixes them, dumping and parsing HTTP dates, loading and dumping cookies with a simple function call (and support for http only) and probably a lot more before the actual 0.2 release.

Another cool thing is (unrelated to Werkzeug) that more and more cool modules come up that make web development a charm. While babel is available for quite a while I haven’t really used it until last weak and it’s really great. Together with Werkzeug’s routing system and the ability to do multiple inheritance in SQLAlchemy there are so cool ways to do web development of internationalized applications. I know the last sentence doesn’t make sense without the context, so I guess I have to blog about that. sooo freaking cool.

Form handling in Python is still a bit strange if you are not using django’s newforms (I know that there is formencode but somehow it isn’t what I’m looking for) but there is now WTForms which looks promising. With some more small changes it could become a very cool form handling system (for example I’m missing some default validators at the moment and I’m not completely sure how to pass choices to a select box in a per form instance basis). WTForms was derived from an application that is already in production so it solves already many problems nicely. It’s a library I want to watch closely the next weeks.

And I think that approach should become the way Werkzeug is developed in the future. Implement features a release earlier and mark them as “under consideration” if they are not yet used in production applications. If you adopt them early you can give feedback and we can improve it to the next Werkzeug release and streamline the API.

What’s to do until the next Werkzeug release? Georg is currently working on making the sphinx documentation builder independent from the CPython documentation so that other projects can use it too. I then want to semi-automatically build an API documentation for Werkzeug and combine them with hand written rst pages for the Werkzeug 0.2 documentation. I got some feedback for the Werkzeug docs and looks like they are a bit too chaotic and misleading. Especially getting started with Werkzeug is still too complicated so I hope we can address this with a new documentation that combines automatically generated documentation with tutorials.

The documentation tool could probably be useful for other projects too, I guess Georg will drop some lines in his blog once it’s ready.

Updates regarding Jinja will be up shortly, there is currently a branch developed by Lakin Wecker to speed up Jinja template evaluation. And if you already know what GHRML/XAML will gonna be: I will try to get that running this weekend.

That’s it for the moment ;-)

I was small-talking with zzzeek about some things when I told him that I’m using Jinja, Genshi and Mako depending on what I’m doing. He told me that it’s unusual to switch tools like that but I don’t think it’s that unusual.

All three template engines are totally different but have a one thing in common: All three are the “second generation” of template engines. Genshi is the formal successor of kid, Mako somewhat replaced Mygthy and Jinja was inspired by the django templates. All three of them are framework agnostic, use unicode internally and have a cool API you can use in WSGI applications without scratching your head. But what inspired those template engines and which template engine to chose for which situation?

I often used PHP in the past to do simple header/footer inclusion. But what always drove me nuts was that I had to use mod_rewrite to get nice URLs or use a bunch of folders with index.php files or use files and folders and drop the extension in the apache config. While this is nice, this is now that portable and you can’t have dynamic parts in the URL and once you want some more dynamic stuff such as RSS feeds etc. you notice that you made a mistake by choosing PHP. Some days ago I then started working on the website for TextPress (not yet online) and wanted to try something new: I wrote a tiny WSGI application (about 50 lines of code) that just uses werkzeug’s routing system and uses template names as endpoints. These templates are then loaded with Mako, rendered and returned as responses. This is not possible in the same way with Jinja because you don’t have python blocks and not so simple and straightforward with Genshi because you have to think about XML or use a rather limited text based template engine. Another very cool feature of Mako is that you can do dynamic inheritance which is not possible in Jinja.

Mako is a great template engine if you know Python, if you need some logic in templates (and you know: logic in templates is not bad in every situation) and if you need the best performance. Without a doubt Mako is one of the fastest template engines for Python and the fastest template engine that does not use a C extension.

Then there is Jinja which is also a text based template engine like Mako. However the focus is on a completely different level. When Mako is like PHP, Jinja is like Smarty (even though Mako is a million times better than PHP as template engine). When I stated working with Python as programming language for web applications I stumbled about django. I looked at the template engine and thought: WTF is that? The syntax seemed odd and the restrictions ridiculous. Later on I loved the syntax (and apparently others do to: the mini template engine by Ian Bicking (tempita if I recall correctly) and the Genshi text templates are using that syntax or a similar one too) but some of the restrictions seem still ridiculous. When I looked at all those Django templates I created over the time I noticed that I often moved calculations into template tags that could be function calls, that I did other calculations in in the view functions that did not belong there and even more important: that you could replace 95% of the custom template tags with function calls or function calls with an enclosed template block if the template engine had proper expressions. This lead to the development of what is now known as Jinja. The syntax, the fact that it’s sandboxed and the designed friendliness is still very similar to Django, but unlike Django python like expressions are possible in Jinja.

I’m using Jinja wherever I think web designers want to work on later on. For example as template engine for TextPress or other applications that should be styled by third party web designers.

Genshi on the other hand is an XML template engine. As a result of that it’s slower but also “context aware”. It knows when it’s processing a CDATA section, it knows when it’s inside a tag or an attribute etc. This makes it possible to defend XSS in an automatic way. Per default Genshi inserts the text into the output stream as text and not as markup. That means all the HTML entities are automatically escaped for you. And because it’s stream based you can rewrite streams during the rendering process. This makes it possible to fill form fields automatically, use XInclude for simple layout templates and a lot more. You can even translate your XML based templates into HTML4 on the fly. So you can use your XML tool chain internally and output HTML4 and use the best of both worlds. But because of this high flexibility Genshi also has some problems to fight: You need to have XML knowledge to use it. No problem if you are a programmer, but not that good if you are a web designer doing fancy layouts. You are also forced to use XML templates everywhere. It’s true that Genshi has text templates too to fill the gaps, but they are not comparable with real text template engines and you are still operating on an XML stream, just that you don’t see it. And lastly: this whole stream processing makes Genshi slow. Not so slow that you can’t use it for big applications, but noticeably slower than Mako or Jinja.

If you are using XML anyways in your application, Genshi is a very good idea. Also if you don’t have template designers that don’t know XML or if performance is not that much of a problem. Most of the time the bottleneck is the database anyways. I never had real problems with Genshi performance so far.

I hope this post sums up why I’m using all three template engines and why I think we should be happy that we can chose between a couple of template engines :-) Why I’m not covering other template engines like Cheetah or SimpleTAL? Mostly because I looked at them, tried them out and never used them for something big. Mostly because Mako looks a lot nicer than Cheetah to me and SimpleTAL is far too much away from Python for me.

Finally Jinja 1.2 is released. It took a long time because this release introduces basically a complete rewrite of the parser and parts of the lexer. With Jinja 1.2 onwards the template engine has its own parser which makes it possible to introduce new syntactical elements and change operator precedence (especially regarding the filter operator which finally binds harder than plus). We added more unittests to check if Jinja stays compatible with the python semantics we had previously and all 153 tests pass :-)

With the new parser we also simplified the syntax a bit, made the “call” a keyword now and added a couple of new features. For example the debugger (with the optional c extension compiled) is now able to rewrite all jinja frames in a traceback so you will never find yourself in the situation where line numbers don’t match the template line numbers.

Another new syntax feature are conditional expressions derived from python2.5 ({{ foo if expr else bar }}). foo.0 is now the same as foo[0] for compatibility with django and tuples are finally tuples and not lists which makes it possible to use string formattings with more than one value. Test functions with one parameter don’t need parentheses so {{ foo is feeling(well) }} can now be written as {{ foo is feeling well }}.

We also added a concatenation operator that converts all arguments into a string automatically to concatenate them. This is very useful if you have some integers and other values you want to join: {{ foo ~ bar ~ baz }}, which is equivalent to {{ foo|string + bar|string + baz|string }}.

Another great new feature (if you are a django user) is the new django support module which makes it a lot easier to integrate Jinja into django. It makes it possible to use django filters in Jinja and load from the same folders.

Aside from that there are couple of improvements and bugfixes. As always you can easy install Jinja directly from pypi or download it.

From yesterday 18:00 until now pocoo.org and all the related domains (including wiki.python.de and pygments.org) where down because we moved all the xen instances to a new hetzner server. However the RAM usage by the instances is still unchanged so there will be another small downtime in the next few days. And small is ~20 minutes. Because I expected that the migration will take less time I have to postpone the upcoming Jinja release until Saturday.

Additionally we’re struggling with a mod_wsgi bug that is probably not mod_wsgi’s fault. Under certain conditions a C extension seems to not release the GIL which the result that one apache process consumes all the processor power available.

Sorry for the inconveniences caused.

Update: resource relocation done (for the moment at least)

One of the big advantages of django templates is that they are sandboxed. And the django sandbox is pretty secure because templates provide nearly no possibility to screw things up, especially because there is no way to put logic into django templates. The only possibility you have to add a security problem to django templates is writing broken template tags or stuff like that. Now Jinja has real expressions and with them the possibility to screw things up. But the Jinja core itself restricts access to python internals and as long as you keep your objects save nobody will be able to execute arbitrary python code in a template or access the filesystem.

Some time ago Christopher Lenz blogged about logic in templates, and one of the things he stated there was the following sentence about Genshi:

But even though I personally prefer working with a template language that allows me to use a real programming language (Python, Ruby, etc), there is definitely room for template languages that put severe restrictions on what you can do with them. An obvious example is that you’re running a site such as Typepad, and want to allow users to manage their own custom templates. As things currently stand, you wouldn’t be able to do that using Genshi.

Just because Genshi doesn’t support it by now this doesn’t mean it will stay like that. In fact the great architecture of Genshi makes sandboxing Genshi quite simple. To see how secure we can get Genshi I started a Genshi branch today. Maybe you can use Genshi soon for user provided templates :-)

Other notice, I got quite a few mails that my blog is/was broken. As you might now we have had some problems with the load of our server the last months. To resolve that problem i tweaked the apache the last two days and while doing that various services behaved strangely or caused problems. By now everything should work again.

Jinja 1.2 is about to be released, I just want to switch all the pocoo projects from subversion to mercurial first. One of the new features in Jinja 1.2 is the django support. Thanks to Bryan McLemore from curse for his contributions.

I refactored his code a bit so that it integrates a little nicer into django applications, the original code had some other names for functions and used idioms from curse that feel unnatural in django applications.

Basically to get started in Jinja 1.2 you have to do nothing more than adding the following thing into your urls.py file, right at the beginning:

from jinja.contrib import djangosupport
djangosupport.configure()

That automatically configures Jinja for django and adds a virtual module called “django.contrib.jinja” that contains versions of the django shortcuts that work with Jinja. This module also provides the environment that is automatically set up based on your django configuration.

Basically if you have used in your views this code so far:

from django.shortcuts import render_to_response

def foo(req):
    return render_to_response("foo.html", context_dict)

The only thing you have to change is the import:

from django.contrib.jinja import render_to_response

Another change is that Jinja users a plain old dictionary instead of a Context instance. So there is no RequestContext. Just pass it a third parameter that points to the request object to get your context processors called.

What the jinja contrib module also provides is a `render_to_string` method that works like render_to_response, but returns a unicode object, a register object that you can use to register filters, objects and tests on the jinja environment. If you want to use some django filters that are not known to Jinja you can use the `convert_django_filter` function that returns a converted filter if you pass it a django one. The Jinja environment instance is available as “env” in that module too.

If you want to try it out right now you can check out the new-parser branch. Information about the usage of the django integration are so far only in the module docstring. Happy testing :-)