lucumr

Georg already blogged about the new ast to code compilation feature of Python so I won’t cover that any more. Basically the old compiler package is nor surpassed by python’s real internal AST which is freaking awesome :)

The only thing that is missing is some sort of high level interface to it which makes manipulating and debugging it easier. The motivation behind it is that template engines such as Genshi, Jinja or Mako which all operate on the AST don’t have to write that much boilerplate code just to modify small pieces of the AST.

I don’t know if this module will make it into the standard library but even if not it will surely be available via the cheeseshop. So what does it do? It provides classes and utility functions to manipulate or traverse the AST in a way that is actually useful for real world applications. For example with the transformer it’s not possible to replace and remove nodes. Additionally it’s possible to generate python sourcecode from an AST to simplify debugging.

Here a small example of a transformer that brings Genshi like Loop behavior to a piece of Python sourcecode:

class GenshiSemantics(ast.NodeTransformer):

    def context(self, method):
        return ast.Expr(ast.Call(
            ast.Attribute(ast.Name('data', ast.Load()), method, ast.Load()),
            [], [], None, None
        ))

    def visit_For(self, node):
        self.generic_visit(node)
        return [self.context('push'), node, self.context('pop')]

    def visit_Name(self, node):
        self.generic_visit(node)
        return ast.Subscript(
            ast.Name('data', ast.Load()),
            ast.Index(ast.Str(node.id)),
            node.ctx
        )

When applied this piece of code:

seq = [1, 2, 3, 4, 5]
item = 42
for item in seq:
    print 'inside', item
print 'outside', item

becomes

data['seq'] = [1, 2, 3, 4, 5]
data['item'] = 42
data.push()
for data['item'] in data['seq']:
    print 'inside', data['item']
data.pop()
print 'outside', data['item']

The sourcecode of the module is in the Pocoo sandbox as usual. Unfortunately there is a bug in the initialization function of the ASTs right now so you will need a patch for it. And obviously this requires Python 2.6.

The full sourcecode for the example above can be found here.

The TurboGears project has been accepted as a mentoring organization for the 2008 Google Summer of Code. Even if you’re not interested in TurboGears because your framework of choice is something else you might still be interested in that one as it includes two Genshi project ideas: Performance and Jython compatibility.

If you have a solid knowledge of XML/HTML, Python and you’re looking for a GSoC project that’s interesting, read Christopher’s blog post about it and go for it :-)

May I introduce: GHRML, the Genshi Human readable markup language. name not final of course, mika would kill me ^^.

What’s GHRML? First of all it’s work in progress. But beside that it’s a pretty cool clone of Haml for Python. But no, it’s not another templating language, just a different representation for genshi markup templates. This way we can reuse all the genshi features like the directives or serializers. This gives you the full capabilities of the genshi templating language in a nice alternative syntax.

So how does it look like? Here a small example:

%html
  %head
    %title Hello World
    %style{'type': 'text/css'}
      body { font-family: sans-serif; }
    %script{'type': 'text/javascript', 'src': 'foo.js'}

  %body
    #header
      %h1 Hello World
    %ul.navigation
      %li[for item in navigation]
        %a{'href': item.href} $item.caption

    #contents
      Hello World!

Most of the syntax is directly stolen from Haml. The only real change is that we don’t need explicitly marked self closing elements because genshi knows that already in the serializer. Additionally you can use brackets for genshi directives like the for-loop above. And the parser works with variable indentation, if you want an indention level of four, just do so, the parser will recognize that.

Sourcecode in the sandbox for those of you who want to try it, some updates later ;-)

I was small-talking with zzzeek about some things when I told him that I’m using Jinja, Genshi and Mako depending on what I’m doing. He told me that it’s unusual to switch tools like that but I don’t think it’s that unusual.

All three template engines are totally different but have a one thing in common: All three are the “second generation” of template engines. Genshi is the formal successor of kid, Mako somewhat replaced Mygthy and Jinja was inspired by the django templates. All three of them are framework agnostic, use unicode internally and have a cool API you can use in WSGI applications without scratching your head. But what inspired those template engines and which template engine to chose for which situation?

I often used PHP in the past to do simple header/footer inclusion. But what always drove me nuts was that I had to use mod_rewrite to get nice URLs or use a bunch of folders with index.php files or use files and folders and drop the extension in the apache config. While this is nice, this is now that portable and you can’t have dynamic parts in the URL and once you want some more dynamic stuff such as RSS feeds etc. you notice that you made a mistake by choosing PHP. Some days ago I then started working on the website for TextPress (not yet online) and wanted to try something new: I wrote a tiny WSGI application (about 50 lines of code) that just uses werkzeug’s routing system and uses template names as endpoints. These templates are then loaded with Mako, rendered and returned as responses. This is not possible in the same way with Jinja because you don’t have python blocks and not so simple and straightforward with Genshi because you have to think about XML or use a rather limited text based template engine. Another very cool feature of Mako is that you can do dynamic inheritance which is not possible in Jinja.

Mako is a great template engine if you know Python, if you need some logic in templates (and you know: logic in templates is not bad in every situation) and if you need the best performance. Without a doubt Mako is one of the fastest template engines for Python and the fastest template engine that does not use a C extension.

Then there is Jinja which is also a text based template engine like Mako. However the focus is on a completely different level. When Mako is like PHP, Jinja is like Smarty (even though Mako is a million times better than PHP as template engine). When I stated working with Python as programming language for web applications I stumbled about django. I looked at the template engine and thought: WTF is that? The syntax seemed odd and the restrictions ridiculous. Later on I loved the syntax (and apparently others do to: the mini template engine by Ian Bicking (tempita if I recall correctly) and the Genshi text templates are using that syntax or a similar one too) but some of the restrictions seem still ridiculous. When I looked at all those Django templates I created over the time I noticed that I often moved calculations into template tags that could be function calls, that I did other calculations in in the view functions that did not belong there and even more important: that you could replace 95% of the custom template tags with function calls or function calls with an enclosed template block if the template engine had proper expressions. This lead to the development of what is now known as Jinja. The syntax, the fact that it’s sandboxed and the designed friendliness is still very similar to Django, but unlike Django python like expressions are possible in Jinja.

I’m using Jinja wherever I think web designers want to work on later on. For example as template engine for TextPress or other applications that should be styled by third party web designers.

Genshi on the other hand is an XML template engine. As a result of that it’s slower but also “context aware”. It knows when it’s processing a CDATA section, it knows when it’s inside a tag or an attribute etc. This makes it possible to defend XSS in an automatic way. Per default Genshi inserts the text into the output stream as text and not as markup. That means all the HTML entities are automatically escaped for you. And because it’s stream based you can rewrite streams during the rendering process. This makes it possible to fill form fields automatically, use XInclude for simple layout templates and a lot more. You can even translate your XML based templates into HTML4 on the fly. So you can use your XML tool chain internally and output HTML4 and use the best of both worlds. But because of this high flexibility Genshi also has some problems to fight: You need to have XML knowledge to use it. No problem if you are a programmer, but not that good if you are a web designer doing fancy layouts. You are also forced to use XML templates everywhere. It’s true that Genshi has text templates too to fill the gaps, but they are not comparable with real text template engines and you are still operating on an XML stream, just that you don’t see it. And lastly: this whole stream processing makes Genshi slow. Not so slow that you can’t use it for big applications, but noticeably slower than Mako or Jinja.

If you are using XML anyways in your application, Genshi is a very good idea. Also if you don’t have template designers that don’t know XML or if performance is not that much of a problem. Most of the time the bottleneck is the database anyways. I never had real problems with Genshi performance so far.

I hope this post sums up why I’m using all three template engines and why I think we should be happy that we can chose between a couple of template engines :-) Why I’m not covering other template engines like Cheetah or SimpleTAL? Mostly because I looked at them, tried them out and never used them for something big. Mostly because Mako looks a lot nicer than Cheetah to me and SimpleTAL is far too much away from Python for me.

One of the big advantages of django templates is that they are sandboxed. And the django sandbox is pretty secure because templates provide nearly no possibility to screw things up, especially because there is no way to put logic into django templates. The only possibility you have to add a security problem to django templates is writing broken template tags or stuff like that. Now Jinja has real expressions and with them the possibility to screw things up. But the Jinja core itself restricts access to python internals and as long as you keep your objects save nobody will be able to execute arbitrary python code in a template or access the filesystem.

Some time ago Christopher Lenz blogged about logic in templates, and one of the things he stated there was the following sentence about Genshi:

But even though I personally prefer working with a template language that allows me to use a real programming language (Python, Ruby, etc), there is definitely room for template languages that put severe restrictions on what you can do with them. An obvious example is that you’re running a site such as Typepad, and want to allow users to manage their own custom templates. As things currently stand, you wouldn’t be able to do that using Genshi.

Just because Genshi doesn’t support it by now this doesn’t mean it will stay like that. In fact the great architecture of Genshi makes sandboxing Genshi quite simple. To see how secure we can get Genshi I started a Genshi branch today. Maybe you can use Genshi soon for user provided templates :-)

Other notice, I got quite a few mails that my blog is/was broken. As you might now we have had some problems with the load of our server the last months. To resolve that problem i tweaked the apache the last two days and while doing that various services behaved strangely or caused problems. By now everything should work again.