lucumr

One of the big advantages of django templates is that they are sandboxed. And the django sandbox is pretty secure because templates provide nearly no possibility to screw things up, especially because there is no way to put logic into django templates. The only possibility you have to add a security problem to django templates is writing broken template tags or stuff like that. Now Jinja has real expressions and with them the possibility to screw things up. But the Jinja core itself restricts access to python internals and as long as you keep your objects save nobody will be able to execute arbitrary python code in a template or access the filesystem.

Some time ago Christopher Lenz blogged about logic in templates, and one of the things he stated there was the following sentence about Genshi:

But even though I personally prefer working with a template language that allows me to use a real programming language (Python, Ruby, etc), there is definitely room for template languages that put severe restrictions on what you can do with them. An obvious example is that you’re running a site such as Typepad, and want to allow users to manage their own custom templates. As things currently stand, you wouldn’t be able to do that using Genshi.

Just because Genshi doesn’t support it by now this doesn’t mean it will stay like that. In fact the great architecture of Genshi makes sandboxing Genshi quite simple. To see how secure we can get Genshi I started a Genshi branch today. Maybe you can use Genshi soon for user provided templates :-)

Other notice, I got quite a few mails that my blog is/was broken. As you might now we have had some problems with the load of our server the last months. To resolve that problem i tweaked the apache the last two days and while doing that various services behaved strangely or caused problems. By now everything should work again.

Jinja 1.2 is about to be released, I just want to switch all the pocoo projects from subversion to mercurial first. One of the new features in Jinja 1.2 is the django support. Thanks to Bryan McLemore from curse for his contributions.

I refactored his code a bit so that it integrates a little nicer into django applications, the original code had some other names for functions and used idioms from curse that feel unnatural in django applications.

Basically to get started in Jinja 1.2 you have to do nothing more than adding the following thing into your urls.py file, right at the beginning:

from jinja.contrib import djangosupport
djangosupport.configure()

That automatically configures Jinja for django and adds a virtual module called “django.contrib.jinja” that contains versions of the django shortcuts that work with Jinja. This module also provides the environment that is automatically set up based on your django configuration.

Basically if you have used in your views this code so far:

from django.shortcuts import render_to_response

def foo(req):
    return render_to_response("foo.html", context_dict)

The only thing you have to change is the import:

from django.contrib.jinja import render_to_response

Another change is that Jinja users a plain old dictionary instead of a Context instance. So there is no RequestContext. Just pass it a third parameter that points to the request object to get your context processors called.

What the jinja contrib module also provides is a `render_to_string` method that works like render_to_response, but returns a unicode object, a register object that you can use to register filters, objects and tests on the jinja environment. If you want to use some django filters that are not known to Jinja you can use the `convert_django_filter` function that returns a converted filter if you pass it a django one. The Jinja environment instance is available as “env” in that module too.

If you want to try it out right now you can check out the new-parser branch. Information about the usage of the django integration are so far only in the module docstring. Happy testing :-)

The new parser works quite well so far, in fact too well. So far all unittests pass which is something i really, really hate. Usually it means that there are unittests missing :)

The new parser supports some new stuff. For example {{ foo.0 }} is supported for easier django template transition. A regular expression literal so that the “matching” filter finally makes sense, a set literal, conditional expressions and the debugger is finally a real help.

It’s still not in the trunk because some of the changes are too big for a simple merging. If someone has really, really complex templates, try the new-parser branch and try to render the templates there. If you encounter any errors, just poke me, i’ll fix it and add a unittest :D

Uha. One day of programming and Jinja got a new parser :D 129 out of 131 tests pass, and nearly all of the old semantics still work. I changed some things so far that probably make more sense. Syntax changes from 1.1 to 1.2:

• call and endcall are keywords now. I wanted to do that in 1.3, not 1.2 but because it’s easy to change it’s a good thing to do the change now.
• tuples are tuples now, not lists. That will make strings formatting easier too — it really was a bad idea to uniform lists and tuples. This change shouldn’t cause many problems because the namespace is more or less read only. So there are no concatenation issues that will appear because a list and tuple are mixed together.
• __getslice__ is currently not supported. Parsing the python slice rules is not that simple and for the moment slice objects are easier. So foo[1:2] results in get_attribute(foo, slice(1, 2, None)). I will add some more unittests to check if this breaks stuff.
• foo|escape + bar|escape is possible now.

The code is not in trunk by now because there are still some things to fix (string escaping, streaming system) and i don’t know if the change has any side effects on bigger code. If someone wants to try out the new parser tough you can check it out from here: new-parser.

Backward incompatible changes: the lexer tokenstream, the lexer tokenize method (not the tokeniter one which is public) and the parse functions because the new parser looks different and has different nodes from different modules (no compiler.ast any more). I hope that there are few hardcore users that analyse the Jinja ast but if there are some, you probably have to change some code.

First of all: Pocoo gets some love recently. EnTeQuAk joined the team and is working on finishing the changes I haven’t finished (conversion to werkzeug and splitting up the packages). That’s great news, but don’t expect checkins the next nine days, he’s on holiday currently.

The other news affects Jinja. The plan was to introduce a new module system for the next Jinja version so that the loaders reload better and that you can use macros without including a module into the current namespace. So {% include foo = “foo.html” %} {{ foo.macro() }} should work.
This however was postponed for a simple reason. The compiler package Jinja is using internally is deprecated because with Python 2.5 the AST is a core Python feature and automatically in sync with the parser Python uses itself. But because maintaining a version for python2.5/2.6, python3 and python2.4 and lower is too much work Jinja will get a hand-written parser for the current syntax.

I expect that this will Jinja also a lot more lightweight because some of the trickeries I do with the AST will become redundant. So that will become the next thing to do.

So what’s one of the greatest features Python has? Correct. Exceptions and Tracebacks. Especially the latter is a cool thing but it’s also limited. Say you have a template engine that generates code before executing the template. So if an runtime error occurs the message is correct but the line number points to the generated code and not the source line.

So how to fix? The plan basically is:

• Have a generated line <-> source line mapping somewhere
• catch exceptions that happen in the code execution and skip the first one or two frames until you reach the frame where the generated code is
• reraise the exception in a isolated namespace but in the new line (hackish)
• patch the traceback and inject your new traceback from the isolated frame (nearly impossible)

Now the mapping is the smallest problem, as well as the frame skipping (just access tb_next a couple of times). The real problems is reraising the exception in a different line and patching that new traceback into the traceback chain. There are solutions but it requires a C extension module and some code that you better hide. Still, it works :D

If you want to see how it’s implemented check those files:

• _debugger.c
• debugger.py

So yes. You can patch tracebacks and customize them, but it’s deeper magic.

Jinja 1.1 released, it’s arrived in debian too (Thanks Piotr Ożarowski for uploading it that quick) and I can start thinking about what to put into 1.2. Actually the plans for 1.2 exist for longer than 1.0 but implementing the Features for 1.0 and 1.1 took incredible long because more and more ideas came up.

Actually the Jinja code is quite stable by now and hopefully stable enough for the import system which will be the major change for 1.2. Basically what Jinja does by now is doing inheritance before generating the bytecode. Basically what I want to have in 1.2 is an import system so that you can import macros into a namespace and do dynamic layouts. This could slow things down a bit that’s why I want to keep the static inheritance if possible and switch to the dynamic one if necessary. Currently Jinja does that implementation switching in other situations too, like in the Jinja 1.1 shorthand block syntax.

Basically what I want it to do is on the one hand this:

{% include helpers='helpers.html' %}
{% call helpers.dialog('foo') %}
    This is the dialog text.
{% endcall %}

And on the other hand dynamic inheritance is another thing I want in Jinja:

{% if show_details %}
  {% set template_to_use='with_details.html' %}
{% else %}
  {% set template_to_use='without_details.html' %}
{% endif %}
{% extends template_to_use %}
{% block body %}content goes here{% endblock %}

Also the limitation for trans tags that simple variables must now follow variable expressions should go (currently it must be {% trans foo, bar=baz, blah=blah %} instead of {% trans foo, bar=baz, blah %} because Jinja just reuses the rule for function calls)

I guess not all of that will make it into Jinja 1.2, and maybe I come up with better ideas but that’s the plan so far.

Jinja 1.1 codenname sinka is out! And with more changes then ever. Here a
small summary of the new features and improvements:

• blocks now support {{ super() }} to render the parent output.
• the template lexer keeps not track of brace, parenthesis and bracket balance in order to not break variable tags apart if they are configured to look like this: ${expr}. This also fixes the problem with nested dicts in variable expressions.
• added whitespace management system for the template designer.
• many new filters and helpers such as lipsum, batch, slice, sum, abs, round, striptags and others.
• reimplemented Buffet plugin so that you can use Jinja in pylons.
• added optional C-implementation of the context baseclass.
• it’s now possible to stream templates.
• reworked loader layer. All the cached loaders now have "private" non cached baseclasses so that you can easily mix your own caching layers in.
• added MemcachedLoaderMixin and MemcachedFileSystemLoader contributed by Bryan McLemore.
• many new unittests, bugfixes and improvements.

The whole list of changes can be found in the changelog. Get it while it’s hot from the cheeseshop.