Cogitations

Jinja2 RC1

June 9th, 2008

I was talking about Jinja2 in the past already but that was in the middle of developing it and many things changed in the last weeks. And now I’m proud to announce the first release candidate :)

As I said earlier Jinja2 is Jinja1 without the design mistakes. But what exactly is new and why is it so cool?

It’s dynamic. Unlike Jinja1 template inheritance and inclusion is handled at runtime now. That means you can do something like {% extends master_template %} to extend from the template with the name stored in the “master_template” variable. Previously that was not possible which clearly was a design mistake. This also affects includes (and now imports). Besides the obvious advantage that this is much more flexible, it also reduces the memory usage of Jinja. This dynamic inheritance / including directly lead to an import statement to load macros and variables from other templates.

It’s fast. The sandboxed evaluation mode in Jinja2 is now optional and the evaluation rules are simplified which gave it a huge performance improvement. Jinja2 renders a test template in under 0.04 seconds where the latest Django version needs nearly a second for.

Macros are more useful now. Macros in Jinja1 weren’t that great, but they are now! On the one hand because you can import them with a python like import syntax ({% from 'helpers.html' import input_field, textarea %} or {% import 'helpers.html' as helpers %}) on the other because they have their own namespace now. That means they are easier to debug and understand.

Includes and imports separated. Includes just include the template and render it at the current position, imports pull variables and macros from there. This makes it easier to understand templates as it’s clear from where a variable is coming from.

For loops are even cooler now. Jinja1 already had a pretty neat for loop which supported recursion and an optional else block that was rendered if the sequence was empty. It also featured a special loop variable that allowed the template designer to access the number of the current loop iteration, the total length of the loop and much more. Jinja2 extends that by adding support for loop filtering. The following example only lists the users that are visible or all if the current user is an administrator:

{% for user in users if user.is_visible or request.user.is_administrator %}
  <li>{{ user.username|e }}</li>
{% else %}
  <li><em>no users online right now</em></li>
{% endfor %}

The advantage of filtering directly in the loop head is that the special loop variable will count proplery (eg: excluding invisible users for counting). Additionally the else block will work as expected.

Better undefined behavior. Jinja1 had a very silent undefined behavior. If a variable was undefined you were able to call it without getting errors or access any attribute. Jinja2 ships three undefined types that make it easier to debug templates. The default undefined types allows you to print the undefined variables (which when printed outputs nothing) and loop over it (works like iterating over an empty list). However every other operatation raises an UndefinedError. Additionally there an undefined type with the same behavior but it prints the name of the variable or attribute missing if it’s printed. The third builtin undefined type is the strict type which doesn’t allow any operation except of testing if it’s undefined which is the closest you can get to the default python behavior. The following doctests shows the behavior of those types:

The normal undefined type:

>>> x = Undefined(name="x")
>>> unicode(x)
u''
>>> x.attribute
Traceback (most recent call last):
  ...
UndefinedError: 'x' is undefined

The strict undefined doesn’t allow anything except of testing if it’s defined from within a template. You can’t even compare it to an arbitrary value without getting an exception:

>>> x = StrictUndefined(name="x")
>>> unicode(x)
Traceback (most recent call last):
  ...
UndefinedError: 'x' is undefined
>>> x == 42
Traceback (most recent call last):
  ...

UndefinedError: 'x' is undefined

A better sandbox. The sandboxed environment if enabled is now easier to secure. In the default configuration everything starting with an underscore is considered insecure so it’s not needed any longer to mark those attributes as insecure. It’s also a lot faster now and easier to customize.

Better i18n support. Jinja2 now integrates neatly into Babel. No need to write custom string extraction scripts.

Extension interface. Like Django it’s now possible to write custom tags. I doubt anyone wants to use it as most functionality is already possible out of the box by using the expressions and tags provided, but it may be useful for some people, especially those switching from Django.

Line statements. Jinja2 allows to to specify a line statement prefix that marks a whole line as statement. This concept is inspired from Mako and Cheetah and allows very clean templates in many situations. The following example shows a template with the line statement prefix set to “%”:

<ul>
% for item in seq
  <li>{{ item }}</li>
% endfor
</ul>

Easier filters. Filters are regular functions now that get the value as first argument and the parameters provided as extra positional arguments or keyword arguments. In Jinja1 we had functions that return functions which turned out to be unnecessary complex and slow. Additionally you can now use “namespaces” for filters so do stuff like {{ variable|tools.something }}.

Automatic escaping. Jinja2 supports the special __html__ method as specified by pylons which makes it possible to use automatic escaping. (We still don’t recommend it though)

Easier API. The API is a lot easier to use now, custom loaders are nearly one-liners now and the caching is builtin automatically for all loaders. No need to funny mixins that add caching.

Grab it while it’s hot from the temporary Jinja2 website.

Upgrading Postgres from 8.1 to 8.3 on Debian/Ubuntu

June 5th, 2008

Yesterday we upgraded hammet (the server powering the Pocoo projects) to Ubuntu LTS. Unfortunately Ubuntu 8.04 aka hardy doesn’t ship Postgres 8.1 any longer so we had to upgrade to 8.3. If you are ever in the same situation: DON’T UNINSTALL 8.1. I did that because my applications showed an auth error and I guessed that was because of two running postgres servers. Turns out that postgres databases are incompatible between versions and the only way to upgrade a cluster is having both versions running and using pg_upgradecluster to upgrade them. Reinstalling in hardy won’t work because there is no package providing an 8.1 server.

What I did then was installing the 8.1 server again from dapper sources, delete the new cluster with “pg_dropcluster 8.3 main” and migrate with “pg_upgradecluster 8.1 main”. After that getting rid of the 8.1 cluster and server and most of it works again.

I fought that auth error by setting local socket connections to trusted in the config responsible for authentication and had to upgrade trac because implicit string casts no longer work.

Those non portable, version incompatible database files and the fact that I hate psql and those admin tools are by the way the reason why I feel more comfortable with MySQL.

The New %GHRML [haml for Genshi]

May 31st, 2008

I started GHRML some time ago as port of haml to Genshi but the initial reception was not that good, from the few blog comments and discussions on IRC I pretty much lost the interest in the project and haven’t used it myself.

However I’m pleased to announce that Richard Davies took over the GHRML maintenance and hosts the project at ghrml.org. He’s also using the project for a Django powered website and wrote a simple django integration module for it.

%GHRML still looks the same but some bugs are fixed now and template inheritance works. Appetizer:

%html
  %head
    %title Hello World
    %style{'type': 'text/css'}
      body { font-family: sans-serif; }
    %script{'type': 'text/javascript', 'src': 'foo.js'}

  %body
    #header
      %h1 Hello World
    %ul.navigation
      %li[for item in navigation]
        %a{'href': item.href} $item.caption

    #contents
      Hello World!

Stupid Ruby Leak

def leak
  "a string".split(/s+/)
end

leak while true

Workaround: assign the return value to a temporary variable and then return it.

How can I compile my Python Scripts?

I see that question about once a week, maybe a bit less often, on the German Python forum. Very often it turns out that py2exe is what the poster was looking for because he doesn’t want to the user to install Python. I can understand that because for the average windows user it doesn’t really make sense why he should install that Python thing. But Python is a runtime environment for the application, like the JVM or the .NET framework windows users do install. So if the windows version of Python would automatically upgrade itself somehow that wouldn’t be a big problem.

However the motivation is very often a different one: “I don’t want others to see my code” aka “I want to obfuscate my code”. Often with the addition that they are afraid others steal their code. The first point is understandable, I just have to look at the code they post in their questions they ask in that forum to see that they really have to be ashamed of. However that’s part of the learning process and by sharing your applications with others including the source you can learn from that. The second argument is that people can steal your code. But surprise: they can do it even if they don’t have the sourcecode. Python has very high-level bytecode and you can decompile it into very nice looking code, even with the original variable names preserved. I would argue that it’s even easier for them to steal code from small closed source Python applications because with a little bit of work the resulting source code looks so fundamentally different when directly compared with the original that one wouldn’t see many differences.

But really. The solution is a proper license, not trying to lock people out. Especially for small applications. Either pick one of the really good open source licenses that range from having a heavy copyleft such as the GPL to something like the do-what-the-fuck-you-want license which allows everything. Or pick a license that doesn’t allow any modifications and put your Copyright into every file. If a bad person actually steals your (probably still crappy) code you can then write a blog post and show that you were the first person that wrote that code. I doubt you want to sue the person that stole your code because that’s pretty expensive, but with the power of the blogosphere you can at least give that copycat a very bad reputation.

By having an open sourced codebase (even if it’s bad) you will attract some other developers that will probably provide patches and help you improve the product. And if it’s good enough, you can make money with it. Just look at WordPress. Some parts of the code look like written by bozos not much better than the average person asking on the python forum how to obfuscate Python code. And still, they are the number one blogging platform and I can only admire what they achieved.

Do you still want to obfuscate your code?

Mail Subjects

I found two mails in my Junk folder today that where ham. I spotted them by accident as the subjects where very stupid. The first mail arrived with “Re:” as subject, the second with “Hi”. Please make my life easier and use subjects that don’t look like spam. Thanks in advance

And no, I haven’t written a mail without subject in the first place.

Command of the Day

ssh-keygen -t dsa -b 1024 -f /etc/ssh/ssh_host_dsa_key -N '' &&
  ssh-keygen -t rsa -b 1024 -f /etc/ssh/ssh_host_rsa_key -N '' &&
  /etc/init.d/ssh restart

I really don’t think distributors should try to patch cryptographic stuff, especially not to silence debuggers.

Mail Problems

May 12th, 2008

Small notice for persons trying to mail me the last ~three days: While I was off to Vienna I noticed that HE disabled the E-Mail routing for the domains I moved to domainfactory. Mails send to me between Friday and today are probably lost.

Jinja2 Documentation Online

May 7th, 2008

I now uploaded the documentation for Jinja2 to the website for those of you who are eager and want to play with it :-) On jinja.pocoo.org you have now the choice to chose between Jinja1 and Jinja2.

The new docs are powered by Sphinx and Jinja2 with a custom templating bridge.

Read the documenation.

Simple batch function for Python

Often I have an iterable i want to group. For example a list of integers and i want to process two at once. That’s a pretty nice idom I found in the documentation translated to itertools:

from itertools import izip, repeat

def batch(iterable, n):
    return izip(*repeat(iter(iterable), n))

Use it like that:

>>> for key, value in batch([1, 2, 3, 4], 2):
...  print key, value
... 
1 2
3 4
cogitations driven by wordpress