lucumr

It took less than a day. And yes I was an asshole. I can’t blame the django team for fixing things too slow because being Python’s biggest framework you can break existing code easily and fixes often requires careful consideration.

All in all I’m very happy with django and love using it, but the trac always gives me a feeling I cannot really describe. If you query open tickets you can find around 800 of them and tickets I posted so far never got real attention. However that’s not a big problem because most of them where proposals or feature requests.

Two days ago someone posted that URL bug in the IRC Channel and I bookmarked it. I thought that someone of the developers would have the timeline in the RSS reader and fix that. Yesterday I then added a patch that ignores malformed unicode in those URLs and thought I could get that fixed quickly if I sent the link to the ticket to ubernostrum and got as response something like: “that requires discussing on the mailing list, I’m not sure if ignoring is the correct behavior.” And I guess my answer was something like “I’ve better things to do”.

That and the following blog post was just rude and unacceptable. I promise that I won’t do that again :-)

disclaimer: Europe/Vienna

At the moment I’m working together with the rest of the webteam of the ubuntuusers Team on the new portal of ubuntuusers.de based on django. One of the things we will do is consolidating all templates. And while doing so we have to decide to use an HTML/XHTML standard which we will use including the correct mimetype and doctype.

And selecting that is the hardest part because once you’ve decided on something you have to live with the consequences and cannot really change. For example HTML and XHTML have a slightly different DOM or different rules for CSS (CSS for example has an exception that allows background colors on the body-tag to affect the whole page, this exception does not exist for XHTML). Without a doubt many people use XHTML in a wrong way. Just have a look how many people serve their webpages as text/html and only use HTML semantics. They break if you serve them as application/xml+xhtml or render in a wrong way.

But why does XML and SGML have different semantics? SGML itself was created long ago (I assume IBM has something to do with it, at least it’s predecessor was created there) and is an insane specification. At least that’s what the web told me. I cannot tell you if that’s true or not because the standard itself is not available without paying for it :-/

From what sources tell me XML is an subset of SGML. I wonder how that’s possible tough, because there are syntactic elements that in my opinion are not compatible. For example clash XML’s self closing tags with null end tags in SGML:

XML <br /><br />
SGML: <p/This is some text in a paragraph/

Because the slash has a special meaning in tags in SGML it clashes with the closing slash of XML tags. Also SGML is apparently case insensitive where XML is not. Maybe I’m also wrong there and that part is up to the DTD, but quite frankly. I don’t care. I don’t even are about clashing slashes in tags because no browser implements the correct SGML behavior. And if they would do, we would all see invalid output because the web is not valid. It’s not and it will never be.

But what’s indeed ridiculous is that it’s incredible hard to write pages that are semantical and syntactical correct to both HTML4 and XHTML. However you have to make your documents compatible to both if you want to your page to be valid XHTML and render correctly. The reason is that no browser today selects the render mode by Doctype, and even if they would do, other browsers would break then on the huge number of pages that incorrectly use XHTML.

XML is strict, very strict. Syntactical errors appear as big red error messages. I for myself have to work on the wiki markup for the new portal and one of the things I have to deal with is balancing elements. That is possible and simple, but what’s harder is adding paragraphs without breaking things. And that’s not that easy because not every element is allowed in a paragraph and a paragraph cannot be mixed with every element thanks to inline versus block elements.

Even HTML5 disallows that mixing of different element types but at least it doesn’t complain. Sure, I could send the output through a validator and tell the user that his markup is bullshit and he should correct it. But I won’t do that. Users give a fuck about their markup. And I cannot bloat the parser more than it is now. Server resources are limited and additional validation for such a high traffic site is nearly impossible.

Fortunately browsers will never show you those errors because they parse XHTML with their tagsoup parser they use for HTML too. Even tough, if we cannot ensure that all of our pages are valid XML and XHTML we are not allowed to use the doctype because it would break browsers that support XHTML.

While this is hard for webdesigners and especially for programmers that want to create parsers that generate XHTML it’s an even harder job for the developers of browsers. In the end they have to have two independent parsers for HTML and XHTML. This makes it hard enough for the big browser vendors Microsoft, Mozilla, Opera and Apple, but even harder if you are new to the market and want to ship your own one. Because you not only have to be compatible with the new XHTML standard, but also the old HTML one. Nobody will translate all the old documents to XHTML I’m sure ;-)

Details about the issues are summarized here:

• The <?xml> prolog, strict mode, and XHTML in IE — Chris Wilson, Microsoft
• Understanding HTML, XML and XHTML — Surfin’ Safari Weblog
• Sending XHTML as text/html Considered Harmful — Ian Hickson
• Quick Guide to XHTML — Anne van Kesteren
• The Road to XHTML 2.0: MIME Types — Mark Pilgrim

Without a doubt we will have fun with XHTML in the future. Probably the web stays like it’s today, we will still use the tag soup parsers, people will write XHTML that is HTML in fact and browsers will interpret it like that.

For me the decision is HTML4 at the moment, with the subset that is valid for both HTML4 and HTML5. That could make it easier for transition once the standard is ready (and I hope it’s earlier than 2022) and it’s good idea now too. Who needs an u-Tag anyway?

One of the big advantages of django templates is that they are sandboxed. And the django sandbox is pretty secure because templates provide nearly no possibility to screw things up, especially because there is no way to put logic into django templates. The only possibility you have to add a security problem to django templates is writing broken template tags or stuff like that. Now Jinja has real expressions and with them the possibility to screw things up. But the Jinja core itself restricts access to python internals and as long as you keep your objects save nobody will be able to execute arbitrary python code in a template or access the filesystem.

Some time ago Christopher Lenz blogged about logic in templates, and one of the things he stated there was the following sentence about Genshi:

But even though I personally prefer working with a template language that allows me to use a real programming language (Python, Ruby, etc), there is definitely room for template languages that put severe restrictions on what you can do with them. An obvious example is that you’re running a site such as Typepad, and want to allow users to manage their own custom templates. As things currently stand, you wouldn’t be able to do that using Genshi.

Just because Genshi doesn’t support it by now this doesn’t mean it will stay like that. In fact the great architecture of Genshi makes sandboxing Genshi quite simple. To see how secure we can get Genshi I started a Genshi branch today. Maybe you can use Genshi soon for user provided templates :-)

Other notice, I got quite a few mails that my blog is/was broken. As you might now we have had some problems with the load of our server the last months. To resolve that problem i tweaked the apache the last two days and while doing that various services behaved strangely or caused problems. By now everything should work again.

A few minutes ago I pushed CleverCSS to the cheeseshop. So far there is no framework support, once again you have to hack up the bridge yourself, but in this post I’ll show a simple way to get CleverCSS running in django and simplify your CSS files.

The preferred way to use CleverCSS is compiling the clevercss files into css, and not doing that dynamically. But because during development this can be annoying, if you have to recompile your files by hand all the time, I suggest using a view that serves those files dynamically during development. In production usage you just overlay the view URL with an apache directive etc. that points to the folder with the static css files in.

Here the view function, you can store it directly in your urls.py because it’s that small. Otherwise move it into a application responsible for static stuff:

import clevercss
from os import path
from django.http import HttpResponse, Http404

def serve_ccss_file(request, filename):
    fn = path.join(path.dirname(__file__), path.pardir, 'styles', filename + '.dcss')
    if not path.exists(fn):
        raise Http404()
    f = file(fn):
    try:
        css = clevercss.convert(f.read().decode('utf-8'))
        return HttpResponse(css, mimetype='text/css')
    finally:
        f.close()

Now you only have to create an URL rule for that:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'style/([a-zA-Z_]+).css', serve_ccss_file)
)

All you clevercss files now should go to yourproject/styles, as filename.ccss. If you enter production mode you just open your shell, go to that directory and execute clevercss.py *. (That requires that you have symlinked the clevercss.py file into your PATH.)

To test that, just drop a test.ccss file with the following contents in your styles folder and open http://localhost:8080/styles/test.css:

link_color = red

body:
  background-color: white.darken(10)
  font-family: 'Lucida Sans', Verdana, sans-serif;

a:
  color: $link_color
  &:hover:
    color: $link_color.brighten(10)
  &:visited:
    color: $link_color.darken(20)

As you might now we switched the pocoo trac to mercurial, mod_wsgi and splitted it in the same go. The new structure is can be found at dev.pocoo.org. What you cannot see there is how all that is implemented. And I tell you. It’s dead simple.

Basically we use mod_wsgi for hosting the tracs. There are many reasons for that but the most important one is that you can host multiple trac instances without much configuration. Basically the configuration binds a wsgi application to a URL match rule. One important thing is the maximum-requests setting. To understand this parameter you have to know that mod_wsgi does not only keep a pool of running python interpreters, but also your application with all data in the memory. Now that’s a big difference to mod_php where your application is sourced on request and removed from the memory after the reqest. So basically you cannot create memory holes, which you can do in python. If your application leaks memory (and trac tends to do so) you can tell mod_wsgi to restart one python interpreter after 500 requests for example. That setting of course depends on your trac version, the number of plugins etc. And especially how many memory you have in your application. Here the Apache config:

<VirtualHost *:80>
    ServerName dev.example.org
    RewriteEngine On

    WSGIDaemonProcess tracs threads=10 maximum-requests=500

    RewriteCond %{REQUEST_URI} ^/([a-z_]+)
    RewriteRule . - [E=TRAC_ID:%1]
    WSGIScriptAliasMatch ^/([a-z_]+) /var/trac/trac.wsgi

    <Directory /var/trac>
        WSGIProcessGroup tracs
        WSGIApplicationGroup %{GLOBAL}
        Order deny,allow
        Allow from all
    </Directory>

    <LocationMatch /([a-z_]+)/login>
        AuthType "Basic"
        AuthName "Trac Instances Login"
        AuthUserFile /var/trac/users
        Require valid-user
    </LocationMatch>
</VirtualHost>

Then we need a trac.wsgi file which is basically just a minimal WSGI application that dispatches our key. Say we have our trac instances in /var/trac/instances, every trac in it’s own folder. Then we can use this code:

#!/usr/bin/python
from os import environ, path
from trac.web.main import dispatch_request

def application(environ, start_response):
    trac_path = path.join('/var/trac/instances', environ['TRAC_ID'])
    if path.exists(trac_path):
        environ['trac.env_path'] = trac_path
        return dispatch_request(environ, start_response)
    start_response('404 NOT FOUND', [('Content-Type', 'text/plain')])
    return ['Not Found']

You can of course modify that not found message, maybe render a fancy HTML page or just redirect to the index of that domain or whatever. The important thing is that you set the path to the trac instance before calling the dispatch_request function. In theory you can do that from the apache config too, but in that situation you cannot check if the trac really exists.

And now about the django hosting part. Basically you can do the same for django. Django basically has a environment key called the DJANGO_SETTINGS_MODULE key. This key basically controls what settings module django will import. Unfortunately the whole django core is not process safe, so you cannot run two different django powered applications in the same python interpreter. This however is not that much of an issue with mod_wsgi, because you can tell mod_wsgi to not share the interpreter. (In the trac config above we shared the interpreter to save some memory)

Your config could look like this:

<VirtualHost *:80>
    ServerName www.example.org
    WSGIDaemonProcess django_app1 threads=10 maximum-requests=5000
    WSGIScriptAlias /app1 /var/www/django_app1.wsgi
    WSGIDaemonProcess django_app2 threads=10 maximum-requests=5000
    WSGIScriptAlias /app2 /var/www/django_app2.wsgi

    <Location /app1>
        WSGIProcessGroup django_app1
        WSGIApplicationGroup %{GLOBAL}
    </Location>
    <Location /app2>
        WSGIProcessGroup django_app2
        WSGIApplicationGroup %{GLOBAL}
    </Location>
</VirtualHost>

The actual “django_appX.wsgi” file is very, very simple. It just adds the folder and instanciates the django wsgi app:

#!/usr/bin/python
import sys, os
sys.path.insert(0, '/path/to/django_appX')
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_appX.settings'

from django.core.handlers.wsgi import WSGIHandler
application = WSGIHandler()

Hope I could help a little bit, if you have some questions to our server setup just send me a mail or write a comment. Finally, we first encountered some problems with mod_wsgi two months ago, but at the moment everything is working well, a lot better than any other server setup we used. You can even put python applications into the context of another user which basically replaces fastcgi + suexec.
And btw, the support we got from Graham is really, really good :)

Update: removed WSGIPassAuthorization like Graham suggested in the comments below.

Wohoo. We finally migrated all of our subversion hosted projects to mercurial. And the best part: we now have a separate trac instance for each of those projects. mod_wsgi does a great job hosting all those tracs without having to add each trac to the config separately. I will post a commented version of our mod_wsgi config in the next days, for the moment still some other things are in the pipeline.

The new developer platform of Jinja, Pygments and all the other projects is on dev.pocoo.org. Note that most tracs are still empty because we haven’t had the time to move the tickets and wiki pages over. For that we have to write a conversion script first.

The plan for the next weeks is releasing Jinja 1.2 and Werkzeug 0.1. The latter is missing some unittests. After that I want to work on pocoo to finally get a working version. TextPress itself is now in the repositories (if you don’t know what it is, just ignore it for now), guess most of use will contribute some code in the next weeks but don’t expect an release. Want to convert this blog here soon though.

Last but not least: hello planet django :)

Jinja 1.2 is about to be released, I just want to switch all the pocoo projects from subversion to mercurial first. One of the new features in Jinja 1.2 is the django support. Thanks to Bryan McLemore from curse for his contributions.

I refactored his code a bit so that it integrates a little nicer into django applications, the original code had some other names for functions and used idioms from curse that feel unnatural in django applications.

Basically to get started in Jinja 1.2 you have to do nothing more than adding the following thing into your urls.py file, right at the beginning:

from jinja.contrib import djangosupport
djangosupport.configure()

That automatically configures Jinja for django and adds a virtual module called “django.contrib.jinja” that contains versions of the django shortcuts that work with Jinja. This module also provides the environment that is automatically set up based on your django configuration.

Basically if you have used in your views this code so far:

from django.shortcuts import render_to_response

def foo(req):
    return render_to_response("foo.html", context_dict)

The only thing you have to change is the import:

from django.contrib.jinja import render_to_response

Another change is that Jinja users a plain old dictionary instead of a Context instance. So there is no RequestContext. Just pass it a third parameter that points to the request object to get your context processors called.

What the jinja contrib module also provides is a `render_to_string` method that works like render_to_response, but returns a unicode object, a register object that you can use to register filters, objects and tests on the jinja environment. If you want to use some django filters that are not known to Jinja you can use the `convert_django_filter` function that returns a converted filter if you pass it a django one. The Jinja environment instance is available as “env” in that module too.

If you want to try it out right now you can check out the new-parser branch. Information about the usage of the django integration are so far only in the module docstring. Happy testing :-)