lucumr

I stumbled about this thread on django-developers which proposes calling the Django 1.0 release Django 2.0. One the one hand version numbers say nothing. Just take Wine or Trac as two examples that are already very stable but still below the magical 1.0 release. Open Source software often takes some time until a 1.0 is released and that’s perfectly fine. However skipping a version number is purely a marketing trick IMO. Just think of Java which currently names 1.6 Java 6 whereas 1.4 still was Java 2.

With django it looks like the plan is to keep up with Rails which went to 2.0 a few days ago. While I love to see that django kicks ass and it’s moving toward a stable release I have a bad feeling naming it 2.0. Because there is currently a huge gap between rails and django unfortunately. Rails has gained really good integrated migrations, REST webservices, a debugger and many other things django is still lacking.

Django makes an incredible good framework if you get your problem into the use case of django. But as soon as you break out of it and need something that goes beyond what’s possible in django you wish you have chosen something else. The django ORM is far from optimal, the admin rocks but as soon as the number of users exceeds 10.000 users it’s impossible to use it (chose yourself in a dropdown of 50.000 users …) or becomes utterly complex. Complex data models also look awkward in the admin or become too complicated to manage. And if you want to stick with the admin you cannot replace the user model. Now what do you do if you have a forum and want to count the posts? Use a UserProfile module? And how do you want to display a list of users sorted by their number of posts?

Yes there are ways to hack around it but the more complex the application becomes the more you of django’s strengths become obsolete. The application I’m working on right now now is only using two more contrib modules. The auth and the admin, and it looks like we have to drop them too, due to the limitations. All applications in that project hack around ORM limitations, we have an incredible number of recreated base middlewares, we have to monkey patch the request object to hack in subdomain support.

I was talking with David Cramer from curse gaming about some of the issues and he told me that they have forked django at a given point and patched the ORM. The django template engine was replaced by Jinja (our application does the same) and they are caching the hell out of the application to scale it. Bryan McLemore from the curse team told me some time ago that some pages have up to 30 queries on a page.

I don’t want to say that django fails in what it’s doing. But it’s far from 2.0.

Finally (sorry for the long delay) Werkzeug 0.1 is out. Here parts of the current featurelist:

• Provides Request and Response objects for WSGI
• Handles file uploads by using temporary files for incoming data.
• Provides a middleware for static data for development purposes
• Tiny wrapper around wsgiref for easier development (autoreload, optional
  multithreaded enviornment)
• Unicode aware data processing. Just use unicode everywhere, werkzeug
  handles that for you.
• Mini template engine. Sometimes string formattings just are not enough
  and real template engines are too big for that tiny task.
• Context locals. Don’t pass request/user/database connections and
  other objects around. Put them on a global context local object and
  werkzeug makes sure that everyting is cleaned up end delivered well.
• Test utilities. Create fake WSGI environments and requests to test
  your application.
• Interactive debugger. Application dies with an error? Hook the debugger
  in and inspect every frame.

Here a screenshot of the debugger in action:

Small werkzeug example applications can be found in the trac. In the werkzeug.contrib package are also some pieces of code that can be useful for django developers. For example there is a stream wrapper that limits incoming form data to a given number of bytes. This is useful for django because django streams into the memory and not to the file system.

Have fun and report bugs / feature wishes in the trac :-) Get it while it’s hot from the Cheeseshop.

Finally Jinja 1.2 is released. It took a long time because this release introduces basically a complete rewrite of the parser and parts of the lexer. With Jinja 1.2 onwards the template engine has its own parser which makes it possible to introduce new syntactical elements and change operator precedence (especially regarding the filter operator which finally binds harder than plus). We added more unittests to check if Jinja stays compatible with the python semantics we had previously and all 153 tests pass :-)

With the new parser we also simplified the syntax a bit, made the “call” a keyword now and added a couple of new features. For example the debugger (with the optional c extension compiled) is now able to rewrite all jinja frames in a traceback so you will never find yourself in the situation where line numbers don’t match the template line numbers.

Another new syntax feature are conditional expressions derived from python2.5 ({{ foo if expr else bar }}). foo.0 is now the same as foo[0] for compatibility with django and tuples are finally tuples and not lists which makes it possible to use string formattings with more than one value. Test functions with one parameter don’t need parentheses so {{ foo is feeling(well) }} can now be written as {{ foo is feeling well }}.

We also added a concatenation operator that converts all arguments into a string automatically to concatenate them. This is very useful if you have some integers and other values you want to join: {{ foo ~ bar ~ baz }}, which is equivalent to {{ foo|string + bar|string + baz|string }}.

Another great new feature (if you are a django user) is the new django support module which makes it a lot easier to integrate Jinja into django. It makes it possible to use django filters in Jinja and load from the same folders.

Aside from that there are couple of improvements and bugfixes. As always you can easy install Jinja directly from pypi or download it.

From yesterday 18:00 until now pocoo.org and all the related domains (including wiki.python.de and pygments.org) where down because we moved all the xen instances to a new hetzner server. However the RAM usage by the instances is still unchanged so there will be another small downtime in the next few days. And small is ~20 minutes. Because I expected that the migration will take less time I have to postpone the upcoming Jinja release until Saturday.

Additionally we’re struggling with a mod_wsgi bug that is probably not mod_wsgi’s fault. Under certain conditions a C extension seems to not release the GIL which the result that one apache process consumes all the processor power available.

Sorry for the inconveniences caused.

Update: resource relocation done (for the moment at least)

It took less than a day. And yes I was an asshole. I can’t blame the django team for fixing things too slow because being Python’s biggest framework you can break existing code easily and fixes often requires careful consideration.

All in all I’m very happy with django and love using it, but the trac always gives me a feeling I cannot really describe. If you query open tickets you can find around 800 of them and tickets I posted so far never got real attention. However that’s not a big problem because most of them where proposals or feature requests.

Two days ago someone posted that URL bug in the IRC Channel and I bookmarked it. I thought that someone of the developers would have the timeline in the RSS reader and fix that. Yesterday I then added a patch that ignores malformed unicode in those URLs and thought I could get that fixed quickly if I sent the link to the ticket to ubernostrum and got as response something like: “that requires discussing on the mailing list, I’m not sure if ignoring is the correct behavior.” And I guess my answer was something like “I’ve better things to do”.

That and the following blog post was just rude and unacceptable. I promise that I won’t do that again :-)

disclaimer: Europe/Vienna

At the moment I’m working together with the rest of the webteam of the ubuntuusers Team on the new portal of ubuntuusers.de based on django. One of the things we will do is consolidating all templates. And while doing so we have to decide to use an HTML/XHTML standard which we will use including the correct mimetype and doctype.

And selecting that is the hardest part because once you’ve decided on something you have to live with the consequences and cannot really change. For example HTML and XHTML have a slightly different DOM or different rules for CSS (CSS for example has an exception that allows background colors on the body-tag to affect the whole page, this exception does not exist for XHTML). Without a doubt many people use XHTML in a wrong way. Just have a look how many people serve their webpages as text/html and only use HTML semantics. They break if you serve them as application/xml+xhtml or render in a wrong way.

But why does XML and SGML have different semantics? SGML itself was created long ago (I assume IBM has something to do with it, at least it’s predecessor was created there) and is an insane specification. At least that’s what the web told me. I cannot tell you if that’s true or not because the standard itself is not available without paying for it :-/

From what sources tell me XML is an subset of SGML. I wonder how that’s possible tough, because there are syntactic elements that in my opinion are not compatible. For example clash XML’s self closing tags with null end tags in SGML:

XML <br /><br />
SGML: <p/This is some text in a paragraph/

Because the slash has a special meaning in tags in SGML it clashes with the closing slash of XML tags. Also SGML is apparently case insensitive where XML is not. Maybe I’m also wrong there and that part is up to the DTD, but quite frankly. I don’t care. I don’t even are about clashing slashes in tags because no browser implements the correct SGML behavior. And if they would do, we would all see invalid output because the web is not valid. It’s not and it will never be.

But what’s indeed ridiculous is that it’s incredible hard to write pages that are semantical and syntactical correct to both HTML4 and XHTML. However you have to make your documents compatible to both if you want to your page to be valid XHTML and render correctly. The reason is that no browser today selects the render mode by Doctype, and even if they would do, other browsers would break then on the huge number of pages that incorrectly use XHTML.

XML is strict, very strict. Syntactical errors appear as big red error messages. I for myself have to work on the wiki markup for the new portal and one of the things I have to deal with is balancing elements. That is possible and simple, but what’s harder is adding paragraphs without breaking things. And that’s not that easy because not every element is allowed in a paragraph and a paragraph cannot be mixed with every element thanks to inline versus block elements.

Even HTML5 disallows that mixing of different element types but at least it doesn’t complain. Sure, I could send the output through a validator and tell the user that his markup is bullshit and he should correct it. But I won’t do that. Users give a fuck about their markup. And I cannot bloat the parser more than it is now. Server resources are limited and additional validation for such a high traffic site is nearly impossible.

Fortunately browsers will never show you those errors because they parse XHTML with their tagsoup parser they use for HTML too. Even tough, if we cannot ensure that all of our pages are valid XML and XHTML we are not allowed to use the doctype because it would break browsers that support XHTML.

While this is hard for webdesigners and especially for programmers that want to create parsers that generate XHTML it’s an even harder job for the developers of browsers. In the end they have to have two independent parsers for HTML and XHTML. This makes it hard enough for the big browser vendors Microsoft, Mozilla, Opera and Apple, but even harder if you are new to the market and want to ship your own one. Because you not only have to be compatible with the new XHTML standard, but also the old HTML one. Nobody will translate all the old documents to XHTML I’m sure ;-)

Details about the issues are summarized here:

• The <?xml> prolog, strict mode, and XHTML in IE — Chris Wilson, Microsoft
• Understanding HTML, XML and XHTML — Surfin’ Safari Weblog
• Sending XHTML as text/html Considered Harmful — Ian Hickson
• Quick Guide to XHTML — Anne van Kesteren
• The Road to XHTML 2.0: MIME Types — Mark Pilgrim

Without a doubt we will have fun with XHTML in the future. Probably the web stays like it’s today, we will still use the tag soup parsers, people will write XHTML that is HTML in fact and browsers will interpret it like that.

For me the decision is HTML4 at the moment, with the subset that is valid for both HTML4 and HTML5. That could make it easier for transition once the standard is ready (and I hope it’s earlier than 2022) and it’s good idea now too. Who needs an u-Tag anyway?

One of the big advantages of django templates is that they are sandboxed. And the django sandbox is pretty secure because templates provide nearly no possibility to screw things up, especially because there is no way to put logic into django templates. The only possibility you have to add a security problem to django templates is writing broken template tags or stuff like that. Now Jinja has real expressions and with them the possibility to screw things up. But the Jinja core itself restricts access to python internals and as long as you keep your objects save nobody will be able to execute arbitrary python code in a template or access the filesystem.

Some time ago Christopher Lenz blogged about logic in templates, and one of the things he stated there was the following sentence about Genshi:

But even though I personally prefer working with a template language that allows me to use a real programming language (Python, Ruby, etc), there is definitely room for template languages that put severe restrictions on what you can do with them. An obvious example is that you’re running a site such as Typepad, and want to allow users to manage their own custom templates. As things currently stand, you wouldn’t be able to do that using Genshi.

Just because Genshi doesn’t support it by now this doesn’t mean it will stay like that. In fact the great architecture of Genshi makes sandboxing Genshi quite simple. To see how secure we can get Genshi I started a Genshi branch today. Maybe you can use Genshi soon for user provided templates :-)

Other notice, I got quite a few mails that my blog is/was broken. As you might now we have had some problems with the load of our server the last months. To resolve that problem i tweaked the apache the last two days and while doing that various services behaved strangely or caused problems. By now everything should work again.