Prerequisites

Things you have to have installed on your computer in order to make the following examples work correctly:

• PyGTK — in a quite new version. Don't ask me which version we need here but I doubt that older versions will work.
• Glade — for designing the user interface. Doing that in the sourcecode is just dumb.
• Python — and in version 2.4 or higher.

Motivation

Although I have en_GB.UTF-8 as my locale and publish articles on this webpage in English my mother-tongue is German. As a result of that I love applications that are available in more than one language. And the OpenSource community has plenty of applications, libraries and online platforms for creating multilanguage GUI and web applications. When I tried to translate one of my applications some days ago I was surprised that there were no tutorials (or maybe I'm too stupid for googleing) that explained the issue. I then checked other projects and read some man pages.

So I hope that this article will make life easier for you.

Organizing the Application

The first (and hardest) part of any application is the organization. It's that hard because the organization should support development and production mode. Having to symlink or copy files into global folders makes things a lot harder and is usually a stupid idea. For my application I decided to not put the translation files into the locale folder but the application shared data directory.

My idea (or better, the idea of Leonard Ritter, who did a similar thing for his aldrin project) was that you have the whole folder structure in your development folder and just copy the files to prefix (usually /usr) when installing the application.

So basically the folder structure looks like this:

bin/
  APPLICATION             the executable that starts the application
share/
  APPLICATION/
    glade/                the folder that contains all the glade files
    lib/
      APPLICATION/        the python package with the application data
        __init__.py       make it a package and add bootstrapping code.
        application.py    the actual application code.
        ...               other files.
      ...                 dependencies you maybe want to bundle
    i18n/                 contains all the compiled translations
    res/                  other resources (images etc)

The executable (bin/APPLICATION) now looks at it's own filename, and looks up the parent folder of the folder the executable is located in. So for the development mode this will be ., for production mode it will probably be something like /usr, /usr/local etc.

Then it calculates the absolute path to the lib folder and adds it to the pythonpath to import the main function from the package and call it:

#!/usr/bin/env python
import sys
from os.path import dirname, join, pardir
sys.path.insert(0, join(dirname(__file__), pardir, 'share',
                        'APPLICATION', 'lib'))
from APPLICATION import main
main()

The main() function comes directly from the __init__.py file of the APPLICATION package located in the lib folder.

Bootstrapping

Now it comes to the second hard part. Before we can use all the libraries we have to do some bootstrapping to get the gettext system working in the python interpreter and the GTK system. This happens in the __init__.py file of the package. There we set up gettext for the whole python interpreter, get the path to the shared files and import the main function from the application submodule:

from os.path import abspath, dirname, join, pardir

# here we define the path constants so that other modules can use it.
# this allows us to get access to the shared files without having to
# know the actual location, we just use the location of the current
# file and use paths relative to that.
SHARED_FILES = abspath(join(dirname(__file__), pardir, pardir))
LOCALE_PATH = join(SHARED_FILES, 'i18n')
RESOURCE_PATH = join(SHARED_FILES, 'res')

# the name of the gettext domain. because we have our translation files
# not in a global folder this doesn't really matter, setting it to the
# application name is a good idea tough.
GETTEXT_DOMAIN = 'APPLICATION'

# setup PyGTK by requiring GTK2
import pygtk
pygtk.require('2.0')

# set up the gettext system and locales
from gtk import glade
import gettext
import locale

locale.setlocale(locale.LC_ALL, '')
for module in glade, gettext:
    module.bindtextdomain(GETTEXT_DOMAIN, LOCALE_PATH)
    module.textdomain(GETTEXT_DOMAIN)

# register the gettext function for the whole interpreter as "_"
import __builtin__
__builtin__._ = gettext.gettext

# import the main function
from APPLICATION.application import main

Alright. The code above should be straightforward, what you probably not know is the __builtin__ module. Basically if python cannot find a name in the locale or global namespace (and also not in a closure) it will look for it in the __builtin__ module before giving up. This is the module where the stuff like float/dir etc. comes from.

By binding the gettext function to __builtin__._ we can use _('Hello World') to mark as translatable and translatate a string from every module in the python interpreter without having to import the function first.

Writing Internationalizable Code

i18n of Linux works pretty well because it's something gettext solves in a very good way. The only problem you have is that there are usually situations where something looks like it would work for your language but certainly not in another language. This usually affects pluralization. The rule "n != 1" for plural forms is something you have in English and German and some other languages but not in Slavic and also not in Asian languages.

Also it's possible that you have two strings that look exactly the same but mean different things in the source language. In that case you have to "annotate" them by adding additional information to it to have two differnent strings. In that case you will have to introduce an translation file for your source language too and cut of the annotated stuff. An example for that is "May" and "May" in Calendars. One of them is the three letter version, the other is the full version. Unfortunately both look the same in English because too short. Solution:

_('May::abbreviated') and _('May::full')

I won't cover plural forms here because that would make this article too long but if you have the situation that you must distinquish between singular and plural have a look at the ngettext function provided.

So what you have to do now is wrapping all translatable stuff in _() as explained here:

# this:
foo = 'Hello World!'
# becomes:
foo = _('Hello World!')

# and this:
foo = 'Hello %s!' % bar
# becomes:
foo = _('Hello %s!') % bar

# because placeholders could move in other languages, this:
foo = 'Expected %d items, got %d.' % (expected, found)
# becomes:
foo = _('Expected %(expected)d items, got %(found)s.') % {
    'expected':   expected,
    'found':      found
}

Creating Internationalizable GTK GUIs using Glade

Alright. That's easy. It happens out of the box. What you can do additionally is leaving comments for translators or marking strings as untranslatable in the property editor but otherwise all labels etc are marked as translatable automatically.

Collecting Strings

Once the application is created we have to collect all the translatable strings and create so-called "pot" files (po templates). A translator can then use that pot file to translate the application and send it back with the translated strings in. A pot file with translations is called a "po" file.

For python there is a cool called pygettext in the distribution that helps collecting translatable strings and creating a pot file. For glade there should be a configuration to generate pot files too but I was unable to find that in my Glade version (3.0.2). But even if there was such a setting or button it would still generate two pot files which are more complex to merge (requires some POTFILES.in foobar and tools with manpages longer than I want to read) I wrote a small tool myself especially for PyGTK which scans all python and glade files for translatable strings.

You can find this tool here: generate_pot.py. Usage is quite simple:

$ python generate_pot.py share/APPLICATION APPLICATION VERSION > APPLICATION.pot

Replace APPLICATION and VERSION with your application name of course.

Translating

So the process of translating an pot file is straigtforward. If it's a new translation (so there is now it.po or de.po) by now you can simply copy the pot file to it's target name and open it with your favorite translator application or text editor. The format is straightfoward, it's easily possible to translate it with vim or whatever editor you use. the msgid part is the source string, the msgstr part is the translated string.

If you already have an translated po file and you added or modified translatable strings in the application (and an updated pot file generated by generate_pot.py) you can use the msgmerge utility that comes with the gettext system:

$ msgmerge -U de.po APPLICATION.pot

If you changed the casing of an source string or did another minor modification msgmerge resolves that and marks the string as fuzzy. Just have a look at the comment above the string.

Compiling Translations

Now you have to compile a mo file gettext can use. It's basically a binary version of the po file with the translated strings in. For that you can use the msgfmt application:

$ msgfmt do.po -o share/APPLICATION/i18n/de/LC_MESSAGES/GETTEXT_DOMAIN.mo

As you can see you need a quite complicated folder structure for gettext so make sure that de/LC_MESSAGES exists in the i18n folder. and replace GETTEXT_DOMAIN with the domain you defined in the package's __init__.py.

Testing Translations

If you have a system with a different default locale and you don't want to logout and login with another locale just use the LANG variable to tell gettext to use another language:

$ LANG=de_DE.UTF-8 ./bin/APPLICATION

Note that this requires that you have the German locales installed on your system because otherwise the setlocale call will raise an error.

Deployment

So now where the application is ready to distribute all you have to do is to create a Makefile (or what ever you want to use) and copy bin and share to the specified prefix.

Get the Code, Don't get Lost

I uploaded a small example application that uses the foundation explained above. You can download it here: pygtki18nexample.tar.gz

It contains a small application that just displays a window and connects some basic signals.

What's WSGI?

Basically WSGI is lower level than CGI which you probably know. But in difference to CGI, WSGI does scale and can work in both multithreaded and multi process environments because it's a specification that doesn't mind how it's implemented. In fact WSGI is not CGI because it's between your web application and the webserver layer which can be CGI, mod_python, FastCGI or a webserver that implements WSGI in the core like the python stdlib standalone WSGI server called wsgiref.

WSGI is specified in the PEP 333 and adapted by various frameworks including the well known frameworks django and pylons.

If you are too lazy to read the pep 333 here's a short summary:

• WSGI application are callable python objects (functions or classes with a __call__ method that are passed two arguments: a WSGI environment as first argument and a function that starts the response.
• the application has to start a response using the function provided and return an iterable where each yielded item means writing and flushing.
• The WSGI environment is like a CGI environment just with some additional keys that are either provided by the server or a middleware.
• you can add middlewares to your application by wrapping it.

Because that's a lot of information let's ignore it for now and have a look at a basic WSGI application:

Extended Hello World

Here a simple, but not too simple example of a WSGI application that says Hello World! where World can be specified via url parameter.

from cgi import parse_qs, escape

def hello_world(environ, start_response):
    parameters = parse_qs(environ.get('QUERY_STRING', ''))
    if 'subject' in parameters:
        subject = escape(parameters['subject'][0])
    else:
        subject = 'World'
    start_response('200 OK', [('Content-Type', 'text/html')])
    return ['''<title>Hello %(subject)s</title>
    <p>Hello %(subject)s!</p>''' % {'subject': subject}]

As you can see the start_response function takes two arguments. A status string and a list of tuples that represent the response headers. What you cannot see because it's not used here and nowhere else is that the start_response function returns something. It returns a write function that directly writes to the webserver output stream. Because it bypasses middlewares (we'll cover that later) it's a terrible bad idea to use that function. For debugging purposes however it can be useful.

But how to start that application now? A webserver doesn't know how to handle that and neither does python because nothing calls that function. Because we're lazy we don't setup a server with WSGI support now but use the wsgiref WSGI standalone server bundled with python2.5 and higher. (You can also download it for python2.3 or 2.4)

Just add this to your file:

if __name__ == '__main__':
    from wsgiref.simple_server import make_server
    srv = make_server('localhost', 8080, hello_world)
    srv.serve_forever()

When you now start the file you should be able to get a Hello John! on http://localhost:8080/?subject=John.

Path Dispatching

You probably worked with CGI or PHP before. If you did so you know that you most of the time have multiple public files (.pl / .php) a user can access and that do something. Not so in WSGI. There you only have one file which consumes all paths. Thus if you have your server from the previous example still running you should get the same content on http://localhost:8080/foo?subject=John.

The accessed path is saved in the PATH_INFO variable in the WSGI environment, the real path to the application in SCRIPT_NAME. In case of the development server SCRIPT_NAME will be empty, but if you have a wiki that is mounted on http://example.com/wiki the SCRIPT_NAME variable would be /wiki. This information can now be used to serve multiple indepentent pages with nice URLs.

In this example we have a bunch of regular expressions and match the current request against that:

import re
from cgi import escape

def index(environ, start_response):
    """This function will be mounted on "/" and display a link
    to the hello world page."""
    start_response('200 OK', [('Content-Type', 'text/html')])
    return ['''<title>Hello World Application</title>
               <p>This is the Hello World application:</p>
               <p><a href="hello/">continue</a></p>''']

def hello(environ, start_response):
    """Like the example above, but it uses the name specified in the URL."""
    # get the name from the url if it was specified there.
    args = environ['myapp.url_args']
    if args:
        subject = escape(args[0])
    else:
        subject = 'World'
    start_response('200 OK', [('Content-Type', 'text/html')])
    return ['''<title>Hello %(subject)s</title>
            <p>Hello %(subject)s!</p>''' % {'subject': subject}]

def not_found(environ, start_response):
    """Called if no URL matches."""
    start_response('404 NOT FOUND', [('Content-Type', 'text/plain')])
    return ['Not Found']

# map urls to functions
urls = [
    (r'^$', index),
    (r'hello/?$', hello),
    (r'hello/(.+)$', hello)
]

def application(environ, start_response):
    """
    The main WSGI application. Dispatch the current request to
    the functions from above and store the regular expression
    captures in the WSGI environment as  `myapp.url_args` so that
    the functions from above can access the url placeholders.

    If nothing matches call the `not_found` function.
    """
    path = environ.get('PATH_INFO', '').lstrip('/')
    for regex, callback in urls:
        match = re.search(regex, path)
        if match is not None:
            environ['myapp.url_args'] = match.groups()
            return callback(environ, start_response)
    return not_found(environ, start_response)

Now that's a bunch of code. But you should get the idea how URL dispatching works. Basically if you now visit http://localhost:8080/hello/John you should get the same as above but with a nicer URL and a error 404 page if you enter the wrong url. Now you could improve that further by encapsulating environ in a request object and replacing the start_response call and the return iterator with a response objects. This is also what WSGI libraries like werkzeug and paste do.

By adding something to the environment we did something normally middlewares do. So let's try to write one that catches exceptions and renders them in the browser:

# import the helper functions we need to get and render tracebacks
from sys import exc_info
from traceback import format_tb

class ExceptionMiddleware(object):
    """The middleware we use."""

    def __init__(self, app):
        self.app = app

    def __call__(self, environ, start_response):
        """Call the application can catch exceptions."""
        appiter = None
        # just call the application and send the output back
        # unchanged but catch exceptions
        try:
            appiter = self.app(environ, start_response)
            for item in appiter:
                yield item
        # if an exception occours we get the exception information
        # and prepare a traceback we can render
        except:
            e_type, e_value, tb = exc_info()
            traceback = ['Traceback (most recent call last):']
            traceback += format_tb(tb)
            traceback.append('%s: %s' % (e_type.__name__, e_value))
            # we might have not a stated response by now. try
            # to start one with the status code 500 or ignore an
            # raised exception if the application already started one.
            try:
                start_response('500 INTERNAL SERVER ERROR', [
                               ('Content-Type', 'text/plain')])
            except:
                pass
            yield '\n'.join(traceback)

        # wsgi applications might have a close function. If it exists
        # it *must* be called.
        if hasattr(appiter, 'close'):
            appiter.close()

So how can we use that middleware now? If our WSGI application is called application like in the previous example all we have to do is to wrap it:

application = ExceptionMiddleware(application)

Now all occouring exceptions will be catched and displayed in the browser. Of course you don't have to do that because there are many libraries that do exactly that and with more features.

Deployment

Now where the application is "finished" it must be installed on the production server somehow. You can of course use wsgiref behind mod_proxy but there are also more sophisticated solutions available. Many people for example prefer using WSGI applications on top of FastCGI. If you have flup installed all you have to do is to defined a myapplication.fcgi with this code in:

#!/usr/bin/python
from flup.server.fcgi import WSGIServer
from myapplication import application
WSGIServer(application).run()

The apache config then could look like this:

<VirtualHost *>
    ServerName www.example.com
    Alias /public /path/to/the/static/files
    ScriptAlias / /path/to/myapplication.fcgi/
</VirtualHost>

As you can see there is also a clause for static files. If you are in development mode and want to serve static files in your WSGI application there are a couple of middlewares (werkzeug and paste as well as "static" from Luke Arno's tools provide that) available.

NIH / DRY

Avoid the "Not Invented Here" problem and don't repeat yourself. Use the libraries that exist and their utilities! But there are so many! Which one to use? Here my suggestions:

Frameworks

Since Ruby on Rails appeared on the web everybody is talking about frameworks. Python has two major ones too. One that abstracts stuff very much and is called Django and the other that is much nearer to WSGI and called pylons. Django is an awesome framework but only as long as you don't want to distribute your application. It's if you have to create a webpage in no time. Pylons on the other hand requires more developer interaction and your applications are a lot easier to deploy.

There are other frameworks too but my experiences with them are quite bad or the community is too small.

Utility Libraries

For many situations you don't want a full blown framework. Either because it's too big for your application or your application is too complex that you can solve it with a framework. (You can solve any application with a framework but it could be that the way you have to solve it is a lot more complex than without the "help" of the framework)

For that some utility libraries exist:

• paste — used by pylons behind the scenes. Implements request and response objects. Ships many middlewares.
• werkzeug — minimal WSGI library we wrote for pocoo. Ships unicode away request and response objects as well as an advanced URL mapper and a interactive debugger.
• Luke Arno's WSGI helpers — various WSGI helpers in independent modules by Luke Arno.

There are also many middlewares out there. Just look for them at the cheeseshop.

Template Engines

Here a list of template engines I often use and recommend:

• Genshi — the world's best XML template engine. But quite slow, so if you need a really good performance you have to go with something else.
• Mako — stupidely fast text based template engine. It's a mix of ERB, Mason and django templates.
• Jinja — sandboxed, designer friendly and quite fast, text based template engine. Of course my personal choice :D

Conclusion

WSGI rocks. You can simply create your own personal stack. If you think it's too complicated have a look at werkzeug and paste, they make things a lot easier without limiting you.

I hope this article was useful.

JavaScript is a great language! But there are many design mistakes in, but probably different mistakes than you think. Having no class syntax etc is not a flaw, it's genious. Also one namespace is no issue at all since you can easly add closures around your stuff and return one element to the global namespace that has multiple members and substructures. This works pretty much like ruby namespaces if used in a sane way.

But where are the mistakes then?

this

this is the biggest mistake in that language. It always points to something and in most situations it points to the global namespace which makes no sense since it makes working with closures a lot harder.

In a situation where you call a function without sending a message from an object (just bar() and not foo.bar()) this should point to the this of the scope where the function is defined in:

/* current solution, one has to bind this to a different variable */
MyNamespace.User.makeCounter() {
  var counter = -1;
  var self = this;
  return function() {
    if (self.userCounter > counter)
      counter = self.userCounter;
    return ++counter;
  }
}

/* would make much more sense when this points to the outer this: */
MyNamespace.User.makeCounter() {
  var counter = -1;
  return function() {
    if (this.userCounter > counter)
      counter = self.userCounter;
    return ++counter;
  }
}

Inheritance

Inheritance in JavaScript works via prototypes. But it doesn't work in a nice way because there is no syntax for it. It would help if you have a syntax for inheriting stuff:

/* old school inheritance by setting the internal link to
   an existing object by creating a new instance of an object: */
MyNamespace.NewClass = (function() {
  var cls = function() {};
  cls.prototype = MyNamespace.ParentClass;
  cls.fooMethod = function() { ... };
  cls.barMember = 42;
  return new cls();
})();

/* with syntactic sugar: */
MyNamespace.NewClass = MyNamespace.ParentClass >> {
  fooMethod : function() { ... },
  barMemner : 42
};

That syntax might not be the best one but it says what happens. If if there is no syntax, at least there should be a factory function that does something similar (Object.extend).

More Functional Helpers

JavaScript is a functional language. That's what the intenion was and what it does best. So please add partial/curry, map and some other helpers either to the global namespace or to the arrays/functions itself.

Better Lambda

JavaScript has a function keyword that works exactly like a lambda should work. But it could be better by removing that keyword or making it a lot smaller. Possible solutions:

/* current solution */
var result = map([1, 2, 3, 4], function(item) {
  return item * item;
});

/* ruby like solution */
var result = map([1, 2, 3, 4], { |item| item * item });

/* just making the keyword smaller */
var result = map([1, 2, 3, 4], def(item) { return item * item });

As you can see one of them has implicit defaults. And the ruby like solution would rewrite the pipe symbols, even if no arguments are passed because otherwise it would have a different meaning.

Iteration

JavaScript iteration works the wrong way. Especially since arrays are objects and iterate over their keys which is wrong and silly. Do what ruby does, call `each()` and pass it a function:

/* old solution for array iteration */
for (var i = 0, n = foo.length; i < n; i++) {
  var item = foo[i];
  ...
}

/* old solution for object iteration */
for (var key in foo) {
  var value = foo[key];
  ...
}

/* ruby like solution for iteration */
for (var key, value in foo) {
  ...
}

/* which would be the same as */
foo.each(function(key, value) { ... })

This of corse raises the question how continue and break should work then. The continue statement would translate to return and break to throw new StopIteration().

Everything as Object / Operators

Throw away that stupid typeof operator and make Numbers, Booleans null, undefined whatever objects. Operator overloading would be a nice to have but it would make things much more complicated. At least the + Operator should not add and concatenate at the same time. The simple rshift Operator would be extend for other objects as seen above.

Also == should become === and vice versa. Likewise for != and !==.

Conclusion

This might not be the answer but it makes a lot more sense for a web centric language (which JavaScript certainly is) than another Java like language implementation.

What jQuery looks like

The most important part regarding JavaScript libraries in my eye is how verbose they are. I don't want to have cryptic constructs but I also don't want to type too much. Especially since it's pretty easy to create huge files and I don't want my application to load hundreds of kilobytes of JavaScript to the client's browser.

So here a pretty nice example for jQuery that adds an anchor to all headlines:

document.ready(function() {
  for (var i = 0; i <= 6; i++)
    $('h' + i + ' [@id]').each(function() {
      $('<a class="anchorlink">\u00B6</a>').
        attr('href', '#' + this.id).
        attr('title', 'Permalink to this headline').
        appendTo(this);
    });
});

This is beautyful code in my mind. For several reasons. For one I can simply query elements by a mixture of XPath and CSS syntax. Then I can iterate over all result elements by passing an anonymous function to the each method. I don't even have to give it an variable name because jQuery automatically binds the value to this. Furthermore I can use $() to create an element and use the chaining possibilities to add attributes and append it to the headline in one go. Gorgeous!

Doing Things Right

I like cherrypicking a lot. Just combine the best of different libraries to do what you want. And JavaScript allows you to do that. Unlike PHP it's possible to have multiple namespaces (by adding objects to window) and binding values to that. jQuery uses that. It just creates one global objects and plays nice with others. The default alias from jQuery to $ is completely optional, thus you can combine jQuery with many other libraries without triggering namespace clashes etc.

It doesn't patch a single builtin. That's awesome. Also jQuery includes basic support for effects and nice event support. While the library itself is quite big with 57KB, the compressed version is only 19KB big.

It's also darn fun to work with. I don't want to miss this and firebug any more :-)

Plugins

jQuery has a list of plugins in the wiki and that page is really awesome long :-) From Since jQuery was developed in an extensible way plugins integrate nicely into the jQuery core. For example if you have interface loaded you can do this:

document.ready(function() {
  $('#message').click(function() {
    $(this).Fold(500, 30);
  });
});

This will fold the element with id message once it's clicked.

What I don't like about Others

prototype.js

Prototype, the default answer you get these days by any rails developer. It's one of the first flashy javascript libraries but it sucks for many reasons. Until the most recent version it was nearly impossible to find a good documentation of it. Also the way you work with the library is too rubysh. That's not bad if you are a rails developer but it makes no sense outside the rails environment. JavaScript itself behaves a lot more like Python or LISP due to first class functions.

It also includes dozens of functions I really don't use or want to use. So why do I have to add those to my requirements too? A huge stdlib makes sense for an offline language but clearly not for javascript.

AJS

We used it in pocoo before we discovered jQuery. Really a nice library but the lack of XPath support is a real show stopper over jQuery. If you don't need chainability or XPath you could go with this library. It provides some nice things like forceArray :-)

mootools

Awesome library but I still like jQuery better. If there wasn't jQuery I would use Mootools :-) It's also a bit too flashy for my needs. I want a non sucking javascript and not blinking elements everywhere ^^

mochikit

Was a nice library some years ago but it's not lightweight any more.

dojo

Awesome but huge library. The API sucks in comparison with jQuery but it provides stunning stuff. Especially regarding vector graphics. But it's huge...

Conclusion

jQuery is really the answer. For my requirements :-) Maybe you have similar and don't know jQuery so far, give it a try. It's more than fun to work with.

I know that there are many other libraries out there but most of them have different views of some things (especially patching prototypes of public objects) which are not compatible with my point of view. This doesn't mean that those libraries are bad. But they just don't work that well for me.

And it breaks copy/paste too. And i cannot read it from my mobile phone. And it's awefully slow to download it via GPRS (and expensive too). And it's hard to read, makes my eyes sick et cetera.

So please:

This image has an alt tag!

Update

Peter Cooper wants you to know that:

There was no point in including the Ruby Inside link in that post. It’s not a code example, it's a /screenshot/ to give an impression. The actual code is on the page linked to ready to copy and paste.

Synopsis

"Digital Fortress" is a book by Dan Brown, written back in 1998.

Susan Fletcher, a brilliant, beautiful mathematician and head of the National Security Agency's cryptography departement, finds herself faced with an unbreakable code resistant to brute-force attacks by the NSA's 3 million processor supercomputer.

The code is written by Japanese cryptographer Ensei Tankado, a former employee of her departement, who now uses his knowledge to work against the NSA to avoid further intrusion into people's privacy. Tankado publishes the algorithm on his website, encrypted with itself. He sells the passphrase for encrypted data on an auction platform, threatening that his accomplice, "North Dakota", will release the algorithm for free if he dies.

This algorithm would be the end of the NSA and the TRANSLTR supercomputer. Unfortunately Tankado is found dead in Seville, Spain. Susan Fletcher, along with her boyfriend, David Becker, a skilled linguist, must find a solution to stop the spread of the code.

Sounds Interesting. What's the Problem?

Alright. Here comes the problem. But first a small information: I'm not a cryptographer and thus not completely informed about that topic. Nonetheless I did enough encryption/decryption for my IT lessons in the past that I can accuse Dan Brown for not having the slightest knowledge about cryptography, information technology, computer viruses, worms and also not Spain. So let's start with non technology related bullshit in that book.

Seville — Spain

I personally was never in Seville itself but I'm darn sure that the description of Spain in that book is terrible wrong. On my last visit to Spain it was a country like Italy or any other southern European country thus quite safe, friendly and modern.

There are two sentences in that book that would hurt every Spanish:

"He'd forgotten: Getting an international connection from Spain was like roulette, all a matter of timing and luck. He'd have to try again in a few minutes."

"A punctured lung was fatal, maybe not in more medically advanced parts of the world, but in Spain, it was fatal."

I don't think I have to further comment that sentence.

Technology

Before heading over to cryptography let's have a look at the dozens of other flaws in that book. Let's start with the obvious ones.

About Viruses And Worms

Half of the story is about a computer that overheats because it's processing a virus instead of an encrypted file. Now this is not a virus, it's not even a worm like mentioned later. It's just bullshit. First of all in order to brute-force something you have to know how it works, but that's covered later. However if you then brute force something you don't execute it. Say that this file contains indeed malicious code. Then nobody would have executed it with root rights. I doubt NSA would break code in something that is not a sandboxed environment.

Also that this program then heads over to the database in order to bypass the security systems is not only unlikely, it's impossible. If it was possible for the program (and TRANSLATR would execute the code during the decoding process) to gain access to the database it would not be able to shut down the security systems of that database. Not even in the worst computer setup the database software would run in the same security context as the security systems like the firewall.

Reading those parts of the book really hurts. Especially at the end when he starts throwing buzzwords around. Then there is a countdown and the security systems for X11 and FTP go down. Not before all systems are down anyone can gain access. Well, and then the hackers try to break into that system all day long. Thus as soon as the system is down the first attacker would already have access to all data.

An attacker would still need credentials to log into an FTP system. And I doubt that NSA would use FTP and X11 in their mainframe. FTP is a insecure protocol and X11 is the system used by UNIX systems to display graphical user interfaces over either the network or on the local computer. I doubt they would make X11 listen on anything else than localhost...

Mutation Strings

Now what the hell are mutation strings? The only thing I know sounding familiar are mutable strings. But I doubt Dan Brown thought about that, especially since those are not virus specific. There is one thing viruses do that has to do with mutation: Changing executables at runtime, but that's not called mutation string.

Passwords and Keyloggers

Greg Hale installed a keylogger in nearly every keyboard on the NSA complex according to the book. While this is a realistic scenario there is still something you should think about: We're talking about the NSA, they should use more sophisticated logon mechanisms than 5 character long passwords.

I know many people (including myself) who use passwords with more than 15 characters. I doubt someone working for the NSA would use a 5 character password.

Tracers

According to the book it's possible to find out the real address of a person by sending him a "tracer". A small programm forwarded via the remailer to the destination address which then sends an invisible message back to the NSA revealing the real mail address. This is complete bullshit of course. Unless there is a security hole in the mail client that executes the code, deletes the mail and sends a message back this can never happen. And that the mail client is that buggy is doubtable. This could be possible via social engineering but since the target is neither stupid nor alive Susan won't have received a mail.

Cryptography

Now things are getting worse. If you don't have a clue about what you're writing regarding technology that's bad. But if your book is all about cryptograhy and you don't know anything about it and mix up words that's really bad.

Brute Force

Dan Brown thinks you can break everything with brute force, out of the box. That's wrong. In order to break something using Brute Force you have to know how it works. Brute Force is a great method to recover passwords to break into systems as long as nobody locks you out because of those many login attempts, but you cannot break ciphers using brute force if you don't know how they works.

And Dan Brown is looking for a method you cannot break: That's called One-Time-Pad, is mathematically unbreakable and easily implemented. The idea is that you use a polyalphabethic cipher where the key is completely random and as long as the clear text and only used one time.

Rotating Cleartext

Flaw 1: the rotating cleartext algorithm. There is no such thing. There is an addition to the Vigenère cipher which is called "autokey" and uses the cleartext to expand the key instead of repeating the key. As soon as the key is exhausted it appends the cleartext. But it's breakable too.

bit ≠ characters ≠ passphrase

A 64bit key does not mean that the passphrase is 64 characters long. And breaking a 64bit key takes quite long. Today we're using more than 64bit to encode messages. Wikipedia has a nice example calculation of a bruteforcing 256bit key:

AES permits the use of 256 bit keys. A 256 bit key requires not merely twice as long to crack as a 128 bit key, but rather 2¹²⁸ times as long. If a device could be built that could check a billion billion (10¹⁸) AES keys per second, it would require 3,671,743,063,080,802,746,815,416,825,491,118,336,290,905,145,409,708 years to exhaust the 256 bit key space.

That in mind smashes the whole plot of the book.

Bergofsky Principle

Something Dan Brown mentiones often in the book which says that "if a computer tried enough keys, it was mathematically guaranteed to find the right one.". As mentioned about that is wrong. (One-Time Pad)

Public-Key Encryption

Dan thinks that you need the senders pass key to decrypt a message encrypted using public-key cryptography. This is just wrong...

Auctioning the Pass-Key

The "bad guy" sells the key on ebay. But the algorithm is encoded with itself. So how should someone every unlock that algorithm if you need the algorithm to unlock it which is locked by itself?

Conclusion

Normally I don't mind if a book contains wrong facts. But this book uses wrong facts in an inflationary way. It would be easy to add new items to this rant since there are enough inconsistencies and other wrong things in the book, but I'm out of time.

The book is boring and the plot childish. Not worth the money.

Introduction

First of all I have to defend vim. Many people think vim is a hacker tool and real developers (apparently hackers are not developers?) use IDEs. Well, that's only partially true. I use a IDE for C# and Java too but that has to do with the fact that those languages are completely different to ruby, python, HTML markup or whatever I use vim for. And if there would be an vim Ecplipse plugin I'm sure that I would use it.

Vim is still the best editor for me. The graphical version has support for many colors (i like them ^^), many different syntax schemes and it handles nested syntax blocks hell (eg, jinja templates in HTML).

Theme

For the plugins listened below I wrote a theme for myself which adds more colors to the syntax elements provided by them. The theme has different colors for most of the syntax elements and is optimized for python, HTML, django/Jinja, JavaScript and Ruby.

It's quite colorful (too colorful for many ^^) and is a GUI only theme. See the screenshots below for some examples.

• fruity.vim

Syntax Files

The default vim syntax files are nice but not colorful enough for me. Especially the default HTML and Python files are crap in comparison what you can get on vim.org. However I did some local changes on some of the files for my needs and I don't upgrade those scripts because of that. If you want exactly the files I use have a look at them.

Note that some of them mainly exist because they are a requirement for another file (all those svn files belong to each other, mathml and svg belong to html). The eruby.vim file is my own one with support for % lines not supported in the version from the vim webpage and it also handles comments correctly.

• eruby.vim
• html.vim
• javascript.vim
• mathml.vim
• python.vim
• svg.vim
• svnannotate.vim
• svnlog.vim
• svn.vim

Plugins

The same applies for the plugins. I use the minibufexplorer (interestingly I don't use vim 7 tab feature myself), the close tag plugin and the textmate snippets emulation called SnippetsEmu. Both plugin unchanged from the vimscripts webpage:

• SnippetsEmu
• minibufexplorer
• closetag
• Jinja Highlight Suppoprt

Snippets for SnippetsEmu

I don't use the original snippets because when I first tried out SnippetsEmu it shipped some i don't wanted to use and those I would have used were completely broken. Because I haven't upgraded the plugin I still use mine. If are interested in, here they are:

• css_snippets.vim
• eruby_snippets.vim
• htmljinja_snippets.vim
• javascript_snippets.vim
• python_snippets.vim
• django_snippets.vim
• htmldjango_snippets.vim
• html_snippets.vim
• jinja_snippets.vim
• ruby_snippets.vim

vimrc

Okay. Now to my vimrc. In order to get that thing working you have to install all of the plugins from that page. At least I guess so because it configures a lot of the plugins here.

• .vimrc

Tips And Tricks

Here some various tips and tricks for vim I use every day.

The * and # Keys

In normal mode you can use * and # to search for the word under the cursor. * search forwards, # backwards.

The . Key

Also useful is the . key. In normal mode it repeats the last change. Very useful for XML editing and where you have to do similar stuff.

The % Key

Pressing the % key in normal mode while being on a parenthesis or a similar construct jumps to the opposite parenthesis. There are a couple of plugins that extend that functionallity but the default behavior is usually good enough for most of the programming languages.

The < and > Keys

When working with Python you often have to indent or outdent a couple of lines. Just mark them using the visual mode and press > to indent them. press 4< to outdent them 4 steps etc.

Advanced Undo Features

Vim has a different system of undoing changes. The normal undo is u in normal mode and ^r is redo. But what happens if you undo a few things and then change something? A normal editor forgets about the new changes. It's not possible to redo that again.

In Vim it's different. Vim starts a new undo branch. Using :undolist you can have a look at the possible undo states. With :earlier 20s / 1m / 2h you can then go back 20 seconds, one minute etc. Traveling forward in time works using :later and a timedelta as argument.

The Search Feature

I really like the firefox search feature. Just hit "/" and you can search in the current file and get the results in real time. That works with vim exactly the same. All you have to to is to either add :set incsearch into your vimrc or type it into the command prompt. Using n you can go to the next result. If you want to have all the search results highlighted use :set hlsearch. Hiding the results works using :nohl

Closing XML Tags

If you have the closetag.vim plugin installed (link above) you can add this to your vimrc in order to get the feature working: autocmd FileType html,xhtml,xml source ~/.vim/scripts/closetag.vim (update the path to your installation and your filetypes of course) Once this is done you can close open tags using ^_.

Regexp Replacement after Search

If you searched for a text using /foo and you now want to replace the found results with something you don't have to write this regexp again. Just do :%s/<c-r>//replacement/g and <c-r> is automatically replaced with the last search regular expression.

Using Bookmarks

Bookmarking in vim is darn easy. If you are on the current line just bookmark it with mX where X is a lowercase letter from a to z. Go to that mark using 'X where X is the same letter again. Using '' you can jump back to the position you were before jumping to the bookmark. You can get the list of bookmarks using :marks.

The Vim File Browser

The Vim File browser is a nice thing. If you open a file using :e and that file is a directory you get a nice file browser for the files in that directory. If you are a python developer you probably want to filter some files out (*.pyc etc). Add this to your vimrc: let g:explHideFiles='^\.,.*\.pyc$' This hides pyc files as well as hidden files.

The Wildmenu

The best vim feature is the wildmenu. Add a set wildmenu to your vimrc and discover the possibilities of filesystem surfing ^^ Enter :e in the command line and press ^D. Vim will show you all possibitilites in a nice little window. By entering the start of a filename and pressing tab it completes for you then. If it was a folder you can now press ^D again to get the contents. Once you finished the command this window will disappear again and you can continue working. Works of course for all commands not only the open command.

Vim Completion Features

Last tip but maybe the most powerful one :). If you have vim7 and omni completion for your language you can use ^X^O to get the completion similar to intelli sense or how it's called. Note that the default color (pink) can be overridden in the theme. In my fruity.vim file i use this to get white letters on a dark red background: hi Pmenu guifg=#ffffff guibg=#cb2f27

Also nice is ^N. It looks up all used words in the open buffers and presents then in a dropdown completion. Useful if you have long variable names and don't remember them. Just type in the start of the variable name and press ^N. Vim will either complete it or show the list of possibilities.

Screenshots

Language Design

A recent discussion with someone (which will go unnamed) again showed up that many people think python has a bad language design. Things that are "wrong" or missing in python are:

• continuation support
• ruby like blocks
• anonymous functions
• weak functional programming support
• statements that aren't expressions (assignments)
• bad object orientation

Interestingly just one of this points is a problem for me. The latter. I would really like to have assignments as expressions but I can also understand Guido who wants to avoid mistakes in the application logic that are caused by an assignment which should be an comparison.

Now to the items on the list. Especially the missing continuation support. Because I never was in a situation where I needed them. By now I could solve all of my problems using generators or context managers.

Because of the fact that python functions are objects I can define them in any namespace. A function in a function doesn't look like an anonymous one but semantically there are few differences. And if you say there is a weak functional programming support I cannot assent. Python has list comprehensions, generator expressions, first class functions and there are libraries that enable currying.

And now to the bad object orientation. Some people think the python object system is bad because it's flange mounted (the explicit self) or because some functions like len() are globals and not instance methods. The excplicit self is something you have to live with. There are some reasons for that implementation detail, one of them is that python doesn't have a ruby like super. Instead of calling a method of the superclass you just call a method of any class in the MRO and provide the self (MyClass.my_method(self, arguments, go='here')). That is much nicer than MyClass.method(:my_method).bind(self).call(arguments, :go => "here") like you do in ruby.

The global functions on the other side are something that is indeed interesting. When I first worked with Python i wondered why some functions are global. Among those are len(), str(), abs() etc. But the idea is in my eyes a very good one. Instead of adding a property called .length to an object you provide a method named __len__() which will be called by the global len() function. The big advantage: You can still use .length for whatever you want, type checks for hasattr(foo, '__len__') won't yield wrong results because __len__ is a special method and doing something else than returning the length of an object is just silly. I know a lot of programming languages (For example C#) where different objects have different size methods/properties (Size, Length, Count) and without and IDE or a good memory you have to have a look at the documentation then.

The str() in the last paragraph is global too. But not a function. It's the class of strings and because of that not an instance method. To control the string conversion you can however provide a __str__ method. This is something i personally really like.

Things I miss in ruby are:

• support for boolean value overloading
• support for overloading the function call operator
• unicode support
• slightly better threading support (at least native threads)
• native generators
• first-class methods. (I'm not talking about Proc or code blocks)
• slightly cleaner syntax

Maintainability

Say what you want but python is easier to maintain. For sure there are some libraries that are written like the developer was stoned but those are exceptions. Python doesn't support complex constructs like assignments in expressions, magic local variables like $~ and there is no implicit return value which can screw things up or make it hard to find out where exactly the return value is coming from. There is also no callcc which might be a cool thing but is hard to understand for people not using it. But there are also constructs in python that can really fuck up semantics. For example metaclasses. They are really great if you use them wisely but never try to do something with them nobody undestands.

Object System

Python and Ruby are both object orientated languages. But there are a few differences. Ruby doesn't have metaclasses and no multiple inheritance. On the other side there are special "singleton" objects which support some basic "per class" manipulation. This might be enough for ActiveRecord and other libraries to modify classes but I don't really like the idea of it because it's not clean in my eyes. Here the difference explained.

The following example shows how metaclasses in python work:

>>> class ClassOfFoo(type):
...  def method_of_foo(self):
...   print repr(self)
... 
>>> class Foo(object):
...  __metaclass__ = ClassOfFoo
... 
>>> isinstance(Foo, ClassOfFoo)
True
>>> Foo.method_of_foo()
<class '__main__.Foo'>

And here the same functionallity in ruby:

irb(main):001:0> class Foo
irb(main):002:1>  class << self
irb(main):003:2>   def method_of_foo
irb(main):004:3>    puts self.inspect
irb(main):005:3>   end
irb(main):006:2>  end
irb(main):007:1> end
=> nil
irb(main):008:0> Foo.method_of_foo
Foo
=> nil

But that's not the class of a class. This just modifies the singleton of a class object to add a new method. Metaclasses as such don't work, also not without syntactic sugar because you cannot subclass Class:

irb(main):009:0> class FooType < Class
irb(main):010:1> end
TypeError: can't make subclass of Class
        from (irb):9
        from :0

This might be only a small detail but I like it a lot and I don't want to miss it.

Regular Expressions

In Ruby 1.8 regular expressions are terrible limited. For me (who uses regular expressions in nearly every library) this is a major annoyance. fullstop.

Libraries and Webdevelopment

At least for the moment Python has the bigger library database. And even although easy_install is something I find a bit broken it's a lot better designed than gems. Especially because it doesn't require changes in the ruby libraries that use gems and it doesn't have to unpack eggs to import them. (Which is also faster)

Also webdevelopment with python makes more fun. At least for the moment. For basic CRUD stuff you can use django, for advanced applications there are hundreds of libraries and the great WSGI specs. Ruby got a WSGI clone too some time ago called Rack, but it's still not as accepted as WSGI.

Access To Internals

Python allows accessing many aspects of the interpreter core. Some of them are forwarded to python (frames via sys._getframe or tracebacks), tracebacks itself, a python compiler and parser (via the compiler package or the new _ast module). Some of them are only available via the C API. But that doesn't really matter since you can use ctypes.pythonapi to access the python C API. (Still, you don't really have to do that)

Conclusion

I hope you can now understand why I personally like Python more than ruby. But don't get me wrong. I like ruby too and I already wrote some libs for it and love to play with it. But for real applications I prefer Python.

Disclaimer: I wrote this article not to tell anybody how great python is. I don't want to tell anybody how stuipd Ruby is. And I certainly don't want to convert any Ruby developer into a Python developer! The world isn't black and white and I'm pretty sure other people have different requirements and will love Ruby more than Python.

RTF is really one of the worst markup languages ever created. It looks like the human unreadable version of LaTeX and is heavily bound to Microsoft® Word®. Until I guess Microsoft® Word® 97 RTF just saved documents in ANSI charset. So. What the hell is an ANSI charset? I have no idea. Looks like ANSI is a mixin charset where each font definition can refer to a different codepage. In fact I don't mind because there is a way to embed unicode:

Binärcode

Looks like this in RTF:

Bin\ud{\u228\'e4}rcode

First you start the small adventure by opening a new block prefixed with the \ud keyword that let the RTF reader know that here starts an Unicode Part. The next one is the 16 bit signed integer number of the character code point in the unicode table. Yeah. Singed. And 16 bit long... So not even utf8 fits into what RTF thinks is unicode.

Anyway. The next part after the unicode codepoint number is the ANSI codepoint number. Because I'm lazy in my example I just encoded that again into iso-8859-1. If the character isn't in ANSI it should use the closest possible character. (Just replace it with ? ^^)

Check out the module for more information about the implementation if you like.

And here some funny facts from the specification (which is btw a Microsoft® Word® document packed in a zip exe file which is packed in a cab extract exe file):

The hidden style property can only be accessed using Microsoft® Visual Basic® for Applications.

That's the footnote for the \shidden style control word

\oldas

Use Word 95 Auto spacing

\lnbrkrule

Don’t use Word 97 line breaking rules for Asian text.

\bdrrlswsix

Use Word 6.0/Word 95 borders rules.

Great huh? I'm looking forward to discover the differences in the spacing and text breaking rules between different Microsoft® Word® versions.

Here the code for dumping and restoring a roster. It requires that you have xmpppy installed

import sys
from xmpp import JID, Client
from pickle import dump, load

try:
    MODE, FILE, USERNAME, PASSWORD = sys.argv[1:5]
except ValueError:
    print 'usage: %s <mode> <file> <username> <password> % sys.argv[0]
    sys.exit(1)

jid = JID(USERNAME)
cl = Client(jid.getDomain(), debug=[])
if not cl.connect() or not cl.auth(jid.getNode(), PASSWORD):
    print 'couldn not connect to server'
else:
    cl.sendInitPresence()
    roster = cl.getRoster()
    if MODE == 'dump':
        data = {}
        for jid in roster.getItems():
            data[jid] = roster.getRawItem(jid)
        f = file(FILE, 'w')
        dump(data, f)
        f.close()
    elif MODE == 'load':
        for jid, data in load(file(FILE)).iteritems():
            roster.setItem(jid, data['name'], data['groups'] or [])
            roster.Authorize(jid)
            roster.Subscribe(jid)
    else:
        print 'error: mode must be load or dump'
        sys.exit(1)

Use it like this:

python rosterdump.py dump roster.dump yourname@example.com yourpassword

And restore it using this command:

python rosterdump.py load roster.dump yourname@otherserver.com yournewpassword

Note that i won't further work on that script and it's possible that are some errors in the script. I don't guarantee that it actually works :D Nevertheless I hope it might be useful for you.