lucumr

Imagine for the moment there was a PHP blog software that has the ability to dump the blog posts into some sort of extended RSS 2 feed and import from there later and probably from a different installation. That’s nice, XML is a flexible format and RSS allows extensions via namespaces. Even better, there are XML parsers for all major programming languages and from python working with XML is especially cool because of lxml and element tree. But there is a problem with that…

…that XML, is not XML. It’s called WordPress eXtended RSS (WXR) but it’s not XML? And why in god’s name did nobody notice so far? I mean, WordPress must have an importer for that.

Why it’s not XML? It has XML syntax, XML namespace declarations but what doesn’t it have? A doctype. What’s the problem? It’s referencing HTML entities! So step one for parsing: inject an inline DTD that defines those entities. Great fun isn’t it? Then it parses. I was happy and finished my work. That XML doesn’t have HTML entities is something PHP developers probably don’t know and their parser isn’t resolving any entities during the parsing process. Or worse, their XML parser expands HTML entites.

But it’s worse! I loaded another dump that happened to have some broken HTML in comments (could happen, does happen, thanks broken trackback support). What happens next? THE XML DOESN’T PARSE ANY MORE! Why? Because comments are neither escaped nor marked as CDATA. I wonder why, especially because it’s so much easier to handle embedded HTML/XHTML for dumping as cdata and not XML, especially if you are working with PHP.

But WordPress was able to import that…. so I looked at their parser…. WORDPRESS PARSES THAT WXR FILE USING REGULAR EXPRESSIONS!!! Argharhgarhghargh. That’s not XML what you are doing there, that’s nothing. WordPress can’t even parse it’s own file if you bind the WordPress exporter namespace to a different prefix! WordPress can’t handle it’s own file if you replace their CDATA foobar against properly escaped stuff. Dammit!

I can’t even write a proper exporter using XML tools because what my XML tools generate is not compatible to WordPress. And what tops it all?

Reading that in the #wordpress channel:

<nickname_deleted> why does it matter what wp's xml format has flaws?
                   adapt your importer to the flaws

ARGHARGHARGHARGH. and then the webpage says:

WordPress is a state-of-the-art semantic personal publishing platform with a focus on aesthetics, web standards, and usability.

Without further comments… I lost my faith into standards that moment. Wait a second, I lost it earlier. Still sad.

Some time ago I wrote a blog engine in Python to replace this wordpress installation. As you can see that never really happened although the blog software is already in a usable state (at least the basic administration and plugin interface works). One of the reasons is that i would have to port over the theme which is one of those stupid tasks i just hate, another one is that i haven’t had time to improve it any further. Another piece of software I recently discovered again is a python powered wiki called ordo which I wrote one year ago i guess. It’s a shame that those applications never where released or properly licensed. ordo lingers around in my svn repository but textpress does not.

TextPress won’t for a few reasons. One is the name, it’s obviously taken from wordpress and want to rename it before I release it, the other reason is that i don’t have the time to maintain the code right now. There are already many other projects and I don’t feel like I want to maintain too much code at the same time.

But because TextPress has some really unique features I at least want to show what it does :)

The screenshot above shows the default theme, the admin panel looks like this and this. It has a pretty interesting plugin interface. The event system was inspired by dokuwiki, the way the syntax based plugins work is IMHO unique :)

Basically what it does is lexing the post sgml markup into a DOM like structure which you can query and transform. Here for example the complete sourcecode of the pygments plugin:

from textpress.api import *
from textpress.htmlprocessor import DataNode
try:
    from pygments import highlight
    from pygments.lexers import get_lexer_by_name
    from pygments.formatters import HtmlFormatter
    have_pygments = True
except ImportError:
    have_pygments = False

class PygmentsHighlighter(object):

    def __init__(self, style):
        self.formatter = HtmlFormatter(style=style)

    def process_doc_tree(self, event):
        for node in event.data['doctree'].query('pre[@tp:lang]'):
            lexer = get_lexer_by_name(node.attributes.pop('tp:lang'))
            output = highlight(node.text, lexer, self.formatter)
            node.parent.children.replace(node, DataNode(output))

    def get_style(self, req):
        return Response(self.formatter.get_style_defs(), mimetype='text/css')

    def inject_style(self, event):
        add_link('stylesheet', url_for('pygments_support/style'), 'text/css')

def setup(app, plugin):
    if not have_pygments:
        return
    app.add_config_var('pygments_support/style', unicode, u'default')
    app.add_url_rule('/_shared/pygments_support/style.css',
                     endpoint='pygments_support/style')

    c = PygmentsHighlighter(app.cfg['pygments_support/style'])
    app.connect_event('process-doc-tree', c.process_doc_tree)
    app.connect_event('after-request-setup', c.inject_style)
    app.add_view('pygments_support/style', c.get_style)

You can see the event and doc tree system in action in the snippet above. The way the DOM is queried is inspired by jQuery ;-)