Armin Ronacher

I'm started using trac short after the 0.8 release. And damn, it rocks. the pocoo trac is currently running version 0.10dev with 10 plugins. I never saw a web application with such a great plugin interface before. hope the pocoo plugin interface isn't worse ;). In the following post I'll cover some of the great plugins we use for our trac

I think everybody knows trac. trac is the whizkid under all bugtrackers. It features a flexible plugin interface, support for multiple versioning systems and has the far most initiutive user interface among all OpenSource trackers I know.

For the pocoo trac I installed many plugins. Here the list of plugins we use and why we use them ;-)

trac 0.10dev

The trac core in version 0.10dev. Why not the stable version? Because I havn't found any bigger bugs in the trunk version we're running and some plugins depend on this version.

TracSpamFilter

Everybody hates Spam. And in the last months many tracs suffered under spambot attacks. This plugins supports akismet and a ip blacklist for catching spam. But also the moin antitrac system which means it helds a central BadContent page with a number of regexes. If one of those regexes matches the user can't save the page. Moin automatically syncs that page with moinmaster, trac doesn't. Because of this I've written a small plugin that automatically syncs the BadContent page with an other wiki or trac defined in the trac config:

spamsync

Like mentioned before this plugin allows the trac to sync the BadContent page with a master wiki. It's not stable, use it at your own risk, but it seems to work in the pocoo trac.

TracTicketDelete

If a spammer was successfull (yeah. even with a antispam plugin this can happen) you can't delete a ticket using tracs techiques. This plugin gives you a "ticket delete" button in the webadmin interface which you can use to delete tickets including all comments on it. Very useful.

WebAdmin

Required by the TicketDeletePlugin and very usefull you don't want to ssh into the server to update components, milestones or other settings. Also the only way if you want all TRAC_ADMINs to edit the settings without giving them ssh access.

XmlRpcPlugin

Allows you to connect using xmlrpc to your trac and query the database or update tickets. Very nice when you want to write some cleanup scripts for the trac etc.

CodeTagsPlugin

Everybody knows those XXX, TODO, etc. tags in the sourcecode. This plugin allows you to display those tags on one page. IMHO very useful. But note: It's not supported by me ;-) But if you found a bug, query me on irc.freenode.net (nickname mitsuhiko) or send me a mail.

IrcLogsPlugin

This plugin parses the logfiles of a supybot and displays them in the trac: #pocoo logs. Also not supported by now ;)

The other plugins we use: trac blog and trac repo search very useful too.

Last but not least: thank you trac team ;-)

There are news in mitsuhiko land. I've now more time for my projects and am angry about the current situation in Libanon, start writing on my pypy "Fachbereichsarbeit" try start driving cars and much more.

First the positive aspects of this blog posts. Since two weeks I have summer holidays. That means no school until the September 9th. I'll use my spare time for starting writing on the "Fachbereichsarbeit" for my school leaving examination which topic will be "pypy". In Austria you have to posibillity to write a 30 page long article about a topic you can choose as replacement for one final examination. So I'll write about pypy (at least I hope I can write about it, the teacher has to approve it).

The second great thing about the holidays: I'll spend two weeks in Ireland. The funny thing about it: Alexander Schremmer informed me that one week after I'll leave Ireland there will be a pypy sprint. Maybe I manage it to extend it somehow. At least I'll print out the pypy docs and read it.

Since Monday I'm also participating in the lessons of the local driving school. I hope I can use my holydays to drive at least 2000 kilometers. I need 3000 too be able to gain the driving license before I'm 18. (this is called L17 here in Austria. You drive 3000km and gain the driving license with 17)

I'm also working again with the rest of the pocoo team on... pocoo ^^. A more or less up to date version is online at area51.pocoo.org. beewee thinks that we can do an early feature freezy in three months. Hopefully he is right. I'm really sick of our phpbb.

And now to something that currently makes me really angry. It's about the situation in Libanon. Beside that I really hate war at all (I hated the war against Iraq, Afghanistan as well as any other war I heard of since I'm able to switch on the television or read the newspaper) I'm especially angered about this one. When you read the newspaper you just read everything from the view of Israel. It really looks like there is a country (Israel) and a anticountry (Libanon) which just brews terrorists. When I found the blog codedemigod by a programmer from Libanon it was the first chance for me to see the situation form the other side too. Goddamn. Those are people too. And they don't do anything wrong. In my oppinion the wholeh situation down there is just crazy. And after reading the response to my comment I really think about informing myself about the situation down there and not just to trust the newspapers and there all-mighty-point-of-view.

I had a discussion about that thingy on #pythonpaste a few hours ago. Looks like I'm the only person hating thread.local. Maybe I'm ultraconservative but I think I've some good argument agaist the use of this magic type.

preamble: This is my point of view. I know that thread local objects are a nice way to fix the problem with not passing request/session/or any other objects to methods but I don't like them. I'm not fighting against pylons or any other application using thread locals nor do I want the pylons developers to remove thread local objects.

In this particular case I want to talk about the thread local object in pylons. Pylons uses some magic thread local objects named m, g, h, c, cache, session, request, response, params and buffet. You can get those object by importing them from the yourapplication.lib.base package.

Some of them are deprecated and exist just because of backwards compatiblitity.

I was told that pylons uses paste.registry to be able to have more than one application in the same thread without getting threading issues. So it should be technically possible to write something to m without touching the m of another pylons application.

Looks like (beside the fact that the variable names are not that easy to understand when first looking at the code) pylons doesn't have any problems with thread.local since they workaround it with paste.registry

But what about web.py for example. It overwrites sys.stdout to redirect printed data directly to the request of the current thread. Nice idea but using this technique it's only possible to have one running web.py application in the same thread. *grmbl*

django doesn't use thread.local objects (not true, afair it uses thread.local objects somewhere in the core) but lacks a deployable system. You can't mix up django application since they mostly end up in name(space) clashes of either models, templatetags, templates or other magic tags. It's not possible to run two django installations in the same process since they relay on os.environ['DJANGO_SETTINGS_MODULE'] or a directly passed settings module.

I really like django, but never try to write an application you want to distribute.

Same for SQLObject. Imagine MoinMoin and pocoo would both use SQLobject and a pocoo plugin imports the moinmoin package, create a moinmoin wikiconfig and try to parse a wikipage. If you overwrite the global sqlhub the later importer moinmoin would overwrite the connection settings of pocoo. Someone on #python.de also reported stupid threading issues with sqlobject which appeared from time to time. It ended up that he switched to sqlalchemy which doesn't use thread locals any more (sice 0.2). (You can still use the magic technologie, but you have to enable it first)

As and programmer and admin I really like to have applications you can just-deploy (tm).

As just deploy i see something you can put into a debian repository and update using the normal update procedure (apt-get update, aptitute, synaptic...) And I don't want a site-packages for apps like pocoo, moinmoin since they arn't libs but applications. And I want to mix up applications in the same process like I do on ubuntuusers to be able to share user accounts, user settings, the same mailing system etc. between many applications.

And thread.local as such breaks this. paste.registry fixes it again I was told, but one thing is still left:

It's magic, hard to understand for newcommers, breaks my dir() and is slower then passing the given objects to the app

Okay. One of this points is premature optimization ;-)

• magic / hard for newcommers - immagine that you have a project using paste.registry for a global request and response variable. You also have a globals variable to put request variables in. How do I tell a PHP developer switching to this project (this currently happens at ubuntuusers.de where it was impossible to find any python devs with web experience) that accessing myapp.request, myapp.resposne and myapp.globals is safe whereas accessing any other global namespace, object etc. isn't. It does not make sence at all ^^ Why not pass a request object to the view? Seven chars more to type? or three in case of just writing req.
• dir() broken - i like dir(), it is the best to debug ^^ Just raise an exception, use a middleware with eval() support and navigate to the frame with the problem. dir() the object and see what's broken. Works as long as you don't use a magic thread local object which breaks dir()
• slower - it actually is premature optimization but fetching current thread and looking up an object in a dict each time you access it is not very fast.

And here an example of how pocoo defines its views:

from pocoo.pkg.core.models import Forum
from pocoo.application import RequestHandler
from pocoo.template import PagePublisher
from pocoo.http import TemplateResponse


class IndexPage(RequestHandler, PagePublisher):

    def get_page_name(self):
        return 'index'

    def get_relative_url(self):
        return ''

    def get_handler_regexes(self):
        yield u'$'

    def handle_request(self, req):
        categories = req.db.select(Forum, Forum.c.parent_id == None)
        return TemplateResponse('index.html',
            categories=categories
        )

Oki. Maybe the imports suck. But everyone who is reading that source the first time knows where the objects are comming from.

And here a pylons/thread.local version of the above code:

from pocoo.pkg.core.models import Forum
from pocoo.template import PagePublisher
from pocoo.http import *


class IndexController(RegexController, PagePublisher):

    def get_page_name(self):
        return 'index'
        
    def get_relative_url(self):
        return ''

    def get_handler_regexes(self):
        yield u'$'

    def handle_request(self):
        c.categories = d.select(Forum, Forum.c.parent_id == None)
        m.subexec('/index.myt')

This isn't valid "pylons" :-) But it uses the ideas of pylons. Those global c, m etc objects. Of course it's less to write. But don't tell me it's easier to understand

jinja 0.8 is out now. Here the changes from the 0.7 release. read on

Here the changes in Jinja 0.8:

• a new EggLoader by Jon Rosebaugh which allows you to load templates from inside eggs
• simple comparison tags: {% if variable equals "some value" %} and {% if not variable equals other_variable %}
• support for memcaching of extended templates by improving the loader interface
• loaders now get passed the name of the parent template which loader developers can use to allow relative imports and much more.
• Added filters to convert variables to other types. |bool, |int, |float and |makebool which converts yes, on, true and 1 to a real boolean True.
• jinja is now licensed unter the BSD license which allows the use of jinja in non gpl projects too

On the roadmap for jinja 0.9:

• better syntax error messages
• *insert your wish here*

The last list item is very important because I don't know what to implement ^^

Hiho to my first python tutorial. Julian Krause aka thecrypto aka Mr. RhubarbTart mentioned the eigenclass article Plugins in your Ruby Application and noticed that there is no such tutorial for python. So: here it is ;-)

Since version 2.2 python ships a new object model (explained in the documentation) which allows very complex class definitions including metaclasses and other features you probably won't use. But some of them are very useful if you want to create a plugin system.

The hardest part is the loading of these plugins. There are basically three ways for loading plugins:

• using an additional folder `plugins/` which gets added to the PYTHONPATH (sys.path)
• writing an import hook which will handle that (I won't talk about this method here since it's very complex and not necessary in most cases)
• Using setuptools as plugin system

This tutorial will just cover the "old" approach without the use of setuptools. A latter tutorial will introduce setuptools as well, but at first just the plain-old-python way.

Each plugin should inherit from a special base class which keeps track for all of it's childclasses.

class Plugin(object): 
    pass

Now we have a base plugin :-) What is so special about this class? Nothing except that this class inherits from the special base class called object which no metaclass. If you don't inherit from object (if you don't define a parent for your class which is object or a subclass from object) python adds types.ClassType as metaclass to your class.

Every newtype class knows about it's children:

>>> class MyPlugin(Plugin): pass
...
>>> class OtherPlugin(Plugin): pass
...
>>> Plugin.__subclasses__()
[<class '__main__.MyPlugin'>, <class '__main__.OtherPlugin'>]

Very useful for plugins. As long as they where actually loaded. So we need to load all required plugins. Now we have two possibilities:

• use a config file where you enter each plugin
• scan the plugin folder on application launch and import everything that ends with .py or is a folder with a __init__.py in it.

In this tutorial I'm just covering the first case since the setuptools approach automagically imports plugins later.

import sys
import os


class Plugin(object):
    pass


def load_plugins(plugins):
    for plugin in plugins:
        __import__(plugin, None, None, [''])


def init_plugin_system(cfg):
    if not cfg['plugin_path'] in sys.path:
        sys.path.insert(0, cfg['plugin_path'])
    load_plugins(cfg['plugins'])
    

def find_plugins():
    return Plugin.__subclasses__()

Now save it as plugins.py. To enable the plugin system you can now use init_plugin_system:

>>> from plugins import init_plugin_system, find_plugins
>>> init_plugin_system({'plugin_path': 'plugins/', 'plugins': ['testplugin']})
>>> find_plugins()
[<class 'testplugin.SpecialPlugin'>, <class 'testplugin.AnotherPlugin'>]

testplugin.py was just a small test file with two classes inheriting from plugins.Plugin.

The next a plugin needs to do is to tell the programmer of the application it's capabilities. Therefore you could use Interfaces, duck typing, subclassing or just a list with exported capabilities. The latter is very basic and convenient therefore we implement it:

class Plugin(object):

    capabilities = []

    def __repr__(self):
        return '<%s %r>' % (
            self.__class__.__name__,
            self.capabilities
        )

def get_plugins_by_capability(capability):
    result = []
    for plugin in Plugin.__subclasses__():
        if capability in plugin.capabilities:
            result.append(plugin)
    return result

Now an example plugin defines one or more capabilities. The names of the capabillities should be explained somewhere. As well as the methods the plugin should export.

from plugins import Plugin

class ExamplePlugin(Plugin):
    capabilities = ['foo']

    def do_foo(self, name):
        return 'Hello %s!' % name

Saved as testplugin.py you should be able to load it:

>>> from plugins import init_plugin_system, get_plugins_by_capability
>>> init_plugin_system({'plugin_path': '.', 'plugins': ['testplugin']})
>>> for plugin in get_plugins_by_capability('foo'):
...     plg = plugin()
...     print plg.do_foo('Huhu')
...
Hello Huhu!

This example also demonstrates the current problem. You have to create a new instance after each iteration since get_plugins_by_capability yields classes, not instances.

But it's possible to fix this:

_instances = {}

def get_plugins_by_capability(capability):
    result = []
    for plugin in Plugin.__subclasses__():
        if capability in plugin.capabilities:
            if not plugin in _instances:
                _instances[plugin] = plugin()
            result.append(_instances[plugin])
    return result

Now the function returns singleton instances, first created when required:

>>> from plugins import init_plugin_system, get_plugins_by_capability
>>> init_plugin_system({'plugin_path': '.', 'plugins': ['testplugin']})
>>> for plugin in get_plugins_by_capability('foo'):
...     print plugin.do_foo('Huhu')
...
Hello Huhu!
>>> from plugins import _instances
>>> _instances
{<class 'testplugin.ExamplePlugin'>: <ExamplePlugin ['foo']>}

done :-) You now should have your own plugin system in 38 lines of code ^^ Of course there is still much too do but it's enough for a very basic plugin system. I'll introduce you to the setuptools plugin way tomorrow :-)