Setuptools Plugins
Welcome to part #2 of the Python Plugin System Tutorial. In this part of the tutorial I'll explain how to use setuptools to create a plugin system.
It really looks like eggs are the most hyped python related thing next to WSGI at the moment. Many applications and Frameworks like TurboGears, Trac, Paste and Pylons use eggs for their plugins. This isn't surprising since eggs are very easy to distribute.
The core of setuptools are the two modules setuptools and pkg_resources. The former is required for the setup.py which is the core of each egg. The latter allows us to query those created eggs when they export entrypoints.
WTF are entrypoints?
The peak wiki explains entry points in this manner:
setuptools supports creating libraries that "plug in" to extensible applications and frameworks, by letting you register "entry points" in your project that can be imported by the application or framework.
For example, suppose that a blogging tool wants to support plugins that provide translation for various file types to the blog's output format. The framework might define an "entry point group" called blogtool.parsers, and then allow plugins to register entry points for the file extensions they support.
This would allow people to create distributions that contain one or more parsers for different file types, and then the blogging tool would be able to find the parsers at runtime by looking up an entry point for the file extension (or mime type, or however it wants to).
So if we look back to the first part of the tutorial we used a list of capabilities. Basically we could Now create an entrypoint of each of those capabilitities. But we won't do that. Why? Because of three reasons:
- Each time you change one entrypoint you have to rebuild the egg-info folder.
- When the developer adds an feature to the plugin he would have to edit the setup.py file too.
- It's more complex to implement
You still want to use more than one entry point you can do so but this tutorial won't cover that case.
In this example the application is called myapplication.py and all plugins lay in an subfolder called plugins. We will call the entrypoint for the plugins myapplication.plugins.
So we should create a folder structure like that:
myapplication/ myapplication.py an empty file by now plugins/ an empty folder for all plugins
The first plugin
At first we have to create a new plugin, in this case we call it foo. Therefore we create a new folder structure below myapplication/plugins:
foo/
foo/
__init__.py an empty file
setup.py an empty file
The next step will be adding content to the setup.py file:
# -*- coding: utf-8 -*-
"""
A small example plugin
"""
from setuptools import setup
__author__ = 'Your Name Here'
setup(
name='Foo',
version='1.0',
description=__doc__,
author=__author__,
packages=['foo'],
entry_points='''
[myapplication.plugins]
Foo = foo:FooPlugin
'''
)
This file should be simple to understand, basically it's a normal distutils setup file. The difference is that we use distutils over setuptools and that he have defined an entrypoint. This entrypoint basically tells the application that the "Foo" plugin is a class called FooPlugin and located inside of the "foo" package.
Now we open the empty __init__.py file and add the following content:
# -*- coding: utf-8 -*-
class FooPlugin(object):
capabilities = ['foo_capability']
def do_something(self):
return 'Hello from %s' % self.__class__.__name__
This file looks like the plugins from the first part of the tutorial but it doesn't have to inherit from a base class. Here we just use the generic object baseclass which is the parent of all new style classes.
Now we have a folder structure for an egg. but not an egg. Since we don't want to rebuild an egg each time we change something on the sourcecode we use the develop option of the setup.py file. Therefore we open a shell and go to the folder with the setup.py file and execute the following command:
$ python setup.py develop --install-dir .. -m
The -m option is required, otherwise setuptools will fail because it can't find the ".." folder in the PYTHONPATH.
The Main Application
Now we have an egg in development mode. Basically it's a file called Foo.egg-link with the path to the sources in it.
Time to create the application:
# -*- coding: utf-8 -*-
import os
import sys
import pkg_resources
sys.modules['myapplication'] = __import__(__name__)
ENTRYPOINT = 'myapplication.plugins'
PLUGIN_DIR = os.path.join(os.path.dirname(__file__), 'plugins')
def init_plugins():
pkg_resources.working_set.add_entry(PLUGIN_DIR)
pkg_env = pkg_resources.Environment([PLUGIN_DIR])
plugins = {}
for name in pkg_env:
egg = pkg_env[name][0]
egg.activate()
modules = []
for name in egg.get_entry_map(ENTRYPOINT):
entry_point = egg.get_entry_info(ENTRYPOINT, name)
cls = entry_point.load()
if not hasattr(cls, 'capabilities'):
cls.capabilities = []
instance = cls()
for c in cls.capabilities:
plugins.setdefault(c, []).append(instance)
return plugins
plugins = init_plugins()
That looks complex. But it isn't. The sys.modules thingy is not needed in real applications, but since this application is a one file application it won't appear in sys.modules and plugins would have problems importing it. So we put the current file into sys.modules.
The init_plugins function basically scans the plugins folder for eggs whose entry point is myapplication.plugins, load that egg and use the returned class to look up the capabilities. After that it instanciates the class and creates a dict in the following structure:
{'capability': [list, of, plugins]}
Basically this now allows you to select all plugins of a given capability by doing this:
for plugin in plugins.get('my_capability', []):
For the sake of simplicity we can create two methods for querying plugins:
def get_plugins_by_capability(capability):
return plugins.get(capability, [])
def get_all_plugins():
result = set()
for p in plugins.itervalues():
for plugin in p:
result.add(plugin)
return list(result)
Other plugins now could query plugins on their own by importing those methods from the myapplication module. For example here a Bar plugin:
# -*- coding: utf-8 -*-
import myapplication
class BarPlugin(object):
capabilities = ['foo_capability']
def do_something(self):
plugins = myapplication.get_all_plugins()
return 'Number of plugins: %d' % len(plugins)
The main application now can gain a main method:
def main():
for plugin in get_plugins_by_capability('foo_capability'):
print '%s: %s' % (plugin, plugin.do_something())
if __name__ == '__main__':
main()
And the output would look like this:
$ python myapplication.py <foo.FooPlugin object at 0xb7ca59ec>: Hello from FooPlugin <bar.BarPlugin object at 0xb7ca5cec>: Number of plugins: 2
I personally don't use egg files beside for working on trac plugins so I'm not sure if my method of loading egg files is the best. If someone has a better way for loading egg files from a given plugin folder drop me a line :)
You can download an archive of all files here: setuptools_plugins.tar.gz