lucumr

I found two mails in my Junk folder today that where ham. I spotted them by accident as the subjects where very stupid. The first mail arrived with “Re:” as subject, the second with “Hi”. Please make my life easier and use subjects that don’t look like spam. Thanks in advance

And no, I haven’t written a mail without subject in the first place.

I’m deeply sorry for the title of that post, but I hope that gives the topic the awareness I think it should get. In the last weeks something remarkable happened in the Python3 sources: self kinda became implicit. Not in function definitions, but in super calls. But not only self: also the class passed to super. That’s remarkable because it means that the language shifts into a completely different direction.

super was rarely used in the past, mainly because it was weird to use. In the most common use case the current class and the current instance where passed to it, and the super typed returned looked up the parent methods on the MRO for you. It was useful for multiple inheritance and mixin classes that don’t know their parent but confusing for many.

The main problem with replacing super(Foo, self).bar() with something like super.bar() is that self is explicit and the class (in that case Foo) can’t be determined by the caller. Furthermore the Python principle was always against functions doing stack introspection to find the caller. There are few examples in the stdlib or builtins that do some sort of caller introspection. Those are the special functions vars(), locals(), globals(), and __import__ and some functions in the inspect module. Four functions, and all of them do nothing more than getting the current frame and accessing the dict of locals or globals. What super in current Python 3 builds does goes way beyond that.

Currently if super is called without arguments Python performs these steps:

• getting the current frame of the caller as well as the code object.
• looking at “co_argcount” to make sure there is a first argument, if there is one it gets the object from the “f_localsplus” array on the frame object. This is btw an attribute not accessible from the Python code.
• then it checks the “co_freevars” of the code object and iterates over all of them to check if one of them is “__class__” (because accessing __class__ in Python 3 creates a special bytecode that returns the class the function was defined in).
• It it can’t find the __class__ in there it dies. How does __class__ end up there? Apparently the compiler checks if “super” or “__class__” is accessed. That’s right. It breaks if you alias super to another name and try to call that name.
• Once it has that information it uses that as two first arguments. The class and the reference to self

I’m sorry, but that’s a very, very bad idea. It’s way more magical than anything we’ve had in Python in the past and just doesn’t fit into the language. We do have an explicit self in methods and we do not have methods. Our methods are functions, just that a descriptor puts a method object around it to pass the self as first arguments. That’s an incredible cool thing and makes things very simple and non-magical. Breaking that principle by coming up with an automatic super harms the whole thing a lot. Defs in classes are not completely differently from defs in the global scope or within another def.

Another odd thing is that Python 3 starts keeping information on the C Layer we can’t access from within Python which is a shame. Super is one example — it’s currently impossible to implement that from within Python. The other good example in Python 3 are methods. They don’t have a descriptor that wraps them if they are accessed via their classes. This as such is not a problem as you can call them the same (just that you can call them with completely different receivers now) but it becomes a problem if some of the functions are marked as staticmethods. Then they look completely the same when looking at them from a classes perspective:

>>> class C:
...  normal = lambda x: None
...  static = staticmethod(lambda x: None)
... 
>>> type(C.normal) is type(C.static)
True
>>> C.normal
<function <lambda> at 0×4da150>

As far as I can see a documentation tool has no chance to keep them apart even though they are completely different on an instance:

>>> type(C().normal) is type(C().static)
False
>>> C().normal
<bound method C.<lambda> of <__main__.C object at 0×4dbcf0>>
>>> C().static
<function <lambda> at 0×4da198>

While I was quite happy with the Python 3 progress so far, these two things are a major, major step into the wrong direction. I really hope that will be rolled back. If there is need for an automatic super self has to go away and __class__ become a free variable all the time or super a keyword. Everything else is too magical and more magical.

Update: I posted the subject on the python-dev mailing list.

There’s something that’s been bugging me for a long time that I need to get off my chest. Some of you may hate me for it, but perhaps there are others out there with the same complaint, silently in agony, wishing for death to take the pain away. It’s time to set the record straight, and prove once and for all that the Pythonistas are wrong.

Pythons almost NEVER look like this:

The frog shown here is what the Python Foundation refers to as a “snake” (though it looks more like a frog), more specifically a blue/yellow one. The name “Python” however refers to a group of six British Gentleman* and something like 86.43% people know that. The name was chosen because snakes just suck. Get it? It’s not a snake, they are British.

Pythons however are better represented by a 16-ton weight or a dead parrot. But they are NOT represented by snakes.

See that one in the scipy logo? That’s a public domain circle someone added a white snake to. A SNAKE. Look at the wikipedia article and search for “snake”. Yeah, no match.

Even the Pycon (where Guido van Rossum himself spoke) has made the mistake of choosing this stupid snake.

lxml is doing it wrong too.

And probably your favourite Python module too. So keep in mind: Pythons are not Snakes!. And I think that proves once and for all that there are tons of projects with the wrong logo out there.

Sorry headius for taking advantage of your blog post but I wanted to blog about that for quite some time anyways ;-)

Update: fixed my mistake about all Pythons being British. Thanks Joe Pantuso.
Update 2: apparently they are all British now. *Terry Gilliam renounced his American citizenship. Thanks meow

Imagine for the moment there was a PHP blog software that has the ability to dump the blog posts into some sort of extended RSS 2 feed and import from there later and probably from a different installation. That’s nice, XML is a flexible format and RSS allows extensions via namespaces. Even better, there are XML parsers for all major programming languages and from python working with XML is especially cool because of lxml and element tree. But there is a problem with that…

…that XML, is not XML. It’s called WordPress eXtended RSS (WXR) but it’s not XML? And why in god’s name did nobody notice so far? I mean, WordPress must have an importer for that.

Why it’s not XML? It has XML syntax, XML namespace declarations but what doesn’t it have? A doctype. What’s the problem? It’s referencing HTML entities! So step one for parsing: inject an inline DTD that defines those entities. Great fun isn’t it? Then it parses. I was happy and finished my work. That XML doesn’t have HTML entities is something PHP developers probably don’t know and their parser isn’t resolving any entities during the parsing process. Or worse, their XML parser expands HTML entites.

But it’s worse! I loaded another dump that happened to have some broken HTML in comments (could happen, does happen, thanks broken trackback support). What happens next? THE XML DOESN’T PARSE ANY MORE! Why? Because comments are neither escaped nor marked as CDATA. I wonder why, especially because it’s so much easier to handle embedded HTML/XHTML for dumping as cdata and not XML, especially if you are working with PHP.

But WordPress was able to import that…. so I looked at their parser…. WORDPRESS PARSES THAT WXR FILE USING REGULAR EXPRESSIONS!!! Argharhgarhghargh. That’s not XML what you are doing there, that’s nothing. WordPress can’t even parse it’s own file if you bind the WordPress exporter namespace to a different prefix! WordPress can’t handle it’s own file if you replace their CDATA foobar against properly escaped stuff. Dammit!

I can’t even write a proper exporter using XML tools because what my XML tools generate is not compatible to WordPress. And what tops it all?

Reading that in the #wordpress channel:

<nickname_deleted> why does it matter what wp's xml format has flaws?
                   adapt your importer to the flaws

ARGHARGHARGHARGH. and then the webpage says:

WordPress is a state-of-the-art semantic personal publishing platform with a focus on aesthetics, web standards, and usability.

Without further comments… I lost my faith into standards that moment. Wait a second, I lost it earlier. Still sad.

About four years ago I stumbled about a promising game called “Söldner — Secret Wars” in a magazine. It was an online tactical shooter and featured a fully destroyable environment, more than one hundred weapons I think, many different vehicles, aircrafts and much more. It was hyped back then in the German gaming community, probably because it was developed by a German studio called “Wings”. Long before the game was released the developers updated their online dairy about recent changes in the code, about what the game will feature and a lot more. Unfortunately they decided to release their game by Jowood, an Austrian publisher.

Why was that bad? Back then Jowood once again had financial problems and they tried to solve it by convincing the sharedholders that their next game will be a major success. They set an totally unachievable release date and the developer team at wings had to live with that. It went that far that they had to design the box are themselves even though the game was still buggy. The game was released and was still buggy. I think GameStar even omitted the rating because they wanted to wait for the first patch.

However the game sold pretty well the first two weeks despite the bad quality of the release version and the bad reviews. It even outsold the greatest competitor at that time (Joint Operations). I was one of the early adopters and I loved to play it. Owning a bad ISDN connection at that time it was terrible lagging, especially because their netcode was a catastrophe. Still, it was great fun and the destructible environment was great. Not only that, the game was somewhat revolutionary. It was the first online game I played with an (albeit bad configured) physics engine, it used python as scripting language, it had one big map that was generated from satellite images with detailed areas where the fights took place. You could buy yourself a jet and leave the map and visit all off Siberia. Not like Battlefield Vietnam where you hit the borders of the map with a jet after roughly 15 seconds. Dammit, it would still be fun and I bet it would have been bug-less by now.

The community that appeared around that game was incredible. A wiki appeared, people tried to hack around on the code to get extra features into the game, unofficial mods appeared. There was even a project that wanted to create a mod for a medieval setting. Hell, it was great back then. But instead of fixing the bugs Jowood forced the developers to start an addon project…

Shortly after the addon was finished the company developing that game was liquidated. The developers lost their job and the community continued developing the closed code. The group around “project zero” continued maintaining the game until that group broke apart and the gEasy team took over the work. The gEasy team was basically just the project zero team without the founder. They did an tremendous job but Jowood once again destroyed everything. They refused to pay their bills, then the gEasy team took down the master server and Ivan Ertlov appeared on the scene. He managed to resolve the problems between Jowood and gEasy team to some extend that the master server was up and running again.

Who’s that Ivan Ertlov? Very good question indeed and I personally don’t know the full story but from what I’ve read on the forums he is called Johann Ertl and owns a company that sells guerrilla marketing and similar services. And he worked for Jowood and apparently still does, at least he is listed as community manager in the official forums. He also brought up the topic “Open Sourcing” the code after it was clear that there will be no Söldner 2. A sequel was actually under consideration and gEasy started working on that till the day they took down the master server because there was no payment by Jowood. But since some time there was no feedback any more and an open source version is somewhat unlikely.

Frankly I don’t know if Jowood payed or didn’t. What Jowood did was destroying the game by forcing an early release, forcing an addon when the code was still unfinished, fired the developers. Even worse: they are telling their shareholders that everything is working perfectly and that they are releasing dozens of new games. Hell, they released Spellforce 1 and 2 which both are awesome games (just happen to have the worst copy protection ever designed), the Gothic games (the 3rd part was buggy like Söldner but probably also because Jowood forced an early release). Jowood however claims that they payed and that gEasy was lying. Who is running the portal now? Apparently Ivan’s company.

So why am I blogging about all that? Mainly because I think that topic hasn’t gained a lot of attraction. I stopped playing that game after Wings was liquidated. For one because I switched to ubuntu and on the other hand because my Söldner plugin (an in-game Winamp controller) disappeared when the forum was updated to reflect the new ownership. All the old topics where either deleted or made unreadable when Wings was closed down.

What nobody really notices is that the game had a tremendous German community. Hell, some of them even took over the development! If Jowood would have noticed that earlier they could have made that game the freaking best online shooter available at that time. But because of their small horizon they just thought about their next quarter and decided to do what shareholders want, not what players want. But not only Jowood is to blame but the root of all evil in that case.

Krawall supported Jowood in the beginning to host their initial infrastructure needed for the game. Unfortunately the server software needed Direct X to run properly which caused a lot of trouble. One the one hand it was hard to get multiple servers running on one machine do to the way the server was designed, on the other hand you needed a windows server to host games. And because the initial code base was that buggy all the magazines flamed to game. Despite the good sales figures it magazines never wrote about the game and forced an early dead which harmed the community. A release six month later, a linux server from the beginning, no addon and good press coverage would have avoided all the problems this game was facing in the past. And it would have saved Wings, the game they were working on beside Söldner, and in the end of course Jowood which would have had a lot less bad press.

Especially the gamers hate Jowood for their buggy products now and I doubt that they will be able to continue to ignore all the user feedback and push alpha versions as release versions for much longer.

I’m especially interested what the Wings guys are doing now. I know that former Wings community manager and sound designed Marc Olbertz is working at Blizzard but that’s about it. It’s sad what happened there and that Austrian’s only game publisher caused all that.

Funky fact: The communication with the master server worked via jabber, the physics engine was ODE and the scripting language was Python. All open source technologies I never saw in a commercial game before :-)

Disclaimer: the information on this page should be accurate but it’s hard to say for sure because there is few information about this topic actually available. If you are able to understand German you can find some information in this thread on the Söldner forums: Söldner ein Abenteuer mit ungewissem Ausgang

And unfortunately also the only day for me because I can’t be there tomorrow. Which is especially stupid because I’m very interested in the talk about broadband 2.0 by Alexander List.

Well. It was my first barcamp so I presented a topic: mercurial. Good: There was at least one person in the audience that considers switching and I hope I whetted appetite to the others too, Bad: the talk was badly organized IMO, hope I will do better the next time. But now to the positive aspects of the day.

I think the main reason why people go to barcamps are not the talks, but the possibility to meet people with similar interests and share experiences and knowledge. I meet a couple people there I didn’t know before or only from IRC/Jabber/whatever and he have a couple of great discussions, especially about the currently problematic political situation here in Austria. And despite what I noticed in the past there are several groups operating here that try to fix the situation here.

I’ll hopefully blog a bit about it more soon :-)

I will not blog about it, because it just sucks.

For a long time I thought I’m the only person disliking the stdlib, but apparently there are more :-) Not just that every library has it’s own code style, it also has different names, some are implemented insanely bad (Cookie.py anyone?) or Javaish (threading, unittest) or just plain stupid (codeop). I know we can’t just get rid of that stdlib, wait a moment, why can’t we? Python 3 is upcoming and we all have to adapt our libraries and applications anyways.

It wouldn’t be too hard actually. Python loads modules from zip files anyways. Just move all the stuff into a zip file, move it into “old” or somethign and old applications just have to alter their imports “from cgi import escape” to “from old.cgi import escape”. Or the other way round, the old libraries stay where they are and the new stdlib goes into “py.*” or “std.*”.

Insane or a good idea? You be the judge.

Again we’re coming close to December the 24th, a day many people will spend with their families or friends to celebrate. I want to use this Christmas to say thank you to a couple of persons that helped me a lot the last years, either directly or indirectly due to their tremendous work in the Open Source community.

First of all I want to thank you Georg Brandl, who you thought me a lot about Python’s internals. You had an answer to pretty all of my questions and you were the first person I worked with on a real Open Source project. Even though we still don’t have Pocoo released it was great fun to work with you.

Thank you Benjamin Wiegand, Christoph Hack and Christopher Grebs for all your code contributions and your work on Pocoo and the Pocoo libraries.

A big thank you to the whole ubuntuusers team, also those of you who left the team in the past. We had many problems in the past but I looking back those three years I think most of the time we worked together in the past were plesant :-)

Thank you Marek Kubica for all those nice discussions we’ve had. And thanks for your work on the German Python community and your patches to the Pocoo libraries. I guess the python forum and wiki wouldn’t be the same without you.

Of course a big thank you to Alexander Schremmer who keeps the pocoo server running. Learned a lot from you :-)

Also a big thank you to Fritz Cizmarov who introduced me to ubuntu. Unfortunately he passed away two years ago but I’ll never forget him.

I also want to thank Mike Bayers for his work on SQLAlchemy which I use on a daily basis, all the guys over at edgewall for Genshi and Trac, the mercurial people for their ass-kicking DVCS, Martijn Faassen for lxml and all the people working on ubuntu.

Keep up the good work and Merry Christmas to you all!