How super() in Python3 works and why it’s retarded
I'm deeply sorry for the title of that post, but I hope that gives the topic the awareness I think it should get. In the last weeks something remarkable happened in the Python3 sources: self kinda became implicit. Not in function definitions, but in super calls. But not only self: also the class passed to super. That's remarkable because it means that the language shifts into a completely different direction.
super was rarely used in the past, mainly because it was weird to use. In the most common use case the current class and the current instance where passed to it, and the super typed returned looked up the parent methods on the MRO for you. It was useful for multiple inheritance and mixin classes that don't know their parent but confusing for many.
The main problem with replacing super(Foo, self).bar() with something like super.bar() is that self is explicit and the class (in that case Foo) can't be determined by the caller. Furthermore the Python principle was always against functions doing stack introspection to find the caller. There are few examples in the stdlib or builtins that do some sort of caller introspection. Those are the special functions vars(), locals(), globals(), and __import__ and some functions in the inspect module. Four functions, and all of them do nothing more than getting the current frame and accessing the dict of locals or globals. What super in current Python 3 builds does goes way beyond that.
Currently if super is called without arguments Python performs these steps:
- getting the current frame of the caller as well as the code object.
- looking at "co_argcount" to make sure there is a first argument, if there is one it gets the object from the "f_localsplus" array on the frame object. This is btw an attribute not accessible from the Python code.
- then it checks the "co_freevars" of the code object and iterates over all of them to check if one of them is "__class__" (because accessing __class__ in Python 3 creates a special bytecode that returns the class the function was defined in).
- It it can't find the __class__ in there it dies. How does __class__ end up there? Apparently the compiler checks if "super" or "__class__" is accessed. That's right. It breaks if you alias super to another name and try to call that name.
- Once it has that information it uses that as two first arguments. The class and the reference to self
I'm sorry, but that's a very, very bad idea. It's way more magical than anything we've had in Python in the past and just doesn't fit into the language. We do have an explicit self in methods and we do not have methods. Our methods are functions, just that a descriptor puts a method object around it to pass the self as first arguments. That's an incredible cool thing and makes things very simple and non-magical. Breaking that principle by coming up with an automatic super harms the whole thing a lot. Defs in classes are not completely differently from defs in the global scope or within another def.
Another odd thing is that Python 3 starts keeping information on the C Layer we can't access from within Python which is a shame. Super is one example -- it's currently impossible to implement that from within Python. The other good example in Python 3 are methods. They don't have a descriptor that wraps them if they are accessed via their classes. This as such is not a problem as you can call them the same (just that you can call them with completely different receivers now) but it becomes a problem if some of the functions are marked as staticmethods. Then they look completely the same when looking at them from a classes perspective:
>>> class C: ... normal = lambda x: None ... static = staticmethod(lambda x: None) ... >>> type(C.normal) is type(C.static) True >>> C.normal <function <lambda=""> at 0x4da150>; </function>
As far as I can see a documentation tool has no chance to keep them apart even though they are completely different on an instance:
>>> type(C().normal) is type(C().static) False >>> C().normal <bound c.<lambda="" method=""> of <__main__.C object at 0x4dbcf0>> >>> C().static <function <lambda=""> at 0x4da198> </function></bound>
While I was quite happy with the Python 3 progress so far, these two things are a major, major step into the wrong direction. I really hope that will be rolled back. If there is need for an automatic super self has to go away and __class__ become a free variable all the time or super a keyword. Everything else is too magical and more magical.
Update: I posted the subject on the python-dev mailing list.
I'd have thought python-dev was the lace for this particular "cogitation", unless the purpose is to gain publicity rather than advance the Python language. It is an open list, after all.
— Steve Holden on Wednesday, April 30, 2008 11:05 #
Or possibly python-3000 would be better, but that's open too.
— Steve Holden on Wednesday, April 30, 2008 11:05 #
I do agree that the python-dev list would be a better place for that to discuss and I'm sure I will bring it up there once I played with it a bit more. I was just confused when I discovered that and hope it brings awareness to this quite radical change.
(And I'm still hoping that this is a feature that is under consideration and not final)
— Armin Ronacher on Wednesday, April 30, 2008 11:11 #
I'm inclined to agree with you, based on your description: "Explicit is better than implicit"!
However, I really think you need to submit these comments to the python-3000 mailing list (python-3000@python.org mail.python.org/mailman/listinfo/python-3000) to get their reactions to it. Don't worry about clogging it up. Your post here is well written and on the level of a lot of the other things they discuss there. (Ie. Don't worry about being the guy that says, "Der, why you guys always use whitespace? Try { }s." Your concerns are not the concerns of a troll/newb.)
— Carl on Wednesday, April 30, 2008 11:55 #
Implementing super has proven to be a bit of a tricky problem for Python. I had a brief look at it myself about a year ago, and it did my head in. You're right, though, about there being a lot of magic involved here. You might also be right about it being a little "too magic". I'd have to third the idea of pushing this to python-3000 and/or python-dev for further discussion if you're really keen to make a difference here. :)
— Thomas Lee on Wednesday, April 30, 2008 13:01 #
Keep in mind that super retains the exact same semantics as before when calling with arguments. Not passing args to super is a convenience feature that will do The Right Thing most of the time. To quote the docs: "You can now invoke super() without arguments and the right class and instance will automatically be chosen. With arguments, its behavior is unchanged."
— fneh on Wednesday, April 30, 2008 14:18 #
Not In My Python!
— Ars on Thursday, May 1, 2008 1:51 #
I was also pretty turned off by the PEP, but never piped up, mostly because the current Py3k doesn't actually implement it like this. Instaed, super now works with zero arguments, which I find fine; it remains just a builtin.
My impression was that the PEP, even though 'accepted', is not and won't actually become part of Py3k. I think that's for the better, too.
— Adam Gomaa on Thursday, May 1, 2008 3:23 #
Modify my last comment: the noargs super() is what you don't like; I thought it was the attribute-access super that you didn't like.
I agree the noargs super() implementation is ugly, but it also works exactly as it should. The attribute-access super, on the other hand, is something I would have found hard to justify. (super().foo==OK, super.foo==not OK)
— Adam Gomaa on Thursday, May 1, 2008 3:33 #
@Adam: there's some difference between "works as it should" and "fits into the language". The current solution may look nice to language users who only do basic usage, but to those who know the details it cries out as a kludge.
There are three things that won't work with this argumentless super(), and they are not justifiable from a language designing point of view:
— Georg on Thursday, May 1, 2008 19:32 #
I was really surprised that Guido approved this awkward hack, considering his past insistance that Python remain as transparent as possible.
Besides the limitations mentioned by Georg, there's also the problem of calling super() from inner functions, because super() expects the first argument to be self. I certainly hope the devs document all the idiosyncrasies, because it will certainly bite people not expecting them.
I had posted an alternative based on inserting an extra argument (which could be implemented in only 26 lines of pure Python) in the python-ideas mailing list, but it was rejected outright. You can find it here: thread.gmane.org/gmane.comp.python.ideas/1498
Of course, I'll continue to use Python in the future (like a proper addict), but you won't find me using the new super().
— Anthony Tolle on Wednesday, May 14, 2008 17:57 #