We Keep Coding

How to prevent a RuntimeError when unpickling a QObject?

Trying to unpickle a pickled QObject (using Python 2.7, PyQt4 or 5, pickle or cPickle), raises the following RuntimeError:
RuntimeError: super-class __init__() of type QObject was never called
A minimal example:
cPickle.loads(cPickle.dumps(QtCore.QObject(), cPickle.HIGHEST_PROTOCOL))
I am aware that unpickling an object, by design, does not call the object's __init__() method.
How, then, can I make sure the superclass __init__() is called, in this case?
A seemingly similar question was asked here, but was not answered.

One possible solution, based on this answer, can be implemented as follows (a slightly more elaborate example, with a custom attribute):
import cPickle
from PyQt5 import QtCore
class MyQObject(QtCore.QObject):
def __init__(self, parent=None):
super(MyQObject, self).__init__(parent)
# Add some custom attribute
self.some_attribute = 'something'
def __setstate__(self, state):
# Restore attributes
self.__dict__.update(state)
# Call the superclass __init__()
super(MyQObject, self).__init__()
original = MyQObject()
pickle_string = cPickle.dumps(original, cPickle.HIGHEST_PROTOCOL)
restored = cPickle.loads(pickle_string)
The parent can then be set using setParent(), if necessary.

How to inherit from a class instantiated with a builder?

I have a class Document, this class is really complex to instantiate so I have a builder object to create them. Both elements are not mine, so I can't change them
Now, I want to create a subclass of Document, just to add some specific methods. In order to keep using the provided builder I tried this:
class SpecialDocument(Document):
def __new__(cls, *args):
return DocumentBuilder(*args)
def __init__(self, *args, **kwargs):
#My initialization
The problem here is that the __init__ method never executes cause the __new__ method doesn't return a SpecialDocument (It returns a Document)
In my particular case I don't need to build my SpecialDocument differently from how I build a Document. Is there a way to use the same builder? If not, how can I achieve this? I just want to inherit from Document to add particular functionalities, maybe it could be achieved with metaclasses but I never used them (Probably cause I don't fully understand it), a little insight on them would be nice if it can help solving my problem

You don't actually need a metaclass here - you just have to proper call the superclass' __new__ method. The way you are doing it, the instantiation of the superclass does not "know" it is being called from a subclass at all.
So, just write your code like this instead:
class SpecialDocument(Document):
def __new__(cls, *args):
return super().__new__(cls, *args)
def __init__(self, *args, **kwargs):
#My initialization
Now, that is the ordinary way to do it - and would work if the code in your "builder" function was correctly placed inside Docment's __new__ or __init__.
Since the code there does nt do that, and you can[ t pass your subclass as a parameter to the builder, a working solution might be to create a normal document, and swap its class after it has been built:
def special_document_init(special_document):
...
class SpecialDocument(Document):
def my_special_method(self, ...):
...
def overriden_method(self):
...
result = super().overriden_method()
...
def build_special_document(*args):
document = DocumentBuilder(*args)
document.__class__ = SpecialDocument
special_document_init(document)
return document

Python: How to handle method calls on a class that could not initialize correctly?

If an object relies on a module that is not included with Python (like win32api, gstreamer, gui toolkits, etc.), and a class/function/method from that module may fail, what should the object do?
Here's an example:
import guimodule # Just an example; could be anything
class RandomWindow(object):
def __init__(self):
try:
self.dialog = guimodule.Dialog() # I might fail
except: guimodule.DialogError:
self.dialog = None # This can't be right
def update(self):
self.dialog.prepare()
self.dialog.paint()
self.dialog.update()
# ~30 more methods
This class would only be a tiny (and unnecessary, but useful) part of a bigger program.
Let's assume we have an imaginary module called guimodule, with a class called Dialog, that may fail to instantiate. If our RandomWindow class has say, 30 methods that manipulate this window, checking if self.dialog is not None will be a pain, and will slow down the program when implemented in constantly used methods (like the update method in the example above). Calling .paint() on a NoneType (when the Dialog fails to load) will raise an error, and making a dummy Dialog class with all of the original's methods and attributes would be absurd.
How can I modify my class to handle a failed creation of the Dialog class?

Rather than creating an invalid object, you should have allowed the exception raised in __init__ to propogate out so the error could be handled in an appropriate manner. Or you could have raised a different exception.
See also Python: is it bad form to raise exceptions within __init__?

You may find it useful to have two subclasses of it; one that uses that module and one that does not. A "factory" method could determine which subclass was appropriate, and return an instance of that subclass.
By subclassing, you allow them to share code that is independent of whether that module is available.

I agree that "checking if self.dialog is not None will be pain" but I don't agree that it will slow down things because if self.dialog existed it will be more slower. So forget about slowness for time being. so one way to handle is to create a MockDialog which does nothing on function calls e.g.
class RandomWindow(object):
def __init__(self):
try:
self.dialog = guimodule.Dialog() # I might fail
except: guimodule.DialogError:
self.dialog = DummyDialog() # create a placeholder
class DummyDialog(object):
# either list all methods or override __getattr__ to create a mock object

Making a dummy Dialog class is not as absurd as you might thing if you consider using Pythons __getattr__ feature. This following dummy-implementation would completely fit your needs:
class DummyDialog:
def __getattr__(self, name):
def fct(*args, **kwargs):
pass
return fct

How can I ensure that one of my class's methods is always called even if a subclass overrides it?

For example, I have a
class BaseHandler(object):
def prepare(self):
self.prepped = 1
I do not want everyone that subclasses BaseHandler and also wants to implement prepare to have to remember to call
super(SubBaseHandler, self).prepare()
Is there a way to ensure the superclass method is run even if the subclass also implements prepare?

I have solved this problem using a metaclass.
Using a metaclass allows the implementer of the BaseHandler to be sure that all subclasses will call the superclasses prepare() with no adjustment to any existing code.
The metaclass looks for an implementation of prepare on both classes and then overwrites the subclass prepare with one that calls superclass.prepare followed by subclass.prepare.
class MetaHandler(type):
def __new__(cls, name, bases, attrs):
instance = type.__new__(cls, name, bases, attrs)
super_instance = super(instance, instance)
if hasattr(super_instance, 'prepare') and hasattr(instance, 'prepare'):
super_prepare = getattr(super_instance, 'prepare')
sub_prepare = getattr(instance, 'prepare')
def new_prepare(self):
super_prepare(self)
sub_prepare(self)
setattr(instance, 'prepare', new_prepare)
return instance
class BaseHandler(object):
__metaclass__ = MetaHandler
def prepare(self):
print 'BaseHandler.prepare'
class SubHandler(BaseHandler):
def prepare(self):
print 'SubHandler.prepare'
Using it looks like this:
>>> sh = SubHandler()
>>> sh.prepare()
BaseHandler.prepare
SubHandler.prepare

Tell your developers to define prepare_hook instead of prepare, but
tell the users to call prepare:
class BaseHandler(object):
def prepare(self):
self.prepped = 1
self.prepare_hook()
def prepare_hook(self):
pass
class SubBaseHandler(BaseHandler):
def prepare_hook(self):
pass
foo = SubBaseHandler()
foo.prepare()
If you want more complex chaining of prepare calls from multiple subclasses, then your developers should really use super as that's what it was intended for.

Just accept that you have to tell people subclassing your class to call the base method when overriding it. Every other solution either requires you to explain them to do something else, or involves some un-pythonic hacks which could be circumvented too.
Python’s object inheritance model was designed to be open, and any try to go another way will just overcomplicate the problem which does not really exist anyway. Just tell everybody using your stuff to either follow your “rules”, or the program will mess up.

One explicit solution without too much magic going on would be to maintain a list of prepare call-backs:
class BaseHandler(object):
def __init__(self):
self.prepare_callbacks = []
def register_prepare_callback(self, callback):
self.prepare_callbacks.append(callback)
def prepare(self):
# Do BaseHandler preparation
for callback in self.prepare_callbacks:
callback()
class MyHandler(BaseHandler):
def __init__(self):
BaseHandler.__init__(self)
self.register_prepare_callback(self._prepare)
def _prepare(self):
# whatever

In general you can try using __getattribute__ to achive something like this (until the moment someone overwrites this method too), but it is against the Python ideas. There is a reason to be able to access private object members in Python. The reason is mentioned in import this

How to auto register a class when it's defined

I want to have an instance of class registered when the class is defined. Ideally the code below would do the trick.
registry = {}
def register( cls ):
registry[cls.__name__] = cls() #problem here
return cls
#register
class MyClass( Base ):
def __init__(self):
super( MyClass, self ).__init__()
Unfortunately, this code generates the error NameError: global name 'MyClass' is not defined.
What's going on is at the #problem here line I'm trying to instantiate a MyClass but the decorator hasn't returned yet so it doesn't exist.
Is the someway around this using metaclasses or something?

Yes, meta classes can do this. A meta class' __new__ method returns the class, so just register that class before returning it.
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(MetaClass, cls).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class MyClass(object):
__metaclass__ = MetaClass
The previous example works in Python 2.x. In Python 3.x, the definition of MyClass is slightly different (while MetaClass is not shown because it is unchanged - except that super(MetaClass, cls) can become super() if you want):
#Python 3.x
class MyClass(metaclass=MetaClass):
pass
As of Python 3.6 there is also a new __init_subclass__ method (see PEP 487) that can be used instead of a meta class (thanks to #matusko for his answer below):
class ParentClass:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
register(cls)
class MyClass(ParentClass):
pass
[edit: fixed missing cls argument to super().__new__()]
[edit: added Python 3.x example]
[edit: corrected order of args to super(), and improved description of 3.x differences]
[edit: add Python 3.6 __init_subclass__ example]

Since python 3.6 you don't need metaclasses to solve this
In python 3.6 simpler customization of class creation was introduced (PEP 487).
An __init_subclass__ hook that initializes all subclasses of a given class.
Proposal includes following example of subclass registration
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
In this example, PluginBase.subclasses will contain a plain list of
all subclasses in the entire inheritance tree. One should note that
this also works nicely as a mixin class.

The problem isn't actually caused by the line you've indicated, but by the super call in the __init__ method. The problem remains if you use a metaclass as suggested by dappawit; the reason the example from that answer works is simply that dappawit has simplified your example by omitting the Base class and therefore the super call. In the following example, neither ClassWithMeta nor DecoratedClass work:
registry = {}
def register(cls):
registry[cls.__name__] = cls()
return cls
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(cls, MetaClass).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class Base(object):
pass
class ClassWithMeta(Base):
__metaclass__ = MetaClass
def __init__(self):
super(ClassWithMeta, self).__init__()
#register
class DecoratedClass(Base):
def __init__(self):
super(DecoratedClass, self).__init__()
The problem is the same in both cases; the register function is called (either by the metaclass or directly as a decorator) after the class object is created, but before it has been bound to a name. This is where super gets gnarly (in Python 2.x), because it requires you to refer to the class in the super call, which you can only reasonably do by using the global name and trusting that it will have been bound to that name by the time the super call is invoked. In this case, that trust is misplaced.
I think a metaclass is the wrong solution here. Metaclasses are for making a family of classes that have some custom behaviour in common, exactly as classes are for making a family of instances that have some custom behavior in common. All you're doing is calling a function on a class. You wouldn't define a class to call a function on a string, neither should you define a metaclass to call a function on a class.
So, the problem is a fundamental incompatibility between: (1) using hooks in the class creation process to create instances of the class, and (2) using super.
One way to resolve this is to not use super. super solves a hard problem, but it introduces others (this is one of them). If you're using a complex multiple inheritance scheme, super's problems are better than the problems of not using super, and if you're inheriting from third-party classes that use super then you have to use super. If neither of those conditions are true, then just replacing your super calls with direct base class calls may actually be a reasonable solution.
Another way is to not hook register into class creation. Adding register(MyClass) after each of your class definitions is pretty equivalent to adding #register before them or __metaclass__ = Registered (or whatever you call the metaclass) into them. A line down the bottom is much less self-documenting than a nice declaration up the top of the class though, so this doesn't feel great, but again it may actually be a reasonable solution.
Finally, you can turn to hacks that are unpleasant, but will probably work. The problem is that a name is being looked up in a module's global scope just before it's been bound there. So you could cheat, as follows:
def register(cls):
name = cls.__name__
force_bound = False
if '__init__' in cls.__dict__:
cls.__init__.func_globals[name] = cls
force_bound = True
try:
registry[name] = cls()
finally:
if force_bound:
del cls.__init__.func_globals[name]
return cls
Here's how this works:
We first check to see whether __init__ is in cls.__dict__ (as opposed to whether it has an __init__ attribute, which will always be true). If it's inherited an __init__ method from another class we're probably fine (because the superclass will already be bound to its name in the usual way), and the magic we're about to do doesn't work on object.__init__ so we want to avoid trying that if the class is using a default __init__.
We lookup the __init__ method and grab it's func_globals dictionary, which is where global lookups (such as to find the class referred to in a super call) will go. This is normally the global dictionary of the module where the __init__ method was originally defined. Such a dictionary is about to have the cls.__name__ inserted into it as soon as register returns, so we just insert it ourselves early.
We finally create an instance and insert it into the registry. This is in a try/finally block to make sure we remove the binding we created whether or not creating an instance throws an exception; this is very unlikely to be necessary (since 99.999% of the time the name is about to be rebound anyway), but it's best to keep weird magic like this as insulated as possible to minimise the chance that someday some other weird magic interacts badly with it.
This version of register will work whether it's invoked as a decorator or by the metaclass (which I still think is not a good use of a metaclass). There are some obscure cases where it will fail though:
I can imagine a weird class that doesn't have an __init__ method but inherits one that calls self.someMethod, and someMethod is overridden in the class being defined and makes a super call. Probably unlikely.
The __init__ method might have been defined in another module originally and then used in the class by doing __init__ = externally_defined_function in the class block. The func_globals attribute of the other module though, which means our temporary binding would clobber any definition of this class' name in that module (oops). Again, unlikely.
Probably other weird cases I haven't thought of.
You could try to add more hacks to make it a little more robust in these situations, but the nature of Python is both that these kind of hacks are possible and that it's impossible to make them absolutely bullet proof.

The answers here didn't work for me in python3, because __metaclass__ didn't work.
Here's my code registering all subclasses of a class at their definition time:
registered_models = set()
class RegisteredModel(type):
def __new__(cls, clsname, superclasses, attributedict):
newclass = type.__new__(cls, clsname, superclasses, attributedict)
# condition to prevent base class registration
if superclasses:
registered_models.add(newclass)
return newclass
class CustomDBModel(metaclass=RegisteredModel):
pass
class BlogpostModel(CustomDBModel):
pass
class CommentModel(CustomDBModel):
pass
# prints out {<class '__main__.BlogpostModel'>, <class '__main__.CommentModel'>}
print(registered_models)

Calling the Base class directly should work (instead of using super()):
def __init__(self):
Base.__init__(self)

It can be also done with something like this (without a registry function)
_registry = {}
class MetaClass(type):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class MyClass1(metaclass=MetaClass): pass
class MyClass2(metaclass=MetaClass): pass
print(_registry)
# {'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}
Additionally, if we need to use a base abstract class (e.g. Base() class), we can do it this way (notice the metacalss inherits from ABCMeta instead of type)
from abc import ABCMeta
_registry = {}
class MetaClass(ABCMeta):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class Base(metaclass=MetaClass): pass
class MyClass1(Base): pass
class MyClass2(Base): pass
print(_registry)
# {'Base': <class '__main__.Base'>, 'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}