Pickle a dynamically parameterized sub-class - python

I have a system which commonly stores pickled class types.
I want to be able to save dynamically-parameterized classes in the same way, but I can't because I get a PicklingError on trying to pickle a class which is not globally found (not defined in simple code).
My problem can be modeled as the following example code:
class Base(object):
def m(self):
return self.__class__.PARAM
def make_parameterized(param_value):
class AutoSubClass(Base):
PARAM = param_value
return AutoSubClass
cls = make_parameterized(input("param value?"))
When I try to pickle the class, I get the following error:
# pickle.PicklingError: Can't pickle <class '__main__.AutoSubClass'>: it's not found as __main__.AutoSubClass
import pickle
print pickle.dumps(cls)
I am looking for some method to declare Base as a ParameterizableBaseClass which should define the params needed (PARAM in above example). A dynamic parameterized subclass (cls above) should then be picklable by saving the "ParameterizableBaseClass" type and the different param-values (dynamic param_value above).
I am sure that in many cases, this can be avoided altogether... And I can avoid this in my code as well if I really (really) have to. I was playing with __metaclass__, copyreg and even __builtin__.issubclass at some point (don't ask), but was unable to crack this one.
I feel like I wouldn't be true to the python spirit if I wasn't to ask: how can this be achieved, in a relatively clean way?

I know this is a very old question, but I think it is worth sharing a better means of pickling the parameterised classes than the one that is the currently accepted solution (making the parameterised class a global).
Using the __reduce__ method, we can provide a callable which will return an uninitialised instance of our desired class.
class Base(object):
def m(self):
return self.__class__.PARAM
def __reduce__(self):
return (_InitializeParameterized(), (self.PARAM, ), self.__dict__)
def make_parameterized(param_value):
class AutoSub(Base):
PARAM = param_value
return AutoSub
class _InitializeParameterized(object):
"""
When called with the param value as the only argument, returns an
un-initialized instance of the parameterized class. Subsequent __setstate__
will be called by pickle.
"""
def __call__(self, param_value):
# make a simple object which has no complex __init__ (this one will do)
obj = _InitializeParameterized()
obj.__class__ = make_parameterized(param_value)
return obj
if __name__ == "__main__":
from pickle import dumps, loads
a = make_parameterized("a")()
b = make_parameterized("b")()
print a.PARAM, b.PARAM, type(a) is type(b)
a_p = dumps(a)
b_p = dumps(b)
del a, b
a = loads(a_p)
b = loads(b_p)
print a.PARAM, b.PARAM, type(a) is type(b)
It is worth reading the __reduce__ docs a couple of times to see exactly what is going on here.
Hope somebody finds this useful.

Yes, it is possible -
Whenever you want to custom the Pickle and Unpickle behaviors for your objects, you just have to set the "__getstate__" and "__setstate__" methods on the class itself.
In this case it is a bit trickier:
There need, as you observed - to exist a class on the global namespace that is the class of the currently being pickled object: it has to be the same class, with the same name. Ok - the deal is that gthis class existing in the globalname space can be created at Pickle time.
At Unpickle time the class, with the same name, have to exist - but it does not have to be the same object - just behave like it does - and as __setstate__ is called in the Unpickling proccess, it can recreate the parameterized class of the orignal object, and set its own class to be that one, by setting the __class__ attribute of the object.
Setting the __class__ attribute of an object may seen objectionable but it is how OO works in Python and it is officially documented, it even works accross implementations. (I tested this snippet in both Python 2.6 and Pypy)
class Base(object):
def m(self):
return self.__class__.PARAM
def __getstate__(self):
global AutoSub
AutoSub = self.__class__
return (self.__dict__,self.__class__.PARAM)
def __setstate__(self, state):
self.__class__ = make_parameterized(state[1])
self.__dict__.update(state[0])
def make_parameterized(param_value):
class AutoSub(Base):
PARAM = param_value
return AutoSub
class AutoSub(Base):
pass
if __name__ == "__main__":
from pickle import dumps, loads
a = make_parameterized("a")()
b = make_parameterized("b")()
print a.PARAM, b.PARAM, type(a) is type(b)
a_p = dumps(a)
b_p = dumps(b)
del a, b
a = loads(a_p)
b = loads(b_p)
print a.PARAM, b.PARAM, type(a) is type(b)

I guess it's too late now, but pickle is a module I'd rather avoid for anything complex, because it has problems like this one and many more.
Anyways, since pickle wants the class in a global it can have it:
import cPickle
class Base(object):
def m(self):
return self.__class__.PARAM
#classmethod
def make_parameterized(cls,param):
clsname = "AutoSubClass.%s" % param
# create a class, assign it as a global under the same name
typ = globals()[clsname] = type(clsname, (cls,), dict(PARAM=param))
return typ
cls = Base.make_parameterized('asd')
import pickle
s = pickle.dumps(cls)
cls = pickle.loads(s)
print cls, cls.PARAM
# <class '__main__.AutoSubClass.asd'> asd
But yeah, you're probably overcomplicating things.

Classes that are not created in the top level of a module cannot be pickled, as shown in the Python documentation.
Furthermore, even for an instance of a top level module class the class attributes are not stored. So in your example PARAM wouldn't be stored anyway. (Explained in the Python documentation section linked above as well)

Related

How to Inherit multiple classes in python dynamically [duplicate]

This article has a snippet showing usage of __bases__ to dynamically change the inheritance hierarchy of some Python code, by adding a class to an existing classes collection of classes from which it inherits. Ok, that's hard to read, code is probably clearer:
class Friendly:
def hello(self):
print 'Hello'
class Person: pass
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
That is, Person doesn't inherit from Friendly at the source level, but rather this inheritance relation is added dynamically at runtime by modification of the __bases__attribute of the Person class. However, if you change Friendly and Person to be new style classes (by inheriting from object), you get the following error:
TypeError: __bases__ assignment: 'Friendly' deallocator differs from 'object'
A bit of Googling on this seems to indicate some incompatibilities between new-style and old style classes in regards to changing the inheritance hierarchy at runtime. Specifically: "New-style class objects don't support assignment to their bases attribute".
My question, is it possible to make the above Friendly/Person example work using new-style classes in Python 2.7+, possibly by use of the __mro__ attribute?
Disclaimer: I fully realise that this is obscure code. I fully realize that in real production code tricks like this tend to border on unreadable, this is purely a thought experiment, and for funzies to learn something about how Python deals with issues related to multiple inheritance.
Ok, again, this is not something you should normally do, this is for informational purposes only.
Where Python looks for a method on an instance object is determined by the __mro__ attribute of the class which defines that object (the M ethod R esolution O rder attribute). Thus, if we could modify the __mro__ of Person, we'd get the desired behaviour. Something like:
setattr(Person, '__mro__', (Person, Friendly, object))
The problem is that __mro__ is a readonly attribute, and thus setattr won't work. Maybe if you're a Python guru there's a way around that, but clearly I fall short of guru status as I cannot think of one.
A possible workaround is to simply redefine the class:
def modify_Person_to_be_friendly():
# so that we're modifying the global identifier 'Person'
global Person
# now just redefine the class using type(), specifying that the new
# class should inherit from Friendly and have all attributes from
# our old Person class
Person = type('Person', (Friendly,), dict(Person.__dict__))
def main():
modify_Person_to_be_friendly()
p = Person()
p.hello() # works!
What this doesn't do is modify any previously created Person instances to have the hello() method. For example (just modifying main()):
def main():
oldperson = Person()
ModifyPersonToBeFriendly()
p = Person()
p.hello()
# works! But:
oldperson.hello()
# does not
If the details of the type call aren't clear, then read e-satis' excellent answer on 'What is a metaclass in Python?'.
I've been struggling with this too, and was intrigued by your solution, but Python 3 takes it away from us:
AttributeError: attribute '__dict__' of 'type' objects is not writable
I actually have a legitimate need for a decorator that replaces the (single) superclass of the decorated class. It would require too lengthy a description to include here (I tried, but couldn't get it to a reasonably length and limited complexity -- it came up in the context of the use by many Python applications of an Python-based enterprise server where different applications needed slightly different variations of some of the code.)
The discussion on this page and others like it provided hints that the problem of assigning to __bases__ only occurs for classes with no superclass defined (i.e., whose only superclass is object). I was able to solve this problem (for both Python 2.7 and 3.2) by defining the classes whose superclass I needed to replace as being subclasses of a trivial class:
## T is used so that the other classes are not direct subclasses of object,
## since classes whose base is object don't allow assignment to their __bases__ attribute.
class T: pass
class A(T):
def __init__(self):
print('Creating instance of {}'.format(self.__class__.__name__))
## ordinary inheritance
class B(A): pass
## dynamically specified inheritance
class C(T): pass
A() # -> Creating instance of A
B() # -> Creating instance of B
C.__bases__ = (A,)
C() # -> Creating instance of C
## attempt at dynamically specified inheritance starting with a direct subclass
## of object doesn't work
class D: pass
D.__bases__ = (A,)
D()
## Result is:
## TypeError: __bases__ assignment: 'A' deallocator differs from 'object'
I can not vouch for the consequences, but that this code does what you want at py2.7.2.
class Friendly(object):
def hello(self):
print 'Hello'
class Person(object): pass
# we can't change the original classes, so we replace them
class newFriendly: pass
newFriendly.__dict__ = dict(Friendly.__dict__)
Friendly = newFriendly
class newPerson: pass
newPerson.__dict__ = dict(Person.__dict__)
Person = newPerson
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
We know that this is possible. Cool. But we'll never use it!
Right of the bat, all the caveats of messing with class hierarchy dynamically are in effect.
But if it has to be done then, apparently, there is a hack that get's around the "deallocator differs from 'object" issue when modifying the __bases__ attribute for the new style classes.
You can define a class object
class Object(object): pass
Which derives a class from the built-in metaclass type.
That's it, now your new style classes can modify the __bases__ without any problem.
In my tests this actually worked very well as all existing (before changing the inheritance) instances of it and its derived classes felt the effect of the change including their mro getting updated.
I needed a solution for this which:
Works with both Python 2 (>= 2.7) and Python 3 (>= 3.2).
Lets the class bases be changed after dynamically importing a dependency.
Lets the class bases be changed from unit test code.
Works with types that have a custom metaclass.
Still allows unittest.mock.patch to function as expected.
Here's what I came up with:
def ensure_class_bases_begin_with(namespace, class_name, base_class):
""" Ensure the named class's bases start with the base class.
:param namespace: The namespace containing the class name.
:param class_name: The name of the class to alter.
:param base_class: The type to be the first base class for the
newly created type.
:return: ``None``.
Call this function after ensuring `base_class` is
available, before using the class named by `class_name`.
"""
existing_class = namespace[class_name]
assert isinstance(existing_class, type)
bases = list(existing_class.__bases__)
if base_class is bases[0]:
# Already bound to a type with the right bases.
return
bases.insert(0, base_class)
new_class_namespace = existing_class.__dict__.copy()
# Type creation will assign the correct ‘__dict__’ attribute.
del new_class_namespace['__dict__']
metaclass = existing_class.__metaclass__
new_class = metaclass(class_name, tuple(bases), new_class_namespace)
namespace[class_name] = new_class
Used like this within the application:
# foo.py
# Type `Bar` is not available at first, so can't inherit from it yet.
class Foo(object):
__metaclass__ = type
def __init__(self):
self.frob = "spam"
def __unicode__(self): return "Foo"
# … later …
import bar
ensure_class_bases_begin_with(
namespace=globals(),
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
Use like this from within unit test code:
# test_foo.py
""" Unit test for `foo` module. """
import unittest
import mock
import foo
import bar
ensure_class_bases_begin_with(
namespace=foo.__dict__,
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
class Foo_TestCase(unittest.TestCase):
""" Test cases for `Foo` class. """
def setUp(self):
patcher_unicode = mock.patch.object(
foo.Foo, '__unicode__')
patcher_unicode.start()
self.addCleanup(patcher_unicode.stop)
self.test_instance = foo.Foo()
patcher_frob = mock.patch.object(
self.test_instance, 'frob')
patcher_frob.start()
self.addCleanup(patcher_frob.stop)
def test_instantiate(self):
""" Should create an instance of `Foo`. """
instance = foo.Foo()
The above answers are good if you need to change an existing class at runtime. However, if you are just looking to create a new class that inherits by some other class, there is a much cleaner solution. I got this idea from https://stackoverflow.com/a/21060094/3533440, but I think the example below better illustrates a legitimate use case.
def make_default(Map, default_default=None):
"""Returns a class which behaves identically to the given
Map class, except it gives a default value for unknown keys."""
class DefaultMap(Map):
def __init__(self, default=default_default, **kwargs):
self._default = default
super().__init__(**kwargs)
def __missing__(self, key):
return self._default
return DefaultMap
DefaultDict = make_default(dict, default_default='wug')
d = DefaultDict(a=1, b=2)
assert d['a'] is 1
assert d['b'] is 2
assert d['c'] is 'wug'
Correct me if I'm wrong, but this strategy seems very readable to me, and I would use it in production code. This is very similar to functors in OCaml.
This method isn't technically inheriting during runtime, since __mro__ can't be changed. But what I'm doing here is using __getattr__ to be able to access any attributes or methods from a certain class. (Read comments in order of numbers placed before the comments, it makes more sense)
class Sub:
def __init__(self, f, cls):
self.f = f
self.cls = cls
# 6) this method will pass the self parameter
# (which is the original class object we passed)
# and then it will fill in the rest of the arguments
# using *args and **kwargs
def __call__(self, *args, **kwargs):
# 7) the multiple try / except statements
# are for making sure if an attribute was
# accessed instead of a function, the __call__
# method will just return the attribute
try:
return self.f(self.cls, *args, **kwargs)
except TypeError:
try:
return self.f(*args, **kwargs)
except TypeError:
return self.f
# 1) our base class
class S:
def __init__(self, func):
self.cls = func
def __getattr__(self, item):
# 5) we are wrapping the attribute we get in the Sub class
# so we can implement the __call__ method there
# to be able to pass the parameters in the correct order
return Sub(getattr(self.cls, item), self.cls)
# 2) class we want to inherit from
class L:
def run(self, s):
print("run" + s)
# 3) we create an instance of our base class
# and then pass an instance (or just the class object)
# as a parameter to this instance
s = S(L) # 4) in this case, I'm using the class object
s.run("1")
So this sort of substitution and redirection will simulate the inheritance of the class we wanted to inherit from. And it even works with attributes or methods that don't take any parameters.

Hash a python new-style class instance?

Given a custom, new-style python class instance, what is a good way to hash it and get a unique ID-like value from it to use for various purposes? Think md5sum or sha1sum of a given class instance.
The approach I am currently using pickles the class and runs that through hexdigest, storing the resultant hash string into a class property (this property is never part of the pickle/unpickle procedures, fyi). Except now I've run into a case where a third-party module uses nested classes, and there is no really good way to pickle those without some hacks. I figure that I am missing out on some clever little Python trick somewhere to accomplish this.
Edit:
Example code because it seems to be a requirement around here to get any traction on a question. The below class can be initialized and the self._uniq_id property can be properly setup.
#!/usr/bin/env python
import hashlib
# cPickle or pickle.
try:
import cPickle as pickle
except:
import pickle
# END try
# Single class, pickles fine.
class FooBar(object):
__slots__ = ("_foo", "_bar", "_uniq_id")
def __init__(self, eth=None, ts=None, pkt=None):
self._foo = "bar"
self._bar = "bar"
self._uniq_id = hashlib.sha1(pickle.dumps(self, -1)).hexdigest()[0:16]
def __getstate__(self):
return {'foo':self._foo, 'bar':self._bar}
def __setstate__(self, state):
self._foo = state['foo']
self._bar = state['bar']
self._uniq_id = hashlib.sha1(pickle.dumps(self, -1)).hexdigest()[0:16]
def _get_foo(self): return self._foo
def _get_bar(self): return self._bar
def _get_uniq_id(self): return self._uniq_id
foo = property(_get_foo)
bar = property(_get_bar)
uniq_id = property(_get_uniq_id)
# End
This next class, however, cannot be initialized because of Bar being nested in Foo:
#!/usr/bin/env python
import hashlib
# cPickle or pickle.
try:
import cPickle as pickle
except:
import pickle
# END try
# Nested class, can't pickle for hexdigest.
class Foo(object):
__slots__ = ("_foo", "_bar", "_uniq_id")
class Bar(object):
pass
def __init__(self, eth=None, ts=None, pkt=None):
self._foo = "bar"
self._bar = self.Bar()
self._uniq_id = hashlib.sha1(pickle.dumps(self, -1)).hexdigest()[0:16]
def __getstate__(self):
return {'foo':self._foo, 'bar':self._bar}
def __setstate__(self, state):
self._foo = state['foo']
self._bar = state['bar']
self._uniq_id = hashlib.sha1(pickle.dumps(self, -1)).hexdigest()[0:16]
def _get_foo(self): return self._foo
def _get_bar(self): return self._bar
def _get_uniq_id(self): return self._uniq_id
foo = property(_get_foo)
bar = property(_get_bar)
uniq_id = property(_get_uniq_id)
# End
The error I receive is:
Traceback (most recent call last):
File "./nest_test.py", line 70, in <module>
foobar2 = Foo()
File "./nest_test.py", line 49, in __init__
self._uniq_id = hashlib.sha1(pickle.dumps(self, -1)).hexdigest()[0:16]
cPickle.PicklingError: Can't pickle <class '__main__.Bar'>: attribute lookup __main__.Bar failed
(nest_test.py) has both classes in it, hence the line number offset).
Pickling requires the __getstate__() method I found out, so I also implemented __setstate__() for completeness as well. But given the already existing warnings about security and pickle, there's got to be a better way to do this.
Based on what I have read so far, the error stems from Python not being able to resolve the nested classes. It tries to look up the attribute __main__.Bar, which doesn't exist. It really needs to be able to find __main__.Foo.Bar instead, but there is no really good way to do this. I bumped into another SO answer here that provides a "hack" to trick Python, but it came with a stern warning that such an approach is not advisable, and to either use something other than pickling or to move the nested class definition to the outside versus the inside.
However, the original question of that SO answer, I believe, was for pickling and unpickling to a file. I only need to pickle in order to use the requisite hashlib functions, which seem to operate on a bytearray (much like I am used to in .NET), and pickling (Especially cPickle) is fast and optimized versus writing my own bytearray routine.
That depends entirely on what properties the ID should have.
For instance, you can use id(foo) to get an ID which is guaranteed to be unique as long as foo is active in memory, or you could use repr(instance.__dict__) if all of the fields have sensible repr values.
What specifically do you need it for?
While you're using hexdigests of pickles at the moment, you make it sound like the id doesn't actually need to be related to the object, it just needs to be unique. Why not simply use the uuid module, specifically uuid.uuid4 to generate unique IDs and assign them to a uuid field in the object...

How to create a new instance from a class object in Python

I need to dynamically create an instance of a class in Python. Basically I am using the load_module and inspect module to import and load the class into a class object, but I can't figure out how to create an instance of this class object.
I figured out the answer to the question I had that brought me to this page. Since no one has actually suggested the answer to my question, I thought I'd post it.
class k:
pass
a = k()
k2 = a.__class__
a2 = k2()
At this point, a and a2 are both instances of the same class (class k).
Just call the "type" built in using three parameters, like this:
ClassName = type("ClassName", (Base1, Base2,...), classdictionary)
update
as stated in the comment bellow this is not the answer to this question at all. I will keep it undeleted, since there are hints some people get here trying to dynamically create classes - which is what the line above does.
To create an object of a class one has a reference too, as put in the accepted answer, one just have to call the class:
instance = ClassObject()
The mechanism for instantiation is thus:
Python does not use the new keyword some languages use - instead it's data model explains the mechanism used to create an instantance of a class when it is called with the same syntax as any other callable:
Its class' __call__ method is invoked (in the case of a class, its class is the "metaclass" - which is usually the built-in type). The normal behavior of this call is to invoke the (pseudo) static __new__ method on the class being instantiated, followed by its __init__. The __new__ method is responsible for allocating memory and such, and normally is done by the __new__ of object which is the class hierarchy root.
So calling ClassObject() invokes ClassObject.__class__.call() (which normally will be type.__call__) this __call__ method will receive ClassObject itself as the first parameter - a Pure Python implementation would be like this: (the cPython version is of course, done in C, and with lots of extra code for cornercases and optimizations)
class type:
...
def __call__(cls, *args, **kw):
constructor = getattr(cls, "__new__")
instance = constructor(cls) if constructor is object.__new__ else constructor(cls, *args, **kw)
instance.__init__(cls, *args, **kw)
return instance
(I don't recall seeing on the docs the exact justification (or mechanism) for suppressing extra parameters to the root __new__ and passing it to other classes - but it is what happen "in real life" - if object.__new__ is called with any extra parameters it raises a type error - however, any custom implementation of a __new__ will get the extra parameters normally)
This is how you can dynamically create a class named Child in your code, assuming Parent already exists... even if you don't have an explicit Parent class, you could use object...
The code below defines __init__() and then associates it with the class.
>>> child_name = "Child"
>>> child_parents = (Parent,)
>>> child body = """
def __init__(self, arg1):
# Initialization for the Child class
self.foo = do_something(arg1)
"""
>>> child_dict = {}
>>> exec(child_body, globals(), child_dict)
>>> childobj = type(child_name, child_parents, child_dict)
>>> childobj.__name__
'Child'
>>> childobj.__bases__
(<type 'object'>,)
>>> # Instantiating the new Child object...
>>> childinst = childobj()
>>> childinst
<__main__.Child object at 0x1c91710>
>>>
If you have a module with a class you want to import, you can do it like this.
module = __import__(filename)
instance = module.MyClass()
If you do not know what the class is named, you can iterate through the classes available from a module.
import inspect
module = __import__(filename)
for c in module.__dict__.values():
if inspect.isclass(c):
# You may need do some additional checking to ensure
# it's the class you want
instance = c()
If you have some class object, you can instantiate it by just calling it (with parenthesis).
class MyClass: pass
some_class = MyClass
some_instance = some_class() # -> instance of MyClass
I think the neat way would be using type. Here is an example:
>>> class Foo:
... def __init__(self, s):
... self.s = s
...
>>> a = Foo("hello")
>>> a.s
'hello'
>>> b = type(a)("world")
>>> b.s
'world'
>>> assert isinstance(a, Foo)
>>> assert isinstance(b, Foo)
b is an instance which has the same type as a.

How can I pickle a dynamically created nested class in python?

I have a nested class:
class WidgetType(object):
class FloatType(object):
pass
class TextType(object):
pass
.. and an object that refers the nested class type (not an instance of it) like this
class ObjectToPickle(object):
def __init__(self):
self.type = WidgetType.TextType
Trying to serialize an instance of the ObjectToPickle class results in:
PicklingError: Can't pickle <class
'setmanager.app.site.widget_data_types.TextType'>
Is there a way to pickle nested classes in python?
I know this is a very old question, but I have never explicitly seen a satisfactory solution to this question other than the obvious, and most likely correct, answer to re-structure your code.
Unfortunately, it is not always practical to do such a thing, in which case as a very last resort, it is possible to pickle instances of classes which are defined inside another class.
The python documentation for the __reduce__ function states that you can return
A callable object that will be called to create the initial version of the object. The next element of the tuple will provide arguments for this callable.
Therefore, all you need is an object which can return an instance of the appropriate class. This class must itself be picklable (hence, must live on the __main__ level), and could be as simple as:
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
and the name of the nested class as the second argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name):
nested_class = getattr(containing_class, class_name)
# return an instance of a nested_class. Some more intelligence could be
# applied for class construction if necessary.
return nested_class()
All that is left therefore, is to return the appropriate arguments in a __reduce__ method on FloatType:
class WidgetType(object):
class FloatType(object):
def __reduce__(self):
# return a class which can return this class when called with the
# appropriate tuple of arguments
return (_NestedClassGetter(), (WidgetType, self.__class__.__name__, ))
The result is a class which is nested but instances can be pickled (further work is needed to dump/load the __state__ information, but this is relatively straightforward as per the __reduce__ documentation).
This same technique (with slight code modifications) can be applied for deeply nested classes.
A fully worked example:
import pickle
class ParentClass(object):
class NestedClass(object):
def __init__(self, var1):
self.var1 = var1
def __reduce__(self):
state = self.__dict__.copy()
return (_NestedClassGetter(),
(ParentClass, self.__class__.__name__, ),
state,
)
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
and the name of the nested class as the second argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name):
nested_class = getattr(containing_class, class_name)
# make an instance of a simple object (this one will do), for which we can change the
# __class__ later on.
nested_instance = _NestedClassGetter()
# set the class of the instance, the __init__ will never be called on the class
# but the original state will be set later on by pickle.
nested_instance.__class__ = nested_class
return nested_instance
if __name__ == '__main__':
orig = ParentClass.NestedClass(var1=['hello', 'world'])
pickle.dump(orig, open('simple.pickle', 'w'))
pickled = pickle.load(open('simple.pickle', 'r'))
print type(pickled)
print pickled.var1
My final note on this is to remember what the other answers have said:
If you are in a position to do so, consider re-factoring your code to
avoid the nested classes in the first place.
The pickle module is trying to get the TextType class from the module. But since the class is nested it doesn't work. jasonjs's suggestion will work.
Here are the lines in pickle.py responsible for the error message:
try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
klass = getattr(mod, name) will not work in the nested class case of course. To demonstrate what is going on try to add these lines before pickling the instance:
import sys
setattr(sys.modules[__name__], 'TextType', WidgetType.TextType)
This code adds TextType as an attribute to the module. The pickling should work just fine. I don't advice you to use this hack though.
If you use dill instead of pickle, it works.
>>> import dill
>>>
>>> class WidgetType(object):
... class FloatType(object):
... pass
... class TextType(object):
... pass
...
>>> class ObjectToPickle(object):
... def __init__(self):
... self.type = WidgetType.TextType
...
>>> x = ObjectToPickle()
>>>
>>> _x = dill.dumps(x)
>>> x_ = dill.loads(_x)
>>> x_
<__main__.ObjectToPickle object at 0x10b20a250>
>>> x_.type
<class '__main__.TextType'>
Get dill here: https://github.com/uqfoundation/dill
In Sage (www.sagemath.org), we have many instances of this pickling issue. The way we decided to systematically solve it is to put the outer class inside a specific metaclass whose goal is to implement and hide the hack. Note that this automatically propagate through nested classes if there are several level of nesting.
Pickle only works with classes defined in module scope (top level). In this case, it looks like you could define the nested classes in module scope and then set them as properties on WidgetType, assuming there's a reason not to just reference TextType and FloatType in your code. Or, import the module they're in and use widget_type.TextType and widget_type.FloatType.
Nadia's answer is pretty complete - it is practically not something you want to be doing; are you sure you can't use inheritance in WidgetTypes instead of nested classes?
The only reason to use nested classes is to encapsulate classes working together closely, your specific example looks like an immediate inheritance candidate to me - there is no benefit in nesting WidgetType classes together; put them in a module and inherit from the base WidgetType instead.
This seems to work fine in newer versions of Python. I tried it in v3.8 and it was able to pickle and unpickle the nested class.

How to do Obj-C Categories in Python?

Obj-C (which I have not used for a long time) has something called categories to extend classes. Declaring a category with new methods and compiling it into your program, all instances of the class suddenly have the new methods.
Python has mixin possibilities, which I use, but mixins must be used from the bottom of the program: the class has to declare it itself.
Foreseen category use-case: Say you have a big class hierarchy that describe different ways of interacting with data, declaring polymorphic ways to get at different attributes. Now a category can help the consumer of these describing classes by implementing a convenient interface to access these methods in one place. (A category method could for example, try two different methods and return the first defined (non-None) return value.)
Any way to do this in Python?
Illustrative code
I hope this clarifies what I mean. The point is that the Category is like an aggregate interface, that the consumer of AppObj can change in its code.
class AppObj (object):
"""This is the top of a big hierarchy of subclasses that describe different data"""
def get_resource_name(self):
pass
def get_resource_location(self):
pass
# dreaming up class decorator syntax
#category(AppObj)
class AppObjCategory (object):
"""this is a category on AppObj, not a subclass"""
def get_resource(self):
name = self.get_resource_name()
if name:
return library.load_resource_name(name)
else:
return library.load_resource(self.get_resource_location())
Why not just add methods dynamically ?
>>> class Foo(object):
>>> pass
>>> def newmethod(instance):
>>> print 'Called:', instance
...
>>> Foo.newmethod = newmethod
>>> f = Foo()
>>> f.newmethod()
Called: <__main__.Foo object at 0xb7c54e0c>
I know Objective-C and this looks just like categories. The only drawback is that you can't do that to built-in or extension types.
I came up with this implementation of a class decorator. I'm using python2.5 so I haven't actually tested it with decorator syntax (which would be nice), and I'm not sure what it does is really correct. But it looks like this:
pycategories.py
"""
This module implements Obj-C-style categories for classes for Python
Copyright 2009 Ulrik Sverdrup <ulrik.sverdrup#gmail.com>
License: Public domain
"""
def Category(toclass, clobber=False):
"""Return a class decorator that implements the decorated class'
methods as a Category on the class #toclass
if #clobber is not allowed, AttributeError will be raised when
the decorated class already contains the same attribute.
"""
def decorator(cls):
skip = set(("__dict__", "__module__", "__weakref__", "__doc__"))
for attr in cls.__dict__:
if attr in toclass.__dict__:
if attr in skip:
continue
if not clobber:
raise AttributeError("Category cannot override %s" % attr)
setattr(toclass, attr, cls.__dict__[attr])
return cls
return decorator
Python's setattr function makes this easy.
# categories.py
class category(object):
def __init__(self, mainModule, override = True):
self.mainModule = mainModule
self.override = override
def __call__(self, function):
if self.override or function.__name__ not in dir(self.mainModule):
setattr(self.mainModule, function.__name__, function)
# categories_test.py
import this
from categories import category
#category(this)
def all():
print "all things are this"
this.all()
>>> all things are this

Categories