import copyreg, pickle
class cars:
def __init__(self, name):
self.name = name
def myfunc(x):
return cars , (x.name,)
copyreg.pickle(cars, myfunc)
test = cars('hello')
print(pickle.dumps(test))
1 - Why did the class get called twise in this code ?
def myfunc(x):
#why does it need to be called here too ?
return cars , (x.name,)
#we already called the class here!!!
copyreg.pickle(cars, myfunc)
2 - and why comma in the tuple ?
The following is solely based on the documentations of copyreg.pickle and pickle dispatch_table.
The copyreg module offers a way to define functions used while pickling specific objects. The pickle and copy modules use those functions when pickling/copying those objects. The module provides configuration information about object constructors which are not classes.
This is mentioned in the copyreg.pickle docs, it basically tells that the function defined is used when pickling specified objects, here, it is cars.
A pickler object’s dispatch table is a registry of reduction functions of the kind which can be declared using copyreg.pickle(). It is a mapping whose keys are classes and whose values are reduction functions.
This is mentioned in the pickle.dispatch_table docs. It basically tells us that cars is the class being used as a key, and myfunc as the reduction function used as its value.
Here, to your doubts, copyreg is not calling your class. It is just creating/editing the dispatch_table with the keys(class) and values(func).
If you are still confused, here's a little experiment, where I added a print statement in the function:
import copyreg, pickle
class cars:
def __init__(self, name):
self.name = name
def myfunc(x):
print("This is a test")
return cars, (x.name,)
copyreg.pickle(cars, myfunc)
test = cars('hello')
print(pickle.dumps(test))
Output:
This is a test
b'\x80\x04\x95!\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x04cars\x94\x93\x94\x8c\x05hello\x94\x85\x94R\x94.'
Now, if I just remove the copyreg.pickle line. The dispatch_table won't be created:
import copyreg, pickle
class cars:
def __init__(self, name):
self.name = name
def myfunc(x):
print("This is a test")
return cars, (x.name,)
test = cars('hello')
print(pickle.dumps(test))
Ouput:
b'\x80\x04\x95+\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x04cars\x94\x93\x94)\x81\x94}\x94\x8c\x04name\x94\x8c\x05hello\x94sb.'
As you can see, the function is not at all triggered. Because the relation is not yet created.
To summarize:
copyreg.pickle just creates a dispatch_table with class as key and func as value, and the function defined is used when pickling specified class.
For the tuple, the comment by #Jean-FrançoisFabre is the most suitable.
EDIT: Upon more research, here are the addons.
copyreg.pickle makes the myfunc function act as a reduction function of cars. You can skip that part if you define __reduce__ method in the class itself.
Reduction functions are used by pickle for instruction on how to reconstruct the original object from the pickle object if it fails automatically.
import copyreg, pickle
class cars:
def __init__(self, name):
self.name = name
def __reduce__(self):
print("This is a test")
return cars, (self.name,)
test = cars('hello')
print(pickle.dumps(test))
Output:
This is a test
b'\x80\x04\x95!\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x04cars\x94\x93\x94\x8c\x05hello\x94\x85\x94R\x94.'
You can skip the copyreg.pickle if a __reduce__ instance is mentioned in your class, and it will act the same.
If you want to know, why these values have to be returned in the function, this documentation will give you insight on it.
Related
I don't think it is because of the scope of the function, but I get a
Unresolved reference at get_all_predicates(examples).count(predicate_list[0])
inside get_entropy_of_attributes(examples, predicate_list) function in my class Tree:
class Tree:
def get_examples(examples, attributes):
for value in examples:
yield dict(zip(attributes, value.strip().replace(" ", "").split(',')))
def get_all_predicates(examples):
return [d['Predicate'] for d in examples]
def get_entropy_of_attributes(examples, predicate_list):
get_all_predicates(examples).count(predicate_list[0])
return 0
examples = list(get_examples(all_examples, name_of_attributes))
predicate_list = list(set(get_all_predicates(examples)))
get_entropy_of_attributes(examples, predicate_list)
all_examples is a list of dictionary and name_of_attributes is a list, that holds values imported from a text file.
all_examples = [{'P_Length': '1.4', 'P_Width': '0.2', 'Predicate': 'I-setosa', 'Sepal_Width': '3.5', 'S_Length': '5.1'}, ...]
name_of_attributes = ["Check","P-Width"]
Any help?
Classes do not have scopes, only namespaces. This means that functions defined within them cannot see other class variables automatically.
class Foo(object):
var = 1 # lets create a class variable
def foo():
print(var) # this doesn't work!
To access a class variable, you need use attribute syntax: either Foo.var (to access via the class) or, if you're writing an instance method, with self.var (to access via the current instance, which will be passed in as the first argument).
class Bar(object):
var = 1
def bar1():
print(Bar.var) # works
def bar2(self):
print(self.var) # also works, if called on an instance, e.g. `Bar().bar2()`
With this kind of setup you can almost fix your current code (but not quite).
def get_entropy_of_attributes(examples, predicate_list):
Tree.get_all_predicates(examples).count(predicate_list[0]) # name the class
return 0
If you call this after the class is fully initialized, it will work without any exceptions (though it's implementation seems a bit nonsensical). However, it doesn't work when you call it to define a class variable, as your current code does. That's because the class object is only created and bound to the class name after all of the class body has been run.
I think the fix for that is probably to redesign your class in a more conventional way. Rather than having class variables set up based on various globals (like all_examples), you should probably create instances of your class by passing in arguments to the constructor and making the other variables you calculate from them instance attributes. I'd try to write it out, but frankly I don't understand what you're doing well enough.
If you want to call class methods, you have to call them with self, e.g.
class myClass:
def __init__(self):
pass
def get_all_predicates(self):
print('asd')
def do_something(self):
self.get_all_predicates() # working
get_all_predicates() # → Unresolved reference
test = myClass()
test.do_something()
See this link for examples for Python classes.
This article has a snippet showing usage of __bases__ to dynamically change the inheritance hierarchy of some Python code, by adding a class to an existing classes collection of classes from which it inherits. Ok, that's hard to read, code is probably clearer:
class Friendly:
def hello(self):
print 'Hello'
class Person: pass
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
That is, Person doesn't inherit from Friendly at the source level, but rather this inheritance relation is added dynamically at runtime by modification of the __bases__attribute of the Person class. However, if you change Friendly and Person to be new style classes (by inheriting from object), you get the following error:
TypeError: __bases__ assignment: 'Friendly' deallocator differs from 'object'
A bit of Googling on this seems to indicate some incompatibilities between new-style and old style classes in regards to changing the inheritance hierarchy at runtime. Specifically: "New-style class objects don't support assignment to their bases attribute".
My question, is it possible to make the above Friendly/Person example work using new-style classes in Python 2.7+, possibly by use of the __mro__ attribute?
Disclaimer: I fully realise that this is obscure code. I fully realize that in real production code tricks like this tend to border on unreadable, this is purely a thought experiment, and for funzies to learn something about how Python deals with issues related to multiple inheritance.
Ok, again, this is not something you should normally do, this is for informational purposes only.
Where Python looks for a method on an instance object is determined by the __mro__ attribute of the class which defines that object (the M ethod R esolution O rder attribute). Thus, if we could modify the __mro__ of Person, we'd get the desired behaviour. Something like:
setattr(Person, '__mro__', (Person, Friendly, object))
The problem is that __mro__ is a readonly attribute, and thus setattr won't work. Maybe if you're a Python guru there's a way around that, but clearly I fall short of guru status as I cannot think of one.
A possible workaround is to simply redefine the class:
def modify_Person_to_be_friendly():
# so that we're modifying the global identifier 'Person'
global Person
# now just redefine the class using type(), specifying that the new
# class should inherit from Friendly and have all attributes from
# our old Person class
Person = type('Person', (Friendly,), dict(Person.__dict__))
def main():
modify_Person_to_be_friendly()
p = Person()
p.hello() # works!
What this doesn't do is modify any previously created Person instances to have the hello() method. For example (just modifying main()):
def main():
oldperson = Person()
ModifyPersonToBeFriendly()
p = Person()
p.hello()
# works! But:
oldperson.hello()
# does not
If the details of the type call aren't clear, then read e-satis' excellent answer on 'What is a metaclass in Python?'.
I've been struggling with this too, and was intrigued by your solution, but Python 3 takes it away from us:
AttributeError: attribute '__dict__' of 'type' objects is not writable
I actually have a legitimate need for a decorator that replaces the (single) superclass of the decorated class. It would require too lengthy a description to include here (I tried, but couldn't get it to a reasonably length and limited complexity -- it came up in the context of the use by many Python applications of an Python-based enterprise server where different applications needed slightly different variations of some of the code.)
The discussion on this page and others like it provided hints that the problem of assigning to __bases__ only occurs for classes with no superclass defined (i.e., whose only superclass is object). I was able to solve this problem (for both Python 2.7 and 3.2) by defining the classes whose superclass I needed to replace as being subclasses of a trivial class:
## T is used so that the other classes are not direct subclasses of object,
## since classes whose base is object don't allow assignment to their __bases__ attribute.
class T: pass
class A(T):
def __init__(self):
print('Creating instance of {}'.format(self.__class__.__name__))
## ordinary inheritance
class B(A): pass
## dynamically specified inheritance
class C(T): pass
A() # -> Creating instance of A
B() # -> Creating instance of B
C.__bases__ = (A,)
C() # -> Creating instance of C
## attempt at dynamically specified inheritance starting with a direct subclass
## of object doesn't work
class D: pass
D.__bases__ = (A,)
D()
## Result is:
## TypeError: __bases__ assignment: 'A' deallocator differs from 'object'
I can not vouch for the consequences, but that this code does what you want at py2.7.2.
class Friendly(object):
def hello(self):
print 'Hello'
class Person(object): pass
# we can't change the original classes, so we replace them
class newFriendly: pass
newFriendly.__dict__ = dict(Friendly.__dict__)
Friendly = newFriendly
class newPerson: pass
newPerson.__dict__ = dict(Person.__dict__)
Person = newPerson
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
We know that this is possible. Cool. But we'll never use it!
Right of the bat, all the caveats of messing with class hierarchy dynamically are in effect.
But if it has to be done then, apparently, there is a hack that get's around the "deallocator differs from 'object" issue when modifying the __bases__ attribute for the new style classes.
You can define a class object
class Object(object): pass
Which derives a class from the built-in metaclass type.
That's it, now your new style classes can modify the __bases__ without any problem.
In my tests this actually worked very well as all existing (before changing the inheritance) instances of it and its derived classes felt the effect of the change including their mro getting updated.
I needed a solution for this which:
Works with both Python 2 (>= 2.7) and Python 3 (>= 3.2).
Lets the class bases be changed after dynamically importing a dependency.
Lets the class bases be changed from unit test code.
Works with types that have a custom metaclass.
Still allows unittest.mock.patch to function as expected.
Here's what I came up with:
def ensure_class_bases_begin_with(namespace, class_name, base_class):
""" Ensure the named class's bases start with the base class.
:param namespace: The namespace containing the class name.
:param class_name: The name of the class to alter.
:param base_class: The type to be the first base class for the
newly created type.
:return: ``None``.
Call this function after ensuring `base_class` is
available, before using the class named by `class_name`.
"""
existing_class = namespace[class_name]
assert isinstance(existing_class, type)
bases = list(existing_class.__bases__)
if base_class is bases[0]:
# Already bound to a type with the right bases.
return
bases.insert(0, base_class)
new_class_namespace = existing_class.__dict__.copy()
# Type creation will assign the correct ‘__dict__’ attribute.
del new_class_namespace['__dict__']
metaclass = existing_class.__metaclass__
new_class = metaclass(class_name, tuple(bases), new_class_namespace)
namespace[class_name] = new_class
Used like this within the application:
# foo.py
# Type `Bar` is not available at first, so can't inherit from it yet.
class Foo(object):
__metaclass__ = type
def __init__(self):
self.frob = "spam"
def __unicode__(self): return "Foo"
# … later …
import bar
ensure_class_bases_begin_with(
namespace=globals(),
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
Use like this from within unit test code:
# test_foo.py
""" Unit test for `foo` module. """
import unittest
import mock
import foo
import bar
ensure_class_bases_begin_with(
namespace=foo.__dict__,
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
class Foo_TestCase(unittest.TestCase):
""" Test cases for `Foo` class. """
def setUp(self):
patcher_unicode = mock.patch.object(
foo.Foo, '__unicode__')
patcher_unicode.start()
self.addCleanup(patcher_unicode.stop)
self.test_instance = foo.Foo()
patcher_frob = mock.patch.object(
self.test_instance, 'frob')
patcher_frob.start()
self.addCleanup(patcher_frob.stop)
def test_instantiate(self):
""" Should create an instance of `Foo`. """
instance = foo.Foo()
The above answers are good if you need to change an existing class at runtime. However, if you are just looking to create a new class that inherits by some other class, there is a much cleaner solution. I got this idea from https://stackoverflow.com/a/21060094/3533440, but I think the example below better illustrates a legitimate use case.
def make_default(Map, default_default=None):
"""Returns a class which behaves identically to the given
Map class, except it gives a default value for unknown keys."""
class DefaultMap(Map):
def __init__(self, default=default_default, **kwargs):
self._default = default
super().__init__(**kwargs)
def __missing__(self, key):
return self._default
return DefaultMap
DefaultDict = make_default(dict, default_default='wug')
d = DefaultDict(a=1, b=2)
assert d['a'] is 1
assert d['b'] is 2
assert d['c'] is 'wug'
Correct me if I'm wrong, but this strategy seems very readable to me, and I would use it in production code. This is very similar to functors in OCaml.
This method isn't technically inheriting during runtime, since __mro__ can't be changed. But what I'm doing here is using __getattr__ to be able to access any attributes or methods from a certain class. (Read comments in order of numbers placed before the comments, it makes more sense)
class Sub:
def __init__(self, f, cls):
self.f = f
self.cls = cls
# 6) this method will pass the self parameter
# (which is the original class object we passed)
# and then it will fill in the rest of the arguments
# using *args and **kwargs
def __call__(self, *args, **kwargs):
# 7) the multiple try / except statements
# are for making sure if an attribute was
# accessed instead of a function, the __call__
# method will just return the attribute
try:
return self.f(self.cls, *args, **kwargs)
except TypeError:
try:
return self.f(*args, **kwargs)
except TypeError:
return self.f
# 1) our base class
class S:
def __init__(self, func):
self.cls = func
def __getattr__(self, item):
# 5) we are wrapping the attribute we get in the Sub class
# so we can implement the __call__ method there
# to be able to pass the parameters in the correct order
return Sub(getattr(self.cls, item), self.cls)
# 2) class we want to inherit from
class L:
def run(self, s):
print("run" + s)
# 3) we create an instance of our base class
# and then pass an instance (or just the class object)
# as a parameter to this instance
s = S(L) # 4) in this case, I'm using the class object
s.run("1")
So this sort of substitution and redirection will simulate the inheritance of the class we wanted to inherit from. And it even works with attributes or methods that don't take any parameters.
I have a system which commonly stores pickled class types.
I want to be able to save dynamically-parameterized classes in the same way, but I can't because I get a PicklingError on trying to pickle a class which is not globally found (not defined in simple code).
My problem can be modeled as the following example code:
class Base(object):
def m(self):
return self.__class__.PARAM
def make_parameterized(param_value):
class AutoSubClass(Base):
PARAM = param_value
return AutoSubClass
cls = make_parameterized(input("param value?"))
When I try to pickle the class, I get the following error:
# pickle.PicklingError: Can't pickle <class '__main__.AutoSubClass'>: it's not found as __main__.AutoSubClass
import pickle
print pickle.dumps(cls)
I am looking for some method to declare Base as a ParameterizableBaseClass which should define the params needed (PARAM in above example). A dynamic parameterized subclass (cls above) should then be picklable by saving the "ParameterizableBaseClass" type and the different param-values (dynamic param_value above).
I am sure that in many cases, this can be avoided altogether... And I can avoid this in my code as well if I really (really) have to. I was playing with __metaclass__, copyreg and even __builtin__.issubclass at some point (don't ask), but was unable to crack this one.
I feel like I wouldn't be true to the python spirit if I wasn't to ask: how can this be achieved, in a relatively clean way?
I know this is a very old question, but I think it is worth sharing a better means of pickling the parameterised classes than the one that is the currently accepted solution (making the parameterised class a global).
Using the __reduce__ method, we can provide a callable which will return an uninitialised instance of our desired class.
class Base(object):
def m(self):
return self.__class__.PARAM
def __reduce__(self):
return (_InitializeParameterized(), (self.PARAM, ), self.__dict__)
def make_parameterized(param_value):
class AutoSub(Base):
PARAM = param_value
return AutoSub
class _InitializeParameterized(object):
"""
When called with the param value as the only argument, returns an
un-initialized instance of the parameterized class. Subsequent __setstate__
will be called by pickle.
"""
def __call__(self, param_value):
# make a simple object which has no complex __init__ (this one will do)
obj = _InitializeParameterized()
obj.__class__ = make_parameterized(param_value)
return obj
if __name__ == "__main__":
from pickle import dumps, loads
a = make_parameterized("a")()
b = make_parameterized("b")()
print a.PARAM, b.PARAM, type(a) is type(b)
a_p = dumps(a)
b_p = dumps(b)
del a, b
a = loads(a_p)
b = loads(b_p)
print a.PARAM, b.PARAM, type(a) is type(b)
It is worth reading the __reduce__ docs a couple of times to see exactly what is going on here.
Hope somebody finds this useful.
Yes, it is possible -
Whenever you want to custom the Pickle and Unpickle behaviors for your objects, you just have to set the "__getstate__" and "__setstate__" methods on the class itself.
In this case it is a bit trickier:
There need, as you observed - to exist a class on the global namespace that is the class of the currently being pickled object: it has to be the same class, with the same name. Ok - the deal is that gthis class existing in the globalname space can be created at Pickle time.
At Unpickle time the class, with the same name, have to exist - but it does not have to be the same object - just behave like it does - and as __setstate__ is called in the Unpickling proccess, it can recreate the parameterized class of the orignal object, and set its own class to be that one, by setting the __class__ attribute of the object.
Setting the __class__ attribute of an object may seen objectionable but it is how OO works in Python and it is officially documented, it even works accross implementations. (I tested this snippet in both Python 2.6 and Pypy)
class Base(object):
def m(self):
return self.__class__.PARAM
def __getstate__(self):
global AutoSub
AutoSub = self.__class__
return (self.__dict__,self.__class__.PARAM)
def __setstate__(self, state):
self.__class__ = make_parameterized(state[1])
self.__dict__.update(state[0])
def make_parameterized(param_value):
class AutoSub(Base):
PARAM = param_value
return AutoSub
class AutoSub(Base):
pass
if __name__ == "__main__":
from pickle import dumps, loads
a = make_parameterized("a")()
b = make_parameterized("b")()
print a.PARAM, b.PARAM, type(a) is type(b)
a_p = dumps(a)
b_p = dumps(b)
del a, b
a = loads(a_p)
b = loads(b_p)
print a.PARAM, b.PARAM, type(a) is type(b)
I guess it's too late now, but pickle is a module I'd rather avoid for anything complex, because it has problems like this one and many more.
Anyways, since pickle wants the class in a global it can have it:
import cPickle
class Base(object):
def m(self):
return self.__class__.PARAM
#classmethod
def make_parameterized(cls,param):
clsname = "AutoSubClass.%s" % param
# create a class, assign it as a global under the same name
typ = globals()[clsname] = type(clsname, (cls,), dict(PARAM=param))
return typ
cls = Base.make_parameterized('asd')
import pickle
s = pickle.dumps(cls)
cls = pickle.loads(s)
print cls, cls.PARAM
# <class '__main__.AutoSubClass.asd'> asd
But yeah, you're probably overcomplicating things.
Classes that are not created in the top level of a module cannot be pickled, as shown in the Python documentation.
Furthermore, even for an instance of a top level module class the class attributes are not stored. So in your example PARAM wouldn't be stored anyway. (Explained in the Python documentation section linked above as well)
I have a nested class:
class WidgetType(object):
class FloatType(object):
pass
class TextType(object):
pass
.. and an object that refers the nested class type (not an instance of it) like this
class ObjectToPickle(object):
def __init__(self):
self.type = WidgetType.TextType
Trying to serialize an instance of the ObjectToPickle class results in:
PicklingError: Can't pickle <class
'setmanager.app.site.widget_data_types.TextType'>
Is there a way to pickle nested classes in python?
I know this is a very old question, but I have never explicitly seen a satisfactory solution to this question other than the obvious, and most likely correct, answer to re-structure your code.
Unfortunately, it is not always practical to do such a thing, in which case as a very last resort, it is possible to pickle instances of classes which are defined inside another class.
The python documentation for the __reduce__ function states that you can return
A callable object that will be called to create the initial version of the object. The next element of the tuple will provide arguments for this callable.
Therefore, all you need is an object which can return an instance of the appropriate class. This class must itself be picklable (hence, must live on the __main__ level), and could be as simple as:
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
and the name of the nested class as the second argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name):
nested_class = getattr(containing_class, class_name)
# return an instance of a nested_class. Some more intelligence could be
# applied for class construction if necessary.
return nested_class()
All that is left therefore, is to return the appropriate arguments in a __reduce__ method on FloatType:
class WidgetType(object):
class FloatType(object):
def __reduce__(self):
# return a class which can return this class when called with the
# appropriate tuple of arguments
return (_NestedClassGetter(), (WidgetType, self.__class__.__name__, ))
The result is a class which is nested but instances can be pickled (further work is needed to dump/load the __state__ information, but this is relatively straightforward as per the __reduce__ documentation).
This same technique (with slight code modifications) can be applied for deeply nested classes.
A fully worked example:
import pickle
class ParentClass(object):
class NestedClass(object):
def __init__(self, var1):
self.var1 = var1
def __reduce__(self):
state = self.__dict__.copy()
return (_NestedClassGetter(),
(ParentClass, self.__class__.__name__, ),
state,
)
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
and the name of the nested class as the second argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name):
nested_class = getattr(containing_class, class_name)
# make an instance of a simple object (this one will do), for which we can change the
# __class__ later on.
nested_instance = _NestedClassGetter()
# set the class of the instance, the __init__ will never be called on the class
# but the original state will be set later on by pickle.
nested_instance.__class__ = nested_class
return nested_instance
if __name__ == '__main__':
orig = ParentClass.NestedClass(var1=['hello', 'world'])
pickle.dump(orig, open('simple.pickle', 'w'))
pickled = pickle.load(open('simple.pickle', 'r'))
print type(pickled)
print pickled.var1
My final note on this is to remember what the other answers have said:
If you are in a position to do so, consider re-factoring your code to
avoid the nested classes in the first place.
The pickle module is trying to get the TextType class from the module. But since the class is nested it doesn't work. jasonjs's suggestion will work.
Here are the lines in pickle.py responsible for the error message:
try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
klass = getattr(mod, name) will not work in the nested class case of course. To demonstrate what is going on try to add these lines before pickling the instance:
import sys
setattr(sys.modules[__name__], 'TextType', WidgetType.TextType)
This code adds TextType as an attribute to the module. The pickling should work just fine. I don't advice you to use this hack though.
If you use dill instead of pickle, it works.
>>> import dill
>>>
>>> class WidgetType(object):
... class FloatType(object):
... pass
... class TextType(object):
... pass
...
>>> class ObjectToPickle(object):
... def __init__(self):
... self.type = WidgetType.TextType
...
>>> x = ObjectToPickle()
>>>
>>> _x = dill.dumps(x)
>>> x_ = dill.loads(_x)
>>> x_
<__main__.ObjectToPickle object at 0x10b20a250>
>>> x_.type
<class '__main__.TextType'>
Get dill here: https://github.com/uqfoundation/dill
In Sage (www.sagemath.org), we have many instances of this pickling issue. The way we decided to systematically solve it is to put the outer class inside a specific metaclass whose goal is to implement and hide the hack. Note that this automatically propagate through nested classes if there are several level of nesting.
Pickle only works with classes defined in module scope (top level). In this case, it looks like you could define the nested classes in module scope and then set them as properties on WidgetType, assuming there's a reason not to just reference TextType and FloatType in your code. Or, import the module they're in and use widget_type.TextType and widget_type.FloatType.
Nadia's answer is pretty complete - it is practically not something you want to be doing; are you sure you can't use inheritance in WidgetTypes instead of nested classes?
The only reason to use nested classes is to encapsulate classes working together closely, your specific example looks like an immediate inheritance candidate to me - there is no benefit in nesting WidgetType classes together; put them in a module and inherit from the base WidgetType instead.
This seems to work fine in newer versions of Python. I tried it in v3.8 and it was able to pickle and unpickle the nested class.
Obj-C (which I have not used for a long time) has something called categories to extend classes. Declaring a category with new methods and compiling it into your program, all instances of the class suddenly have the new methods.
Python has mixin possibilities, which I use, but mixins must be used from the bottom of the program: the class has to declare it itself.
Foreseen category use-case: Say you have a big class hierarchy that describe different ways of interacting with data, declaring polymorphic ways to get at different attributes. Now a category can help the consumer of these describing classes by implementing a convenient interface to access these methods in one place. (A category method could for example, try two different methods and return the first defined (non-None) return value.)
Any way to do this in Python?
Illustrative code
I hope this clarifies what I mean. The point is that the Category is like an aggregate interface, that the consumer of AppObj can change in its code.
class AppObj (object):
"""This is the top of a big hierarchy of subclasses that describe different data"""
def get_resource_name(self):
pass
def get_resource_location(self):
pass
# dreaming up class decorator syntax
#category(AppObj)
class AppObjCategory (object):
"""this is a category on AppObj, not a subclass"""
def get_resource(self):
name = self.get_resource_name()
if name:
return library.load_resource_name(name)
else:
return library.load_resource(self.get_resource_location())
Why not just add methods dynamically ?
>>> class Foo(object):
>>> pass
>>> def newmethod(instance):
>>> print 'Called:', instance
...
>>> Foo.newmethod = newmethod
>>> f = Foo()
>>> f.newmethod()
Called: <__main__.Foo object at 0xb7c54e0c>
I know Objective-C and this looks just like categories. The only drawback is that you can't do that to built-in or extension types.
I came up with this implementation of a class decorator. I'm using python2.5 so I haven't actually tested it with decorator syntax (which would be nice), and I'm not sure what it does is really correct. But it looks like this:
pycategories.py
"""
This module implements Obj-C-style categories for classes for Python
Copyright 2009 Ulrik Sverdrup <ulrik.sverdrup#gmail.com>
License: Public domain
"""
def Category(toclass, clobber=False):
"""Return a class decorator that implements the decorated class'
methods as a Category on the class #toclass
if #clobber is not allowed, AttributeError will be raised when
the decorated class already contains the same attribute.
"""
def decorator(cls):
skip = set(("__dict__", "__module__", "__weakref__", "__doc__"))
for attr in cls.__dict__:
if attr in toclass.__dict__:
if attr in skip:
continue
if not clobber:
raise AttributeError("Category cannot override %s" % attr)
setattr(toclass, attr, cls.__dict__[attr])
return cls
return decorator
Python's setattr function makes this easy.
# categories.py
class category(object):
def __init__(self, mainModule, override = True):
self.mainModule = mainModule
self.override = override
def __call__(self, function):
if self.override or function.__name__ not in dir(self.mainModule):
setattr(self.mainModule, function.__name__, function)
# categories_test.py
import this
from categories import category
#category(this)
def all():
print "all things are this"
this.all()
>>> all things are this