I am trying to update a class that is supposed to be used as a custom type for the multiprocessing.manager and imitate a basic dictionary. All works well on Linux, but things fail on Windows and I understood that the problem lies in a possibly suboptimal creation mechanism that it uses that involves a closure. With forking, Linux gets around serializing something that pickle cannot cope with, while this does not happen on Windows. I am using Python 3.6 and feel like it is better to improve the class rather than force a new package dependency that has more robust serialization than pickle.
An example that I think demonstrates this is presented below. It involves a class that is meant to act like a dict, but have an additional method and a class attribute. These are bound in a factory method that the code calls and passes to multiprocessing.manager.register. I get AttributeError: Can't pickle local object 'foo_factory.<locals>.Foo' as a result here.
import abc
import pickle
class FooTemplate(abc.ABC, dict):
bar = None
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.foo = 'foo'
#abc.abstractmethod
def special_method(self, arg1, arg2):
pass
def foo_factory(dynamic_special_method):
class Foo(FooTemplate):
bar = 'bar'
def special_method(self, arg1, arg2):
print(self.foo, ' ', self.bar, ' ', dynamic_special_method(arg1, arg2))
return Foo
def method_to_pass(a1, a2):
return a1 + a2
if __name__ == '__main__':
foo = foo_factory(method_to_pass)()
pickle.dumps(foo)
I attempted to fix the problem by creating a class dynamically, but this throws a new error that I am not sure I understand and it makes things look even worse with all honesty. Using the main part from above with the code below produces error _pickle.PicklingError: Can't pickle <class '__main__.Foo'>: attribute lookup Foo on __main__ failed.
class FooTemplate(dict):
bar = None
method_map = None
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.foo = 'foo'
def special_method(self, arg1, arg2):
print(self.foo, ' ', self.bar, ' ', self.method_map[self.bar](arg1, arg2))
def foo_factory(dynamic_special_method):
return type('Foo', (FooTemplate,), {'bar': 'bar', 'method_map': {'bar': dynamic_special_method}})
Error above aside, I feel like I am missing something fundamental and that I took a wrong direction. Even if this worked, it feels wrong to introduce a new attribute with a nested structure to simply keep a method which avoids calls to this method as a class method with self in the front...
Maybe someone can suggest a better direction how to create a preferably serializable class which imitates a dictionary and that can also get parameters dynamically? An explanation of the error that I get would be very useful too, but I think this is not the biggest problem I am facing here. Thank you for any help in advance.
With a lot of help and encouragement from #CharchitAgarwal, I have figured out two solutions to the problem.
If you are looking for the solution to the error in my last attempt, then Pickle a dynamically parameterized sub-class can be consulted. Tried it, all worked with minimal alterations, but the final result was harder to follow and not satisfying. Due to ease of copy/pasting and the bitter taste afterwards, I will not share the solution here.
Better answer to the question is to stop for a moment and to reflect if you are on the right path. Had this doubt when posting the question and, fortunately, I am not committed to the structure above. I think it is better to use composition instead of inheritance. Code is shorter and much more simple. If necessary, one can bind the special parameters with functools.partial and objects are still going to be serializable in Python 3.6 that was used here.
import pickle
class Foo(dict):
def __init__(self, *args, special_method=None, bar=None, **kwargs):
super().__init__(*args, **kwargs)
self.foo = 'foo'
self._special_method = special_method
self.bar = bar
def special_method(self, arg1, arg2):
print(self.foo, ' ', self.bar, ' ', self._special_method(arg1, arg2))
def method_to_pass(a1, a2):
return a1 + a2
if __name__ == '__main__':
foo = Foo({'foo': 'bar'}, special_method=method_to_pass, bar='bar')
print(foo)
foo.special_method(4, 2)
pickle.dumps(foo)
Related
This is how it works for me:
class SomeName:
def __init__(self):
self.value = "something"
def some_method(self):
print self.value
def external_func(instance, method):
method(instance)
external_func(SomeName(), SomeName.some_method)
This appears to work correctly. Is this the right way to do this?
Your code is "technically correct" (it does what you ask for) but - at least in your example - pretty useless:
def external_func(instance, method):
method(instance)
external_func(SomeName(), SomeName.some_method)
is the same as:
def external_func(method):
method()
external_func(SomeName().some_method)
which FWIW is the same as:
SomeName().some_method()
but I assume you understood this already .
Now you probably have a reason to try to pass both the method AND instance to external_func(), or there might be a better way to solve your real problem...
I of course don't know what you're doing exactly, but it sounds to me like you're trying to do too much inside of one function. Your problem might be better solved by simply splitting up the contents of external_func.
The goals here, as I understand them, are you don't know ahead of time what the object/method pair will be, and want to reduce code repetition.
Perhaps something like this would be better:
def main():
obj = SomeName()
# do the setting up portion
complex_object = external_func_set_up(obj)
# presumably at some point you have to designate the method to be used:
method = get_method_name(obj)
# run the method:
getattr(obj, method)()
# finish up the external operation:
external_func_complete(***args***)
I understand this is more code, but I think in the end it's a lot clearer what is happening, and also might force you to think through your problem a bit more (and potentially come up with an even better solution).
You could pass SomeName().some_method or make some_metod staticmethod or classmethod if there is no instance data used in your method.
Check documentation to know more about staticmethod and classmethod:
https://docs.python.org/3/library/functions.html#staticmethod
https://docs.python.org/3/library/functions.html#classmethod
Depending on what you're doing. Because functions are also objects in Python it is possible to do so.
But is it a good solution? It seems though that you're trying to handle a problem which maybe could be better solved with more of an object oriented approach:
class A:
def __init__(self):
self.value = "class A"
def some_method(self):
print self.value
class B:
def __init__(self):
self.value = "class B"
def some_method(self):
print self.value
some_class = A()
some_class.some_method()
some_class = B()
some_class.some_method()
Output:
"class A"
"class B"
In my view this would be a better approach (if this is possible/reasonable in your case): You just call some_method() on your class, maybe without even knowing what exact type of object you're dealing with (regarding inheritance). The class itself knows what to do and reacts accordingly when its method has been called.
This of course doesn't work when you work with external libraries which you have no influence on.
I frequently have simple classes which I'll only ever want a single instance of. As a simple example:
import datetime
import sys
class PS1(object):
def __repr__(self):
now = datetime.datetime.now()
return str(now.strftime("%H:%M:%S"))
sys.ps1 = PS1()
Is there a way that I could somehow combine the definition and instantiation into a single step and achieve the same results?
As another example, just as something that is simple enough to understand.
class Example(object):
def methodOne(self, a, b):
return a + b
def methodTwo(self, a, b):
return a * b
example = Example()
I googled around and found nothing (lots of people throwing around the words one-off and anonymous but nobody seems to be talking about the same thing I am). I tried this, but it didn't work:
example = class(object):
def methodOne(self, a, b):
return a + b
def methodTwo(self, a, b):
return a * b
I realize I don't gain much, just one line I don't have to type plus one fewer things in my namespace, so I understand if this doesn't exist.
I think you don't see this often because it's really hard to read, but ...
sys.ps1 = type('PS1', (object,), {'__repr__': lambda self: datetime.datetime.now().strftime('%H:%M:%S')})()
would do the trick here...
I use type to dynmically create a class (the arguments are name, base classes, class dictionary). The class dictionary just consists of a single function __repr__ in this case.
Hopefully we can agree that the full format is much easier to grok and use ;-).
You could use a simple class decorator to replace the class with an instance of it:
def instantiator(cls):
return cls()
Then use it like this:
#instantiator
class PS1(object):
def __repr__(self):
now = datetime.datetime.now()
return str(now.strftime("%H:%M:%S"))
Then:
>>> PS1
11:53:37
If you do this, you might want to make the class name lowercase, since it will ultimately be used to name an instance, not a class.
This still requires an extra line, but not an extra name in the namespace.
If you really wanted to, you could write a metaclass that does the same thing, but automatically. However, I don't really think this would save much effort over just instantiating the class manually, and it would definitely make the code more complex and difficult to understand.
You could use a metaclass, so you can still use prettier syntax in comparison to #mgilson's answer.
class OneOff(type):
def __new__(cls, name, bases, attrs):
klass = type.__new__(cls, name, bases, attrs)
return klass()
class PS1(object):
__metaclass__ = OneOff
...
However, I'm with the others saying that I'm not sure this is a great idea. I did something like this once, but it was for a very specific usecase, and I'd really think about exploring other avenues first. Also, this looks an awful lot like a singleton/borg, so maybe that would be the better way for you to go.
(#mgilson's answer achieves what you're looking for in the most direct way. I second him on the opinion your original code is better than any of the answers here)
A simpler, more readable alternative, only if you don't need to use any special functions (e.g. __repr__), just use a dict of functions (playing the role of the methods):
fake_obj = dict(method_one = lambda a,b: a+b, method_two = lambda a,b: a*b)
There are two ways to do this in python. One is to instantiate a singleton object which can be done with a decorator, another is to make the class itself the used object with class methods and class variables.
The first option (singleton) looks like this:
def apply_class(*args, **kwargs):
def myclass(c):
c(*args,**kwargs)
return myclass
#apply_class(5)
class mysingleton(object):
def __init__(self, x):
print x
The second option (class methods/variables) looks like this:
class mysingleton:
myvariable = 5
#classmethod
def mymethod(cls):
print cls.myvariable
I have a class (Bar) which effectively has its own state and callback(s) and is used by another class (Foo):
class Foo(object):
def __init__(self):
self._bar = Bar(self.say, 10)
self._bar.work()
def say(self, msg):
print msg
class Bar(object):
def __init__(self, callback, value):
self._callback = callback
self._value = value
self._more = { 'foo' : 1, 'bar': 3, 'baz': 'fubar'}
def work(self):
# Do some work
self._more['foo'] = 5
self._value = 10
self._callback('FooBarBaz')
Foo()
Obviously I can't pickle the class Foo since Bar has an instancemethod, so I'm left with the following solution of implementing __getstate__ & __setstate__ in Bar to save self._value & self._more, but I have to instantiate the self._callback method as well (i.e. call __init__() from the outer class Foo passing the callback function.
But I cannot figure out how to achieve this.
Any help is much appreciated.
Thanks.
I think if you need to serialize something like this you need to be able to define your callback as a string. For example, you might say that callback = 'myproject.callbacks.foo_callback'.
Basically in __getstate__ you'd replace the _callback function with something you could use to look up the function later like self._callback.__name__.
In __setstate__ you'd replace _callback with a function.
This depends on your functions all having real names so you couldn't use a lambda as a callback and expect it to be serialized. You'd also need a reasonable mechanism for looking up your functions by name.
You could potentially use __import__ (something like: 'myproject.somemodule.somefunc' dotted name syntax could be supported that way, see http://code.google.com/p/mock/source/browse/mock.py#1076) or just define a lookup table in your code.
Just a quick (untested, sorry!) example assuming you have a small set of possible callbacks defined in a lookup table:
def a():
pass
callbacks_to_name = {a: 'a'
# ...
}
callbacks_by_name = {'a': a,
# ...
}
class C:
def __init__(self, cb):
self._callback = cb
def __getstate__(self):
self._callback = callbacks_to_name[self._callback]
return self.__dict__
def __setstate__(self, state):
state[_callback] = callbacks_by_name[self._callback]
I'm not sure what your use case is but I'd recommend doing this by serializing your work items to JSON or XML and writing a simple set of functions to serialize and deserialize them yourself.
The benefit is that the serialized format can be read and understood by humans and modified when you upgrade your software. Pickle is tempting because it seems close enough, but by the time you have a serious pile of __getstate__ and __setstate__ you haven't really saved yourself much effort or headache over building your own scheme specifically for your application.
Is there no magic python way of accessing the instance of the class that has a reference to the current self inside it?
ie:
class A(object):
def __init__(self):
self.B = B()
def say_hi(self):
print "Hi"
class B(object)
def __init__(self):
__get_owner_ref__.say_hi()
A()
get_owner_ref being the magic bullet that does not exist.
Is there a feature in python for this behaviour?
Yes I know I could pass a reference in to the constructor, but I'm looking for a more elegant solution.
No, You'd have to do something like this
class A(object):
def __init__(self):
self.B = B(parent=self)
def say_hi(self):
print "Hi"
class B(object)
def __init__(self, parent):
self.parent = parent # you don't need to do this, but it might be a good idea
parent.say_hi()
A()
On the second thought, what you're looking for pretty closely resembles descriptors. Consider:
class Agent(object):
def __get__(self, obj, objtype):
print 'Agent %s called from %s ' % (id(self), obj.name)
class X(object):
agent = Agent()
def __init__(self, name):
self.name = name
a = X('Foo')
a.agent
b = X('Bar')
b.agent
Here the agent is attached to two different instances and "knows" each time which instance wants to talk to him.
No, there is no nice way of doing this. Pass a reference in to the initializer.
To preclude questions, it's probably possible in most cases to find the owner heuristically by inspecting the stack, something like in this question. But it will be fragile, buggy and difficult to understand. And it goes against the "explicit > implicit" philosophy.
As far as I know such a feature does not exist. Also, passing it in as reference to the constructor and calling self.parent.say_hi() is much more explicit and (indeed) elegant. And explicit is better than implicit or using magic language features.
Technically, you can use sys._getframe:
class B(object):
def __init__(self):
import sys
a = sys._getframe(1).f_locals['self']
a.say_hi()
But you should not do that. It's bound to lead to confusion, will break on new Python implementations, will complicate debugging, and is prone to break.There's a reason why sys._getframe is listed in 5 Years of Bad Ideas.
Instead, pass a reference, either to the parent object, or to the say_hi method.
Many times I have member functions that copy parameters into object's fields. For Example:
class NouveauRiches(object):
def __init__(self, car, mansion, jet, bling):
self.car = car
self.mansion = mansion
self.jet = jet
self.bling = bling
Is there a python language construct that would make the above code less tedious?
One could use *args:
def __init__(self, *args):
self.car, self.mansion, self.jet, self.bling = args
+: less tedious
-: function signature not revealing enough. need to dive into function code to know how to use function
-: does not raise a TypeError on call with wrong # of parameters (but does raise a ValueError)
Any other ideas? (Whatever your suggestion, make sure the code calling the function does stays simple)
You could do this with a helper method, something like this:
import inspect
def setargs(func):
f = inspect.currentframe(1)
argspec = inspect.getargspec(func)
for arg in argspec.args:
setattr(f.f_locals["self"], arg, f.f_locals[arg])
Usage:
class Foo(object):
def __init__(self, bar, baz=4711):
setargs(self.__init__)
print self.bar # Now defined
print self.baz # Now defined
This is not pretty, and it should probably only be used when prototyping. Please use explicit assignment if you plan to have others read it.
It could probably be improved not to need to take the function as an argument, but that would require even more ugly hacks and trickery :)
I would go for this, also you could override already defined properties.
class D:
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
But i personally would just go the long way.
Think of those:
- Explicit is better than implicit.
- Flat is better than nested.
(The Zen of Python)
Try something like
d = dict(locals())
del d['self']
self.__dict__.update(d)
Of course, it returns all local variables, not just function arguments.
I am not sure this is such a good idea, but it can be done:
import inspect
class NouveauRiches(object):
def __init__(self, car, mansion, jet, bling):
arguments = inspect.getargvalues(frame)[0]
values = inspect.getargvalues(frame)[3];
for name in arguments:
self.__dict__[name] = values[name]
It does not read great either, though I suppose you could put this in a utility method that is reused.
You could try something like this:
class C(object):
def __init__(self, **kwargs):
for k in kwargs:
d = {k: kwargs[k]}
self.__dict__.update(d)
Or using setattr you can do:
class D(object):
def __init__(self, **kwargs):
for k in kwargs:
setattr(self, k, kwargs[k])
Both can then be called like:
myclass = C(test=1, test2=2)
So you have to use **kwargs, rather than *args.
I sometimes do this for classes that act "bunch-like", that is, they have a bunch of customizable attributes:
class SuperClass(object):
def __init__(self, **kw):
for name, value in kw.iteritems():
if not hasattr(self, name):
raise TypeError('Unexpected argument: %s' % name)
setattr(self, name, value)
class SubClass(SuperClass):
instance_var = None # default value
class SubClass2(SubClass):
other_instance_var = True
#property
def something_dynamic(self):
return self._internal_var
#something_dynamic.setter # new Python 2.6 feature of properties
def something_dynamic(self, value):
assert value is None or isinstance(value, str)
self._internal_var = value
Then you can call SubClass2(instance_var=[], other_instance_var=False) and it'll work without defining __init__ in either of them. You can use any property as well. Though this allows you to overwrite methods, which you probably wouldn't intend (as they return True for hasattr() just like an instance variable).
If you add any property or other other descriptor it will work fine. You can use that to do type checking; unlike type checking in __init__ it'll be applied any time that value is updated. Note you can't use any positional arguments for these unless you override __init__, so sometimes what would be a natural positional argument won't work. formencode.declarative covers this and other issues, probably with a thoroughness I would not suggest you attempt (in retrospect I don't think it's worth it).
Note that any recipe that uses self.__dict__ won't respect properties and descriptors, and if you use those together you'll just get weird and unexpected results. I only recommend using setattr() to set attributes, never self.__dict__.
Also this recipe doesn't give a very helpful signature, while some of the ones that do frame and function introspection do. With some work it is possible to dynamically generate a __doc__ that clarifies the arguments... but again I'm not sure the payoff is worth the addition of more moving parts.
I am a fan of the following
import inspect
def args_to_attrs(otherself):
frame = inspect.currentframe(1)
argvalues = inspect.getargvalues(frame)
for arg in argvalues.args:
if arg == 'self':
continue
value = argvalues.locals[arg]
setattr(otherself, arg, value)
class MyClass:
def __init__(self, arga="baf", argb="lek", argc=None):
args_to_attrs(self)
Arguments to __init__ are explicitly named, so it is clear what attributes are being set. Additionally, it is a little bit streamlined over the currently accepted answer.