Original problem description
The problem arises when I implement some machine learning algorithm with numpy. I want some new class ludmo which works the same as numpy.ndarray, but with a few more properties. For example, with a new property ludmo.foo. I've tried several methods below, but none is satisfactory.
1. Wrapper
First I created a wrapper class for numpy.ndarray, as
import numpy as np
class ludmo(object):
def __init__(self)
self.foo = None
self.data = np.array([])
But when I use some function (in scikit-learn which I cannot modify) to manipulate a list of np.ndarray instance, I have to first extract all data field of each ludmo object and collect them into a list. After that the list is sorted and I lost the correspondence between the data and original ludmo objects.
2. Inheritance
Then I tried to make ludmo a subclass of numpy.ndarray, as
import numpy as np
class ludmo(np.ndarray):
def __init__(self, shape, dtype=float, buffer=None, offset=0, strides=None, order=None)
super().__init__(shape, dtype, buffer, offset, strides, order)
self.foo = None
But another problem arises then: the most common way to create a numpy.ndarray object is numpy.array(some_list), which returns a numpy.ndarray object, and I have to convert it to a ludmo object. But till now I found no good way to do this; simply changing the __class__ attribute will result in an error.
I'm new to Python and numpy, so there must be some elegant way that I don't know. Any advice is appreciated.
It's better if anyone can give an generic solution, which not only applies to the numpy.ndarray class but also all kinds of classes.
As explained in the docs you could add your own methods to np.ndarray doing:
import numpy as np
class Ludmo(np.ndarray):
def sumcols(self):
return self.sum(axis=1)
def sumrows(self):
return self.sum(axis=0)
def randomize(self):
self[:] = np.random.rand(*self.shape)
and then creating the instances using the np.ndarray.view() method:
a = np.random.rand(4,5).view(Ludmo)
And use the __array_finalize__() method to define new attributes:
def __array_finalize__(self, arr):
self.foo = 'foo'
Since you ask about a generic solution, here's a generic wrapper class that you can use: (from http://code.activestate.com/recipes/577555-object-wrapper-class/ )
class Wrapper(object):
'''
Object wrapper class.
This a wrapper for objects. It is initialiesed with the object to wrap
and then proxies the unhandled getattribute methods to it.
Other classes are to inherit from it.
'''
def __init__(self, obj):
'''
Wrapper constructor.
#param obj: object to wrap
'''
# wrap the object
self._wrapped_obj = obj
def __getattr__(self, attr):
# see if this object has attr
# NOTE do not use hasattr, it goes into
# infinite recurrsion
if attr in self.__dict__:
# this object has it
return getattr(self, attr)
# proxy to the wrapped object
return getattr(self._wrapped_obj, attr)
the way this works is:
when e.g. skicit would call ludmo.data python actually calls
ludmo.__getattribute__('data')
if ludmo doesn't have the 'data' attribute, python will call
ludmo.__getattr__('data')
by overridding the __getattr__ function you intercept this call, check if your ludmo has the data attribute (again, you could get into recursion otherwise), and send the call to your internal object. So you should have covered every possible call to your internal numpy object.
update:
You would also have to implement __setattr__ the same way, or you would get this
>>> class bla(object):
... def __init__(self):
... self.a = 1
... def foo(self):
... print self.a
...
>>> d = Wrapper(bla())
>>> d.a
1
>>> d.foo()
1
>>> d.a = 2
>>> d.a
2
>>> d.foo()
1
and you probably also want to set a new metaclass that intercepts calls to magic functions of new style classes (for full class see https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/wrapper.py
for info see How can I intercept calls to python's "magic" methods in new style classes?
)
however, this is only needed if you still want to be able to access x.__name__ or x.__file__ and get the magic attribute from the wrapped class, and not your class.
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
Related
Is there some more generic and in general best practice to deal with a problem when numpy changing datatype results in loss of instance attributes?
(MyCustomClass inherits from np.ndarray)
Problem:
array = MyCustomClass(shape=[4,4,4],dtype=np.float16)
array.variable = 5
array_uint8 = np.uint8(array)
print(array_uint8.variable)
>> AttributeError: 'MyCustomClass' object has no attribute 'variable'
Note, that changing datatype doesnt result in changing array's class, which is MyCustomClass
My very bad solution
class uint8(np.uint8):
def __new__(cls,*args, **kwargs):
instance = super().__new__(cls, *args, **kwargs)
print('instance class', instance.__class__)
# Moving attributes from one instance to another
instance.__dict__ = args[0].__dict__ # args[0] is MyCustomClass instance
return instance
>>> instance class <class '__main__.MyCustomClass'>
I would prefer a solution that would work in such a way, that I wouldn't have to modify np.uint8 (or other numpy classes) at all.
Thanks to #kwinkunks I was able to find solution.
As #kwinkunks pointed out, it is necessary to define __array_finalize__.
Inside it, it is possible to exchange __dict__'s between obj's -> self represents new array and obj represents old one
class MyCustomClass(np.ndarray):
def __array_finalize__(self, obj):
if isinstance(obj, (MyCustomClass,)):
self.__dict__ = obj.__dict__
I tried writing a decorator as such (going off memory, excuse any problems in code):
def required(fn):
def wrapped(self):
self.required_attributes += [fn.__name__]
fn(self)
return wrapped
and I used this to decorate #property attributes in classes, e.g.:
#property
#required
def some_property(self):
return self._some_property
...so that I could do something like this:
def validate_required_attributes(instance):
for attribute in instance.required_attributes:
if not hasattr(instance, attribute):
raise ValueError(f"Required attribute {attribute} was not set!")
Now I forgot that this wouldn't work because in order for the required_attributes to be updated with the name of the property, I would have to retrieve the property first. So in essence, when I do init in the class, I can just do a self.propertyname to add it... but this solution is not nice at all, I might as well create a list of required attribute names in the init.
From what I know, the decorator is applied at compile time so I wouldn't be able to modify the required_attributes before defining the wrapped function. Is there another way I can make this work? I just want a nice, elegant solution.
Thanks!
I think the attrs library does what you want. You can define a class like this, where x and y are required and z is optional.
from attr import attrs, attrib
#attrs
class MyClass:
x = attrib()
y = attrib()
z = attrib(default=0)
Testing it out:
>>> instance = MyClass(1, 2)
>>> print(instance)
MyClass(x=1, y=2, z=0)
Here's my take at doing it with a class decorator and a method decorator. There's probably a nicer way of doing this using metaclasses (nice being the API not the implementation ;)).
def requiredproperty(f):
setattr(f, "_required", True)
return property(f)
def hasrequiredprops(cls):
props = [x for x in cls.__dict__.items() if isinstance(x[1], property)]
cls._required_props = {k for k, v in props if v.fget._required}
return cls
#hasrequiredprops
class A(object):
def __init__(self):
self._my_prop = 1
def validate(self):
print("required attributes are", ",".join(self._required_props))
#requiredproperty
def my_prop(self):
return self._my_prop
This should make validation work without the requiring the caller to touch the property first:
>>> a = A()
>>> a.validate()
required attributes are my_prop
>>> a.my_prop
1
The class decorator is required to make sure it has the required property names duing instantiation. The requiredproperty function is just a way to mark the properties as required.
That being said, I'm not completely sure what you are trying to achieve here. Perhaps validation of the instance attribute values that the property should return?
Is there a way to make a Python #property act as a setter and getter all at once?
I feel like I've seen this somewhere before but can't remember and can't recreate the solution myself.
For example, instead of:
class A(object):
def __init__(self, b): self.b = b
def get_c(self): return self.b.c
def set_c(self, value): self.b.c = value
c = property(get_c, set_c)
we could somehow signal that for A objects, the c attribute is really equivalent to b.c for getter, setter (and deleter if we like).
Motivation:
This would be particularly useful when we need A to be a proxy wrapper around B objects (of which b is an instance) but share only the data attributes and no methods. Properties such as these would allow the A and B objects' data to stay completely in sync while both are used by the same code.
I think you are looking for this forwardTo class as posted on ActiveState.
This recipe lets you transparently forward attribute access to another
object in your class. This way, you can expose functionality from some
member of your class instance directly, e.g. foo.baz() instead of
foo.bar.baz().
class forwardTo(object):
"""
A descriptor based recipe that makes it possible to write shorthands
that forward attribute access from one object onto another.
>>> class C(object):
... def __init__(self):
... class CC(object):
... def xx(self, extra):
... return 100 + extra
... foo = 42
... self.cc = CC()
...
... localcc = forwardTo('cc', 'xx')
... localfoo = forwardTo('cc', 'foo')
...
>>> print C().localcc(10)
110
>>> print C().localfoo
42
Arguments: objectName - name of the attribute containing the second object.
attrName - name of the attribute in the second object.
Returns: An object that will forward any calls as described above.
"""
def __init__(self, objectName, attrName):
self.objectName = objectName
self.attrName = attrName
def __get__(self, instance, owner=None):
return getattr(getattr(instance, self.objectName), self.attrName)
def __set__(self, instance, value):
setattr(getattr(instance, self.objectName), self.attrName, value)
def __delete__(self, instance):
delattr(getattr(instance, self.objectName), self.attrName)
For a more robust code, you may want to consider replacing getattr(instance, self.objectName) with operator.attrgetter(self.objectName)(instance). This would allow objectName to be a dotted name (e.g., so you could have A.c be a proxy for A.x.y.z.d).
If you're trying to delegate a whole slew of properties from any A object to its b member, it's probably easier to do that inside __getattr__, __setattr__, and __delattr__, e.g.:
class A(object):
delegated = ['c', 'd', 'e', 'f']
def __getattr__(self, attr):
if attr in A.delegated:
return getattr(self.b, attr)
raise AttributeError()
I haven't shown the __setattr__ and __delattr__ definitions here, for brevity, and to avoid having to explain the difference between __getattr__ and __getattribute__. See the docs if you need more information.
This is readily extensible to classes that want to proxy different attributes to different members:
class A(object):
b_delegated = ['c', 'd', 'e', 'f']
x_delegated = ['y', 'z']
def __getattr__(self, attr):
if attr in A.b_delegated:
return getattr(self.b, attr)
elif attr in A.x_delegated:
return getattr(self.x, attr)
else:
raise AttributeError()
If you need to delegate all attributes, dynamically, that's almost as easy. You just get a list of self.b's attributes (or self.b.__class__'s) at init time or at call time (which of the four possibilities depends on exactly what you want to do), and use that in place of the static list b_delegated.
You can of course filter this by name (e.g., to remove _private methods), or by type, or any arbitrary predicate (e.g., to remove any callable attributes).
Or combine any of the above.
At any rate, this is the idiomatic way to do (especially dynamic) proxying in Python. It's not perfect, but trying to invent a different mechanism is probably not a good idea.
And in fact, it's not really meant to be perfect. This is something you shouldn't be doing too often, and shouldn't be trying to disguise when you do it. It's obvious that a ctypes.cdll or a pyobjc module is actually delegating to something else, because it's actually useful for the user to know that. If you really need to delegate most of the public interface of one class to another, and don't want the user to know about the delegation… maybe you don't need it. Maybe it's better to just expose the private object directly, or reorganize your object model so the user is interacting with the right things in the first place.
There's the decorator syntax for creating properties, then there are full blown custom-defined descriptors, but since the setter/getter pseudo-private pattern is actively discouraged in Python and the Python community, there isn't really a widely distributed or commonly used way to do what you are looking for.
For proxy objects, you can use __getattr__, __setattr__, and __getattribute__, or try to manipulate things earlier in the process by fooling around with __new__ or a metaclass.
def make_property(parent, attr):
def get(self):
return getattr(getattr(self, parent), attr)
def set(self, value):
setattr(getattr(self, parent), attr, value)
return property(get, set)
class A(object):
def __init__(self, b): self.b = b
c = make_property('b', 'c')
Here's another way of doing it, statically forwarding properties from one object to another, but with economy.
It allows to forward get/set property in two lines, and aread-only property in one line, making use of dynamic property definition at the class level and lambdas.
class A:
"""Classic definition of property, with decorator"""
_id = ""
_answer = 42
#property
def id(self):
return self._id
#id.setter
def id(self, value):
self._id = value
#property
def what(self):
return self._answer
class B:
obj = A()
# Forward "id" from self.obj
id = property(lambda self: self.obj.id,
lambda self, value: setattr(self.obj, "id", value))
# Forward read-only property from self.obj
what = property(lambda self: self.obj.what)
According to the docs, super(cls, obj) returns
a proxy object that delegates method calls to a parent or sibling
class of type cls
I understand why super() offers this functionality, but I need something slightly different: I need to create a proxy object that delegates methods calls (and attribute lookups) to class cls itself; and as in super, if cls doesn't implement the method/attribute, my proxy should continue looking in the MRO order (of the new not the original class). Is there any function I can write that achieves that?
Example:
class X:
def act():
#...
class Y:
def act():
#...
class A(X, Y):
def act():
#...
class B(X, Y):
def act():
#...
class C(A, B):
def act():
#...
c = C()
b = some_magic_function(B, c)
# `b` needs to delegate calls to `act` to B, and look up attribute `s` in B
# I will pass `b` somewhere else, and have no control over it
Of course, I could do b = super(A, c), but that relies on knowing the exact class hierarchy and the fact that B follows A in the MRO. It would silently break if any of these two assumptions change in the future. (Note that super doesn't make any such assumptions!)
If I just needed to call b.act(), I could use B.act(c). But I am passing b to someone else, and have no idea what they'll do with it. I need to make sure it doesn't betray me and start acting like an instance of class C at some point.
A separate question, the documentation for super() (in Python 3.2) only talks about its method delegation, and does not clarify that attribute lookups for the proxy are also performed the same way. Is it an accidental omission?
EDIT
The updated Delegate approach works in the following example as well:
class A:
def f(self):
print('A.f')
def h(self):
print('A.h')
self.f()
class B(A):
def g(self):
self.f()
print('B.g')
def f(self):
print('B.f')
def t(self):
super().h()
a_true = A()
# instance of A ends up executing A.f
a_true.h()
b = B()
a_proxy = Delegate(A, b)
# *unlike* super(), the updated `Delegate` implementation would call A.f, not B.f
a_proxy.h()
Note that the updated class Delegate is closer to what I want than super() for two reasons:
super() only does it proxying for the first call; subsequent calls will happen as normal, since by then the object is used, not its proxy.
super() does not allow attribute access.
Thus, my question as asked has a (nearly) perfect answer in Python.
It turns out that, at a higher level, I was trying to do something I shouldn't (see my comments here).
This class should cover the most common cases:
class Delegate:
def __init__(self, cls, obj):
self._delegate_cls = cls
self._delegate_obj = obj
def __getattr__(self, name):
x = getattr(self._delegate_cls, name)
if hasattr(x, "__get__"):
return x.__get__(self._delegate_obj)
return x
Use it like this:
b = Delegate(B, c)
(with the names from your example code.)
Restrictions:
You cannot retrieve some special attributes like __class__ etc. from the class you pass in the constructor via this proxy. (This restistions also applies to super.)
This might behave weired if the attribute you want to retrieve is some weired kind of descriptor.
Edit: If you want the code in the update to your question to work as desired, you can use the foloowing code:
class Delegate:
def __init__(self, cls):
self._delegate_cls = cls
def __getattr__(self, name):
x = getattr(self._delegate_cls, name)
if hasattr(x, "__get__"):
return x.__get__(self)
return x
This passes the proxy object as self parameter to any called method, and it doesn't need the original object at all, hence I deleted it from the constructor.
If you also want instance attributes to be accessible you can use this version:
class Delegate:
def __init__(self, cls, obj):
self._delegate_cls = cls
self._delegate_obj = obj
def __getattr__(self, name):
if name in vars(self._delegate_obj):
return getattr(self._delegate_obj, name)
x = getattr(self._delegate_cls, name)
if hasattr(x, "__get__"):
return x.__get__(self)
return x
A separate question, the documentation for super() (in Python 3.2)
only talks about its method delegation, and does not clarify that
attribute lookups for the proxy are also performed the same way. Is it
an accidental omission?
No, this is not accidental. super() does nothing for attribute lookups. The reason is that attributes on an instance are not associated with a particular class, they're just there. Consider the following:
class A:
def __init__(self):
self.foo = 'foo set from A'
class B(A):
def __init__(self):
super().__init__()
self.bar = 'bar set from B'
class C(B):
def method(self):
self.baz = 'baz set from C'
class D(C):
def __init__(self):
super().__init__()
self.foo = 'foo set from D'
self.baz = 'baz set from D'
instance = D()
instance.method()
instance.bar = 'not set from a class at all'
Which class "owns" foo, bar, and baz?
If I wanted to view instance as an instance of C, should it have a baz attribute before method is called? How about afterwards?
If I view instance as an instance of A, what value should foo have? Should bar be invisible because was only added in B, or visible because it was set to a value outside the class?
All of these questions are nonsense in Python. There's no possible way to design a system with the semantics of Python that could give sensible answers to them. __init__ isn't even special in terms of adding attributes to instances of the class; it's just a perfectly ordinary method that happens to be called as part of the instance creation protocol. Any method (or indeed code from another class altogether, or not from any class at all) can create attributes on any instance it has a reference to.
In fact, all of the attributes of instance are stored in the same place:
>>> instance.__dict__
{'baz': 'baz set from C', 'foo': 'foo set from D', 'bar': 'not set from a class at all'}
There's no way to tell which of them were originally set by which class, or were last set by which class, or whatever measure of ownership you want. There's certainly no way to get at "the A.foo being shadowed by D.foo", as you would expect from C++; they're the same attribute, and any writes to to it by one class (or from elsewhere) will clobber a value left in it by the other class.
The consequence of this is that super() does not perform attribute lookups the same way it does method lookups; it can't, and neither can any code you write.
In fact, from running some experiments, neither super nor Sven's Delegate actually support direct attribute retrieval at all!
class A:
def __init__(self):
self.spoon = 1
self.fork = 2
def foo(self):
print('A.foo')
class B(A):
def foo(self):
print('B.foo')
b = B()
d = Delegate(A, b)
s = super(B, b)
Then both work as expected for methods:
>>> d.foo()
A.foo
>>> s.foo()
A.foo
But:
>>> d.fork
Traceback (most recent call last):
File "<pyshell#43>", line 1, in <module>
d.fork
File "/tmp/foo.py", line 6, in __getattr__
x = getattr(self._delegate_cls, name)
AttributeError: type object 'A' has no attribute 'fork'
>>> s.spoon
Traceback (most recent call last):
File "<pyshell#45>", line 1, in <module>
s.spoon
AttributeError: 'super' object has no attribute 'spoon'
So they both only really work for calling some methods on, not for passing to arbitrary third party code to pretend to be an instance of the class you want to delegate to.
They don't behave the same way in the presence of multiple inheritance unfortunately. Given:
class Delegate:
def __init__(self, cls, obj):
self._delegate_cls = cls
self._delegate_obj = obj
def __getattr__(self, name):
x = getattr(self._delegate_cls, name)
if hasattr(x, "__get__"):
return x.__get__(self._delegate_obj)
return x
class A:
def foo(self):
print('A.foo')
class B:
pass
class C(B, A):
def foo(self):
print('C.foo')
c = C()
d = Delegate(B, c)
s = super(C, c)
Then:
>>> d.foo()
Traceback (most recent call last):
File "<pyshell#50>", line 1, in <module>
d.foo()
File "/tmp/foo.py", line 6, in __getattr__
x = getattr(self._delegate_cls, name)
AttributeError: type object 'B' has no attribute 'foo'
>>> s.foo()
A.foo
Because Delegate ignores the full MRO of whatever class _delegate_obj is an instance of, only using the MRO of _delegate_cls. Whereas super does what you asked in the question, but the behaviour seems quite strange: it's not wrapping an instance of C to pretend it's an instance of B, because direct instances of B don't have foo defined.
Here's my attempt:
class MROSkipper:
def __init__(self, cls, obj):
self.__cls = cls
self.__obj = obj
def __getattr__(self, name):
mro = self.__obj.__class__.__mro__
i = mro.index(self.__cls)
if i == 0:
# It's at the front anyway, just behave as getattr
return getattr(self.__obj, name)
else:
# Check __dict__ not getattr, otherwise we'd find methods
# on classes we're trying to skip
try:
return self.__obj.__dict__[name]
except KeyError:
return getattr(super(mro[i - 1], self.__obj), name)
I rely on the __mro__ attribute of classes to properly figure out where to start from, then I just use super. You could walk the MRO chain from that point yourself checking class __dict__s for methods instead if the weirdness of going back one step to use super is too much.
I've made no attempt to handle unusual attributes; those implemented with descriptors (including properties), or those magic methods looked up behind the scenes by Python, which often start at the class rather than the instance directly. But this behaves as you asked moderately well (with the caveat expounded on ad nauseum in the first part of my post; looking up attributes this way will not give you any different results than looking them up directly in the instance).
In Python, is there a way to bind an unbound method without calling it?
I am writing a wxPython program, and for a certain class I decided it'd be nice to group the data of all of my buttons together as a class-level list of tuples, like so:
class MyWidget(wx.Window):
buttons = [("OK", OnOK),
("Cancel", OnCancel)]
# ...
def Setup(self):
for text, handler in MyWidget.buttons:
# This following line is the problem line.
b = wx.Button(parent, label=text).Bind(wx.EVT_BUTTON, handler)
The problem is, since all of the values of handler are unbound methods, my program explodes in a spectacular blaze and I weep.
I was looking around online for a solution to what seems like should be a relatively straightforward, solvable problem. Unfortunately I couldn't find anything. Right now, I'm using functools.partial to work around this, but does anyone know if there's a clean-feeling, healthy, Pythonic way to bind an unbound method to an instance and continue passing it around without calling it?
All functions are also descriptors, so you can bind them by calling their __get__ method:
bound_handler = handler.__get__(self, MyWidget)
Here's R. Hettinger's excellent guide to descriptors.
As a self-contained example pulled from Keith's comment:
def bind(instance, func, as_name=None):
"""
Bind the function *func* to *instance*, with either provided name *as_name*
or the existing name of *func*. The provided *func* should accept the
instance as the first argument, i.e. "self".
"""
if as_name is None:
as_name = func.__name__
bound_method = func.__get__(instance, instance.__class__)
setattr(instance, as_name, bound_method)
return bound_method
class Thing:
def __init__(self, val):
self.val = val
something = Thing(21)
def double(self):
return 2 * self.val
bind(something, double)
something.double() # returns 42
This can be done cleanly with types.MethodType. Example:
import types
def f(self):
print(self)
class C:
pass
meth = types.MethodType(f, C(), C) # Bind f to an instance of C
print(meth) # prints <bound method C.f of <__main__.C object at 0x01255E90>>
Creating a closure with self in it will not technically bind the function, but it is an alternative way of solving the same (or very similar) underlying problem. Here's a trivial example:
self.method = (lambda self: lambda args: self.do(args))(self)
This will bind self to handler:
bound_handler = lambda *args, **kwargs: handler(self, *args, **kwargs)
This works by passing self as the first argument to the function. object.function() is just syntactic sugar for function(object).
Late to the party, but I came here with a similar question: I have a class method and an instance, and want to apply the instance to the method.
At the risk of oversimplifying the OP's question, I ended up doing something less mysterious that may be useful to others who arrive here (caveat: I'm working in Python 3 -- YMMV).
Consider this simple class:
class Foo(object):
def __init__(self, value):
self._value = value
def value(self):
return self._value
def set_value(self, value):
self._value = value
Here's what you can do with it:
>>> meth = Foo.set_value # the method
>>> a = Foo(12) # a is an instance with value 12
>>> meth(a, 33) # apply instance and method
>>> a.value() # voila - the method was called
33