I'm trying to have a class act in every way like a tuple that's an attribute of the class, so len(instance) would be the same as len(instance.tup), instance[3] would return instance.tup[3], etc. Here's the class:
class mytup(object):
def __init__(self, a):
self.tup = tuple(a)
def __getattr__(self, nm):
f = types.MethodType(lambda self:getattr(self.tup, nm)(), self, type(self))
f.__func__.func_name = nm
setattr(self, nm, f)
return f
I can
mt = mytup(range(10))
But if I try to:
In [253]: len(mt)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-253-67688b907b8a> in <module>()
----> 1 len(mt)
TypeError: object of type 'mytup' has no len()
mt does in fact have a __len__ that I can call:
In [254]: mt.__len__
Out[254]: <bound method mytup.__len__ of <__main__.mytup object at 0x2e85150>>
In [255]: mt.__len__()
Out[255]: 10
(I even renamed it __len__). As near as I can tell, this should look just as if I did:
def __len__(self, *a):
return self.tup.__len__(*a)
But python won't let me len(mt) (or mt[2] or mt [1:5] for that matter).
New-style classes look-up "special methods"—those that start and end with two underscore characters—on an instance's class not the instance involved, so when len() is called it tries to call typeof(mt).__len__(). So the proper way to do what you want would be to use one of the Abstract Base Classes for Containers in the collections module (since Python 3.3)
import collections.abc
class MyTuple(collections.abc.Sequence):
def __init__(self, a):
self.tup = tuple(a)
def __len__(self):
return len(self.tup)
def __getitem__(self, index):
return self.tup[index]
mt = MyTuple(range(10))
print(len(mt)) # -> 10
print(mt[4]) # -> 4
The reason this isn't working as you have hoped is because doing:
setattr(self, nm, f)
Is not equivalent to
def __len__(self, *a):
return self.tup.__len__(*a)
In the latter case, your method is a property of the class because it is defined in class scope. It would be the equivlanet of setattr(cls, nm, f). If you check MyTup.__dict__ you will see it there. However, in the former case, __len__ is a property of the instance. So it will be in my_instance.__dict__. len checks the class for a __len__ method, and doesn't find one. Hence the error. Your __getattr__ is never actually called, and even if it were, it wouldn't allow you to use len. You can use an_instanec.__len__ diretly, though.
len does not use __getattr__ to get the __len__ function - it calls __len__ directly.
Calling x.__len__ is like calling getattr(x, '__len__') - which will return the x.__len__ method object.
len works behind the scene, so it can access this method directly, without invoking the __getattr__ helper.
Try to add a print statement in your __getattr__ to see what is printed when calling len (hint: nothing).
Related
I have a python module
helpers.py
def str_to_num(s: str):
'''Converts to int if possible else converts to float if possible
Returns back the string if not possible to convert to a number.
'''
# NOTE: These are not really funcs, but classes.
funcs = [int, float]
for func in funcs:
try:
res = func(s)
break
except ValueError:
continue
else:
res = s
return(res)
I have another module string_number.py
from helpers import str_to_num
class StringNumber:
def __init__(self, s):
self.s = s
str_to_num = str_to_num
#property
def value(self):
return(self.str_to_num(self.s))
def __repr__(self):
return(f'{self.__class__.__name__}({repr(self.s)})')
>>> from string_number import StringNumber
>>> sn = StringNumber(1)
>>> sn.value
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "string_number.py", line 19, in value
return(self.str_to_num(self.s))
TypeError: str_to_num() takes 1 positional argument but 2 were given
However this works when accessing the function from the class:
>>> StringNumber.str_to_num(1)
1
Q.1: Why does the str_to_num attribute require two arguments when accessing it from the instance? Is self being passed to it? If so, why?
Now, I know I can add modify the __init__ method to make it an attribute of the instance
def __init__(self, s):
self.s = s
self.str_to_num = str_to_num
Further, I can resolve this by making a class of Helper functions and then inheriting from it.
from helpers import str_to_num
class Helper:
#staticmethod
def str_to_num(s):
return(str_to_num(s))
class StringNumber(Helper):
def __init__(self, s):
self.s = s
#property
def value(self):
return(self.str_to_num(self.s))
def __repr__(self):
return(f'{self.__class__.__name__}({repr(self.s)})')
Q: 2 Is there a way to make module functions, staticmethods of a class, without using inheritance? Or is this a really bad practice?
Q: 3 Assuming I had a helpers.py module, with a large amount of module functions. To incorporate them as staticmethods into my class, what would be the best way, without making a separate Helper class?
Q.1: Why does the str_to_num attribute require two arguments when accessing it from the instance? Is self being passed to it? If so, why?
You wrote "However this works when accessing the function from the class: StringNumber.str_to_num(1)". It works because you declared your method as a static method by defining it under your class definition.
As contrary to static method, instance method does pass the instance as a first argument when it's called. So when you called instance.str_to_num(1) your str_to_num(s: str) - no matter your type hinted it as a string - received instance as s argument and complained that value 1 hasn't got variable to hold it.
i want the next code to work
class A(object):
def __getattr__(self, item):
print item
return self.item
def x(self):
print 4
a = A()
a.x()
and the output will ber
x
4
i know its not working becuase x is like a static variable and not an instance variable.
I saw this __getattr__ for static/class variables in python and it doesn't seem to work in my case
how can it be done?
thx
There are a couple of obvious problems with your code:
class A(object):
def __getattr__(self, item): # 1
print item
return self.item # 2
def x(self): # 1 again
print 4
__getattr__ will only be invoked if item cannot be found the normal way. For item == 'x', therefore, it is never invoked.
Which is probably just as well, since self.item looks for the attribute item, not the attribute corresponding to whatever is assigned to item. This doesn't exist, so would invoke __getattr__. If you try A().y() you'll get RuntimeError: maximum recursion depth exceeded while calling a Python object.
Instead, I think you want to use __getattribute__, which is always invoked. You need to be careful not to get the same runtime error, though; here I avoid it by calling the superclass implementation of __getattribute__, the naïve way of calling getattr(self, item) would fail:
class A(object):
def __getattribute__(self, item):
print item
return super(A, self).__getattribute__(item)
def x(self):
print 4
Which gives:
>>> A().x()
x
4
>>> A().y()
y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __getattribute__
AttributeError: 'A' object has no attribute 'y'
Note that both __getattr__ and __getattribute__ apply equally to attributes and methods (which are, more or less, just callable attributes).
Original problem description
The problem arises when I implement some machine learning algorithm with numpy. I want some new class ludmo which works the same as numpy.ndarray, but with a few more properties. For example, with a new property ludmo.foo. I've tried several methods below, but none is satisfactory.
1. Wrapper
First I created a wrapper class for numpy.ndarray, as
import numpy as np
class ludmo(object):
def __init__(self)
self.foo = None
self.data = np.array([])
But when I use some function (in scikit-learn which I cannot modify) to manipulate a list of np.ndarray instance, I have to first extract all data field of each ludmo object and collect them into a list. After that the list is sorted and I lost the correspondence between the data and original ludmo objects.
2. Inheritance
Then I tried to make ludmo a subclass of numpy.ndarray, as
import numpy as np
class ludmo(np.ndarray):
def __init__(self, shape, dtype=float, buffer=None, offset=0, strides=None, order=None)
super().__init__(shape, dtype, buffer, offset, strides, order)
self.foo = None
But another problem arises then: the most common way to create a numpy.ndarray object is numpy.array(some_list), which returns a numpy.ndarray object, and I have to convert it to a ludmo object. But till now I found no good way to do this; simply changing the __class__ attribute will result in an error.
I'm new to Python and numpy, so there must be some elegant way that I don't know. Any advice is appreciated.
It's better if anyone can give an generic solution, which not only applies to the numpy.ndarray class but also all kinds of classes.
As explained in the docs you could add your own methods to np.ndarray doing:
import numpy as np
class Ludmo(np.ndarray):
def sumcols(self):
return self.sum(axis=1)
def sumrows(self):
return self.sum(axis=0)
def randomize(self):
self[:] = np.random.rand(*self.shape)
and then creating the instances using the np.ndarray.view() method:
a = np.random.rand(4,5).view(Ludmo)
And use the __array_finalize__() method to define new attributes:
def __array_finalize__(self, arr):
self.foo = 'foo'
Since you ask about a generic solution, here's a generic wrapper class that you can use: (from http://code.activestate.com/recipes/577555-object-wrapper-class/ )
class Wrapper(object):
'''
Object wrapper class.
This a wrapper for objects. It is initialiesed with the object to wrap
and then proxies the unhandled getattribute methods to it.
Other classes are to inherit from it.
'''
def __init__(self, obj):
'''
Wrapper constructor.
#param obj: object to wrap
'''
# wrap the object
self._wrapped_obj = obj
def __getattr__(self, attr):
# see if this object has attr
# NOTE do not use hasattr, it goes into
# infinite recurrsion
if attr in self.__dict__:
# this object has it
return getattr(self, attr)
# proxy to the wrapped object
return getattr(self._wrapped_obj, attr)
the way this works is:
when e.g. skicit would call ludmo.data python actually calls
ludmo.__getattribute__('data')
if ludmo doesn't have the 'data' attribute, python will call
ludmo.__getattr__('data')
by overridding the __getattr__ function you intercept this call, check if your ludmo has the data attribute (again, you could get into recursion otherwise), and send the call to your internal object. So you should have covered every possible call to your internal numpy object.
update:
You would also have to implement __setattr__ the same way, or you would get this
>>> class bla(object):
... def __init__(self):
... self.a = 1
... def foo(self):
... print self.a
...
>>> d = Wrapper(bla())
>>> d.a
1
>>> d.foo()
1
>>> d.a = 2
>>> d.a
2
>>> d.foo()
1
and you probably also want to set a new metaclass that intercepts calls to magic functions of new style classes (for full class see https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/wrapper.py
for info see How can I intercept calls to python's "magic" methods in new style classes?
)
however, this is only needed if you still want to be able to access x.__name__ or x.__file__ and get the magic attribute from the wrapped class, and not your class.
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
I'm trying to intercept calls to python's double underscore magic methods in new style classes. This is a trivial example but it show's the intent:
class ShowMeList(object):
def __init__(self, it):
self._data = list(it)
def __getattr__(self, name):
attr = object.__getattribute__(self._data, name)
if callable(attr):
def wrapper(*a, **kw):
print "before the call"
result = attr(*a, **kw)
print "after the call"
return result
return wrapper
return attr
If I use that proxy object around list I get the expected behavior for non-magic methods but my wrapper function is never called for magic methods.
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
<__main__.ShowMeList object at 0x9640eac>
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
If I don't inherit from object (first line class ShowMeList:) everything works as expected:
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
before the call
after the call
[0, 1, 2, 3, 4, 5, 6, 7]
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
How do I accomplish this intercept with new style classes?
For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.
class Wrapper(object):
"""Wrapper class that provides proxy access to some internal instance."""
__wraps__ = None
__ignore__ = "class mro new init setattr getattr getattribute"
def __init__(self, obj):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
elif isinstance(obj, self.__wraps__):
self._obj = obj
else:
raise ValueError("wrapped object must be of %s" % self.__wraps__)
# provide proxy access to regular attributes of wrapped object
def __getattr__(self, name):
return getattr(self._obj, name)
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
Usage:
class DictWrapper(Wrapper):
__wraps__ = dict
wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))
# make sure it worked....
assert "b" in wrapped_dict # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3) # __eq__
assert "'a': 1" in str(wrapped_dict) # __str__
assert wrapped_dict.__doc__.startswith("dict()") # __doc__
Using __getattr__ and __getattribute__ are the last resources of a class to respond to getting an attribute.
Consider the following:
>>> class C:
x = 1
def __init__(self):
self.y = 2
def __getattr__(self, attr):
print(attr)
>>> c = C()
>>> c.x
1
>>> c.y
2
>>> c.z
z
The __getattr__ method is only called when nothing else works (It will not work on operators, and you can read about that here).
On your example, the __repr__ and many other magic methods are already defined in the object class.
One thing can be done, thought, and it is to define those magic methods and make then call the __getattr__ method. Check this other question by me and its answers (link) to see some code doing that.
As of the answers to Asymmetric behavior for __getattr__, newstyle vs oldstyle classes (see also the Python docs), modifying access to "magic" methods with __getattr__ or __getattribute__ is just not possible with new-style classes. This restriction makes the interpreter much faster.
Cut and copy from the documentation:
For old-style classes, special methods are always looked up in exactly the same way as any other method or attribute.
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
I want to override access to one variable in a class, but return all others normally. How do I accomplish this with __getattribute__?
I tried the following (which should also illustrate what I'm trying to do) but I get a recursion error:
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else:
return self.__dict__[name]
>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp
You get a recursion error because your attempt to access the self.__dict__ attribute inside __getattribute__ invokes your __getattribute__ again. If you use object's __getattribute__ instead, it works:
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else:
return object.__getattribute__(self, name)
This works because object (in this example) is the base class. By calling the base version of __getattribute__ you avoid the recursive hell you were in before.
Ipython output with code in foo.py:
In [1]: from foo import *
In [2]: d = D()
In [3]: d.test
Out[3]: 0.0
In [4]: d.test2
Out[4]: 21
Update:
There's something in the section titled More attribute access for new-style classes in the current documentation, where they recommend doing exactly this to avoid the infinite recursion.
Actually, I believe you want to use the __getattr__ special method instead.
Quote from the Python docs:
__getattr__( self, name)
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.
Note: for this to work, the instance should not have a test attribute, so the line self.test=20 should be removed.
Python language reference:
In order to avoid infinite recursion
in this method, its implementation
should always call the base class
method with the same name to access
any attributes it needs, for example,
object.__getattribute__(self, name).
Meaning:
def __getattribute__(self,name):
...
return self.__dict__[name]
You're calling for an attribute called __dict__. Because it's an attribute, __getattribute__ gets called in search for __dict__ which calls __getattribute__ which calls ... yada yada yada
return object.__getattribute__(self, name)
Using the base classes __getattribute__ helps finding the real attribute.
How is the __getattribute__ method used?
It is called before the normal dotted lookup. If it raises AttributeError, then we call __getattr__.
Use of this method is rather rare. There are only two definitions in the standard library:
$ grep -Erl "def __getattribute__\(self" cpython/Lib | grep -v "/test/"
cpython/Lib/_threading_local.py
cpython/Lib/importlib/util.py
Best Practice
The proper way to programmatically control access to a single attribute is with property. Class D should be written as follows (with the setter and deleter optionally to replicate apparent intended behavior):
class D(object):
def __init__(self):
self.test2=21
#property
def test(self):
return 0.
#test.setter
def test(self, value):
'''dummy function to avoid AttributeError on setting property'''
#test.deleter
def test(self):
'''dummy function to avoid AttributeError on deleting property'''
And usage:
>>> o = D()
>>> o.test
0.0
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
A property is a data descriptor, thus it is the first thing looked for in the normal dotted lookup algorithm.
Options for __getattribute__
You several options if you absolutely need to implement lookup for every attribute via __getattribute__.
raise AttributeError, causing __getattr__ to be called (if implemented)
return something from it by
using super to call the parent (probably object's) implementation
calling __getattr__
implementing your own dotted lookup algorithm somehow
For example:
class NoisyAttributes(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self, name):
print('getting: ' + name)
try:
return super(NoisyAttributes, self).__getattribute__(name)
except AttributeError:
print('oh no, AttributeError caught and reraising')
raise
def __getattr__(self, name):
"""Called if __getattribute__ raises AttributeError"""
return 'close but no ' + name
>>> n = NoisyAttributes()
>>> nfoo = n.foo
getting: foo
oh no, AttributeError caught and reraising
>>> nfoo
'close but no foo'
>>> n.test
getting: test
20
What you originally wanted.
And this example shows how you might do what you originally wanted:
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else:
return super(D, self).__getattribute__(name)
And will behave like this:
>>> o = D()
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
>>> del o.test
Traceback (most recent call last):
File "<pyshell#216>", line 1, in <module>
del o.test
AttributeError: test
Code review
Your code with comments. You have a dotted lookup on self in __getattribute__.
This is why you get a recursion error. You could check if name is "__dict__" and use super to workaround, but that doesn't cover __slots__. I'll leave that as an exercise to the reader.
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else: # v--- Dotted lookup on self in __getattribute__
return self.__dict__[name]
>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp
Are you sure you want to use __getattribute__? What are you actually trying to achieve?
The easiest way to do what you ask is:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
test = 0
or:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
#property
def test(self):
return 0
Edit:
Note that an instance of D would have different values of test in each case. In the first case d.test would be 20, in the second it would be 0. I'll leave it to you to work out why.
Edit2:
Greg pointed out that example 2 will fail because the property is read only and the __init__ method tried to set it to 20. A more complete example for that would be:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
_test = 0
def get_test(self):
return self._test
def set_test(self, value):
self._test = value
test = property(get_test, set_test)
Obviously, as a class this is almost entirely useless, but it gives you an idea to move on from.
Here is a more reliable version:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
def __getattribute__(self, name):
if name == 'test':
return 0.
else:
return super(D, self).__getattribute__(name)
It calls __getattribute__ method from parent class, eventually falling back to object.__getattribute__ method if other ancestors don't override it.