I was writing a class and at some point I decided that it would be nice to have a possibility to create an instance of the class using other instance of this same class. So basically the class would look like this:
class Test:
def __init__(self, a, b):
if type(b) == Test:
self = a / b
else:
self.a = a
self.b = b
def __rtruediv__(self, other):
return Test(other * self.b, self.a)
def __str__(self):
return f"{self.a = }, {self.b = }"
if __name__ == '__main__':
hf = Test(1, 2)
print(hf) # self.a = 1, self.b = 2
print(Test(1, hf)) # AttributeError: 'Test' object has no attribute 'a'
print(1 / Test(1, hf)) # AttributeError: 'Test' object has no attribute 'b'
However, when i tried to do it, I got the AttributeError (AttributeError: 'Test' object has no attribute 'a'). Interestingly enough, the code print(1/Test(1, hf)) gives the same attribute error about attribute b while going into the rtruediv func, so the object Test(1, hf) has methods that I defined. Why does that happen? Is something wrong with "self = ..."?
The reason this doesn't work is that self is just an ordinary variable like any other.
if type(b) == Test:
self = a / b
else:
self.a = a
self.b = b
In the if branch, we reassign a local variable, but it doesn't change the instance being constructed. It just makes a new instance, that will be discarded in a moment. The actual instance being constructed (the original value of self) is still there and remains in an uninitialized state (i.e. doesn't have a and b like it should).
I think your goal here is well-intentioned, though it will likely confuse Python programmers as it's not incredibly idiomatic. My recommendation is to simply set self.a and self.b in the constructor in all cases, as it's a more intuitive code design for the average Python coder.
However, what you want can be done. In Python, when you call Test(a, b), some complicated internal things happen.
First, we invoke __call__ on the metaclass of Test. In your example, the metaclass of Test isn't specified, so it defaults to [type], the built-in metaclass.
The default behavior of type.__call__ is to invoke the class method __new__ on your class (__new__ is implicitly a class method, even if you don't ask it to be, so you don't need the #classmethod decorator). Your class doesn't currently define __new__, so we inherit the default one from object.
object.__new__ actually creates the new object. This is written in low-level C and can't be replicated in Python. It truly is primitive.
Finally, object.__new__ calls Test.__init__ on the newly-constructed object and then returns the new object.
Aside from (3), which is truly a C-level primitive, we can easily inject into any part of this. We could write a metaclass and redefine __call__, but that's overkill for what we're doing. We're going to hook into (2) and define __new__. So rather than your __init__, consider
class Test:
def __new__(cls, a, b):
if type(b) == Test:
return a / b
else:
obj = object.__new__(cls)
obj.a = a
obj.b = b
return obj
In __init__, we're given the object to construct, and our only option is to construct that object (or throw an exception if we can't). But __new__ is lower-level. In __new__, it's our choice whether and how to create a new object. In the case where we're given two numbers, we delegate to the default __new__ on object (creating a new object, using the primitive C-level code discussed above), and then we initialize its instance variables. But in the case where b is a Test, we do something different and short-circuit the process entirely, calling a different method instead.
Again, I don't necessarily think this is super idiomatic Python code, but it is possible to do what you want.
Related
I find a good desc for python property in this link
How does the #property decorator work in Python?
below example shows how it works, while I find an exception for class attr 'name'
now I have a reload function which will raise an error
#property
def foo(self): return self._foo
really means the same thing as
def foo(self): return self._foo
foo = property(foo)
here is my example
class A(object):
#property
def __name__(self):
return 'dd'
a = A()
print(a.__name__)
dd
this works, however below cannot work
class B(object):
pass
def test(self):
return 'test'
B.t = property(test)
print(B.t)
B.__name__ = property(test)
<property object at 0x7f71dc5e1180>
Traceback (most recent call last):
File "<string>", line 23, in <module>
TypeError: can only assign string to B.__name__, not 'property'
Does anyone knows the difference for builtin name attr, it works if I use normal property decorator, while not works for the 2nd way. now I have a requirement to reload the function when code changes, however this error will block the reload procedure. Can anyone helps? thanks.
The short answer is: __name__ is deep magic in CPython.
So, first, let's get the technicalities out of the way. To quote what you said
#property
def foo(self): return self._foo
really means the same thing as
def foo(self): return self._foo
foo = property(foo)
This is correct. But it can be a bit misleading. You have this A class
class A(object):
#property
def __name__(self):
return 'dd'
And you claim that it's equivalent to this B class
class B(object):
pass
def test(self):
return 'test'
B.__name__ = property(test)
which is not correct. It's actually equivalent to this
def test(self):
return 'test'
class B(object):
__name__ = property(test)
which works and does what you expect it to. And you're also correct that, for most names in Python, your B and my B would be the same. What difference does it make whether I'm assigning to a name inside the class or immediately after its declaration? Replace __name__ with ravioli in the above snippets and either will work. So what makes __name__ special?
That's where the magic comes in. When you define a name inside the class, you're working directly on the class' internal dictionary, so
class A:
foo = 1
def bar(self):
return 1
This defines two things on the class A. One happens to be a number and the other happens to be a function (which will likely be called as a bound method). Now we can access these.
A.foo # Returns 1, simple access
A.bar # Returns the function object bar
A().foo # Returns 1
A().bar # Returns a bound method object
When we look up the names directly on A, we simply access the slots like we would on any object. However, when we look them up on A() (an instance of A), a multi-step process happens
Look up the name on the instance's __dict__ directly.
If that failed, then look up the name on the class' __dict__.
If we found it on the class, see if there's a __get__ on the result and call it.
That third step is what allows bound method objects to work, and it's also the mechanism underlying the property decorators in Python.
Let's go through this whole process with a property called ravioli. No magic here.
class A(object):
#property
def ravioli(self):
return 'dd'
When we do A().ravioli, first we see if there's a ravioli on the instance we just made. There isn't, so we check the class' __dict__, and indeed we find a property object at that position. That property object has a __get__, so we call it, and it returns 'dd', so indeed we get the string 'dd'.
>>> A().ravioli
'dd'
Now I would expect that, if I do A.ravioli, we will simply get the property object. Since we're not calling it on an instance, we don't call __get__.
>>> A.ravioli
<property object at 0x7f5bd3690770>
And indeed, we get the property object, as expected.
Now let's do the exact same thing but replace ravioli with __name__.
class A(object):
#property
def __name__(self):
return 'dd'
Great! Now let's make an instance.
>>> A().__name__
'dd'
Sensible, we looked up __name__ on A's __dict__ and found a property, so we called its __get__. Nothing weird.
Now
>>> A.__name__
'A'
Um... what? If we had just found the property on A's __dict__, then we should see that property here, right?
Well, no, not always. See, in the abstract, foo.bar normally looks in foo.__dict__ for a field called bar. But it doesn't do that if the type of foo defines a __getattribute__. If it defines that, then that method is always called instead.
Now, the type of A is type, the type of all Python types. Read that sentence a few times and make sure it makes sense. And if we do a bit of spelunking into the CPython source code, we see that type actually defines __getattribute__ and __setattr__ for the following names:
__name__
__qualname__
__bases__
__module__
__abstractmethods__
__dict__
__doc__
__text_signature__
__annotations__
That explains how __name__ can serve double duty as a property on the class instances and also as an accessible field on the same class. It also explains why you get that highly specialized error message when reassigning to B.__name__: the line
B.__name__ = property(test)
is actually equivalent to
type.__setattr__(B, '__name__', property(test))
which is calling our special-case checker in CPython.
For any other type in Python, in particular for user-defined types, we could get around this with object.__setattr__. Unfortunately,
>>> object.__setattr__(B, '__name__', property(test))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't apply this __setattr__ to type object
There's a really specific check to make sure we don't do exactly this, and the comment reads
/* Reject calls that jump over intermediate C-level overrides. */
We also can't use metaclasses to override __setattr__ and __getattribute__, because the instance lookup procedure specifically doesn't call those (in the above examples, __getattribute__ was called in every case except the one we care about for property purposes). I even tried subclassing str to trick __setattr__ into accepting our made-up value
class NameProperty(str):
def __new__(cls, value, **kwargs):
return str.__new__(cls, value)
def __init__(self, value, method):
self.method = method
def __get__(self, instance, owner):
return self.method(instance)
B.__name__ = NameProperty(B.__name__, method=test)
This actually passes the __setattr__ check, but it doesn't assign to B.__dict__ (since the __setattr__ still assigns to the actual CPython-level name, not to B.__dict__['__name__']), so the property lookup doesn't work.
So... that's how I reached my conclusion of: __name__ is deep magic in CPython. All of the usual Python metaprogramming techniques have failed, and all of the methods getting called are written deep down in C. My advice to you is: Stop using __name__ for things it's not intended for, or be prepared to write some C code and hack on CPython directly.
I want to call inspect.signature within the __new__ of a python metaclass, before calling super().__new__. This seems to be working fine with non-static methods but raises errors on static methods. Is this intended behaviour? Is there a way to work around it?
For example, this code:
import inspect
class M(type):
def __new__(mcs, name, bases, namespace, **kwargs):
result = super().__new__(mcs, name, bases, namespace, **kwargs)
print(name, 'dot', repr(inspect.signature(result.x)))
print(name, 'get', repr(inspect.signature(namespace['x'])))
return result
class C1(metaclass=M):
def x(self, a: int):
pass
class C2(metaclass=M):
#staticmethod
def x(self, a: int):
pass
Produces this error:
C1 dot <Signature (self, a: int)>
C1 get <Signature (self, a: int)>
C2 dot <Signature (self, a: int)>
TypeError: <staticmethod object at 0x0000014ED0693940> is not a callable object
When attempting to get the signature of the static method before calling __new__, there is a type error.
This seems to work fine on non-static methods, but raises errors on static methods. Is this intended behavior?
The fact that staticmethods are not callable is indeed intended - the staticmethod object is a descriptor whose __get__ method just returns the function it decorates.
As to why it works this way, this is because of how "methods" are implemented in Python, which forces functions intended to be staticmethod to be wrapped to prevent the function's own descriptors protocol to be invoked at lookup time. It would of course have been possible to make staticmethod objects callable (just like methods and classmethods), but this would have implied a supplementary, technically useless, function call (and function calls don't come for free...).
Is there a way to work around it?
There are a couple indeed. You can force the invocation of the protocol descriptor either implicitely by triggering attribute resolution - which is actually what you're doing with result.x - or manually using :
x = namespace["x"]
x = x.__get__(result)
Or you can (as answered by Craig Gidney) do some typechecking and directly retrieve the .__func__ attribute of the staticmethod.
I'd personnaly advise to stick to the first solution if possible as it's the one that's the most generic and doesn't break encapsulation. Note that if your issue to find out which names are defined in the class, you can still use the namespace dict's keys together with getattr:
for name in namespace:
val = getattr(result, name)
if callable(val):
print("{} is a callable".format(val))
One workaround is to manually check for and unwrap staticmethod, like this:
x = namespace['x']
if isinstance(x, staticmethod):
x = x.__func__
print(inspect.signature(x))
Why doesn't Python have an instancemethod function analogous to staticmethod and classmethod?
Here is how this arose for me. Suppose I have an object which I know will be hashed frequently and whose hash is expensive to calculate. Under this assumption, it is reasonable to compute the hash value once and cache it, as in the following toy example:
class A:
def __init__(self, x):
self.x = x
self._hash_cache = hash(self.x)
def __hash__(self):
return self._hash_cache
The __hash__ function in this class does very little, just an attribute lookup and a return. Naively, it seems it ought to be equivalent to instead write:
class B:
def __init__(self, x):
self.x = x
self._hash_cache = hash(self.x)
__hash__ = operator.attrgetter('_hash_cache')
According to the documentation, operator.attrgetter returns a callable object that fetches the given attribute from its operand. If its operand is self, then it will return self._hash_cache, which is the desired result. Unfortunately this does not work:
>>> hash(A(1))
1
>>> hash(B(1))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: attrgetter expected 1 arguments, got 0
The reason for this is as follows. If one reads the descriptor HOWTO, one finds that class dictionaries store methods as functions; functions are non-data descriptors whose __get__ method returns a bound method. But operator.attrgetter does not return a function; it returns a callable object. And in fact, it is a callable object with no __get__ method:
>>> hasattr(operator.attrgetter('_hash_cache'), '__get__')
False
Lacking a __get__ method, this of course will not automatically be turned into a bound method. We can make a bound method from it using types.MethodType, but using it in our class B would require creating a bound method for every object instance and assigning it to __hash__.
We can see the fact that operator.attrgetter has no __get__ directly if we browse the CPython source. I'm not very familiar with the CPython API, but I believe that what's going on is as follows. The definition of the attrgetter_type is in Modules/_operator.c, at line 1439 as I write this. This type sets tp_descr_get to 0. And according to the type object documentation, that means an object whose type is attrgetter_type will not have a __get__.
Of course, if we give ourselves a __get__ method, then everything works. This is the case in the first example above, where __hash__ is actually a function and not just a callable. It's also true in some other cases. For example, if we want to lookup a class attribute, we could write the following:
class C:
y = 'spam'
get_y = classmethod(operator.attrgetter('y'))
As written this is terribly un-Pythonic (though it might be defensible if there were a strange custom __getattr__ for which we wanted to provide convenience functions). But at least it gives the desired result:
>>> C.get_y()
'spam'
I can't think of any reason why it would be bad for attrgetter_type to implement __get__. But on the other hand, even if it did, there would be other situations where we run into trouble. For example, suppose we have a class whose instances are callable:
class D:
def __call__(self, other):
...
We can't use an instance of this class as a class attribute and expect instance lookups to generate bound methods. For instance,
d = D()
class E:
apply_d = d
When D.__call__ is called, it will receive self but not other, and that generates a TypeError. This example might be a little far-fetched, but I'd be a little surprised if nobody had ever encountered something like this in practice. It could be fixed by giving D a __get__ method; but if D is from a third-party library that could be inconvenient.
It seems that the easiest solution would be to have an instancemethod function. Then we could write __hash__ = instancemethod(operator.attrgetter('_hash_cache')) and apply_d = instancemethod(d) and they would both work as intended. Yet, as far as I know, no such function exists. Hence my question: Why is there no instancemethod function?
EDIT: Just to be clear, the functionality of instancemethod would be equivalent to:
def instancemethod(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
This could be applied as in the original question above. One could also imagine writing a class decorator that could be applied to D that would give it a __get__ method; but this code doesn't do this.
So I'm not talking about adding a new feature to Python. Really the question is one of language design: Why not provide it as, say, functools.instancemethod? If the answer is simply, "The use cases are so obscure that nobody's bothered," that's okay. But I would be happy to learn about other reasons, if there are any.
There is no instancemethod decorator because this is the default behaviour for functions declared inside a class.
class A:
...
# This is an instance method
def __hash__(self):
return self._hash_cache
Any callable which does not have a __get__ method can thus be wrapped into an instance method like so.
class A:
def instance_method(*args):
return any_callable(*args)
Thus creating an instancemethod decorator would just add another syntax for a feature which already exists. This would go against the saying that there should be one-- and preferably only one --obvious way to do it.
Side note
If it is so expensive to hash your instances, you might want to avoid calling you hash function on instantiation and delay it for when the object are hashed.
One way to do that could be to set the attribute _hash_cache in __hash__ instead of __init__. Although, let me suggest a slightly more self-contained methods which relies on caching your hash.
from weakref import finalize
class CachedHash:
def __init__(self, x):
self.x = x
def __hash__(self, _cache={}):
if id(self) not in _cache:
finalize(self, _cache.pop, id(self))
_cache[id(self)] = hash(self.x) # or some complex hash function
return _cache[id(self)]
The use of finalize ensures the cache is cleared of an id when its instance is garbage collected.
I have a satisfying answer to my question. Python does have the internal interface necessary for an instancemethod function, but it's not exposed by default.
import ctypes
import operator
instancemethod = ctypes.pythonapi.PyInstanceMethod_New
instancemethod.argtypes = (ctypes.py_object,)
instancemethod.restype = ctypes.py_object
class A:
def __init__(self, x):
self.x = x
self._hash_cache = hash(x)
__hash__ = instancemethod(operator.attrgetter('_hash_cache'))
a = A(1)
print(hash(a))
The instancemethod function this creates works in essentially the same way as classmethod and staticmethod. These three functions return new objects of types instancemethod, classmethod, and staticmethod, respectively. We can see how they work by looking at Objects/funcobject.c. These objects all have __func__ members which store a callable object. They also have a __get__. For a staticmethod object, the __get__ returns __func__ unchanged. For a classmethod object, __get__ returns a bound method object, where the binding is to the class object. And for a staticmethod object, __get__ returns a bound method object, where the binding is to the object instance. This is precisely the same behavior as __get__ for a function object and is exactly what we want.
The only documentation on these objects seems to be in the Python C API here. My guess is that they're not exposed because they're so rarely needed. I think it would be nice to have PyInstanceMethod_New available as functools.instancemethod.
In the code below, class A has a member function which is set to point to a function defined outside the class.
in class B, the same function is set to the external pointer in the class definition.
Calling the function for an object of type A will fail, because the self does not get passed to the function. But for B, the self gets passed.
Why does the self get passed for B, but not for A?
def f1(s,a):
print s
print a
class A(object):
def __init__(self):
self.fp1 = f1
class B(object):
fp1 = f1
a=A()
b=B()
try:a.fp1("blaa")
except Exception, e: print `e`
try:b.fp1("bluu")
except Exception, e: print `e`
Output:
TypeError('f1() takes exactly 2 arguments (1 given)',)
<__main__.B object at 0x2ab0dacabed0>
bluu
When you did self.fp1 = f1 you just assigned a function to an instance variable of the class A. So when you call it you have to pass two arguments.
When you did:
class B(object):
fp1 = f1
during creation process of the class B python found a function fp1 in the class scope and created an instancemethod from it (replaced the variable with name fp1 with an instancemethod created from the function that it held before). When you call an instancemethod on an object self gets automatically passed as the first argument.
You can check this by typing:
>>> a = A()
>>> b = B()
>>> type(a.fp1)
function
>>> type(b.fp1)
instancemethod
In class A you bind a function to an instance. This could be really considered as "function pointer" so all arguments must be passed explicitly. In class B you bind the function to the class which will cause the function to work as method. You could modify class definition A to
class A(object):
def __init__(self):
A.fp1 = f1
which will give the same behavior has class B, i.e. fp1 of all instances points to f1, or you could wrap f1.
class A(object):
def __init__(self):
self.fp1 = lambda a: f1(self, a)
This will allow to change fp1 for each instance individually. The latter variant is probably what you were looking for.
The magic that makes instance.method(...) equivalent to Class.method(instance, ...) is dependent on the function object being an attribute of the class. The details vary (and with them, the ugly workaround by which you can create such a method). In Python 3, all functions are descriptors. In Python 2, there are special unbound method objects which are implicitly created to wrap functions stored as class attributes and do roughly what all functions do by themselves in Python 3.
In either case, accessing it through an instance creates a bound method which passes the instance along as first argument when called. In either case, a function accessed through an instance attribute is in no way special, it's just another object which can be passed around and used.
You can achieve similar behavior by either using partial (a bit of a leaky abstraction):
from functools import partial
# in __init__
self.fp1 = partial(f1, self)
or by creating a method which delegates:
def __init__(self):
self._fp1 = f1
def fp1(*args, **kwds):
return self._fp1(self, *args, **kwds)
In the first case you create a field in the class, that has a method object stored in it. So "a.fp1" is not a method call and therefore "a" is not put as the first argument. It's a retrieval of a method object, and then calling it.
For the second case, you can refer to the documentation:
Any function object that is a class attribute defines a method for
instances of that class.
So, for b "fp1" becomes a method for instances of class b.
You can find more detailed explanation here: method objects vs function objects , Python class instances vs class
Please look at the simple example:
class A:
def __init__(self, flag):
self.flag = flag
def func(self):
print self.flag
a = A(1)
b = A(2)
callback_a = a.func
callback_b = b.func
callback_a()
callback_b()
The result is:
1
2
It runs as expected. But I have a question. In C, the callback function is passed as a pointer. In Python, it should have a similar way to do this, so the caller knows the address of the function. But in my example, not only the function pointer is passed, but also the parameter (self) is passed, because the same method of the same class prints different results. So my questions are:
Does such a method in Python only has one copy in memory? My meaning is that the code of any method only has one copy, and in my example the method won't be cloned itself. I think it should have only one copy, but here I still make this question in order to get more inputs.
I remember everything in Python is an object. So in my example, are there two function instances with different parameters but only one copy of code?
In Python, the callback is not simply a reference to a member function. Instead, it is "bound" to the object that it refers to when it was created. So a.func creates a callable that is bound to a, and b.func creates a callable that is bound to b.
Python only needs one implementation of func() in memory, but it will probably create one or more "trampoline" functions at runtime to accomplish the binding (I'm not certain of the internal details on this, and it would differ between Python implementations anyway).
If you print id(callback_a) and id(callback_b) you will get different results, showing that they are indeed different callable objects.
Specific to CPython, there is only one copy of the function object. During instance creation, the class wraps the unbound functions in its namespace as bound methods. But they all wrap the same function.
Here's your example expanded to show what's going on.
class A(object):
def __init__(self, flag):
self.flag = flag
def func(self):
print self.flag
a = A(1)
b = A(2)
callback_a = a.func
callback_b = b.func
print "typeof(callback_a) = {0}".format(type(callback_a))
print "typeof(callback_b) = {0}".format(type(callback_b))
print "typeof(callback_a.__func__) = {0}".format(type(callback_a.__func__))
print "typeof(callback_b.__func__) = {0}".format(type(callback_b.__func__))
print "'callback_a.__func__ is callback_b.__func__' is {0}".format(callback_a.__func__ is callback_b.__func__)
callback_a()
callback_b()
This code outputs
typeof(callback_a) = <type 'instancemethod'>
typeof(callback_b) = <type 'instancemethod'>
typeof(callback_a.__func__) = <type 'function'>
typeof(callback_b.__func__) = <type 'function'>
'callback_a.__func__ is callback_b.__func__' is True
You can clearly see, using the is operator, that both instancemethod classes are sharing the same function object.