I am unsuccessfully trying to get the magic with-statement methods __enter__ and __exit__ running on class-level:
class Spam():
#classmethod
def __enter__(cls):
return cls
#classmethod
def __exit__(cls, typ, value, tb):
cls.cleanup_stuff()
with Spam:
pass
However, this will result in an AttributeError:
Traceback (most recent call last):
File "./test.py", line 15, in <module>
with Spam:
AttributeError: __exit__
Is it possible to use the __enter__ and __exit__ methods on class-level anyway?
__enter__ and __exit__ are special methods, and as such only work correctly when defined on a object's type, not in it's instance dictionary.
Now Spam is a instance of type, and type(Spam).__enter__ and type(Spam).__exit__ do not exist. Therefore you get an attribute error.
To make this work, the methods would need to be declared on the metaclass of the class you want to use. Example:
class Spam(type):
def __enter__(cls):
print('enter')
return cls
def __exit__(cls, typ, value, tb):
print('exit')
class Eggs(metaclass=Spam):
pass
with Eggs:
pass
Now Eggs is an instance of Spam (type(Eggs) == Spam, and therefore type(Eggs).__enter__ and type(Eggs).__exit__ do exist).
However defining a metaclass just to use an instance of it as a context manager seems a little over the top. The more straight forward solution starting from your example would be to just use
with Spam():
pass
Or if you want to reuse the same instance later:
spam = Spam()
with spam:
pass
It seems that CPython doesn't call a bound method like instance.__exit__, it seeks over instance type, doing something like type(instance).__dict__['__exit__'] than calls it. And since type(Spam) is a special type object (not a Spam itself), it doesn't contain __exit__ method.
I tried to workaround that using metaclasses, but wasn't successful. __getattr__ doesn't work either.
See here: https://github.com/python/cpython/blob/2545fdbd4b4a6a77b132fccf816578f59b609be5/Objects/typeobject.c#L1362
Py_TYPE is similiar to type(self)
_PyType_LookupId walks over type(self).__dict__ (no __getattr__ call here)
Python 2 implementation is different, but main idea about getting type(self) applies to it too
Related
I find a good desc for python property in this link
How does the #property decorator work in Python?
below example shows how it works, while I find an exception for class attr 'name'
now I have a reload function which will raise an error
#property
def foo(self): return self._foo
really means the same thing as
def foo(self): return self._foo
foo = property(foo)
here is my example
class A(object):
#property
def __name__(self):
return 'dd'
a = A()
print(a.__name__)
dd
this works, however below cannot work
class B(object):
pass
def test(self):
return 'test'
B.t = property(test)
print(B.t)
B.__name__ = property(test)
<property object at 0x7f71dc5e1180>
Traceback (most recent call last):
File "<string>", line 23, in <module>
TypeError: can only assign string to B.__name__, not 'property'
Does anyone knows the difference for builtin name attr, it works if I use normal property decorator, while not works for the 2nd way. now I have a requirement to reload the function when code changes, however this error will block the reload procedure. Can anyone helps? thanks.
The short answer is: __name__ is deep magic in CPython.
So, first, let's get the technicalities out of the way. To quote what you said
#property
def foo(self): return self._foo
really means the same thing as
def foo(self): return self._foo
foo = property(foo)
This is correct. But it can be a bit misleading. You have this A class
class A(object):
#property
def __name__(self):
return 'dd'
And you claim that it's equivalent to this B class
class B(object):
pass
def test(self):
return 'test'
B.__name__ = property(test)
which is not correct. It's actually equivalent to this
def test(self):
return 'test'
class B(object):
__name__ = property(test)
which works and does what you expect it to. And you're also correct that, for most names in Python, your B and my B would be the same. What difference does it make whether I'm assigning to a name inside the class or immediately after its declaration? Replace __name__ with ravioli in the above snippets and either will work. So what makes __name__ special?
That's where the magic comes in. When you define a name inside the class, you're working directly on the class' internal dictionary, so
class A:
foo = 1
def bar(self):
return 1
This defines two things on the class A. One happens to be a number and the other happens to be a function (which will likely be called as a bound method). Now we can access these.
A.foo # Returns 1, simple access
A.bar # Returns the function object bar
A().foo # Returns 1
A().bar # Returns a bound method object
When we look up the names directly on A, we simply access the slots like we would on any object. However, when we look them up on A() (an instance of A), a multi-step process happens
Look up the name on the instance's __dict__ directly.
If that failed, then look up the name on the class' __dict__.
If we found it on the class, see if there's a __get__ on the result and call it.
That third step is what allows bound method objects to work, and it's also the mechanism underlying the property decorators in Python.
Let's go through this whole process with a property called ravioli. No magic here.
class A(object):
#property
def ravioli(self):
return 'dd'
When we do A().ravioli, first we see if there's a ravioli on the instance we just made. There isn't, so we check the class' __dict__, and indeed we find a property object at that position. That property object has a __get__, so we call it, and it returns 'dd', so indeed we get the string 'dd'.
>>> A().ravioli
'dd'
Now I would expect that, if I do A.ravioli, we will simply get the property object. Since we're not calling it on an instance, we don't call __get__.
>>> A.ravioli
<property object at 0x7f5bd3690770>
And indeed, we get the property object, as expected.
Now let's do the exact same thing but replace ravioli with __name__.
class A(object):
#property
def __name__(self):
return 'dd'
Great! Now let's make an instance.
>>> A().__name__
'dd'
Sensible, we looked up __name__ on A's __dict__ and found a property, so we called its __get__. Nothing weird.
Now
>>> A.__name__
'A'
Um... what? If we had just found the property on A's __dict__, then we should see that property here, right?
Well, no, not always. See, in the abstract, foo.bar normally looks in foo.__dict__ for a field called bar. But it doesn't do that if the type of foo defines a __getattribute__. If it defines that, then that method is always called instead.
Now, the type of A is type, the type of all Python types. Read that sentence a few times and make sure it makes sense. And if we do a bit of spelunking into the CPython source code, we see that type actually defines __getattribute__ and __setattr__ for the following names:
__name__
__qualname__
__bases__
__module__
__abstractmethods__
__dict__
__doc__
__text_signature__
__annotations__
That explains how __name__ can serve double duty as a property on the class instances and also as an accessible field on the same class. It also explains why you get that highly specialized error message when reassigning to B.__name__: the line
B.__name__ = property(test)
is actually equivalent to
type.__setattr__(B, '__name__', property(test))
which is calling our special-case checker in CPython.
For any other type in Python, in particular for user-defined types, we could get around this with object.__setattr__. Unfortunately,
>>> object.__setattr__(B, '__name__', property(test))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't apply this __setattr__ to type object
There's a really specific check to make sure we don't do exactly this, and the comment reads
/* Reject calls that jump over intermediate C-level overrides. */
We also can't use metaclasses to override __setattr__ and __getattribute__, because the instance lookup procedure specifically doesn't call those (in the above examples, __getattribute__ was called in every case except the one we care about for property purposes). I even tried subclassing str to trick __setattr__ into accepting our made-up value
class NameProperty(str):
def __new__(cls, value, **kwargs):
return str.__new__(cls, value)
def __init__(self, value, method):
self.method = method
def __get__(self, instance, owner):
return self.method(instance)
B.__name__ = NameProperty(B.__name__, method=test)
This actually passes the __setattr__ check, but it doesn't assign to B.__dict__ (since the __setattr__ still assigns to the actual CPython-level name, not to B.__dict__['__name__']), so the property lookup doesn't work.
So... that's how I reached my conclusion of: __name__ is deep magic in CPython. All of the usual Python metaprogramming techniques have failed, and all of the methods getting called are written deep down in C. My advice to you is: Stop using __name__ for things it's not intended for, or be prepared to write some C code and hack on CPython directly.
I have following class with a function:
class A:
def myfn():
print("In myfn method.")
Here, the function does not have self as argument. It also does not have #classmethod or #staticmethod as decorator. However, it works if called with class:
A.myfn()
Output:
In myfn method.
But give an error if called from any instance:
a = A()
a.myfn()
Error output:
Traceback (most recent call last):
File "testing.py", line 16, in <module>
a.myfn()
TypeError: myfn() takes 0 positional arguments but 1 was given
probably because self was also sent as an argument.
What kind of function will this be called? Will it be a static function? Is it advisable to use function like this in classes? What is the drawback?
Edit: This function works only when called with class and not with object/instance. My main question is what is such a function called?
Edit2: It seems from the answers that this type of function, despite being the simplest form, is not accepted as legal. However, as no serious drawback is mentioned in any of many answers, I find this can be a useful construct, especially to group my own static functions in a class that I can call as needed. I would not need to create any instance of this class. In the least, it saves me from typing #staticmethod every time and makes code look less complex. It also gets derived neatly for someone to extend my class. Although all such functions can be kept at top/global level, keeping them in class is more modular. However, I feel there should be a specific name for such a simple construct which works in this specific way and it should be recognized as legal. It may also help beginners understand why self argument is needed for usual functions in a Python class. This will only add to the simplicity of this great language.
The function type implements the descriptor protocol, which means when you access myfn via the class or an instance of the class, you don't get the actual function back; you get instead the result of that function's __get__ method. That is,
A.myfn == A.myfn.__get__(None, A)
Here, myfn is an instance method, though one that hasn't been defined properly to be used as such. When accessed via the class, though, the return value of __get__ is simply the function object itself, and the function can be called the same as a static method.
Access via an instance results in a different call to __get__. If a is an instance of A, then
a.myfn() == A.myfn.__get__(a, A)
Here , __get__ tries to return, essentially, a partial application of myfn to a, but because myfn doesn't take any arguments, that fails.
You might ask, what is a static method? staticmethod is a type that wraps a function and defines its own __get__ method. That method returns the underlying function whether or not the attribute is accessed via the class or an instance. Otherwise, there is very little difference between a static method and an ordinary function.
This is not a true method. Correctly declarated instance methods should have a self argument (the name is only a convention and can be changed if you want hard to read code), and classmethods and staticmethods should be introduced by their respective decorator.
But at a lower level, def in a class declaration just creates a function and assigns it to a class member. That is exactly what happens here: A.my_fn is a function and can successfully be called as A.my_fn().
But as it was not declared with #staticmethod, it is not a true static method and it cannot be applied on a A instance. Python sees a member of that name that happens to be a function which is neither a static nor a class method, so it prepends the current instance to the list of arguments and tries to execute it.
To answer your exact question, this is not a method but just a function that happens to be assigned to a class member.
Such a function isn't the same as what #staticmethod provides, but is indeed a static method of sorts.
With #staticmethod you can also call the static method on an instance of the class. If A is a class and A.a is a static method, you'll be able to do both A.a() and A().a(). Without this decorator, only the first example will work, because for the second one, as you correctly noticed, "self [will] also [be] sent as an argument":
class A:
#staticmethod
def a():
return 1
Running this:
>>> A.a() # `A` is the class itself
1
>>> A().a() # `A()` is an instance of the class `A`
1
On the other hand:
class B:
def b():
return 2
Now, the second version doesn't work:
>>> B.b()
2
>>> B().b()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: b() takes 0 positional arguments but 1 was given
further to #chepnet's answer, if you define a class whose objects implement the descriptor protocol like:
class Descr:
def __get__(self, obj, type=None):
print('get', obj, type)
def __set__(self, obj, value):
print('set', obj, value)
def __delete__(self, obj):
print('delete', obj)
you can embed an instance of this in a class and invoke various operations on it:
class Foo:
foo = Descr()
Foo.foo
obj = Foo()
obj.foo
which outputs:
get None <class '__main__.Foo'>
get <__main__.Foo object at 0x106d4f9b0> <class '__main__.Foo'>
as functions also implement the descriptor protocol, we can replay this by doing:
def bar():
pass
print(bar)
print(bar.__get__(None, Foo))
print(bar.__get__(obj, Foo))
which outputs:
<function bar at 0x1062da730>
<function bar at 0x1062da730>
<bound method bar of <__main__.Foo object at 0x106d4f9b0>>
hopefully that complements chepnet's answer which I found a little terse/opaque
This may be a silly question but i am curious to know the answer.
As per official documentation, __init__ doesn't need return statement. Any particular reason why is it that way.
>>> class Complex:
... def __init__(self, realpart, imagpart):
... self.r = realpart
... self.i = imagpart
...
>>> x = Complex(3.0, -4.5)
>>> x.r, x.i
(3.0, -4.5)
__init__() is not a normal function. It is a special method Python uses to customize an instance of a class. It is part of Python's data model:
Called after the instance has been created (by __new__()), but before it is returned to the caller[...].
As you can see from above, when you create a new instance of a class, Python first calls __new_() - which is also a special method - to create a new instance of the class. Then __init__() is called to customize the new instance.
It wouldn't make sense to return anything from __init__(), since the class instance is already created. In fact, Python goes as far as raising an error to prevent this:
>>> class A:
... def __init__(self):
... return 'foo'
...
>>> A()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() should return None, not 'str'
>>>
If you want to know what exactly is going on behind the scenes, #eryksun provides a nice explanation:
To completely explain this story, you have to step back to the metaclass __call__ method. In particular the default type.__call__ in CPython calls __new__ and __init__ via their C slot functions, and it's slot_tp_init (defined in Objects/typeobject.c) that enforces the return value to be None. If you use a custom metaclass that overrides type.__call__, it can manually call the __new__ and __init__ methods of the class with no restriction on what __init__ can return -- as silly as that would be.
__init__ is called when you create a new instance of a class.
It's main use is initializing the instance variables, and it can be called only with an instance - so you can't call it before you create an instance anyways (what triggers it automatically).
For these reasons, __init__s have no reason to be able to return any value - it's simply not their use case.
I came across this as a bit of a surprise while trying to work out another question.
This seemed extremely odd to me, I thought it was worth asking the question. Why doesn't __getattr__ appear to work with with?
if I make this object:
class FileHolder(object):
def __init__(self,*args,**kwargs):
self.f= file(*args,**kwargs)
def __getattr__(self,item):
return getattr(self.f,item)
and using it with with,
>>> a= FileHolder("a","w")
>>> a.write
<built-in method write of file object at 0x018D75F8>
>>> with a as f:
... print f
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __exit__
>>> a.__exit__
<built-in method __exit__ of file object at 0x018D75F8>
Why does this happen?
EDIT
>>> object.__exit__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'object' has no attribute '__exit__'
It definitely isn't inheriting __exit__
The with statement opcode SETUP_WITH looks up __exit__ as a "special method lookup", which ignores __getattr__ and __getattribute__ on new-style classes (but not on old-style classes). See this mailing list thread for more information, where they discuss adding the special method lookup semantics to with (which they eventually do). See also special method lookup for new-style classes for a detailed discussion on why these special methods are looked up in this way.
In particular, special method lookup also bypasses __getattr__ on the type object. So, even though the documentation says the method is looked up as type(mgr).__exit__, this code doesn't work:
class M(type):
def __getattr__(*args): return lambda: 0
class X(object):
__metaclass__ = M
x = X()
type(x).__exit__ # works, returns a lambda
with x: pass # fails, AttributeError
I can't say for sure, but after reading over the PEP describing the with statement:
http://www.python.org/dev/peps/pep-0343/
This jumped out at me:
A new statement is proposed with the syntax:
with EXPR as VAR:
BLOCK
....
The translation of the above statement is:
mgr = (EXPR)
exit = type(mgr).__exit__ # Not calling it yet
value = type(mgr).__enter__(mgr)
....
Right there. The with statement does not call __getattr__(__exit__) but calls type(a).__exit__ which does not exist giving the error.
So you just need to define those:
class FileHolder(object):
def __init__(self,*args,**kwargs):
self.f= file(*args,**kwargs)
def __enter__(self,*args,**kwargs):
return self.f.__enter__(*args,**kwargs)
def __exit__(self,*args,**kwargs):
self.f.__exit__(*args,**kwargs)
def __getattr__(self,item):
return getattr(self.f,item)
The previous answers has explained the fact that __getattr__ does not work with __enter__ and __exit__. I'm here to give my thinking of why it SHOULD NOT work.
The only reason we define __enter__ and __exit__ methods on an object is that we need to use it in with statement. The two methods help us get and release a resource implicitly, so we usually define them like this:
class Resource(object):
...
def __enter__(self):
return self
def __exit__(self, *exc):
self.close()
then you can write some code like this:
with Resource() as resource: # __enter__ is called and returns a value as `resource`
do_something_with_resource()
# `resource.__exit__` is called
As you have noticed, the resource we get and release is exactly an instance of the class we defined.
What if we hold a resource as an attribute and proxy its __enter__ and __exit__ with __getattr__? We write some code like this:
class ResourceProxy(object):
def __init__(self):
self._resource = Resource()
def __getattr__(self, key):
return getattr(self._resource, key)
Assuming __getattr__ works fine with __enter__ and __exit__, here is what will happen in with statement:
with ResourceProxy() as resource: # proxied __enter__ is called
# now `resource` is NOT a ResourceProxy instance, because what we called is `_resource.__enter__`
do_something_with_resource()
# `_resource.__exit__` is called and closed itself properly.
# Here is nothing to do with ResourceProxy, because it has never enter `with` context
The behavior above is strange and probably not as the user expected, for the following two reasons:
the resource entered into with context is not the object we sent in.
when exiting with context, __exit__ method of the proxied object is called, instead of the outer object we sent in. You may think it might help if we add an __exit__ definition on the outer class, but the answer is not, because the outer class has never enter with context.
To conclude, if we make __getattr__ works with __enter__ and __exit__, it will result in bad behaviors. It's not a good design.
Why doesn't this work?
>>> class A (unicode):
... def __init__ (self, value):
... super(unicode, self).__init__(str(value).upper())
...
>>> assert A('hello') == u'HELLO'
Traceback (most recent call last):
File "<pyshell#39>", line 1, in <module>
assert A('hello') == u'HELLO'
AssertionError
>>>
I know that without the init method initializing the object would default to the unicode classes init method (thank you MRO) and I know that my init method is the one being called, however it doesn't seem to want to work. Even when I add an extra line and explicitly set the value of the value parameter to upper it still doesn't want to work.
Although I can (and probably will have to) define the repr and str special classes in order to get the desired effect, I'm a little curious as to why this doesn't work.
Thanks in advance.
unicode are immutable object use __new__ instead of __init__:
class A (unicode):
def __new__ (cls, value):
return unicode.__new__(cls, str(value).upper())
print A('hello')
u'HELLO'
Instead of super(unicode, self), you probably want super(A, self) - the former is object.__init__.
Additionally, since unicode objects are immutable, __init__ does nothing. Instead of the regular initialization with __init__ (which would also prevent interning), the constructor you want to override is __new__:
class A (unicode):
def __new__(cls, value):
return super(A, cls).__new__(cls, str(value).upper())