We Keep Coding

Confused about the property decorator and its related getter/setter methods

I've been doing some Python, and I realised I Haven't actually know a lot about the property decorator, so I tried making a simple example. This is the code I used:
class foo():
def __init__(self):
self.__test = 0
#property
def test(self):
return self.__test
#test.setter
def test(self, value):
self.__test = value
#test.getter
def test(self):
self.__test += 1
return self.__test
Then I started playing with it in the interactive shell:
>>> bar = foo()
>>> bar.test
1
>>> bar.test
2
So far the object behaved as I expected it to.
Then I tried checking out the setter method
>>> bar.test = 5
>>> bar.test
5
>>> bar.test
5
Weird. For some reason the value of __test wasn't incremented.
>>> bar._foo__test
2
I thought I had set __test to be equal to 5.
What's going on?

The problem is that your foo class is an old style class, descriptors (and as such properties) are only intended to work with new style classes.
From the doc:
Note that descriptors are only invoked for new style objects or classes (a class is new style if it inherits from object or type)
In this case, with an old style class setting bar.test = 5 creates a test attribute in the instance dict, which shadows the property from the class dict:
>>> bar = foo()
>>> foo.__dict__
{'test': <property object at 0x7f302e64c628>, '__module__': '__main__', '__doc__': None, '__init__': <function __init__ at 0x7f302e658b18>}
>>> bar.test # test property from class dict is used
1
>>> bar.__dict__
{'_foo__test': 1}
>>> bar.test = 5 # sets test on instance
>>> bar.__dict__
{'test': 5, '_foo__test': 1}
So the solution is simple: make foo a new style class by inheriting from object

Adding a new attribute to Python object?

Being from OOPS background, It looks strange to see below code from link
def f():
f.beencalled = True
return 0
My question:
1)
From the above code,
Is f a reference variable pointing to an object f of class 'function'?
2)
We add a new attribute beencalled to an object f, so now 'function' class does not have this attribute beencalled defined and we say that object f is an object of class 'function'? Does it make sense?

1) Yes:
>>> def f():
print(type(f))
>>> f()
>>> <class 'function'>
2) The function class does not have a new attribute, but the object f does. Adding or removing attributes to/from an object does not affect which attributes other objects of that class will have:
>>> class A: pass
>>> a = A()
>>> a.var = 7
>>> b = A()
>>> b.var
Traceback (most recent call last):
File "<pyshell#19>", line 1, in <module>
b.newvar
AttributeError: 'A' object has no attribute 'var'
Classes are much more flexible in python than in Java or C++. Objects can have attributes not defined in their class, or even lack attributes that were defined in their class! Look at this:
>>> class A:
def __init__(self, a):
self.var = a
>>> obj = A(7)
>>> del obj.var #deletes the var attribute from obj, does not change the A class
>>> obj.var
Traceback (most recent call last):
File "<pyshell#28>", line 1, in <module>
obj.var
AttributeError: 'A' object has no attribute 'var'
>>> obj2 = A(6)
>>> obj2.var #obj2 is a new object, so the fact we deleted var from obj doesn't affect it
6
EDIT: after a bit of searching I found an explanation for why this behavior was chosen (source):
To implement user-defined objects, I settled on the simplest possible
design; a scheme where objects were represented by a new kind of
built-in object that stored a class reference pointing to a "class
object" shared by all instances of the same class, and a dictionary,
dubbed the "instance dictionary", that contained the instance
variables.
In this implementation, the instance dictionary would contain the
instance variables of each individual object whereas the class object
would contain stuff shared between all instances of the same class--in
particular, methods. In implementing class objects, I again chose the
simplest possible design; the set of methods of a class were stored in
a dictionary whose keys are the method names. This, I dubbed the class
dictionary. To support inheritance, class objects would additionally
store a reference to the class objects corresponding to the base
classes. At the time, I was fairly naïve about classes, but I knew
about multiple inheritance, which had recently been added to C++. I
decided that as long as I was going to support inheritance, I might as
well support a simple-minded version of multiple inheritance. Thus,
every class object could have one or more base classes.
In this implementation, the underlying mechanics of working with
objects are actually very simple. Whenever changes are made to
instance or class variables, those changes are simply reflected in the
underlying dictionary object. For example, setting an instance
variable on an instance updates its local instance dictionary.
Likewise, when looking up the value of a instance variable of an
object, one merely checks its instance dictionary for the existence of
that variable. If the variable is not found there, things become a
little more interesting. In that case, lookups are performed in the
class dictionary and then in the class dictionaries of each of the
base classes.

On a slightly different note, you can change this behavior for custom classes.
class FooBar(object):
__slots__ = ["foo","bar","baz"]
# if you don't define __slots__, you can add attr to the object as needed
# if you do, the object can only contain those attributes.
def __init__(self,foo=None,bar=None,baz=None):
self.foo = foo
self.bar = bar
self.baz = baz
def __str__(self):
return "I'm a FooBar with id {0} with foo: {1.foo}, bar: {1.bar}, baz: {1.baz}".format(id(self),self)
>>> a = FooBar("a","B","CCC")
>>> print(a)
I'm a FooBar with id 47260256 with foo: a, bar: B, baz: CCC
>>> a.spam = "eggs"
Traceback (most recent call last):
File "<pyshell#13>", line 1, in <module>
a.spam = "eggs"
AttributeError: 'FooBar' object has no attribute 'spam'
Alternately, without defining __slots__:
class BooFar(object):
def __str__(self):
return "I'm a BooFar with the following attributes:\n{}".format(self.__dict__)
>>> b = BooFar()
>>> print(b)
I'm a BooFar with the following attributes:
{}
>>> b.spam = "eggs"
>>> print(b)
I'm a BooFar with the following attributes:
{'spam': 'eggs'}

f() in just an instance of types.FunctionType, and instances can have their own attributes.
Adding an attribute to an instance won't affect its class unless you've overridden the __setattr__ method of that class and doing something evil there.
>>> import types
>>> def func(): pass
>>> isinstance(func, types.FunctionType)
True

Why 'declare' variables in Python?

I was Googling some Python-related questions earlier, and stumbled upon this page. The author does something like the following:
class TestClass(object):
first = str()
def __init__(self):
self.first = "Hello"
What's the point of "declaring" the variable first like that? I've never seen this done before, and I can't for the life of me think of a situation where it is beneficial to create a variable before assigning it some value.
The above example could just as well have looked like this:
class TestClass(object):
def __init__(self, first="Hello"):
self.first = first
...or am I missing something?

The fact that the author uses
first = str()
as opposed to
first = ''
shows, alongside setting self.first in __init__ anyway, that there that is no purpose in doing this.
Maybe the author is confused and thinks python variable need to be declared first -_- (evident when viewing the link)

That's not a declaration, that's an assignment ... to a variable inside the class, as opposed to a variable inside an instance.
Consider the following output:
>>> class K1(object):
... def __init__(self):
... self.attr = 'value'
...
>>> x = K1()
>>> x.__dict__
{'attr': 'value'}
>>> class K2(object):
... attr = 'value'
... def __init__(self):
... self.another = 'value2'
...
>>> y = K2()
>>> y.__dict__
{'another': 'value2'}
Here x is an instance of class K1 and has an attribute named attr, and y is an instance of class K2 and has a different attribute named another. But:
>>> y.attr
'value'
Where did that come from? It came from the class:
>>> y.__class__.__dict__
dict_proxy({'__module__': '__main__', 'attr': 'value',
'__dict__': <attribute '__dict__' of 'K2' objects>,
'__weakref__': <attribute '__weakref__' of 'K2' objects>,
'__doc__': None, '__init__': <function __init__ at 0x80185b9b0>})
That's kind of messy but you can see the attr sitting in there. If you look at x.__class__.__dict__ there's no attr:
>>> x.__class__.__dict__
dict_proxy({'__dict__': <attribute '__dict__' of 'K1' objects>,
'__module__': '__main__',
'__weakref__': <attribute '__weakref__' of 'K1' objects>,
'__doc__': None, '__init__': <function __init__ at 0x80185b938>})
When you get an attribute on an instance, like x.attr or y.attr, Python first looks for something attached to the instance itself. If nothing is found, though, it "looks upward" to see if something else defines that attribute. For classes with inheritance, that involves going through the "member resolution order" list. In this case there is no inheritance to worry about, but the next step is to look at the class itself. Here, in K2, there's an attribute in the class named attr, so that's what y.attr produces.
You can change the class attribute to change what shows up in y.attr:
>>> K2.attr = 'newvalue'
>>> y.attr
'newvalue'
And in fact, if you make another instance of K2(), it too will pick up the new value:
>>> z = K2()
>>> z.attr
'newvalue'
Note that changing x's attr does not affect new instances of K1():
>>> w = K1()
>>> w.attr = 'private to w'
>>> w.attr
'private to w'
>>> x.attr
'value'
That's because w.attr is really w.__dict__['attr'], and x.attr is really x.__dict__['attr']. On the other hand, y.attr and z.attr are both really y.__class__.__dict__['attr'] and z.__class__.__dict__['attr'], and since y.__class__ and z.__class__ are both K2, changing K2.attr changes both.
(I'm not sure the guy who wrote the page referenced in the original question realizes all this, though. Creating a class-level attribute and then creating an instance-level one with the same name is kind of pointless.)

str() is equal to ""
>>> str()
''
I think the author wants to show that instance attributes override class attributes having same name. So on executing
test = testclass()
print test.__dict__
you'll get:
{'second': 'weird', 'third': 'test', 'first': 'Some'}
not
{'second': '', 'third': '', 'first': ''}
but
print testclass.__dict__
will print the class attributes:
{'__module__': '__main__', 'third': '', 'second': '', '__doc__': None, '__init__': <function __init__ at 0xb5fed6bc>, 'first': ''}

There is indeed a little difference between the two examples:
class TestClass(object):
first = 'foo'
def __init__(self):
self.first = "Hello"
print(TestClass.first)
Output:
foo
However with:
class TestClass(object):
def __init__(self, first="Hello"):
self.first = "Hello"
print(TestClass.first)
Output:
Traceback (most recent call last):
File "C:\Users\...\Desktop\test.py", line 5, in <module>
print(TestClass.first)
AttributeError: type object 'TestClass' has no attribute 'first'
Note: But that doesn't mean that the author's code make sense. Just wanted to point out the difference.

Obtaining object reference for a method using getattr

Say, I have the following class called Test with a method called start
>>> class Test:
... def __init__(self, *args, **kwargs):
... pass
... def start(self):
... pass
...
Now, I have a standalone independent function called func
>>> def func():
... print 'this is a func and not a method!!!'
...
>>>
[1]
Now, t.start is a method of an instance of __main__.Test which belong to 0xb769678c
>>> t = Test()
>>> t.start
<bound method Test.start of <__main__.Test instance at 0xb769678c>>
>>>
[2]
func is a function which belong to the location 0xb767ec6c
>>> func
<function func at 0xb767ec6c>
>>>
Now, we can extract the __module__ from t.start and func by using builtin __module__. Not surprisingly, func and t.start belong to the same module i.e. __main__
>>> func.__module__
'__main__'
>>> t.__module__
'__main__'
>>>
[3]
Now, lets store __module__ for t.start in a variable obj
>>> obj = __import__(t.start.__module__)
>>> obj
<module '__main__' (built-in)>
>>>
Now, I use getattr() to get the func handle <function func at 0xb767ec6c> for function func as follows and the output of getattr() is the identical to [2]
>>> print getattr(obj, 'func')
<function func at 0xb767ec6c>
>>>
>>> print getattr(__import__('__main__'), 'func')
<function func at 0xb767ec6c>
>>>
Question:
How do I use getattr() and the module name [3] to get the handle of Test.start [1] which should be <bound method Test.start of <__main__.Test instance at 0xb769678c>>
When I tried using getattr() on 't.start' I got the following Traceback
>>> print getattr(obj, 'Test.start')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'Test.start'
>>>
>>>
>>> print getattr(__import__('__main__'), 'Test.start')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'Test.start'
>>>
In other words, I have two data with me. They are
__import__('__main__')
The sting 'Test.start'
Now, how do I get the handle for t.start (note the instance here) which should be <bound method Test.start of <__main__.Test instance at 0xb769678c>>

I'm not sure if I understand your question(s), but I think this does what you want:
class Test:
def __init__(self, *args, **kwargs):
pass
def start(self):
pass
def func():
print('this is a func and not a method!!!')
t = Test()
module = __import__(t.start.__module__)
print(vars(module)['Test'].start)
print(vars(module)['func'])
print(vars(module)['t'].start)
(Python 3) output:
<function Test.start at 0x00E52460>
<function func at 0x00E524F0>
<bound method Test.start of <__main__.Test object at 0x008DF670>>

obj = __import__(t.start.__module__)
test_class = getattr(obj, 'Test')
print getattr(test_class, 'start')
I'm not sure if you need it straight from the module (or even if that is possible) :/
You could also use:
obj = __import__(t.start.__module__)
print obj.__dict__["Test"].__dict__["start"]
but you asked for getattr() so...

I wondered why you wrote obj = __import__(t.start.__module__)
I think it's to hold a name for the main module, in order to have the possibility to obtain the objects in the module's namespace as attributes of the module thanks to the getattr( ) function.
I inform you that you don't have to do this trick.
globals( ) is a dictionary that represents the global namespace of the main space.
Then you can write, for example, globals()["func"] to obtain the object func
I also inform you that there is another way than with getattr(N,"xyz") to obtain an object xyz in the namespace of the object N thanks to its name xyz ,
it's by the __dict__ method:
N.__dict__ gives access to the namespace of the object N
I wonder if your problem doesn't lies in a subtlety of Python that induced you in an error.
In your question, at a place you wrote t.__module__ and at another place you wrote t.start.__module__ .
For the two cases, the result is the same and equal to "__main__"
However,
1) t has no attribute of name __module__
2) t has even no attribute of name start !
If you print t.__dict__ you'll see the result is { } !
start isn't really belonging to the namespace of the instance t because it belongs in fact to the namespace of Test
The following code proves these affirmations:
class Test:
def __init__(self, *args, **kwargs):
pass
def start(self):
pass
y = Test()
y.ku = 102
print 'y.__dict__',y.__dict__
print 'Test.__dict__',Test.__dict__
result
y.__dict__ {'ku': 102}
Test.__dict__ {'start': <function start at 0x011E11B0>,
'__module__': '__main__',
'__doc__': None,
'__init__': <function __init__ at 0x011E1170>}
.
The reason why the expressions t.__module__ and t.start.__module__ give answers anyway is explained by this citation:
A class instance has a namespace implemented as a dictionary which is
the first place in which attribute references are searched.
When an
attribute is not found there, and the instance’s class has an
attribute by that name, the search continues with the class
attributes.
http://docs.python.org/2/reference/datamodel.html#index-55
Hence:
1) t has no attribute of name __module__ then Python searches for it in the class of t
2) t has no attribute of name start then Python searches for it in the class of t
.
Well, so, Python goes to the class of t to find the values of t.__module__ and t.start.__module__
This mechanism, described by the above citation, can be observed with the following code:
class Test:
def __init__(self, *args, **kwargs):
pass
def start(self):
pass
t = Test()
print 'getattr(t,"start")'
print getattr(t,"start")
# result is
getattr(t,"start")
<bound method Test.start of <__main__.Test instance at 0x011DF288>>
I mean that Python answers that the attribute of t whose name is "start" is Test.start , not t.start !
.
Thereby, from the fact that t.start.__module__ is "__main__", it seems to me that you may have believed that t.start is belonging to the module namespace, and hence you should be able to write something similar to getattr(obj,"func") to obtain the object start
But it is false.
You can't find the method start (I write the METHOD) in the global namespace = namespace of the module, since it is in the namespace of Test. You must attain it, as a method, through the instance or the class or through the class itself.
One more thing.
I precised : the METHOD, because I believe that the method t.start is based on a FUNCTION that is something different from it and that lies in the module's namespace. In reality, a method is a wrapper of pointers pointing to an instance and to this global namespace's function:
If you still don’t understand how methods work, a look at the
implementation can perhaps clarify matters. When an instance attribute
is referenced that isn’t a data attribute, its class is searched. If
the name denotes a valid class attribute that is a function object, a
method object is created by packing (pointers to) the instance object
and the function object just found together in an abstract object:
this is the method object. When the method object is called with an
argument list, a new argument list is constructed from the instance
object and the argument list, and the function object is called with
this new argument list.
http://docs.python.org/2/tutorial/classes.html#method-objects
Well what I want to underline is that it seems a function (not a method) exists somewhere, on which a method is based.
What I believe is that this function lies in the module namespace and that it's why Test.__dict__["start"]
gives
<function start at 0x011E40F0>
while
getattr(Test,"start")
and getattr(t,"start")
give
<unbound method Test.start>
and <bound method Test.start of <__main__.Test instance at 0x011DF530>>
without localizing the bound and unbound methods.
So, in my opinion, as far as I have correctly understood the doc (which is not explaining these points very well, I find, by the way) a method is a wrapper, and is based on a real function that lies in the module namespace, and that's why t.start.__module__ and Test.start.__module__ are "__main__"
It may seem a little weird that there exists a function in the module namespace that doesn't appear among the attributes of the module when we print them with globals() :
class Test:
def __init__(self, *args, **kwargs):
pass
def start(self):
pass
def func():
print 'this is a func and not a method!!!'
t = Test()
print '* globals()["func"]'
print globals()["func"]
print id(globals()["func"])
print "===================================="
print '* Test.__dict__["start"]'
print Test.__dict__["start"]
print id(Test.__dict__["start"])
print '----------------------------------------------'
print '* getattr(Test,"start")'
print getattr(Test,"start")
print id(getattr(Test,"start"))
print '----------------------------------------------'
print '* getattr(t,"start")'
print getattr(t,"start")
print id(getattr(t,"start"))
print "===================================="
print globals()
result
* globals()["func"]
<function func at 0x011C27B0>
18622384
====================================
* Test.__dict__["start"]
<function start at 0x011DEFB0>
18739120
----------------------------------------------
* getattr(Test,"start")
<unbound method Test.start>
18725304
----------------------------------------------
* getattr(t,"start")
<bound method Test.start of <__main__.Test instance at 0x011DF418>>
18725304
{'__builtins__': <module '__builtin__' (built-in)>,
'__package__': None,
't': <__main__.Test instance at 0x011DF418>,
'func': <function func at 0x011C27B0>,
'Test': <class __main__.Test at 0x011DC538>,
'__name__': '__main__',
'__doc__': None}
But the fact is that we indeed see a Test.__dict__["start"] function whose address 18739120 is different from the bound and unbound methods address 18725304.
Moreover, I don't see which other explanation could explain the totality of the facts exposed.
There's in fact nothing weird in the fact that a function that didn't receive any assigned name doesn't appear in the module namespace.
It is an internal function, necessary to Python, not at the disposal of the programmer, that's all.
Concerning func, using all the methods you employed:
class Test:
def __init__(self, *args, **kwargs):
pass
def start(self):
pass
def func():
print 'this is a func and not a method!!!'
t = Test()
print 'func --->',func
print id(func)
print '\nobj = __import__(t.start.__module__) done\n'
obj = __import__(t.start.__module__)
print '* globals()["func"]'
print globals()["func"]
print id(globals()["func"])
print '* obj.__dict__["func"]'
print obj.__dict__["func"]
print "* getattr(obj, 'func')"
print getattr(obj, 'func')
print "* getattr(__import__('__main__'), 'func')"
print getattr(__import__('__main__'), 'func')
result
func ---> <function func at 0x011C2470>
18621552
obj = __import__(t.start.__module__) done
* globals()["func"]
<function func at 0x011C2470> # <== address in hexadecimal
18621552 # <== address in decimal
* obj.__dict__["func"]
<function func at 0x011C2470>
* getattr(obj, 'func')
<function func at 0x011C2470>
* getattr(__import__('__main__'), 'func')
<function func at 0x011C2470>
.
Now taking apart your methods using obj :
class Test:
def __init__(self, *args, **kwargs):
pass
def start(self):
pass
print 'Test.start -->',Test.start
print id(Test.start)
print '----------------------------------------------'
print '* Test.__dict__["start"]'
print Test.__dict__["start"]
print id(Test.__dict__["start"])
print '* getattr(Test,"start")'
print getattr(Test,"start")
print id(getattr(Test,"start"))
print '\n'
print 't.start -->',t.start
print id(t.start)
print '----------------------------------------------'
print '* t.__dict__["start"]'
try:
print t.__dict__["start"]
print id(t.__dict__["start"])
except KeyError as e:
print 'KeyError :',e
print '* getattr(t,"start")'
print getattr(t,"start")
print id(getattr(t,"start"))
result
Test.start --> <unbound method Test.start>
18725264
----------------------------------------------
* Test.__dict__["start"]
<function start at 0x011E40F0>
18759920
* getattr(Test,"start")
<unbound method Test.start>
18725264
t.start --> <bound method Test.start of <__main__.Test instance at 0x011DB940>>
18725264
----------------------------------------------
* t.__dict__["start"]
KeyError : 'start'
* getattr(t,"start")
<bound method Test.start of <__main__.Test instance at 0x011DB940>>
18725264
Until now, I had seldom used getattr( ).
I realize on these results that getattr( ) gives results in a specific manner: the attribute start is described as a method, bound or unbound according if it is attained through an instance or through a class.
However, only __dict__ gives precise information on the real attributes of an object.
Thus, we see that start isn't really in the namespace of the instance, but only in the namespace of the class.
It is described as being a function when it is as element of the namespace Test.__dict__ and as being a method when given as an attribute through getattr( ). There are no addresses given in this last case.
The use of the vars( ) function gives the same results.

Why is getattr() so much slower than self.__dict__.get()?

The example below is from a REST database driver on Python 2.7.
In the __setattr__ method below, if I use the commented out getattr() line, it reduces the object instantiation performance from 600 rps to 230.
Why is getattr() so much slower than self.__dict__.get() in this case?
class Element(object):
def __init__(self, client):
self._client = client
self._data = {}
self._initialized = True
def __setattr__(self, key, value):
#_initialized = getattr(self, "_initialized", False)
_initialized = self.__dict__.get("_initialized", False)
if key in self.__dict__ or _initialized is False:
# set the attribute normally
object.__setattr__(self, key, value)
else:
# set the attribute as a data property
self._data[key] = value

In short: because getattr(foo,bar) does the same thing as foo.bar, which is not the same thing as just accessing the __dict__ property (for a start, getattr has to select the right __dict__, but there's a whole lot more going on).
An example for illustration:
>>> class A:
... a = 1
...
>>> class B(A):
... b = 2
...
>>> dir(B)
['__doc__', '__module__', 'a', 'b']
>>> B.a
1
>>> B.__dict__
{'__module__': '__main__', 'b': 2, '__doc__': None}
>>> B.__dict__['a']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'a'
>>> B.__dict__.get('a')
>>>
Details contained in, or linked to here: http://docs.python.org/reference/datamodel.html (search for "getattr").