Turning nested class into descriptor without instantiating it

Turning nested class into descriptor without instantiating it - python

Note: Python version is 2.7
Problem
I want to have {hierarchy of descriptor classes} integrated within {hierarchy of widget classes}, and overriding descriptor behaviour must be as easy as defining nested derived class. Example:
class A(object):
class width(Attribute):
def changed(self, obj, value, old):
print 'A.width changed:', value
class B(A):
class width(A.width):
def changed(self, obj, value, old):
super(B.width, self).changed(obj, value, old)
print 'B.width changed:', value
B().width = 10
# must print:
# A.width.changed: 10
# B.width.changed: 10
Here is my custom descriptor class:
class Attribute(object):
def __init__(self):
if not hasattr(self, '_name'):
self._name = self.__class__.__name__
def __get__(self, obj, cls):
if obj is None:
print 'Attribute getter (class):', cls
return self
print 'Attribute getter (class, inst):', (cls, obj)
print 'Attribute getter returning:', self.get(obj)
return self.get(obj)
def __set__(self, obj, value):
print 'Attribute setter (inst, value):', (obj, value)
self.set(obj, value)
def get(self, obj):
try:
return obj.__dict__[self._name]
except KeyError:
raise AttributeError("attribute '%s' referenced before assigment" % (self._name))
def set(self, obj, value):
try:
old = obj.__dict__[self._name]
except KeyError:
obj.__dict__[self._name] = value
self.changed(obj, value, None)
else:
obj.__dict__[self._name] = value
if old != value:
self.changed(obj, value, old)
def changed(self, obj, value, old):
pass
Problem is that Python don't want to use __get__ and __set__, while they are attributes of class. It may be seen from this test:
# `A` and `B` were defined above
A.width_ = A.width()
B.width_ = B.width()
to_test = (
# good:
'Aw_ = A.width_',
'Bw_ = B.width_',
'B().width_ = 10',
# silent:
'Aw = A.width',
'Bw = B.width',
'B().width = 10',
)
for expr in to_test:
print "Testing:", expr
exec expr
So, my Attribute works only when instantiated.
What I have tried already
Decorating __get__ and __set__ with either staticmethod or classmethod. No changes in silent section. Good section fails: methods aren't callable. Wat.
Adding __get__ and __set__ to class Attribute from outside, as methods bound to Attribute. Nothing changed.
Code:
# `__get__` was renamed to `_unbound__get__`
Attribute.__get__ = Attribute._unbound__get__.__get__(Attribute, Attribute.__class__)
# `__set__` was renamed to `_unbound__set__`
Attribute.__set__ = Attribute._unbound__set__.__get__(Attribute, Attribute.__class__)
Using instantiated descriptors. This approach requires 2 symbols: one for descriptor (sub)class and one for descriptor. It also requires instantiating descriptor after subclassing it.
Code:
class B(A):
class Width(A.width):
def changed(self, obj, value, old):
super(B.width, self).changed(obj, value, old)
print 'B.width.changed:', value
B.width = B.Width()
More background.
I have growing hierarchy of widgets, where some propertries must be tracked for changes, and response to change may be extended in subclassses. So I am trying to create automated approach, with modular machinery. Because keeping relevant variables and methods per each propertry inside actual widgets is just horrible annoying mess.
Question
Are there workarounds to achieve my needs? Or may be I am doing something wrong?

Descriptors require instances, so you cannot achieve exactly what you want. In particular you want A.width to be, at the same time, both a class and an instance of that class, which is impossible. You must have a way to access the class and the instance separately.
It's pretty simple to automatically create the instances with a known naming scheme. For example using a class decorator:
def track_attributes(cls):
for attr in dir(cls):
value = getattr(cls, attr)
if isinstance(value, type) and issubclass(value, Attribute):
# probably require more checks and/or different naming scheme
setattr(cls, attr.lower(), value())
return cls
Used as:
#track_attributes
class A(object):
class Width(Attribute):
def changed(self, obj, value, old):
print 'A.width changed:', value
#track_attributes
class B(A):
class Width(A.Width):
def changed(self, obj, value, old):
super(B.Width, self).changed(obj, value, old)
print 'B.width changed:', value
B().width = 10
output:
$python test_attributes.py
Attribute getter (class): <class '__main__.B'>
Attribute setter (inst, value): (<__main__.B object at 0x7fc225cd7d50>, 10)
A.width changed: 10
B.width changed: 10
An other way to achieve this without explicit decorators is to create a metaclass and have a Widget base class for all widgets.

As mata says, descriptors needs to be instances. One possible solution would be to use a class decorator or a custom metaclass (in your case more probably a custom metaclass) that would lookup the class namespace for Attribute subclasses and instanciate them. The whole thing smells a bit like overengineering to me but sometimes well you just need that level of complexity.

Related

Overloading get on a custom Python class not working [duplicate]

I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
What is instance and owner here? (in __get__). What is the purpose of these parameters?
How would I call/use this example?

The descriptor is how Python's property type is implemented. A descriptor simply implements __get__, __set__, etc. and is then added to another class in its definition (as you did above with the Temperature class). For example:
temp=Temperature()
temp.celsius #calls celsius.__get__
Accessing the property you assigned the descriptor to (celsius in the above example) calls the appropriate descriptor method.
instance in __get__ is the instance of the class (so above, __get__ would receive temp, while owner is the class with the descriptor (so it would be Temperature).
You need to use a descriptor class to encapsulate the logic that powers it. That way, if the descriptor is used to cache some expensive operation (for example), it could store the value on itself and not its class.
An article about descriptors can be found here.
EDIT: As jchl pointed out in the comments, if you simply try Temperature.celsius, instance will be None.

Why do I need the descriptor class?
It gives you extra control over how attributes work. If you're used to getters and setters in Java, for example, then it's Python's way of doing that. One advantage is that it looks to users just like an attribute (there's no change in syntax). So you can start with an ordinary attribute and then, when you need to do something fancy, switch to a descriptor.
An attribute is just a mutable value. A descriptor lets you execute arbitrary code when reading or setting (or deleting) a value. So you could imagine using it to map an attribute to a field in a database, for example – a kind of ORM.
Another use might be refusing to accept a new value by throwing an exception in __set__ – effectively making the "attribute" read only.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
This is pretty subtle (and the reason I am writing a new answer here - I found this question while wondering the same thing and didn't find the existing answer that great).
A descriptor is defined on a class, but is typically called from an instance. When it's called from an instance both instance and owner are set (and you can work out owner from instance so it seems kinda pointless). But when called from a class, only owner is set – which is why it's there.
This is only needed for __get__ because it's the only one that can be called on a class. If you set the class value you set the descriptor itself. Similarly for deletion. Which is why the owner isn't needed there.
How would I call/use this example?
Well, here's a cool trick using similar classes:
class Celsius:
def __get__(self, instance, owner):
return 5 * (instance.fahrenheit - 32) / 9
def __set__(self, instance, value):
instance.fahrenheit = 32 + 9 * value / 5
class Temperature:
celsius = Celsius()
def __init__(self, initial_f):
self.fahrenheit = initial_f
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
(I'm using Python 3; for python 2 you need to make sure those divisions are / 5.0 and / 9.0). That gives:
100.0
32.0
Now there are other, arguably better ways to achieve the same effect in python (e.g. if celsius were a property, which is the same basic mechanism but places all the source inside the Temperature class), but that shows what can be done...

I am trying to understand what Python's descriptors are and what they can be useful for.
Descriptors are objects in a class namespace that manage instance attributes (like slots, properties, or methods). For example:
class HasDescriptors:
__slots__ = 'a_slot' # creates a descriptor
def a_method(self): # creates a descriptor
"a regular method"
#staticmethod # creates a descriptor
def a_static_method():
"a static method"
#classmethod # creates a descriptor
def a_class_method(cls):
"a class method"
#property # creates a descriptor
def a_property(self):
"a property"
# even a regular function:
def a_function(some_obj_or_self): # creates a descriptor
"create a function suitable for monkey patching"
HasDescriptors.a_function = a_function # (but we usually don't do this)
Pedantically, descriptors are objects with any of the following special methods, which may be known as "descriptor methods":
__get__: non-data descriptor method, for example on a method/function
__set__: data descriptor method, for example on a property instance or slot
__delete__: data descriptor method, again used by properties or slots
These descriptor objects are attributes in other object class namespaces. That is, they live in the __dict__ of the class object.
Descriptor objects programmatically manage the results of a dotted lookup (e.g. foo.descriptor) in a normal expression, an assignment, or a deletion.
Functions/methods, bound methods, property, classmethod, and staticmethod all use these special methods to control how they are accessed via the dotted lookup.
A data descriptor, like property, can allow for lazy evaluation of attributes based on a simpler state of the object, allowing instances to use less memory than if you precomputed each possible attribute.
Another data descriptor, a member_descriptor created by __slots__, allows memory savings (and faster lookups) by having the class store data in a mutable tuple-like datastructure instead of the more flexible but space-consuming __dict__.
Non-data descriptors, instance and class methods, get their implicit first arguments (usually named self and cls, respectively) from their non-data descriptor method, __get__ - and this is how static methods know not to have an implicit first argument.
Most users of Python need to learn only the high-level usage of descriptors, and have no need to learn or understand the implementation of descriptors further.
But understanding how descriptors work can give one greater confidence in one's mastery of Python.
In Depth: What Are Descriptors?
A descriptor is an object with any of the following methods (__get__, __set__, or __delete__), intended to be used via dotted-lookup as if it were a typical attribute of an instance. For an owner-object, obj_instance, with a descriptor object:
obj_instance.descriptor invokes
descriptor.__get__(self, obj_instance, owner_class) returning a value
This is how all methods and the get on a property work.
obj_instance.descriptor = value invokes
descriptor.__set__(self, obj_instance, value) returning None
This is how the setter on a property works.
del obj_instance.descriptor invokes
descriptor.__delete__(self, obj_instance) returning None
This is how the deleter on a property works.
obj_instance is the instance whose class contains the descriptor object's instance. self is the instance of the descriptor (probably just one for the class of the obj_instance)
To define this with code, an object is a descriptor if the set of its attributes intersects with any of the required attributes:
def has_descriptor_attrs(obj):
return set(['__get__', '__set__', '__delete__']).intersection(dir(obj))
def is_descriptor(obj):
"""obj can be instance of descriptor or the descriptor class"""
return bool(has_descriptor_attrs(obj))
A Data Descriptor has a __set__ and/or __delete__.
A Non-Data-Descriptor has neither __set__ nor __delete__.
def has_data_descriptor_attrs(obj):
return set(['__set__', '__delete__']) & set(dir(obj))
def is_data_descriptor(obj):
return bool(has_data_descriptor_attrs(obj))
Builtin Descriptor Object Examples:
classmethod
staticmethod
property
functions in general
Non-Data Descriptors
We can see that classmethod and staticmethod are Non-Data-Descriptors:
>>> is_descriptor(classmethod), is_data_descriptor(classmethod)
(True, False)
>>> is_descriptor(staticmethod), is_data_descriptor(staticmethod)
(True, False)
Both only have the __get__ method:
>>> has_descriptor_attrs(classmethod), has_descriptor_attrs(staticmethod)
(set(['__get__']), set(['__get__']))
Note that all functions are also Non-Data-Descriptors:
>>> def foo(): pass
...
>>> is_descriptor(foo), is_data_descriptor(foo)
(True, False)
Data Descriptor, property
However, property is a Data-Descriptor:
>>> is_data_descriptor(property)
True
>>> has_descriptor_attrs(property)
set(['__set__', '__get__', '__delete__'])
Dotted Lookup Order
These are important distinctions, as they affect the lookup order for a dotted lookup.
obj_instance.attribute
First the above looks to see if the attribute is a Data-Descriptor on the class of the instance,
If not, it looks to see if the attribute is in the obj_instance's __dict__, then
it finally falls back to a Non-Data-Descriptor.
The consequence of this lookup order is that Non-Data-Descriptors like functions/methods can be overridden by instances.
Recap and Next Steps
We have learned that descriptors are objects with any of __get__, __set__, or __delete__. These descriptor objects can be used as attributes on other object class definitions. Now we will look at how they are used, using your code as an example.
Analysis of Code from the Question
Here's your code, followed by your questions and answers to each:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
Your descriptor ensures you always have a float for this class attribute of Temperature, and that you can't use del to delete the attribute:
>>> t1 = Temperature()
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
Otherwise, your descriptors ignore the owner-class and instances of the owner, instead, storing state in the descriptor. You could just as easily share state across all instances with a simple class attribute (so long as you always set it as a float to the class and never delete it, or are comfortable with users of your code doing so):
class Temperature(object):
celsius = 0.0
This gets you exactly the same behavior as your example (see response to question 3 below), but uses a Pythons builtin (property), and would be considered more idiomatic:
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
What is instance and owner here? (in get). What is the purpose of these parameters?
instance is the instance of the owner that is calling the descriptor. The owner is the class in which the descriptor object is used to manage access to the data point. See the descriptions of the special methods that define descriptors next to the first paragraph of this answer for more descriptive variable names.
How would I call/use this example?
Here's a demonstration:
>>> t1 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1
>>>
>>> t1.celsius
1.0
>>> t2 = Temperature()
>>> t2.celsius
1.0
You can't delete the attribute:
>>> del t2.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
And you can't assign a variable that can't be converted to a float:
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __set__
ValueError: invalid literal for float(): 0x02
Otherwise, what you have here is a global state for all instances, that is managed by assigning to any instance.
The expected way that most experienced Python programmers would accomplish this outcome would be to use the property decorator, which makes use of the same descriptors under the hood, but brings the behavior into the implementation of the owner class (again, as defined above):
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
Which has the exact same expected behavior of the original piece of code:
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1.0
>>> t2.celsius
1.0
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't delete attribute
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in celsius
ValueError: invalid literal for float(): 0x02
Conclusion
We've covered the attributes that define descriptors, the difference between data- and non-data-descriptors, builtin objects that use them, and specific questions about use.
So again, how would you use the question's example? I hope you wouldn't. I hope you would start with my first suggestion (a simple class attribute) and move on to the second suggestion (the property decorator) if you feel it is necessary.

Before going into the details of descriptors it may be important to know how attribute lookup in Python works. This assumes that the class has no metaclass and that it uses the default implementation of __getattribute__ (both can be used to "customize" the behavior).
The best illustration of attribute lookup (in Python 3.x or for new-style classes in Python 2.x) in this case is from Understanding Python metaclasses (ionel's codelog). The image uses : as substitute for "non-customizable attribute lookup".
This represents the lookup of an attribute foobar on an instance of Class:
Two conditions are important here:
If the class of instance has an entry for the attribute name and it has __get__ and __set__.
If the instance has no entry for the attribute name but the class has one and it has __get__.
That's where descriptors come into it:
Data descriptors which have both __get__ and __set__.
Non-data descriptors which only have __get__.
In both cases the returned value goes through __get__ called with the instance as first argument and the class as second argument.
The lookup is even more complicated for class attribute lookup (see for example Class attribute lookup (in the above mentioned blog)).
Let's move to your specific questions:
Why do I need the descriptor class?
In most cases you don't need to write descriptor classes! However you're probably a very regular end user. For example functions. Functions are descriptors, that's how functions can be used as methods with self implicitly passed as first argument.
def test_function(self):
return self
class TestClass(object):
def test_method(self):
...
If you look up test_method on an instance you'll get back a "bound method":
>>> instance = TestClass()
>>> instance.test_method
<bound method TestClass.test_method of <__main__.TestClass object at ...>>
Similarly you could also bind a function by invoking its __get__ method manually (not really recommended, just for illustrative purposes):
>>> test_function.__get__(instance, TestClass)
<bound method test_function of <__main__.TestClass object at ...>>
You can even call this "self-bound method":
>>> test_function.__get__(instance, TestClass)()
<__main__.TestClass at ...>
Note that I did not provide any arguments and the function did return the instance I had bound!
Functions are Non-data descriptors!
Some built-in examples of a data-descriptor would be property. Neglecting getter, setter, and deleter the property descriptor is (from Descriptor HowTo Guide "Properties"):
class Property(object):
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
Since it's a data descriptor it's invoked whenever you look up the "name" of the property and it simply delegates to the functions decorated with #property, #name.setter, and #name.deleter (if present).
There are several other descriptors in the standard library, for example staticmethod, classmethod.
The point of descriptors is easy (although you rarely need them): Abstract common code for attribute access. property is an abstraction for instance variable access, function provides an abstraction for methods, staticmethod provides an abstraction for methods that don't need instance access and classmethod provides an abstraction for methods that need class access rather than instance access (this is a bit simplified).
Another example would be a class property.
One fun example (using __set_name__ from Python 3.6) could also be a property that only allows a specific type:
class TypedProperty(object):
__slots__ = ('_name', '_type')
def __init__(self, typ):
self._type = typ
def __get__(self, instance, klass=None):
if instance is None:
return self
return instance.__dict__[self._name]
def __set__(self, instance, value):
if not isinstance(value, self._type):
raise TypeError(f"Expected class {self._type}, got {type(value)}")
instance.__dict__[self._name] = value
def __delete__(self, instance):
del instance.__dict__[self._name]
def __set_name__(self, klass, name):
self._name = name
Then you can use the descriptor in a class:
class Test(object):
int_prop = TypedProperty(int)
And playing a bit with it:
>>> t = Test()
>>> t.int_prop = 10
>>> t.int_prop
10
>>> t.int_prop = 20.0
TypeError: Expected class <class 'int'>, got <class 'float'>
Or a "lazy property":
class LazyProperty(object):
__slots__ = ('_fget', '_name')
def __init__(self, fget):
self._fget = fget
def __get__(self, instance, klass=None):
if instance is None:
return self
try:
return instance.__dict__[self._name]
except KeyError:
value = self._fget(instance)
instance.__dict__[self._name] = value
return value
def __set_name__(self, klass, name):
self._name = name
class Test(object):
#LazyProperty
def lazy(self):
print('calculating')
return 10
>>> t = Test()
>>> t.lazy
calculating
10
>>> t.lazy
10
These are cases where moving the logic into a common descriptor might make sense, however one could also solve them (but maybe with repeating some code) with other means.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
It depends on how you look up the attribute. If you look up the attribute on an instance then:
the second argument is the instance on which you look up the attribute
the third argument is the class of the instance
In case you look up the attribute on the class (assuming the descriptor is defined on the class):
the second argument is None
the third argument is the class where you look up the attribute
So basically the third argument is necessary if you want to customize the behavior when you do class-level look-up (because the instance is None).
How would I call/use this example?
Your example is basically a property that only allows values that can be converted to float and that is shared between all instances of the class (and on the class - although one can only use "read" access on the class otherwise you would replace the descriptor instance):
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius = 20 # setting it on one instance
>>> t2.celsius # looking it up on another instance
20.0
>>> Temperature.celsius # looking it up on the class
20.0
That's why descriptors generally use the second argument (instance) to store the value to avoid sharing it. However in some cases sharing a value between instances might be desired (although I cannot think of a scenario at this moment). However it makes practically no sense for a celsius property on a temperature class... except maybe as purely academic exercise.

Why do I need the descriptor class?
Inspired by Fluent Python by Buciano Ramalho
Imaging you have a class like this
class LineItem:
price = 10.9
weight = 2.1
def __init__(self, name, price, weight):
self.name = name
self.price = price
self.weight = weight
item = LineItem("apple", 2.9, 2.1)
item.price = -0.9 # it's price is negative, you need to refund to your customer even you delivered the apple :(
item.weight = -0.8 # negative weight, it doesn't make sense
We should validate the weight and price in avoid to assign them a negative number, we can write less code if we use descriptor as a proxy as this
class Quantity(object):
__index = 0
def __init__(self):
self.__index = self.__class__.__index
self._storage_name = "quantity#{}".format(self.__index)
self.__class__.__index += 1
def __set__(self, instance, value):
if value > 0:
setattr(instance, self._storage_name, value)
else:
raise ValueError('value should >0')
def __get__(self, instance, owner):
return getattr(instance, self._storage_name)
then define class LineItem like this:
class LineItem(object):
weight = Quantity()
price = Quantity()
def __init__(self, name, weight, price):
self.name = name
self.weight = weight
self.price = price
and we can extend the Quantity class to do more common validating

You'd see https://docs.python.org/3/howto/descriptor.html#properties
class Property(object):
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)

Easy to digest (with example) Explanation for __get__ & __set__ & __call__ in classes, what is Owner, Instance?
Some points to mug up before diving in:
__get__ __set__ are called descriptors of the class to work/save their internal attributes namely: __name__ (name of class/owner class), variables - __dict__ etc. I will explain what is an owner later
Descriptors are used in design patterers more commonly, for example, with decorators (to abstract things out). You can consider it's more often used in software architecture design to make things less redundant and more readable (seems ironical). Thus abiding SOLID and DRY principles.
If you are not designing software that should abide by SOLID and DRY principles, you probably don't need them, but it's always wise to understand them.
1. Conside this code:
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints:TypeError: __call__() missing 1 required positional argument: 'arg2'
So, when instance.method("first", "second") is called, __call__ method is called from the Method class (call method makes a class object just callable like a function - whenever a class instance is called __call__ gets instiantiated), and following arguments are assigned: instance: "first", arg1: "second", and the last arg2 is left out, this prints out the error: TypeError: __call__() missing 1 required positional argument: 'arg2'
2. how to solve it?
Since __call__ takes instance as first argument (instance, arg1, arg2), but instance of what?
Instance is the instance of main class (MyClass) which is calling the descriptor class (Method). So, instance = MyClass() is the instance and so who is the owner? the class holding the discriptor class - MyClass, However, there is no method in our descriptor class (Method) to recognise it as an instance. So that is where we need __get__ method. Again consider the code below:
from types import MethodType
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
def __set__(self, instance, value):
self.value = value
instance.__dict__["method"] = value
def __get__(self, instance, owner):
if instance is None:
return self
print (instance, owner)
return MethodType(self, instance)
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints: Internal call: <__main__.MyClass object at 0x7fb7dd989690> called with first and second
forget about set for now according to docs:
__get__ "Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access)."
if you do: instance.method.__get__(instance)
Prints:<__main__.MyClass object at 0x7fb7dd9eab90> <class '__main__.MyClass'>
this means instance: object of MyClass which is instance
and Owner is MyClass itself
3. __set__ Explaination:
__set__ is used to set some value in the class __dict__ object (let's say using a command line). command for setting the internal value for set is: instance.descriptor = 'value' # where descriptor is method in this case
(instance.__dict__["method"] = value in the code just update the __dict__ object of the descriptor)
So do: instance.method = 'value' now to check if the value = 'value' is set in the __set__ method we can access __dict__ object of the descriptor method.
Do:
instance.method.__dict__ prints: {'_name': 'Internal call', 'value': 'value'}
Or you can check the __dict__ value using vars(instance.method)
prints: {'name': 'Internal call', 'value': 'value'}
I hope things are clear now:)

I tried (with minor changes as suggested) the code from Andrew Cooke's answer. (I am running python 2.7).
The code:
#!/usr/bin/env python
class Celsius:
def __get__(self, instance, owner): return 9 * (instance.fahrenheit + 32) / 5.0
def __set__(self, instance, value): instance.fahrenheit = 32 + 5 * value / 9.0
class Temperature:
def __init__(self, initial_f): self.fahrenheit = initial_f
celsius = Celsius()
if __name__ == "__main__":
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
The result:
C:\Users\gkuhn\Desktop>python test2.py
<__main__.Celsius instance at 0x02E95A80>
212
With Python prior to 3, make sure you subclass from object which will make the descriptor work correctly as the get magic does not work for old style classes.

Why overriding the `get` method of a class in Python affects what is returned? [duplicate]

I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
What is instance and owner here? (in __get__). What is the purpose of these parameters?
How would I call/use this example?

The descriptor is how Python's property type is implemented. A descriptor simply implements __get__, __set__, etc. and is then added to another class in its definition (as you did above with the Temperature class). For example:
temp=Temperature()
temp.celsius #calls celsius.__get__
Accessing the property you assigned the descriptor to (celsius in the above example) calls the appropriate descriptor method.
instance in __get__ is the instance of the class (so above, __get__ would receive temp, while owner is the class with the descriptor (so it would be Temperature).
You need to use a descriptor class to encapsulate the logic that powers it. That way, if the descriptor is used to cache some expensive operation (for example), it could store the value on itself and not its class.
An article about descriptors can be found here.
EDIT: As jchl pointed out in the comments, if you simply try Temperature.celsius, instance will be None.

Why do I need the descriptor class?
It gives you extra control over how attributes work. If you're used to getters and setters in Java, for example, then it's Python's way of doing that. One advantage is that it looks to users just like an attribute (there's no change in syntax). So you can start with an ordinary attribute and then, when you need to do something fancy, switch to a descriptor.
An attribute is just a mutable value. A descriptor lets you execute arbitrary code when reading or setting (or deleting) a value. So you could imagine using it to map an attribute to a field in a database, for example – a kind of ORM.
Another use might be refusing to accept a new value by throwing an exception in __set__ – effectively making the "attribute" read only.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
This is pretty subtle (and the reason I am writing a new answer here - I found this question while wondering the same thing and didn't find the existing answer that great).
A descriptor is defined on a class, but is typically called from an instance. When it's called from an instance both instance and owner are set (and you can work out owner from instance so it seems kinda pointless). But when called from a class, only owner is set – which is why it's there.
This is only needed for __get__ because it's the only one that can be called on a class. If you set the class value you set the descriptor itself. Similarly for deletion. Which is why the owner isn't needed there.
How would I call/use this example?
Well, here's a cool trick using similar classes:
class Celsius:
def __get__(self, instance, owner):
return 5 * (instance.fahrenheit - 32) / 9
def __set__(self, instance, value):
instance.fahrenheit = 32 + 9 * value / 5
class Temperature:
celsius = Celsius()
def __init__(self, initial_f):
self.fahrenheit = initial_f
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
(I'm using Python 3; for python 2 you need to make sure those divisions are / 5.0 and / 9.0). That gives:
100.0
32.0
Now there are other, arguably better ways to achieve the same effect in python (e.g. if celsius were a property, which is the same basic mechanism but places all the source inside the Temperature class), but that shows what can be done...

I am trying to understand what Python's descriptors are and what they can be useful for.
Descriptors are objects in a class namespace that manage instance attributes (like slots, properties, or methods). For example:
class HasDescriptors:
__slots__ = 'a_slot' # creates a descriptor
def a_method(self): # creates a descriptor
"a regular method"
#staticmethod # creates a descriptor
def a_static_method():
"a static method"
#classmethod # creates a descriptor
def a_class_method(cls):
"a class method"
#property # creates a descriptor
def a_property(self):
"a property"
# even a regular function:
def a_function(some_obj_or_self): # creates a descriptor
"create a function suitable for monkey patching"
HasDescriptors.a_function = a_function # (but we usually don't do this)
Pedantically, descriptors are objects with any of the following special methods, which may be known as "descriptor methods":
__get__: non-data descriptor method, for example on a method/function
__set__: data descriptor method, for example on a property instance or slot
__delete__: data descriptor method, again used by properties or slots
These descriptor objects are attributes in other object class namespaces. That is, they live in the __dict__ of the class object.
Descriptor objects programmatically manage the results of a dotted lookup (e.g. foo.descriptor) in a normal expression, an assignment, or a deletion.
Functions/methods, bound methods, property, classmethod, and staticmethod all use these special methods to control how they are accessed via the dotted lookup.
A data descriptor, like property, can allow for lazy evaluation of attributes based on a simpler state of the object, allowing instances to use less memory than if you precomputed each possible attribute.
Another data descriptor, a member_descriptor created by __slots__, allows memory savings (and faster lookups) by having the class store data in a mutable tuple-like datastructure instead of the more flexible but space-consuming __dict__.
Non-data descriptors, instance and class methods, get their implicit first arguments (usually named self and cls, respectively) from their non-data descriptor method, __get__ - and this is how static methods know not to have an implicit first argument.
Most users of Python need to learn only the high-level usage of descriptors, and have no need to learn or understand the implementation of descriptors further.
But understanding how descriptors work can give one greater confidence in one's mastery of Python.
In Depth: What Are Descriptors?
A descriptor is an object with any of the following methods (__get__, __set__, or __delete__), intended to be used via dotted-lookup as if it were a typical attribute of an instance. For an owner-object, obj_instance, with a descriptor object:
obj_instance.descriptor invokes
descriptor.__get__(self, obj_instance, owner_class) returning a value
This is how all methods and the get on a property work.
obj_instance.descriptor = value invokes
descriptor.__set__(self, obj_instance, value) returning None
This is how the setter on a property works.
del obj_instance.descriptor invokes
descriptor.__delete__(self, obj_instance) returning None
This is how the deleter on a property works.
obj_instance is the instance whose class contains the descriptor object's instance. self is the instance of the descriptor (probably just one for the class of the obj_instance)
To define this with code, an object is a descriptor if the set of its attributes intersects with any of the required attributes:
def has_descriptor_attrs(obj):
return set(['__get__', '__set__', '__delete__']).intersection(dir(obj))
def is_descriptor(obj):
"""obj can be instance of descriptor or the descriptor class"""
return bool(has_descriptor_attrs(obj))
A Data Descriptor has a __set__ and/or __delete__.
A Non-Data-Descriptor has neither __set__ nor __delete__.
def has_data_descriptor_attrs(obj):
return set(['__set__', '__delete__']) & set(dir(obj))
def is_data_descriptor(obj):
return bool(has_data_descriptor_attrs(obj))
Builtin Descriptor Object Examples:
classmethod
staticmethod
property
functions in general
Non-Data Descriptors
We can see that classmethod and staticmethod are Non-Data-Descriptors:
>>> is_descriptor(classmethod), is_data_descriptor(classmethod)
(True, False)
>>> is_descriptor(staticmethod), is_data_descriptor(staticmethod)
(True, False)
Both only have the __get__ method:
>>> has_descriptor_attrs(classmethod), has_descriptor_attrs(staticmethod)
(set(['__get__']), set(['__get__']))
Note that all functions are also Non-Data-Descriptors:
>>> def foo(): pass
...
>>> is_descriptor(foo), is_data_descriptor(foo)
(True, False)
Data Descriptor, property
However, property is a Data-Descriptor:
>>> is_data_descriptor(property)
True
>>> has_descriptor_attrs(property)
set(['__set__', '__get__', '__delete__'])
Dotted Lookup Order
These are important distinctions, as they affect the lookup order for a dotted lookup.
obj_instance.attribute
First the above looks to see if the attribute is a Data-Descriptor on the class of the instance,
If not, it looks to see if the attribute is in the obj_instance's __dict__, then
it finally falls back to a Non-Data-Descriptor.
The consequence of this lookup order is that Non-Data-Descriptors like functions/methods can be overridden by instances.
Recap and Next Steps
We have learned that descriptors are objects with any of __get__, __set__, or __delete__. These descriptor objects can be used as attributes on other object class definitions. Now we will look at how they are used, using your code as an example.
Analysis of Code from the Question
Here's your code, followed by your questions and answers to each:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
Your descriptor ensures you always have a float for this class attribute of Temperature, and that you can't use del to delete the attribute:
>>> t1 = Temperature()
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
Otherwise, your descriptors ignore the owner-class and instances of the owner, instead, storing state in the descriptor. You could just as easily share state across all instances with a simple class attribute (so long as you always set it as a float to the class and never delete it, or are comfortable with users of your code doing so):
class Temperature(object):
celsius = 0.0
This gets you exactly the same behavior as your example (see response to question 3 below), but uses a Pythons builtin (property), and would be considered more idiomatic:
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
What is instance and owner here? (in get). What is the purpose of these parameters?
instance is the instance of the owner that is calling the descriptor. The owner is the class in which the descriptor object is used to manage access to the data point. See the descriptions of the special methods that define descriptors next to the first paragraph of this answer for more descriptive variable names.
How would I call/use this example?
Here's a demonstration:
>>> t1 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1
>>>
>>> t1.celsius
1.0
>>> t2 = Temperature()
>>> t2.celsius
1.0
You can't delete the attribute:
>>> del t2.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
And you can't assign a variable that can't be converted to a float:
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __set__
ValueError: invalid literal for float(): 0x02
Otherwise, what you have here is a global state for all instances, that is managed by assigning to any instance.
The expected way that most experienced Python programmers would accomplish this outcome would be to use the property decorator, which makes use of the same descriptors under the hood, but brings the behavior into the implementation of the owner class (again, as defined above):
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
Which has the exact same expected behavior of the original piece of code:
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1.0
>>> t2.celsius
1.0
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't delete attribute
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in celsius
ValueError: invalid literal for float(): 0x02
Conclusion
We've covered the attributes that define descriptors, the difference between data- and non-data-descriptors, builtin objects that use them, and specific questions about use.
So again, how would you use the question's example? I hope you wouldn't. I hope you would start with my first suggestion (a simple class attribute) and move on to the second suggestion (the property decorator) if you feel it is necessary.

Before going into the details of descriptors it may be important to know how attribute lookup in Python works. This assumes that the class has no metaclass and that it uses the default implementation of __getattribute__ (both can be used to "customize" the behavior).
The best illustration of attribute lookup (in Python 3.x or for new-style classes in Python 2.x) in this case is from Understanding Python metaclasses (ionel's codelog). The image uses : as substitute for "non-customizable attribute lookup".
This represents the lookup of an attribute foobar on an instance of Class:
Two conditions are important here:
If the class of instance has an entry for the attribute name and it has __get__ and __set__.
If the instance has no entry for the attribute name but the class has one and it has __get__.
That's where descriptors come into it:
Data descriptors which have both __get__ and __set__.
Non-data descriptors which only have __get__.
In both cases the returned value goes through __get__ called with the instance as first argument and the class as second argument.
The lookup is even more complicated for class attribute lookup (see for example Class attribute lookup (in the above mentioned blog)).
Let's move to your specific questions:
Why do I need the descriptor class?
In most cases you don't need to write descriptor classes! However you're probably a very regular end user. For example functions. Functions are descriptors, that's how functions can be used as methods with self implicitly passed as first argument.
def test_function(self):
return self
class TestClass(object):
def test_method(self):
...
If you look up test_method on an instance you'll get back a "bound method":
>>> instance = TestClass()
>>> instance.test_method
<bound method TestClass.test_method of <__main__.TestClass object at ...>>
Similarly you could also bind a function by invoking its __get__ method manually (not really recommended, just for illustrative purposes):
>>> test_function.__get__(instance, TestClass)
<bound method test_function of <__main__.TestClass object at ...>>
You can even call this "self-bound method":
>>> test_function.__get__(instance, TestClass)()
<__main__.TestClass at ...>
Note that I did not provide any arguments and the function did return the instance I had bound!
Functions are Non-data descriptors!
Some built-in examples of a data-descriptor would be property. Neglecting getter, setter, and deleter the property descriptor is (from Descriptor HowTo Guide "Properties"):
class Property(object):
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
Since it's a data descriptor it's invoked whenever you look up the "name" of the property and it simply delegates to the functions decorated with #property, #name.setter, and #name.deleter (if present).
There are several other descriptors in the standard library, for example staticmethod, classmethod.
The point of descriptors is easy (although you rarely need them): Abstract common code for attribute access. property is an abstraction for instance variable access, function provides an abstraction for methods, staticmethod provides an abstraction for methods that don't need instance access and classmethod provides an abstraction for methods that need class access rather than instance access (this is a bit simplified).
Another example would be a class property.
One fun example (using __set_name__ from Python 3.6) could also be a property that only allows a specific type:
class TypedProperty(object):
__slots__ = ('_name', '_type')
def __init__(self, typ):
self._type = typ
def __get__(self, instance, klass=None):
if instance is None:
return self
return instance.__dict__[self._name]
def __set__(self, instance, value):
if not isinstance(value, self._type):
raise TypeError(f"Expected class {self._type}, got {type(value)}")
instance.__dict__[self._name] = value
def __delete__(self, instance):
del instance.__dict__[self._name]
def __set_name__(self, klass, name):
self._name = name
Then you can use the descriptor in a class:
class Test(object):
int_prop = TypedProperty(int)
And playing a bit with it:
>>> t = Test()
>>> t.int_prop = 10
>>> t.int_prop
10
>>> t.int_prop = 20.0
TypeError: Expected class <class 'int'>, got <class 'float'>
Or a "lazy property":
class LazyProperty(object):
__slots__ = ('_fget', '_name')
def __init__(self, fget):
self._fget = fget
def __get__(self, instance, klass=None):
if instance is None:
return self
try:
return instance.__dict__[self._name]
except KeyError:
value = self._fget(instance)
instance.__dict__[self._name] = value
return value
def __set_name__(self, klass, name):
self._name = name
class Test(object):
#LazyProperty
def lazy(self):
print('calculating')
return 10
>>> t = Test()
>>> t.lazy
calculating
10
>>> t.lazy
10
These are cases where moving the logic into a common descriptor might make sense, however one could also solve them (but maybe with repeating some code) with other means.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
It depends on how you look up the attribute. If you look up the attribute on an instance then:
the second argument is the instance on which you look up the attribute
the third argument is the class of the instance
In case you look up the attribute on the class (assuming the descriptor is defined on the class):
the second argument is None
the third argument is the class where you look up the attribute
So basically the third argument is necessary if you want to customize the behavior when you do class-level look-up (because the instance is None).
How would I call/use this example?
Your example is basically a property that only allows values that can be converted to float and that is shared between all instances of the class (and on the class - although one can only use "read" access on the class otherwise you would replace the descriptor instance):
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius = 20 # setting it on one instance
>>> t2.celsius # looking it up on another instance
20.0
>>> Temperature.celsius # looking it up on the class
20.0
That's why descriptors generally use the second argument (instance) to store the value to avoid sharing it. However in some cases sharing a value between instances might be desired (although I cannot think of a scenario at this moment). However it makes practically no sense for a celsius property on a temperature class... except maybe as purely academic exercise.

Why do I need the descriptor class?
Inspired by Fluent Python by Buciano Ramalho
Imaging you have a class like this
class LineItem:
price = 10.9
weight = 2.1
def __init__(self, name, price, weight):
self.name = name
self.price = price
self.weight = weight
item = LineItem("apple", 2.9, 2.1)
item.price = -0.9 # it's price is negative, you need to refund to your customer even you delivered the apple :(
item.weight = -0.8 # negative weight, it doesn't make sense
We should validate the weight and price in avoid to assign them a negative number, we can write less code if we use descriptor as a proxy as this
class Quantity(object):
__index = 0
def __init__(self):
self.__index = self.__class__.__index
self._storage_name = "quantity#{}".format(self.__index)
self.__class__.__index += 1
def __set__(self, instance, value):
if value > 0:
setattr(instance, self._storage_name, value)
else:
raise ValueError('value should >0')
def __get__(self, instance, owner):
return getattr(instance, self._storage_name)
then define class LineItem like this:
class LineItem(object):
weight = Quantity()
price = Quantity()
def __init__(self, name, weight, price):
self.name = name
self.weight = weight
self.price = price
and we can extend the Quantity class to do more common validating

You'd see https://docs.python.org/3/howto/descriptor.html#properties
class Property(object):
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)

Easy to digest (with example) Explanation for __get__ & __set__ & __call__ in classes, what is Owner, Instance?
Some points to mug up before diving in:
__get__ __set__ are called descriptors of the class to work/save their internal attributes namely: __name__ (name of class/owner class), variables - __dict__ etc. I will explain what is an owner later
Descriptors are used in design patterers more commonly, for example, with decorators (to abstract things out). You can consider it's more often used in software architecture design to make things less redundant and more readable (seems ironical). Thus abiding SOLID and DRY principles.
If you are not designing software that should abide by SOLID and DRY principles, you probably don't need them, but it's always wise to understand them.
1. Conside this code:
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints:TypeError: __call__() missing 1 required positional argument: 'arg2'
So, when instance.method("first", "second") is called, __call__ method is called from the Method class (call method makes a class object just callable like a function - whenever a class instance is called __call__ gets instiantiated), and following arguments are assigned: instance: "first", arg1: "second", and the last arg2 is left out, this prints out the error: TypeError: __call__() missing 1 required positional argument: 'arg2'
2. how to solve it?
Since __call__ takes instance as first argument (instance, arg1, arg2), but instance of what?
Instance is the instance of main class (MyClass) which is calling the descriptor class (Method). So, instance = MyClass() is the instance and so who is the owner? the class holding the discriptor class - MyClass, However, there is no method in our descriptor class (Method) to recognise it as an instance. So that is where we need __get__ method. Again consider the code below:
from types import MethodType
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
def __set__(self, instance, value):
self.value = value
instance.__dict__["method"] = value
def __get__(self, instance, owner):
if instance is None:
return self
print (instance, owner)
return MethodType(self, instance)
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints: Internal call: <__main__.MyClass object at 0x7fb7dd989690> called with first and second
forget about set for now according to docs:
__get__ "Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access)."
if you do: instance.method.__get__(instance)
Prints:<__main__.MyClass object at 0x7fb7dd9eab90> <class '__main__.MyClass'>
this means instance: object of MyClass which is instance
and Owner is MyClass itself
3. __set__ Explaination:
__set__ is used to set some value in the class __dict__ object (let's say using a command line). command for setting the internal value for set is: instance.descriptor = 'value' # where descriptor is method in this case
(instance.__dict__["method"] = value in the code just update the __dict__ object of the descriptor)
So do: instance.method = 'value' now to check if the value = 'value' is set in the __set__ method we can access __dict__ object of the descriptor method.
Do:
instance.method.__dict__ prints: {'_name': 'Internal call', 'value': 'value'}
Or you can check the __dict__ value using vars(instance.method)
prints: {'name': 'Internal call', 'value': 'value'}
I hope things are clear now:)

I tried (with minor changes as suggested) the code from Andrew Cooke's answer. (I am running python 2.7).
The code:
#!/usr/bin/env python
class Celsius:
def __get__(self, instance, owner): return 9 * (instance.fahrenheit + 32) / 5.0
def __set__(self, instance, value): instance.fahrenheit = 32 + 5 * value / 9.0
class Temperature:
def __init__(self, initial_f): self.fahrenheit = initial_f
celsius = Celsius()
if __name__ == "__main__":
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
The result:
C:\Users\gkuhn\Desktop>python test2.py
<__main__.Celsius instance at 0x02E95A80>
212
With Python prior to 3, make sure you subclass from object which will make the descriptor work correctly as the get magic does not work for old style classes.

Need help understanding a piece of code with class decorators and descriptors [duplicate]

I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
What is instance and owner here? (in __get__). What is the purpose of these parameters?
How would I call/use this example?

The descriptor is how Python's property type is implemented. A descriptor simply implements __get__, __set__, etc. and is then added to another class in its definition (as you did above with the Temperature class). For example:
temp=Temperature()
temp.celsius #calls celsius.__get__
Accessing the property you assigned the descriptor to (celsius in the above example) calls the appropriate descriptor method.
instance in __get__ is the instance of the class (so above, __get__ would receive temp, while owner is the class with the descriptor (so it would be Temperature).
You need to use a descriptor class to encapsulate the logic that powers it. That way, if the descriptor is used to cache some expensive operation (for example), it could store the value on itself and not its class.
An article about descriptors can be found here.
EDIT: As jchl pointed out in the comments, if you simply try Temperature.celsius, instance will be None.

Why do I need the descriptor class?
It gives you extra control over how attributes work. If you're used to getters and setters in Java, for example, then it's Python's way of doing that. One advantage is that it looks to users just like an attribute (there's no change in syntax). So you can start with an ordinary attribute and then, when you need to do something fancy, switch to a descriptor.
An attribute is just a mutable value. A descriptor lets you execute arbitrary code when reading or setting (or deleting) a value. So you could imagine using it to map an attribute to a field in a database, for example – a kind of ORM.
Another use might be refusing to accept a new value by throwing an exception in __set__ – effectively making the "attribute" read only.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
This is pretty subtle (and the reason I am writing a new answer here - I found this question while wondering the same thing and didn't find the existing answer that great).
A descriptor is defined on a class, but is typically called from an instance. When it's called from an instance both instance and owner are set (and you can work out owner from instance so it seems kinda pointless). But when called from a class, only owner is set – which is why it's there.
This is only needed for __get__ because it's the only one that can be called on a class. If you set the class value you set the descriptor itself. Similarly for deletion. Which is why the owner isn't needed there.
How would I call/use this example?
Well, here's a cool trick using similar classes:
class Celsius:
def __get__(self, instance, owner):
return 5 * (instance.fahrenheit - 32) / 9
def __set__(self, instance, value):
instance.fahrenheit = 32 + 9 * value / 5
class Temperature:
celsius = Celsius()
def __init__(self, initial_f):
self.fahrenheit = initial_f
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
(I'm using Python 3; for python 2 you need to make sure those divisions are / 5.0 and / 9.0). That gives:
100.0
32.0
Now there are other, arguably better ways to achieve the same effect in python (e.g. if celsius were a property, which is the same basic mechanism but places all the source inside the Temperature class), but that shows what can be done...

I am trying to understand what Python's descriptors are and what they can be useful for.
Descriptors are objects in a class namespace that manage instance attributes (like slots, properties, or methods). For example:
class HasDescriptors:
__slots__ = 'a_slot' # creates a descriptor
def a_method(self): # creates a descriptor
"a regular method"
#staticmethod # creates a descriptor
def a_static_method():
"a static method"
#classmethod # creates a descriptor
def a_class_method(cls):
"a class method"
#property # creates a descriptor
def a_property(self):
"a property"
# even a regular function:
def a_function(some_obj_or_self): # creates a descriptor
"create a function suitable for monkey patching"
HasDescriptors.a_function = a_function # (but we usually don't do this)
Pedantically, descriptors are objects with any of the following special methods, which may be known as "descriptor methods":
__get__: non-data descriptor method, for example on a method/function
__set__: data descriptor method, for example on a property instance or slot
__delete__: data descriptor method, again used by properties or slots
These descriptor objects are attributes in other object class namespaces. That is, they live in the __dict__ of the class object.
Descriptor objects programmatically manage the results of a dotted lookup (e.g. foo.descriptor) in a normal expression, an assignment, or a deletion.
Functions/methods, bound methods, property, classmethod, and staticmethod all use these special methods to control how they are accessed via the dotted lookup.
A data descriptor, like property, can allow for lazy evaluation of attributes based on a simpler state of the object, allowing instances to use less memory than if you precomputed each possible attribute.
Another data descriptor, a member_descriptor created by __slots__, allows memory savings (and faster lookups) by having the class store data in a mutable tuple-like datastructure instead of the more flexible but space-consuming __dict__.
Non-data descriptors, instance and class methods, get their implicit first arguments (usually named self and cls, respectively) from their non-data descriptor method, __get__ - and this is how static methods know not to have an implicit first argument.
Most users of Python need to learn only the high-level usage of descriptors, and have no need to learn or understand the implementation of descriptors further.
But understanding how descriptors work can give one greater confidence in one's mastery of Python.
In Depth: What Are Descriptors?
A descriptor is an object with any of the following methods (__get__, __set__, or __delete__), intended to be used via dotted-lookup as if it were a typical attribute of an instance. For an owner-object, obj_instance, with a descriptor object:
obj_instance.descriptor invokes
descriptor.__get__(self, obj_instance, owner_class) returning a value
This is how all methods and the get on a property work.
obj_instance.descriptor = value invokes
descriptor.__set__(self, obj_instance, value) returning None
This is how the setter on a property works.
del obj_instance.descriptor invokes
descriptor.__delete__(self, obj_instance) returning None
This is how the deleter on a property works.
obj_instance is the instance whose class contains the descriptor object's instance. self is the instance of the descriptor (probably just one for the class of the obj_instance)
To define this with code, an object is a descriptor if the set of its attributes intersects with any of the required attributes:
def has_descriptor_attrs(obj):
return set(['__get__', '__set__', '__delete__']).intersection(dir(obj))
def is_descriptor(obj):
"""obj can be instance of descriptor or the descriptor class"""
return bool(has_descriptor_attrs(obj))
A Data Descriptor has a __set__ and/or __delete__.
A Non-Data-Descriptor has neither __set__ nor __delete__.
def has_data_descriptor_attrs(obj):
return set(['__set__', '__delete__']) & set(dir(obj))
def is_data_descriptor(obj):
return bool(has_data_descriptor_attrs(obj))
Builtin Descriptor Object Examples:
classmethod
staticmethod
property
functions in general
Non-Data Descriptors
We can see that classmethod and staticmethod are Non-Data-Descriptors:
>>> is_descriptor(classmethod), is_data_descriptor(classmethod)
(True, False)
>>> is_descriptor(staticmethod), is_data_descriptor(staticmethod)
(True, False)
Both only have the __get__ method:
>>> has_descriptor_attrs(classmethod), has_descriptor_attrs(staticmethod)
(set(['__get__']), set(['__get__']))
Note that all functions are also Non-Data-Descriptors:
>>> def foo(): pass
...
>>> is_descriptor(foo), is_data_descriptor(foo)
(True, False)
Data Descriptor, property
However, property is a Data-Descriptor:
>>> is_data_descriptor(property)
True
>>> has_descriptor_attrs(property)
set(['__set__', '__get__', '__delete__'])
Dotted Lookup Order
These are important distinctions, as they affect the lookup order for a dotted lookup.
obj_instance.attribute
First the above looks to see if the attribute is a Data-Descriptor on the class of the instance,
If not, it looks to see if the attribute is in the obj_instance's __dict__, then
it finally falls back to a Non-Data-Descriptor.
The consequence of this lookup order is that Non-Data-Descriptors like functions/methods can be overridden by instances.
Recap and Next Steps
We have learned that descriptors are objects with any of __get__, __set__, or __delete__. These descriptor objects can be used as attributes on other object class definitions. Now we will look at how they are used, using your code as an example.
Analysis of Code from the Question
Here's your code, followed by your questions and answers to each:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
Your descriptor ensures you always have a float for this class attribute of Temperature, and that you can't use del to delete the attribute:
>>> t1 = Temperature()
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
Otherwise, your descriptors ignore the owner-class and instances of the owner, instead, storing state in the descriptor. You could just as easily share state across all instances with a simple class attribute (so long as you always set it as a float to the class and never delete it, or are comfortable with users of your code doing so):
class Temperature(object):
celsius = 0.0
This gets you exactly the same behavior as your example (see response to question 3 below), but uses a Pythons builtin (property), and would be considered more idiomatic:
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
What is instance and owner here? (in get). What is the purpose of these parameters?
instance is the instance of the owner that is calling the descriptor. The owner is the class in which the descriptor object is used to manage access to the data point. See the descriptions of the special methods that define descriptors next to the first paragraph of this answer for more descriptive variable names.
How would I call/use this example?
Here's a demonstration:
>>> t1 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1
>>>
>>> t1.celsius
1.0
>>> t2 = Temperature()
>>> t2.celsius
1.0
You can't delete the attribute:
>>> del t2.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
And you can't assign a variable that can't be converted to a float:
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __set__
ValueError: invalid literal for float(): 0x02
Otherwise, what you have here is a global state for all instances, that is managed by assigning to any instance.
The expected way that most experienced Python programmers would accomplish this outcome would be to use the property decorator, which makes use of the same descriptors under the hood, but brings the behavior into the implementation of the owner class (again, as defined above):
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
Which has the exact same expected behavior of the original piece of code:
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1.0
>>> t2.celsius
1.0
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't delete attribute
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in celsius
ValueError: invalid literal for float(): 0x02
Conclusion
We've covered the attributes that define descriptors, the difference between data- and non-data-descriptors, builtin objects that use them, and specific questions about use.
So again, how would you use the question's example? I hope you wouldn't. I hope you would start with my first suggestion (a simple class attribute) and move on to the second suggestion (the property decorator) if you feel it is necessary.

Before going into the details of descriptors it may be important to know how attribute lookup in Python works. This assumes that the class has no metaclass and that it uses the default implementation of __getattribute__ (both can be used to "customize" the behavior).
The best illustration of attribute lookup (in Python 3.x or for new-style classes in Python 2.x) in this case is from Understanding Python metaclasses (ionel's codelog). The image uses : as substitute for "non-customizable attribute lookup".
This represents the lookup of an attribute foobar on an instance of Class:
Two conditions are important here:
If the class of instance has an entry for the attribute name and it has __get__ and __set__.
If the instance has no entry for the attribute name but the class has one and it has __get__.
That's where descriptors come into it:
Data descriptors which have both __get__ and __set__.
Non-data descriptors which only have __get__.
In both cases the returned value goes through __get__ called with the instance as first argument and the class as second argument.
The lookup is even more complicated for class attribute lookup (see for example Class attribute lookup (in the above mentioned blog)).
Let's move to your specific questions:
Why do I need the descriptor class?
In most cases you don't need to write descriptor classes! However you're probably a very regular end user. For example functions. Functions are descriptors, that's how functions can be used as methods with self implicitly passed as first argument.
def test_function(self):
return self
class TestClass(object):
def test_method(self):
...
If you look up test_method on an instance you'll get back a "bound method":
>>> instance = TestClass()
>>> instance.test_method
<bound method TestClass.test_method of <__main__.TestClass object at ...>>
Similarly you could also bind a function by invoking its __get__ method manually (not really recommended, just for illustrative purposes):
>>> test_function.__get__(instance, TestClass)
<bound method test_function of <__main__.TestClass object at ...>>
You can even call this "self-bound method":
>>> test_function.__get__(instance, TestClass)()
<__main__.TestClass at ...>
Note that I did not provide any arguments and the function did return the instance I had bound!
Functions are Non-data descriptors!
Some built-in examples of a data-descriptor would be property. Neglecting getter, setter, and deleter the property descriptor is (from Descriptor HowTo Guide "Properties"):
class Property(object):
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
Since it's a data descriptor it's invoked whenever you look up the "name" of the property and it simply delegates to the functions decorated with #property, #name.setter, and #name.deleter (if present).
There are several other descriptors in the standard library, for example staticmethod, classmethod.
The point of descriptors is easy (although you rarely need them): Abstract common code for attribute access. property is an abstraction for instance variable access, function provides an abstraction for methods, staticmethod provides an abstraction for methods that don't need instance access and classmethod provides an abstraction for methods that need class access rather than instance access (this is a bit simplified).
Another example would be a class property.
One fun example (using __set_name__ from Python 3.6) could also be a property that only allows a specific type:
class TypedProperty(object):
__slots__ = ('_name', '_type')
def __init__(self, typ):
self._type = typ
def __get__(self, instance, klass=None):
if instance is None:
return self
return instance.__dict__[self._name]
def __set__(self, instance, value):
if not isinstance(value, self._type):
raise TypeError(f"Expected class {self._type}, got {type(value)}")
instance.__dict__[self._name] = value
def __delete__(self, instance):
del instance.__dict__[self._name]
def __set_name__(self, klass, name):
self._name = name
Then you can use the descriptor in a class:
class Test(object):
int_prop = TypedProperty(int)
And playing a bit with it:
>>> t = Test()
>>> t.int_prop = 10
>>> t.int_prop
10
>>> t.int_prop = 20.0
TypeError: Expected class <class 'int'>, got <class 'float'>
Or a "lazy property":
class LazyProperty(object):
__slots__ = ('_fget', '_name')
def __init__(self, fget):
self._fget = fget
def __get__(self, instance, klass=None):
if instance is None:
return self
try:
return instance.__dict__[self._name]
except KeyError:
value = self._fget(instance)
instance.__dict__[self._name] = value
return value
def __set_name__(self, klass, name):
self._name = name
class Test(object):
#LazyProperty
def lazy(self):
print('calculating')
return 10
>>> t = Test()
>>> t.lazy
calculating
10
>>> t.lazy
10
These are cases where moving the logic into a common descriptor might make sense, however one could also solve them (but maybe with repeating some code) with other means.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
It depends on how you look up the attribute. If you look up the attribute on an instance then:
the second argument is the instance on which you look up the attribute
the third argument is the class of the instance
In case you look up the attribute on the class (assuming the descriptor is defined on the class):
the second argument is None
the third argument is the class where you look up the attribute
So basically the third argument is necessary if you want to customize the behavior when you do class-level look-up (because the instance is None).
How would I call/use this example?
Your example is basically a property that only allows values that can be converted to float and that is shared between all instances of the class (and on the class - although one can only use "read" access on the class otherwise you would replace the descriptor instance):
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius = 20 # setting it on one instance
>>> t2.celsius # looking it up on another instance
20.0
>>> Temperature.celsius # looking it up on the class
20.0
That's why descriptors generally use the second argument (instance) to store the value to avoid sharing it. However in some cases sharing a value between instances might be desired (although I cannot think of a scenario at this moment). However it makes practically no sense for a celsius property on a temperature class... except maybe as purely academic exercise.

Why do I need the descriptor class?
Inspired by Fluent Python by Buciano Ramalho
Imaging you have a class like this
class LineItem:
price = 10.9
weight = 2.1
def __init__(self, name, price, weight):
self.name = name
self.price = price
self.weight = weight
item = LineItem("apple", 2.9, 2.1)
item.price = -0.9 # it's price is negative, you need to refund to your customer even you delivered the apple :(
item.weight = -0.8 # negative weight, it doesn't make sense
We should validate the weight and price in avoid to assign them a negative number, we can write less code if we use descriptor as a proxy as this
class Quantity(object):
__index = 0
def __init__(self):
self.__index = self.__class__.__index
self._storage_name = "quantity#{}".format(self.__index)
self.__class__.__index += 1
def __set__(self, instance, value):
if value > 0:
setattr(instance, self._storage_name, value)
else:
raise ValueError('value should >0')
def __get__(self, instance, owner):
return getattr(instance, self._storage_name)
then define class LineItem like this:
class LineItem(object):
weight = Quantity()
price = Quantity()
def __init__(self, name, weight, price):
self.name = name
self.weight = weight
self.price = price
and we can extend the Quantity class to do more common validating

You'd see https://docs.python.org/3/howto/descriptor.html#properties
class Property(object):
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)

Easy to digest (with example) Explanation for __get__ & __set__ & __call__ in classes, what is Owner, Instance?
Some points to mug up before diving in:
__get__ __set__ are called descriptors of the class to work/save their internal attributes namely: __name__ (name of class/owner class), variables - __dict__ etc. I will explain what is an owner later
Descriptors are used in design patterers more commonly, for example, with decorators (to abstract things out). You can consider it's more often used in software architecture design to make things less redundant and more readable (seems ironical). Thus abiding SOLID and DRY principles.
If you are not designing software that should abide by SOLID and DRY principles, you probably don't need them, but it's always wise to understand them.
1. Conside this code:
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints:TypeError: __call__() missing 1 required positional argument: 'arg2'
So, when instance.method("first", "second") is called, __call__ method is called from the Method class (call method makes a class object just callable like a function - whenever a class instance is called __call__ gets instiantiated), and following arguments are assigned: instance: "first", arg1: "second", and the last arg2 is left out, this prints out the error: TypeError: __call__() missing 1 required positional argument: 'arg2'
2. how to solve it?
Since __call__ takes instance as first argument (instance, arg1, arg2), but instance of what?
Instance is the instance of main class (MyClass) which is calling the descriptor class (Method). So, instance = MyClass() is the instance and so who is the owner? the class holding the discriptor class - MyClass, However, there is no method in our descriptor class (Method) to recognise it as an instance. So that is where we need __get__ method. Again consider the code below:
from types import MethodType
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
def __set__(self, instance, value):
self.value = value
instance.__dict__["method"] = value
def __get__(self, instance, owner):
if instance is None:
return self
print (instance, owner)
return MethodType(self, instance)
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints: Internal call: <__main__.MyClass object at 0x7fb7dd989690> called with first and second
forget about set for now according to docs:
__get__ "Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access)."
if you do: instance.method.__get__(instance)
Prints:<__main__.MyClass object at 0x7fb7dd9eab90> <class '__main__.MyClass'>
this means instance: object of MyClass which is instance
and Owner is MyClass itself
3. __set__ Explaination:
__set__ is used to set some value in the class __dict__ object (let's say using a command line). command for setting the internal value for set is: instance.descriptor = 'value' # where descriptor is method in this case
(instance.__dict__["method"] = value in the code just update the __dict__ object of the descriptor)
So do: instance.method = 'value' now to check if the value = 'value' is set in the __set__ method we can access __dict__ object of the descriptor method.
Do:
instance.method.__dict__ prints: {'_name': 'Internal call', 'value': 'value'}
Or you can check the __dict__ value using vars(instance.method)
prints: {'name': 'Internal call', 'value': 'value'}
I hope things are clear now:)

I tried (with minor changes as suggested) the code from Andrew Cooke's answer. (I am running python 2.7).
The code:
#!/usr/bin/env python
class Celsius:
def __get__(self, instance, owner): return 9 * (instance.fahrenheit + 32) / 5.0
def __set__(self, instance, value): instance.fahrenheit = 32 + 5 * value / 9.0
class Temperature:
def __init__(self, initial_f): self.fahrenheit = initial_f
celsius = Celsius()
if __name__ == "__main__":
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
The result:
C:\Users\gkuhn\Desktop>python test2.py
<__main__.Celsius instance at 0x02E95A80>
212
With Python prior to 3, make sure you subclass from object which will make the descriptor work correctly as the get magic does not work for old style classes.

What tests can I perform to ensure I have overridden setattr correctly?

I have been studying Python for a little while now, and I've come to understand that overriding __setattr__ correctly can be troublesome (to say the least!).
What are some effective ways to ensure/prove to myself the override has been done correctly? I'm specifically concerned about ensuring the override remains consistent with the descriptor protocol and MRO.
(Tagged as Python 3.x since that's what I am using, but the question is certainly applicable to other versions as well.)
Example code in which the "override" exhibits default behavior (but how can I prove it?):
class MyClass():
def __setattr__(self,att,val):
print("I am exhibiting default behavior!")
super().__setattr__(att,val)
Contrived example in which the override violates the descriptor protocol (instance storage lookup occurs prior to the descriptor lookup - but how can I test it?):
class MyClass():
def __init__(self,mydict):
self.__dict__['mydict'] = mydict
#property
def mydict(self):
return self._mydict
def __setattr__(self,att,val):
if att in self.mydict:
self.mydict[att] = val
else:
super().__setattr__(att, val)
The ideal answer will provide a general test that will succeed when __setattr__ has been overridden correctly, and fail otherwise.

In this case there's a simple solution: add a binding descriptor with a name that's in mydict and test that assigning to that name goes thru the descriptor (NB : Python 2.x code, I don't have a Python 3 install here):
class MyBindingDescriptor(object):
def __init__(self, key):
self.key = key
def __get__(self, obj, cls=None):
if not obj:
return self
return obj.__dict__[self.key]
def __set__(self, obj, value):
obj.__dict__[self.key] = value
sentinel = object()
class MyClass(object):
test = MyBindingDescriptor("test")
def __init__(self, mydict):
self.__dict__['mydict'] = mydict
self.__dict__["test"] = sentinel
def __setattr__(self, att, val):
if att in self.mydict:
self.mydict[att] = val
else:
super(MyClass, self).__setattr__(att, val)
# first test our binding descriptor
instance1 = MyClass({})
# sanity check
assert instance1.test is sentinel, "instance1.test should be sentinel, got '%s' instead" % instance1.test
# this one should pass ok
instance1.test = NotImplemented
assert instance1.test is NotImplemented, "instance1.test should be NotImplemented, got '%s' instead" % instance1.test
# now demonstrate that the current implementation is broken:
instance2 = MyClass({"test":42})
instance2.test = NotImplemented
assert instance2.test is NotImplemented, "instance2.test should be NotImplemented, got '%s' instead" % instance2.test

If you define overriding __setattr__ correctly as calling the __setattr__ of the parent class then you could graft your method into a class hierarchy that defines its own custom __setattr__:
def inject_tester_class(cls):
def __setattr__(self, name, value):
self._TesterClass__setattr_args.append((name, value))
super(intermediate, self).__setattr__(name, value)
def assertSetAttrDelegatedFor(self, name, value):
assert \
[args for args in self._TesterClass__setattr_args if args == (name, value)], \
'__setattr__(name, value) was never delegated'
body = {
'__setattr__': __setattr__,
'assertSetAttrDelegatedFor': assertSetAttrDelegatedFor,
'_TesterClass__setattr_args': []
}
intermediate = type('TesterClass', cls.__bases__, body)
testclass = type(cls.__name__, (intermediate,), vars(cls).copy())
# rebind the __class__ closure
def closure():
testclass
osa = testclass.__setattr__
new_closure = tuple(closure.__closure__[0] if n == '__class__' else c
for n, c in zip(osa.__code__.co_freevars, osa.__closure__))
testclass.__setattr__ = type(osa)(
osa.__code__, osa.__globals__, osa.__name__,
osa.__defaults__, new_closure)
return testclass
This function jumps through a few hoops to insert an intermediate class that'll intercept any properly delegated __setattr__ call. It'll work even if you don't have any base classes other than the default object (which wouldn't let us replace __setattr__ for an easier path to test this).
It does make the assumption that you are using super().__setattr__() to delegate, where you used super() without arguments. It also assumes there is no meta class involved.
The extra __setattr__ is injected in a manner consistent with the existing MRO; the extra intermediate class is injected between the original class and the rest of the MRO, and delegates the __setattr__ call onwards.
To use this in a test, you'd produce a new class with the above function, create an instance then set attributes on that instance:
MyTestClass = inject_tester_class(MyClass)
my_test_instance = MyTestClass()
my_test_instance.foo = 'bar'
my_test_instance.assertSetAttrDelegatedFor('foo', 'bar')
If setting foo is not delegated, an AssertionError exception is raised, which the unittest test runner records as a test failure.

Prevent creating new attributes outside init

I want to be able to create a class (in Python) that once initialized with __init__, does not accept new attributes, but accepts modifications of existing attributes. There's several hack-ish ways I can see to do this, for example having a __setattr__ method such as
def __setattr__(self, attribute, value):
if not attribute in self.__dict__:
print "Cannot set %s" % attribute
else:
self.__dict__[attribute] = value
and then editing __dict__ directly inside __init__, but I was wondering if there is a 'proper' way to do this?

I wouldn't use __dict__ directly, but you can add a function to explicitly "freeze" a instance:
class FrozenClass(object):
__isfrozen = False
def __setattr__(self, key, value):
if self.__isfrozen and not hasattr(self, key):
raise TypeError( "%r is a frozen class" % self )
object.__setattr__(self, key, value)
def _freeze(self):
self.__isfrozen = True
class Test(FrozenClass):
def __init__(self):
self.x = 42#
self.y = 2**3
self._freeze() # no new attributes after this point.
a,b = Test(), Test()
a.x = 10
b.z = 10 # fails

Slots is the way to go:
The pythonic way is to use slots instead of playing around with the __setter__. While it may solve the problem, it does not give any performance improvement. The attributes of objects are stored in a dictionary "__dict__", this is the reason, why you can dynamically add attributes to objects of classes that we have created so far. Using a dictionary for attribute storage is very convenient, but it can mean a waste of space for objects, which have only a small amount of instance variables.
Slots are a nice way to work around this space consumption problem. Instead of having a dynamic dict that allows adding attributes to objects dynamically, slots provide a static structure which prohibits additions after the creation of an instance.
When we design a class, we can use slots to prevent the dynamic creation of attributes. To define slots, you have to define a list with the name __slots__. The list has to contain all the attributes, you want to use. We demonstrate this in the following class, in which the slots list contains only the name for an attribute "val".
class S(object):
__slots__ = ['val']
def __init__(self, v):
self.val = v
x = S(42)
print(x.val)
x.new = "not possible"
=> It fails to create an attribute "new":
42
Traceback (most recent call last):
File "slots_ex.py", line 12, in <module>
x.new = "not possible"
AttributeError: 'S' object has no attribute 'new'
Notes:
Since Python 3.3 the advantage optimizing the space consumption is not as impressive any more. With Python 3.3 Key-Sharing Dictionaries are used for the storage of objects. The attributes of the instances are capable of sharing part of their internal storage between each other, i.e. the part which stores the keys and their corresponding hashes. This helps to reduce the memory consumption of programs, which create many instances of non-builtin types. But still is the way to go to avoid dynamically created attributes.
Using slots come also with it's own cost. It will break serialization (e.g. pickle). It will also break multiple inheritance. A class can't inherit from more than one class that either defines slots or has an instance layout defined in C code (like list, tuple or int).

If someone is interested in doing that with a decorator, here is a working solution:
from functools import wraps
def froze_it(cls):
cls.__frozen = False
def frozensetattr(self, key, value):
if self.__frozen and not hasattr(self, key):
print("Class {} is frozen. Cannot set {} = {}"
.format(cls.__name__, key, value))
else:
object.__setattr__(self, key, value)
def init_decorator(func):
#wraps(func)
def wrapper(self, *args, **kwargs):
func(self, *args, **kwargs)
self.__frozen = True
return wrapper
cls.__setattr__ = frozensetattr
cls.__init__ = init_decorator(cls.__init__)
return cls
Pretty straightforward to use:
#froze_it
class Foo(object):
def __init__(self):
self.bar = 10
foo = Foo()
foo.bar = 42
foo.foobar = "no way"
Result:
>>> Class Foo is frozen. Cannot set foobar = no way

Actually, you don't want __setattr__, you want __slots__. Add __slots__ = ('foo', 'bar', 'baz') to the class body, and Python will make sure that there's only foo, bar and baz on any instance. But read the caveats the documentation lists!

The proper way is to override __setattr__. That's what it's there for.

I like very much the solution that uses a decorator, because it's easy to use it for many classes across a project, with minimum additions for each class. But it doesn't work well with inheritance.
So here is my version: It only overrides the __setattr__ function - if the attribute doesn't exist and the caller function is not __init__, it prints an error message.
import inspect
def froze_it(cls):
def frozensetattr(self, key, value):
if not hasattr(self, key) and inspect.stack()[1][3] != "__init__":
print("Class {} is frozen. Cannot set {} = {}"
.format(cls.__name__, key, value))
else:
self.__dict__[key] = value
cls.__setattr__ = frozensetattr
return cls
#froze_it
class A:
def __init__(self):
self._a = 0
a = A()
a._a = 1
a._b = 2 # error

What about this:
class A():
__allowed_attr=('_x', '_y')
def __init__(self,x=0,y=0):
self._x=x
self._y=y
def __setattr__(self,attribute,value):
if not attribute in self.__class__.__allowed_attr:
raise AttributeError
else:
super().__setattr__(attribute,value)

Here is approach i came up with that doesn't need a _frozen attribute or method to freeze() in init.
During init i just add all class attributes to the instance.
I like this because there is no _frozen, freeze(), and _frozen also does not show up in the vars(instance) output.
class MetaModel(type):
def __setattr__(self, name, value):
raise AttributeError("Model classes do not accept arbitrary attributes")
class Model(object):
__metaclass__ = MetaModel
# init will take all CLASS attributes, and add them as SELF/INSTANCE attributes
def __init__(self):
for k, v in self.__class__.__dict__.iteritems():
if not k.startswith("_"):
self.__setattr__(k, v)
# setattr, won't allow any attributes to be set on the SELF/INSTANCE that don't already exist
def __setattr__(self, name, value):
if not hasattr(self, name):
raise AttributeError("Model instances do not accept arbitrary attributes")
else:
object.__setattr__(self, name, value)
# Example using
class Dog(Model):
name = ''
kind = 'canine'
d, e = Dog(), Dog()
print vars(d)
print vars(e)
e.junk = 'stuff' # fails

I like the "Frozen" of Jochen Ritzel. The inconvenient is that the isfrozen variable then appears when printing a Class.__dict
I went around this problem this way by creating a list of authorized attributes (similar to slots):
class Frozen(object):
__List = []
def __setattr__(self, key, value):
setIsOK = False
for item in self.__List:
if key == item:
setIsOK = True
if setIsOK == True:
object.__setattr__(self, key, value)
else:
raise TypeError( "%r has no attributes %r" % (self, key) )
class Test(Frozen):
_Frozen__List = ["attr1","attr2"]
def __init__(self):
self.attr1 = 1
self.attr2 = 1

The FrozenClass by Jochen Ritzel is cool, but calling _frozen() when initialing a class every time is not so cool (and you need to take the risk of forgetting it). I added a __init_slots__ function:
class FrozenClass(object):
__isfrozen = False
def _freeze(self):
self.__isfrozen = True
def __init_slots__(self, slots):
for key in slots:
object.__setattr__(self, key, None)
self._freeze()
def __setattr__(self, key, value):
if self.__isfrozen and not hasattr(self, key):
raise TypeError( "%r is a frozen class" % self )
object.__setattr__(self, key, value)
class Test(FrozenClass):
def __init__(self):
self.__init_slots__(["x", "y"])
self.x = 42#
self.y = 2**3
a,b = Test(), Test()
a.x = 10
b.z = 10 # fails

None of the answers mention the performance impact of overriding __setattr__, which can be an issue when creating many small objects. (And __slots__ would be the performant solution but limits pickle/inheritance).
So I came up with this variant which installs our slower settatr after init:
class FrozenClass:
def freeze(self):
def frozen_setattr(self, key, value):
if not hasattr(self, key):
raise TypeError("Cannot set {}: {} is a frozen class".format(key, self))
object.__setattr__(self, key, value)
self.__setattr__ = frozen_setattr
class Foo(FrozenClass): ...
If you don't want to call freeze at the end of __init__, if inheritance is an issue, or if you don't want it in vars(), it can also be adapted: for example here is a decorator version based on the pystrict answer:
import functools
def strict(cls):
cls._x_setter = getattr(cls, "__setattr__", object.__setattr__)
cls._x_init = cls.__init__
#functools.wraps(cls.__init__)
def wrapper(self, *args, **kwargs):
cls._x_init(self, *args, **kwargs)
def frozen_setattr(self, key, value):
if not hasattr(self, key):
raise TypeError("Class %s is frozen. Cannot set '%s'." % (cls.__name__, key))
cls._x_setter(self, key, value)
cls.__setattr__ = frozen_setattr
cls.__init__ = wrapper
return cls
#strict
class Foo: ...

I wrote pystrict as a solution to this problem. It's too large to paste all of the code in stackoverflow.
pystrict is a pypi installable decorator that can be used with classes to freeze them. Many solutions here don't properly support inheritance.
If __slots__ doesn't work for you (because of inheritance issues), this is a good alternative.
There is an example to the README that shows why a decorator like this is needed even if you have mypy and pylint running on your project:
pip install pystrict
Then just use the #strict decorator:
from pystrict import strict
#strict
class Blah
def __init__(self):
self.attr = 1

#dataclass(slots=True) Nirvana (Python 3.10)
I'm in love with this #dataclass thing:
main.py
from dataclasses import dataclass
#dataclass(slots=True)
class C:
n: int
s: str
c = C(n=1, s='one')
assert c.n == 1
assert c.s == 'one'
c.n == 2
c.s == 'two'
c.asdf = 2
Outcome:
Traceback (most recent call last):
File "/home/ciro/main.py", line 15, in <module>
c.asdf = 2
AttributeError: 'C' object has no attribute 'asdf'
Note how #dataclass only requires use to define our attributes once with type annotations
n: int
s: str
and then, without any repetition we get for free:
def __init__(n, s):
self.n = n
self.s = s
__slots__ = ['n', 's']
Other free things not shown in this example:
__str__
__eq__: Compare object instances for equality by their attributes
__hash__ if you also use frozen=True: Object of custom type as dictionary key
Tested on Python 3.10.7, Ubuntu 22.10.

We Keep Coding

Python is a programming language that lets you work quickly and integrate systems more effectively.

Turning nested class into descriptor without instantiating it - python

Related

Overloading get on a custom Python class not working [duplicate]

Why overriding the `get` method of a class in Python affects what is returned? [duplicate]

Need help understanding a piece of code with class decorators and descriptors [duplicate]

What tests can I perform to ensure I have overridden setattr correctly?

Prevent creating new attributes outside init

Categories

Resources

We Keep Coding

Python is a programming language that lets you work quickly and integrate systems more effectively.

Turning nested class into descriptor without instantiating it - python

Related

Overloading __get__ on a custom Python class not working [duplicate]

Why overriding the `__get__` method of a class in Python affects what is returned? [duplicate]

Need help understanding a piece of code with class decorators and descriptors [duplicate]

What tests can I perform to ensure I have overridden __setattr__ correctly?

Prevent creating new attributes outside __init__

Categories

Resources

Overloading get on a custom Python class not working [duplicate]

Why overriding the `get` method of a class in Python affects what is returned? [duplicate]

What tests can I perform to ensure I have overridden setattr correctly?

Prevent creating new attributes outside init