According to Python's documentation,
Data descriptors with __set__() and __get__() defined always override a redefinition in an instance dictionary.
I have no problem understanding this sentence, but can someone clarify for me why such a rule is in place? After all, if I want to override an attribute in an instance dictionary, I already need to do that explicitely (inst.__dict__["attr"] = val), as a naive inst.attr = val would call the descriptor's __set__ method, which would (usually) not override the attribute in the instance dictionary.
edit: just to make it clear, I understand what is happening, my question is about why such a rule was put in place.
The override applies to descriptors that are part of the class __dict__.
Python will always look up type(instance).__dict__[attributename].__get__(instance, type(instance)), and will not use instance.__dict__ to search for a instance-override.
Here is an example using a contrived Descriptor class and a property (which is a descriptor with a __get__ and a __set__:
>>> class Descriptor(object):
... def __init__(self, name):
... self.name = name
... def __get__(self, instance, cls):
... print 'Getting %s, with instance %r, class %r' % (self.name, instance, cls)
...
>>> class Foo(object):
... _spam = 'eggs'
... #property
... def spam(self):
... return self._spam
... #spam.setter
... def spam(self, val):
... self._spam = val
...
>>> Foo().spam
'eggs'
>>> foo = Foo()
>>> foo.__dict__['spam'] = Descriptor('Override')
>>> foo.spam
'eggs'
As you can see, even though I add a spam entry in the instance __dict__, it is completely ignored and the Foo.spam property is used still. Python is ignoring the instance __dict__ because the spam property defines both __get__ and a __set__.
If you use a descriptor that doesn't define a __set__ the override works (but it's __get__ is not called:
>>> class Foo(object):
... desc = Descriptor('Class-stored descriptor')
...
>>> Foo.desc
Getting Class-stored descriptor, with instance None, class <class '__main__.Foo'>
>>> Foo().desc
Getting Class-stored descriptor, with instance <__main__.Foo object at 0x1018df510>, class <class '__main__.Foo'>
>>> foo = Foo()
>>> foo.__dict__['desc'] = Descriptor('Instance-stored descriptor')
>>> foo.desc
<__main__.Descriptor object at 0x1018df1d0>
Related
Is it at all possible to monkey patch the value of a #property of an instance of a class that I do not control?
class Foo:
#property
def bar(self):
return here().be['dragons']
f = Foo()
print(f.bar) # baz
f.bar = 42 # MAGIC!
print(f.bar) # 42
Obviously the above would produce an error when trying to assign to f.bar. Is # MAGIC! possible in any way? The implementation details of the #property are a black box and not indirectly monkey-patchable. The entire method call needs to be replaced. It needs to affect a single instance only (class-level patching is okay if inevitable, but the changed behaviour must only selectively affect a given instance, not all instances of that class).
Subclass the base class (Foo) and change single instance's class to match the new subclass using __class__ attribute:
>>> class Foo:
... #property
... def bar(self):
... return 'Foo.bar'
...
>>> f = Foo()
>>> f.bar
'Foo.bar'
>>> class _SubFoo(Foo):
... bar = 0
...
>>> f.__class__ = _SubFoo
>>> f.bar
0
>>> f.bar = 42
>>> f.bar
42
from module import ClassToPatch
def get_foo(self):
return 'foo'
setattr(ClassToPatch, 'foo', property(get_foo))
To monkey patch a property, there is an even simpler way:
from module import ClassToPatch
def get_foo(self):
return 'foo'
ClassToPatch.foo = property(get_foo)
Idea: replace property descriptor to allow setting on certain objects. Unless a value is explicitly set this way, original property getter is called.
The problem is how to store the explicitly set values. We cannot use a dict keyed by patched objects, since 1) they are not necessarily comparable by identity; 2) this prevents patched objects from being garbage-collected. For 1) we could write a Handle that wraps objects and overrides comparison semantics by identity and for 2) we could use weakref.WeakKeyDictionary. However, I couldn't make these two work together.
Therefore we use a different approach of storing the explicitly set values on the object itself, using a "very unlikely attribute name". It is of course still possible that this name would collide with something, but that's pretty much inherent to languages such as Python.
This won't work on objects that lack a __dict__ slot. Similar problem would arise for weakrefs though.
class Foo:
#property
def bar (self):
return 'original'
class Handle:
def __init__(self, obj):
self._obj = obj
def __eq__(self, other):
return self._obj is other._obj
def __hash__(self):
return id (self._obj)
_monkey_patch_index = 0
_not_set = object ()
def monkey_patch (prop):
global _monkey_patch_index, _not_set
special_attr = '$_prop_monkey_patch_{}'.format (_monkey_patch_index)
_monkey_patch_index += 1
def getter (self):
value = getattr (self, special_attr, _not_set)
return prop.fget (self) if value is _not_set else value
def setter (self, value):
setattr (self, special_attr, value)
return property (getter, setter)
Foo.bar = monkey_patch (Foo.bar)
f = Foo()
print (Foo.bar.fset)
print(f.bar) # baz
f.bar = 42 # MAGIC!
print(f.bar) # 42
It looks like you need to move on from properties to the realms of data descriptors and non-data descriptors. Properties are just a specialised version of data descriptors. Functions are an example of non-data descriptors -- when you retrieve them from an instance they return a method rather than the function itself.
A non-data descriptor is just an instance of a class that has a __get__ method. The only difference with a data descriptor is that it has a __set__ method as well. Properties initially have a __set__ method that throws an error unless you provide a setter function.
You can achieve what you want really easily just by writing your own trivial non-data descriptor.
class nondatadescriptor:
"""generic nondata descriptor decorator to replace #property with"""
def __init__(self, func):
self.func = func
def __get__(self, obj, objclass):
if obj is not None:
# instance based access
return self.func(obj)
else:
# class based access
return self
class Foo:
#nondatadescriptor
def bar(self):
return "baz"
foo = Foo()
another_foo = Foo()
assert foo.bar == "baz"
foo.bar = 42
assert foo.bar == 42
assert another_foo.bar == "baz"
del foo.bar
assert foo.bar == "baz"
print(Foo.bar)
What makes all this work is that logic under the hood __getattribute__. I can't find the appropriate documentation at the moment, but order of retrieval is:
Data descriptors defined on the class are given the highest priority (objects with both __get__ and __set__), and their __get__ method is invoked.
Any attribute of the object itself.
Non-data descriptors defined on the class (objects with only a __get__ method).
All other attributes defined on the class.
Finally the __getattr__ method of the object is invoked as a last resort (if defined).
You can also patch property setters. Using #fralau 's answer:
from module import ClassToPatch
def foo(self, new_foo):
self._foo = new_foo
ClassToPatch.foo = ClassToPatch.foo.setter(foo)
reference
In case someone needs to patch a property while being able to call the original implementation, here is an example:
#property
def _cursor_args(self, __orig=mongoengine.queryset.base.BaseQuerySet._cursor_args):
# TODO: remove this hack when we upgrade MongoEngine
# https://github.com/MongoEngine/mongoengine/pull/2160
cursor_args = __orig.__get__(self)
if self._timeout:
cursor_args.pop("no_cursor_timeout", None)
return cursor_args
mongoengine.queryset.base.BaseQuerySet._cursor_args = _cursor_args
This is the setup I want:
A should be an abstract base class with a static & abstract method f(). B should inherit from A. Requirements:
1. You should not be able to instantiate A
2. You should not be able to instantiate B, unless it implements a static f()
Taking inspiration from this question, I've tried a couple of approaches. With these definitions:
class abstractstatic(staticmethod):
__slots__ = ()
def __init__(self, function):
super(abstractstatic, self).__init__(function)
function.__isabstractmethod__ = True
__isabstractmethod__ = True
class A:
__metaclass__ = abc.ABCMeta
#abstractstatic
def f():
pass
class B(A):
def f(self):
print 'f'
class A2:
__metaclass__ = abc.ABCMeta
#staticmethod
#abc.abstractmethod
def f():
pass
class B2(A2):
def f(self):
print 'f'
Here A2 and B2 are defined using usual Python conventions and A & B are defined using the way suggested in this answer. Following are some operations I tried and the results that were undesired.
With classes A/B:
>>> B().f()
f
#This should have thrown, since B doesn't implement a static f()
With classes A2/B2:
>>> A2()
<__main__.A2 object at 0x105beea90>
#This should have thrown since A2 should be an uninstantiable abstract class
>>> B2().f()
f
#This should have thrown, since B2 doesn't implement a static f()
Since neither of these approaches give me the output I want, how do I achieve what I want?
You can't do what you want with just ABCMeta. ABC enforcement doesn't do any type checking, only the presence of an attribute with the correct name is enforced.
Take for example:
>>> from abc import ABCMeta, abstractmethod, abstractproperty
>>> class Abstract(object):
... __metaclass__ = ABCMeta
... #abstractmethod
... def foo(self): pass
... #abstractproperty
... def bar(self): pass
...
>>> class Concrete(Abstract):
... foo = 'bar'
... bar = 'baz'
...
>>> Concrete()
<__main__.Concrete object at 0x104b4df90>
I was able to construct Concrete() even though both foo and bar are simple attributes.
The ABCMeta metaclass only tracks how many objects are left with the __isabstractmethod__ attribute being true; when creating a class from the metaclass (ABCMeta.__new__ is called) the cls.__abstractmethods__ attribute is then set to a frozenset object with all the names that are still abstract.
type.__new__ then tests for that frozenset and throws a TypeError if you try to create an instance.
You'd have to produce your own __new__ method here; subclass ABCMeta and add type checking in a new __new__ method. That method should look for __abstractmethods__ sets on the base classes, find the corresponding objects with the __isabstractmethod__ attribute in the MRO, then does typechecking on the current class attributes.
This'd mean that you'd throw the exception when defining the class, not an instance, however. For that to work you'd add a __call__ method to your ABCMeta subclass and have that throw the exception based on information gathered by your own __new__ method about what types were wrong; a similar two-stage process as what ABCMeta and type.__new__ do at the moment. Alternatively, update the __abstractmethods__ set on the class to add any names that were implemented but with the wrong type and leave it to type.__new__ to throw the exception.
The following implementation takes that last tack; add names back to __abstractmethods__ if the implemented type doesn't match (using a mapping):
from types import FunctionType
class ABCMetaTypeCheck(ABCMeta):
_typemap = { # map abstract type to expected implementation type
abstractproperty: property,
abstractstatic: staticmethod,
# abstractmethods return function objects
FunctionType: FunctionType,
}
def __new__(mcls, name, bases, namespace):
cls = super(ABCMetaTypeCheck, mcls).__new__(mcls, name, bases, namespace)
wrong_type = set()
seen = set()
abstractmethods = cls.__abstractmethods__
for base in bases:
for name in getattr(base, "__abstractmethods__", set()):
if name in seen or name in abstractmethods:
continue # still abstract or later overridden
value = base.__dict__.get(name) # bypass descriptors
if getattr(value, "__isabstractmethod__", False):
seen.add(name)
expected = mcls._typemap[type(value)]
if not isinstance(namespace[name], expected):
wrong_type.add(name)
if wrong_type:
cls.__abstractmethods__ = abstractmethods | frozenset(wrong_type)
return cls
With this metaclass you get your expected output:
>>> class Abstract(object):
... __metaclass__ = ABCMetaTypeCheck
... #abstractmethod
... def foo(self): pass
... #abstractproperty
... def bar(self): pass
... #abstractstatic
... def baz(): pass
...
>>> class ConcreteWrong(Abstract):
... foo = 'bar'
... bar = 'baz'
... baz = 'spam'
...
>>> ConcreteWrong()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class ConcreteWrong with abstract methods bar, baz, foo
>>>
>>> class ConcreteCorrect(Abstract):
... def foo(self): return 'bar'
... #property
... def bar(self): return 'baz'
... #staticmethod
... def baz(): return 'spam'
...
>>> ConcreteCorrect()
<__main__.ConcreteCorrect object at 0x104ce1d10>
For dictionary we can use .get method. What about a class's attribute and provide a default value?
You can use hasattr and getattr.
For example:
hasattr(foo, 'bar')
would return True if foo has an attribute named bar, otherwise False and
getattr(foo, 'bar', 'quux')
would return foo.bar if it exists, otherwise defaults to quux.
It's dead simple: use a keyword argument.
>>> class Foo(object):
... def __init__(self, my_foo=None):
... self.my_foo = my_foo
...
>>> f = Foo()
>>> f.my_foo
>>> repr(f.my_foo)
'None'
>>> f2 = Foo(2)
>>> f2.my_foo
2
If you are looking for an object which does not have an attribute until you set it, Jason's idea is pretty good unless you directly refer to the attribute. You'll have to work around the AttributeError you'll get. Personally, I'm not a fan of creating objects which must be constantly surrounded by try/except blocks just for the sake of not setting an instance attribute.
Python isn't much for getters and setters. However, you can use property() to work around this problem:
>>> class Foo(object):
... def __init__(self):
... pass
... def get_my_foo(self):
... return getattr(self, "_my_foo", "there's no my_foo")
... def set_my_foo(self, foo):
... self._my_foo = foo
... my_foo = property(get_my_foo, set_my_foo)
...
>>> f = Foo()
>>> f.my_foo
"there's no my_foo"
>>> f.my_foo = "foo!"
>>> f.my_foo
'foo!'
It works just as well to call get_my_foo and set_my_foo, if you like. An added benefit is that you can omit the setter to make a read-only attribute, provided someone using your object doesn't change the underlying _my_foo.
I want to wrap a model class of a legacy codebase. The model class has a dictionary with meta-information and properties that access that dictionary as well as attributes. I want to unify the access to meta information, properties, and attributes with the an_object[some_key] syntax using __getitem__. The problem is, that some of the properties have getters but not setters. So trying to check if an attribute exists (via hasattr) returns True, but then setting that attribute fails because there is no property defined.
How can I decide if I can set an attribute safely or if it is an property that I need to set in the meta-dictionary?
You can detect if something is a property, by looking at the same attribute on the class:
class_attribute = getattr(type(instance), some_key, None)
if isinstance(class_attribute, property):
# this is a property
if class_attribute.fset is None:
print "Read-only"
You can also test .fget and .fdel to test if the property has a getter and a deleter, respectively.
However, you can always catch the AttributeError exception to deal with the setter missing:
>>> class Foo(object):
... #property
... def bar(self):
... return 'spam'
...
>>> f = Foo()
>>> class_attribute = getattr(type(f), 'bar', None)
>>> isinstance(class_attribute, property)
True
>>> class_attribute.fget
<function bar at 0x10aa8c668>
>>> class_attribute.fset is None
True
>>> try:
... f.bar = 'baz'
... except AttributeError:
... print 'Read-only'
...
Read-only
Is there some way to make a class-level read-only property in Python? For instance, if I have a class Foo, I want to say:
x = Foo.CLASS_PROPERTY
but prevent anyone from saying:
Foo.CLASS_PROPERTY = y
EDIT:
I like the simplicity of Alex Martelli's solution, but not the syntax that it requires. Both his and ~unutbu's answers inspired the following solution, which is closer to the spirit of what I was looking for:
class const_value (object):
def __init__(self, value):
self.__value = value
def make_property(self):
return property(lambda cls: self.__value)
class ROType(type):
def __new__(mcl,classname,bases,classdict):
class UniqeROType (mcl):
pass
for attr, value in classdict.items():
if isinstance(value, const_value):
setattr(UniqeROType, attr, value.make_property())
classdict[attr] = value.make_property()
return type.__new__(UniqeROType,classname,bases,classdict)
class Foo(object):
__metaclass__=ROType
BAR = const_value(1)
BAZ = 2
class Bit(object):
__metaclass__=ROType
BOO = const_value(3)
BAN = 4
Now, I get:
Foo.BAR
# 1
Foo.BAZ
# 2
Foo.BAR=2
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# AttributeError: can't set attribute
Foo.BAZ=3
#
I prefer this solution because:
The members get declared inline instead of after the fact, as with type(X).foo = ...
The members' values are set in the actual class's code as opposed to in the metaclass's code.
It's still not ideal because:
I have to set the __metaclass__ in order for const_value objects to be interpreted correctly.
The const_values don't "behave" like the plain values. For example, I couldn't use it as a default value for a parameter to a method in the class.
The existing solutions are a bit complex -- what about just ensuring that each class in a certain group has a unique metaclass, then setting a normal read-only property on the custom metaclass. Namely:
>>> class Meta(type):
... def __new__(mcl, *a, **k):
... uniquemcl = type('Uniq', (mcl,), {})
... return type.__new__(uniquemcl, *a, **k)
...
>>> class X: __metaclass__ = Meta
...
>>> class Y: __metaclass__ = Meta
...
>>> type(X).foo = property(lambda *_: 23)
>>> type(Y).foo = property(lambda *_: 45)
>>> X.foo
23
>>> Y.foo
45
>>>
this is really much simpler, because it's based on nothing more than the fact that when you get an instance's attribute descriptors are looked up on the class (so of course when you get a class's attribute descriptors are looked on the metaclass), and making class/metaclass unique isn't terribly hard.
Oh, and of course:
>>> X.foo = 67
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't set attribute
just to confirm it IS indeed read-only!
The ActiveState solution that Pynt references makes instances of ROClass have read-only attributes. Your question seems to ask if the class itself can have read-only attributes.
Here is one way, based on Raymond Hettinger's comment:
#!/usr/bin/env python
def readonly(value):
return property(lambda self: value)
class ROType(type):
CLASS_PROPERTY = readonly(1)
class Foo(object):
__metaclass__=ROType
print(Foo.CLASS_PROPERTY)
# 1
Foo.CLASS_PROPERTY=2
# AttributeError: can't set attribute
The idea is this: Consider first Raymond Hettinger's solution:
class Bar(object):
CLASS_PROPERTY = property(lambda self: 1)
bar=Bar()
bar.CLASS_PROPERTY=2
It shows a relatively simple way to give bar a read-only property.
Notice that you have to add the CLASS_PROPERTY = property(lambda self: 1)
line to the definition of the class of bar, not to bar itself.
So, if you want the class Foo to have a read-only property, then the parent class of Foo has to have CLASS_PROPERTY = property(lambda self: 1) defined.
The parent class of a class is a metaclass. Hence we define ROType as the metaclass:
class ROType(type):
CLASS_PROPERTY = readonly(1)
Then we make Foo's parent class be ROType:
class Foo(object):
__metaclass__=ROType
Found this on ActiveState:
# simple read only attributes with meta-class programming
# method factory for an attribute get method
def getmethod(attrname):
def _getmethod(self):
return self.__readonly__[attrname]
return _getmethod
class metaClass(type):
def __new__(cls,classname,bases,classdict):
readonly = classdict.get('__readonly__',{})
for name,default in readonly.items():
classdict[name] = property(getmethod(name))
return type.__new__(cls,classname,bases,classdict)
class ROClass(object):
__metaclass__ = metaClass
__readonly__ = {'a':1,'b':'text'}
if __name__ == '__main__':
def test1():
t = ROClass()
print t.a
print t.b
def test2():
t = ROClass()
t.a = 2
test1()
Note that if you try to set a read-only attribute (t.a = 2) python will raise an AttributeError.