I have classes that have attributes set with #property decorator. They function as getter and setter using try and except clauses inside them. If attribute is not set, it gets data from database and uses it to instatiate objects from other classes. I tried to keep the example short, but the code used to instantiate attribute objects is a little different with each attribute. What they have in common is the try-except at the beginning.
class SubClass(TopClass):
#property
def thing(self):
try:
return self._thing
except AttributeError:
# We don't have any thing yet
pass
thing = get_some_thing_from_db('thing')
if not thing:
raise AttributeError()
self._thing = TheThing(thing)
return self._thing
#property
def another_thing(self):
try:
return self._another_thing
except AttributeError:
# We don't have things like this yet
pass
another_thing = get_some_thing_from_db('another')
if not another_thing:
raise AttributeError()
self._another_thing = AnotherThing(another_thing)
return self._another_thing
...etc...
#property
def one_more_thing(self):
try:
return self._one_more_thing
except AttributeError:
# We don't have this thing yet
pass
one_thing = get_some_thing_from_db('one')
if not one_thing:
raise AttributeError()
self._one_more_thing = OneThing(one_thing)
return self._one_more_thing
My question: is this a proper (e.g. pythonic) way of doing stuff? To me it seems a bit awkward to add the try-except-segment on top of everything. On the other hand it keeps the code short. Or is there a better way of defining attributes?
So long as you are using at least Python 3.2, use the functools.lru_cache() decorator.
import functools
class SubClass(TopClass):
#property
#functools.lru_cache()
def thing(self):
thing = get_some_thing_from_db('thing')
if not thing:
raise AttributeError()
return TheThing(thing)
A quick runnable example:
>>> import functools
>>> class C:
#property
#functools.lru_cache()
def foo(self):
print("Called foo")
return 42
>>> c = C()
>>> c.foo
Called foo
42
>>> c.foo
42
If you have a lot of these you can combine the decorators:
>>> def lazy_property(f):
return property(functools.lru_cache()(f))
>>> class C:
#lazy_property
def foo(self):
print("Called foo")
return 42
>>> c = C()
>>> c.foo
Called foo
42
>>> c.foo
42
If you are still on an older version of Python there's a fully featured backport of lru_cache on ActiveState although as in this case you're not passing any parameters when you call it you could probably replace it with something much simpler.
#YAmikep asks how to access the cache_info() method of lru_cache. It's a little bit messy, but you can still access it through the property object:
>>> C.foo.fget.cache_info()
CacheInfo(hits=0, misses=1, maxsize=128, currsize=1)
Related
I have following code:
a = A()
a.foo(123)
A.foo(a, 123)
How can I detect which line caused foo() execution: 2 or 3? Thank you for your answers.
Preface: There are very few use cases where the difference between these is actually important. Having different behavior based on whether a method is called directly on an instance or by explicitly passing the instance to the method accessed through the class would likely violate the Principle of least astonishment
This solution uses the fact that we can tell whether a descriptor was accessed through an instance or through a class by whether the instance parameter is None or not.
from __future__ import annotations
from functools import partial
from typing import Optional, Type
class Method:
def __init__(self, func):
self.func = func
def __get__(self, instance: Optional[B], owner: Type[B]):
via_instance = instance is not None
if via_instance:
return partial(self.func, instance, via_instance=via_instance)
else:
return partial(self.func, via_instance=via_instance)
class B:
#Method
def foo(self, bar, via_instance):
if via_instance:
return 'via instance'
return 'via class'
Test:
>>> b = B()
>>> b.foo(123)
'via instance'
>>> B.foo(b, 123)
'via class'
It may still be technically possible to work around this detection, but this approach is not as hacky/brittle as the one below that uses inspect.
Alternate Solution
Note: This solution over-complicates the problem. This is extremely hacky, and there are many other cases that this does not cover. This guesses based on the defined parameters of this question. It is not definitive.
import ast
import inspect
from ast import Assign, Call, Expr
class A:
def foo(self, bar):
code = inspect.stack()[1].code_context[0]
parsed = ast.parse(code).body[0]
if isinstance(parsed, Assign):
parsed = parsed.value
if isinstance(parsed, Expr):
parsed = parsed.value
if not isinstance(parsed, Call):
raise TypeError(f'Unexpected type={type(parsed)} for {parsed=}')
if len(parsed.args) == 1:
return 'via instance'
return 'via class'
Test:
>>> a = A()
>>> a.foo(123)
'via instance'
>>> A.foo(a, 123)
'via class'
There are many other ways that the method could be called, and this approach is not guaranteed to catch all of them. Some examples of other ways :
super().foo(123) in a subclass
foo = partial(A.foo, bar=123); foo(a)
foo = partial(A.foo, a); foo(123)
args = (a, 123); A.foo(*args)
etc...
assume following class definition:
class A:
def f(self):
return 'this is f'
#staticmethod
def g():
return 'this is g'
a = A()
So f is a normal method and g is a static method.
Now, how can I check if the funcion objects a.f and a.g are static or not? Is there a "isstatic" funcion in Python?
I have to know this because I have lists containing many different function (method) objects, and to call them I have to know if they are expecting "self" as a parameter or not.
Lets experiment a bit:
>>> import types
>>> class A:
... def f(self):
... return 'this is f'
... #staticmethod
... def g():
... return 'this is g'
...
>>> a = A()
>>> a.f
<bound method A.f of <__main__.A instance at 0x800f21320>>
>>> a.g
<function g at 0x800eb28c0>
>>> isinstance(a.g, types.FunctionType)
True
>>> isinstance(a.f, types.FunctionType)
False
So it looks like you can use types.FunctionType to distinguish static methods.
Your approach seems a bit flawed to me, but you can check class attributes:
(in Python 2.7):
>>> type(A.f)
<type 'instancemethod'>
>>> type(A.g)
<type 'function'>
or instance attributes in Python 3.x
>>> a = A()
>>> type(a.f)
<type 'method'>
>>> type(a.g)
<type 'function'>
To supplement the answers here, in Python 3 the best way is like so:
import inspect
class Test:
#staticmethod
def test(): pass
isstatic = isinstance(inspect.getattr_static(Test, "test"), staticmethod)
We use getattr_static rather than getattr, since getattr will retrieve the bound method or function, not the staticmethod class object. You can do a similar check for classmethod types and property's (e.g. attributes defined using the #property decorator)
Note that even though it is a staticmethod, don't assume it was defined inside the class. The method source may have originated from another class. To get the true source, you can look at the underlying function's qualified name and module. For example:
class A:
#staticmethod:
def test(): pass
class B: pass
B.test = inspect.getattr_static(A, "test")
print("true source: ", B.test.__qualname__)
Technically, any method can be used as "static" methods, so long as they are called on the class itself, so just keep that in mind. For example, this will work perfectly fine:
class Test:
def test():
print("works!")
Test.test()
That example will not work with instances of Test, since the method will be bound to the instance and called as Test.test(self) instead.
Instance and class methods can be used as static methods as well in some cases, so long as the first arg is handled properly.
class Test:
def test(self):
print("works!")
Test.test(None)
Perhaps another rare case is a staticmethod that is also bound to a class or instance. For example:
class Test:
#classmethod
def test(cls): pass
Test.static_test = staticmethod(Test.test)
Though technically it is a staticmethod, it is really behaving like a classmethod. So in your introspection, you may consider checking the __self__ (recursively on __func__) to see if the method is bound to a class or instance.
I happens to have a module to solve this. And it's Python2/3 compatible solution. And it allows to test with method inherit from parent class.
Plus, this module can also test:
regular attribute
property style method
regular method
staticmethod
classmethod
For example:
class Base(object):
attribute = "attribute"
#property
def property_method(self):
return "property_method"
def regular_method(self):
return "regular_method"
#staticmethod
def static_method():
return "static_method"
#classmethod
def class_method(cls):
return "class_method"
class MyClass(Base):
pass
Here's the solution for staticmethod only. But I recommend to use the module posted here.
import inspect
def is_static_method(klass, attr, value=None):
"""Test if a value of a class is static method.
example::
class MyClass(object):
#staticmethod
def method():
...
:param klass: the class
:param attr: attribute name
:param value: attribute value
"""
if value is None:
value = getattr(klass, attr)
assert getattr(klass, attr) == value
for cls in inspect.getmro(klass):
if inspect.isroutine(value):
if attr in cls.__dict__:
bound_value = cls.__dict__[attr]
if isinstance(bound_value, staticmethod):
return True
return False
Why bother? You can just call g like you call f:
a = A()
a.f()
a.g()
Is it at all possible to monkey patch the value of a #property of an instance of a class that I do not control?
class Foo:
#property
def bar(self):
return here().be['dragons']
f = Foo()
print(f.bar) # baz
f.bar = 42 # MAGIC!
print(f.bar) # 42
Obviously the above would produce an error when trying to assign to f.bar. Is # MAGIC! possible in any way? The implementation details of the #property are a black box and not indirectly monkey-patchable. The entire method call needs to be replaced. It needs to affect a single instance only (class-level patching is okay if inevitable, but the changed behaviour must only selectively affect a given instance, not all instances of that class).
Subclass the base class (Foo) and change single instance's class to match the new subclass using __class__ attribute:
>>> class Foo:
... #property
... def bar(self):
... return 'Foo.bar'
...
>>> f = Foo()
>>> f.bar
'Foo.bar'
>>> class _SubFoo(Foo):
... bar = 0
...
>>> f.__class__ = _SubFoo
>>> f.bar
0
>>> f.bar = 42
>>> f.bar
42
from module import ClassToPatch
def get_foo(self):
return 'foo'
setattr(ClassToPatch, 'foo', property(get_foo))
To monkey patch a property, there is an even simpler way:
from module import ClassToPatch
def get_foo(self):
return 'foo'
ClassToPatch.foo = property(get_foo)
Idea: replace property descriptor to allow setting on certain objects. Unless a value is explicitly set this way, original property getter is called.
The problem is how to store the explicitly set values. We cannot use a dict keyed by patched objects, since 1) they are not necessarily comparable by identity; 2) this prevents patched objects from being garbage-collected. For 1) we could write a Handle that wraps objects and overrides comparison semantics by identity and for 2) we could use weakref.WeakKeyDictionary. However, I couldn't make these two work together.
Therefore we use a different approach of storing the explicitly set values on the object itself, using a "very unlikely attribute name". It is of course still possible that this name would collide with something, but that's pretty much inherent to languages such as Python.
This won't work on objects that lack a __dict__ slot. Similar problem would arise for weakrefs though.
class Foo:
#property
def bar (self):
return 'original'
class Handle:
def __init__(self, obj):
self._obj = obj
def __eq__(self, other):
return self._obj is other._obj
def __hash__(self):
return id (self._obj)
_monkey_patch_index = 0
_not_set = object ()
def monkey_patch (prop):
global _monkey_patch_index, _not_set
special_attr = '$_prop_monkey_patch_{}'.format (_monkey_patch_index)
_monkey_patch_index += 1
def getter (self):
value = getattr (self, special_attr, _not_set)
return prop.fget (self) if value is _not_set else value
def setter (self, value):
setattr (self, special_attr, value)
return property (getter, setter)
Foo.bar = monkey_patch (Foo.bar)
f = Foo()
print (Foo.bar.fset)
print(f.bar) # baz
f.bar = 42 # MAGIC!
print(f.bar) # 42
It looks like you need to move on from properties to the realms of data descriptors and non-data descriptors. Properties are just a specialised version of data descriptors. Functions are an example of non-data descriptors -- when you retrieve them from an instance they return a method rather than the function itself.
A non-data descriptor is just an instance of a class that has a __get__ method. The only difference with a data descriptor is that it has a __set__ method as well. Properties initially have a __set__ method that throws an error unless you provide a setter function.
You can achieve what you want really easily just by writing your own trivial non-data descriptor.
class nondatadescriptor:
"""generic nondata descriptor decorator to replace #property with"""
def __init__(self, func):
self.func = func
def __get__(self, obj, objclass):
if obj is not None:
# instance based access
return self.func(obj)
else:
# class based access
return self
class Foo:
#nondatadescriptor
def bar(self):
return "baz"
foo = Foo()
another_foo = Foo()
assert foo.bar == "baz"
foo.bar = 42
assert foo.bar == 42
assert another_foo.bar == "baz"
del foo.bar
assert foo.bar == "baz"
print(Foo.bar)
What makes all this work is that logic under the hood __getattribute__. I can't find the appropriate documentation at the moment, but order of retrieval is:
Data descriptors defined on the class are given the highest priority (objects with both __get__ and __set__), and their __get__ method is invoked.
Any attribute of the object itself.
Non-data descriptors defined on the class (objects with only a __get__ method).
All other attributes defined on the class.
Finally the __getattr__ method of the object is invoked as a last resort (if defined).
You can also patch property setters. Using #fralau 's answer:
from module import ClassToPatch
def foo(self, new_foo):
self._foo = new_foo
ClassToPatch.foo = ClassToPatch.foo.setter(foo)
reference
In case someone needs to patch a property while being able to call the original implementation, here is an example:
#property
def _cursor_args(self, __orig=mongoengine.queryset.base.BaseQuerySet._cursor_args):
# TODO: remove this hack when we upgrade MongoEngine
# https://github.com/MongoEngine/mongoengine/pull/2160
cursor_args = __orig.__get__(self)
if self._timeout:
cursor_args.pop("no_cursor_timeout", None)
return cursor_args
mongoengine.queryset.base.BaseQuerySet._cursor_args = _cursor_args
I would like to know what is the python way of initializing a class member but only when accessing it, if accessed.
I tried the code below and it is working but is there something simpler than that?
class MyClass(object):
_MY_DATA = None
#staticmethod
def _retrieve_my_data():
my_data = ... # costly database call
return my_data
#classmethod
def get_my_data(cls):
if cls._MY_DATA is None:
cls._MY_DATA = MyClass._retrieve_my_data()
return cls._MY_DATA
You could use a #property on the metaclass instead:
class MyMetaClass(type):
#property
def my_data(cls):
if getattr(cls, '_MY_DATA', None) is None:
my_data = ... # costly database call
cls._MY_DATA = my_data
return cls._MY_DATA
class MyClass(metaclass=MyMetaClass):
# ...
This makes my_data an attribute on the class, so the expensive database call is postponed until you try to access MyClass.my_data. The result of the database call is cached by storing it in MyClass._MY_DATA, the call is only made once for the class.
For Python 2, use class MyClass(object): and add a __metaclass__ = MyMetaClass attribute in the class definition body to attach the metaclass.
Demo:
>>> class MyMetaClass(type):
... #property
... def my_data(cls):
... if getattr(cls, '_MY_DATA', None) is None:
... print("costly database call executing")
... my_data = 'bar'
... cls._MY_DATA = my_data
... return cls._MY_DATA
...
>>> class MyClass(metaclass=MyMetaClass):
... pass
...
>>> MyClass.my_data
costly database call executing
'bar'
>>> MyClass.my_data
'bar'
This works because a data descriptor like property is looked up on the parent type of an object; for classes that's type, and type can be extended by using metaclasses.
This answer is for a typical instance attribute/method only, not for a class attribute/classmethod, or staticmethod.
For Python 3.8+, how about using the cached_property decorator? It memoizes.
from functools import cached_property
class MyClass:
#cached_property
def my_lazy_attr(self):
print("Initializing and caching attribute, once per class instance.")
return 7**7**8
For Python 3.2+, how about using both property and lru_cache decorators? The latter memoizes.
from functools import lru_cache
class MyClass:
#property
#lru_cache()
def my_lazy_attr(self):
print("Initializing and caching attribute, once per class instance.")
return 7**7**8
Credit: answer by Maxime R.
Another approach to make the code cleaner is to write a wrapper function that does the desired logic:
def memoize(f):
def wrapped(*args, **kwargs):
if hasattr(wrapped, '_cached_val'):
return wrapped._cached_val
result = f(*args, **kwargs)
wrapped._cached_val = result
return result
return wrapped
You can use it as follows:
#memoize
def expensive_function():
print "Computing expensive function..."
import time
time.sleep(1)
return 400
print expensive_function()
print expensive_function()
print expensive_function()
Which outputs:
Computing expensive function...
400
400
400
Now your classmethod would look as follows, for example:
class MyClass(object):
#classmethod
#memoize
def retrieve_data(cls):
print "Computing data"
import time
time.sleep(1) #costly DB call
my_data = 40
return my_data
print MyClass.retrieve_data()
print MyClass.retrieve_data()
print MyClass.retrieve_data()
Output:
Computing data
40
40
40
Note that this will cache just one value for any set of arguments to the function, so if you want to compute different values depending on input values, you'll have to make memoize a bit more complicated.
Consider the pip-installable Dickens package which is available for Python 3.5+. It has a descriptors package which provides the relevant cachedproperty and cachedclassproperty decorators, the usage of which is shown in the example below. It seems to work as expected.
from descriptors import cachedproperty, classproperty, cachedclassproperty
class MyClass:
FOO = 'A'
def __init__(self):
self.bar = 'B'
#cachedproperty
def my_cached_instance_attr(self):
print('Initializing and caching attribute, once per class instance.')
return self.bar * 2
#cachedclassproperty
def my_cached_class_attr(cls):
print('Initializing and caching attribute, once per class.')
return cls.FOO * 3
#classproperty
def my_class_property(cls):
print('Calculating attribute without caching.')
return cls.FOO + 'C'
Ring gives lru_cache-like interface but working with any kind of descriptor supports: https://ring-cache.readthedocs.io/en/latest/quickstart.html#method-classmethod-staticmethod
class Page(object):
(...)
#ring.lru()
#classmethod
def class_content(cls):
return cls.base_content
#ring.lru()
#staticmethod
def example_dot_com():
return requests.get('http://example.com').content
See the link for more details.
Imagine that i have f which is a function of a member of a class instance:
class A:
def b(self):
print 'hey'
a = A()
f = a.b
If I have another instance of the same class, let's say c = A() how can I reconstruct a new ff only using f and c, so calling ff() would result in c.b() instead of a.b()?
c = A()
ff = some_python_kungfu(f,c)
ff() #it is calling c.b()
Can you use a method reference for the class instead of the instance reference?
class A:
def whoami(self):
print 'I am %s' % id(self)
a = A()
c = A()
func = A.whoami
func(a)
func(c)
So you want to know how to rebind an already bound method to another instance, using only the bound method and the other instance. It can be done like this:
def some_python_kungfu(meth, obj):
return type(meth)(meth.__func__, obj, obj.__class__)
The __func__ attribute is really the same as Ned Batchelders im_func, but __func__ is forward-compatible with python 3.
There is one case where this will not work: methods of built-in classes. The __func__ and im_func attributes are only available on user-defined classes. Therefore, this will fail:
a = "that's no ordinary rabbit"
b = "consult the book of armaments"
b_split = some_python_kungfu(a.split, b)
A slight modification of Ned's solution will work on both built-in and user-defined classes:
def some_python_kungfu(meth, obj):
return getattr(obj, meth.__name__)
So will this always work then? Well... no, but the stumbling block a rather obscure and (I guess) seldom occuring problem: if the name of the method (meth.__name__) is not the same as the name it has in the class dictionary ('b'), then getattr will either return the wrong attribute or raise an AttributeError. For example:
def external(self):
pass
class A(object):
b = external
Here A.b.__name__ == 'external' instead of 'b', so getattr(obj, 'external') will be called instead of getattr(obj, 'b').
While both previous approaches have problems, one with built-in classes and one with patched-together classes, both problems do not occur simultaneously in any circumstance. Therefore, a combination will work in all cases:
def some_python_kungfu(meth, obj):
try:
return type(meth)(meth.__func__, obj, obj.__class__)
except AttributeError:
# meth is a built-in method, so meth.__name__ is always correct
return getattr(obj, meth.__name__)
As explained elsewhere on this page, your best bet would probably be to ignore this whole mess and do it some cleaner way, like for instance using the unbound methods and passing in the first argument (self) manually, as in Cixates answer. But who knows, this may prove useful to some of you some day perhaps, in a somewhat bizarre set of circumstances. ;)
I'm not sure this would work in all cases, but:
def some_python_kungfu(meth, obj):
"""Get a bound method on `obj` corresponding to the method `meth`."""
return getattr(obj, meth.im_func.__name__)