Often in Python it is helpful to make use of duck typing, for instance, imagine I have an object spam, whose prompt attribute controls the prompt text in my application. Normally, I would say something like:
spam.prompt = "fixed"
for a fixed prompt. However, a dynamic prompt can also be achived - while I can't change the spam class to use a function as the prompt, thanks to duck typing, because the userlying spam object calls str, I can create a dynamic prompt like so:
class MyPrompt:
def __str__( self ):
return eggs.get_user_name() + ">"
spam.prompt = MyPrompt()
This principal could be extended to make any attribute dynamic, for instance:
class MyEnabled:
def __bool__( self ):
return eggs.is_logged_in()
spam.enabled = MyEnabled()
Sometimes though, it would be more succinct to have this inline, i.e.
spam.prompt = lambda: eggs.get_user_name() + ">"
spam.enabled = eggs.is_logged_in
These of course don't work, because neither the __str__ of the lambda or the __bool__ of the function return the actual value of the call.
I feel like a solution for this should be simple, am I missing something, or do I need to wrap my function in a class every time?
What you want are computed attributes. Python's support for computed attributes is the descriptor protocol, which has a generic implementation as the builtin property type.
Now the trick is that, as documented (cf link above), descriptors only work when they are class attributes. Your code snippet is incomplete as it doesn't contains the definition of the spam object but I assume it's a class instance, so you cannot just do spam.something = property(...) - as the descriptor protocol wouldn't then be invoked on property().
The solution here is the good old "strategy" design pattern: use properties (or custom descriptors, but if you only have a couple of such attributes the builtin property will work just fine) that delegates to a "strategy" function:
def default_prompt_strategy(obj):
return "fixed"
def default_enabled_strategy(obj):
return False
class Spam(object):
def __init__(self, prompt_strategy=default_prompt_strategy, enabled_strategy=default_enabled_strategy):
self.prompt = prompt_strategy
self.enabled = enabled_strategy
#property
def prompt(self):
return self._prompt_strategy(self)
#prompt.setter
def prompt(self, value):
if not callable(value):
raise TypeError("PromptStrategy must be a callable")
self._prompt_strategy = value
#property
def enabled(self):
return self._enabled_strategy(self)
#enabled.setter
def enabled(self, value):
if not callable(value):
raise TypeError("EnabledtStrategy must be a callable")
self._enabled_strategy = value
class Eggs(object):
def is_logged_in(self):
return True
def get_user_name(self):
return "DeadParrot"
eggs = Eggs()
spam = Spam(enabled_strategy=lambda obj: eggs.is_logged_in())
spam.prompt = lambda obj: "{}>".format(eggs.get_user_name())
Related
I am writing a python library used by importing and (optionally) sub-classing some of the 'helper classes' it provides. I fail to come up with a design that would properly let static analysis tools properly recognise the types that my 'helper classes' methods deal with. Here's a MWE illustrating (one of) the issues I run into:
My lib
from typing import Dict
class Thing:
def shout(self):
print(f"{self} says AAAAAAAAAaaaaaaaa")
class ContainerOfThings:
def __init__(self):
thing_cls = self._thing_cls = get_unique_subclass(Thing)
self._things: Dict[str, thing_cls] = {}
def add_thing(self, id_: str):
self._things[id_] = self._thing_cls()
def get_thing(self, id_: str):
return self._things[id_]
def get_unique_subclass(cls):
# this works but maybe there's a better way to do this?
classes = cls.__subclasses__()
if len(classes) == 0:
return cls
elif len(classes) == 1:
return classes[0]
elif len(classes) > 1:
raise RuntimeError(
"This class should only be subclassed once", cls, classes
)
What I expect users to do with it
class BetterThing(Thing):
def be_civilized(self):
print(f"{self} says howdy!")
container = ContainerOfThings()
container.add_thing("some_id")
thingy = container.get_thing("some_id")
thingy.be_civilized()
thingy.do_something_invalid() # here I would like mypy to detect that this will not work
This snippet does not alarm static analysis tools, because thingy is detected as Any, but fails at runtime on the last line because do_something_invalid() is not defined. Isn't it possible to give hints that thingy is in fact an instance of BetterThing here?
My attempts so far:
Attempt 1
Annotate ContainerOfThings._things as Dict[str, Thing] instead of Dict[str, thing_cls]
This passes mypy, but pycharm detects thingy as an instance of Thing and thus complains about "Unresolved attribute reference 'be_civilized' for class 'Thing'"
Attempt 2
Annotate ContainerOfThings.get_thing() return value as Thing
Less surprisingly, this triggers errors from both pycharm and mypy about Thing not having the 'be_civilized' attribute.
Attempt 3
Use ThingType = TypeVar("ThingType", bound=Thing) as return value for ContainerOfThings.get_thing()
I believe (?) that this is what TypeVar is intended for, and it works, except for the fact that mypy then requires thingy to be be annotated with BetterThing, along with every return value of ContainerOfThings.get_thing(), which will be quite cumbersome with my 'real' library.
Is there an elegant solution for this? Is get_unique_subclass() too dirty a trick to play nice with static analysis? Is there something clever to do with typing_extensions.Protocol that I could not come up with?
Thanks for your suggestions.
Basically you need ContainerOfThings to be generic:
https://mypy.readthedocs.io/en/stable/generics.html#defining-generic-classes
And then I think it would be better for ContainerOfThings to be explicit about the type of thing that it will generate instead of auto-magically locating some sub-class that has been defined.
We can put this together in a way that will satisfy mypy (and I would expect pycharm too, though I haven't tried it)...
from typing import Dict, Generic, Type, TypeVar
class Thing:
def shout(self):
print(f"{self} says AAAAAAAAAaaaaaaaa")
T = TypeVar('T', bound=Thing)
class ContainerOfThings(Generic[T]):
def __init__(self, thing_cls: Type[T]):
self._thing_cls = thing_cls
self._things: Dict[str, T] = {}
def add_thing(self, id_: str):
self._things[id_] = self._thing_cls()
def get_thing(self, id_: str) -> T:
return self._things[id_]
class BetterThing(Thing):
def be_civilized(self):
print(f"{self} says howdy!")
container = ContainerOfThings(BetterThing)
container.add_thing("some_id")
thingy = container.get_thing("some_id")
thingy.be_civilized() # OK
thingy.do_something_invalid() # error: "BetterThing" has no attribute "do_something_invalid"
Here is a simplified version of what I am dealing with:
def mydecorator(prop):
def wrapper(self_arg):
return prop(self_arg) + 'bar'
return wrapper
class Foo(object):
def __init__(self):
self.value = 'foo'
#property
#mydecorator
def foo(self):
return self.value
#property
def doesnt_have_my_decorator(self):
return 'something that requires a lot of computation'
f = Foo()
print f.foo # prints 'foobar'
Now what I would like to be able to do, is introspect f.foo, without actually accessing its value, and check if it is decorated with #mydecorator. Is this possible?
The use case is to be able whitelist a property as "safe" in a certain context, without actually accessing it's value in case it is "unsafe".
I have seen this great post, but it seems like that method requires foo to have already been accessed.
I have found that I can see that it is a property with:
f.__class__.__dict__['foo'].__class__
but I haven't been able to find any references to mydecorator. Since it's Python, I am sure there is a way, but I haven't been able to figure it out so far...
In Python a decorator is merely a shortened syntax. E.g.:
#property
#mydecorator
def foo(self):
return self.value
is exactly the same as this:
def foo(self):
return self.value
foo = property (mydecorator (foo))
Since your decorator doesn't leave any detectable trace on its return value, it is not possible to determine if it was applied. You could instead rewrite to e.g.:
def mydecorator(prop):
def wrapper(self_arg):
return prop(self_arg) + 'bar'
wrapper._mydecorator = True
return wrapper
and then use this expression to test it:
hasattr (type (f).foo.fget, '_mydecorator')
I have written a Python class for parsing a specialized text format.
class Parser(object):
def __init__(self):
# Initialize parser instance
def parseFile(self , filename):
pass
def modifyParser(self , *args , **kwargs):
pass
#Classmethod has same name as instance method - this does not work.
#classmethod
def parseFile(cls , filename)
parser = Parser( )
return parser.parseFile( filename )
As indicated the parser can be modified with the modifyParser method - but in most cases I will just use the Parser instance as it comes from the Parser.__init__(). I would like to be able to do this:
# Parse file using 'custom' parser:
parser = Parser( )
parser.modifyParser( ... )
result = parser.parseFile("file.input")
# Parse using the default parser - do not explicitly instantiate an object:
result = Parser.parseFile("file.input")
This requires that the parseFile( ) method can be called both as an instance method - with a self - and as a classmethod. Is this possible? Bad form?
If I were you, I'd offer two distinct functions:
mymodule.Parser().parseFile() (instance method), and
mymodule.parseFile() (module-level function that uses the default instance).
This is what happens for example with the standard json module, where you have json.JSONDecoder().decode() and json.loads(). Offering two distinct functions makes the code less ambiguous, more explicit and more predictable (in my opinion).
However, yes: what you want to do is possible. You have to implement your own descriptor using __get__. Here's an example:
from functools import partial
class class_and_instance_method(object):
def __init__(self, func):
self.func = func
def __get__(self, obj, type=None):
first_arg = obj if obj is not None else type
return partial(self.func, first_arg)
class Parser(object):
#class_and_instance_method
def parseFile(self):
if isinstance(self, type):
print('using default parser')
else:
print('using the current instance')
>>> Parser.parseFile()
using default parser
>>> p = Parser()
>>> p.parseFile()
using the current instance
You'll have to use two separate names. In python due to it's dynamic nature there's no operator overloading as in C++, when one function has same name with different arguments.
When you say def in your script, you tell Python "set the following object(function object) to this name". So in your code you just redefine the name to reference classmethod and your instance method function object is lost.
Solution: use different names for instace method and class method.
Recently, I faced a problem which was similar to this question:
Accessing the class that owns a decorated method from the decorator
My rep was not high enough to comment there, so I am starting a new question to address some improvements to the answer to that problem.
This is what I needed:
def original_decorator(func):
# need to access class here
# for eg, to append the func itself to class variable "a", to register func
# or say, append func's default arg values to class variable "a"
return func
class A(object):
a=[]
#classmethod
#original_decorator
def some_method(self,a=5):
''' hello'''
print "Calling some_method"
#original_decorator
def some_method_2(self):
''' hello again'''
print "Calling some_method_2"
The solution would need to work both with class methods and instance methods, the method returned from the decorator should work and behave just the same way if it was undecorated i.e. method signature should be preserved.
The accepted answer for that question returned a Class from the decorator and the metaclass identified that specific Class, and did the "class-accessing" operations.
The answer did mention itself as a rough solution, but clearly it had a few caveats :
Decorator returned a class and it was not callable. Obviously, it can be made callable easily, but the returned value is still a class - it just behaves the same way while calling, but its properties and behaviors would be different. Essentially, it would not work the same way as the undecorated method.
It forced the decorator to return a custom-type class and all the "class-accessing" code was put inside the metaclass directly. It is simply not nice, writing the decorator should not enforce touching the metaclass directly.
I have tried to come up with a better solution, documented in the answer.
Here is the solution.
It uses a decorator (which would work on "class-accessing" decorators) and a metaclass, which would fulfill all my requirements and address the problems of that answer. Probably the best advantage is that the "class-accessing" decorators can just access the class, without even touching the metaclass.
# Using metaclass and decorator to allow class access during class creation time
# No method defined within the class should have "_process_meta" as arg
# Potential problems: Using closures, function.func_globals is read-only
from functools import partial
import inspect
class meta(type):
def __new__(cls, name, base, clsdict):
temp_cls = type.__new__(cls, name, base, clsdict)
methods = inspect.getmembers(temp_cls, inspect.ismethod)
for (method_name, method_obj) in methods:
tmp_spec = inspect.getargspec(method_obj)
if "__process_meta" in tmp_spec.args:
what_to_do, main_func = tmp_spec.defaults[:-1]
f = method_obj.im_func
f.func_code, f.func_defaults, f.func_dict, f.func_doc, f.func_name = main_func.func_code, main_func.func_defaults, main_func.func_dict, main_func.func_doc, main_func.func_name
mod_func = what_to_do(temp_cls, f)
f.func_code, f.func_defaults, f.func_dict, f.func_doc, f.func_name = mod_func.func_code, mod_func.func_defaults, mod_func.func_dict, mod_func.func_doc, mod_func.func_name
return temp_cls
def do_it(what_to_do, main_func=None):
if main_func is None:
return partial(do_it, what_to_do)
def whatever(what_to_do=what_to_do, main_func=main_func, __process_meta=True):
pass
return whatever
def original_classmethod_decorator(cls, func):
# cls => class of the method
# appends default arg values to class variable "a"
func_defaults = inspect.getargspec(func).defaults
cls.a.append(func_defaults)
func.__doc__ = "This is a class method"
print "Calling original classmethod decorator"
return func
def original_method_decorator(cls, func):
func_defaults = inspect.getargspec(func).defaults
cls.a.append(func_defaults)
func.__doc__ = "This is a instance method" # Can change func properties
print "Calling original method decorator"
return func
class A(object):
__metaclass__ = meta
a = []
#classmethod
#do_it(original_classmethod_decorator)
def some_method(cls, x=1):
''' hello'''
print "Calling original class method"
#do_it(original_method_decorator)
def some_method_2(self, y=2):
''' hello again'''
print "Calling original method"
# signature preserved
print(inspect.getargspec(A.some_method))
print(inspect.getargspec(A.some_method_2))
Open to suggestions on whether this approach has any ceveats.
I've started to use the python descriptor protocol more extensively in the code I've been writing. Typically, the default python lookup magic is what I want to happen, but sometimes I'm finding I want to get the descriptor object itself instead the results of its __get__ method. Wanting to know the type of the descriptor, or access state stored in the descriptor, or somesuch thing.
I wrote the code below to walk the namespaces in what I believe is the correct ordering, and return the attribute raw regardless of whether it is a descriptor or not. I'm surprised though that I can't find a built-in function or something in the standard library to do this -- I figure it has to be there and I just haven't noticed it or googled for the right search term.
Is there functionality somewhere in the python distribution that already does this (or something similar)?
Thanks!
from inspect import isdatadescriptor
def namespaces(obj):
obj_dict = None
if hasattr(obj, '__dict__'):
obj_dict = object.__getattribute__(obj, '__dict__')
obj_class = type(obj)
return obj_dict, [t.__dict__ for t in obj_class.__mro__]
def getattr_raw(obj, name):
# get an attribute in the same resolution order one would normally,
# but do not call __get__ on the attribute even if it has one
obj_dict, class_dicts = namespaces(obj)
# look for a data descriptor in class hierarchy; it takes priority over
# the obj's dict if it exists
for d in class_dicts:
if name in d and isdatadescriptor(d[name]):
return d[name]
# look for the attribute in the object's dictionary
if obj_dict and name in obj_dict:
return obj_dict[name]
# look for the attribute anywhere in the class hierarchy
for d in class_dicts:
if name in d:
return d[name]
raise AttributeError
Edit Wed, Oct 28, 2009.
Denis's answer gave me a convention to use in my descriptor classes to get the descriptor objects themselves. But, I had an entire class hierarchy of descriptor classes, and I didn't want to begin every __get__ function with a boilerplate
def __get__(self, instance, instance_type):
if instance is None:
return self
...
To avoid this, I made the root of the descriptor class tree inherit from the following:
def decorate_get(original_get):
def decorated_get(self, instance, instance_type):
if instance is None:
return self
return original_get(self, instance, instance_type)
return decorated_get
class InstanceOnlyDescriptor(object):
"""All __get__ functions are automatically wrapped with a decorator which
causes them to only be applied to instances. If __get__ is called on a
class, the decorator returns the descriptor itself, and the decorated
__get__ is not called.
"""
class __metaclass__(type):
def __new__(cls, name, bases, attrs):
if '__get__' in attrs:
attrs['__get__'] = decorate_get(attrs['__get__'])
return type.__new__(cls, name, bases, attrs)
Most descriptors do their job when accessed as instance attribute only. So it's convenient to return itself when it's accessed for class:
class FixedValueProperty(object):
def __init__(self, value):
self.value = value
def __get__(self, inst, cls):
if inst is None:
return self
return self.value
This allows you to get descriptor itself:
>>> class C(object):
... prop = FixedValueProperty('abc')
...
>>> o = C()
>>> o.prop
'abc'
>>> C.prop
<__main__.FixedValueProperty object at 0xb7eb290c>
>>> C.prop.value
'abc'
>>> type(o).prop.value
'abc'
Note, that this works for (most?) built-in descriptors too:
>>> class C(object):
... #property
... def prop(self):
... return 'abc'
...
>>> C.prop
<property object at 0xb7eb0b6c>
>>> C.prop.fget
<function prop at 0xb7ea36f4>
Accessing descriptor could be useful when you need to extent it in subclass, but there is a better way to do this.
The inspect library provides a function to retrieve an attribute without any descriptor magic: inspect.getattr_static.
Documentation: https://docs.python.org/3/library/inspect.html#fetching-attributes-statically
(This is an old question, but I keep coming across it when trying to remember how to do this, so I'm posting this answer so I can find it again!)
The above method
class FixedValueProperty(object):
def __init__(self, value):
self.value = value
def __get__(self, inst, cls):
if inst is None:
return self
return self.value
Is a great method whenever you control the code of the property, but there are some cases, such as when the property is part of a library controlled by someone else, where another approach is useful. This alternative approach can also be useful in other situations such as implementing object mapping, walking a name-space as described in the question, or other specialised libraries.
Consider a class with a simple property:
class ClassWithProp:
#property
def value(self):
return 3
>>>test=ClassWithProp()
>>>test.value
3
>>>test.__class__.__dict__.['value']
<property object at 0x00000216A39D0778>
When accessed from the container objects class dict, the 'descriptor magic' is bypassed. Note also that if we assign the property to a new class variable, it behaves just like the original with 'descriptor magic', but if assigned to an instance variable, the property behaves as any normal object and also bypasses 'descriptor magic'.
>>> test.__class__.classvar = test.__class__.__dict__['value']
>>> test.classvar
3
>>> test.instvar = test.__class__.__dict__['value']
>>> test.instvar
<property object at 0x00000216A39D0778>
Let's say we want to get the descriptor for obj.prop where type(obj) is C.
C.prop usually works because the descriptor usually returns itself when accessed via C (i.e., bound to C). But C.prop may trigger a descriptor in its metaclass. If prop were not present in obj, obj.prop would raise AttributeError while C.prop might not. So it's better to use inspect.getattr_static(obj, 'prop').
If you are not satisfied with that, here's a CPython-specific method (from _PyObject_GenericGetAttrWithDict in Objects/object.c):
import ctypes, _ctypes
_PyType_Lookup = ctypes.pythonapi._PyType_Lookup
_PyType_Lookup.argtypes = (ctypes.py_object, ctypes.py_object)
_PyType_Lookup.restype = ctypes.c_void_p
def type_lookup(ty, name):
"""look for a name through the MRO of a type."""
if not isinstance(ty, type):
raise TypeError('ty must be a type')
result = _PyType_Lookup(ty, name)
if result is None:
raise AttributeError(name)
return _ctypes.PyObj_FromPtr(result)
type_lookup(type(obj), 'prop') returns the descriptor in the same way when CPython uses it at obj.prop if obj is a usual object (not class, for example).