We Keep Coding

Count calls of a method that may or may not be called inside a decorator

I have this class:
class SomeClass(object):
def __init__(self):
self.cache = {}
def check_cache(method):
def wrapper(self):
if method.__name__ in self.cache:
print('Got it from the cache!')
return self.cache[method.__name__]
print('Got it from the api!')
self.cache[method.__name__] = method(self)
return self.cache[method.__name__]
return wrapper
#check_cache
def expensive_operation(self):
return get_data_from_api()
def get_data_from_api():
"This would call the api."
return 'lots of data'
The idea is that I can use the #check_cache decorator to keep the expensive_operation method from calling an api additional times if the result is already cached.
This works fine, it seems.
>>> sc.expensive_operation()
Got it from the api!
'lots of data'
>>> sc.expensive_operation()
Got it from the cache!
'lots of data'
But I would love to be able to test it with another decorator:
import unittest
class SomeClassTester(SomeClass):
def counted(f):
def wrapped(self, *args, **kwargs):
wrapped.calls += 1
return f(self, *args, **kwargs)
wrapped.calls = 0
return wrapped
#counted
def expensive_operation(self):
return super().expensive_operation()
class TestSomeClass(unittest.TestCase):
def test_api_is_only_called_once(self):
sc = SomeClassTester()
sc.expensive_operation()
self.assertEqual(sc.expensive_operation.calls, 1) # is 1
sc.expensive_operation()
self.assertEqual(sc.expensive_operation.calls, 1) # but this goes to 2
unittest.main()
The problem is that the counted decorator counts the number of times the wrapper function is called, not this inner function.
How do I count that from SomeClassTester?

There's no easy way to do this. Your current test applies the decorators in the wrong order. You want check_cache(counted(expensive_operation)), but you're getting the counted decorator on the outside instead: counted(check_cache(expensive_operation)).
There's no easy way to fix this within the counted decorator, because by the time it gets called, the original function is already wrapped up by the check_cache decorator, and there's no easy way to change the wrapper (it holds its reference to the original function in a closure cell, which is read-only from the outside).
One possible way to make it work is to rebuild the whole method with the decorators in the desired order. You can get a reference to the original method from the closure cell:
class SomeClassTester(SomeClass):
def counted(f):
def wrapped(self, *args, **kwargs):
wrapped.calls += 1
return f(self, *args, **kwargs)
wrapped.calls = 0
return wrapped
expensive_operation = SomeClass.check_cache(
counted(SomeClass.expensive_operation.__closure__[0].cell_value)
)
This is of course far from ideal, since you need to know exactly what decorators are being applied on the method in SomeClass in order to apply them again properly. You also need to know the internals of those decorators so that you can get the right closure cell (the [0] index may not be correct if the other decorator gets changed to differently).
Another (perhaps better) approach might be to change SomeClass in such a way that you can inject your counting code in between the changed method and the expensive bit you want to count. For example, you could have the real expensive part be in _expensive_method_implementation, while the decorated expensive_method is just a simple wrapper that calls it. The test class can override the _implementation method with its own decorated version (which might even skip the actually expensive part and just return dummy data). It doesn't need to override the regular method or mess with its decorators.

It is impossible to do this, without modifying the base class to provide hooks or changing the whole decorated function in derived class based on internal knowledge of base class. Though there is a third way based on internal working of cache decorator, basically change your cache dict so that it counts
class CounterDict(dict):
def __init__(self, *args):
super().__init__(*args)
self.count = {}
def __setitem__(self, key, value):
try:
self.count[key] += 1
except KeyError:
self.count[key] = 1
return super().__setitem__(key, value)
class SomeClassTester(SomeClass):
def __init__(self):
self.cache = CounterDict()
class TestSomeClass(unittest.TestCase):
def test_api_is_only_called_once(self):
sc = SomeClassTester()
sc.expensive_operation()
self.assertEqual(sc.cache.count['expensive_operation'], 1) # is 1
sc.expensive_operation()
self.assertEqual(sc.cache.count['expensive_operation'], 1) # is 1

Python - extending properties like you'd extend a function

Question
How can you extend a python property?
A subclass can extend a super class's function by calling it in the overloaded version, and then operating on the result. Here's an example of what I mean when I say "extending a function":
# Extending a function (a tongue-in-cheek example)
class NormalMath(object):
def __init__(self, number):
self.number = number
def add_pi(self):
n = self.number
return n + 3.1415
class NewMath(object):
def add_pi(self):
# NewMath doesn't know how NormalMath added pi (and shouldn't need to).
# It just uses the result.
n = NormalMath.add_pi(self)
# In NewMath, fractions are considered too hard for our users.
# We therefore silently convert them to integers.
return int(n)
Is there an analogous operation to extending functions, but for functions that use the property decorator?
I want to do some additional calculations immediately after getting an expensive-to-compute attribute. I need to keep the attribute's access lazy. I don't want the user to have to invoke a special routine to make the calculations. basically, I don't want the user to ever know the calculations were made in the first place. However, the attribute must remain a property, since i've got legacy code I need to support.
Maybe this is a job for decorators? If I'm not mistaken, decorator is a function that wraps another function, and I'm looking to wrap a property with some more calculations, and then present it as a property again, which seems like a similar idea... but I can't quite figure it out.
My Specific Problem
I've got a base class LogFile with an expensive-to-construct attribute .dataframe. I've implemented it as a property (with the property decorator), so it won't actually parse the log file until I ask for the dataframe. So far, it works great. I can construct a bunch (100+) LogFile objects, and use cheaper methods to filter and select only the important ones to parse. And whenever I'm using the same LogFile over and over, i only have to parse it the first time I access the dataframe.
Now I need to write a LogFile subclass, SensorLog, that adds some extra columns to the base class's dataframe attribute, but I can't quite figure out the syntax to call the super class's dataframe construction routines (without knowing anything about their internal workings), then operate on the resulting dataframe, and then cache/return it.
# Base Class - rules for parsing/interacting with data.
class LogFile(object):
def __init__(self, file_name):
# file name to find the log file
self.file_name = file_name
# non-public variable to cache results of parse()
self._dataframe = None
def parse(self):
with open(self.file_name) as infile:
...
...
# Complex rules to interpret the file
...
...
self._dataframe = pandas.DataFrame(stuff)
#property
def dataframe(self):
"""
Returns the dataframe; parses file if necessary. This works great!
"""
if self._dataframe is None:
self.parse()
return self._dataframe
#dataframe.setter
def dataframe(self,value):
self._dataframe = value
# Sub class - adds more information to data, but does't parse
# must preserve established .dataframe interface
class SensorLog(LogFile):
def __init__(self, file_name):
# Call the super's constructor
LogFile.__init__(self, file_name)
# SensorLog doesn't actually know about (and doesn't rely on) the ._dataframe cache, so it overrides it just in case.
self._dataframe = None
# THIS IS THE PART I CAN'T FIGURE OUT
# Here's my best guess, but it doesn't quite work:
#property
def dataframe(self):
# use parent class's getter, invoking the hidden parse function and any other operations LogFile might do.
self._dataframe = LogFile.dataframe.getter()
# Add additional calculated columns
self._dataframe['extra_stuff'] = 'hello world!'
return self._dataframe
#dataframe.setter
def dataframe(self, value):
self._dataframe = value
Now, when these classes are used in an interactive session, the user should be able to interact with either in the same way.
>>> log = LogFile('data.csv')
>>> print log.dataframe
#### DataFrame with 10 columns goes here ####
>>> sensor = SensorLog('data.csv')
>>> print sensor.dataframe
#### DataFrame with 11 columns goes here ####
I have lots of existing code that takes a LogFile instance which provides a .dataframe attribute and dos something interesting (mostly plotting). I would LOVE to have SensorLog instances present the same interface so they can use the same code. Is it possible to extend the super-class's dataframe getter to take advantage of existing routines? How? Or am I better off doing this a different way?
Thanks for reading that huge wall of text. You are an internet super hero, dear reader. Got any ideas?

You should be calling the superclass properties, not bypassing them via self._dataframe. Here's a generic example:
class A(object):
def __init__(self):
self.__prop = None
#property
def prop(self):
return self.__prop
#prop.setter
def prop(self, value):
self.__prop = value
class B(A):
def __init__(self):
super(B, self).__init__()
#property
def prop(self):
value = A.prop.fget(self)
value['extra'] = 'stuff'
return value
#prop.setter
def prop(self, value):
A.prop.fset(self, value)
And using it:
b = B()
b.prop = dict((('a', 1), ('b', 2)))
print(b.prop)
Outputs:
{'a': 1, 'b': 2, 'extra': 'stuff'}
I would generally recommend placing side-effects in setters instead of getters, like this:
class A(object):
def __init__(self):
self.__prop = None
#property
def prop(self):
return self.__prop
#prop.setter
def prop(self, value):
self.__prop = value
class B(A):
def __init__(self):
super(B, self).__init__()
#property
def prop(self):
return A.prop.fget(self)
#prop.setter
def prop(self, value):
value['extra'] = 'stuff'
A.prop.fset(self, value)
Having costly operations within a getter is also generally to be avoided (such as your parse method).

If I understand correctly what you want to do is call the parent's method from the child instance. The usual way to do that is by using the super built-in.
I've taken your tongue-in-cheek example and modified it to use super in order to show you:
class NormalMath(object):
def __init__(self, number):
self.number = number
def add_pi(self):
n = self.number
return n + 3.1415
class NewMath(NormalMath):
def add_pi(self):
# this will call NormalMath's add_pi with
normal_maths_pi_plus_num = super(NewMath, self).add_pi()
return int(normal_maths_pi_plus_num)
In your Log example, instead of calling:
self._dataframe = LogFile.dataframe.getter()
you should call:
self._dataframe = super(SensorLog, self).dataframe
You can read more about super here
Edit: Even thought the example I gave you deals with methods, to do the same with #properties shouldn't be a problem.

You have some possibilities to consider:
1/ Inherit from logfile and override parse in your derived sensor class. It should be possible to modify your methods that work on dataframe to work regardless of the number of members that dataframe has - as you are using pandas a lot of it is done for you.
2/ Make sensor an instance of logfile then provide its own parse method.
3/ Generalise parse, and possibly some of your other methods, to use a list of data descriptors and possibly a dictionary of methods/rules either set in your class initialiser or set by a methods.
4/ Look at either making more use of the methods already in pandas, or possibly, extending pandas to provide the missing methods if you and others think that they would be accepted into pandas as useful extensions.
Personally I think that you would find the benefits of options 3 or 4 to be the most powerful.

The problem is that you're missing a self going into the parent class. If your parent is a singleton then a #staticmethod should work.
class X():
x=1
#staticmethod
def getx():
return X.x
class Y(X):
y=2
def getyx(self):
return X.getx()+self.y
wx = Y()
wx.getyx()
3

How can I programmatically change the argspec of a function *not* in a python decorator?

Very closely related to: How can I programmatically change the argspec of a function in a python decorator?
The decorator module provides the means to make a decorator function that preserves the argspec of the decorated function.
If I define a function that is not used as a decorator, is there a way to copy another function's argspec?
Example use case:
class Blah(object):
def foo(self, *args, **kwargs):
""" a docstr """
result = bar(*args, **kwargs)
result = result**2 # just so it's clear we're doing something extra here...
return result
def bar(x, y, z=1, q=2):
""" a more useful docstr, saying what x,y,z,q do """
return x+y*z+q
I would like to have foo's argspec look like bar's, but the source to stay unchanged (i.e., inspect.getsource(foo) would still show the result junk). The main purpose for this is to get sphinx docs and ipython's interactive help to show the appropriate arguments.
As the answers to the other question said, the decorator package shows a way to do this, but I got lost within the meat of that code. It seems that the decorator package is recompiling the source, or something like that. I had hoped a simpler approach, e.g. something like foo.argspec = bar.argspec, would be possible.

A decorator is simply a function that does something with another function. So, technically, you could put the required code directly underneath the foo method and then, technically, you would be changing foo without using a decorator, but it would be a horrible mess.
The easiest way to do what you want is going to be to make a decorator that takes a second function (bar in this case) as an argument so it knows which signature to copy. The class code would then look something like:
class Blah(object):
#copy_argspec(bar)
def foo(self, *args, **kwargs):
""" a docstr """
result = bar(*args, **kwargs)
result = result**2 # just so it's clear we're doing something extra here...
return result
You'll have to have bar defined before instead of after the class.
.
.
.
. . . time passes . . .
.
.
.
Okay, luckily I found an old decorator I could adapt.
help(Blah.foo) looks like this before decoration:
Help on method foo in module __main__:
foo(self, *args, **kwargs) unbound __main__.Blah method
a docstr
and after decoration it looks like this:
Help on method foo in module __main__:
foo(self, x, y, z=1, q=2) unbound __main__.Blah method
a more useful docstr, saying what x,y,z,q do
Here's the decorator I used:
import inspect
class copy_argspec(object):
"""
copy_argspec is a signature modifying decorator. Specifically, it copies
the signature from `source_func` to the wrapper, and the wrapper will call
the original function (which should be using *args, **kwds). The argspec,
docstring, and default values are copied from src_func, and __module__ and
__dict__ from tgt_func.
"""
def __init__(self, src_func):
self.argspec = inspect.getargspec(src_func)
self.src_doc = src_func.__doc__
self.src_defaults = src_func.func_defaults
def __call__(self, tgt_func):
tgt_argspec = inspect.getargspec(tgt_func)
need_self = False
if tgt_argspec[0][0] == 'self':
need_self = True
name = tgt_func.__name__
argspec = self.argspec
if argspec[0][0] == 'self':
need_self = False
if need_self:
newargspec = (['self'] + argspec[0],) + argspec[1:]
else:
newargspec = argspec
signature = inspect.formatargspec(
formatvalue=lambda val: "",
*newargspec
)[1:-1]
new_func = (
'def _wrapper_(%(signature)s):\n'
' return %(tgt_func)s(%(signature)s)' %
{'signature':signature, 'tgt_func':'tgt_func'}
)
evaldict = {'tgt_func' : tgt_func}
exec new_func in evaldict
wrapped = evaldict['_wrapper_']
wrapped.__name__ = name
wrapped.__doc__ = self.src_doc
wrapped.func_defaults = self.src_defaults
wrapped.__module__ = tgt_func.__module__
wrapped.__dict__ = tgt_func.__dict__
return wrapped

Python: renaming method names on-the-fly

I have many files using classes with the following syntax:
o = module.CreateObject()
a = o.get_Field
and now the implementation has changed from 'get_XXX' and 'set_XXX' to just 'XXX':
o = module.CreateObject()
a = o.Field
This implementation is an external package, which I don't want to change. Is it possible to write a wrapper which will on-the-fly intercept all calls to 'get_XXX' and replace then with calls to the new name 'XXX'?
o = MyRenamer(module.CreateObject())
a = o.get_Field # works as before, o.Field is called
a = o.DoIt() # works as before, o.DoIt is called
It needs to intercept all calls, not just to a finite-set of fields, decide based on the method name if to modify it and cause a method with a modified name to be called.

If you want to continue to use get_Field and set_Field on an object that has switched to using properties (where you simply access or assign to Field), it's possible to use an wrapper object:
class NoPropertyAdaptor(object):
def __init__(self, obj):
self.obj = obj
def __getattr__(self, name):
if name.startswith("get_"):
return lambda: getattr(self.obj, name[4:])
elif name.startswith("set_"):
return lambda value: setattr(self.obj, name[4:], value)
else:
return getattr(self.obj, name)
This will have problems if you are using extra syntax, like indexing or iteration on the object, or if you need to recognize the type of the object using isinstance.
A more sophisticated solution would be to create a subclass that does the name rewriting and force the object to use it. This isn't exactly a wrapping, since outside code will still deal with the object directly (and so magic methods and isinstance) will work as expected. This approach will work for most objects, but it might fail for types that have fancy metaclass magic going on and for some builtin types:
def no_property_adaptor(obj):
class wrapper(obj.__class__):
def __getattr__(self, name):
if name.startswith("get_"):
return lambda: getattr(self, name[4:])
elif name.startswith("set_"):
return lambda value: setattr(self, name[4:], value)
else:
return super(wrapper, self).__getattr__(name)
obj.__class__ = wrapper
return obj

You can 'monkey patch' any python class; import the class directly and add a property:
import original_module
#property
def get_Field(self):
return self.Field
original_module.OriginalClass.get_Field = get_Field
You'd need to enumerate what fields you wanted to access this way:
def addField(fieldname, class):
#property
def get_Field(self):
return getattr(self, fieldname)
setattr(original_module.OriginalClass, 'get_{}'.format(fieldname), get_Field)
for fieldname in ('Foo', 'Bar', 'Baz'):
addField(fieldname, original_module.OriginalClass)

Subclassing method decorators in python

I am having trouble thinking of a way that's good python and consistent with oop principles as I've been taught to figure out how to create a family of related method decorators in python.
The mutually inconsistent goals seem to be that I want to be able to access both decorator attributes AND attributes of the instance on which the decorated method is bound. Here's what I mean:
from functools import wraps
class AbstractDecorator(object):
"""
This seems like the more natural way, but won't work
because the instance to which the wrapped function
is attached will never be in scope.
"""
def __new__(cls,f,*args,**kwargs):
return wraps(f)(object.__new__(cls,*args,**kwargs))
def __init__(decorator_self, f):
decorator_self.f = f
decorator_self.punctuation = "..."
def __call__(decorator_self, *args, **kwargs):
decorator_self.very_important_prep()
return decorator_self.f(decorator_self, *args, **kwargs)
class SillyDecorator(AbstractDecorator):
def very_important_prep(decorator_self):
print "My apartment was infested with koalas%s"%(decorator_self.punctuation)
class UsefulObject(object):
def __init__(useful_object_self, noun):
useful_object_self.noun = noun
#SillyDecorator
def red(useful_object_self):
print "red %s"%(useful_object_self.noun)
if __name__ == "__main__":
u = UsefulObject("balloons")
u.red()
which of course produces
My apartment was infested with koalas...
AttributeError: 'SillyDecorator' object has no attribute 'noun'
Note that of course there is always a way to get this to work. A factory with enough arguments, for example, will let me attach methods to some created instance of SillyDecorator, but I was kind of wondering whether there is a reasonable way to do this with inheritance.

#miku got the key idea of using the descriptor protocol. Here is a refinement that keeps the decorator object separate from the "useful object" -- it doesn't store the decorator info on the underlying object.
class AbstractDecorator(object):
"""
This seems like the more natural way, but won't work
because the instance to which the wrapped function
is attached will never be in scope.
"""
def __new__(cls,f,*args,**kwargs):
return wraps(f)(object.__new__(cls,*args,**kwargs))
def __init__(decorator_self, f):
decorator_self.f = f
decorator_self.punctuation = "..."
def __call__(decorator_self, obj_self, *args, **kwargs):
decorator_self.very_important_prep()
return decorator_self.f(obj_self, *args, **kwargs)
def __get__(decorator_self, obj_self, objtype):
return functools.partial(decorator_self.__call__, obj_self)
class SillyDecorator(AbstractDecorator):
def very_important_prep(decorator_self):
print "My apartment was infested with koalas%s"%(decorator_self.punctuation)
class UsefulObject(object):
def __init__(useful_object_self, noun):
useful_object_self.noun = noun
#SillyDecorator
def red(useful_object_self):
print "red %s"%(useful_object_self.noun)
>>> u = UsefulObject("balloons")
... u.red()
My apartment was infested with koalas...
red balloons
The descriptor protocol is the key here, since it is the thing that gives you access to both the decorated method and the object on which it is bound. Inside __get__, you can extract the useful object identity (obj_self) and pass it on to the __call__ method.
Note that it's important to use functools.partial (or some such mechanism) rather than simply storing obj_self as an attribute of decorator_self. Since the decorated method is on the class, only one instance of SillyDecorator exists. You can't use this SillyDecorator instance to store useful-object-instance-specific information --- that would lead to strange errors if you created multiple UsefulObjects and accessed their decorated methods without immediately calling them.
It's worth pointing out, though, that there may be an easier way. In your example, you're only storing a small amount of information in the decorator, and you don't need to change it later. If that's the case, it might be simpler to just use a decorator-maker function: a function that takes an argument (or arguments) and returns a decorator, whose behavior can then depend on those arguments. Here's an example:
def decoMaker(msg):
def deco(func):
#wraps(func)
def wrapper(*args, **kwargs):
print msg
return func(*args, **kwargs)
return wrapper
return deco
class UsefulObject(object):
def __init__(useful_object_self, noun):
useful_object_self.noun = noun
#decoMaker('koalas...')
def red(useful_object_self):
print "red %s"%(useful_object_self.noun)
>>> u = UsefulObject("balloons")
... u.red()
koalas...
red balloons
You can use the decoMaker ahead of time to make a decorator to reuse later, if you don't want to retype the message every time you make the decorator:
sillyDecorator = decoMaker("Some really long message about koalas that you don't want to type over and over")
class UsefulObject(object):
def __init__(useful_object_self, noun):
useful_object_self.noun = noun
#sillyDecorator
def red(useful_object_self):
print "red %s"%(useful_object_self.noun)
>>> u = UsefulObject("balloons")
... u.red()
Some really long message about koalas that you don't want to type over and over
red balloons
You can see that this is much less verbose than writing a whole class inheritance tree for different kinds of decoratorts. Unless you're writing super-complicated decorators that store all sorts of internal state (which is likely to get confusing anyway), this decorator-maker approach might be an easier way to go.

Adapted from http://metapython.blogspot.de/2010/11/python-instance-methods-how-are-they.html. Note that this variant sets attributes on the target instance, hence, without checks, it is possible to overwrite target instance attributes. The code below does not contain any checks for this case.
Also note that this example sets the punctuation attribute explicitly; a more general class could auto-discover it's attributes.
from types import MethodType
class AbstractDecorator(object):
"""Designed to work as function or method decorator """
def __init__(self, function):
self.func = function
self.punctuation = '...'
def __call__(self, *args, **kw):
self.setup()
return self.func(*args, **kw)
def __get__(self, instance, owner):
# TODO: protect against 'overwrites'
setattr(instance, 'punctuation', self.punctuation)
return MethodType(self, instance, owner)
class SillyDecorator(AbstractDecorator):
def setup(self):
print('[setup] silly init %s' % self.punctuation)
class UsefulObject(object):
def __init__(self, noun='cat'):
self.noun = noun
#SillyDecorator
def d(self):
print('Hello %s %s' % (self.noun, self.punctuation))
obj = UsefulObject()
obj.d()
# [setup] silly init ...
# Hello cat ...