This question already has answers here:
How can I decorate an instance method with a decorator class?
(2 answers)
Closed 4 years ago.
I'm trying to memoize using a decorator with the decorator being a class not a function, but I'm getting the error
TypeError: seqLength() takes exactly 2 arguments (1 given)
I'm guessing this has something to do with the classes, but not sure what's wrong from there.
The code:
import sys
class memoize(object):
'''memoize decorator'''
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args):
try:
return self.cache[args]
except KeyError:
value = self.func(self, *args)
self.cache[args] = value
return value
class collatz(object):
def __init__(self, n):
self.max = 1
self.n = n
#memoize
def seqLength(self, n):
if n>1:
if n%2 == 0:
return 1+self.seqLength(n/2)
else:
return 1+self.seqLength(3*n+1)
else:
return 1
def maxLength(self):
for n in xrange(1, self.n):
l = self.seqLength(n)
if l > self.max:
self.max = n
return self.max
n = int(sys.argv[1])
c = collatz(n)
print c.maxLength()
This is confusing, syntactically. It's not clear if self.func is part of your memoize or a separate function that's part of some other object of some other class. (You mean the latter, BTW)
value = self.func(self, *args)
Do this to make it clear that the_func is just a function, not a member of the memoize class.
the_func= self.func
value= the_func( *args )
That kind of thing prevents confusion over the class to which self. is bound.
Also, please spell it Memoize. With a leading capital letter. It is a class definition, after all.
Using a class as a decorator is tricky, because you have to implement the descriptor protocol correctly (the currently accepted answer doesn't.) A much, much easier solution is to use a wrapper function, because they automatically implement the descriptor protocol correctly. The wrapper equivalent of your class would be:
import functools
def memoize(func):
cache = {}
#functools.wraps(func)
def wrapper(*args):
try:
return cache[args]
except KeyError:
value = func(*args)
cache[args] = value
return value
return wrapper
When have so much state you want to encapsulate it in a class anyway, you can still use a wrapper function, for example like so:
import functools
class _Memoize(object):
'''memoize decorator helper class'''
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args):
try:
return self.cache[args]
except KeyError:
value = self.func(*args)
self.cache[args] = value
return value
def memoize(func):
o = _Memoize(func)
#functools.wraps(func)
def wrapper(*args):
return o(*args)
return wrapper
A decorator is just syntactic sugar for foo = decorator(foo), so in this case you're ending up making the self of seqLength be memoize instead of collatz. You need to use descriptors. This code works for me:
class memoize(object):
'''memoize descriptor'''
def __init__(self, func):
self.func = func
def __get__(self, obj, type=None):
return self.memoize_inst(obj, self.func)
class memoize_inst(object):
def __init__(self, inst, fget):
self.inst = inst
self.fget = fget
self.cache = {}
def __call__(self, *args):
# if cache hit, done
if args in self.cache:
return self.cache[args]
# otherwise populate cache and return
self.cache[args] = self.fget(self.inst, *args)
return self.cache[args]
More on descriptors:
http://docs.python.org/howto/descriptor.html#descriptor-example
Related
I am trying to use a simple validation decorator inside a class:
import numpy as np
def validate_decorator(func):
def func_wrapper(value):
if value < 0:
raise Exception("Not valid!")
return func(value)
return func_wrapper
class MyClass:
def __init__(self):
self.my_array = np.zeros(shape=(10,))
self.idx = 0
#validate_decorator
def insert_value(self, value):
self.my_array[self.idx] = value
self.idx += 1
def __str__(self):
return f"{self.my_array[:self.idx]}"
a = MyClass()
a.insert_value(3.14)
This gives
TypeError: func_wrapper() takes 1 positional argument but 2 were given
I do not understand this error.
If I modify the func_wrapper to also take a self argument and return self (instead of func(value)) then there is no TypeError but, as expected, the insert_value is not executed.
Can someone help me understand how to use this simple decorator inside my class?
I've added self argument to func_wrapper and changed func_wrapper's return value to func(self, value). It works now, insert_value is executed.
Class instance method always has this required self attribute unless converted to class method or static method: What is the purpose of the word 'self', in Python?
import numpy as np
def validate_decorator(func):
def func_wrapper(self, value):
if value < 0:
raise Exception("Not valid!")
return func(self, value)
return func_wrapper
class MyClass:
def __init__(self):
self.my_array = np.zeros(shape=(10,))
self.idx = 0
#validate_decorator
def insert_value(self, value):
self.my_array[self.idx] = value
self.idx += 1
print('EXECUTED!')
def __str__(self):
return f"{self.my_array[:self.idx]}"
a = MyClass()
a.insert_value(3.14)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I would like to list all methods from a given class that return self.
I am aware that in Python it is impossible to check what is the return type of a function before it was run.
Perhaps it would be possible to check for return self in the function body (source) or maybe there is some other way?
I need it to work with Python 3.5 and above
EDIT:
This class (not a full code) is a part of gremlinpython library.
class GraphTraversal(Traversal):
def __init__(self, graph, traversal_strategies, bytecode):
super(GraphTraversal, self).__init__(graph, traversal_strategies, bytecode)
def __getitem__(self, index):
if isinstance(index, int):
return self.range(long(index), long(index + 1))
elif isinstance(index, slice):
low = long(0) if index.start is None else long(index.start)
high = long(sys.maxsize) if index.stop is None else long(index.stop)
if low == long(0):
return self.limit(high)
else:
return self.range(low,high)
else:
raise TypeError("Index must be int or slice")
def __getattr__(self, key):
return self.values(key)
def V(self, *args):
self.bytecode.add_step("V", *args)
return self
def addE(self, *args):
self.bytecode.add_step("addE", *args)
return self
def addV(self, *args):
self.bytecode.add_step("addV", *args)
return self
It allows to query the graph data base using fluent API like this
g.V().hasLabel('label').has('id','xxx').addE('relation').to(g.V().hasLabel('otherlabel').has('id','yyy')
So far I was able to get the methods like this:
from inspect import getsourcelines, signature
def contains_return_self(f):
lines, _ = getsourcelines(f)
return any("return self" in line for line in lines)
def check_signature(f):
sig = signature(f)
if(len(sig.parameters) == 2
and 'self' in sig.parameters.keys()
and 'args' in sig.parameters.keys()):
return True
return False
fluent_methods = [
method_name for method_name in dir(GraphTraversal)
if callable(getattr(GraphTraversal, method_name))
and ('__' not in method_name)
and contains_return_self(getattr(GraphTraversal, method_name))
and check_signature(getattr(GraphTraversal, method_name))]
I would like to return all the methods that have the following signature:
def foo(self, *args)
# some code
return self
Although, like #RafaelC, I strongly suspect this is likely an XY Problem, here's something — based partially on the inspect module — that seems to work (within the inherent limitations of such an approach). For testing I added a definition of the Traversal base class as well as some non-matching methods to both it and the derived GraphTraversal class.
from collections import namedtuple
import inspect
import re
class Traversal:
def inherited_method1(self, *args):
return self
def inherited_method2(self, foobar):
return foobar + 13
class GraphTraversal(Traversal):
def __init__(self, graph, traversal_strategies, bytecode):
super(GraphTraversal, self).__init__(graph, traversal_strategies, bytecode)
def __getitem__(self, index):
if isinstance(index, int):
return self.range(long(index), long(index + 1))
elif isinstance(index, slice):
low = long(0) if index.start is None else long(index.start)
high = long(sys.maxsize) if index.stop is None else long(index.stop)
if low == long(0):
return self.limit(high)
else:
return self.range(low,high)
else:
raise TypeError("Index must be int or slice")
def __getattr__(self, key):
return self.values(key)
def non_match1(self, *args):
self.bytecode.add_step("V", *args)
return 42
def non_match2(self, arg1, arg2):
self.bytecode.add_step("V", *args)
return self
def V(self, *args):
self.bytecode.add_step("V", *args)
return self
def addE(self, *args):
self.bytecode.add_step("addE", *args)
return self
def addV(self, *args):
self.bytecode.add_step("addV", *args)
return self
### Introspect class
DUNDER = re.compile(r"^_{2,}\w*_{2,}\Z", re.UNICODE)
MethInfo = namedtuple('MethInfo', ['name', 'value'])
methods = [MethInfo(*pair) for pair in inspect.getmembers(GraphTraversal, inspect.isfunction)
if not DUNDER.match(pair[0])]
def contains_return_self(meth_info):
src = inspect.getsource(meth_info.value)
for line in src.splitlines():
if 'return self' in line.strip():
return True
else:
return False
def check_signature(meth_info):
sig = inspect.signature(meth_info.value)
return str(sig) == '(self, *args)'
fluent_methods = [meth_info.name for meth_info in methods
if contains_return_self(meth_info) and check_signature(meth_info)]
print('fluent_methods:', fluent_methods)
Output:
fluent_methods: ['V', 'addE', 'addV', 'inherited_method1']
I'm trying to use a subclass for the int class to attach an additional label ("headline"). All works if I access the individual object, but if I collect several in a list, they all have the same property, while I would expect them to have the one I specified when creating the object.
I also tried with methods instead of properties to no further results.
I'm using Python 3.4.3.
import unittest
class LabeledInt(int):
def __new__(cls, *args, **kwargs):
cls._headline = args[1]
return super(LabeledInt, cls).__new__(cls, args[0])
#property
def headline(self):
return self._headline
class SomeNumbers:
def __init__(self, arg):
self.arg = arg
#property
def something(self):
return LabeledInt(self.arg, "Something")
#property
def something_squared(self):
return LabeledInt(self.arg ** 2, "Squared")
#property
def something_exponential(self):
return LabeledInt(self.arg ** self.arg, "Exp.")
def all_numbers(self):
array = [
LabeledInt(self.arg, "Something"),
LabeledInt(self.arg ** 2, "Squared"),
LabeledInt(self.arg ** self.arg, "Exp.")
]
return array
S = SomeNumbers(2)
class Test(unittest.TestCase):
def test_something(self):
self.assertEqual(2, S.something)
self.assertEqual("Something", S.something.headline)
def test_something_squard(self):
self.assertEqual(4, S.something_squared)
self.assertEqual("Squared", S.something_squared.headline)
def test_exp(self):
self.assertEqual(4, S.something_exponential)
self.assertEqual("Exp.", S.something_exponential.headline)
def test_all_numbers_1(self):
self.assertEqual(2, S.all_numbers()[0])
def test_all_numbers_2(self):
self.assertEqual("Something", S.all_numbers()[0].headline)
def test_all_numbers_3(self):
self.assertEqual(4, S.all_numbers()[1])
def test_all_numbers_4(self):
self.assertEqual("Squared", S.all_numbers()[1].headline)
def test_all_numbers_5(self):
self.assertEqual(4, S.all_numbers()[2])
def test_all_numbers_6(self):
self.assertEqual("Exp.", S.all_numbers()[2].headline)
for n in S.all_numbers():
print(n.headline)
>>>
Exp.
Exp.
Exp.
Tests "test_all_numbers_2" und "...4" fail.
Why does this happen? And what's the best way around it? Thanks a lot.
class LabeledInt(int):
def __new__(cls, *args, **kwargs):
cls._headline = args[1]
# ^^^
return super(LabeledInt, cls).__new__(cls, args[0])
You are setting the attribute of the class, not of the instance. Try this:
class LabeledInt(int):
def __new__(cls, *args, **kwargs):
self = super(LabeledInt, cls).__new__(cls, args[0])
self._headline = args[1]
# ^^^^
return self
PS: don't use *args and **kwargs if you are neither using them, nor passing them around. Also, Python 3's super() doesn't need arguments anymore. Consider using this code:
class LabeledInt(int):
def __new__(cls, value, headline):
self = super().__new__(cls, value)
self._headline = headline
return self
I've got a decorator that I've implemented as a class:
class Cached(object):
def __init__(self, func):
self.cache = None
self.func = func
def __call__(self, *args, **kwargs):
if self.cache is None or (time.time() - self.cache[0] >= 1000):
res = self.f(*args, **kwargs)
self.cache = (time.time(), res)
else:
res = self.cache[1]
return res
I want to use this decorator to decorate a method of a class, e.g.:
class Foo(object):
def __init__(self, x):
self.x = x
#cached
def bar(self, y):
return self.x + y
As it stands,
f = Foo(10)
f.bar(11)
throws TypeError: foo() takes exactly 2 arguments (1 given). f.bar(f, 11) works, but is the code smell equivalent of summer in New York City during a sanitation worker strike. What am I missing?
ETA: Originally, I was trying to implement Cached as a function:
def cached(cache):
def w1(func):
def w2(*args, **kwargs):
# same
return w2
return w1
but I kept getting weird scoping errors about cache being used before it's defined, which switching to a decorator class fixed.
You need to add this to your decorator class:
def __get__(self, obj, objtype):
"""support instance methods"""
from functools import partial
return partial(self.__call__, obj)
What I want is a memoization decorator that:
can memoize instance methods with both arguments and keyword arguments
has a cache that can be cleared (globally) with one call (vs. this one that uses a per-function cache: python resettable instance method memoization decorator)
is reasonably efficient
I've tweaked an example I saw and came up with the following:
import functools
class Memoized(object):
"""Decorator that caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned, and
not re-evaluated.
"""
__cache = {}
def __init__(self, func):
self.func = func
self.key = (func.__module__, func.__name__)
def __call__(self, *args):
try:
return Memoized.__cache[self.key][args]
except KeyError:
value = self.func(*args)
Memoized.__cache[self.key] = {args : value}
return value
except TypeError:
# uncachable -- for instance, passing a list as an argument.
# Better to not cache than to blow up entirely.
return self.func(*args)
def __get__(self, obj, objtype):
"""Support instance methods."""
return functools.partial(self.__call__, obj)
#staticmethod
def reset():
Memoized.__cache = {}
My problem with it is that the caching part seems to involve a lot of overhead (eg. for recursive functions). Using the following function as an example, I can call fib(30) ten times with the non-memoized version in less time than the memoized version.
def fib(n):
if n in (0, 1):
return n
return fib(n-1) + fib(n-2)
Can anyone suggest a better way to write this decorator? (or point me to a better (ie. faster) decorator that does what I want).
I'm not interested in preserving method signatures, or helping introspection tools "know" anything about the decorated function.
Thanks.
P.S. Using python 2.7
You're not actually caching any data, because each time you set a new cached value you overwrite the previous:
Memoized.__cache[self.key] = {args : value}
eg.
import functools
class Memoized(object):
"""Decorator that caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned, and
not re-evaluated.
"""
cache = {}
def __init__(self, func):
self.func = func
self.key = (func.__module__, func.__name__)
self.cache[self.key] = {}
def __call__(self, *args):
try:
return Memoized.cache[self.key][args]
except KeyError:
value = self.func(*args)
Memoized.cache[self.key][args] = value
return value
except TypeError:
# uncachable -- for instance, passing a list as an argument.
# Better to not cache than to blow up entirely.
return self.func(*args)
def __get__(self, obj, objtype):
"""Support instance methods."""
return functools.partial(self.__call__, obj)
#staticmethod
def reset():
Memoized.cache = {}
fib(30) without caching: 2.86742401123
fib(30) with caching: 0.000198125839233
Some other notes:
Don't use __prefix; there's no reason to here and it only uglifies the code.
Instead of using a single, monolithic, class-attribute dict, give each instance of Memoized its own dict, and keep a registry of Memoized objects. This improves encapsulation, and removes the oddity of depending on the module and function names.
.
import functools
import weakref
class Memoized(object):
"""Decorator that caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned, and
not re-evaluated.
>>> counter = 0
>>> #Memoized
... def count():
... global counter
... counter += 1
... return counter
>>> counter = 0
>>> Memoized.reset()
>>> count()
1
>>> count()
1
>>> Memoized.reset()
>>> count()
2
>>> class test(object):
... #Memoized
... def func(self):
... global counter
... counter += 1
... return counter
>>> testobject = test()
>>> counter = 0
>>> testobject.func()
1
>>> testobject.func()
1
>>> Memoized.reset()
>>> testobject.func()
2
"""
caches = weakref.WeakSet()
def __init__(self, func):
self.func = func
self.cache = {}
Memoized.caches.add(self)
def __call__(self, *args):
try:
return self.cache[args]
except KeyError:
value = self.func(*args)
self.cache[args] = value
return value
except TypeError:
# uncachable -- for instance, passing a list as an argument.
# Better to not cache than to blow up entirely.
return self.func(*args)
def __get__(self, obj, objtype):
"""Support instance methods."""
return functools.partial(self.__call__, obj)
#staticmethod
def reset():
for memo in Memoized.caches:
memo.cache = {}
if __name__ == '__main__':
import doctest
doctest.testmod()
Edited: add tests, and use weakref.WeakSet. Note that WeakSet is only available in 2.7 (which the OP is using); for a version that works in 2.6, see the edit history.
Here is a version that is significantly faster. Unfortunately reset can no longer actually completely clear the cache since all the instances are storing their own local copy of the reference to the per-function dictionary. Though you can sort of get it to work:
import functools
class Memoized(object):
"""Decorator that caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned, and
not re-evaluated.
"""
__cache = {}
def __init__(self, func):
self.func = func
key = (func.__module__, func.__name__)
if key not in self.__cache:
self.__cache[key] = {}
self.mycache = self.__cache[key]
def __call__(self, *args):
try:
return self.mycache[args]
except KeyError:
value = self.func(*args)
self.mycache[args] = value
return value
except TypeError:
# uncachable -- for instance, passing a list as an argument.
# Better to not cache than to blow up entirely.
return self.func(*args)
def __get__(self, obj, objtype):
"""Support instance methods."""
return functools.partial(self.__call__, obj)
#classmethod
def reset(cls):
for v in cls.__cache.itervalues():
v.clear()