I am using retry function from the package retrying. I want to pass the arguments of the retry decorator from a function and I am not sure how to achieve that.
#retry # (wait_exponential_multiplier=x,wait_exponential_max=y)
def post(url, json, exponential_multiplier, exponential_max):
...
return(abc)
I want to pass the arguments of retry when calling the post(). I know when the function is compiled, the resulting function object is passed to the decorator so I am not sure if this is possible- or if I should may be approach it differently.
If you just want to use the library as is, then you cannot really use the decorator like this. It's arguments are constant from when it is invoked (excepting messing about with mutable arguments). Instead, you could always invoke the decorator before calling the function each time. This allows you to change the retrying arguments as and when you need to.
eg.
def post(url, json):
...
rety(post, wait_exponential_multiplier=...)(url=..., json=...)
But at that point, you might as well just skip the decorator altogether, and use what the decorator is using.
from retrying import Retrying
def post(url, json):
...
Retrying(wait_exponential_multiplier=...).call(post, url=..., json=...)
Either of these ways allow you to keep the post function pure and abstracted away from the concept of retrying (making it easier to call post when you don't want retrying behaviour).
You could also write a convenience function that wrapper that fills in defaults for your program. eg.
def retrier(wait_exponential_multiplier=2, **kwargs):
return Retrying(wait_exponential_multiplier=wait_exponential_multiplier, **kwargs)
retrier(wait_exponential_max=10).call(post, url=..., json=...)
retrier(wait_exponential_multiplier=3, wait_exponential_max=10).call(post, url=..., json=...)
Generally speaking there no good ways to achieve this. You surely can write code like this:
def post(url, json):
...
return(abc)
...
decorated_func = retry(wait_exponential_max=1)(post)
a = decorated_func(url, json)
and it will work. But it looks rather ugly and will construct decorated object for every call ("regular" decorators are executed once in import time).
If decorator itself is not very complex - you can use this approach in some more user-friendly manner:
def _post(url, json):
return(abc)
def post(url, json, wait_exponential_max=None, **kwargs):
return retry(wait_exponential_max=wait_exponential_max, **kwargs)(_post)(url, json)
You have to create a new decorator which pass its own arguments down to the decorated function and transforms the function using the retry decorator:
def retry_that_pass_down_arguments(**decorator_arguments):
def internal_decorator(f):
def decorated_function(*args, **kwargs):
# Add the decorator key-word arguments to key-word arguments of the decorated function
kwargs.update(decorator_arguments)
return retry(**decorator_arguments)(f)(*args, **kwargs)
return decorated_function
return internal_decorator
Then you can just do:
#retry_that_pass_down_arguments(wait_exponential_multiplier=x, wait_exponential_max=y)
def post(url, json, exponential_multiplier=None, exponential_max=None):
...
return(abc)
This is a complement to Jundiaius's answers to show that you can even use the inspect module to correctly handle the signature of the decorated function:
def deco_and_pass(deco, **kwparams):
"""Decorates a function with a decorator and parameter.
The parameters are passed to the decorator and forwarded to the function
The function must be prepared to receive those parameters, but they will
be removed from the signature of the decorated function."""
def outer(f):
sig = inspect.signature(f) # remove parameters from the function signature
params = collections.OrderedDict(sig.parameters)
for k in kwparams:
del params[k]
def inner(*args, **kwargs): # define the decorated function
kwargs.update(kwparams) # add the parameters
# and call the function through the parameterized decorator
return deco(**kwparams)(f)(*args, **kwargs)
inner.__signature__ = inspect.signature(f).replace(
parameters = params.values())
inner.__doc__ = f.__doc__ # update doc and signature
return inner
return outer
Example usage:
#deco_and_pass(retry,wait_exponential_multiplier=x,wait_exponential_max=y)
def post(url, json, exponential_multiplier, exponential_max):
...
return(abc)
...
post(url, json)
The signature of the decorated function is only def post(url, json)
Limits: the above code only accepts and passes keyword arguments for the decorator
Related
I have a function that has a decorator #retry, which retries the function if a certain Exception happened. I want to test that this function executes the correct amount of times, for which I have the following code which is working:
#pytest.mark.asyncio
async def test_redis_failling(mocker):
sleep_mock = mocker.patch.object(retry, '_sleep')
with pytest.raises(ConnectionError):
retry_store_redis()
assert sleep_mock.call_count == 4
#retry(ConnectionError, initial_wait=2.0, attempts=5)
def retry_store_redis():
raise ConnectionError()
But, if I modify retry_store_redis() to be an async function, the return value of sleep_mock.call_count is 0.
So you define "retry" as a function. Then you define a test, then you define some code that uses #retry.
#retry, as a decorator, is being called at import time. So the order of operations is
declare retry
declare test
call retry with retry_store_redis as an argument
start your test
patch out retry
call the function you defined in step 3
so "retry" gets called once (at import time), your mock gets called zero times. To get the behavior you want, (ensuring that retry is actually re-calling the underlying function) I would do
#pytest.mark.asyncio
async def test_redis_failling(mocker):
fake_function = MagicMock(side_effect=ConnectionError)
decorated_function = retry(ConnectionError, initial_wait=2.0, attempts=5)(fake_function)
with pytest.raises(ConnectionError):
decorated_function()
assert fake_function.call_count == 4
if you wanted to test this as built (instead of a test specifically for the decorator) you would have to mock out the original function inside the decorated function- which would depend on how you implemented the decorator. The default way (without any libraries) means you would have to inspect the "closure" attribute. You can build the object to retain a reference to the original function though, here is an example
def wrap(func):
class Wrapper:
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
return Wrapper(func)
#wrap
def wrapped_func():
return 42
in this scenario, you could patch the wrapped function at wrapped_func.func
I want use a same param in decorater and function like:
def outer(deco_param):
def decorator(func):
print("param:%s" % deco_param)
return func
return decorator
#outer(deco_param=s) # problems here: 's' is unreferenced
def test(s):
print("test:%s" % s)
I got this idea when using Flask but I didn't know how did they make it, which supports using a param of view function in its decorater like:
app.route(rule="/user/<int:uid>")
def access_user(uid): # if I use any other name except from 'uid', IDE would raise a tip
...
What's more, Flask seems to make a static check. If I miss uid argument or use any other name, IDE(pycharm) would raise a tip of "Function 'access_user` doesn't have a parameter 'int:uid'". That's the effect I want.
Can anyone please give me some advice? Thanks
You don't need to pass the same parameter to both the outer function and the inner function. To call the parameters via decorator, you should add a wrapper function to pass the parameters (*args) from inner one to outer one.
You can write like this:
def outer():
def decorator(func):
def wrapper(*args):
print("param:%s" % func.__param__)
return func(*args)
return wrapper
return decorator
#outer
def test(s):
print("test:%s" % s)
I want to make a Python decorator that can take an argument (the name of an argument in the function it is decorating) and use that argument to find the value of the argument in the function. I know that sounds super confusing, ha! But what I want to do would look like this (it's for a Flask web application):
This is what I have right now:
#app.route('/admin/courses/<int:course_id>')
def view_course(course_id):
... view code here
And I want to do something like this:
#app.route('/admin/courses/<int:course_id>')
#access_course_permission(course_id)
def view_course(course_id):
... view code here
Right now I know I can do something like:
def access_course_permission(f):
#wraps(f)
def decorated_function(*args, **kwargs):
# check args[0] right here to find the course ID
return f(*args, **kwargs)
return decorated_function
which works great as long as course_id is always the first parameter. However, it isn't. That's why I want to be able to specify the name.
If this is not possible, I guess I could just pass the index of the parameter to the decorator, but that isn't very nice...
You can use the inspect.getfullargspec() function to access the names used in a function:
try:
# Python 3
from inspect import getfullargspec
except ImportError:
# Python 2, use inspect.getargspec instead
# this is the same function really, without support for annotations
# and keyword-only arguments
from inspect import getargspec as getfullargspec
from functools import wraps
def access_course_permission(argument_name):
def decorator(f):
argspec = getfullargspec(f)
argument_index = argspec.args.index(argument_name)
#wraps(f)
def wrapper(*args, **kwargs):
try:
value = args[argument_index]
except IndexError:
value = kwargs[argument_name]
# do something with value
return f(*args, **kwargs)
return wrapper
return decorator
The above finds out at what index your specific argument is positioned; this covers both positional and keyword arguments (because in Python, you can pass in a value for a keyword argument by position too).
Note however that for your specific example, Flask will call view_course with course_id as a keyword argument, so using kwargs[argument_name] would suffice.
You'll have to pass in a string to name that argument:
#app.route('/admin/courses/<int:course_id>')
#access_course_permission('course_id')
def view_course(course_id):
# ... view code here
Note however, that in Flask you could just access request.view_args, without the need to parse this information out of the function arguments:
course_id = requests.view_args[argument_name]
I am using nose test generators feature to run the same test with different contexts. Since it requires the following boiler plate for each test:
class TestSample(TestBase):
def test_sample(self):
for context in contexts:
yield self.check_sample, context
def check_sample(self, context):
"""The real test logic is implemented here"""
pass
I decided to write the following decorator:
def with_contexts(contexts=None):
if contexts is None:
contexts = ['twitter', 'linkedin', 'facebook']
def decorator(f):
#wraps(f)
def wrapper(self, *args, **kwargs):
for context in contexts:
yield f, self, context # The line which causes the error
return wrapper
return decorator
The decorator is used in the following manner:
class TestSample(TestBase):
#with_contexts()
def test_sample(self, context):
"""The real test logic is implemented here"""
var1 = self.some_valid_attribute
When the tests executed an error is thrown specifying that the attribute which is being accessed is not available. However If I change the line which calls the method to the following it works fine:
yield getattr(self, f.__name__), service
I understand that the above snippet creates a bound method where as in the first one self is passed manually to the function. However as far as my understanding goes the first snippet should work fine too. I would appreciate if anyone could clarify the issue.
The title of the question is related to calling instance methods in decorators in general but I have kept the description specific to my context.
You can use functools.partial to tie the wrapped function to self, just like a method would be:
from functools import partial
def decorator(f):
#wraps(f)
def wrapper(self, *args, **kwargs):
for context in contexts:
yield partial(f, self), context
return wrapper
Now you are yielding partials instead, which, when called as yieldedvalue(context), will call f(self, context).
As far as I can tell, some things don't fit together. First, your decorator goes like
def with_contexts(contexts=None):
if contexts is None:
contexts = ['twitter', 'linkedin', 'facebook']
def decorator(f):
#wraps(f)
def wrapper(self, *args, **kwargs):
for context in contexts:
yield f, self, context # The line which causes the error
return wrapper
return decorator
but you use it like
#with_contexts
def test_sample(self, context):
"""The real test logic is implemented here"""
var1 = self.some_valid_attribute
This is wrong: this calls with_context(test_sample), but you need with_context()(test_sample). So do
#with_contexts()
def test_sample(self, context):
"""The real test logic is implemented here"""
var1 = self.some_valid_attribute
even if you don't provide the contexts argument.
Second, you decorate the wrong function: your usage shows that the test function yields the check function for each context. The function you want to wrap does the job of the check function, but you have to name it after the test function.
Applying self to a method can be done with partial as Martijn writes, but it can as well be done the way Python does it under the hood: with
method.__get__(self, None)
or maybe better
method.__get__(self, type(self))
you can achieve the same. (Maybe your original version works as well, with yielding the function to be called and the arguments to use. It was not clear to me that this is the way it works.)
This is what i want to do:
#MyDecorator
def f():
pass
for d in f.decorators:
print d
This is not generally possible without the cooperation of the decorators. For example,
def my_decorator(f):
def wrapper(*args, **kwargs):
return f(*args, **kwargs)
wrapper.decorators = [wrapper]
if hasattr(f, 'decorators'):
wrapper.decorators.extend[f.decorators]
return wrapper
Essentially, all the decorator does is wrap the function as usual and then put a decorators attribute on the wrapper function. It then checks the wrapped function for a similar list and propagates it upwards.
This is pretty useless though
What I think you want is
def my_decorator(f):
def wrapper(args):
return f(args)
wrapper.decorated = f
return wrapper
This will allow you to do stuff like
#my_other_decorator # another decorator following the same pattern
#my_decorator
def foo(args):
print args
foo.decorated(args) # calls the function with the inner decorated function (my_decorator)
foo.decorated.decorated(args) # original function
You can actually abstract this pattern into a decorator
def reversable(decorator):
def wrapper(func):
ret = decorator(func) # manually apply the decorator
ret.decorated = func # save the original function
return ret
return wrapper
Now when you are writing your decorator:
#reversable
def my_decorator(f):
def wrapper(x):
return f(x + 1)
return wrapper
The #MyDecorator syntax is just shorthand for writing the following Python code:
def f():
pass
f = MyDecorator(f)
Written in this form, you can see that the decorators applied to the function are not kept track of in any way. You could make your decorators remember when they're applied (Aaron's answer has a couple good ideas on how to do this), but you'd have to wrap all third-party decorators with your own.