I need to know when a function is called and do something after calling the function. It seems Interceptor can do it.
How can I use Interceptor in python ?
This can be done using decorators:
from functools import wraps
def iterceptor(func):
print('this is executed at function definition time (def my_func)')
#wraps(func)
def wrapper(*args, **kwargs):
print('this is executed before function call')
result = func(*args, **kwargs)
print('this is executed after function call')
return result
return wrapper
#iterceptor
def my_func(n):
print('this is my_func')
print('n =', n)
my_func(4)
Output:
this is executed at function definition time (def my_func)
this is executed before function call
this is my_func
n = 4
this is executed after function call
#iterceptor replaces my_func with the result of execution of the iterceptor function, that is with wrapper function. wrapper wraps the given function in some code, usually preserving the arguments and execution result of wrappee, but adds some additional behavior.
#wraps(func) is there to copy the signature/docstring data of the function func onto the newly created wrapper function.
More info:
http://python-3-patterns-idioms-test.readthedocs.io/en/latest/PythonDecorators.html
https://www.python.org/dev/peps/pep-0318/
Related
I have some questions regarding what happens inside decorators in Python.
Consider code, which saves sum in log file:
def logger(func):
def wrapped(*args, **kwargs):
result = func(*args, **kwargs)
with open('log.txt', 'w') as f:
f.write(str(result))
return result
return wrapped
#logger
def summator(numlist):
return sum(numlist)
print(f"Summator: {summator([1, 2, 3, 4, 5])}")
How does it work when I run summator([1, 2, 3, 4, 5, 6])? I suppose the logger to send the summator([1, 2, 3, 4, 5, 6]) function instance inside wrapped, where the summator function is executed and it's result is someway modified. But def logger(func) is strange for me: does it mean, that it takes func with func arguments attached? I see that logger itself doesn't take * args, ** kwargs, only wrapped does...
Consider similar code:
def logger(filename):
def decorator(func):
def wrapped(*args, **kwargs):
result = func(*args, **kwargs)
with open(filename, 'w') as f:
f.write(str(result))
return result
return wrapped
return decorator
#logger('new_log.txt')
def summator(numlist):
return sum(numlist)
How does decorator get func? It is wrapped by logger, which accepts only a filename.
This is my simple explanation of how decorators work.
You are going to decorate a function, here summator and basically add some functionalities to it without touching the body of it.
The expression:
#logger
def summator(numlist):
return sum(numlist)
is equivalent to:
def summator(numlist):
return sum(numlist)
summator = logger(summator)
# is just a syntactic sugar.
What happened?
Look at logger function. What does it do ? it takes a function and returns another function (here named wrapped). Not the result of that function, just the function itself.
Then you assign this returned function from calling logger(summator) to a symbol called summator. From now on, whenever you call summator you are calling the returned function from the logger. You are calling the wrapped.
Inside wrapped there is a func. What is it ? isn't it the original summator function which you passed to the logger in the previous paragraph? YES.
Q: How wrapped function can access it? it is not its local variable!
A: Because wrapped is a closure and have access to the enclosing scope's local variables. read more here
So wrapped calls func, stores the result, write it to the file then returns it.
note: we normally define our wrapped function with this signature : def wrapped(*args, **kwargs): because we want to be free to pass all parameters(both positional and keyword arguments) down to the func function. At the end of the day func and original summator are the same object. In your case you could write it like:
def logger(func):
def wrapped(lst): # <--------
result = func(lst) # <--------
with open('log.txt', 'w') as f:
f.write(str(result))
return result
return wrapped
You second question is referring to what is known as decorator factory. It's a function which returns the actual decorator. Q: When do we use it? A: When we want to have additional parameters from the decorator factory to the decorator or it's inner function. (Here you passed filename as a parameter to the decorator factory but used it inside the wrapped function.)
The story is the same, it is just another layer above our decorator. The equivalent of that syntactic sugar would be:
def summator(numlist):
return sum(numlist)
summator = logger('new_log.txt')(summator)
# In two steps, it would be:
# decorator = logger('new_log.txt')
# summator = decorator(summator)
note: In the first question, name logger was the actual decorator(Because it takes the summator function for decorating) But in second question it is not. It is now a decorator factory.
The way wrapped can access filename is the same way it can access func as I mentioned previously.
Read more about it here.
I have a really simple function, defined as
def test(x):
return x
I would like to wrap it with a decorator, that returns a function that expects another kwargs param.
#simple_dec
def test(x):
return x
Inside that decorator function, i would pop that param from the kwargs dict, and that just call test function with the params test would expect to get, without breaking it:
def simple_dec():
def simple_dec_logic(func, *args, **kwargs):
kwargs.pop("extra_param")
return func(*args, **kwargs)
return decorator.decorate(_func, simple_dec_logic)
My issue is - after wrapping it, if I call:
test(1, extra_param=2)
It fails on "test got unexpected param extra_param", although if the code would actually run, the decorator would handle this and call test func, without that param. If I get it correctly, the interpreter just fails it, before running the code and knowing it's defined with a decorator.
Is there any way to work around this? I would like the decorator to let me call the test function with more params, without defining them in the test function.
This works fine:
import functools
def decorator(func):
#functools.wraps(func)
def wrapper_decorator(*args, **kwargs):
kwargs.pop('extra_param')
value = func(*args, **kwargs)
return value
return wrapper_decorator
#decorator
def test(x):
return x
test(1, extra_param=2)
I have a function that has a decorator #retry, which retries the function if a certain Exception happened. I want to test that this function executes the correct amount of times, for which I have the following code which is working:
#pytest.mark.asyncio
async def test_redis_failling(mocker):
sleep_mock = mocker.patch.object(retry, '_sleep')
with pytest.raises(ConnectionError):
retry_store_redis()
assert sleep_mock.call_count == 4
#retry(ConnectionError, initial_wait=2.0, attempts=5)
def retry_store_redis():
raise ConnectionError()
But, if I modify retry_store_redis() to be an async function, the return value of sleep_mock.call_count is 0.
So you define "retry" as a function. Then you define a test, then you define some code that uses #retry.
#retry, as a decorator, is being called at import time. So the order of operations is
declare retry
declare test
call retry with retry_store_redis as an argument
start your test
patch out retry
call the function you defined in step 3
so "retry" gets called once (at import time), your mock gets called zero times. To get the behavior you want, (ensuring that retry is actually re-calling the underlying function) I would do
#pytest.mark.asyncio
async def test_redis_failling(mocker):
fake_function = MagicMock(side_effect=ConnectionError)
decorated_function = retry(ConnectionError, initial_wait=2.0, attempts=5)(fake_function)
with pytest.raises(ConnectionError):
decorated_function()
assert fake_function.call_count == 4
if you wanted to test this as built (instead of a test specifically for the decorator) you would have to mock out the original function inside the decorated function- which would depend on how you implemented the decorator. The default way (without any libraries) means you would have to inspect the "closure" attribute. You can build the object to retain a reference to the original function though, here is an example
def wrap(func):
class Wrapper:
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
return Wrapper(func)
#wrap
def wrapped_func():
return 42
in this scenario, you could patch the wrapped function at wrapped_func.func
I would like to write a decorator that limits the number of calls to the wrapped function. Say, if I want the wrapped function to be called maximum 10 times, the decorator should execute that function the first 10 times, then it should return None instead.
Here's what I've come up with:
from functools import wraps
def max_calls(num):
"""Decorator which allows its wrapped function to be called `num` times"""
def decorator(func):
#wraps(func)
def wrapper(*args, **kwargs):
calls = getattr(wrapper, 'calls', 0)
calls += 1
if calls == num:
return None
setattr(wrapper, 'calls', calls)
return func(*args, **kwargs)
setattr(wrapper, 'calls', 0)
return wrapper
return decorator
However, this count calls properly, returns None when the limit is reached, but...it doesn't reset between program runs. That is, if I execute the program once, the counter reaches 5, and then re-execute the program, it continues from 5. What do I need to change so that the decorator works properly?
The problem is that you maintain just one set of call counts. But that means each Flask request shares call counts with all the other requests. What you need to do is to maintain a separate set of call counts for each Flask request.
From reading the API documentation it looks as though the way to make this work is to carry out these three steps:
Make a subclass of flask.Request that can store your function call counts:
import collections
import flask
class MyRequest(flask.Request):
"""A subclass of Request that maintains function call counts."""
def __init__(self, *args, **kwargs):
super(MyRequest, self).__init__(*args, **kwargs)
self.call_counts = collections.defaultdict(int)
Set request_class to your subclass when you initialize your application:
app = flask.Flask(__name__, request_class=MyRequest, ...)
Rewrite your decorator to store its counts in the global flask.request object:
import functools
def max_calls(num, default=None):
"""Decorator which allows its wrapped function to be called at most `num`
times per Flask request, but which then returns `default`.
"""
def decorator(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
if flask.request.call_counts[func] == num:
return default
flask.request.call_counts[func] += 1
return func(*args, **kwargs)
return wrapper
return decorator
But having written that, it would be remiss for me not to point out that your question seems very strange. Why do you want to restrict the number of times a function can be called by a Flask request? Are you trying to do some kind of rate limiting? It seems likely that whatever you want to do can be done better using some other approach.
I want to use a decorator to handle auditing of various functions (mainly Django view functions, but not exclusively). In order to do this I would like to be able to audit the function post-execution - i.e. the function runs as normal, and if it returns without an exception, then the decorator logs the fact.
Something like:
#audit_action(action='did something')
def do_something(*args, **kwargs):
if args[0] == 'foo':
return 'bar'
else:
return 'baz'
Where audit_action would only run after the function has completed.
Decorators usually return a wrapper function; just put your logic in the wrapper function after invoking the wrapped function.
def audit_action(action):
def decorator_func(func):
def wrapper_func(*args, **kwargs):
# Invoke the wrapped function first
retval = func(*args, **kwargs)
# Now do something here with retval and/or action
print('In wrapper_func, handling action {!r} after wrapped function returned {!r}'.format(action, retval))
return retval
return wrapper_func
return decorator_func
So audit_action(action='did something') is a decorator factory that returns a scoped decorator_func, which is used to decorate your do_something (do_something = decorator_func(do_something)).
After decorating, your do_something reference has been replaced by wrapper_func. Calling wrapper_func() causes the original do_something() to be called, and then your code in the wrapper func can do things.
The above code, combined with your example function, gives the following output:
>>> do_something('foo')
In wrapper_func, handling action 'did something' after wrapped function returned 'bar'
'bar'
Your decorator can handle it here itself, like
def audit_action(function_to_decorate):
def wrapper(*args, **kw):
# Calling your function
output = function_to_decorate(*args, **kw)
# Below this line you can do post processing
print "In Post Processing...."
return output
return wrapper