I have a python module with lots of functions and I want to apply a decorator for all of them.
Is there a way to patch all of them via monkey-patching to apply this decorator for every function without copy-pasting on line of applying decorator?
In other words, I want to replace this:
#logging_decorator(args)
func_1():
pass
#logging_decorator(args)
func_2():
pass
#logging_decorator(args)
func_3():
pass
#logging_decorator(args)
func_n():
pass
With this:
patch_func():
# get all functions of this module
# apply #logging_decorator to all (or not all) of them
func_1():
pass
func_2():
pass
func_3():
pass
func_n():
pass
I'm really not certain that this is a good idea. Explicit is better than implicit, after all.
With that said, something like this should work, using inspect to find which members of a module can be decorated and using __dict__ to manipulate the module's contents.
import inspect
def decorate_module(module, decorator):
for name, member in inspect.getmembers(module):
if inspect.getmodule(member) == module and callable(member):
if member == decorate_module or member == decorator:
continue
module.__dict__[name] = decorator(member)
Sample usage:
def simple_logger(f):
def wrapper(*args, **kwargs):
print("calling " + f.__name__)
f(*args, **kwargs)
return wrapper
def do_something():
pass
decorate_module(sys.modules[__name__], simple_logger)
do_something()
I ain't gonna be pretty ... but you can list all functions using dir() after their definition. Then I can't think of a way to patch them without a wrapper function.
def patched(func):
#logging_decorator
def newfunc(*args, **kwargs):
return func(*args, **kwargs)
return newfunc
funcs=[f in dir() if not '__' in f]
for f in funcs:
exec(f+'=patched(f)')
Related
I want use a same param in decorater and function like:
def outer(deco_param):
def decorator(func):
print("param:%s" % deco_param)
return func
return decorator
#outer(deco_param=s) # problems here: 's' is unreferenced
def test(s):
print("test:%s" % s)
I got this idea when using Flask but I didn't know how did they make it, which supports using a param of view function in its decorater like:
app.route(rule="/user/<int:uid>")
def access_user(uid): # if I use any other name except from 'uid', IDE would raise a tip
...
What's more, Flask seems to make a static check. If I miss uid argument or use any other name, IDE(pycharm) would raise a tip of "Function 'access_user` doesn't have a parameter 'int:uid'". That's the effect I want.
Can anyone please give me some advice? Thanks
You don't need to pass the same parameter to both the outer function and the inner function. To call the parameters via decorator, you should add a wrapper function to pass the parameters (*args) from inner one to outer one.
You can write like this:
def outer():
def decorator(func):
def wrapper(*args):
print("param:%s" % func.__param__)
return func(*args)
return wrapper
return decorator
#outer
def test(s):
print("test:%s" % s)
What I am trying to do is write a wrapper around another module so that I can transform the parameters that are being passed to the methods of the other module. That was fairly confusing, so here is an example:
import somemodule
class Wrapper:
def __init__(self):
self.transforms = {}
self.transforms["t"] = "test"
# This next function is the one I want to exist
# Please understand the lines below will not compile and are not real code
def __intercept__(self, item, *args, **kwargs):
if "t" in args:
args[args.index("t")] = self.transforms["t"]
return somemodule.item(*args, **kwargs)
The goal is to allow users of the wrapper class to make simplified calls to the underlying module without having to rewrite all of the functions in the module. So in this case if somemodule had a function called print_uppercase then the user could do
w = Wrapper()
w.print_uppercase("t")
and get the output
TEST
I believe the answer lies in __getattr__ but I'm not totally sure how to use it for this application.
__getattr__ combined with defining a function on the fly should work:
# somemodule
def print_uppercase(x):
print(x.upper())
Now:
from functools import wraps
import somemodule
class Wrapper:
def __init__(self):
self.transforms = {}
self.transforms["t"] = "test"
def __getattr__(self, attr):
func = getattr(somemodule, attr)
#wraps(func)
def _wrapped(*args, **kwargs):
if "t" in args:
args = list(args)
args[args.index("t")] = self.transforms["t"]
return func(*args, **kwargs)
return _wrapped
w = Wrapper()
w.print_uppercase('Hello')
w.print_uppercase('t')
Output:
HELLO
TEST
I would approach this by calling the intercept method, and entering the desired method to execute, as a parameter for intercept. Then, in the intercept method, you can search for a method with that name and execute it.
Since your Wrapper object doesn't have any mutable state, it'd be easier to implement without a class. Example wrapper.py:
def func1(*args, **kwargs):
# do your transformations
return somemodule.func1(*args, **kwargs)
Then call it like:
import wrapper as w
print w.func1('somearg')
I am trying to test a function that is decorated. Is there a way to mock a decorator and test function in isolation, when decorator is already applied?
import mock
def decorator(func):
def wrapper(*args, **kwargs):
return 1
return wrapper
def mocked(func):
def wrapper(*args, **kwargs):
return 2
return wrapper
#decorator
def f():
return 0
with mock.patch('test.decorator') as d:
d.side_effect = mocked
assert f() == 2 # error
There is not a simple solution.
This is a similar question: How to strip decorators from a function in python
You can either modify the original code just for testing, or use something like this library: https://pypi.python.org/pypi/undecorated in order to write a helper function to switch from the original wrapper to the testing wrapper:
from undecorated import undecorated
mocked(undecorated(f))()
How do I pass a decorator's function into a job?
I have a decorator that would run a job using the function.
#job
def queueFunction(passedFunction, *args, **kwargs):
# Do some stuff
passedFunction(*args, **kwargs)
def myDecorator(async=True):
def wrapper(function):
def wrappedFunc(*args, **kwargs):
data = DEFAULT_DATA
if async:
queueFunction.delay(function, *args, **kwargs)
else:
data = queueFunction(function, *args, **kwargs)
return data
return wrappedFunc
return wrapper
I get an error when trying to use it.
Can't pickle <function Model.passedFunction at 0x7f410ad4a048>: it's not the same object as modelInstance.models.Model.passedFunction
Using Python 3.4
What happens is that you are passing in the original function (or method) to the queueFunction.delay() function, but that's not the same function that it's qualified name says it is.
In order to run functions in a worker, Python RQ uses the pickle module to serialise both the function and its arguments. But functions (and classes) are serialised as importable names, and when deserialising the pickle module simply imports the recorded name. But it does first check that that will result in the right object. So when pickling, the qualified name is tested to double-check it'll produce the exact same object.
If we use pickle.loads as a sample function, then what roughly happens is this:
>>> import pickle
>>> import sys
>>> sample_function = pickle.loads
>>> module_name = sample_function.__module__
>>> function_name = sample_function.__qualname__
>>> recorded_name = f"{module_name}.{function_name}"
>>> recorded_name
'_pickle.loads'
>>> parent, obj = sys.modules[module_name], None
>>> for name in function_name.split("."): # traverse a dotted path of names
... obj = getattr(parent, name)
...
>>> obj is sample_function
True
Note that pickle.loads is really _pickle.loads; that doesn't matter all that much, but what does matter is that _pickle can be accessed and it has an object that can be found by using the qualified name, and it is the same object still. This will work even for methods on classes (modulename.ClassName.method_name).
But when you decorate a function, you are potentially replacing that function object:
>>> def decorator(f):
... def wrapper(*args, **kwargs):
... return f, f(*args, **kwargs)
... return wrapper
...
>>> #decorator
... def foo(): pass
...
>>> foo.__qualname__
'decorator.<locals>.wrapper'
>>> foo()[0].__qualname__ # original function
'foo'
Note that the decorator result has a very different qualified name from the original! Pickle won't be able to map that back to either the decorator result or to the original function.
You are passing in the original, undecorated function to queueFunction.delay(), and it's qualified name will not match that of the wrappedFunc() function you replaced it with; when pickle tries to import the fully qualified name found on that function object, it'll find the wrappedFunc object and that's not the same object.
There are several ways around this, but the easiest is to store the original function as an attribute on the wrapper, and rename it's qualified name to match. This makes the original function available
You'll have to use he #functools.wraps() utility decorator here to copy various attributes from the original, decorated function over to your wrapper function. This includes the original name.
Here is a version that alters the original function qualified name:
from functools import wraps
def myDecorator(async_=True):
def wrapper(function):
#wraps(function)
def wrappedFunc(*args, **kwargs):
data = DEFAULT_DATA
if async:
queueFunction.delay(function, *args, **kwargs)
else:
data = queueFunction(function, *args, **kwargs)
return data
# make the original available to the pickle module as "<name>.original"
wrappedFunc.original = function
wrappedFunc.original.__qualname__ += ".original"
return wrappedFunc
return wrapper
The #wraps(function) decorator makes sure that wrappedFunc.__qualname__ is set to that of function, so if function was named foo, so now is the wrappedFunc function object. The wrappedFunc.original.__qualname__ += ".original" statement then sets the qualified name of wrappedFunc.original to foo.original, and that's exactly where pickle can find it again!
Note: I renamed async to async_ to make the above code work on Python 3.7 and above; as of Python 3.7 async is a reserved keyword.
I also see that you are making the decision to run something synchronous or asynchronous at decoration time. In that case I'd re-write it to not check the aync_ boolean flag each time you call the function. Just return different wrappers:
from functools import wraps
def myDecorator(async_=True):
def decorator(function):
if async_:
#wraps(function)
def wrapper(*args, **kwargs):
queueFunction.delay(wrappedFunc.original, *args, **kwargs)
return DEFAULT_DATA
# make the original available to the pickle module as "<name>.original"
wrapper.original = function
wrapper.original.__qualname__ += ".original"
else:
#wraps(function)
def wrapper(*args, **kwargs):
return queueFunction(function, *args, **kwargs)
return wrapper
return decorator
I also renamed the various inner functions; myDecorator is a decorator factory that returns the actual decorator, and the decorator returns the wrapper.
Either way, the result is that now the .original object can be pickled:
>>> import pickle
>>> #myDecorator(True)
... def foo(): pass
...
>>> foo.original
<function foo.original at 0x10195dd90>
>>> pickle.dumps(foo.original, pickle.HIGHEST_PROTOCOL)
b'\x80\x04\x95\x1d\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x0cfoo.original\x94\x93\x94.'
This is what i want to do:
#MyDecorator
def f():
pass
for d in f.decorators:
print d
This is not generally possible without the cooperation of the decorators. For example,
def my_decorator(f):
def wrapper(*args, **kwargs):
return f(*args, **kwargs)
wrapper.decorators = [wrapper]
if hasattr(f, 'decorators'):
wrapper.decorators.extend[f.decorators]
return wrapper
Essentially, all the decorator does is wrap the function as usual and then put a decorators attribute on the wrapper function. It then checks the wrapped function for a similar list and propagates it upwards.
This is pretty useless though
What I think you want is
def my_decorator(f):
def wrapper(args):
return f(args)
wrapper.decorated = f
return wrapper
This will allow you to do stuff like
#my_other_decorator # another decorator following the same pattern
#my_decorator
def foo(args):
print args
foo.decorated(args) # calls the function with the inner decorated function (my_decorator)
foo.decorated.decorated(args) # original function
You can actually abstract this pattern into a decorator
def reversable(decorator):
def wrapper(func):
ret = decorator(func) # manually apply the decorator
ret.decorated = func # save the original function
return ret
return wrapper
Now when you are writing your decorator:
#reversable
def my_decorator(f):
def wrapper(x):
return f(x + 1)
return wrapper
The #MyDecorator syntax is just shorthand for writing the following Python code:
def f():
pass
f = MyDecorator(f)
Written in this form, you can see that the decorators applied to the function are not kept track of in any way. You could make your decorators remember when they're applied (Aaron's answer has a couple good ideas on how to do this), but you'd have to wrap all third-party decorators with your own.