I am new to python and decorators and am stumped in writing a decorator which reports not only passed args and kwargs but ALSO the unchanged default kwargs.
This is what I have so far.
def document_call(fn):
def wrapper(*args, **kwargs):
print 'function %s called with positional args %s and keyword args %s' % (fn.__name__, args, kwargs)
return fn(*args, **kwargs)
return wrapper
#document_call
def square(n, trial=True, output=False):
# kwargs are a bit of nonsense to test function
if not output:
print 'no output'
if trial:
print n*n
square(6) # with this call syntax, the default kwargs are not reported
# function square called with positional args (6,) and keyword args {}
# no output
36
square(7,output=True) # only if a kwarg is changed from default is it reported
# function square called with positional args (7,) and keyword args {'output': True}
49
The 'problem' is that this decorator reports the args that are passed in the call to square but does not report the default kwargs defined in the square definition. The only way kwargs are reported is if they're changed from their default i.e. passed to the square call.
Any recommendations for how I get the kwargs in the square definition reported too?
Edit after following up on the inspect suggestions, which helped me to the solution below. I changed the output of positional params to include their names because I thought it made the output easier to understand.
import inspect
def document_call(fn):
def wrapper(*args, **kwargs):
argspec = inspect.getargspec(fn)
n_postnl_args = len(argspec.args) - len(argspec.defaults)
# get kwargs passed positionally
passed = {k:v for k,v in zip(argspec.args[n_postnl_args:], args[n_postnl_args:])}
# update with kwargs
passed.update({k:v for k,v in kwargs.iteritems()})
print 'function %s called with \n positional args %s\n passed kwargs %s\n default kwargs %s' % (
fn.__name__, {k:v for k,v in zip(argspec.args, args[:n_postnl_args])},
passed,
{k:v for k,v in zip(argspec.args[n_postnl_args:], argspec.defaults) if k not in passed})
return fn(*args, **kwargs)
return wrapper
That was a good learning experience. It's neat to see three different solutions to the same problem. Thanks to the Answerers!
You'll have to introspect the function that you wrapped, to read the defaults. You can do this with the inspect.getargspec() function.
The function returns a tuple with, among others, a sequence of all argument names, and a sequence of default values. The last of the argument names pair up with the defaults to form name-default pairs; you can use this to create a dictionary and extract unused defaults from there:
import inspect
argspec = inspect.getargspec(fn)
positional_count = len(argspec.args) - len(argspec.defaults)
defaults = dict(zip(argspec.args[positional_count:], argspec.defaults))
You'll need to take into account that positional arguments can specify default arguments too, so the dance to figure out keyword arguments is a little more involved but looks like this:
def document_call(fn):
argspec = inspect.getargspec(fn)
positional_count = len(argspec.args) - len(argspec.defaults)
defaults = dict(zip(argspec.args[positional_count:], argspec.defaults))
def wrapper(*args, **kwargs):
used_kwargs = kwargs.copy()
used_kwargs.update(zip(argspec.args[positional_count:], args[positional_count:]))
print 'function %s called with positional args %s and keyword args %s' % (
fn.__name__, args[:positional_count],
{k: used_kwargs.get(k, d) for k, d in defaults.items()})
return fn(*args, **kwargs)
return wrapper
This determines what keyword paramaters were actually used from both the positional arguments passed in, and the keyword arguments, then pulls out default values for those not used.
Demo:
>>> square(39)
function square called with positional args (39,) and keyword args {'trial': True, 'output': False}
no output
1521
>>> square(39, False)
function square called with positional args (39,) and keyword args {'trial': False, 'output': False}
no output
>>> square(39, False, True)
function square called with positional args (39,) and keyword args {'trial': False, 'output': True}
>>> square(39, False, output=True)
function square called with positional args (39,) and keyword args {'trial': False, 'output': True}
Starting with Python 3.5 you can use BoundArguments.apply_defaults to fill in missing arguments with their default value:
import inspect
def document_call(fn):
def wrapper(*args, **kwargs):
bound = inspect.signature(fn).bind(*args, **kwargs)
bound.apply_defaults()
print(f'{fn.__name__} called with {bound}')
return fn(*args, **kwargs)
return wrapper
Since the decorator function wrapper takes any argument and just passes everything on, of course it does not know anything about the parameters of the wrapped function and its default values.
So without actually looking at the decorated function, you will not get this information. Fortunately, you can use the inspect module to figure out the default arguments of the wrapped function.
You can use the inspect.getargspec function to get the information about the default argument values in the function signature. You just need to match them up properly with the parameter names:
def document_call(fn):
argspec = inspect.getargspec(fn)
defaultArguments = list(reversed(zip(reversed(argspec.args), reversed(argspec.defaults))))
def wrapper(*args, **kwargs):
all_kwargs = kwargs.copy()
for arg, value in defaultArguments:
if arg not in kwargs:
all_kwargs[arg] = value
print 'function %s called with positional args %s and keyword args %s' % (fn.__name__, args, all_kwargs)
# still make the call using kwargs, to let the function handle its default values
return fn(*args, **kwargs)
return wrapper
Note that you could still improve this as right now you are handling positional and named arguments separately. For example, in your square function, you could also set trial by passing it as a positional argument after n. This will make it not appear in the kwargs. So you’d have to match the positional arguments with your kwargs to get the full information. You can get all the information about the positions from the argspec.
In python 3.6 I did it using inspect.getfullargspec:
def document_call(func):
#wraps(func)
def decorator(*args, **kwargs):
fullargspec = getfullargspec(func)
default_kwargs = fullargspec.kwonlydefaults
print('Default kwargs', default_kwargs)
print('Passed kwargs', kwargs)
return func(*args, **kwargs)
return decorator
Be aware about using the * separator when defining the decorated function for this to work
#document_call
def square(n, *, trial=True, output=False):
# kwargs are a bit of nonsense to test function
if not output:
print 'no output'
if trial:
print n*n
Here is the code modified to work with python3
import inspect
import decorator
#decorator.decorator
def log_call(fn,*args, **kwargs):
sign = inspect.signature(fn)
arg_names = list(sign.parameters.keys())
passed = {k:v for k,v in zip(arg_names[:len(args)], args)}
passed.update({k:v for k,v in kwargs.items()})
params_str = ", ".join([f"{k}={passed.get(k, '??')}" for k in arg_names])
print (f"{fn.__name__}({params_str})")
return fn(*args, **kwargs)
Note I'm using additional library "decorator" as it preserves the function signature.
Related
# This version only accepts one argument
# def shout(fn):
# def wrapper(name):
# return fn(name).upper()
# return wrapper
# This version works with any number of args
def shout(fn):
def wrapper(*args, **kwargs):
return fn(*args, **kwargs).upper()
return wrapper
#shout
def greet(name):
return f"Hi, I'm {name}."
#shout
def order(main, side):
return f"Hi, I'd like the {main}, with a side of {side}, please."
#shout
def lol():
return "lol"
print(greet("todd"))
print(order(side="burger", main="fries"))
print(lol())
In the above code,
def wrapper(*args, **kwargs):
print(f"abc is {fn.__name__}")
return fn(*args, **kwargs).upper()
return wrapper
When wrapper functions executed, how does it knows the value of arguments which are to be assigned to , * args and **kwargs. We have not defined the values of arguments here, but instead func is given the parameters.
side="burger", main="fries".
**kwargs open up the dictionary but when did we defined such dictionary?
How does side="burger", main="fries" are set as arguments of wrapper function and Why are they being assigned to args and kwargs ?
Why are arguments given to fync being assigned to parameters of wrapper function?
They are set when you call the function. When you call the wrapper, all positional arguments are packed together into args and all keyword arguments are put into kwargs, which are then unpacked and passed to the wrapped function.
When you call:
order(side="burger", main="fries")
You are actually calling wrapper() with those parameters.
This is the time at which **kwargs is assigned.
wrapper() then goes on to call your actual function order() with *args and **kwargs.
In Python 3.4+, functools.wraps preserves the signature of the function it wraps. Unfortunately, if you create decorators that are meant to be stacked on top of each other, the second (or later) decorator in the sequence will be seeing the generic *args and **kwargs signature of the wrapper and not preserving the signature of the original function all the way at the bottom of the sequence of decorators. Here's an example.
from functools import wraps
def validate_x(func):
#wraps(func)
def wrapper(*args, **kwargs):
assert kwargs['x'] <= 2
return func(*args, **kwargs)
return wrapper
def validate_y(func):
#wraps(func)
def wrapper(*args, **kwargs):
assert kwargs['y'] >= 2
return func(*args, **kwargs)
return wrapper
#validate_x
#validate_y
def foo(x=1, y=3):
print(x + y)
# call the double wrapped function.
foo()
This gives
-------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-5-69c17467332d> in <module>
22
23
---> 24 foo()
<ipython-input-5-69c17467332d> in wrapper(*args, **kwargs)
4 #wraps(func)
5 def wrapper(*args, **kwargs):
----> 6 assert kwargs['x'] <= 2
7 return func(*args, **kwargs)
8 return wrapper
KeyError: 'x'
and if you switch the order of the decorators, you get the same key error for 'y'.
I tried replacing wraps(func) with wraps(func.__wrapped__) in the second decorator, but this still doesn't work (not to mention it requires the programmer to explicitly know where in the stack of decorators they are working for given wrapper functionality).
I also took a look at inspect.signature(foo) and this seems to give the right thing, but I found that this is because inspect.signature has a follow_wrapped parameter that defaults to True so it somehow knows to follow the sequence of wrapped functions, but apparently the regular method call framework for invoking foo() will not follow this same protocol for resolve args and kwargs of the outer decorated wrapper.
How can I just have wraps faithfully passthrough the signature so that wraps(wraps(wraps(wraps(f)))) (so to speak) always faithfully replicated the signature of f?
You are not actually passing any arguments to you function foo so *args and **kwargs are empty for both decorators. If you pass arguments the decorators will work just fine
foo(x=2, y = 3) # prints 5
You can try to get default function arguments using inspect
You can't really get the default values without using inspect and you also need to account for positional args (*args) vs keyword args (**kwargs). So normalize the data if it's there if it's missing then inspect the function
import inspect
from functools import wraps
def get_default_args(func):
signature = inspect.signature(func)
return {
k: v.default
for k, v in signature.parameters.items()
if v.default is not inspect.Parameter.empty
}
def validate_x(func):
#wraps(func)
def wrapper(*args, **kwargs):
if args and not kwargs and len(args) == 2:
kwargs['x'] = args[0]
kwargs['y'] = args[1]
args = []
if not args and not kwargs:
kwargs = get_default_args(func)
assert kwargs['x'] <= 2
return func(*args, **kwargs)
return wrapper
def validate_y(func):
#wraps(func)
def wrapper(*args, **kwargs):
if args and not kwargs and len(args) == 2:
kwargs['x'] = args[0]
kwargs['y'] = args[1]
args = []
if not args and not kwargs:
kwargs = get_default_args(func)
assert kwargs['y'] >= 2
return func(*args, **kwargs)
return wrapper
#validate_x
#validate_y
def foo(x=1, y=3):
print(x + y)
# call the double wrapped function.
foo()
# call with positional args
foo(1, 4)
# call with keyword args
foo(x=2, y=10)
This prints
4
5
12
Your diagnosis is incorrect; in fact, functools.wraps preserves the signature of the double-decorated function:
>>> import inspect
>>> inspect.signature(foo)
<Signature (x=1, y=3)>
We can also observe that it is not a problem with calling the function with the wrong signature, since that would raise a TypeError, not a KeyError.
You seem to be under the impression that when just one decorator is used, kwargs will be populated with the argument default values. This doesn't happen at all:
def test_decorator(func):
#wraps(func)
def wrapper(*args, **kwargs):
print('args:', args)
print('kwargs:', kwargs)
return func(*args, **kwargs)
return wrapper
#test_decorator
def foo(x=1):
print('x:', x)
The output is:
>>> foo()
args: ()
kwargs: {}
x: 1
So as you can see, neither args nor kwargs receives the argument's default value, even when just one decorator is used. They are both empty, because foo() calls the wrapper function with no positional arguments and no keyword arguments.
The actual problem is that your code has a logical error. The decorators validate_x and validate_y expect the arguments to be passed as keyword arguments, but in fact they might be passed as positional arguments or not at all (so the default values would apply), in which case 'x' and/or 'y' won't be present in kwargs.
There is no easy way to make your decorators work with an argument which could be passed as either keyword or positional; if you make the arguments keyword-only, then you can test whether 'x' or 'y' are in kwargs before validating them.
def validate_x(func):
#wraps(func)
def wrapper(*args, **kwargs):
if 'x' in kwargs and kwargs['x'] > 2:
raise ValueError('Invalid x, should be <= 2, was ' + str(x))
return func(*args, **kwargs)
return wrapper
#validate_x
def bar(*, x=1): # keyword-only arg, prevent passing as positional arg
...
It's usually better to explicitly raise an error, instead of using assert, because your program can be run with assert disabled.
Beware also that it's possible to declare a function like #validate_x def baz(*, x=5): ... where the default x is invalid. This won't raise any error because the default argument value isn't checked by the decorator.
I'm not sure if this is possible in Python and I was wondering, is there any method for checking at runtime whether a parameter is passed to a Python function without doing some sort of check on the parameter value?
def my_func(req, opt=0):
return was_opt_passed() # Returns bool
print(my_func(0)) # Prints False
print(my_func(0, 0)) # Prints True
This would be superior in some circumstances, if possible, because it relieves the need to remember and check for sentinel values. Is it possible?
the standard method of detecting if an argument has been passed is sentinel values; however, if you're willing to lose your function signature, you can use **kwargs:
def my_func(**kwargs):
return 'opt' in kwargs
print(my_func()) #=> False
print(my_func(opt=0)) $=> True
As Mark has already stated in his comment the typical convention is to use the default value None. Then you can check if it’s still None upon calling.
def my_func(req, opt=None):
if opt is None:
#opt wasn’t passed.
return False
#opt was passed
return True
Although if you’d like to do more research on other options (most unconventional for most cases) feel free to check out these answers
One solution is to use decorators/wrappers. They allow you to interface with what's being passed to your function at runtime and then handle said things as you see fit Consider this code:
def check_keywords(func):
def wrapper(*args, **kwargs):
if kwargs:
print('Keyword was passed!')
return func(*args, **kwargs)
return wrapper
#check_keywords
def my_func(req, opt=0):
print(req)
check_keywords captures the function and if it detects keywords being passed into my_func, it then prints something. This print statement can be converted to any arbitrary code you want.
e.g.:
my_func(1)
>>> 1
my_func(1, opt = 1)
>>> Keyword was passed!
>>> 1
A great way would be to use *args or **kwargs.
def was_opt_passed(*args, **kwargs):
return len(args) > 0 or 'opt' in kwargs
def my_func(req, *args, **kwargs):
return was_opt_passed(*args, **kwargs) # Returns bool
print(my_func(0)) # Prints False
print(my_func(0, 0)) # Prints True
print(my_func(0, opt=0)) # Prints True
print(my_func(0, not_opt=0)) # Prints False
*args collects any positional arguments passed to your function that are not already enumerated, while **kwargs collects any keyword arguments passed to your function that are not already enumerated. If args contains a positional argument, we assume it was opt, and it must have been passed. Otherwise, if it is in kwargs, it was passed, and then if we didn't find it in either place, it must not have been passed.
See also https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments
Using decorator
def check_opt_passed(methd):
def m(req, *args, **kwarg):
# this will check even if opt is passed as positional argument
# and check if opt is passed not any other keyword
if args or (kwarg and 'opt' in kwarg):
print('opt is passed')
else:
print('opt is not passed')
return methd(req, *args, **kwarg)
return m
#check_opt_passed
def my_func(req, opt=0):
# dummy expression for testing
return req * opt
print(my_func(1)) # opt is not passed
print(my_func(1, 0)) # opt is passed
print(my_func(1, opt=0)) # opt is passed
I`m reading and following Pro Python book, i createad this two decorators annotation_decorator and typesafe using the same code from book, but when i try to run this code i receive:
TypeError: decorator() takes 1 positional argument but 2 were given
The code is the same of the book and i can't figure out why this does work, can you guys spot what is wrong? I hosted the PoC here in case you wanna test: https://repl.it/repls/GlassNotedPayware
import functools
import inspect
from itertools import chain
def annotation_decorator(process):
"""
Creates a decorator that processes annotations for each argument passed
into its target function, raising an exception if there's a problem.
"""
#functools.wraps(process)
def decorator(func):
spec = inspect.getfullargspec(func)
annotations = spec.annotations
defaults = spec.defaults or ()
defaults_zip = zip(spec.args[-len(defaults):], defaults)
kwonlydefaults = spec.kwonlydefaults or {}
for name, value in chain(defaults_zip, kwonlydefaults.items()):
if name in annotations:
process(value, annotations[name])
#functools.wraps(func)
def wrapper(*args, **kwargs):
# Populate a dictionary of explicit arguments passed positionally
explicit_args = dict(zip(spec.args, args))
new_args = []
new_kwargs = {}
keyword_args = kwargs.copy()
# Deal with explicit arguments passed positionally
for name, arg in explicit_args:
if name in annotations:
new_args.append(process(arg, annotations[name]))
# Add all explicit arguments passed by keyword
for name in chain(spec.args, spec.kwonlyargs):
if name in kwargs:
new_kwargs[name] = process(keyword_args.pop(name),
annotations[name])
# Deal with variable positional arguments
if spec.varargs and spec.varargs in annotations:
annotation = annotations[spec.varargs]
for arg in args[len(spec.args):]:
new_args.append(process(arg, annotation))
# Deal with variable keyword arguments
if spec.varkw and spec.varkw in annotations:
annotation = annotations[spec.varkw]
for name, arg in keyword_args.items():
new_kwargs[name] = process(arg, annotation)
r = func(*new_args, **new_kwargs)
if 'return' in annotations:
r = process(r, annotations['return'])
return r
return wrapper
return decorator
#annotation_decorator
def typesafe(value, annotation):
"""
Verify that the function is called with the right argument types and
that it returns a value of the right type, according to its annotations
"""
if not isinstance(value, annotation):
raise TypeError("Expected %s, got %s." % (annotation.__name__,
type(value).__name__))
return value
#annotation_decorator
def coerce_arguments(value, annotation):
return annotation(value)
#typesafe(str, str)
def combine(a, b):
return a + b
combine('spam', 'alot')
I modified two things based on #chepner commentary:
Instead of using typesafe decorator on combine function i simple decorated with annotation_decorator passing typesafe as arg and also add function annotation to the variables of function.
#annotation_decorator(process=typesafe)
def combine(a: str, b: str):
return a + b
I wanted an easy-to-use decorator class that can be used to force certain parameters to be pre-defined types lest an error be raised. The issue is that I have to specify each variable twice, one for each args and one for kwargs. This is true even with default parameters, provided there is no single splat parameter in the original function.
Simplified:
class ParamConstraint:
def __init__(self, *args, error=TypeError, **kwargs):
self.args = args
self.kwargs = kwargs
self.error = error
def __call__(self, function):
def wrap(*args, **kwargs):
for (arg, constraint) in zip(args, self.args):
if not isinstance(arg, constraint):
raise self.error(
"Value '{}' is not of type '{}'.".format(arg, constraint.__name__)
)
for kwarg in self.kwargs:
if kwarg in kwargs:
assert isinstance(kwargs[kwarg], self.kwargs[kwarg])
return function(*args, **kwargs)
return __import__("functools").wraps(function)(wrap)
This is easy enough to use. Say I wanted a function that would multiply a number by 2, but I only want integers, not floats. I can use:
#ParamConstraint(int)
def foo(x):
return x*2
print(foo(5)) # => prints 10
print(foo(x=5.0)) # => prints 10.0
print(foo(5.0)) # => raises TypeError
Now, I can prevent this by passing both positional and non-positional arguments into a function, but it gets very tedious after a while. For example, I'd have to write out 6 constraints for 3 parameters:
#ParamConstraint((int, float), int, FunctionType, value=(int, float), count=int, func=FunctionType):
def foo(value, count, func):
for _ in range(count):
value = func(value)
return value
Ideally, I should be able to just pass it by KWarg and not provide any positional arguments. However, this does not work because I have no way of finding out what the positional arguments of the original function are. I have used the inspect module minorly to no avail.
How can I remediate this?