TL;TR Looking for idioms and patterns to unpack positional and keyword arguments into ordered sequence of positional arguments, based on simple specification, e.g. a list of names. The idea seems similar to scanf-like parsing.
I'm wrapping functions of a Python module, called someapi.
Functions of someapi only expect positional arguments, which are in pain numbers in most cases.
I'd like to enable callers with flexibility of how they can pass arguments to my wrappers.
Here are examples of the wrappers invocations I'd like to allow:
# foo calls someapi.foo()
foo(1, 2, 3, 4)
foo(1, 2, 3, 4, 5) # but forward only 1st 4 to someapi.foo
foo([1, 2, 3, 4])
foo([1, 2, 3, 4, 5, 6]) # but forward only 1st 4 to someapi.foo
foo({'x':1, 'y':2, 'z':3, 'r':4})
foo(x=1, y=2, z=3, r=4)
foo(a=0, b=0, x=1, y=2, z=3, r=4) # but forward only x,y,z,r someapi.foo
I don't see any need to support convoluted case of mixed positional and keyword arguments:
foo(3, 4, x=1, y=2)
Here is my first stab at implementing such arguments handling for the foo wrapper calling someapi.foo:
def foo(*args, **kwargs):
# BEGIN arguments un/re-packing
a = None
kwa = None
if len(args) > 1:
# foo(1, 2, 3, 4)
a = args
elif len(args) == 1:
if isinstance(args[0], (list, tuple)) and len(args[0]) > 1:
# foo([1, 2, 3, 4])
a = args[0]
if isinstance(args[0], dict):
# foo({'x':1, 'y':2, 'z':3, 'r':4})
kwa = args[0]
else:
# foo(x=1, y=2, z=3, r=4)
kwa = kwargs
if a:
(x, y, z, r) = a
elif kwa:
(x, y, z, r) = (kwa['x'], kwa['y'], kwa['z'], kwa['r'])
else:
raise ValueError("invalid arguments")
# END arguments un/re-packing
# make call forwarding unpacked arguments
someapi.foo(x, y, z, r)
It does the job as expected, as far as I can tell, but it there are two issues:
Can I do it better in more Python idiomatic fashion?
I have dozen(s) of someapi functions to wrap, so how to avoid copying and adjusting the whole block between BEGIN/END marks in every wrapper?
I don't know the answer for the question 1, yet.
Here, however, is my attempt to address the issue 2.
So, I defined a generic handler for arguments based on the simple specification of names.
The names specify a couple of things, depending on the actual wrapper invocation:
How many arguments to unpack from *args? (see len(names) test below)
What keyword arguments are expected in **kwargs? (see generator expression returning tuple below)
Here is new version:
def unpack_args(names, *args, **kwargs):
a = None
kwa = None
if len(args) >= len(names):
# foo(1, 2, 3, 4...)
a = args
elif len(args) == 1:
if isinstance(args[0], (list, tuple)) and len(args[0]) >= len(names):
# foo([1, 2, 3, 4...])
a = args[0]
if isinstance(args[0], dict):
# foo({'x':1, 'y':2, 'z':3, 'r':4...})
kwa = args[0]
else:
# foo(x=1, y=2, z=3, r=4)
kwa = kwargs
if a:
return a
elif kwa:
if all(name in kwa.keys() for name in names):
return (kwa[n] for n in names)
else:
raise ValueError("missing keys:", \
[name for name in names if name not in kwa.keys()])
else:
raise ValueError("invalid arguments")
This allows me to implement the wrapper functions in the following way:
def bar(*args, **kwargs):
# arguments un/re-packing according to given of names
zargs = unpack_args(('a', 'b', 'c', 'd', 'e', 'f'), *args, **kwargs)
# make call forwarding unpacked arguments
someapi.bar(*zargs)
I think I have achieved all the advantages over the foo version above that I was looking for:
Enable callers with the requested flexibility.
Compact form, cut down on copy-and-paste.
Flexible protocol for positional arguments: bar can be called with 7, 8 and more positional arguments or a long list of numbers, but only first 6 are taken into account. For example, it would allow iterations processing long list of numbers (e.g. think of geometry coordinates):
# meaw expects 2 numbers
n = [1,2,3,4,5,6,7,8]
for i in range(0, len(n), 2):
meaw(n[i:i+2])
Flexible protocol for keyword arguments: more keywords may be specified than actually used or dictionary can have more items than used.
Getting back to the question 1 above, can I do better and make it more Pythonic?
Also, I'd like to ask for review of my solution: you see any bugs? have I overlooked anything? how to improve it?
Python is a very powerful language that allows you manipulate code in any way you want, but understanding what you're doing is hard. For this you can use the inspect module. So an example of how to wrap a function in someapi. I'll only consider positional arguments in this example, you can intuit how to extend this further. You can do it like this:
import inspect
import someapi
def foo(args*):
argspec = inspect.getargspec(someapi.foo)
if len(args) > len(argspec.args):
args = args[:len(argspec.args)]
return someapi.foo(*args)
This will detect if the number of arguments given to foo is too many and if so, it will get rid of the excess arguments. On the other hand, if there are too few arguments then it will just do nothing and let foo handle the errors.
Now to make it more pythonic. The ideal way to wrap many functions using the same template is to use decorator syntax (familiarity with this subject is assumed, if you want to learn more then see the docs at http://www.python.org/doc). Although since decorator syntax is mostly used on functions that are in development rather than wrapping another API, we'll make a decorator but just use it as a factory (the factory pattern) for our API. To make this factory we'll make use of the functools module to help us out (so the wrapped function looks as it should). So we can turn our example into:
import inspect
import functools
import someapi
def my_wrapper_maker(func):
#functools.wraps(func)
def wrapper(args*):
argspec = inspect.getargspec(func)
if len(args) > len(argspec.args):
args = args[:len(argspec.args)]
return func(*args)
return wrapper
foo = my_wrapper_maker(someapi.foo)
Finally, if someapi has a relatively large API that could change between versions (or we just want to make our source file more modular so it can wrap any API) then we can automate the application of my_wrapper_maker to everything exported by the module someapi. We'll do this like so:
__all__ = ['my_wrapper_maker']
# Add the entire API of someapi to our program.
for func in someapi.__all__:
# Only add in bindings for functions.
if callable(getattr(someapi, func)):
globals()[func] = my_wrapper_maker(getattr(someapi, func))
__all__.append(func)
This probably considered the most pythonic way to implement this, it makes full use of Python's meta-programming resources and allows the programmer to use this API everywhere they want without depending on a specific someapi.
Note: Whether this is most idiomatic way to do this is really up to opinion. I personally believe that this follows the philosophy set out in "The Zen of Python" quite well and so to me it is very idiomatic.
Related
Let's consider a function like this:
def f(*args, **kwargs):
...
*args are a variable number of arguments that must be:
(an object, a float, a float), possibly repeated N times,
followed by 0 to N' objects.
For example this is a valid call:
f(my_obj, 0, 1, other_obj, 2, 3, obj3, obj4)
but this is invalid:
f(my_obj, other_obj, 2, 3)
This function is exposed to users through a Python shell.
So, there is value in checking user input -- I am using
the typeguard library that works with type annotations
(like mypy).
I am trying to use typing module to write the proper annotations...
I thought I could at least express the constraint on the groups of 3 args
like this:
#typeguard.typechecked
f(*args:Tuple[Any,float,float])
But it does not work.
And in anyways I have no idea how to add the constraint on the following objects.
Of course I can craft myself some code to check arguments, but I am sure something better exists for cases of complex variable arguments sequences (either a clever use of the typing module or another Python lib ?)
What I meant by making the validation:
def _validate(a, b, c):
assert isinstance(b, float), f"{b} is not a float!"
assert isinstance(c, float), f"{c} is not a float"
def _validate_args(args):
if (len(args) % 3 != 0): # wrong number of args
raise ValueError("Arguments must be passed in pack of 3")
for idx in range(0, len(args), 3):
a, b, c = args[idx: idx + 3]
_validate(a, b, c)
def func(*args, **kwargs):
_validate_args(args)
func(1, 2.0, 3, 1, 2, 3)
AssertionError: 3 is not a float
You can make any message you want.
How can I check if a function can always be called with the same arguments as another function? For example, b can be called with all arguments provided to a.
def a(a, b, c=None):
pass
def b(a, *args, d=4,**kwargs):
pass
The reason I want this is that I have a base function:
def f(a, b):
print('f', a, b)
and a list of callbacks:
def g(b, a):
print('g', a, b)
def h(*args, **kwargs):
print('h', args, kwargs)
funcs = [g, h]
and a wrapper function that accepts anything:
def call(*args, **kwargs):
f(*args, **kwargs)
for func in funcs:
func(*args, **kwargs)
Now I want to check if all callbacks will accept the arguments provided to call(), assuming they're valid for f().
For performance reasons, I don't want to check the arguments every time call() is called, but rather check each callback before adding it to the list of callbacks.
For example, those calls are ok:
call(1, 2)
call(a=1, b=3)
But this one should fail because g has arguments in wrong order:
call(1, b=3)
This took a bit of fun research, but i think i've covered the corner cases. A number of them arise to keep things compatible with python 2 while new syntax being added.
Most problematic part is the fact that some named (keyword) parameters can be passed in as positional argument or be required based on order passed in.
For more see comments.
Below code will ensure that function b can be called using any possible combination of valid arguments to function a. (does not imply inverse).
Uncomment/add try except block to get true/valse result and not an AssertionError.
import inspect
def check_arg_spec(a,b):
"""
attrs of FullArgSpec object:
sp.args = pos or legacy keyword arguments, w/ keyword at back
sp.varargs = *args
sp.varkw = **kwargs
sp.defaults = default values for legacy keyword arguments #
sp.args
sp.kwdonly = keyword arguments follow *args or *, must be passed in by name
sp.kwdonlydefaults = {name: default_val, .... }
sp.annotatons -> currently not in use, except as standard flag for outside applications
Consume order:
(1) Positional arguments
(2) legacy keyword argument = default (can be filled by both keyword or positional parameter)
[
(3) *args
[
(4) keyword only arguments [=default]
]
]
(5) **kwds
"""
a_sp = inspect.getfullargspec(a)
b_sp = inspect.getfullargspec(b)
kwdfb = b_sp.kwonlydefaults or {}
kwdfa = a_sp.kwonlydefaults or {}
kwddefb = b_sp.defaults or []
kwddefa = a_sp.defaults or []
# try:
akwd = a_sp.kwonlyargs
if len(kwddefa):
akwd += a_sp.args[-len(kwddefa):]
bkwd = b_sp.kwonlyargs
if len(kwddefb):
bkwd += b_sp.args[-len(kwddefb):]
# all required arguments in b must have name match in a spec.
for bkey in (set(b_sp.args) ^ set(bkwd)) & set(b_sp.args) :
assert bkey in a_sp.args
# all required positional in b can be met by a
assert (len(a_sp.args)-len(kwddefb)) >= (len(b_sp.args)-len(kwddefb))
# if a has *args spec, so must b
assert not ( a_sp.varargs and b_sp.varargs is None )
# if a does not take *args, max number of pos args passed to a is len(a_sp.args). b must accept at least this many positional args unless it can consume *args
if b_sp.varargs is None:
# if neither a nor b accepts *args, check that total number of pos plus py2 style keyword arguments for sg of b is more than a can send its way.
assert len(a_sp.args) <= len(b_sp.args)
# Keyword only arguments of b -> they are required, must be present in a.
akws = set(a_sp.kwonlyargs) | set(a_sp.args[-len(kwddefa):])
for nmreq in (set(b_sp.kwonlyargs)^set(kwdfb)) & set(b_sp.kwonlyargs):
assert nmreq in akws
# if a and b both accept an arbitrary number of positional arguments or if b can but a cannot, no more checks neccessary here
# if a accepts optional arbitrary, **kwds, then so must b
assert not (a_sp.varkw and b_sp.varkw is None)
if b_sp.varkw is None:
# neither a nor b can consume arbitrary keyword arguments
# then b must be able to consume all keywords that a can be called w/th.
for akw in akwd:
assert akw in bkwd
# if b accepts **kwds, but not a, then there is no need to check further
# if both accept **kwds, then also no need to check further
# return True
#
# except AssertionError:
#
# return False
Not sure what you are really looking for and I'm pretty sure your issue could be solved in a better way, but anyway:
from inspect import getargspec
def foo(a, b, c=None):
pass
def bar(a, d=4, *args, **kwargs):
pass
def same_args(func1, func2):
return list(set(getargspec(func1)[0]).intersection(set(getargspec(func2)[0])))
print same_args(foo, bar)
# => ['a']
same_args just check arguments from func1 and func2 and returns a new list with only same arguments in both func1 and func2.
Is there a Pythonic way to create a function that accepts both separate arguments and a tuple? i.e to achieve something like this:
def f(*args):
"""prints 2 values
f(1,2)
1 2
f( (1,2) )
1 2"""
if len(args) == 1:
if len(args[0]) != 2:
raise Exception("wrong number of arguments")
else:
print args[0][0],args[0][1]
elif len(args) == 2:
print args[0],args[1]
else:
raise Exception("wrong number of arguments")
First of all I don't know if it is very wise to do so. Say a person calls a function like:
f(*((1,4),(2,5)))
As you can see the tuple contains two elements. But now for some reason, the person calls it with only one element (because for instance the generator did not generated two elements):
f(*((1,4),))
Then the user would likely want your function to report this, but now it will simply accept it (which can lead to complicated and unexpected behavior). Okay printing the elements of course will not do much harm. But in a general case the consequences might be more severe.
Nevertheless an elegant way to do this is making a simple decorator that first checks if one element is fed it checks if one tuple element is feeded and if so expands it:
def one_tuple(f):
def g(*args):
if len(args) == 1 and isinstance(args[0],tuple):
return f(*args[0])
else:
return f(*args)
return g
And apply it to your f:
#one_tuple
def f(*args):
if len(args) == 2:
print args[0],args[1]
else:
raise Exception("wrong number of arguments")
The decorator one_tuple thus checks if one tuple is fed, and if so unpacks it for you before passing it to your f function.
As a result f does not have to take the tuple case into account: it will always be fed expanded arguments and handle these (of course the opposite could be done as well).
The advantage of defining a decorator is its reuse: you can apply this decorator to all kinds of functions (and thus make it easier to implement these).
The Pythonic way would be to use duck typing. This will only work if you are sure that none of the expanded arguments are going to be iterable.
def f(*args):
def g(*args):
# play with guaranteed expanded arguments
if len(args) == 1:
try:
iter(args[0])
except TypeError:
pass
else:
return g(*args[0])
return g(*args)
This implementation offers a slight improvement on #Ev.Kounis's answer for cases where a single non-tuple argument is passed in. It can also easily be turned into the equivalent decorator described by #WillemVanOnsem. Use the decorator version if you have more than one function like this.
I do not agree to the idea myself (even though I like the fact that python does not require the definition of variable types and thus allows such a thing) but there might be cases where such a thing is needed. So here you go:
def my_f(a, *b):
def whatever(my_tuple):
# check tuple for conformity
# do stuff with the tuple
print(my_tuple)
return
if hasattr(a, '__iter__'):
whatever(a)
elif b:
whatever((a,) + b)
else:
raise TypeError('malformed input')
return
Restructured it a bit but the logic stays the same. if "a" is an iterable consider it to be your tuple, if not take "b" into account as well. if "a" is not an iterable and "b" is not defined, raise TypeError
my_f((1, 2, 3)) # (1, 2, 3)
my_f(1, 2, 3) # (1, 2, 3)
What is a most pythonic way to write a function to either pass in arguments or a tuple/list of arguments?
For example, a function add could either take in an argument of add(1, 2) or add((1, 2)) and both output 3.
What I have so far: (it works, but does not look nice)
def add(*args):
if len(args) == 1:
return (args[0][0] + args[0][1])
if len(args) == 2:
return args[0] + args[1]
else:
print "error: add takes in one or two arguments"
What I don't like about it is:
I have to print the error about passing in one or two arguments
The args[0][0] looks very unreadable
This way, it is hard to tell what the arguments passed in represent (they don't have names)
I dont know if this is the most "pythonic" way but it will do what you want:
def add(a, b=None):
return a+b if b is not None else sum(a)
If your function takes a specific number of arguments, then the most pythonic way to do this is to not do it. Rather if the user has a tuple with the arguments, you make them unpack them when they call the function. E.g.
def add(a, b):
return a + b
Then the caller can do
add(1,2)
or
t = (1,2)
add(*t)
The only time you want to accept either a sequence of params or individual params is when you can have any arbitrary (non-zero) number of arguments (e.g. the max and min builtin functions) in which case you'd just use *args
If you can only take a finite number of arguments, it makes more sense to ask for those specifically. If you can accept an arbitrary number of arguments, then the *args paradigm works well if you loop through it. Mixing and matching those two aren't very elegant.
def add(*args):
total = 0
for i in args:
total += i
return total
>>> add(1, 2, 3)
6
(I know we could just use sum() there, but I'm trying to make it look a bit more general)
In the spirit of python duck typing, if you see 1 argument, assume its something that expands to 2 arguments. If its then 2, assume its two things that add together. If it violates your rules, raise an exception like python would do on a function call.
def add(*args):
if len(args) == 1:
args = args[0]
if len(args) != 2:
raise TypeError("add takes 2 arguments or a tuple of 2 arguments")
return args[0] + args[1]
A decorator would be best suited for this job.
from functools import wraps
def tupled_arguments(f):
#wraps(f) # keeps name, docstring etc. of f
def accepts_tuple(tup, *args):
if not args: # only one argument given
return f(*tup)
return f(tup, *args)
return accepts_tuple
#tupled_arguments
def add(a, b):
return a + b
Our professor used this in the assignment. I don't think "The binary version of a function" exist after searching about it in Google. What do you think it means?
Say we have a function add that adds a bunch of numbers. Rather than
writing add(3, 5, 4, 1) we want to use currying to create an adder
function that can be extended using a chain of calls. We would then
have adder(3)(5)(4)(1)(). Let us assume we have the currying function
that can create this adder given the add2 function (the binary version
of add) and a start value. Let us call it curry. Then we have adder =
curry(add2, 0).
I think he means a function that accepts only two arguments, so it just adds two numbers. His example function add(3, 5, 4, 1) would be a function that accepts any number of arguments and adds them all, but add2 would only accept two arguments, so add2(3, 5) would be 8. "The binary version of a function" in this case means a binary function (a function accepting two arguments).
In this case "binary function" refers to an argument that accepts two arguments. In this case your professor is probably referring to something like this:
def add2(x, y):
return x + y
# equivalently: add2 = lambda x,y: x+y
def curry(func, num):
def wrapped(*args):
if len(args) == 0:
return num
elif len(args) > 1:
raise TypeError('{} takes 1 positional argument but '
'{} were given'.format(
func.__name__, len(args)))
arg = args[0]
return curry(func, func(num, arg))
return wrapped
#AdamSmith and #BrenBarn have already pointed out what binary function means. A simple and clear assignment solution can be write by using object instead of decorator.
class curry():
def __init__(self, func, default):
self._f = func
self._default = default
def __call__(self, val=None):
if val is None:
return self._default
return curry(self._f,self._f(self._default,val))
print(curry(lambda x,y:x+y, 0)(3)(5)(4)(1)())
Neat and simple!
IMHO functors should be used only when the increase readability, simplicity or hide tedious work. In that case the object and functor implementations are really the same but the object version is more readable and straight to understand.