I would like to be able to wrap a function, while still allowing optional (keyword) arguments in the outer function. I can do this:
def outer(p1, p2, *a, **k):
inner(*a, **k)
but what if I want for p2 to be an "optional argument"? Obviously:
def outer(p1, p2=None, *a, **k):
inner(*a, **k)
won't work, and I know why it won't work. I'm wondering if there's a nice, clean way to achieve this. Is there some well-known pattern or convention?
Use the functools module:
outer2 = functools.partial(outer, p2=None)
If you just want to ensure that p2 is present when the arguments are passed from outer to inner:
def outer(p1, *a, **k):
if 'p2' not in k: k.update(p2 = None)
inner(*a, **k)
for a more general way you can keep a dict of defaults and apply them when needed:
def outer (p1, *args, **kwargs):
defaults = {'a': 1, 'b':2}
for k, v in defaults.items():
if k not in kwargs: kwargs[k] = v
inner(*a, **k)
If that's the case you might want to edit the title of your post to something along the lines of 'how to include default values in **kwargs arguments' since the wrapping is not really part of the syntax
Related
I want to use a bunch of local variables defined in a function, outside of the function. So I am passing x=locals() in the return value.
How can I load all the variables defined in that dictionary into the namespace outside the function, so that instead of accessing the value using x['variable'], I could simply use variable.
Rather than create your own object, you can use argparse.Namespace:
from argparse import Namespace
ns = Namespace(**mydict)
To do the inverse:
mydict = vars(ns)
Consider the Bunch alternative:
class Bunch(object):
def __init__(self, adict):
self.__dict__.update(adict)
so if you have a dictionary d and want to access (read) its values with the syntax x.foo instead of the clumsier d['foo'], just do
x = Bunch(d)
this works both inside and outside functions -- and it's enormously cleaner and safer than injecting d into globals()! Remember the last line from the Zen of Python...:
>>> import this
The Zen of Python, by Tim Peters
...
Namespaces are one honking great idea -- let's do more of those!
This is perfectly valid case to import variables in
one local space into another local space as long as
one is aware of what he/she is doing.
I have seen such code many times being used in useful ways.
Just need to be careful not to pollute common global space.
You can do the following:
adict = { 'x' : 'I am x', 'y' : ' I am y' }
locals().update(adict)
blah(x)
blah(y)
Importing variables into a local namespace is a valid problem and often utilized in templating frameworks.
Return all local variables from a function:
return locals()
Then import as follows:
r = fce()
for key in r.keys():
exec(key + " = r['" + key + "']")
The Bunch answer is ok but lacks recursion and proper __repr__ and __eq__ builtins to simulate what you can already do with a dict. Also the key to recursion is not only to recurse on dicts but also on lists, so that dicts inside lists are also converted.
These two options I hope will cover your needs (you might have to adjust the type checks in __elt() for more complex objects; these were tested mainly on json imports so very simple core types).
The Bunch approach (as per previous answer) - object takes a dict and converts it recursively. repr(obj) will return Bunch({...}) that can be re-interpreted into an equivalent object.
class Bunch(object):
def __init__(self, adict):
"""Create a namespace object from a dict, recursively"""
self.__dict__.update({k: self.__elt(v) for k, v in adict.items()})
def __elt(self, elt):
"""Recurse into elt to create leaf namespace objects"""
if type(elt) is dict:
return type(self)(elt)
if type(elt) in (list, tuple):
return [self.__elt(i) for i in elt]
return elt
def __repr__(self):
"""Return repr(self)."""
return "%s(%s)" % (type(self).__name__, repr(self.__dict__))
def __eq__(self, other):
if hasattr(other, '__dict__'):
return self.__dict__ == other.__dict__
return NotImplemented
# Use this to allow comparing with dicts:
#return self.__dict__ == (other.__dict__ if hasattr(other, '__dict__') else other)
The SimpleNamespace approach - since types.SimpleNamespace already implements __repr__ and __eq__, all you need is to implement a recursive __init__ method:
import types
class RecursiveNamespace(types.SimpleNamespace):
# def __init__(self, /, **kwargs): # better, but Python 3.8+
def __init__(self, **kwargs):
"""Create a SimpleNamespace recursively"""
self.__dict__.update({k: self.__elt(v) for k, v in kwargs.items()})
def __elt(self, elt):
"""Recurse into elt to create leaf namespace objects"""
if type(elt) is dict:
return type(self)(**elt)
if type(elt) in (list, tuple):
return [self.__elt(i) for i in elt]
return elt
# Optional, allow comparison with dicts:
#def __eq__(self, other):
# return self.__dict__ == (other.__dict__ if hasattr(other, '__dict__') else other)
The RecursiveNamespace class takes keyword arguments, which can of course come from a de-referenced dict (ex **mydict)
Now let's put them to the test (argparse.Namespace added for comparison, although it's nested dict is manually converted):
from argparse import Namespace
from itertools import combinations
adict = {'foo': 'bar', 'baz': [{'aaa': 'bbb', 'ccc': 'ddd'}]}
a = Bunch(adict)
b = RecursiveNamespace(**adict)
c = Namespace(**adict)
c.baz[0] = Namespace(**c.baz[0])
for n in ['a', 'b', 'c']:
print(f'{n}:', str(globals()[n]))
for na, nb in combinations(['a', 'b', 'c'], 2):
print(f'{na} == {nb}:', str(globals()[na] == globals()[nb]))
The result is:
a: Bunch({'foo': 'bar', 'baz': [Bunch({'aaa': 'bbb', 'ccc': 'ddd'})]})
b: RecursiveNamespace(foo='bar', baz=[RecursiveNamespace(aaa='bbb', ccc='ddd')])
c: Namespace(foo='bar', baz=[Namespace(aaa='bbb', ccc='ddd')])
a == b: True
a == c: True
b == c: False
Although those are different classes, because they both (a and b) have been initialized to equivalent namespaces and their __eq__ method compares the namespace only (self.__dict__), comparing two namespace objects returns True. For the case of comparing with argparse.Namespace, for some reason only Bunch works and I'm unsure why (please comment if you know, I haven't looked much further as types.SimpleNameSpace is a built-in implementation).
You might also notice that I recurse using type(self)(...) rather than using the class name - this has two advantages: first the class can be renamed without having to update recursive calls, and second if the class is subclassed we'll be recursing using the subclass name. It's also the name used in __repr__ (type(self).__name__).
EDIT 2021-11-27:
Modified the Bunch.__eq__ method to make it safe against type mismatch.
Added/modified optional __eq__ methods (commented out) to allow comparing with the original dict and argparse.Namespace(**dict) (note that the later is not recursive but would still be comparable with other classes as the sublevel structs would compare fine anyway).
Used following snippet (PY2) to make recursive namespace from my dict(yaml) configs:
class NameSpace(object):
def __setattr__(self, key, value):
raise AttributeError('Please don\'t modify config dict')
def dump_to_namespace(ns, d):
for k, v in d.iteritems():
if isinstance(v, dict):
leaf_ns = NameSpace()
ns.__dict__[k] = leaf_ns
dump_to_namespace(leaf_ns, v)
else:
ns.__dict__[k] = v
config = NameSpace()
dump_to_namespace(config, config_dict)
There's Always this option, I don't know that it is the best method out there, but it sure does work. Assuming type(x) = dict
for key, val in x.items(): # unpack the keys from the dictionary to individual variables
exec (key + '=val')
Let's say we have a function that takes some arguments and a dict that is a superset of values required to invoke the function:
d = {"a": 1, "b": 2, "c": 3}
def foo(a, b):
print(a, b)
# this won't work
foo(**d)
# this works but is tedious the more values there are
foo(**{k: v for k,v in d.items() if k not in ("c")})
Is there a more elegant solution?
Instead of doing this:
def foo(a, b):
print(a, b)
You could do this:
def foo(a, b, **kwargs):
print(a, b)
Then the function would just ignore all the unneeded arguments.
However, in my view, there are some problems with this approach.
Sometimes you don't own the source code of the function. Of course, you can make lots of wrapper functions. If you do that manually, you'll have to change the signatures of your helper functions every time the library authors modify the function signature.
This solution can make some bugs harder to find. Maybe it's better to explicitly state that you're only using some of the elements from the dictionary.
You can still make it easier using the inspect module of the standard library.
a) Make a decorator that makes the function filter its keyword arguments.
For instance:
import inspect
def filter_kwargs(f):
arg_names = inspect.getargspec(f).args
def _f(*args, **kwargs):
return f(*args, **{k: v for k, v in kwargs.items() if k in arg_names})
return _f
# Use it like this:
filter_kwargs(foo)(a_dict)
b) You can create a function that transforms the argument to fit the function
def fit_for(f, kwargs):
arg_names = inspect.getargspec(f).args
return {k: v for k, v in kwargs.items() if k in arg_names}
foo(fit_for(foo, a_dict))
Optionally, you can expand this function to also take into account the *args and **kwargs in the original function.
If this is a recurring pattern in your code, you can just pass the dict as a single argument.
Also, as #chepner pointed out in the comments, consider creating a class.
you could use keyword arguments:
def foo(**kwarg):
print(kwarg['a'], kwarg['b'])
foo(**d)
output:
1 2
Assuming you do not want to modify foo's function signature, you can use foo.__code__.co_varnames to get the argument names of foo if you are using cpython
foo(**{k: d[k] for k in foo.__code__.co_varnames})
You can do something like:
d = {"a": 1, "b": 2, "c": 3}
def foo(a, b, *arg, **kwarg):
print(a, b)
foo(**d)
Let us say I have a function like this:
def helloWorld(**args):
for arg in args:
print(args[arg])
To call this function is easy:
helloWorld(myMsg = 'hello, world')
helloWorld(anotherMessages = 'no, no this is hard')
But the hard part is that I want to dynamically name the args from variables coming from a list or somewhere else. And I'd want for myMsg and anotherMessages to be passed from a list (for example, the hard part that I am clueless and I need help with is how to take strings into variables to be inputs of a function).
list_of_variable_names = ['myMsg','anotherMessages']
for name in list_of_variable_names:
helloWorld(name = 'ooops, this is not easy, how do I pass a variable name that is stored as a string in a list? No idea! help!')
You can create a dict using the variable and then unpack while passing it to the function:
list_of_variable_names = ['myMsg','anotherMessages']
for name in list_of_variable_names:
helloWorld(**{name: '...'})
The ** syntax is a dictionary unpacking.
So in your function, args (which is usually kwargs) is a dictionary.
Therefore, you need to pass an unpacked dictionary to it, which is what is done when f(a=1, b=2) is called.
For instance:
kwargs = {'myMsg': "hello, world", 'anotherMessages': "no, no this is hard"}
helloWorld(**kwargs)
Then, you will get kwargs as a dictionary.
def f(**kwargs):
for k, v in kwargs.items():
print(k, v)
>>> kwargs = {'a': 1, 'b': 2}
>>> f(**kwargs)
a 1
b 2
If you want to do so, you can call the function once for every name as well, by creating a dictionary on the fly and unpacking it, as Moses suggested.
def f(**kwargs):
print("call to f")
for k, v in kwargs.items():
print(k, v)
>>> for k, v in {'a': 1, 'b': 2}:
... kwargs = {k: v}
... f(**kwargs)
...
call to f
a 1
call to f
b 2
How can one define the initial contents of the keyword arguments dictionary? Here's an ugly solution:
def f(**kwargs):
ps = {'c': 2}
ps.update(kwargs)
print(str(ps))
Expected behaviour:
f(a=1, b=2) => {'c': 2, 'a': 1, 'b': 2}
f(a=1, b=2, c=3) => {'c': 3, 'a': 1, 'b': 2}
Yet, I would like to do something a little bit more in the lines of:
def f(**kwargs = {'c': 2}):
print(str(kwargs))
Or even:
def f(c=2, **kwargs):
print(str(???))
Ideas?
First to address the issues with your current solution:
def f(**kwargs):
ps = {'c': 2}
ps.update(kwargs)
print(str(ps))
This creates a new dictionary and then has to take the time to update it with all the values from kwargs. If kwargs is large that can be fairly inefficient and as you pointed out is a bit ugly.
Obviously your second isn't valid.
As for the third option, an implementation of that was already given by Austin Hastings
If you are using kwargs and not keyword arguments for default values there's probably a reason (or at least there should be; for example an interface that defines c without explicitly requiring a and b might not be desired even though the implementation may require a value for c).
A simple implementation would take advantage of dict.setdefault to update the values of kwargs if and only if the key is not already present:
def f(**kwargs):
kwargs.setdefault('c', 2)
print(str(kwargs))
Now as mentioned by the OP in a previous comment, the list of default values may be quite long, in that case you can have a loop set the default values:
def f(**kwargs):
defaults = {
'c': 2,
...
}
for k, v in defaults.items():
kwargs.setdefault(k, v)
print(str(kwargs))
A couple of performance issues here as well. First the defaults dict literal gets created on every call of the function. This can be improved upon by moving the defaults outside of the function like so:
DEFAULTS = {
'c': 2,
...
}
def f(**kwargs):
for k, v in DEFAULTS.items():
kwargs.setdefault(k, v)
print(str(kwargs))
Secondly in Python 2, dict.items returns a copy of the (key, value) pairs so instead dict.iteritems or dict.viewitems allows you to iterate over the contents and thus is more efficient. In Python 3, 'dict.items` is a view so there's no issue there.
DEFAULTS = {
'c': 2,
...
}
def f(**kwargs):
for k, v in DEFAULTS.iteritems(): # Python 2 optimization
kwargs.setdefault(k, v)
print(str(kwargs))
If efficiency and compatibility are both concerns, you can use the six library for compatibility as follows:
from six import iteritems
DEFAULTS = {
'c': 2,
...
}
def f(**kwargs):
for k, v in iteritems(DEFAULTS):
kwargs.setdefault(k, v)
print(str(kwargs))
Additionally, on every iteration of the for loop, a lookup of the setdefault method of kwargs needs to be performed. If you truly have a really large number of default values a micro-optimization is to assign the method to a variable to avoid repeated lookup:
from six import iteritems
DEFAULTS = {
'c': 2,
...
}
def f(**kwargs):
setdefault = kwargs.setdefault
for k, v in iteritems(DEFAULTS):
setdefault(k, v)
print(str(kwargs))
Lastly if the number of default values is instead expected to be larger than the number of kwargs, it would likely be more efficient to update the default with the kwargs. To do this, you can't use the global default or it would update the defaults with every run of the function, so the defaults need to be moved back into the function. This would leave us with the following:
def f(**kwargs):
defaults = {
'c': 2,
...
}
defaults.update(kwargs)
print(str(defaults))
Enjoy :D
A variation possible on your first approach on Python 3.5+ is to define the default and expand the provided arguments on a single line, which also lets you replace kwargs on the same line, e.g.:
def f(**kwargs):
# Start with defaults, then expand kwargs which will overwrite defaults if
# it has them
kwargs = {'c': 2, **kwargs}
print(str(kwargs))
Another approach (that won't produce an identical string) that creates a mapping that behaves the same way using collections.ChainMap (3.3+):
from collections import ChainMap
def f(**kwargs):
# Chain overrides over defaults
# {'c': 2} could be defined outside f to avoid recreating it each call
ps = ChainMap(kwargs, {'c': 2})
print(str(ps))
Like I said, that won't produce the same string output, but unlike the other solutions, it won't become more and more costly as the number of passed keyword arguments increases (it doesn't have to copy them at all).
Have you tried:
def f(c=2, **kwargs):
kwargs['c'] = c
print(kwargs)
Update
Barring that, you can use inspect to access the code object, and get the keyword-only args from that, or even all the args:
import inspect
def f(a,b,c=2,*,d=1,**kwargs):
code_obj = inspect.currentframe().f_code
nposargs = code_obj.co_argcount
nkwargs = code_obj.co_kwonlyargcount
localvars = locals()
kwargs.update({k:localvars[k] for k in code_obj.co_varnames[:nposargs+nkwargs]})
print(kwargs)
g=f
g('a', 'b')
I have, for example, 3 functions, with required arguments (some arguments are shared by the functions, in different order):
def function_one(a,b,c,d,e,f):
value = a*b/c ...
return value
def function_two(b,c,e):
value = b/e ..
return value
def function_three(f,a,c,d):
value = a*f ...
return value
If I have the next dictionary:
argument_dict = {'a':3,'b':3,'c':23,'d':6,'e':1,'f':8}
Is posible to call the functions in this way??:
value_one = function_one(**argument_dict)
value_two = function_two (**argument_dict)
value_three = function_three (**argument_dict)
Not the way you have written those functions, no: they are not expecting the extra arguments so will raise a TypeError.
If you define all the functions as also expecting **kwargs, things will work as you want.
I assume what you're trying to do is to create a function with an undefined number of arguments. You can do this by using args (arguments) or kwargs (key word arguments kind of foo='bar') style so for example:
for arguments
def f(*args): print(args)
f(1, 2, 3)
(1, 2, 3)`
then for kwargs
def f2(**kwargs): print(kwargs)
f2(a=1, b=3)
{'a': 1, 'b': 3}
Let's try a couple more things.
def f(my_dict): print (my_dict['a'])
f(dict(a=1, b=3, c=4))
1
It works!!! so, you could do it that way and complement it with kwargs if you don't know what else the function could receive.
Of course you could do:
argument_dict = {'a':1, 'b':3, 'c':4}
f(argument_dict)
1
So you don't have to use kwargs and args all the time. It all depends the level of abstraction of the object you're passing to the function. In your case, you're passing a dictionary so you can handle that guy without only.