Okay this one is confusing. My old piece of code has something like
map(lambda x:x.func1(arg1), other_args_to_be_mapped)
now I would like to make arg1 -> *args
while other_args_to_be_mapped stays unchanged.
in func1, the length of arguments will be checked different operations. My questions are
1) which length will be checked? arg1 or other_args_to_be_mapped
2) in func1, how should I set up the default? It was like
def func1(arg1=something)
but now with potential multiple arguments, I don't know what to do with the initialization. I want to be able to do something like
def func1(args*=something, something_else)
Is that even possible?
If I understand your question correctly, you're looking for variable arguments. These can be mixed with fixed arguments, provided you obey a logical ordering (fixed arguments first, then keyword arguments or variable arguments).
For example, the following shows how map to a function that takes in one constant argument and one variable argument. If you would like different behaviour, please provide a concrete example of what you are trying to accomplish
import random
class Foo:
def get_variable_parameters(self):
return [1] if random.random() > .5 else [1,2]
def foo( self, arg, *args ):
print("Doing stuff with constant arg", arg)
if len(args) == 1:
print("Good",args)
else:
print("Bad",args)
list(map( lambda x : x.foo( 'Static Argument', *x.get_variable_parameters()), [Foo(),Foo(),Foo()] ))
We don't know how many arguments are going to be passed to foo (in this trivial case, it's one or two), but the "*" notation accepts any number of objects to be passed
Note I've encapsulated map in list so that it gets evaluated, as in python3 it is a generator. List comprehension may be more idiomatic in python. Also don't forget you can always use a simple for loop - an obfuscated or complex map call is far less pythonic than a clear (but several line) for-loop, imo.
If, rather, you're trying to combine multiple arguments in a map call, I would recommend using the same variable argument strategy with the zip function, e.g.,
def foo(a,*b): ...
map(lambda x : foo(x[0],*x[1]), zip(['a','b'],[ [1], [1,2] ]))
In this case, foo will get called first as foo('a',1), and then as foo('b',2,3)
Related
I want to define a function using explicit argument names ff(a,b,c) in the function definition, but I also want to map a function over all arguments to get a list:
ff(a,b,c):
return list(map(myfunc,[a,b,c]))
However, I don't want to explicitly write parameter names inside function as a,b,c. I want to do it like
ff(a,b,c):
return list(map(myfunc,getArgValueList()))
getArgValueList() will retrieve the argument values in order and form a list. How to do this? Is there a built-in function like getArgValueList()?
What you're trying to do is impossible without ugly hacks. You either take *args and get a sequence of parameter values that you can use as args:
def ff(*args):
return list(map(myfunc, args))
… or you take three explicit parameters and use them by name:
def ff(a, b, c):
return list(map(myfunc, (a, b, c)))
… but it's one or the other, not both.
Of course you can put those values in a sequence yourself if you want:
def ff(a, b, c):
args = a, b, c
return list(map(myfunc, args))
… but I'm not sure what that buys you.
If you really want to know how to write a getArgValueList function anyway, I'll explain how to do it. However, if you're looking to make your code more readable, more efficient, more idiomatic, easier to understand, more concise, or almost anything else, it will have the exact opposite effect. The only reason I could imagine doing something like this is if you had to generate functions dynamically or something—and even then, I can't think of a reason you couldn't just use *args. But, if you insist:
def getArgValueList():
frame = inspect.currentframe().f_back
code = frame.f_code
vars = code.co_varnames[:code.co_argcount]
return [frame.f_locals[var] for var in vars]
If you want to know how it works, most of it's in the inspect module docs:
currentframe() gets the current frame—the frame of getArgValueList.
f_back gets the parent frame—the frame of whoever called getArgValueList.
f_code gets the code object compiled from the function body of whoever called getArgValueList.
co_varnames is a list of all local variables in that body, starting with the parameters.
co_argcount is a count of explicit positional-or-keyword parameters.
f_locals is a dict with a copy of the locals() environment of the frame.
This of course only works for a function that takes no *args, keyword-only args, or **kwargs, but you can extend it to work for them as well with a bit of work. (See co_kwonlyargcount, co_flags, CO_VARARGS, and CO_VARKEYWORDS for details.)
Also, this only works for CPython, not most other interpreters. and it could break in some future version, because it's pretty blatantly relying on implementation details of the interpreter.
The *args construction will give you the arguments as a list:
>>> def f(*args): return list(map(lambda x:x+1, args))
>>> f(1,2,3)
[2, 3, 4]
If you are bound with the signature of f, you'll have to use the inspect module:
import inspect
def f(a, b,c):
f_locals = locals()
values = [f_locals[name] for name in inspect.signature(f).parameters]
return list(map(lambda x:x+1, values))
inspect.signature(f).parameters gives you the list of arguments in the correct order. The values are in locals().
I recently attempted Googles foo.bar challenge. After my time was up I decided to try find a solution to the problem I couldn't do and found a solution here (includes the problem statement if you're interested). I'd previously been making a dictionary for every function I wanted to cache but it looks like in this solution any function/input can be cached using the same syntax.
Firstly I'm confused on how the code is even working, the *args variable isn't inputted as an argument (and prints to nothing). Heres an modified minimal example to illustrate my confusion:
mem = {}
def memoize(key, func, *args):
"""
Helper to memoize the output of a function
"""
print(args)
if key not in mem:
# store the output of the function in memory
mem[key] = func(*args)
return mem[key]
def example(n):
return memoize(
n,
lambda: longrun(n),
)
def example2(n):
return memoize(
n,
longrun(n),
)
def longrun(n):
for i in range(10000):
for j in range(100000):
2**10
return n
Here I use the same memoize function but with a print. The function example returns memoize(n, a lambda function,). The function longrun is just an identity function with lots of useless computation so it's easy to see if the cache is working (example(2) will take ~5 seconds the first time and be almost instant after).
Here are my confusions:
Why is the third argument of memoize empty? When args is printed in memoize it prints (). Yet somehow mem[key] stores func(*args) as func(key)?
Why does this behavior only work when using the lambda function (example will cache but example2 won't)? I thought lambda: longrun(n) is just a short way of giving as input a function which returns longrun(n).
As a bonus, does anyone know how you could memoize functions using a decorator?
Also I couldn't think of a more descriptive title, edits welcome. Thanks.
The notation *args stands for a variable number of positional arguments. For example, print can be used as print(1), print(1, 2), print(1, 2, 3) and so on. Similarly, **kwargs stands for a variable number of keyword arguments.
Note that the names args and kwargs are just a convention - it's the * and ** symbols that make them variadic.
Anyways, memoize uses this to accept basically any input to func. If the result of func isn't cached, it's called with the arguments. In a function call, *args is basically the reverse of *args in a function definition. For example, the following are equivalent:
# provide *args explicitly
print(1, 2, 3)
# unpack iterable to *args
arguments = 1, 2, 3
print(*arguments)
If args is empty, then calling print(*args) is the same as calling print() - no arguments are passed to it.
Functions and lambda functions are the same in python. It's simply a different notation for creating a function object.
The problem is that in example2, you are not passing a function. You call a function, then pass on its result. Instead, you have to pass on the function and its argument separately.
def example2(n):
return memoize(
n,
longrun, # no () means no call, just the function object
# all following parameters are put into *args
n
)
Now, some implementation details: why is args empty and why is there a separate key?
The empty args comes from your definition of the lambda. Let's write that as a function for clarity:
def example3(n):
def nonlambda():
return longrun(n)
return memoize(n, nonlambda)
Note how nonlambda takes no arguments. The parameter n is bound from the containing scope as a closure, bound from the containing scope. As such, you don't have to pass it to memoize - it is already bound inside the nonlambda. Thus, args is empty in memoize, even though longrun does receive a parameter, because the two don't interact directly.
Now, why is it mem[key] = f(*args), not mem[key] = f(key)? That's actually slightly the wrong question; the right question is "why isn't it mem[f, args] = f(*args)?".
Memoization works because the same input to the same function leads to the same output. That is, f, args identifies your output. Ideally, your key would be f, args as that's the only relevant information.
The problem is you need a way to look up f and args inside mem. If you ever tried putting a list inside a dict, you know there are some types which don't work in mappings (or any other suitable lookup structure, for that matter). So if you define key = f, args, you cannot memoize functions taking mutable/unhashable types. Python's functools.lru_cache actually has this limitation.
Defining an explicit key is one way of solving this problem. It has the advantage that the caller can select an appropriate key, for example taking n without any modifications. This offers the best optimization potential. However, it breaks easily - using just n misses out the actual function called. Memoizing a second function with the same input would break your cache.
There are alternative approaches, each with pros and cons. Common is the explicit conversion of types: list to tuple, set to frozenset, and so on. This is slow, but the most precise. Another approach is to just call str or repr as in key = repr((f, args, sorted(kwargs.items()))), but it relies on every value having a proper repr.
So I was playing around with currying functions in Python and one of the things that I noticed was that functools.partial returns a partial object rather than an actual function. One of the things that annoyed me about this was that if I did something along the lines of:
five = partial(len, 'hello')
five('something')
then we get
TypeError: len() takes exactly 1 argument (2 given)
but what I want to happen is
TypeError: five() takes no arguments (1 given)
Is there a clean way to make it work like this? I wrote a workaround, but it's too hacky for my taste (doesn't work yet for functions with varargs):
def mypartial(f, *args):
argcount = f.func_code.co_argcount - len(args)
params = ''.join('a' + str(i) + ',' for i in xrange(argcount))
code = '''
def func(f, args):
def %s(%s):
return f(*(args+(%s)))
return %s
''' % (f.func_name, params, params, f.func_name)
exec code in locals()
return func(f, args)
Edit: I think it might be helpful if I added more context. I'm writing a decorator that will automatically curry a function like so:
#curry
def add(a, b, c):
return a + b + c
f = add(1, 2) # f is a function
assert f(5) == 8
I want to hide the fact that f was created from a partial (maybe a bad idea :P). The message that the TypeError message above gives is one example of where whether something is a partial can be revealed. I want to change that.
This needs to be generalizable so EnricoGiampieri's and mgilson's suggestions only work in that specific case.
You definitely don't want to do this with exec.
You can find recipes for partial in pure Python, such as this one—many of them are mislabeled as curry recipes, so look for that as well. At any rate, these will show you the proper way to do it without exec, and you can just pick one and modify it to do what you want.
Or you could just wrap partial…
However, whatever you do, there's no way the wrapper can know that it's defining a function named "five"; that's just the name of the variable you store the function in. So if you want a custom name, you'll have to pass it in to the function:
five = my_partial('five', len, 'hello')
At that point, you have to wonder why this is any better than just defining a new function.
However, I don't think this is what you actually want anyway. Your ultimate goal is to define a #curry decorator that creates a curried version of the decorated function, with the same name (and docstring, arg list, etc.) as the decorated function. The whole idea of replacing the name of the intermediate partial is a red herring; use functools.wraps properly inside your curry function, and it won't matter how you define the curried function, it'll preserve the name of the original.
In some cases, functools.wraps doesn't work. And in fact, this may be one of those times—you need to modify the arg list, for example, so curry(len) can take either 0 or 1 parameter instead of requiring 1 parameter, right? See update_wrapper, and the (very simple) source code for wraps and update_wrapper to see how the basics work, and build from there.
Expanding on the previous: To curry a function, you pretty much have to return something that takes (*args) or (*args, **kw) and parse the args explicitly, and possibly raise TypeError and other appropriate exceptions explicitly. Why? Well, if foo takes 3 params, curry(foo) takes 0, 1, 2, or 3 params, and if given 0-2 params it returns a function that takes 0 through n-1 params.
The reason you might want **kw is that it allows callers to specify params by name—although then it gets much more complicated to check when you're done accumulating arguments, and arguably this is an odd thing to do with currying—it may be better to first bind the named params with partial, then curry the result and pass in all remaining params in curried style…
If foo has default-value or keyword args, it gets even more complicated, but even without those problems, you already need to deal with this problem.
For example, let's say you implement curry as a class that holds the function and all already-curried parameters as instance members. Then you'll have something like this:
def __call__(self, *args):
if len(args) + len(self.curried_args) > self.fn.func_code.co_argcount:
raise TypeError('%s() takes exactly %d arguments (%d given)' %
(self.fn.func_name, self.fn.func_code.co_argcount,
len(args) + len(self.curried_args)))
self.curried_args += args
if len(self.curried_args) == self.fn.func_code.co_argcount:
return self.fn(*self.curried_args)
else:
return self
This is horribly oversimplified, but it shows how to handle the basics.
My guess is that the partial function just delay the execution of the function, do not create a whole new function out of it.
My guess is that is just easier to define directly a new function in place:
def five(): return len('hello')
This is a very simple line, won't clutter your code and is quite clear, so i wouldn't bother writing a function to replace it, especially if you don't need this situation in a large number of cases
I have a Python script which creates a dictionary of its own functions and I'd like it to execute them by reading in a function name and arguments using YARP (knowledge of YARP is irrelevant to this question though).
I create a list of strings called "inc" which is populated by values coming into the program. The first item is a function name, and any other strings in the list are arguments. I create a dictionary called "methods" where the key is the function name and the value is a reference to the function object (using the inspect module). I store the return value of the function in a variable "result".
The snippet below shows a simplified version of what I'm using so far, which works fine, but can't handle functions with more than one argument. To circumvent this I use a list if a function needs more parameters:
if len(inc) == 1:
result = methods[inc[0]]() # call method with 0 arguments
elif len(inc) == 2:
result = methods[inc[0]](inc[1]) # call method passing a string
else:
args = []
result = methods(inc[0])(inc[1:]) # call method passing a list
Ideally, I'd like to change this so that my functions can have any number of arguments, but I can't figure out how I can do this. I'm new to Python and I have looked at the documentation and various websites - I just can't find a solution. I've tried things like creating a tuple of the arguments, but that doesn't work either as it ends up passing the whole tuple in as one parameter.
Is there a better solution to this problem, like creating some kind of object which represents a set of parameters and passing that into the function? Any suggestions would be greatly appreciated!
You should check out https://stackoverflow.com/a/3394898/1395668.
The magic you are looking for is the *. Apply this to your list and it unpacks the items into the argument fields of your function:
a = [ 1, 2, 3]
def myfunc(a, b, c):
return a + b + c
print myfunc(*a)
Check out ** for the same approach for dict
It's a bit strange to have this kind of mixed structure:
inc = [func_name, arg1, arg2, ...]
Wouldn't it be much more natural to have two separate bits of information?
func_name = ...
args = [arg1, arg2, ...]
The you could do
methods[func_name](*args)
(Usually, I wouldn't bind the functions name to a variable, but preferably the function itself.)
Suppose I have a function that I do not control that looks something like the following:
def some_func(foo, bar, bas, baz):
do_something()
return some_val
Now I want to call this function passing elements from a dict that contains keys that are identical to the arguments of this function. I could do something like:
some_func(foo=mydict['foo'],
bar=mydict['bar'],
bas=mydict['bas'],
baz=mydict['baz'])
Is there some elegant way I could take advantage of the fact that the keys match the parms to do this less verbosely? I know I could pass the whole dict, but let's say I either don't want to or can't change the function to accept a single dict rather than the individual arguments.
Thanks,
Jerry
That's what ** argument unpacking is for:
some_func(**mydict)
See also Unpacking argument lists in the Python tutorial.
As Sven notes, you can pass a dict to a function using ** unpacking. But if the dict contains keys that aren't argument names to the target function, this works only if the function you're calling accepts keyword arguments using the ** notation. If it doesn't, you'll get an error.
If the function you're calling doesn't have a **kwargs parameter, the easiest way to handle it is to add one. But if you can't do that, you have a couple approaches:
1) Write a wrapper function:
def wrapper(foo, bar, baz, quux, **kwargs):
return some_func(foo, bar, baz, quux)
wrapper(**mydict)
2) Write a function to extract just the dict keys you need:
def extract(dikt, keys):
return dict((k, dikt[k]) for k in keys.split())
some_func(**extract(mydict, "foo bar baz quux"))
3) Write a function that introspects the function you're calling and extracts the keys you need from the dictionary -- basically the same as #2 except you don't have to "repeat yourself" as much.
def call_with_dict(func, dikt):
func(**dict((k, dikt[k]) for k in
func.func_code.co_varnames[:func.func_code.co_argcount]))
call_with_dict(some_func, my_dict)
Note that this doesn't allow you to omit arguments that have default values in the function signature. You could do some additional introspection to permit that (the length of func.func_defaults determines which of the arguments have default values).
This introspection is for Python 2.x, Python 3 and later will probably need some tweaking. For this reason, I prefer one of the first two methods.
PS -- Thanks to Sven for catching my missing ** in my second approach.