Suppose I have a function that takes two arguments and performs some calculation on them:
def add(a, b):
return a + b
I want to call this function through a multiprocessing library which can only handle functions with a single argument. So, I change the function to take its argument as a single tuple instead:
def add2(ab):
a, b = ab
return a + b
However, this seems clunky to me. The variables essentially need to be defined (and documented) twice. If I were using a lambda function, I could just write the following and it will accept the tuple properly:
add3 = lambda (a, b): a + b
Unfortunately, my function is not trivial enough to implement as a lambda function. Is there any sort of syntactic sugar feature in python that would allow me to write a named function that accepts a tuple but treats each component of that tuple as a separate named argument? My attempts to search for solutions to this have mostly turned up references to the *args operator, but that does not apply here because I do not control the site where the function is called.
Here is an example of how the function is being called. Note that it is called via the multiprocessing library so I cannot pass more than one argument:
import multiprocessing
pool = multiprocessing.Pool(processes=4)
for result in pool.imap_unordered(add, [(1,2),(3,4)]):
print(result)
Answers for either python 2.7 or 3.x are welcome.
It's best not to alter the original function interface, making it less Pythonic.
In Python 2, write a wrapper function to use with multiprocessing.
def _add(args):
return add(*args)
In Python 3, just use Pool.starmap instead:
>>> def add(a, b):
... return a + b
...
>>> p = Pool()
>>> list(p.starmap(add, [(1, 2), ('hello', ' world')]))
[3, 'hello world']
If you are worried about repeating yourself (a and b appear too many times), simply give the incoming tuple a non-descriptive name.
def add(t):
a, b = t
return a + b
Or, in your specific case, you can avoid a and b altogether by indexing the tuple:
def add(addends):
return addends[0] + addends[1]
As an alternative, you could wrap your function so the source code has the familiar argument format, but the function in use has the tuple argument:
def tupleize(func):
def wrapper(tup):
return func(*tup)
return wrapper
#tupleize
def add(a, b):
return a+b
t = 1, 2
assert(add(t) == 3)
As I was writing this question, I found the way to do it in Python 2.7:
def add4((a, b)):
return a + b
However apparently this no longer works in Python 3, so additional answers regarding Python 3 would still be helpful.
You could use a decorator to extend the multiprocessing library function to take multiple arguments, do whatever you want to them, and then call it with a single argument.
For example, a simple decorator that takes any number of arguments, sums them together, then calls the original function with the total as a single argument:
import 3rdpartylib
def sum_args(func):
def inner(*args):
return func(sum(args))
return inner
# Replace imported function with decorated version
3rdpartylib = sum_args(3rdpartylib)
# Decorate your own libraries
#sum_args
def my_own_lib(number):
print("A:", number)
3rdpartylib(1,2,3,4)
my_own_lib(5,10,15)
The main advantage is that you can decorate/replace any number of methods with this same decorator function to achieve the same effect.
Related
In Python we can assign a function to a variable. For example, the math.sine function:
sin = math.sin
rad = math.radians
print sin(rad(my_number_in_degrees))
Is there any easy way of assigning multiple functions (ie, a function of a function) to a variable? For example:
sin = math.sin(math.radians) # I cannot use this with brackets
print sin (my_number_in_degrees)
Just create a wrapper function:
def sin_rad(degrees):
return math.sin(math.radians(degrees))
Call your wrapper function as normal:
print sin_rad(my_number_in_degrees)
I think what the author wants is some form of functional chaining. In general, this is difficult, but may be possible for functions that
take a single argument,
return a single value,
the return values for the previous function in the list is of the same type as that of the input type of the next function is the list
Let us say that there is a list of functions that we need to chain, off of which take a single argument, and return a single argument. Also, the types are consistent. Something like this ...
functions = [np.sin, np.cos, np.abs]
Would it be possible to write a general function that chains all of these together? Well, we can use reduce although, Guido doesn't particularly like the map, reduce implementations and was about to take them out ...
Something like this ...
>>> reduce(lambda m, n: n(m), functions, 3)
0.99005908575986534
Now how do we create a function that does this? Well, just create a function that takes a value and returns a function:
import numpy as np
def chainFunctions(functions):
def innerFunction(y):
return reduce(lambda m, n: n(m), functions, y)
return innerFunction
if __name__ == '__main__':
functions = [np.sin, np.cos, np.abs]
ch = chainFunctions( functions )
print ch(3)
You could write a helper function to perform the function composition for you and use it to create the kind of variable you want. Some nice features are that it can combine a variable number of functions together that each accept a variable number of arguments.
import math
try:
reduce
except NameError: # Python 3
from functools import reduce
def compose(*funcs):
""" Compose a group of functions (f(g(h(...)))) into a single composite func. """
return reduce(lambda f, g: lambda *args, **kwargs: f(g(*args, **kwargs)), funcs)
sindeg = compose(math.sin, math.radians)
print(sindeg(90)) # -> 1.0
I got the following code:
g = lambda x: x+7
foo = lambda f: (lambda x: f(x+1)*2)
print( g(3), (foo(g))(3), (foo(foo(g))((3) )
Could I get an explanation on how (foo(foo(g))((3) works?
The first thing to remember is that lambdas are regular functions that:
Don't automatically have names
Can be used as expressions
Must consist of a single expression
Implicitly return the results of that expression
So you can always rewrite them as normal def functions with names if you're confused. For example, foo can become:
def foo(f):
def foo_inner(x):
return f(x + 1) * 2
return foo_inner
So calling foo with any function (f) returns a new function which takes a numeric type, adds one to it, calls f with the value, and doubles the result.
All the rest of it is just tracing the multiple layers of wrapping here, which I'll leave to you; this isn't an interesting problem in general. In real code that uses factory functions like this, the intent and behavior is generally much more clear (because it's being done for a purpose, rather than as a brainteaser).
def cons(a, b):
def pair(f):
return f(a, b)
return pair
def car(f):
def left(a, b):
return a
return f(left)
def cdr(f):
def right(a, b):
return b
return f(right)
Found this python code on git.
Just want to know what is f(a,b) in cons definition is, and how does it work?
(Not a function I guess)
cons is a function, that takes two arguments, and returns a function that takes another function, which will consume these two arguments.
For example, consider the following function:
def add(a, b):
return a + b
This is just a function that adds the two inputs, so, for instance, add(2, 5) == 7
As this function takes two arguments, we can use cons to call this function:
func_caller = cons(2, 5) # cons receives two arguments and returns a function, which we call func_caller
result = func_caller(add) # func_caller receives a function, that will process these two arguments
print(result) # result is the actual result of doing add(2, 5), i.e. 7
This technique is useful for wrapping functions and executing stuff, before and after calling the appropriate functions.
For example, we can modify our cons function to actually print the values before and after calling add:
def add(a, b):
print('Adding {} and {}'.format(a, b))
return a + b
def cons(a, b):
print('Received arguments {} and {}'.format(a, b))
def pair(f):
print('Calling {} with {} and {}'.format(f, a, b))
result = f(a, b)
print('Got {}'.format(result))
return result
return pair
With this update, we get the following outputs:
func_caller = cons(2, 5)
# prints "Received arguments 2 and 5" from inside cons
result = func_caller(add)
# prints "Calling add with 2 and 5" from inside pair
# prints "Adding 2 and 5" from inside add
# prints "Got 7" from inside pair
This isn't going to make any sense to you until you know what cons, car, and cdr mean.
In Lisp, lists are stored as a very simple form of linked list. A list is either nil (like None) for an empty list, or it's a pair of a value and another list. The cons function takes a value and a list and returns you another list just by making a pair:
def cons(head, rest):
return (head, rest)
And the car and cdr functions (they stand for "Contents of Address|Data Register", because those are the assembly language instructions used to implement them on a particular 1950s computer, but that isn't very helpful) return the first or second value from a pair:
def car(lst):
return lst[0]
def cdr(lst):
return lst[1]
So, you can make a list:
lst = cons(1, cons(2, cons(3, None)))
… and you can get the second value from it:
print(car(cdr(lst))
… and you can even write functions to get the nth value:
def nth(lst, n):
if n == 0:
return car(lst)
return nth(cdr(lst), n-1)
… or print out the whole list:
def printlist(lst):
if lst:
print(car(lst), end=' ')
printlist(cdr(lst))
If you understand how these work, the next step is to try them on those weird definitions you found.
They still do the same thing. So, the question is: How? And the bigger question is: What's the point?
Well, there's no practical point to using these weird functions; the real point is to show you that everything in computer science can be written with just functions, no built-in data structures like tuples (or even integers; that just takes a different trick).
The key is higher-order functions: functions that take functions as values and/or return other functions. You actually use these all the time: map, sort with a key, decorators, partial… they’re only confusing when they’re really simple:
def car(f):
def left(a, b):
return a
return f(left)
This takes a function, and calls it on a function that returns the first of its two arguments.
And cdr is similar.
It's hard to see how you'd use either of these, until you see cons:
def cons(a, b):
def pair(f):
return f(a, b)
return pair
This takes two things and returns a function that takes another function and applies it to those two things.
So, what do we get from cons(3, None)? We get a function that takes a function, and applies it to the arguments 3 and None:
def pair3(f):
return f(3, None)
And if we call cons(2, cons(3, None))?
def pair23(f):
return f(2, pair3)
And what happens if you call car on that function? Trace through it:
def left(a, b):
return a
return pair23(left)
That pair23(left) does this:
return left(2, pair3)
And left is dead simple:
return 2
So, we got the first element of (2, cons(3, None)).
What if you call cdr?
def right(a, b):
return a
return pair23(right)
That pair23(right) does this:
return right(2, pair3)
… and right is dead simple, so it just returns pair3.
You can work out that if we call car(cdr(pair23)), we're going to get the 3 out of it.
And now you can write lst = cons(1, cons(2, cons(3, None))), write the recursive nth and printlist functions above, and trace through how they work on lst.
I mentioned above that you can even get rid of integers. How do you do that? Read about Church numerals. You define zero and successor functions. Then you can define one as successor(zero) and two as successor(one). You can even recursively define add so that add(x, zero) is x but add(x, successor(y)) is successor(add(x, y)), and go on to define mul, etc.
You also need a special function you can use as a value for nil.
Anyway, once you've done that, using all of the other definitions above, you can do lst = cons(zero(cons(one, cons(two, cons(three, nil)))), and nth(lst, two) will give you back one. (Of course writing printlist will be a bit trickier…)
Obviously, this is all going to be a lot slower than just using tuples and integers and so on. But theoretically, it’s interesting.
Consider this: we could write a tiny dialect of Python that has only three kinds of statements—def, return, and expression statements—and only three kinds of expressions—literals, identifiers, and function calls—and it could do everything normal Python does. (In fact, you could get rid of statements altogether just by having a function-defining expression, which Python already has.) That tiny language would be a pain to use, but it would a lot easier to write a program to reason about programs in that tiny language. And we even know how to translate code using tuples, loops, etc. into code in this tiny subset language, which means we can write a program that reasons about that real Python code.
In fact, with a couple more tricks (curried functions and/or static function types, and lazy evaluation), the compiler/interpreter could do that kind of reasoning on the fly and optimize our code for us. It’s easy to tell programmatically that car(cdr(cons(2, cons(3, None)) is going to return 3 without having to actually evaluate most of those function calls, so we can just skip evaluating them and substitute 3 for the whole expression.
Of course this breaks down if any function can have side effects. You obviously can’t just substitute None for print(3) and get the same results. So instead, you need some clever trick where IO is handled by some magic object that evaluates functions to figure out what it should read and write, and then the whole rest of the program, the part that users write, becomes pure and can be optimized however you want. With a couple more abstractions, we can even make IO something that doesn’t have to be magical to do that.
And then you can build a standard library that gives you back all those things we gave up, written in terms of defining and calling functions, so it’s actually usable—but under the covers it’s all just reducing pure function calls, which is simple enough for a computer to optimize. And then you’ve basically written Haskell.
(disclaimed: not a Python kid, so please be gentle)
I am trying to compose functions using the following:
def compose(*functions):
return functools.reduce(lambda acc, f: lambda x: acc(f(x)), functions, lambda x: x)
which works as expected for scalar functions. I'd like to work with functions returning tuples and others taking multiple arguments, eg.
def dummy(name):
return (name, len(name), name.upper())
def transform(name, size, upper):
return (upper, -size, name)
# What I want to achieve using composition,
# ie. f = compose(transform, dummy)
transform(*dummy('Australia'))
=> ('AUSTRALIA', -9, 'Australia')
Since dummy returns a tuple and transform takes three arguments, I need to unpack the value.
How can I achieve this using my compose function above? If I try like this, I get:
f = compose(transform, dummy)
f('Australia')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in <lambda>
File "<stdin>", line 2, in <lambda>
TypeError: transform() takes exactly 3 arguments (1 given)
Is there a way to change compose such that it will unpack where needed?
This one works for your example but it wont handle just any arbitrary function - it will only works with positional arguments and (of course) the signature of any function must match the return value of the previous (wrt/ application order) one.
def compose(*functions):
return functools.reduce(
lambda f, g: lambda *args: f(*g(*args)),
functions,
lambda *args: args
)
Note that using reduce here, while certainly idiomatic in functional programming, is rather unpythonic. The "obvious" pythonic implementation would use iteration instead:
def itercompose(*functions):
def composed(*args):
for func in reversed(functions):
args = func(*args)
return args
return composed
Edit:
You ask "Is there a way to make have a compose function which will work in both cases" - "both cases" here meaning wether the functions returns an iterable or not (what you call "scalar functions", a concept that has no meaning in Python).
Using the iteration-based implementation, you could just test if the return value is iterable and wrap it in a tuple ie:
import collections
def itercompose(*functions):
def composed(*args):
for func in reversed(functions):
if not isinstance(args, collections.Iterable):
args = (args,)
args = func(*args)
return args
return composed
but this is not garanteed to work as expected - actually this is even garanteed to NOT work as expected for most use cases. There are a lot of builtin iterable types in Python (and even more user-defined ones) and just knowing an object is iterable doesn't say much about it's semantic.
For example a dict or str are iterable but in this case should obviously be considered a "scalar". A list is iterable too, and how it should be interpreted in this case is actually just undecidable without knowing exactly what it contains and what the "next" function in composition order expects - in some cases you will want to treat it as a single argument, in other cases ase a list of args.
IOW only the caller of the compose() function can really tell how each function result should be considered - actually you might even have cases where you want a tuple to be considered as a "scalar" value by the next function. So to make a long story short: no, there's no one-size-fits-all generic solution in Python. The best I could think of requires a combination of result inspection and manual wrapping of composed functions so the result is properly interpreted by the "composed" function but at this point manually composing the functions will be both way simpler and much more robust.
FWIW remember that Python is first and mostly a dynamically typed object oriented language so while it does have a decent support for functional programming idioms it's obviously not the best tool for real functional programming.
You might consider inserting a "function" (really, a class constructor) in your compose chain to signal the unpacking of the prior/inner function's results. You would then adjust your composer function to check for that class to determine if the prior result should be unpacked. (You actually end up doing the reverse: tuple-wrap all function results except those signaled to be unpacked -- and then have the composer unpack everything.) It adds overhead, it's not at all Pythonic, it's written in a terse lambda style, but it does accomplish the goal of being able to properly signal in a function chain when the composer should unpack a result. Consider the following generic code, which you can then adapt to your specific composition chain:
from functools import reduce
from operator import add
class upk: #class constructor signals composer to unpack prior result
def __init__(s,r): s.r = r #hold function's return for wrapper function
idt = lambda x: x #identity
wrp = lambda x: x.r if isinstance(x, upk) else (x,) #wrap all but unpackables
com = lambda *fs: ( #unpackable compose, unpacking whenever upk is encountered
reduce(lambda a,f: lambda *x: a(*wrp(f(*x))), fs, idt) )
foo = com(add, upk, divmod) #upk signals divmod's results should be unpacked
print(foo(6,4))
This circumvents the problem, as called out by prior answers/comments, of requiring your composer to guess which types of iterables should be unpacked. Of course, the cost is that you must explicitly insert upk into the callable chain whenever unpacking is required. In that sense, it is by no means "automatic", but it is still a fairly simple/terse way of achieving the intended result while avoiding unintended wraps/unwraps in many corner cases.
The compose function in the answer contributed by Bruno did do the job for functions with multiple arguments but didn't work any more for scalar ones unfortunately.
Using the fact that Python `unpacks' tuples into positional arguments, this is how I solved it:
import functools
def compose(*functions):
def pack(x): return x if type(x) is tuple else (x,)
return functools.reduce(
lambda acc, f: lambda *y: f(*pack(acc(*pack(y)))), reversed(functions), lambda *x: x)
which now works just as expected, eg.
#########################
# scalar-valued functions
#########################
def a(x): return x + 1
def b(x): return -x
# explicit
> a(b(b(a(15))))
# => 17
# compose
> compose(a, b, b, a)(15)
=> 17
########################
# tuple-valued functions
########################
def dummy(x):
return (x.upper(), len(x), x)
def trans(a, b, c):
return (b, c, a)
# explicit
> trans(*dummy('Australia'))
# => ('AUSTRALIA', 9, 'Australia')
# compose
> compose(trans, dummy)('Australia')
# => ('AUSTRALIA', 9, 'Australia')
And this also works with multiple arguments:
def add(x, y): return x + y
# explicit
> b(a(add(5, 3)))
=> -9
# compose
> compose(b, a, add)(5, 3)
=> -9
I currently have the following structure:
Inside a class I need to handle several types of functions with two special variables and an arbitrary number of parameters. To wrap these for the methods I apply them on I scan the function signatures first (that works very reliable) and decide what the parameters and what my variables are.
I then bind them back with a lambda expression in the following way. Let func(x, *args) be my function, then I'll bind
f = lambda x, t: func(x=x, **func_parameter)
In the case that I get func(x, t, *args) I bind
f = lambda x, t: func(x=x, t=t, **func_parameter)
and similar if I have neither variables.
It is essential that I hand a function of the form f(x,t) to my methods inside that class.
I would like to use functools.partial for that - it is the more pythonic way to do it and the performance when executing is better (the function f is potentially called a couple of million times...). The problem that I have is that I don't know what to do if I have a basis function which is independent of one of the variables t and x, that's why I went with lambda functions at all, they just map the other variable 'blind'. It's still two function calls and while definitions with lambda and partial take the same time, execution is a lot faster with partial.
Does anyone knoe how to use partial in that case? Performance is kind of an issue here.
EDIT: A little later. I figured out that function evaluation with tuple arguments are faster than with keyword arguments, so that was a plus.
And then, in the end, as a user I would just take some of the guess work from Python, i.e. directly define
def func(x):
return 2*x
instead of
def func(x, a):
return a*x
And call it directly. In that way I can use the function directly. Second case would be if I implement the case where x and t are both present as partial mapping.
That might be a compromise.
You could write adapter classes that have an f(x,t) call signature. The result is similar to functools.partial but much more flexible. __call__ gives you a consistent call signature and lets you add, drop, and map parameters. Arguments can be fixed when an instance is made. It seems like it should execute as fast as a normal function, but I have no basis for that.
A toy version:
class Adapt:
'''return a function with call signature f(x,t)'''
def __init__(self, func, **kwargs):
self.func = func
self.kwargs = kwargs
def __call__(self, x, t):
# mapping magic goes here
return self.func(x, t, **self.kwargs)
#return self.func(a=x, b=t, **self.kwargs)
def f(a, b, c):
print(a, b, c)
Usage:
>>> f_xt = Adapt(f, c = 4)
>>> f_xt(3, 4)
3 4 4
>>>
Don't know how you could make that generic for arbitrary parameters and mappings, maybe someone will chime in with an idea or an edit.
So if you end up writing an adapter specific to each function, the function can be embedded in the class instead of an instance parameter.
class AdaptF:
'''return a function with call signature f(x,t)'''
def __init__(self, **kwargs):
self.kwargs = kwargs
def __call__(self, x, t):
'''does stuff with x and t'''
# mapping magic goes here
return self.func(a=x, b=t, **self.kwargs)
def func(self, a, b, c):
print(a, b, c)
>>> f_xt = AdaptF(c = 4)
>>> f_xt(x = 3, t = 4)
3 4 4
>>>
I just kinda made this up from stuff I have read so I don't know if it is viable. I feel like I should give credit to the source I read but for the life of me I can't find it - I probably saw it on a pyvideo.
.