Let's say I have three functions and it needs to process a list one after another.
def f1(lst):
lst_processed = do_something_of_type1(lst)
return lst_processed
def f2(lst):
lst_processed = do_something_of_type2(lst)
return lst_processed
def f3(lst):
lst_processed = do_something_of_type3(lst)
return lst_processed
I would like to apply these three functions on some input_list in the following order: f1, then f2 since f2 needs the processed list from f1 and finally f3 which needs the return value of f2.
Option 1:
output_list = f3(f2(f1(input_list)))
Option 2:
output_list1 = f1(input_list)
output_list2 = f2(output_list1)
output_list = f3(output_list2)
Does one of these comply with PEP 8 more so than the other?
From this answer: Is there a chain calling method in Python?
def f1(i):
return i + 1
def f2(i):
return i + 1
def f3(i):
return i + 1
def f4(i):
return i + 1
from functools import reduce
def chain(*funcs):
def chained_call(arg):
return reduce(lambda r, f: f(r), funcs, arg)
return chained_call
chained = chain(f1, f2, f3, f4)
print(chained(0))
I don't think there's actually anything built-in to do this (functools seemed like the most likely place to find something), but it's also relatively easy to write the kind of thing you'd find there
def chainfunc(funcs, arg):
ret = arg
for f in funcs:
ret = f(arg)
return ret
output_list = chainfunc([f1, f2, f3], lst)
That said, you can also coerce functools.reduce() into doing something like this in a one-liner:
from functools import reduce
output_list = reduce(lambda a, b: b(a), [f1, f2, f3], lst)
I would put option 2 in a function of it’s own, and then call that. Also, I wouldn’t call them f1, f2, etc.
option 1 is more pythonic, but depends for whom you are writing your code.
For my opinion option 2 is more readable for junior developers.
Have you searched the pep8 documentation?
Why do you need 3 functions?
def f(lst):
lst_1 = do_something_of_type1(lst)
lst_2 = do_something_of_type2(lst_1)
lst_processed = do_something_of_type3(lst_2)
return lst_processed
Question
Is there a readily available Pythonic way to compose a multiple nested function g = f3(f2(f1())) from a list of functions [f1, f2, f3] where there are more functions in a list.
If there are a few, I may do:
g = lambda x: f3(f2(f1(x)))
However when I have dozens of functions e.g layers in a deep neural network, it is un-manageable. Prefer not creating another function to compose g but finding an available way.
Update
Based on the answer from #Chris. For sequential neural network layers [ batchnorm, matmul, activation, softmaxloss ], each of which has a forward(X) method to calculate its output to the next layer, the loss function L and loss would be:
L = reduce(lambda f, g: lambda X: g(f(X)), [ layer.forward for layer in layers ] ) # Loss function
network_loss = L(X)
One way using functools.reduce:
from functools import reduce
f1 = lambda x: x+1
f2 = lambda x: x*2
f3 = lambda x: x+3
funcs = [f1, f2, f3]
g = reduce(lambda f, g: lambda x: g(f(x)), funcs)
Output:
g(1)==7 # ((1+1) * 2) + 3
g(2)==9 # ((2+1) * 2) + 3
Insight:
functools.reduce will chain its second argument (funcs here) according to its first argument (lambda here).
That being said, it will start chaining f1 and f2 as f_21(x) = f2(f1(x)), then f3 and f_21 as f3(f_21(x)) which becomes g(x).
One problem with the reduce-baed approach is that you introduce O(n) additional function calls. An alternative is to define a single function that remembers the functions to compose; when called, it simply calls each function in sequence on the given argument.
def compose(*args):
"""compose(f1, f2, ..., fn) == lambda x: fn(...(f2(f1(x))...)"""
def _(x):
result = x
for f in args:
result = f(result)
return result
return _
You can implement it yourself, but you could also try a module named compose which implements this, written by #mtraceur. It takes care to handle various details, such as correct function signature forwarding.
pip install compose
from compose import compose
def doubled(x):
return 2*x
octupled = compose(doubled, doubled, doubled)
print(octupled(1))
# 8
Lets say we have functions in python:
def _abs(iterable): #cause its normally for one element only
return [abs(i) for i in iterable]
def A(list, foo):
return foo(list)
list = [2,3,-5]
print( A(list,foo=sum) )
>> 0
while I may pass foo=sum to A, I am looking for an elegant way to pass something like foo=sum(_abs) to perform sum(_abs(list)).
The only way I see it now is to send a list of functions [sum, _abs] and apply them in order. Is there a better way?
Or, to compose more generally (i.e. with an arbitrary number of argument functions):
from functools import partial, reduce
def compose(*funcs):
return partial(reduce, lambda x, f: f(x), reversed(funcs))
(see the docs on partial and reduce - note you don't need to import reduce in Python 2.x)
Then:
>>> compose(sum, _abs)([2, 3, -5])
10
You could make an explicit compose function
>>> def compose(f, g):
... return lambda x: f(g(x))
...
Then use it
>>> A(list, compose(sum, _abs))
10
def compose(funcs, funcargs):
for func in reversed(funcs):
funcargs = func(funcargs)
return funcargs
I have a bunch of functions in a list:
funcs = [f1, f2, f3, f4, f5]
and all of the functions take in return a single argument, eg.
f1 = lambda x: x*2
I'd like to map all these functions together
result = lambda x: f5(f4(f3(f2(f1(x)))))
or, iterating over funcs
def dispatch(x):
for f in funcs:
x = f(x)
return x
dispatch works fine, but I couldn't figure out a clean way to do this using iterools. Is it possible? Does this sequential function mapping idiom have a name?
There is no point in using itertools here; you are producing one output, and you could not apply this to an infinite iterable. You have to have a finite number of functions in the input iterable for this to work at all.
Use the reduce() function:
from functools import reduce
x = reduce(lambda res, func: func(res), funcs, x)
The functools.reduce() import helps the above work in both Python 2 and 3.
reduce(), together with map(), filter() and, yes, itertools, is an often used tool in functional programming.
Another (less efficient, alas) way of looking at Martijn's answer is to realize that you want to compose the list of functions.
# function composition: compose(f,g)(x) = f(g(x))
def compose(f, g):
return lambda x: f(g(x))
# Identity for function composition
# compose(f, identity)(x) = f(x)
identity = lambda x: x
# result(x) = f1(f2(...fn(x)...))
result = reduce(compose, funcs, identity)
I have an array of functions and I'm trying to produce one function which consists of the composition of the elements in my array.
My approach is:
def compose(list):
if len(list) == 1:
return lambda x:list[0](x)
list.reverse()
final=lambda x:x
for f in list:
final=lambda x:f(final(x))
return final
This method doesn't seems to be working, help will be appreciated.
(I'm reversing the list because this is the order of composition I want the functions to be)
The easiest approach would be first to write a composition of 2 functions:
def compose2(f, g):
return lambda *a, **kw: f(g(*a, **kw))
And then use reduce to compose more functions:
import functools
def compose(*fs):
return functools.reduce(compose2, fs)
Or you can use some library, which already contains compose function.
def compose (*functions):
def inner(arg):
for f in reversed(functions):
arg = f(arg)
return arg
return inner
Example:
>>> def square (x):
return x ** 2
>>> def increment (x):
return x + 1
>>> def half (x):
return x / 2
>>> composed = compose(square, increment, half) # square(increment(half(x)))
>>> composed(5) # square(increment(half(5))) = square(increment(2.5)) = square(3.5) = 12,25
12.25
It doesn't work because all the anonymous functions you create in the loop refer to the same loop variable and therefore share its final value.
As a quick fix, you can replace the assignment with:
final = lambda x, f=f, final=final: f(final(x))
Or, you can return the lambda from a function:
def wrap(accum, f):
return lambda x: f(accum(x))
...
final = wrap(final, f)
To understand what's going on, try this experiment:
>>> l = [lambda: n for n in xrange(10)]
>>> [f() for f in l]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
This result surprises many people, who expect the result to be [0, 1, 2, ...]. However, all the lambdas point to the same n variable, and all refer to its final value, which is 9. In your case, all the versions of final which are supposed to nest end up referring to the same f and, even worse, to the same final.
The topic of lambdas and for loops in Python has been already covered on SO.
One liner:
compose = lambda *F: reduce(lambda f, g: lambda x: f(g(x)), F)
Example usage:
f1 = lambda x: x+3
f2 = lambda x: x*2
f3 = lambda x: x-1
g = compose(f1, f2, f3)
assert(g(7) == 15)
Recursive implementation
Here's a fairly elegant recursive implementation, which uses features of Python 3 for clarity:
def strict_compose(*funcs):
*funcs, penultimate, last = funcs
if funcs:
penultimate = strict_compose(*funcs, penultimate)
return lambda *args, **kwargs: penultimate(last(*args, **kwargs))
Python 2 compatible version:
def strict_compose2(*funcs):
if len(funcs) > 2:
penultimate = strict_compose2(*funcs[:-1])
else:
penultimate = funcs[-2]
return lambda *args, **kwargs: penultimate(funcs[-1](*args, **kwargs))
This is an earlier version which uses lazy evaluation of the recursion:
def lazy_recursive_compose(*funcs):
def inner(*args, _funcs=funcs, **kwargs):
if len(_funcs) > 1:
return inner(_funcs[-1](*args, **kwargs), _funcs=_funcs[:-1])
else:
return _funcs[0](*args, **kwargs)
return inner
Both would seem to make a new tuple and dict of arguments each recursive call.
Comparison of all suggestions:
Let's test some of these implementations and determine which is most performant, first some single argument functions (Thank you poke):
def square(x):
return x ** 2
def increment(x):
return x + 1
def half(x):
return x / 2
Here's our implementations, I suspect my iterative version is the second most efficient (manual compose will naturally be fastest), but that may be in part due to it sidestepping the difficulty of passing any number of arguments or keyword arguments between functions - in most cases we'll only see the trivial one argument being passed.
from functools import reduce
def strict_recursive_compose(*funcs):
*funcs, penultimate, last = funcs
if funcs:
penultimate = strict_recursive_compose(*funcs, penultimate)
return lambda *args, **kwargs: penultimate(last(*args, **kwargs))
def strict_recursive_compose2(*funcs):
if len(funcs) > 2:
penultimate = strict_recursive_compose2(*funcs[:-1])
else:
penultimate = funcs[-2]
return lambda *args, **kwargs: penultimate(funcs[-1](*args, **kwargs))
def lazy_recursive_compose(*funcs):
def inner(*args, _funcs=funcs, **kwargs):
if len(_funcs) > 1:
return inner(_funcs[-1](*args, **kwargs), _funcs=_funcs[:-1])
else:
return _funcs[0](*args, **kwargs)
return inner
def iterative_compose(*functions):
"""my implementation, only accepts one argument."""
def inner(arg):
for f in reversed(functions):
arg = f(arg)
return arg
return inner
def _compose2(f, g):
return lambda *a, **kw: f(g(*a, **kw))
def reduce_compose1(*fs):
return reduce(_compose2, fs)
def reduce_compose2(*funcs):
"""bug fixed - added reversed()"""
return lambda x: reduce(lambda acc, f: f(acc), reversed(funcs), x)
And to test these:
import timeit
def manual_compose(n):
return square(increment(half(n)))
composes = (strict_recursive_compose, strict_recursive_compose2,
lazy_recursive_compose, iterative_compose,
reduce_compose1, reduce_compose2)
print('manual compose', min(timeit.repeat(lambda: manual_compose(5))), manual_compose(5))
for compose in composes:
fn = compose(square, increment, half)
result = min(timeit.repeat(lambda: fn(5)))
print(compose.__name__, result, fn(5))
Results
And we get the following output (same magnitude and proportion in Python 2 and 3):
manual compose 0.4963762479601428 12.25
strict_recursive_compose 0.6564744340721518 12.25
strict_recursive_compose2 0.7216697579715401 12.25
lazy_recursive_compose 1.260614730999805 12.25
iterative_compose 0.614982972969301 12.25
reduce_compose1 0.6768529079854488 12.25
reduce_compose2 0.9890829260693863 12.25
And my expectations were confirmed: the fastest is of course, manual function composition followed by the iterative implementation. The lazy recursive version is much slower - likely since a new stack frame is created by each function call and a new tuple of functions is created for each function.
For a better and perhaps more realistic comparison, if you remove **kwargs and change *args to arg in the functions, the ones that used them will be more performant, and we can better compare apples to apples - here, aside from manual composition, reduce_compose1 wins followed by the strict_recursive_compose:
manual compose 0.443808660027571 12.25
strict_recursive_compose 0.5409777010791004 12.25
strict_recursive_compose2 0.5698030130006373 12.25
lazy_recursive_compose 1.0381018499610946 12.25
iterative_compose 0.619289995986037 12.25
reduce_compose1 0.49532539502251893 12.25
reduce_compose2 0.9633988010464236 12.25
Functions with just one arg:
def strict_recursive_compose(*funcs):
*funcs, penultimate, last = funcs
if funcs:
penultimate = strict_recursive_compose(*funcs, penultimate)
return lambda arg: penultimate(last(arg))
def strict_recursive_compose2(*funcs):
if len(funcs) > 2:
penultimate = strict_recursive_compose2(*funcs[:-1])
else:
penultimate = funcs[-2]
return lambda arg: penultimate(funcs[-1](arg))
def lazy_recursive_compose(*funcs):
def inner(arg, _funcs=funcs):
if len(_funcs) > 1:
return inner(_funcs[-1](arg), _funcs=_funcs[:-1])
else:
return _funcs[0](arg)
return inner
def iterative_compose(*functions):
"""my implementation, only accepts one argument."""
def inner(arg):
for f in reversed(functions):
arg = f(arg)
return arg
return inner
def _compose2(f, g):
return lambda arg: f(g(arg))
def reduce_compose1(*fs):
return reduce(_compose2, fs)
def reduce_compose2(*funcs):
"""bug fixed - added reversed()"""
return lambda x: reduce(lambda acc, f: f(acc), reversed(funcs), x)
The most reliable implementation I have found is in the 3rd party library toolz. The compose function from this library also deals with docstring for the composition of functions.
The source code is freely available. Below is a simple example of usage.
from toolz import compose
def f(x):
return x+1
def g(x):
return x*2
def h(x):
return x+3
res = compose(f, g, h)(5) # 17
You can also create an array of functions and use reduce:
def f1(x): return x+1
def f2(x): return x+2
def f3(x): return x+3
x = 5
# Will print f3(f2(f1(x)))
print reduce(lambda acc, x: x(acc), [f1, f2, f3], x)
# As a function:
def compose(*funcs):
return lambda x: reduce(lambda acc, f: f(acc), funcs, x)
f = compose(f1, f2, f3)
pip install funcoperators is another library to implement it that allows infix notation:
from funcoperators import compose
# display = lambda x: hex(ord(list(x)))
display = hex *compose* ord *compose* list
# also works as a function
display = compose(hex, ord, list)
pip install funcoperators https://pypi.org/project/funcoperators/
Disclaimer: I'm the creator of the module
Suppose you have the following functions:
def square(x):
return x**2
def inc(x):
return x+1
def half(x):
return x/2
Define a compose function as follows:
import functools
def compose(*functions):
return functools.reduce(lambda f, g: lambda x: g(f(x)),
functions,
lambda x: x)
Usage:
composed = compose(square, inc, inc, half)
compose(10)
>>> 51.0
which executes the functions procedurally in the defined order:
square (= 100)
inc (= 101)
inc (= 102)
half (= 51)
Adapted from https://mathieularose.com/function-composition-in-python/.
I prefer this one due to readability/simplicity
from functools import reduce
def compose(*fs):
apply = lambda arg, f: f(arg)
composition = lambda x: reduce(apply, [x, *fs])
return composition
the pipe = compose(a, b, c) will first apply a, then b and then c.
With regard to maintainability (an debugging) I think actually this one is the easiest to use:
def compose(*fs):
def composition(x):
for f in fs:
x = f(x)
return x
return composition
You can use funcy.
Installation:
pip install funcy
Then you can use compose or rcompose as follows:
from funcy import compose, rcompose
def inc(x): return x + 1
def double(x): return x + x
def tripple(x): return x + x + x
print(compose(tripple, double, inc)(1)) # 12
print(rcompose(inc, double, tripple)(1)) # 12
I found this piece of code from GeeksforGeeks here for Python 3. Not sure of how efficient it is, but it is very simple to understand.
# importing reduce() from functools
from functools import reduce
# composite_function accepts N
# number of function as an
# argument and then compose them
def composite_function(*func):
def compose(f, g):
return lambda x : f(g(x))
return reduce(compose, func, lambda x : x)
# Function to add 2
def add(x):
return x + 2
# Function to multiply 2
def multiply(x):
return x * 2
# Function to subtract 2
def subtract(x):
return x - 1
# Here add_subtract_multiply will
# store lambda x : multiply(subtract(add(x)))
add_subtract_multiply = composite_function(multiply,
subtract,
add)
print("Adding 2 to 5, then subtracting 1 and multiplying the result with 2: ",
add_subtract_multiply(5))
You can keep adding more functions to composite_functions e.g.:
print(composite_function(multiply, add, subtract, multiply,subtract, add)(5))
More general solution of Imanol Luengo from my point of view (python notebook example):
from functools import reduce
from functools import partial
def f(*argv, **kwargs):
print('f: {} {}'.format(argv, kwargs))
return argv, kwargs
def g(*argv, **kwargs):
print('g: {} {}'.format(argv, kwargs))
return argv, kwargs
def compose(fs, *argv, **kwargs):
return reduce(lambda x, y: y(*x[0], **x[1]), fs, (argv, kwargs))
h = partial(compose, [f, g])
h('value', key='value')
output:
f: ('value',) {'key': 'value'}
g: ('value',) {'key': 'value'}
m = partial(compose, [h, f, g])
m('value', key='value')
output:
f: ('value',) {'key': 'value'}
g: ('value',) {'key': 'value'}
f: ('value',) {'key': 'value'}
g: ('value',) {'key': 'value'}
Perfectly good question, but the answers sure are unnecessarily complex. It's just:
def compose(*funs):
return (lambda x:
x if len(funs) == 0
else compose(*funs[:-1])(funs[-1](x)))
If you want no dependencies here is a one-liner recursive solution:
def compose(*f):
return f[0] if len(f) <= 1 else lambda *a,**kw: f[0](compose(*f[1:])(*a,**kw))
N.B. len(f) == 1 might seem more reasonable at first sight, but it allows to write compose() (i.e. no arguments) throwing an error only when you apply the empty compose function. On the contrary, with len(f) <= 1, compose() throws an error immediately, which is a more rational behavior.
This is my version
def compose(*fargs):
def inner(arg):
if not arg:
raise ValueError("Invalid argument")
if not all([callable(f) for f in fargs]):
raise TypeError("Function is not callable")
return reduce(lambda arg, func: func(arg), fargs, arg)
return inner
An example of how it's used
def calcMean(iterable):
return sum(iterable) / len(iterable)
def formatMean(mean):
return round(float(mean), 2)
def adder(val, value):
return val + value
def isEven(val):
return val % 2 == 0
if __name__ == '__main__':
# Ex1
rand_range = [random.randint(0, 10000) for x in range(0, 10000)]
isRandIntEven = compose(calcMean, formatMean,
partial(adder, value=0), math.floor.__call__, isEven)
print(isRandIntEven(rand_range))