What is the correct way to wrap mulptiple functions in the loop in Python?
I have universal wrapper and the list of functions. I need to wrap all functions from the list inside one loop but for f in funcs: doesn't work for me.
What will be the correct way to perform this?
def orig_func1(x, y):
print "x=%i y=%i" % (x,y)
def orig_func2(a, b):
print "a=%i b=%i" % (a,b)
def wrapper(func):
f_name = func.__name__
print 'adding hook for function [%s]' % f_name
def inner(*args, **kwargs):
print 'I am before original function'
ret = func(*args, **kwargs)
print 'I am after original function'
return ret
return inner
funcs = [orig_func1, orig_func2]
print funcs
for f in funcs:
f = wrapper(f)
print funcs
and the results showing that functions from the list are not changed:
[<function orig_func1 at 0x022F78F0>, <function orig_func2 at 0x022F7930>]
adding hook for function [orig_func1]
adding hook for function [orig_func2]
[<function orig_func1 at 0x022F78F0>, <function orig_func2 at 0x022F7930>]
x=1 y=2
a=3 b=4
Inside that loop, f is nothing but a local variable. You're not changing anything meaningful unless you modify the list directly. Instead of:
for f in funcs:
f = wrapper(f)
You should do this:
for i, f in enumerate(funcs):
funcs[i] = wrapper(f)
This will change the functions in your list to new, wrapped ones that you can use. But it still won't change the ultimate definition of the function. Nothing will, once it's been defined, short of a complete redefinition or a wrapper used right above the function definition; calling orig_func1 directly will net the same results before and after the for loop. If you want to modify a function at runtime, you'll have to keep referring to this wrapped version of the function that you've just created.
Instead of looping over functions trying to wrap them, you should be using Python Decorators. They are the correct way to modify the behavior of your functions, rather than your current looping method. If the official docs don't make it clear enough, here and here are a couple of tutorials that helped me quite a bit.
Your existing code actually looks a lot like some of the code snippets from my first tutorial link. You should replace your loop with the #decorator syntax instead of the manual wrapping.
That being said, you can accomplish what you originally intended with a comprehension. Replace your loop with this:
funcs = [wrapper(func) for func in funcs]
The other comments and answers were correct that your modification of f in the loop wouldn't work because it had a scope local to that loop and was not modifying your list.
I've implemented required behavior using following approach:
import Module1, Module2
funcs = ['Module1.func1','Module1.func2','Module2.func1','Module2.func2']
hooks = {}
def wrapper(func,f_name):
if not hooks.has_key(f_name):
hooks[f_name] = {'before':[],
'after':[]}
def inner(*args, **kwargs):
for f in hooks[f_name]['before']:
f(*args, **kwargs)
ret = func(*args, **kwargs)
for f in hooks[f_name]['after']:
f(*args, **kwargs)
return ret
return inner
def implementHooks():
for f in funcs:
obj_name, func_name = f.rsplit('.',1)
obj = globals()[obj_name]
func = getattr(obj, func_name)
setattr(obj, func_name, wrapper(func, f))
implementHooks()
def module1_func1_hook():
print 'Before running module1.func1()'
hooks['Module1.func1']['before'] += [module1_func1_hook]
Related
In the code example below, I have two higher level functions, factory1 and factory2, that produce a function with identical behavior. The first factory, factory1, avoids having to explicitly define two different functions by letting the returned function change behavior based on a boolean from the factory. The usefulness of this is not as obvious in this example, but if the function to be produced were more complex, it would be detrimental to both readability and and maintainability to explicitly write out two almost identical copies of the function, like is done in factory2.
However, the factory2 implementation is faster, as can be seen by the timing results.
Is there a way to achieve the performance of factory2 without explicitly defining two alternative functions?
def factory1(condition):
def fn():
if condition:
return "foo"
else:
return "bar"
return fn
def factory2(condition):
def foo_fn():
return "foo"
def bar_fn():
return "bar"
if condition:
return foo_fn
else:
return bar_fn
def test1():
fn = factory1(True)
for _ in range(1000):
fn()
def test2():
fn = factory2(True)
for _ in range(1000):
fn()
if __name__ == '__main__':
import timeit
print(timeit.timeit("test1()", setup="from __main__ import test1"))
# >>> 62.458039999
print(timeit.timeit("test2()", setup="from __main__ import test2"))
# >>> 49.203676939
EDIT: Some more context
The reason I am asking is that I am trying to produce a function that looks something like this:
def function(data):
data = some_transform(data)
if condition:
# condition should be considered invariant at time of definition
data = transform1(data)
else:
data = transform2(data)
data = yet_another_transform(data)
return data
Depending on what you mean by "explicitly defining two functions", note that you don't have to execute a def statement until after you check the condition:
def factory3(condition):
if condition:
def fn():
return "foo"
else:
def fn():
return "bar"
return fn
One might object that this still has to compile two code objects before determining which one gets used to define the function at run-time. In the case, you might fallback on using exec on a dynamically constructed string. NOTE This needs to be done carefully for anything other than the static example I will show here. See the old definition for namedtuple for a good(?) example.
def factory4(condition):
code = """def fn():\n return "{}"\n""".format("foo" if condition else "bar")
exec(code)
return fn
A safer alternative might be to use a closure:
def factory5(condition):
def make_fun(val):
def _():
return val
return _
if condition:
return make_fun("foo")
else:
return make_fun("bar")
make_fun can be define outside of factory5 as well, as it doesn't rely on condition at all.
Based on your edit, I think you are just looking to implement dependency injection. Don't put an if statement inside your function; pass transform1 or transform2 as an argument:
def function(transform):
def _(data):
data = some_transform(data)
data = transform(data)
data = yet_another_transform(data)
return data
return _
if condition:
thing = function(transform1)
else:
thing = function(transform2)
A python wrapper specifically for for-loops and its actions
I write a lot FOR loops that are, well, pretty generic.
For eg:
for x in y:
do something
... and error-prone. eg forgetting the ":", or indentation probs.
Could we put the FOR loop in a def, and call it, supplying it with the something?
An interesting exercise if nothing else.
A basic wrapper...
def wrapper(func,*args):
def wrapped():
return func(*args)
return wrapped
wrapper(print,"bob")
wrapper()
...which works. ie prints bob out
but I don't know how to make the below work - returning a FOR function made lots of syntax errors.
eg something like:
def for_1(y, do_something):
def wrapped():
return for x in y:
do_something
return wrapped
for_1(range(3),print("bob\n"))
for_1()
...and didn't see bob on the screen 3 times.
Could someone point me in the right direction, please? A wrapper is not doing the returned function.
Perhaps use a class for the wrapper? Then have my own methods(??)
...or maybe point me to someone's page who has done this before. I have explored wrappers and decorators but not seen something for passing parameters to a FOR loop function
You can simply restructure your code to not return too early and not call to early.
For this split up the function and parameters as two parameters to your for_1 wrapper.
If you want return value, gather them in your for loop and return them as a list.
def for_1(y, do_something, with_param):
for x in y:
do_something(with_param)
for_1(range(3), print, "bob")
Why make it complicated?
def for_1(y, to_print):
for x in range(y):
print(to_print)
for_1(3, "bob")
OUTPUT:
bob
bob
bob
EDIT:
def square(x):
print(x*x)
def for_1(y, command, param):
for x in range(y):
command(param)
for_1(1, square, 3)
OUTPUT:
9
the print is evaluated immediately and its return value passed in. what you want here is to pass in a callable, and append () to the do_something inside the loop. than you can use lambda for the passed in function.
def for_1(y, do_something):
def wrapped():
return for x in y:
do_something() # so we call whatever is passed in to be executed at this point
return wrapped
f = for_1(range(3),lambda: print("bob\n"))
f()
# or equivalent:
def print_bob():
print("bob\n")
for_1(range(3),print_bob)
I am trying to use currying to make a simple functional add in Python. I found this curry decorator here.
def curry(func):
def curried(*args, **kwargs):
if len(args) + len(kwargs) >= func.__code__.co_argcount:
return func(*args, **kwargs)
return (lambda *args2, **kwargs2:
curried(*(args + args2), **dict(kwargs, **kwargs2)))
return curried
#curry
def foo(a, b, c):
return a + b + c
Now this is great because I can do some simple currying:
>>> foo(1)(2, 3)
6
>>> foo(1)(2)(3)
6
But this only works for exactly three variables. How do I write the function foo so that it can accept any number of variables and still be able to curry the result? I've tried the simple solution of using *args but it didn't work.
Edit: I've looked at the answers but still can't figure out how to write a function that can perform as shown below:
>>> foo(1)(2, 3)
6
>>> foo(1)(2)(3)
6
>>> foo(1)(2)
3
>>> foo(1)(2)(3)(4)
10
Arguably, explicit is better than implicit:
from functools import partial
def example(*args):
print("This is an example function that was passed:", args)
one_bound = partial(example, 1)
two_bound = partial(one_bound, 2)
two_bound(3)
#JohnKugelman explained the design problem with what you're trying to do - a call to the curried function would be ambiguous between "add more curried arguments" and "invoke the logic". The reason this isn't a problem in Haskell (where the concept comes from) is that the language evaluates everything lazily, so there isn't a distinction you can meaningfully make between "a function named x that accepts no arguments and simply returns 3" and "a call to the aforementioned function", or even between those and "the integer 3". Python isn't like that. (You could, for example, use a zero-argument call to signify "invoke the logic now"; but that would break special cases aren't special enough, and require an extra pair of parentheses for simple cases where you don't actually want to do any currying.)
functools.partial is an out-of-box solution for partial application of functions in Python. Unfortunately, repeatedly calling partial to add more "curried" arguments isn't quite as efficient (there will be nested partial objects under the hood). However, it's much more flexible; in particular, you can use it with existing functions that don't have any special decoration.
You can implement the same thing as the functools.partial example for yourself like this:
def curry (prior, *additional):
def curried(*args):
return prior(*(args + additional))
return curried
def add(*args):
return sum(args)
x = curry(add, 3,4,5)
y = curry(b, 100)
print y(200)
# 312
It may be easier to think of curry as a function factory rather than a decorator; technically that's all a decorator does but the decorator usage pattern is static where a factory is something you expect to be invoking as part of a chain of operations.
You can see here that I'm starting with add as an argument to curry and not add(1) or something: the factory signature is <callable>, *<args> . That gets around the problem in the comments to the original post.
FACT 1: It is simply impossible to implement an auto currying function for a variadic function.
FACT 2: You might not be searching for curry, if you want the function that will be passed to it * to know* that its gonna be curried, so as to make it behave differently.
In case what you need is a way to curry a variadic function, you should go with something along these lines below (using your own snipped):
def curryN(arity, func):
"""curries a function with a pre-determined number of arguments"""
def curried(*args, **kwargs):
if len(args) + len(kwargs) >= arity:
return func(*args, **kwargs)
return (lambda *args2, **kwargs2:
curried(*(args + args2), **dict(kwargs, **kwargs2)))
return curried
def curry(func):
"""automatically curries a function"""
return curryN(func.__code__.co_argcount, func);
this way you can do:
def summation(*numbers):
return sum(numbers);
sum_two_numbers = curryN(2, summation)
sum_three_numbers = curryN(3, summation)
increment = curryN(2, summation)(1)
decrement = curryN(2, summation)(-1)
I think this is a decent solution:
from copy import copy
import functools
def curry(function):
def inner(*args, **kwargs):
partial = functools.partial(function, *args, **kwargs)
signature = inspect.signature(partial.func)
try:
signature.bind(*partial.args, **partial.keywords)
except TypeError as e:
return curry(copy(partial))
else:
return partial()
return inner
This just allow you to call functools.partial recursively in an automated way:
def f(x, y, z, info=None):
if info:
print(info, end=": ")
return x + y + z
g = curry_function(f)
print(g)
print(g())
print(g(2))
print(g(2,3))
print(g(2)(3))
print(g(2, 3)(4))
print(g(2)(3)(4))
print(g(2)(3, 4))
print(g(2, info="test A")(3, 4))
print(g(2, info="test A")(3, 4, info="test B"))
Outputs:
<function curry.<locals>.inner at 0x7f6019aa6f28>
<function curry.<locals>.inner at 0x7f6019a9a158>
<function curry.<locals>.inner at 0x7f6019a9a158>
<function curry.<locals>.inner at 0x7f6019a9a158>
<function curry.<locals>.inner at 0x7f6019a9a0d0>
9
9
9
test A: 9
test B: 9
I have a function which performs an expensive operation and is called often; but, the operation only needs to be performed once - its result could be cached.
I tried making an infinite generator but I didn't get the results I expected:
>>> def g():
... result = "foo"
... while True:
... yield result
...
>>> g()
<generator object g at 0x1093db230> # why didn't it give me "foo"?
Why isn't g a generator?
>>> g
<function g at 0x1093de488>
Edit: it's fine if this approach doesn't work, but I need something which performs exactly like a regular function, like so:
>>> [g() for x in range(3)]
["foo", "foo", "foo"]
g() is a generator function. Calling it returns the generator. You then need to use that generator to get your values. By looping, for example, or by calling next() on it:
gen = g()
value = next(gen)
Note that calling g() again will calculate the same value again and produce a new generator.
You may just want to use a global to cache the value. Storing it as an attribute on the function could work:
def g():
if not hasattr(g, '_cache'):
g._cache = 'foo'
return g._cache
A better way: #functools.lru_cache(maxsize=None). It's been backported to python 2.7, or you could just write your own.
I am occasionally guilty of doing:
def foo():
if hasattr(foo, 'cache'):
return foo.cache
# do work
foo.cache = result
return result
Here's a dead-simple caching decorator. It doesn't take into account any variations in parameters, it just returns the same result after the first call. There are fancier ones out there that cache the result for each combination of inputs ("memoization").
import functools
def callonce(func):
result = []
#functools.wraps(func)
def wrapper(*args, **kwargs):
if not result:
result.append(func(*args, **kwargs))
return result[0]
return wrapper
Usage:
#callonce
def long_running_function(x, y, z):
# do something expensive with x, y, and z, producing result
return result
If you would prefer to write your function as a generator for some reason (perhaps the result is slightly different on each call, but there's still a time-consuming initial setup, or else you just want C-style static variables that allow your function to remember some bit of state from one call to the next), you can use this decorator:
import functools
def gen2func(generator):
gen = []
#functools.wraps(generator)
def wrapper(*args, **kwargs):
if not gen:
gen.append(generator(*args, **kwargs))
return next(gen[0])
return wrapper
Usage:
#gen2func
def long_running_function_in_generator_form(x, y, z):
# do something expensive with x, y, and z, producing result
while True:
yield result
result += 1 # for example
A Python 2.5 or later version that uses .send() to allow parameters to be passed to each iteration of the generator is as follows (note that **kwargs are not supported):
import functools
def gen2func(generator):
gen = []
#functools.wraps(generator)
def wrapper(*args):
if not gen:
gen.append(generator(*args))
return next(gen[0])
return gen[0].send(args)
return wrapper
#gen2func
def function_with_static_vars(a, b, c):
# time-consuming initial setup goes here
# also initialize any "static" vars here
while True:
# do something with a, b, c
a, b, c = yield # get next a, b, c
A better option would be to use memoization. You can create a memoize decorator that you can use to wrap any function that you want to cache the results for. You can find some good implementations here.
You can also leverage Beaker and its cache.
Also it has a tons of extensions.
I have a question about decorators. I understand what are decorators and I know how to use it, I have read all this tutorial How to make a chain of function decorators?
I understand that :
>>> def my_decorator(fn):
>>> print 'Do something before'
>>> print fn()
>>> def foo():
>>> return 'Hello World!'
>>> foo = my_decorator(foo)
Is the same at that :
>>> def my_decorator(fn):
>>> print 'Do something before'
>>> print fn()
>>> #my_decorator
>>> def foo():
>>> return 'Hello World!'
I know what are closures and why we use closure in a decorator with parameters (to get the decorator parameters in nested function) but that I don't understand is why we use closure and nested functions to get arguments and the function.
How the closure (or something else) can access parameters and the function outside. I am unable to do the same without the #decorator.
Here for example I can access my foo() and the parameters of the function without passing this function in parameter :
def my_decorator(str):
def wrapper(fn):
def inner_function(*args):
print 'Do something before'
return fn(*args)
return inner_function
return wrapper
#my_decorator('test')
def foo(a, b):
return a + b
print foo(1, 1)
How this is possible ?
I found the solution :
In fact the decorator use the closure functionality :
So here is the solution to do the same thing without the decorator and with parameters (it's just to understand the operation, and to learn)
def wrapper(str):
def decorator_factory(fn):
def inner_function(*args):
print 'Do something before'
return fn(*args)
return inner_function
return decorator_factory
#my_decorator('test')
def foo(a, b):
return a + b
# with decorator
print foo(1, 1)
# without decorator
print wrapper('str')(foo)(1, 1)
Decorator is a function that takes one argument (this argument is function) - this is my definition of decorator. In your case wrapper is decorator. While my_decorator is used to collect arguments for inner_function. inner_function is used to replace original [not decorated] function. Step by step explanation would be:
my_decorator is called to collect options for inner_function
my_decorator returns wrapper
wrapper is responsible for capturing original function or in other words decorate it. In your case original function is foo.
wrapper returns replacement function for original (which is inner_function)
from now foo points to inner_function, therefore inner_function is executed then foo is called
Hope it makes things a bit clearer.