Assume that I don't care about occurring exceptions at all. I have a complex function that calls multiple functions along the way.. I want to test that with certain input parameters, certain functions will be called.
So basically, I am looking for something like:
#patch(
"service.module.class.some_nested_function_1",
new_callable=AsyncMock,
)
#patch(
"service.module.class.some_nested_function_2",
new_callable=AsyncMock,
)
#pytest.mark.asyncio
async def test_complex_function(function2_mock, function1_mock):
some_param = "abc"
another_param = "xyz"
try:
call_complex_function("with_some_configuration")
except:
abort_crashing_function_and_continue_execution_in_complex_function()
assert function2_mock.assert_called_once_with(some_param)
assert function1_mock.assert_called_once_with(another_param)
EDIT: An alternative idea would be to have something like:
...
async def test_complex_function(function2_mock, function1_mock):
...
mock_every_function_call_except_complex_function_to_return_zero()
call_complex_function("with_some_configuration")
assert function2_mock.assert_called_once_with(some_param)
assert function1_mock.assert_called_once_with(another_param)
...
...
Function Call Counting Wrapper
Wrap your nested functions with this so that mocks aren't necessary and you can still see if certain functions were called or not called.
def count_calls_wrapper(fn):
call_count = 0
def _result(*args):
nonlocal call_count
call_count +=1
return fn(*args)
def _get_call_count():
return call_count
_result.get_call_count = _get_call_count
return _result
def f1():
return 10
f1 = count_calls_wrapper(f1)
assert(f1.get_call_count() == 0)
f1()
assert(f1.get_call_count() == 1)
Is there any way to check inside function f1 in my example if calling a function (here decorated or not_decorated) has a specific decorator (in code #out)? Is such information passed to a function?
def out(fun):
def inner(*args, **kwargs):
fun(*args, **kwargs)
return inner
#out
def decorated():
f1()
def not_decorated():
f1()
def f1():
if is_decorated_by_out: # here I want to check it
print('I am')
else:
print('I am not')
decorated()
not_decorated()
Expected output:
I am
I am not
To be clear, this is egregious hackery, so I don't recommend it, but since you've ruled out additional parameters, and f1 will be the same whether wrapped or not, you've left hacks as your only option. The solution is to add a local variable to the wrapper function for the sole purpose of being found by means of stack inspection:
import inspect
def out(fun):
def inner(*args, **kwargs):
__wrapped_by__ = out
fun(*args, **kwargs)
return inner
def is_wrapped_by(func):
try:
return inspect.currentframe().f_back.f_back.f_back.f_locals.get('__wrapped_by__') is func
except AttributeError:
return False
#out
def decorated():
f1()
def not_decorated():
f1()
def f1():
if is_wrapped_by(out):
print('I am')
else:
print('I am not')
decorated()
not_decorated()
Try it online!
This assumes a specific degree of nesting (the manual back-tracking via f_back to account for is_wrapped_by itself, f1, decorated and finally to inner (from out). If you want to determine if out was involved anywhere in the call stack, make is_wrapped_by loop until the stack is exhausted:
def is_wrapped_by(func):
frame = None
try:
# Skip is_wrapped_by and caller
frame = inspect.currentframe().f_back.f_back
while True:
if frame.f_locals.get('__wrapped_by__') is func:
return True
frame = frame.f_back
except AttributeError:
pass
finally:
# Leaving frame on the call stack can cause cycle involving locals
# which delays cleanup until cycle collector runs;
# explicitly break cycle to save yourself the headache
del frame
return False
If you are open to creating an additional parameter in f1 (you could also use a default parameter), you can use functools.wraps and check for the existence of the __wrapped__ attribute. To do so, pass the wrapper function to f:
import functools
def out(fun):
#functools.wraps(fun)
def inner(*args, **kwargs):
fun(*args, **kwargs)
return inner
#out
def decorated():
f1(decorated)
def not_decorated():
f1(not_decorated)
def f1(_func):
if getattr(_func, '__wrapped__', False):
print('I am')
else:
print('I am not')
decorated()
not_decorated()
Output:
I am
I am not
Suppose you have a function decoration like this one
def double_arg(fun):
def inner(x):
return fun(x*2)
return inner
however you can't access it (it's inside a 3rd party lib or something). In this case you can wrap it into another function that adds the name of the decoration to the resulting function
def keep_decoration(decoration):
def f(g):
h = decoration(g)
h.decorated_by = decoration.__name__
return h
return f
and replace the old decoration by the wrapper.
double_arg = keep_decoration(double_arg)
You can even write a helper function that checks whether a function is decorated or not.
def is_decorated_by(f, decoration_name):
try:
return f.decorated_by == decoration_name
except AttributeError:
return False
Example of use...
#double_arg
def inc_v1(x):
return x + 1
def inc_v2(x):
return x + 1
print(inc_v1(5))
print(inc_v2(5))
print(is_decorated_by(inc_v1, 'double_arg'))
print(is_decorated_by(inc_v2, 'double_arg'))
Output
11
6
True
False
In Clojure I can do something like this:
(-> path
clojure.java.io/resource
slurp
read-string)
instead of doing this:
(read-string (slurp (clojure.java.io/resource path)))
This is called threading in Clojure terminology and helps getting rid of a lot of parentheses.
In Python if I try to use functional constructs like map, any, or filter I have to nest them to each other. Is there a construct in Python with which I can do something similar to threading (or piping) in Clojure?
I'm not looking for a fully featured version since there are no macros in Python, I just want to do away with a lot of parentheses when I'm doing functional programming in Python.
Edit: I ended up using toolz which supports pipeing.
Here is a simple implementation of #deceze's idea (although, as #Carcigenicate points out, it is at best a partial solution):
import functools
def apply(x,f): return f(x)
def thread(*args):
return functools.reduce(apply,args)
For example:
def f(x): return 2*x+1
def g(x): return x**2
thread(5,f,g) #evaluates to 121
I wanted to take this to the extreme and do it all dynamically.
Basically, the below Chain class lets you chain functions together similar to Clojure's -> and ->> macros. It supports both threading into the first and last arguments.
Functions are resolved in this order:
Object method
Local defined variable
Built-in variable
The code:
class Chain(object):
def __init__(self, value, index=0):
self.value = value
self.index = index
def __getattr__(self, item):
append_arg = True
try:
prop = getattr(self.value, item)
append_arg = False
except AttributeError:
try:
prop = locals()[item]
except KeyError:
prop = getattr(__builtins__, item)
if callable(prop):
def fn(*args, **kwargs):
orig = list(args)
if append_arg:
if self.index == -1:
orig.append(self.value)
else:
orig.insert(self.index, self.value)
return Chain(prop(*orig, **kwargs), index=self.index)
return fn
else:
return Chain(prop, index=self.index)
Thread each result as first arg
file = Chain(__file__).open('r').readlines().value
Thread each result as last arg
result = Chain(range(0, 100), index=-1).map(lambda x: x * x).reduce(lambda x, y: x + y).value
So here's an extension to this question: https://stackoverflow.com/a/37568895/2290820
on how to optionally Enable or Disable Decorator on a Function.
On those lines, I came up with something like this to make decorator get invoked on a recursive call:
def deco(f):
def fattr(attr):
f.attr = attr
def closure(*args):
f(*args)
f.unwrap = f
f.closure = closure
return f
return fattr
#deco
def printa(x):
if x > 1:
print x
return printa(x-1)
else:
print x
return
printa({1:1})(5)
# do the same call w/o deocorator
def finta(x):
if x > 1:
print x
return finta(x-1)
else:
print x
return
finta(5) # this works
to experiment with decorators on a recursive function. Clearly, printa recursive version is not behaving the way it should be.
I could do
g = printa({1:1})
g.closure(5)
to turn on the decorator option or not use that option. Anyway, regardless of good or bad design, How can I make decorator get invoked on a recursive call?
In your deco you have an assignment f.attr = attr that "eats" your argument after first recursive call. Your should modify your recursive call this way:
def deco(f):
def fattr(attr):
f.attr = attr
def closure(*args):
f(*args)
f.unwrap = f
f.closure = closure
return f
return fattr
#deco
def printa(x):
if x > 1:
print x
return printa(None)(x-1) # None will be assigned to f.attr
else:
print x
return
printa({1:1})(5)
5
4
3
2
1
I asked previously how the nested functions work, but unfortunately I still don't quite get it. To understand it better, can someone please show some real-wold, practical usage examples of nested functions?
Many thanks
Your question made me curious, so I looked in some real-world code: the Python standard library. I found 67 examples of nested functions. Here are a few, with explanations.
One very simple reason to use a nested function is simply that the function you're defining doesn't need to be global, because only the enclosing function uses it. A typical example from Python's quopri.py standard library module:
def encode(input, output, quotetabs, header = 0):
...
def write(s, output=output, lineEnd='\n'):
# RFC 1521 requires that the line ending in a space or tab must have
# that trailing character encoded.
if s and s[-1:] in ' \t':
output.write(s[:-1] + quote(s[-1]) + lineEnd)
elif s == '.':
output.write(quote(s) + lineEnd)
else:
output.write(s + lineEnd)
... # 35 more lines of code that call write in several places
Here there was some common code within the encode function, so the author simply factored it out into a write function.
Another common use for nested functions is re.sub. Here's some code from the json/encode.py standard library module:
def encode_basestring(s):
"""Return a JSON representation of a Python string
"""
def replace(match):
return ESCAPE_DCT[match.group(0)]
return '"' + ESCAPE.sub(replace, s) + '"'
Here ESCAPE is a regular expression, and ESCAPE.sub(replace, s) finds all matches of ESCAPE in s and replaces each one with replace(match).
In fact, any API, like re.sub, that accepts a function as a parameter can lead to situations where nested functions are convenient. For example, in turtle.py there's some silly demo code that does this:
def baba(xdummy, ydummy):
clearscreen()
bye()
...
tri.write(" Click me!", font = ("Courier", 12, "bold") )
tri.onclick(baba, 1)
onclick expects you to pass an event-handler function, so we define one and pass it in.
Decorators are a very popular use for nested functions. Here's an example of a decorator that prints a statement before and after any call to the decorated function.
def entry_exit(f):
def new_f(*args, **kwargs):
print "Entering", f.__name__
f(*args, **kwargs)
print "Exited", f.__name__
return new_f
#entry_exit
def func1():
print "inside func1()"
#entry_exit
def func2():
print "inside func2()"
func1()
func2()
print func1.__name__
Nested functions avoid cluttering other parts of the program with other functions and variables that only make sense locally.
A function that return Fibonacci numbers could be defined as follows:
>>> def fib(n):
def rec():
return fib(n-1) + fib(n-2)
if n == 0:
return 0
elif n == 1:
return 1
else:
return rec()
>>> map(fib, range(10))
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
EDIT: In practice, generators would be a better solution for this, but the example shows how to take advantage of nested functions.
They are useful when using functions that take other functions as input. Say you're in a function, and want to sort a list of items based on the items' value in a dict:
def f(items):
vals = {}
for i in items: vals[i] = random.randint(0,100)
def key(i): return vals[i]
items.sort(key=key)
You can just define key right there and have it use vals, a local variable.
Another use-case is callbacks.
I have only had to use nested functions when creating decorators. A nested function is basically a way of adding some behavior to a function without knowing what the function is that you are adding behavior to.
from functools import wraps
from types import InstanceType
def printCall(func):
def getArgKwargStrings(*args, **kwargs):
argsString = "".join(["%s, " % (arg) for arg in args])
kwargsString = "".join(["%s=%s, " % (key, value) for key, value in kwargs.items()])
if not len(kwargs):
if len(argsString):
argsString = argsString[:-2]
else:
kwargsString = kwargsString[:-2]
return argsString, kwargsString
#wraps(func)
def wrapper(*args, **kwargs):
ret = None
if args and isinstance(args[0], InstanceType) and getattr(args[0], func.__name__, None):
instance, args = args[0], args[1:]
argsString, kwargsString = getArgKwargStrings(*args, **kwargs)
ret = func(instance, *args, **kwargs)
print "Called %s.%s(%s%s)" % (instance.__class__.__name__, func.__name__, argsString, kwargsString)
print "Returned %s" % str(ret)
else:
argsString, kwargsString = getArgKwargStrings(*args, **kwargs)
ret = func(*args, **kwargs)
print "Called %s(%s%s)" % (func.__name__, argsString, kwargsString)
print "Returned %s" % str(ret)
return ret
return wrapper
def sayHello(name):
print "Hello, my name is %s" % (name)
if __name__ == "__main__":
sayHelloAndPrintDebug = printCall(sayHello)
name = "Nimbuz"
sayHelloAndPrintDebug(name)
Ignore all the mumbo jumbo in the "printCall" function for right now and focus only the "sayHello" function and below. What we're doing here is we want to print out how the "sayHello" function was called everytime it is called without knowing or altering what the "sayHello" function does. So we redefine the "sayHello" function by passing it to "printCall", which returns a NEW function that does what the "sayHello" function does AND prints how the "sayHello" function was called. This is the concept of decorators.
Putting "#printCall" above the sayHello definition accomplishes the same thing:
#printCall
def sayHello(name):
print "Hello, my name is %s" % (name)
if __name__ == "__main__":
name = "Nimbuz"
sayHello(name)
Yet another (very simple) example. A function that returns another function. Note how the inner function (that is returned) can use variables from the outer function's scope.
def create_adder(x):
def _adder(y):
return x + y
return _adder
add2 = create_adder(2)
add100 = create_adder(100)
>>> add2(50)
52
>>> add100(50)
150
Python Decorators
This is actually another topic to learn, but if you look at the stuff on 'Using Functions as Decorators', you'll see some examples of nested functions.
OK, besides decorators: Say you had an application where you needed to sort a list of strings based on substrings which varied from time to time. Now the sorted functions takes a key= argument which is a function of one argument: the items (strings in this case) to be sorted. So how to tell this function which substrings to sort on? A closure or nested function, is perfect for this:
def sort_key_factory(start, stop):
def sort_key(string):
return string[start: stop]
return sort_key
Simple eh? You can expand on this by encapsulating start and stop in a tuple or a slice object and then passing a sequence or iterable of these to the sort_key_factory.