I have some functions which, under normal circumstances, are called with arguments provided by user input. It is, however, valid to call some of these functions with certain series of arguments which are determined at runtime based on some system state.
I would like for the user to be able to optionally instruct my program to call these functions with all valid input and return the results of each call. I think a decorator which would work something like an activation switch for functions which have another decorator which indicates which series of arguments to use would work well.
Additionally, I need to preserve the function signature and metadata. It's vital to the operation of my program.
This is what I've tried, but it doesn't work. It is based upon this example.
>>> from decorator import decorator
>>> def the_list():
... return ["foo", "bar", "baz"]
...
>>> import itertools
>>> #decorator
... def do_all(func):
... # this will do nothing (besides wrap in a tuple) unless func is decorated with #gets_arg_from
... if hasattr(func, 'get_from'):
... return tuple(func(*args) for args in itertools.product(*(list_fun() for list_fun in func.get_from)))
... else:
... return (func(),)
...
>>> def gets_arg_from(*list_funcs):
... # this will do nothing to func unless decorated with #do_all
... def gets_arg_from(func, *args, **kwargs):
... func.get_from = list_funcs
... return func(*args, **kwargs)
... return decorator(gets_arg_from)
...
>>> #gets_arg_from(the_list)
... def print_func(word):
... # this will print its argument unless decorated with #do_all
... # at that point it will print every element returned by the_list()
... print word
...
>>> print_func("foo")
foo
>>> all = decorator(do_all, print_func)
>>> all()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: print_func() takes exactly 1 argument (0 given)
>>> print_func.get_from
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'get_from'
What I expected was:
>>> all()
("foo", "bar", "baz")
What I've noticed is wrong:
gets_arg_from doesn't add the get_from attribute to func.
Something about me using the notation #gets_arg_from(the_list) is wrong. It thinks I am trying to pass two arguments (but why would that be a problem anyway?)
As for my motivation, I think of decorators for this because there are literally hundreds of these routines, their implementation details (as well as their functional correctness) is subject to frequent change, and I don't want to use inspect to reason what to do based on their argument names nor do I want to hard-code the do_all functionality for each function for which it makes sense. Class methods might work, but for my purpose, they're semantically contrived. Furthermore, for the sake of others who may have to maintain my code, I think it is easier to ask them to apply a decorator and not to worry about the rest rather than to use certain argument names or place the function in a certain class or whatever. I realize this question may sound strange, so I figured this footnote might help make me look less like a madman.
Isn't this doing the thing you want?
import functools
from itertools import product
def the_list():
return ["foo", "bar", "baz"]
def do_all(func):
if hasattr(func, 'get_from'):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return tuple(func(*args) for args in
product(*(lf() for lf in func.get_from)))
return wrapper
return func
def gets_arg_from(*list_funcs):
def decorator(func):
func.get_from = list_funcs
return func
return decorator
#gets_arg_from(the_list)
def print_func(word):
return word
print print_func('foo')
all = do_all(print_func)
print all()
Edit: Explanation
These two code segments are identical:
#deco
def func(...):
some code
is the same as
func = deco(lambda ...: some code)
#something is just a syntactic sugar for the function call and anonymous function creation...
I'll explain what happened in the next peace of code step by step:
#gets_arg_from(the_list)
def print_func(word):
return word
First Python creates an anonimous function that receives a parameter word and has a body that just returns this word (or does whatever the function body does)
Then the function get_arg_from gets called and the_list gets passed to it as an argument
get_arg_from creates a decorator function and returnes it
The decorator function returned from the get_arg_from is called (this is the syntactic sugar thing) passing as an argument func the anonimous function created in the step 1.
decorator just assigns the list_funcs tuple to the get_from attribute of the anonymous function and returns the anonimous function
The return value of the decorator function is assigned to a variable print_func
Similar effect can be achieved with:
def __anonimous(word):
return word
__decorator = gets_arg_from(the_list)
print_func = __decorator(__anonimous)
So basically gets_arg_from is not a decorator it's a function that returns a decorator.
do_all on the other hand is a decorator, it receives a function as an argument, and returns either the original function (if the function doesn't have the attribute get_from) or a wrapper function which replaces the original function (if it has the get_from attribute).
You can find more examples here.
Related
I am new to the more advanced features of Python like decorators.
I am unable to understand how the Python interpreter actually understands where to put the original function object in a decorator.
Lets look at an example: Examples taken from here.
Simple decorator with no arguments:
def call_counter(func):
def helper(*args, **kwargs):
helper.calls += 1
return func(*args, **kwargs)
helper.calls = 0
return helper
#call_counter
def succ(x):
return x + 1
This makes perfect sense if we can assume that the first/only argument to the decorator call_counter(func) is the function object that needs to wrapped ie. in this case succ() function.
But things become inconsistent when you are talking about "decorators with parameters". Look at the example below:
Decorator with one argument:
def greeting(expr): # Shouldn't expr be the function here ? Or at least isn't there suppose to be another parameter.
def greeting_decorator(func): # How does Python know to pass the function down here ?
def function_wrapper(x):
print(expr + ", " + func.__name__ + " returns:")
func(x)
return function_wrapper
return greeting_decorator
#greeting("Hello")
def foo(x):
print(42)
foo("Hi")
Now we know Python has no concept of data-types, so function parameters give no information about what type of object they will contain.
Am I correct ?
Having said that lets look at the line from the above example:
def greeting(expr):
If for decorators the first argument is the function to be wrapped then by that logic expr should point to foo() right ? Otherwise there should be at least two parameters in greeting(), like:
def greeting(func, expr):
But instead Python can "magically" understand that the inner function needs to be passed the function reference:
def greeting(expr):
def greeting_decorator(func): # How is it correctly put one level down ?
The code has no datatypes or type information specified, so how is it that for decorators without arguments the function is passed as the first argument and for decorators with arguments the function is passed to the inner function ?
How can the interpreter detect that ?
What is going on here ?
This seems like "magic" to me.
What happens if I have 5 or 6 levels of nested functions ?
I am pretty sure I am missing something pretty basic here.
Thanks.
Python evaluates the expression after the # and uses the result as the decorator function.
Python calls the __call__ method of the object that is the decorator with the function as argument.
using
#call_counter
def succ(x):
return x + 1
callcounter is the object looked for __call__ to give the argument func
If you use
#greeting("Hello")
def foo(x):
print(42)
greeting("Hello") is evaluated and its result is an object that Python uses the __call__ method with the func argument.
I am learning about Python decorators and inner functions and have some questions about the lesson I'm learning via a YouTube video from codeacademy.com https://youtu.be/WOHsHaaJ8VQ.
When using inner functions sometimes I have to return the function with parenthesis, and sometimes without.
If I call an inner function without using decorators, I have to use parentheses when returning the inner function, otherwise it seems the inner function is returned as an object(?).
In the YouTube video from codeacademy.com as well as this one https://www.youtube.com/watch?v=FsAPt_9Bf3U, they call the inner function without parentheses and the expected result is output.
If I call an inner function using decorators, I have to not use parentheses when returning the inner function, otherwise it seems to work correctly, but throws an error along with some other weird results.
I've written some code to test different variations and output the results.
You can see the live code here: https://trinket.io/python/af1b47658f
# Test 1: The title function returns inner function wrapper without parentheses.
def title(print_name_function):
def wrapper():
print("Professor:")
print_name_function()
return wrapper # Without parentheses
def print_my_name():
print("John")
print('Test 1')
title(print_my_name)
# Results: Nothing is printed.
# Test 2: The title function returns inner function wrapper with parentheses.
def title(print_name_function):
def wrapper():
print("Professor:")
print_name_function()
return wrapper() # With parentheses
def print_my_name():
print("John")
print('Test 2')
title(print_my_name)
# Results: Professor John is printed.
# Test 3: Using a decorator while the title function returns inner function wrapper without parentheses
def title(print_name_function):
def wrapper():
print("Professor:")
print_name_function()
return wrapper # Without parentheses
#title
def print_my_name():
print("John")
print('Test 3')
print_my_name()
# Results: Professor John is printed.
# Test 4: Using a decorator while the title function returns inner function wrapper with parentheses
def title(print_name_function):
def wrapper():
print("Professor:")
print_name_function()
return wrapper() # With parentheses
#title
def print_my_name():
print("John")
print('Test 4')
print_my_name()
# Results: Professor John is printed and the following error is thrown:
'''
Traceback (most recent call last):
File "decorators.py", line 59, in <module>
print_my_name()
TypeError: 'NoneType' object is not callable.
'''
# Additionally, Professor John is printed before 'Test 4' is printed which seems that print_my_name() runs, then print('Test 4') runs.
In the two videos I've watched listed above about inner functions/decorators I've found...
For inner functions: the inner function was returned without using parentheses and ran correctly. Upon my testing, I have to use the parentheses for it to run correctly.
For decorators: the inner function was returned without using parentheses and ran correctly. Upon my testing, running without using parentheses works. Running with parentheses seems to work, but the output order is mixed up and an error is thrown (see test 4 in my code).
Let's break this down into two parts.
1) Let's ignore decorators for now.
You should use parentheses when you want to call some function.
Without parentheses, a function is just its name.
For example:
Here is a function, where we give it a number, and we get back that number plus 5.
def add_five(x):
return x + 5
We see that add_five, without parentheses, is just the function definition. Think of it as a recipe. It's not the actually cake, just the instructions on how to bake a cake.
>>> add_five
<function add_five at 0x10da3ce18>
Now we give it an ingredient, and it makes a cake:
>>> add_five(1)
6
Let's do a similar thing, but with better names.
>>> def make_cake(cake_type):
>>> print("Making: " + cake_type + " cake!")
>>> make_cake("carrot")
'Making: carrot cake!'
>>> make_cake
<function make_cake at 0x10da3cf28>
Ok, so when we put the function name without any parentheses, we're not actually calling the function, we're just getting the declaration of the function (which is kinda like the function's Birth Certificate, which has its memory address, in this case: 0x10da3cf28.
The same thing applies for functions that don't expect any parameters.
Without the parentheses, you're just asking, "Hey function, you exist?"
With the parentheses (and necessary parameters/variables required), you're saying, "Hey function, do something!"
Now for the second part.
2) What do Decorators do?
#SyntaxVoid has a great explanation about what you're doing. Decorators are a much more complicated thing, so I'll stick to explaining what they're doing in this specific context.
Basically, your decorator, #<Some Function Name> specifies a function to call the decorated function on.
def some_decorator(function_that_I_decorated):
print("I'm going to print this, and then call my decorated function!")
function_that_I_decorated()
#some_decorator
def my_decorated_function():
print("Did I do anything?")
Then we see the results:
>>> def some_decorator(function_that_I_decorated):
... print("I'm going to print this, and then call my decorated function!")
... function_that_I_decorated()
...
>>> #some_decorator
... def my_decorated_function():
... print("Did I do anything?")
...
I'm going to print this, and then call my decorated function!
Did I do anything?
Now here's the important part:
>>> my_decorated_function
>>> my_decorated_function()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable
Wait... didn't we define my_decorated_function?
Yes, we defined the function, but the decorator is reassigning that function name to something else.
Namely, my_decorator_function = some_decorator(my_decorator_function)
Now some_decorator happens to do something before calling my_decorator_function. It prints some stuff. But what is the return value of some_decorator? There's no return statement, so some_decorator returns None by default.
Therefore, my_decorator_function was created, run, and now has a new value.
Why would we want this behavior?
When we want the output to change, when running the same function with the same input(s) multiple times.
For example, maybe I want a function that returns "Go Left" every other time it's called, or "Go Right" every 5-times the function gets called.
If I want to do this with a function with more than one variable, that's easy! Just pass it in and check if num_times == whatever_int.
But life ain't so easy- sometimes other people already have functions written that are much simpler and only allow one variable, because that's more generalizable. Or maybe it's so complex it would take us a really long time to figure out how the function works (and we usually don't want to violate abstraction barriers anyways). In those situations, we need to adapt their function to our needs.
I would encourage you to read more about Currying, as that'll help you understand other uses too.
Let me use this famous quote first.
In python everything is an object.
I've been wrapping my head about two hours, until I remembered this quote. We should think original function and decorated function as objects, given an example:
def decorator(original_func):
def decorated_func():
print("Start decoration")
original_func()
print("End decoration")
return decorated_func # return decorated function as object without calling
#decorator
def func():
print("I will be decorated")
func()
The decorator_func transfers the func object to decorated_func object, and return the decorated_func as an object, so when we call func object with it's original name, we are actually calling the new function object decorated_func, which is equivalent to decorated_func().
Now it is easy to see why return decorated_func() is wrong, if we return decorated_func() in definition of decorator, we are return None, because the default return value of a function is None, and None is not callable as traceback says, so we can't use func() to call func.
Additionally, although the following two codes are equivalent, decorator help us to simplify our code, and without changing the original function object, also don't need to create a new function object mannually
#decorator
def func():
print("I will be decorated")
func()
def func():
print("I will be decorated")
dec_func = decorator(func)
dec_func()
I am going to to write a decorator which evaluates the actual names (not their value) of the variables that are passed to the function call.
Below, you find a skeleton of the code which makes it a bit clearer what I want to do.
import functools
def check_func(func):
# how to get variable names of function call
# s.t. a call like func(arg1, arg2, arg3)
# returns a dictionary {'a':'arg1', 'b':'arg2', 'c':'arg3'} ?
pass
def my_decorator(func):
#functools.wraps(func)
def call_func(*args, **kwargs):
check_func(func)
return func(*args, **kwargs)
return call_func
#my_decorator
def my_function(a, b, c):
pass
arg1='foo'
arg2=1
arg3=[1,2,3]
my_function(arg1,arg2,arg3)
You can't really have what you are asking for.
There are many ways of calling a function, where you won't even get variable names for individual values. For example, what would the names when you use literal values in the call, so:
my_function('foo', 10 - 9, [1] + [2, 3])
or when you use a list with values for argument expansion with *:
args = ['foo', 1, [1, 2, 3]]
my_function(*args)
Or when you use a functools.partial() object to bind some argument values to a callable object:
from functools import partial
func_partial = partial(my_function, arg1, arg2)
func_partial(arg3)
Functions are passed objects (values), not variables. Expressions consisting of just names may have been used to produce the objects, but those objects are independent of the variables.
Python objects can have many different references, so just because the call used arg1, doesn't mean that there won't be other references to the object elsewhere that would be more interesting to your code.
You could try to analyse the code that called the function (the inspect module can give you access to the call stack), but then that presumes that the source code is available. The calling code could be using a C extension, or interpreter only has access to .pyc bytecode files, not the original source code. You still would have to trace back and analyse the call expression (not always that straightforward, functions are objects too and can be stored in containers and retrieved later to be called dynamically) and from there find the variables involved if there are any at all.
For the trivial case, where only direct positional argument names were used for the call and the whole call was limited to a single line of source code, you could use a combination of inspect.stack() and the ast module to parse the source into something useful enough to analyse:
import inspect, ast
class CallArgumentNameFinder(ast.NodeVisitor):
def __init__(self, functionname):
self.name = functionname
self.params = []
self.kwargs = {}
def visit_Call(self, node):
if not isinstance(node.func, ast.Name):
return # not a name(...) call
if node.func.id != self.name:
return # different name being called
self.params = [n.id for n in node.args if isinstance(n, ast.Name)]
self.kwargs = {
kw.arg: kw.value.id for kw in node.keywords
if isinstance(kw.value, ast.Name)
}
def check_func(func):
caller = inspect.stack()[2] # caller of our caller
try:
tree = ast.parse(caller.code_context[0])
except SyntaxError:
# not a complete Python statement
return None
visitor = CallArgumentNameFinder(func.__name__)
visitor.visit(tree)
return inspect.signature(func).bind_partial(
*visitor.params, **visitor.kwargs)
Again, for emphasis: this only works with the most basic of calls, where the call consists of a single line only, and the called name matches the function name. It can be expanded upon but this takes a lot of work.
For your specific example, this produces <BoundArguments (a='arg1', b='arg2', c='arg3')>, so an inspect.BoundArguments instance. Use .arguments to get an OrderedDict mapping with the name-value pairs, or dict(....arguments) to turn that into a regular dictionary.
You'll have to think about your specific problem differently instead. Decorators are not meant to be acting upon the code calling, they act upon the decorated object. There are many other powerful features in the language that can help you deal with the calling context, decorators are not it.
Example code:
# -*- coding: utf-8 -*-
from functools import wraps
class MyClass(object):
def __init__(self):
pass
#decorator inside class
def call(f):
#wraps(f)
def wrapper(*args):
print 'Wrapper: ', args
return wrapper
#decorated 'method' without self
#call
def myfunc(a):
pass
c = MyClass()
c.myfunc(1)
Returns:
Wrapper: (<test3.MyClass object at 0xb788a34c>, 1)
Is this normal? Can someone explain?
If this is a feature I would use it in my library.
This is perfectly normal.
The function myfunc is replacecd by an instance of wrapper. The signature of wrapper is (*args). because it is a bound method, the first argument is the instance of MyClass which is printed out after the string `Wrapper: '.
What's confusing you?
It's worth noting that if you use call as a decorator from outside of MyClass, it will generate a TypeError. One way around this is to apply the staticmethod decorator to it but then you can't call it during class construction.
It's a little bit hacky but I address how to have it both ways here.
update after comment
it gets the instance as the first argument regardless of if you type self in the parameter list because after the class is created, and an instance instantiated, it is a bound method. when you call it in the form
#instance.call
def foo(bar):
return bar + 1
it expands to
def foo(bar):
return bar + 1
foo = instance.call(f)
but note that you are calling it on an instance! This will automatically expand to a call of the form
def foo(bar):
return bar + 1
foo = MyClass.call(instance, f)
This is how methods work. But you only defined call to take one argument so this raises a TypeError.
As for calling it during class construction, it works fine. but the function that it returns gets passed an instance of MyClass when it is called for the same reason that I explained above. Specifically, whatever arguments you explicity pass to it come after the implicit and automatic placement of the instance that it is called upon at the front of the argument list.
#call
def myfunc(a):
...
is equivalent to
def myfunc(a):
...
myfunc=call(myfunc)
The orginial myfunc may have expected only one argument, a, but after being decorated with call, the new myfunc can take any number of positional arguments, and they will all be put in args.
Notice also that
def call(f)
never calls f. So the fact that
def myfunc(a)
lacks the normal self argument is not an issue. It just never comes up.
When you call c.myfunc(1), wrapper(*args) gets called.
What is args? Well, since c.myfunc is a method call, c is sent as the first argument, followed by any subsequent arguments. In this case, the subsequent argument is 1. Both arguments are sent to wrapper, so args is the 2-tuple (c,1).
Thus, you get
Wrapper: (<test3.MyClass object at 0xb788a34c>, 1)
I'd like to do something like this:
class SillyWalk(object):
#staticmethod
def is_silly_enough(walk):
return (False, "It's never silly enough")
def walk(self, appraisal_method=is_silly_enough):
self.do_stuff()
(was_good_enough, reason) = appraisal_method(self)
if not was_good_enough:
self.execute_self_modifying_code(reason)
return appraisal_method
def do_stuff(self):
pass
def execute_self_modifying_code(self, problem):
from __future__ import deepjuju
deepjuju.kiss_booboo_better(self, problem)
with the idea being that someone can do
>>> silly_walk = SillyWalk()
>>> appraise = walk()
>>> is_good_walk = appraise(silly_walk)
and also get some magical machine learning happening; this last bit is not of particular interest to me, it was just the first thing that occurred to me as a way to exemplify the use of the static method in both an in-function context and from the caller's perspective.
Anyway, this doesn't work, because is_silly_enough is not actually a function: it is an object whose __get__ method will return the original is_silly_enough function. This means that it only works in the "normal" way when it's referenced as an object attribute. The object in question is created by the staticmethod() function that the decorator puts in between SillyWalk's is_silly_enough attribute and the function that's originally defined with that name.
This means that in order to use the default value of appraisal_method from within either SillyWalk.walk or its caller, we have to either
call appraisal_method.__get__(instance, owner)(...) instead of just calling appraisal_method(...)
or assign it as the attribute of some object, then reference that object property as a method that we call as we would call appraisal_method.
Given that neither of these solutions seem particularly Pythonic™, I'm wondering if there is perhaps a better way to get this sort of functionality. I essentially want a way to specify that a method should, by default, use a particular class or static method defined within the scope of the same class to carry out some portion of its daily routine.
I'd prefer not to use None, because I'd like to allow None to convey the message that that particular function should not be called. I guess I could use some other value, like False or NotImplemented, but it seems a) hackety b) annoying to have to write an extra couple of lines of code, as well as otherwise-redundant documentation, for something that seems like it could be expressed quite succinctly as a default parameter.
What's the best way to do this?
Maybe all you need is to use the function (and not the method) in the first place?
class SillyWalk(object):
def is_silly_enough(walk):
return (False, "It's never silly enough")
def walk(self, appraisal_function=is_silly_enough):
self.do_stuff()
(was_good_enough, reason) = appraisal_function(self)
if not was_good_enough:
self.execute_self_modifying_code(reason)
return appraisal_function
def do_stuff(self):
pass
def execute_self_modifying_code(self, problem):
deepjuju.kiss_booboo_better(self, problem)
Note that the default for appraisal_function will now be a function and not a method, even though is_silly_enough will be bound as a class method once the class is created (at the end of the code).
This means that
>>> SillyWalk.is_silly_enough
<unbound method SillyWalk.is_silly_enough>
but
>>> SillyWalk.walk.im_func.func_defaults[0] # the default argument to .walk
<function is_silly_enough at 0x0000000002212048>
And you can call is_silly_enough with a walk argument, or call a walk instance with .is_silly_enough().
If you really wanted is_silly_enough to be a static method, you could always add
is_silly_enough = staticmethod(is_silly_enough)
anywhere after the definition of walk.
I ended up writing an (un)wrapper function, to be used within function definition headers, eg
def walk(self, appraisal_method=unstaticmethod(is_silly_enough)):
This actually seems to work, at least it makes my doctests that break without it pass.
Here it is:
def unstaticmethod(static):
"""Retrieve the original function from a `staticmethod` object.
This is intended for use in binding class method default values
to static methods of the same class.
For example:
>>> class C(object):
... #staticmethod
... def s(*args, **kwargs):
... return (args, kwargs)
... def m(self, args=[], kwargs={}, f=unstaticmethod(s)):
... return f(*args, **kwargs)
>>> o = C()
>>> o.s(1, 2, 3)
((1, 2, 3), {})
>>> o.m((1, 2, 3))
((1, 2, 3), {})
"""
# TODO: Technically we should be passing the actual class of the owner
# instead of `object`, but
# I don't know if there's a way to get that info dynamically,
# since the class is not actually declared
# when this function is called during class method definition.
# I need to figure out if passing `object` instead
# is going to be an issue.
return static.__get__(None, object)
update:
I wrote doctests for the unstaticmethod function itself; they pass too. I'm still not totally sure that this is an actual smart thing to do, but it does seem to work.
Not sure if I get exactly what you're after, but would it be cleaner to use getattr?
>>> class SillyWalk(object):
#staticmethod
def ise(walk):
return (False, "boo")
def walk(self, am="ise"):
wge, r = getattr(self, am)(self)
print wge, r
>>> sw = SillyWalk()
>>> sw.walk("ise")
False boo