Just reading the Python doc's in particular this page, in regards to attrgetter and itemgetter.
I'm struggling to understand the usage of the usage of def and return in the code examples. Not the usage of those functions.
It seems like a lamdba, but I don't know how to interpret it.
EDIT
I think it just clicked. The attrgetter is like a constructor for the resolve_attr function? This was a little confusing, as i'd normally expect the code for a def to be between the def and the return. The code on the return is like a lambda. To add to the confusion the arg ojb is the argument passed to the returned object. Arhhhhh.....
What do i need to know for this to make sense?
This is the code example I'm referring to:
def attrgetter(*items):
if any(not isinstance(item, str) for item in items):
raise TypeError('attribute name must be a string')
if len(items) == 1:
attr = items[0]
def g(obj): #What is this
return resolve_attr(obj, attr)
else:
def g(obj):#And this?
return tuple(resolve_attr(obj, attr) for attr in items)
return g
def resolve_attr(obj, attr):
for name in attr.split("."):
obj = getattr(obj, name)
return obj
def is the keyword to define a function.
These are just functions defined inside another function attrgetter()
That's why you have multiple return inside attrgetter()
g looks like lambda because attrgetter() returns a function, and not the result directly.
g could be replaced by a lambda declaration indeed.
In Python, functions are just like normal object. You can pass them around, and (as in the example above) return them from other functions.
The example function, returns another function as its return value. It does so by defining a function (that's the def g(ojb): line) and then simply returning it.
You can do this, because in Python - a function is just like any other object.
Related
I have a class. This class has a list of functions that are to be evaluated by a different program.
class SomeClass(object):
def __init__(self, context):
self.functions_to_evaluate = []
There is a function that adds functions to an instance of SomeClass, via something like:
new_function = check_number(5)
SomeClassInstance.functions_to_evaluate.append(new_function)
Where check_number is a function that will check if number is greater than 10, let's say.
If I take SomeClassInstance.functions_to_evaluate and print it, I get a bunch of python objects, like so:
<some_library.check_number object at 0x07B35B90>
I am wondering if it is possible for me to extract the input given to check_number, so something like:
SomeClassInstance.functions_to_evaluate[0].python_feature() that will return "5" or whatever the input to check_number was to me.
You can use the standard library functools.partial, which creates a new partially applied function *.
>>> from functools import partial
>>> def check_number(input):
... return input > 10
>>> fn = partial(check_number, 5)
>>> fn.args # this attribute gives you back the bound arguments, as a tuple.
(5,)
>>> fn() # calls the function with the bound arguments.
False
*: actually the partial object is not a function instance, but it is a callable, and from a duck-type perspective it's a function.
If new_function = check_number(5) is a closure, then you can extract this value using __closure__[0].cell_contents:
Example:
def foo(x):
def inn(y):
return x
return inn
s = foo(5)
print(s.__closure__[0].cell_contents)
Output:
5
I understand your confusion, but:
new_function = check_number(5)
Is calling the function, and the new_function variable gets assigned the return value of the function.
If you have this check_number function:
def check_number(input):
return input > 10
Then it will return False, and new_function will be False. Never <some_library.check_number object at 0x07B35B90>.
If you're getting <some_library.check_number object at 0x07B35B90> then your check_number() function is returning something else.
There are probably several ways to skin this cat. But I'd observe first and foremost that you're not adding python function objects to the functions_to_evaluate list, you're adding the evaluations of functions.
You could simply add a tuple of function, args to the list:
SomeClassInstace.functions_to_evaluate.append((check_number, 5))
And then you can:
for f, args in SomeClassInstance.functions_to_evaluate:
print(args)
I have a base decorator that takes arguments but that also is built upon by other decorators. I can't seem to figure where to put the functools.wraps in order to preserve the full signature of the decorated function.
import inspect
from functools import wraps
# Base decorator
def _process_arguments(func, *indices):
""" Apply the pre-processing function to each selected parameter """
#wraps(func)
def wrap(f):
#wraps(f)
def wrapped_f(*args):
params = inspect.getargspec(f)[0]
args_out = list()
for ind, arg in enumerate(args):
if ind in indices:
args_out.append(func(arg))
else:
args_out.append(arg)
return f(*args_out)
return wrapped_f
return wrap
# Function that will be used to process each parameter
def double(x):
return x * 2
# Decorator called by end user
def double_selected(*args):
return _process_arguments(double, *args)
# End-user's function
#double_selected(2, 0)
def say_hello(a1, a2, a3):
""" doc string for say_hello """
print('{} {} {}'.format(a1, a2, a3))
say_hello('say', 'hello', 'arguments')
The result of this code should be and is:
saysay hello argumentsarguments
However, running help on say_hello gives me:
say_hello(*args, **kwargs)
doc string for say_hello
Everything is preserved except the parameter names.
It seems like I just need to add another #wraps() somewhere, but where?
I experimented with this:
>>> from functools import wraps
>>> def x(): print(1)
...
>>> #wraps(x)
... def xyz(a,b,c): return x
>>> xyz.__name__
'x'
>>> help(xyz)
Help on function x in module __main__:
x(a, b, c)
AFAIK, this has nothing to do with wraps itself, but an issue related to help. Indeed, because help inspects your objects to provide the information, including __doc__ and other attributes, this is why you get this behavior, although your wrapped function has different argument list. Though, wraps doesn't update that automatically (the argument list) what it really updates is this tuple and the __dict__ which is technically the objects namespace:
WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__',
'__annotations__')
WRAPPER_UPDATES = ('__dict__',)
If you aren't sure about how wraps work, probably it'll help if your read the the source code from the standard library: functools.py.
It seems like I just need to add another #wraps() somewhere, but where?
No, you don't need to add another wraps in your code, help as I stated above works that way by inspecting your objects. The function's arguments are associated with code objects (__code__) because your function's arguments are stored/represented in that object, wraps has no way to update the argument of the wrapper to be like the wrapped function (continuing with the above example):
>>> xyz.__code__.co_varnames
>>> xyz.__code__.co_varnames = x.__code__.co_varnames
AttributeError: readonly attribute
If help displayed that function xyz has this argument list () instead of (a, b, c) then this is clearly wrong! And the same applies for wraps, to change the argument list of the wrapper to the wrapped, would be cumbersome! So this should not be a concern at all.
>>> #wraps(x, ("__code__",))
... def xyz(a,b,c): pass
...
>>> help(xyz)
Help on function xyz in module __main__:
xyz()
But xyz() returns x():
>>> xyz()
1
For other references take a look at this question or the Python Documentation
What does functools.wraps do?
direprobs was correct in that no amount of functools wraps would get me there. bravosierra99 pointed me to somewhat related examples. However, I couldn't find a single example of signature preservation on nested decorators in which the outer decorator takes arguments.
The comments on Bruce Eckel's post on decorators with arguments gave me the biggest hints in achieving my desired result.
The key was in removing the middle function from within my _process_arguments function and placing its parameter in the next, nested function. It kind of makes sense to me now...but it works:
import inspect
from decorator import decorator
# Base decorator
def _process_arguments(func, *indices):
""" Apply the pre-processing function to each selected parameter """
#decorator
def wrapped_f(f, *args):
params = inspect.getargspec(f)[0]
args_out = list()
for ind, arg in enumerate(args):
if ind in indices:
args_out.append(func(arg))
else:
args_out.append(arg)
return f(*args_out)
return wrapped_f
# Function that will be used to process each parameter
def double(x):
return x * 2
# Decorator called by end user
def double_selected(*args):
return _process_arguments(double, *args)
# End-user's function
#double_selected(2, 0)
def say_hello(a1, a2,a3):
""" doc string for say_hello """
print('{} {} {}'.format(a1, a2, a3))
say_hello('say', 'hello', 'arguments')
print(help(say_hello))
And the result:
saysay hello argumentsarguments
Help on function say_hello in module __main__:
say_hello(a1, a2, a3)
doc string for say_hello
[I am using python 2.7]
I wanted to make a little wrapper function that add one output to a function. Something like:
def add_output(fct, value):
return lambda *args, **kargs: (fct(*args,**kargs),value)
Example of use:
def f(a): return a+1
g = add_output(f,42)
print g(12) # print: (13,42)
This is the expected results, but it does not work if the function given to add_ouput return more than one output (nor if it returns no output). In this case, the wrapped function will return two outputs, one contains all the output of the initial function (or None if it returns no output), and one with the added output:
def f1(a): return a,a+1
def f2(a): pass
g1 = add_output(f1,42)
g2 = add_output(f2,42)
print g1(12) # print: ((12,13),42) instead of (12,13,42)
print g2(12) # print: (None,42) instead of 42
I can see this is related to the impossibility to distinguish between one output of type tuple and several output. But this is disappointing not to be able to do something so simple with a dynamic language like python...
Does anyone have an idea on a way to achieve this automatically and nicely enough, or am I in a dead-end ?
Note:
In case this change anything, my real purpose is doing some wrapping of class (instance) method, to looks like function (for workflow stuff). However it is require to add self in the output (in case its content is changed):
class C(object):
def f(self): return 'foo','bar'
def wrap(method):
return lambda self, *args, **kargs: (self,method(self,*args,**kargs))
f = wrap(C.f)
c = C()
f(c) # returns (c,('foo','bar')) instead of (c,'foo','bar')
I am working with python 2.7, so I a want solution with this version or else I abandon the idea. I am still interested (and maybe futur readers) by comments about this issue for python 3 though.
Your add_output() function is what is called a decorator in Python. Regardless, you can use one of the collections module's ABCs (Abstract Base Classes) to distinguish between different results from the function being wrapped. For example:
import collections
def add_output(fct, value):
def wrapped(*args, **kwargs):
result = fct(*args, **kwargs)
if isinstance(result, collections.Sequence):
return tuple(result) + (value,)
elif result is None:
return value
else: # non-None and non-sequence
return (result, value)
return wrapped
def f1(a): return a,a+1
def f2(a): pass
g1 = add_output(f1, 42)
g2 = add_output(f2, 42)
print g1(12) # -> (12,13,42)
print g2(12) # -> 42
Depending of what sort of functions you plan on decorating, you might need to use the collections.Iterable ABC instead of, or in addition to, collections.Sequence.
I appologize, if I didn't express my self clearly. What I want to do is this:
class someClass(object):
def aMethod(self, argument):
return some_data #for example a list or a more complex datastructure
def aMethod_max(self, argument):
var = self.aMethod(argument)
#do someting with var
return altered_var
or I could do:
def aMethod(self, argument):
self.someVar = some_data
return some_data #for example a list or a more complex datastructure
def aMethod_max(self, argument):
if not hasattr(self, someVar):
self.aMethod(argument)
#do someting with self.var
return altered_var
But I considered this too complicated and hoped for a more elegant solution. I hope that it's clear now, what I want to accomplish.
Therefore I phantasized about something like in the following paragraph.
class someClass(object):
someMethod(self):
#doSomething
return result
subMethod(self):
#doSomething with the result of someMethod
Foo = someClass()
Foo.someMethod.subMethod()
or if someMethod has an argument something like
Foo.someMethod(argument).subMethod()
How would I do something like this in python?
EDIT: or like this?
subMethod(self):
var = self.someMethod()
return doSomething(var)
Let's compare the existing solutions already given in your question (e.g. the ones you call "complicated" and "inelegant") with your proposed alternative.
The existing solutions mean you will be able to write:
foo.subMethod() # foo.someMethod() is called internally
but your proposed alternative means you have to write:
foo.someMethod().subMethod()
which is obviously worse.
On the other hand, if subMethod has to be able to modify the result of any method, rather than just someMethod, then the existing solutions would mean you have to write:
foo.subMethod(foo.anyMethod())
with the only disadvantage here being that you have to type foo twice, as opposed to once.
Conclusion: on the whole, the existing solutions are less complicated and inelegant than your proposed alternative - so stick with the existing solutions.
You can do method chaining when the result of someMethod is an instance of someClass.
Simple example:
>>> class someClass:
... def someMethod(self):
... return self
... def subMethod(self):
... return self.__class__
...
>>> x=someClass()
>>> x
<__main__.someClass instance at 0x2aaaaab30d40>
>>> x.someMethod().subMethod()
<class __main__.someClass at 0x2aaaaab31050>
Not sure if I'm understanding it right, but perhaps you mean this:
Foo.subMethod(Foo.someMethod())
This passes the result of someMethod() to subMethod(). You'd have to change your current definition of subMethod() to accept the result of someMethod().
You can achieve something similar using decorators:
def on_result(f):
def decorated(self,other,*args,**kwargs):
result = getattr(self,other)(*args,**kwargs)
return f(result)
return decorated
Usage:
class someClass(object):
def someMethod(self,x,y):
#doSomething
result = [1,2,3,x,y] # example
return result
#on_result
def subMethod(self):
#doSomething with the result of someMethod
print self # example
Foo = someClass()
Foo.subMethod("someMethod",4,5)
Output:
[1, 2, 3, 4, 5]
As you see, the first argument is the name of the method to be chained, and the remaining ones will be passed to it, no matter what its signature is.
EDIT: on second thought, this is rather pointless, since you could always use
Foo.submethod(Foo.someMethod(4,5))...
Maybe I didn't understand what you're trying to achieve. Does the subMethod have to be linked to a specific method only? Or maybe it's the syntatic form
a.b().c()
that's important to you? (in that case, see kojiro's answer)
From the feedback so far, I understand that subMethod will link only to someMethod, right? Maybe you can achieve this combining a decorator with a closure:
def on_result(other):
def decorator(f):
def decorated(self,*a1,**k1):
def closure(*a2,**k2):
return f(self,getattr(self,other)(*a1,**k1),*a2,**k2)
return closure
return decorated
return decorator
class someClass(object):
def someMethod(self,a,b):
return [a,2*b]
#on_result('someMethod')
def subMethod(self,result,c,d):
result.extend([3*c,4*d])
return result
Foo = someClass()
print Foo.subMethod(1,2)(3,4) # prints [1,4,9,16]
The decorator is kinda "ugly", but once written it's usage is quite elegant IMHO (plus, there are no contraints in the signature of either method).
Note: I'm using Python 2.5 and this is the only way I know of writing decorators that take arguments. There's probably a better way, but I'm too lazy to look it up right now...
I've been playing around in depth with attempting to write my own version of a memoizing decorator before I go looking at other people's code. It's more of an exercise in fun, honestly. However, in the course of playing around I've found I can't do something I want with decorators.
def addValue( func, val ):
def add( x ):
return func( x ) + val
return add
#addValue( val=4 )
def computeSomething( x ):
#function gets defined
If I want to do that I have to do this:
def addTwo( func ):
return addValue( func, 2 )
#addTwo
def computeSomething( x ):
#function gets defined
Why can't I use keyword arguments with decorators in this manner? What am I doing wrong and can you show me how I should be doing it?
You need to define a function that returns a decorator:
def addValue(val):
def decorator(func):
def add(x):
return func(x) + val
return add
return decorator
When you write #addTwo, the value of addTwo is directly used as a decorator. However, when you write #addValue(4), first addValue(4) is evaluated by calling the addValue function. Then the result is used as a decorator.
You want to partially apply the function addValue - give the val argument, but not func. There are generally two ways to do this:
The first one is called currying and used in interjay's answer: instead of a function with two arguments, f(a,b) -> res, you write a function of the first arg that returns another function that takes the 2nd arg g(a) -> (h(b) -> res)
The other way is a functools.partial object. It uses inspection on the function to figure out what arguments a function needs to run (func and val in your case ). You can add extra arguments when creating a partial and once you call the partial, it uses all the extra arguments given.
from functools import partial
#partial(addValue, val=2 ) # you can call this addTwo
def computeSomething( x ):
return x
Partials are usually a much simpler solution for this partial application problem, especially with more than one argument.
Decorators with any kinds of arguments -- named/keyword ones, unnamed/positional ones, or some of each -- essentially, ones you call on the #name line rather than just mention there -- need a double level of nesting (while the decorators you just mention have a single level of nesting). That goes even for argument-less ones if you want to call them in the # line -- here's the simplest, do-nothing, double-nested decorator:
def double():
def middling():
def inner(f):
return f
return inner
return middling
You'd use this as
#double()
def whatever ...
note the parentheses (empty in this case since there are no arguments needed nor wanted): they mean you're calling double, which returns middling, which decorates whatever.
Once you've seen the difference between "calling" and "just mentioning", adding (e.g. optional) named args is not hard:
def doublet(foo=23):
def middling():
def inner(f):
return f
return inner
return middling
usable either as:
#doublet()
def whatever ...
or as:
#doublet(foo=45)
def whatever ...
or equivalently as:
#doublet(45)
def whatever ...