In Python I can do something like this:
def wrap(f):
def wrapper(*args, **kwargs):
print "args: ", args, kwargs
res = f(*args, **kwargs)
print "result: ", res
return res
return wrapper
This lets me wrap any function regardless of the arguments they take. For instance:
In [8]: def f(thing):
print "in f:", thing
return 3
In [9]: wrapped_f = wrap(f)
In [10]: wrapped_f(2)
args: (2,) {}
in f: 2
result: 3
Out[10]: 3
Is there a way to do something similar (write a wrapper that can be applied to any function regardless of its input/output types) in Scala?
You could certainly do this with macros. You can convert a method call to a function with partial application:
object Foo {
def bar(i: Int): Int = i + 1
}
val fn = Foo.bar _
defined object Foo
fn: Int => Int = <function1>
Now you have an object, in this case of type Function1[Int, Int], which you can pass to a Scala macro, which would be something like this (not tested):
object DecoratorMacros {
import reflect.macros.blackbox
import language.experimental.macros
def decorate[A <: Function](fn: A): [A] = macro decorate_impl[A]
def decorate_impl[A: c.WeakTypeTag](c: blackbox.Context) = {
import c.universe._
val type = weakTypeOf[A]
...
}
}
In the body of the macro, you can inspect the whole type signature of fn: A, which will include the arguments. You can then write code to do your desired side effects, and return a function which you can then invoke. Something like this:
DecoratorMacros.decorate(Foo.bar _)(42)
Macros are fairly involved, but I can elaborate if you think this is a path you'd like to go down.
There is a fundamental issue here: in Scala you have to know what arguments the function should get and actually pass them so that the compiler can be sure that the types match.
Say there is def f(a: List[Int], b: String) = ... and def g(args: Any*) = f(args). This won't compile! (Any* means any amount of objects with any type). The problem is that Any* is still only one single argument which actually is translated to one kind of Array.
Just to make this more clear you could think of an example situation: you have called the wrap(f) with some function f(a: String, b: String). Then you have the output of the wrapper which would somehow accept any amount of any kind of arguments and you make the call wrapper_f(List(1), "a"). In this situation the wrapper_f(...) call should be correct but inside the wrapper the wrapped function has a completely different parameter list which can not accept a List[Int] and a String. Thus you would get the "Type Error" in runtime which should (in general) be impossible in statically typed programming languages (or at least in Scala).
Related
Why doesnt this give back '12'?
The '+' sign should concatenate two strings, not add them.
def foo(a:str, b:str):
print(a+b)
foo(1,2)
3
That's not what annotations are for. Annotations are metadata, not an instruction to Python to convert data.
From the Function definitions reference documentation:
Parameters may have annotations of the form “: expression” following the parameter name. Any parameter may have an annotation even those of the form *identifier or **identifier. Functions may have “return” annotation of the form “-> expression” after the parameter list. These annotations can be any valid Python expression and are evaluated when the function definition is executed. Annotations may be evaluated in a different order than they appear in the source code. The presence of annotations does not change the semantics of a function.
(Bold emphisis mine).
For example, the Python type hinting framework uses annotations to attach type information to functions for static analysis, validating that code actually passes in the types that are expected to be passed in.
Just convert your values explicitly; in the call:
foo(str(1), str(2))
or in the function itself:
def foo(a, b):
print(str(a) + str(b))
or in a decorator:
import functools
import inspect
def typeconversion(f):
"""Converts arguments with a callable attached in the parameter annotation"""
sig = inspect.signature(f)
#functools.wraps(f)
def wrapper(*args, **kwargs):
# convert any argument (including defaults), for which there is a
# callable annotation
bound = sig.bind(*args, **kwargs)
bound.apply_defaults()
args = bound.arguments
for param in sig.parameters.values():
if param.annotation is not param.empty and callable(param.annotation):
args[param.name] = param.annotation(args[param.name])
# call the function with the converted arguments
result = f(*bound.args, **bound.kwargs)
# convert the return value
if sig.return_annotation is not sig.empty and callable(sig.return_annotation):
result = sig.return_annotation(result)
return result
return wrapper
Demo:
>>> #typeconversion
... def foo(a: str, b: str) -> int:
... return a + b
...
>>> foo(42, 101)
42101
I have a base decorator that takes arguments but that also is built upon by other decorators. I can't seem to figure where to put the functools.wraps in order to preserve the full signature of the decorated function.
import inspect
from functools import wraps
# Base decorator
def _process_arguments(func, *indices):
""" Apply the pre-processing function to each selected parameter """
#wraps(func)
def wrap(f):
#wraps(f)
def wrapped_f(*args):
params = inspect.getargspec(f)[0]
args_out = list()
for ind, arg in enumerate(args):
if ind in indices:
args_out.append(func(arg))
else:
args_out.append(arg)
return f(*args_out)
return wrapped_f
return wrap
# Function that will be used to process each parameter
def double(x):
return x * 2
# Decorator called by end user
def double_selected(*args):
return _process_arguments(double, *args)
# End-user's function
#double_selected(2, 0)
def say_hello(a1, a2, a3):
""" doc string for say_hello """
print('{} {} {}'.format(a1, a2, a3))
say_hello('say', 'hello', 'arguments')
The result of this code should be and is:
saysay hello argumentsarguments
However, running help on say_hello gives me:
say_hello(*args, **kwargs)
doc string for say_hello
Everything is preserved except the parameter names.
It seems like I just need to add another #wraps() somewhere, but where?
I experimented with this:
>>> from functools import wraps
>>> def x(): print(1)
...
>>> #wraps(x)
... def xyz(a,b,c): return x
>>> xyz.__name__
'x'
>>> help(xyz)
Help on function x in module __main__:
x(a, b, c)
AFAIK, this has nothing to do with wraps itself, but an issue related to help. Indeed, because help inspects your objects to provide the information, including __doc__ and other attributes, this is why you get this behavior, although your wrapped function has different argument list. Though, wraps doesn't update that automatically (the argument list) what it really updates is this tuple and the __dict__ which is technically the objects namespace:
WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__',
'__annotations__')
WRAPPER_UPDATES = ('__dict__',)
If you aren't sure about how wraps work, probably it'll help if your read the the source code from the standard library: functools.py.
It seems like I just need to add another #wraps() somewhere, but where?
No, you don't need to add another wraps in your code, help as I stated above works that way by inspecting your objects. The function's arguments are associated with code objects (__code__) because your function's arguments are stored/represented in that object, wraps has no way to update the argument of the wrapper to be like the wrapped function (continuing with the above example):
>>> xyz.__code__.co_varnames
>>> xyz.__code__.co_varnames = x.__code__.co_varnames
AttributeError: readonly attribute
If help displayed that function xyz has this argument list () instead of (a, b, c) then this is clearly wrong! And the same applies for wraps, to change the argument list of the wrapper to the wrapped, would be cumbersome! So this should not be a concern at all.
>>> #wraps(x, ("__code__",))
... def xyz(a,b,c): pass
...
>>> help(xyz)
Help on function xyz in module __main__:
xyz()
But xyz() returns x():
>>> xyz()
1
For other references take a look at this question or the Python Documentation
What does functools.wraps do?
direprobs was correct in that no amount of functools wraps would get me there. bravosierra99 pointed me to somewhat related examples. However, I couldn't find a single example of signature preservation on nested decorators in which the outer decorator takes arguments.
The comments on Bruce Eckel's post on decorators with arguments gave me the biggest hints in achieving my desired result.
The key was in removing the middle function from within my _process_arguments function and placing its parameter in the next, nested function. It kind of makes sense to me now...but it works:
import inspect
from decorator import decorator
# Base decorator
def _process_arguments(func, *indices):
""" Apply the pre-processing function to each selected parameter """
#decorator
def wrapped_f(f, *args):
params = inspect.getargspec(f)[0]
args_out = list()
for ind, arg in enumerate(args):
if ind in indices:
args_out.append(func(arg))
else:
args_out.append(arg)
return f(*args_out)
return wrapped_f
# Function that will be used to process each parameter
def double(x):
return x * 2
# Decorator called by end user
def double_selected(*args):
return _process_arguments(double, *args)
# End-user's function
#double_selected(2, 0)
def say_hello(a1, a2,a3):
""" doc string for say_hello """
print('{} {} {}'.format(a1, a2, a3))
say_hello('say', 'hello', 'arguments')
print(help(say_hello))
And the result:
saysay hello argumentsarguments
Help on function say_hello in module __main__:
say_hello(a1, a2, a3)
doc string for say_hello
[I am using python 2.7]
I wanted to make a little wrapper function that add one output to a function. Something like:
def add_output(fct, value):
return lambda *args, **kargs: (fct(*args,**kargs),value)
Example of use:
def f(a): return a+1
g = add_output(f,42)
print g(12) # print: (13,42)
This is the expected results, but it does not work if the function given to add_ouput return more than one output (nor if it returns no output). In this case, the wrapped function will return two outputs, one contains all the output of the initial function (or None if it returns no output), and one with the added output:
def f1(a): return a,a+1
def f2(a): pass
g1 = add_output(f1,42)
g2 = add_output(f2,42)
print g1(12) # print: ((12,13),42) instead of (12,13,42)
print g2(12) # print: (None,42) instead of 42
I can see this is related to the impossibility to distinguish between one output of type tuple and several output. But this is disappointing not to be able to do something so simple with a dynamic language like python...
Does anyone have an idea on a way to achieve this automatically and nicely enough, or am I in a dead-end ?
Note:
In case this change anything, my real purpose is doing some wrapping of class (instance) method, to looks like function (for workflow stuff). However it is require to add self in the output (in case its content is changed):
class C(object):
def f(self): return 'foo','bar'
def wrap(method):
return lambda self, *args, **kargs: (self,method(self,*args,**kargs))
f = wrap(C.f)
c = C()
f(c) # returns (c,('foo','bar')) instead of (c,'foo','bar')
I am working with python 2.7, so I a want solution with this version or else I abandon the idea. I am still interested (and maybe futur readers) by comments about this issue for python 3 though.
Your add_output() function is what is called a decorator in Python. Regardless, you can use one of the collections module's ABCs (Abstract Base Classes) to distinguish between different results from the function being wrapped. For example:
import collections
def add_output(fct, value):
def wrapped(*args, **kwargs):
result = fct(*args, **kwargs)
if isinstance(result, collections.Sequence):
return tuple(result) + (value,)
elif result is None:
return value
else: # non-None and non-sequence
return (result, value)
return wrapped
def f1(a): return a,a+1
def f2(a): pass
g1 = add_output(f1, 42)
g2 = add_output(f2, 42)
print g1(12) # -> (12,13,42)
print g2(12) # -> 42
Depending of what sort of functions you plan on decorating, you might need to use the collections.Iterable ABC instead of, or in addition to, collections.Sequence.
I've been playing around in depth with attempting to write my own version of a memoizing decorator before I go looking at other people's code. It's more of an exercise in fun, honestly. However, in the course of playing around I've found I can't do something I want with decorators.
def addValue( func, val ):
def add( x ):
return func( x ) + val
return add
#addValue( val=4 )
def computeSomething( x ):
#function gets defined
If I want to do that I have to do this:
def addTwo( func ):
return addValue( func, 2 )
#addTwo
def computeSomething( x ):
#function gets defined
Why can't I use keyword arguments with decorators in this manner? What am I doing wrong and can you show me how I should be doing it?
You need to define a function that returns a decorator:
def addValue(val):
def decorator(func):
def add(x):
return func(x) + val
return add
return decorator
When you write #addTwo, the value of addTwo is directly used as a decorator. However, when you write #addValue(4), first addValue(4) is evaluated by calling the addValue function. Then the result is used as a decorator.
You want to partially apply the function addValue - give the val argument, but not func. There are generally two ways to do this:
The first one is called currying and used in interjay's answer: instead of a function with two arguments, f(a,b) -> res, you write a function of the first arg that returns another function that takes the 2nd arg g(a) -> (h(b) -> res)
The other way is a functools.partial object. It uses inspection on the function to figure out what arguments a function needs to run (func and val in your case ). You can add extra arguments when creating a partial and once you call the partial, it uses all the extra arguments given.
from functools import partial
#partial(addValue, val=2 ) # you can call this addTwo
def computeSomething( x ):
return x
Partials are usually a much simpler solution for this partial application problem, especially with more than one argument.
Decorators with any kinds of arguments -- named/keyword ones, unnamed/positional ones, or some of each -- essentially, ones you call on the #name line rather than just mention there -- need a double level of nesting (while the decorators you just mention have a single level of nesting). That goes even for argument-less ones if you want to call them in the # line -- here's the simplest, do-nothing, double-nested decorator:
def double():
def middling():
def inner(f):
return f
return inner
return middling
You'd use this as
#double()
def whatever ...
note the parentheses (empty in this case since there are no arguments needed nor wanted): they mean you're calling double, which returns middling, which decorates whatever.
Once you've seen the difference between "calling" and "just mentioning", adding (e.g. optional) named args is not hard:
def doublet(foo=23):
def middling():
def inner(f):
return f
return inner
return middling
usable either as:
#doublet()
def whatever ...
or as:
#doublet(foo=45)
def whatever ...
or equivalently as:
#doublet(45)
def whatever ...
My idea of program:
I have a dictionary:
options = { 'string' : select_fun(function pointer),
'float' : select_fun(function pointer),
'double' : select_fun(function pointer)
}
whatever type comes single function select_fun(function pointer) gets called.
Inside select_fun(function pointer),I will have diff functions for float, double and so on.
Depending on function pointers, specified function will get called.
I don't know whether my programming knowledge is good or bad, still I need help.
Could you be more specific on what you're trying to do? You don't have to do anything special to get function pointers in Python -- you can pass around functions like regular objects:
def plus_1(x):
return x + 1
def minus_1(x):
return x - 1
func_map = {'+' : plus_1, '-' : minus_1}
func_map['+'](3) # returns plus_1(3) ==> 4
func_map['-'](3) # returns minus_1(3) ==> 2
You can use the type() built-in function to detect the type of the function.
Say, if you want to check if a certain name hold a string data, you could do this:
if type(this_is_string) == type('some random string'):
# this_is_string is indeed a string
So in your case, you could do it like this:
options = { 'some string' : string_function,
(float)(123.456) : float_function,
(int)(123) : int_function
}
def call_option(arg):
# loop through the dictionary
for (k, v) in options.iteritems():
# if found matching type...
if type(k) == type(arg):
# call the matching function
func = option[k]
func(arg)
Then you can use it like this:
call_option('123') # string_function gets called
call_option(123.456) # float_function gets called
call_option(123) # int_function gets called
I don't have a python interpreter nearby and I don't program in Python much so there may be some errors, but you should get the idea.
EDIT: As per #Adam's suggestion, there are built-in type constants that you can check against directly, so a better approach would be:
from types import *
options = { types.StringType : string_function,
types.FloatType : float_function,
types.IntType : int_function,
types.LongType : long_function
}
def call_option(arg):
for (k, v) in options.iteritems():
# check if arg is of type k
if type(arg) == k:
# call the matching function
func = options[k]
func(arg)
And since the key itself is comparable to the value of the type() function, you can just do this:
def call_option(arg):
func = options[type(arg)]
func(arg)
Which is more elegant :-) save for some error-checking.
EDIT: And for ctypes support, after some fiddling around, I've found that ctypes.[type_name_here] is actually implented as classes. So this method still works, you just need to use the ctypes.c_xxx type classes.
options = { ctypes.c_long : c_long_processor,
ctypes.c_ulong : c_unsigned_long_processor,
types.StringType : python_string_procssor
}
call_option = lambda x: options[type(x)](x)
Looking at your example, it seems to me some C procedure, directly translated to Python.
For this reason, I think there could be some design issue, because usually, in Python, you do not care about type of an object, but only about the messages you can send to it.
Of course, there are plenty of exceptions to this approach, but still in this case I would try encapsulating in some polymorphism; eg.
class StringSomething(object):
data = None
def data_function(self):
string_function_pointer(self.data)
class FloatSomething(object):
data = None
def data_function(self):
float_function_pointer(self.data)
etc.
Again, all of this under the assumption you are translating from a procedural language to python; if it is not the case, then discard my answer :-)
Functions are the first-class objects in Python therefore you can pass them as arguments to other functions as you would with any other object such as string or an integer.
There is no single-precision floating point type in Python. Python's float corresponds to C's double.
def process(anobject):
if isinstance(anobject, basestring):
# anobject is a string
fun = process_string
elif isinstance(anobject, (float, int, long, complex)):
# anobject is a number
fun = process_number
else:
raise TypeError("expected string or number but received: '%s'" % (
type(anobject),))
return fun(anobject)
There is functools.singledispatch that allows to create a generic function:
from functools import singledispatch
from numbers import Number
#singledispatch
def process(anobject): # default implementation
raise TypeError("'%s' type is not supported" % type(anobject))
#process.register(str)
def _(anobject):
# handle strings here
return process_string(anobject)
process.register(Number)(process_number) # use existing function for numbers
On Python 2, similar functionality is available as pkgutil.simplegeneric().
Here's a couple of code example of using generic functions:
Remove whitespaces and newlines from JSON file
Make my_average(a, b) work with any a and b for which f_add and d_div are defined. As well as builtins
Maybe you want to call the same select_fun() every time, with a different argument. If that is what you mean, you need a different dictionary:
>>> options = {'string' : str, 'float' : float, 'double' : float }
>>> options
{'double': <type 'float'>, 'float': <type 'float'>, 'string': <type 'str'>}
>>> def call_option(val, func):
... return func(val)
...
>>> call_option('555',options['float'])
555.0
>>>