I'm trying to make sure running help() at the Python 2.7 REPL displays the __doc__ for a function that was wrapped with functools.partial. Currently running help() on a functools.partial 'function' displays the __doc__ of the functools.partial class, not my wrapped function's __doc__. Is there a way to achieve this?
Consider the following callables:
def foo(a):
"""My function"""
pass
partial_foo = functools.partial(foo, 2)
Running help(foo) will result in showing foo.__doc__. However, running help(partial_foo) results in the __doc__ of a Partial object.
My first approach was to use functools.update_wrapper which correctly replaces the partial object's __doc__ with foo.__doc__. However, this doesn't fix the 'problem' because of how pydoc.
I've investigated the pydoc code, and the issue seems to be that partial_foo is actually a Partial object not a typical function/callable, see this question for more information on that detail.
By default, pydoc will display the __doc__ of the object type, not instance if the object it was passed is determined to be a class by inspect.isclass. See the render_doc function for more information about the code itself.
So, in my scenario above pydoc is displaying the help of the type, functools.partial NOT the __doc__ of my functools.partial instance.
Is there anyway to make alter my call to help() or functools.partial instance that's passed to help() so that it will display the __doc__ of the instance, not type?
I found a pretty hacky way to do this. I wrote the following function to override the __builtins__.help function:
def partialhelper(object=None):
if isinstance(object, functools.partial):
return pydoc.help(object.func)
else:
# Preserve the ability to go into interactive help if user calls
# help() with no arguments.
if object is None:
return pydoc.help()
else:
return pydoc.help(object)
Then just replace it in the REPL with:
__builtins__.help = partialhelper
This works and doesn't seem to have any major downsides, yet. However, there isn't a way with the above naive implementation to support still showing the __doc__ of some functools.partial objects. It's all or nothing, but could probably attach an attribute to the wrapped (original) function to indicate whether or not the original __doc__ should be shown. However, in my scenario I never want to do this.
Note the above does NOT work when using IPython and the embed functionality. This is because IPython directly sets the shell's namespace with references to the 'real' __builtin__, see the code and old mailing list for information on why this is.
So, after some investigation there's another way to hack this into IPython. We must override the site._Helper class, which is used by IPython to explicitly setup the help system. The following code will do just that when called BEFORE IPython.embed:
import site
site._Helper.__call__ = lambda self, *args, **kwargs: partialhelper(*args, **kwargs)
Are there any other downsides I'm missing here?
how bout implementing your own?
def partial_foo(*args):
""" some doc string """
return foo(*((2)+args))
not a perfect answer but if you really want this i suspect this is the only way to do it
You identified the issue - partial functions aren't typical functions, and the dunder variables don't carry over. This applies not just to __doc__, but also __name__, __module__, and more. Not sure if this solution existed when the question was asked, but you can achieve this more elegantly ("elegantly" up to interpretation) by re-writing partial() as a decorator factory. Since decorators (& factories) do not automatically copy over dunder variables, you need to also use #wraps(func):
def wrapped_partial(*args, **kwargs):
def foo(func):
#wraps(func)
def bar(*fargs,**fkwargs):
return func(*args, *fargs, **kwargs, **fkwargs)
return bar
return foo
Usage example:
#wrapped_partial(3)
def multiply_triple(x, y=1, z=0):
"""Multiplies three numbers"""
return x * y * z
# Without decorator syntax: multiply_triple = wrapped_partial(3)(multiply_triple)
With output:
>>>print(multiply_triple())
0
>>>print(multiply_triple(3,z=3))
9
>>>help(multiply_triple)
help(multiply_triple)
Help on function multiply_triple in module __main__:
multiply_triple(x: int, y: int = 1, z: int = 0)
Multiplies three numbers
Thing that didn't work, but informative when using multiple decorators
You might think, as I first did, that based upon the stacking syntax of decorators in PEP-318, you could put the wrapping and the partial function definition in separate decorators, e.g.
def partial_func(*args, **kwargs):
def foo(func):
def bar(*fargs,**fkwargs):
return func(*args, *fargs, **kwargs, **fkwargs)
return bar
return foo
def wrapped(f):
#wraps(f)
def wrapper(*args, **kwargs):
return f(*args, **kwargs)
return wrapper
#wrapped
#partial_func(z=3)
def multiply_triple(x, y=1, z=0):
"""Multiplies three numbers"""
return x * y * z
In these cases (and in reverse order), the decorators are applied one at a time, and the #partial_func interrupts wrapping. This means that if you are trying to use any decorator that you want to wrap, you need to rewrite the decorator in a factory where the decorator's return function is itself decorated by #wraps(func). If you are using multiple decorators, they all have to be turned into wrapped factories.
Alternate method to have decorators "wrap"
Since decorators are just functions, you can write a copy_dunder_vars(obj1, obj2) function that retruns obj2 but with all the dunder variables from obj1. Call as:
def foo()
pass
foo = copy_dunder_vars(decorator(foo), foo)
This goes against the preferred syntax, but practicality beats purity. I think "not forcing you to rewrite decorators that you're borrowing from elsewhere and leaving largely unchanged" fits into that category. After all that wrapping, don't forget ribbon and a bow ;)
Related
I am using functions from external librairies (such as scipy.optimize.minimize) and for conveniance I have wrapped them up to pre-process and post-process the arguments. I am aware of the wraps decorator from functools but I can't find a way to use it properly.
Here is basic example of what I do:
def wrapper(outer_arg, **inner_arguments):
inner_arg = prepare_inputs(outer_arg)
inner_output = inner_function(inner_arg, **inner_arguments)
outer_output = prepare_outputs(inner_output)
return outer_output
The problem with this is that I loose the docstring from inner_function when calling help and I do not want to copy all the arguments in the wrapper declaration.
I have tried to add the wraps decorator like so:
#wraps(inner_function)
def wrapper(outer_arg, **inner_arguments):
inner_arg = prepare_inputs(outer_arg)
inner_output = inner_function(inner_arg, **inner_arguments)
outer_output = prepare_outputs(inner_output)
return outer_output
But then I loose the docstring of wrapper when calling help.
Is there a way to write this function that would give the user insight on both the wrapper and the inner_function (and be able to use autocompletion etc...) ?
Using #deceze idea I built a decorator to format the wrapped function's doctring:
def merge_docstrings(other):
def decorator(func):
func.__doc__ = f'{func.__doc__}'.format(other.__doc__)
return func
return decorator
It does format the wrapper doctring with the one of inner_function
But it doesn't work well when displaying the docstring in PyCharm for example, and obviously you don't get auto-completion with the arguments of inner_function. I am not sure how auto-completion work and if it would be possible to achieve what I want.
A Python class's __call__ method lets us specify how a class member should be behave as a function. Can we do the "opposite", i.e. specify how a class member should behave as an argument to an arbitrary other function?
As a simple example, suppose I have a ListWrapper class that wraps lists, and when I call a function f on a member of this class, I want f to be mapped over the wrapped list. For instance:
x = WrappedList([1, 2, 3])
print(x + 1) # should be WrappedList([2, 3, 4])
d = {1: "a", 2: "b", 3:"c"}
print(d[x]) # should be WrappedList(["a", "b", "c"])
Calling the hypothetical __call__ analogue I'm looking for __arg__, we could imagine something like this:
class WrappedList(object):
def __init__(self, to_wrap):
self.wrapped = to_wrap
def __arg__(self, func):
return WrappedList(map(func, self.wrapped))
Now, I know that (1) __arg__ doesn't exist in this form, and (2) it's easy to get the behavior in this simple example without any tricks. But is there a way to approximate the behavior I'm looking for in the general case?
You can't do this in general.*
You can do something equivalent for most of the builtin operators (like your + example), and a handful of builtin functions (like abs). They're implemented by calling special methods on the operands, as described in the reference docs.
Of course that means writing a whole bunch of special methods for each of your types—but it wouldn't be too hard to write a base class (or decorator or metaclass, if that doesn't fit your design) that implements all those special methods in one place, by calling the subclass's __arg__ and then doing the default thing:
class ArgyBase:
def __add__(self, other):
return self.__arg__() + other
def __radd__(self, other):
return other + self.__arg__()
# ... and so on
And if you want to extend that to a whole suite of functions that you create yourself, you can give them all similar special-method protocols similar to the builtin ones, and expand your base class to cover them. Or you can just short-circuit that and use the __arg__ protocol directly in those functions. To avoid lots of repetition, I'd use a decorator for that.
def argify(func):
def _arg(arg):
try:
return arg.__arg__()
except AttributeError:
return arg
#functools.wraps(func)
def wrapper(*args, **kwargs):
args = map(_arg, args)
kwargs = {kw: _arg(arg) for arg in args}
return func(*args, **kwargs)
return wrapper
#argify
def spam(a, b):
return a + 2 * b
And if you really want to, you can go around wrapping other people's functions:
sin = argify(math.sin)
… or even monkeypatching their modules:
requests.get = argify(requests.get)
… or monkeypatching a whole module dynamically a la early versions of gevent, but I'm not going to even show that, because at this point we're getting into don't-do-this-for-multiple-reasons territory.
You mentioned in a comment that you'd like to do this to a bunch of someone else's functions without having to specify them in advance, if possible. Does that mean every function that ever gets constructed in any module you import? Well, you can even do that if you're willing to create an import hook, but that seems like an even worse idea. Explaining how to write an import hook and either AST-patch each function creation node or insert wrappers around the bytecode or the like is way too much to get into here, but if your research abilities exceed your common sense, you can figure it out. :)
As a side note, if I were doing this, I wouldn't call the method __arg__, I'd call it either arg or _arg. Besides being reserved for future use by the language, the dunder-method style implies things that aren't true here (special-method lookup instead of a normal call, you can search for it in the docs, etc.).
* There are languages where you can, such as C++, where a combination of implicit casting and typed variables instead of typed values means you can get a method called on your objects just by giving them an odd type with an implicit conversion operator to the expected type.
I've only seen examples for setting the __repr__ method in class definitions. Is it possible to change the __repr__ for functions either in their definitions or after defining them?
I've attempted without success...
>>> def f():
pass
>>> f
<function f at 0x1026730c8>
>>> f.__repr__ = lambda: '<New repr>'
>>> f
<function __main__.f>
Yes, if you're willing to forgo the function actually being a function.
First, define a class for our new type:
import functools
class reprwrapper(object):
def __init__(self, repr, func):
self._repr = repr
self._func = func
functools.update_wrapper(self, func)
def __call__(self, *args, **kw):
return self._func(*args, **kw)
def __repr__(self):
return self._repr(self._func)
Add in a decorator function:
def withrepr(reprfun):
def _wrap(func):
return reprwrapper(reprfun, func)
return _wrap
And now we can define the repr along with the function:
#withrepr(lambda x: "<Func: %s>" % x.__name__)
def mul42(y):
return y*42
Now repr(mul42) produces '<Func: mul42>'
No, because repr(f) is done as type(f).__repr__(f) instead.
In order to do that, you'd need to change the __repr__ function for the given class, which in this case is the built-in function class (types.FunctionType). Since in Python you cannot edit built-in classes, only subclass them, you cannot.
However, there are two approaches you could follow:
Wrap some functions as kwatford suggested
Create your own representation protocol with your own repr function. For example, you could define a myrepr function that looks for __myrepr__ methods first, which you cannot add to the function class but you can add it to individual function objects as you suggest (as well as your custom classes and objects), then defaults to repr if __myrepr__ is not found. A possible implementation for this would be:
def myrepr(x):
try:
x.__myrepr__
except AttributeError:
return repr(x)
else:
return x.__myrepr__()
Then you could define __myrepr__ methods and use the myrepr function. Alternatively, you could also do __builtins__.repr = myrepr to make your function the default repr and keep using repr. This approach would end up doing exactly what you want, though editing __builtins__ may not always be desirable.
This appears to be difficult. Kwatford's approach only solves this problem partially since it does not work for functions in classes, becuase self would be treated like a positional argument, as explained in Decorating Python class methods - how do I pass the instance to the decorator? - However, the solution for that question is not applicable to this case, unfortunately, as using __get__() and functools.partial would override the custom __repr__().
I have an class which decorates some methods using a decorator from another library. Specifically, the class subclasses flask-restful resources, decorates the http methods with httpauth.HTTPBasicAuth().login_required(), and does some sensible defaults on a model service.
On most subclasses I want the decorator applied; therefore I'd rather remove it than add it in the subclasses.
My thought is to have a private method which does the operations and a public method which is decorated. The effects of decoration can be avoided by overriding the public method to call the private one and not decorating this override. Mocked example below.
I am curious to know if there's a better way to do this. Is there a shortcut for 'cancelling decorators' in python that gives this effect?
Or can you recommend a better approach?
Some other questions have suitable answers for this, e.g. Is there a way to get the function a decorator has wrapped?. But my question is about broader design - i am interested in any pythonic way to run the operations in decorated methods without the effects of decoration. E.g. my example is one such way but there may be others.
def auth_required(fn):
def new_fn(*args, **kwargs):
print('Auth required for this resource...')
fn(*args, **kwargs)
return new_fn
class Resource:
name = None
#auth_required
def get(self):
self._get()
def _get(self):
print('Getting %s' %self.name)
class Eggs(Resource):
name = 'Eggs'
class Spam(Resource):
name = 'Spam'
def get(self):
self._get()
# super(Spam, self)._get()
eggs = Eggs()
spam = Spam()
eggs.get()
# Auth required for this resource...
# Getting Eggs
spam.get()
# Getting Spam
Flask-HTTPAuth uses functools.wraps in the login_required decorator:
def login_required(self, f):
#wraps(f)
def decorated(*args, **kwargs):
...
From Python 3.2, as this calls update_wrapper, you can access the original function via __wrapped__:
To allow access to the original function for introspection and other
purposes (e.g. bypassing a caching decorator such as lru_cache()),
this function automatically adds a __wrapped__ attribute to the
wrapper that refers to the function being wrapped.
If you're writing your own decorators, as in your example, you can also use #wraps to get the same functionality (as well as keeping the docstrings, etc.).
See also Is there a way to get the function a decorator has wrapped?
Another common option is to have the decorated function keep a copy of the original function that can be accessed:
def auth_required(fn):
def new_fn(*args, **kwargs):
print('Auth required for this resource...')
fn(*args, **kwargs)
new_fn.original_fn = fn
return new_fn
Now, for any function that has been decorated, you can access its original_fn attribute to get a handle to the original, un-decorated function.
In that case, you could define some type of dispatcher that either makes plain function calls (when you are happy with the decorator behavior) or makes calls to thing.original_fn when you prefer to avoid the decorator behavior.
Your proposed method is also a valid way to structure it, and whether my suggestion is "better" depends on the rest of the code you're dealing with, who needs to read it, and other kinds of trade-offs.
I am curious to know if there's a better way to do this. Is there a
shortcut for 'cancelling decorators' in python that gives this effect?
Use the undecorated library. It digs through all the decorators and returns just the original function. The docs should be self-explanatory, basically you just call: undecorated(your_decorated_function)
Is it possible to inspect a function/method to see whether it can be used as a decorator? In that it follows the usual way decorators wrap other functions and return a callable? Specifically, I'm looking to validate 3rd party code.
By applying a suspected decorator, catching exceptions, and then testing whether the result contains a __call__ method, you could produce a guess as to whether a given callable is a decorator or not. But it will be only a guess, not a guarantee.
Beyond that, I do not believe what you want will be possible in general, due to the dynamically typed nature of the Python language and to the special treatment of built-in functions in the CPython interpreter. It is not possible to programmatically tell whether a callable will accept another callable as an argument, or what type its return value will have. Also, in CPython, for functions implemented in C, you cannot even inspect a callable to see how many arguments it accepts.
The word "decorator" can be taken to mean different things. One way to define it is, a decorator is any callable that accepts a single (callable) argument and returns a callable.
Note that I have not even used the word "function" in this definition; it would actually be incorrect to do so. Indeed, some commonly used decorators have strange properties:
The built-in classmethod and staticmethod decorators return descriptor objects, not functions.
Since language version 2.6 you can decorate classes, not just functions and methods.
Any class containing an __init__(self, somecallable) method and a __call__(self, *args, **kwargs) method can be used as a decorator.
Since there is no standardized decorator in Python, there's no real way of telling if a function is a decorator unless you know something about the decorator you're looking for.
If the decorator is under your control, you can add a mark to indicate it's a decorated function. Otherwise there is no real unified way of doing this. Take this example for instance:
def decorator(func):
return g
#decorator
def f()
pass
def g():
pass
In the above example, in run-time, f and g will be identical, and there is no way of telling the two apart.
Any callable with the right number of arguments can be used as a decorator. Remember that
#foo
def bar(...):
is exactly the same as
def bar(...):
...
bar = foo(bar)
Naturally, since foo could return anything, you have no way of checking whether a function has been decorated or not. Although foo could be nice and leave a mark, it has no obligation to do so.
If you are given some Python code and you want to find all the things that are decorators, you can do so by parsing the code into an abstract syntax tree then walking the tree looking for decorated functions. Here's an example, storing the .ids of the decorators. Obviously, you could store the astobjects if you wanted to.
>>> class DecoratorFinder(ast.NodeVisitor):
... def __init__(self, *args, **kwargs):
... super(DecoratorFinder, self).__init__(*args, **kwargs)
... self.decorators = set()
...
... def visit_FunctionDef(self, node):
... self.decorators.update(dec.id for dec in node.decorator_list)
... self.generic_visit(node)
...
>>> finder = DecoratorFinder()
>>> x = ast.parse("""
... #dec
... def foo():
... pass
... """)
>>> finder.visit(x)
>>> finder.decorators
set(['dec'])
No this is not possible. May be instead of checking if f is a decorator, you should think why you need to check that?
If you are expecting some specific decorator, you can directly check that, if you want some specific behavior/methods/attributes you can check that
If you want to check if some callable f can be used as decorator, you can test the decorator behavior by passing some dummy function, but in general it may not work or have different behavior for different inputs.
Here is a such naive check:
def decorator1(func):
def _wrapper(*args, **kwargs):
print "before"
func(*args, **kwargs)
print "after"
return _wrapper
def dummy_func(): pass
out_func = decorator1(dummy_func)
if callable(out_func) and dummy_func != out_func:
print "aha decorated!"
I've never done anything like this, but in general python relies on "duck-typing" in situations like this. So you could just try to decorate a dummy function and see if a callable is returned.