I have a basic understanding of decorators, but as yet they seem superfluous and "hacky", like C macros but for functions.
An article stressing the importance of decorators gives this example of decorator usage:
from myapp.log import logger
def log_order_event(func):
def wrapper(*args, **kwargs):
logger.info("Ordering: %s", func.__name__)
order = func(*args, **kwargs)
logger.debug("Order result: %s", order.result)
return order
return wrapper
#log_order_event
def order_pizza(*toppings):
# let's get some pizza!
order_pizza(*toppings) # Usage
Isn't this equivalent to the decoratorless code below?
from myapp.log import logger
def log_order_event(func, *args, **kwargs):
logger.info("Ordering: %s", func.__name__)
order = func(*args, **kwargs)
logger.debug("Order result: %s", order.result)
return order
def order_pizza(*toppings):
# let's get some pizza!
log_order_event(order_pizza, *toppings) # Usage
In fact, isn't the second snippet easier to write since it's one function instead of a function and a wrapper function? The usage calls are longer, but they more clearly indicate what's actually being called.
Is it purely a matter of taste and syntactic sugar, or am I missing something?
Both snippets are equivalent as far as behavior is concerned. However, I'd use the decorator version for two main reasons:
Readability: With snippet 2, you'd require an explicit function call to a less relevant method log_order_event() whose main purpose is logging. This logging functionality may be required by other methods such as order_appetizers() and therefore, you would require individual calls to log_order_event() for every such method. Wouldn't it be better(from a readability perspective) to have a wrapper defined once & for all, that performs logging and you may call the wrapper(as a decorator) for such methods whenever necessary? Decorators are syntactic sugar but certainly not a hack. In fact, PEP 318 clearly talks about the motivation behind decorators, you must read that.
Scalability: Consider a situation where your code has several methods(say 20) such as order_appetizers(), order_maincourse(), order_desserts() and so on. Let's also assume that you'd like to perform some prior checks before ordering any of these items, say check for toppings(for pizza's), calories(for desserts) and so on. When using decorators, its just a matter of defining the wrapper functions in a separate wrapper class for each check, import it in your main code and decorate the required methods respectively. That way, your wrapper functions are completely isolated and the main logic remains clean and easy to maintain.
#check_toppings
#log_order_event
def order_pizza(*toppings):
# let's get some pizza!
#check_calories
#log_order_event
def order_desserts():
# let's order some desserts!
...and so on for 20 more methods
With a decoratorless logic, while you can separate the check(and log) methods in a separate class and import in your main logic, you would still have to add individual calls to each of these methods inside each order method which from a code maintenance perspective could prove to be daunting, especially for new eyes.
Related
For example:
def get_val(n):
return n
def check_args(func):
# gets the arguments of a function at runtime
get_val(1)
get_val(2)
Note: This is probably bad practice, but I want to understand more about how python works.
Python decorators allow you to do this with minimal effort:
import functools
def check_args(func):
#functools.wraps(func) # Copies documentation and other stuff from wrapped func to wrapper, making it look as much like the wrapped func as possible
def wrapper(*args, **kwargs):
# do stuff to inspect arguments
return func(*args, **kwargs)
return wrapper
#check_args
def get_val(n):
return n
Using #check_args is equivalent to fully defining get_val, then doing:
get_val = check_args(get_val)
which means get_val gets replaced with the wrapper function, that now gets called first, can perform checks, then delegate to the wrapped function (in this case, get_args). Obviously, for just one function, it's kind of pointless (you could just put the checking in get_val), but if you want to check other functions, prefixing their definition with #check_args is a single line of code that doesn't get intermingled with the rest of the code, and keeps boilerplate down.
I came across a peculiar behaviour of functools.update_wrapper: it overwrites the __dict__ of the wrapper object by that of the wrapped object - which may hinder its use when nesting decorators.
As a simple example, assume that we are writing a decorator class that caches data in memory and another decorator class that caches data to a file. The following example demonstrates this (I made the example brief and omitted all cacheing logic, but I hope that it demonstrates the question):
import functools
class cached:
cache_type = 'memory'
def __init__(self, fcn):
super().__init__()
self.fcn = fcn
functools.update_wrapper(self, fcn, updated=())
def __call__(self, *args):
print("Retrieving from", type(self).cache_type)
return self.fcn(*args)
class diskcached(cached):
cache_type = 'disk'
#cached
#diskcached
def expensive_function(what):
print("expensive_function working on", what)
expensive_function("Expensive Calculation")
This example works as intended - its output is
Retrieving from memory
Retrieving from disk
expensive_function working on Expensive Calculation
However, it took me long to make this work - at first, I hat not included the 'updated=()' argument in the functools.update_wrapper call. But when this is left out, then nesting the decorators does not work - in this case, the output is
Retrieving from memory
expensive_function working on Expensive Calculation
I.e. the outer decorator directly calls the innermost wrapped function. The reason (which took me a while to understand) for this is that functools.update_wrapper updates the __dict__ attribute of the wrapper to the __dict__ attribute of the wrapped argument - which short-circuits the inner decorator, unless one adds the updated=() argument.
My question: is this behaviour intended - and why? (Python 3.7.1)
Making a wrapper function look like the function it wraps is the point of update_wrapper, and that includes __dict__ entries. It doesn't replace the __dict__; it calls update.
If update_wrapper didn't do this, then if one decorator set attributes on a function and another decorator wrapped the modified function:
#decorator_with_update_wrapper
#decorator_that_sets_attributes
def f(...):
...
the wrapper function wouldn't have the attributes set, rendering it incompatible with the code that looks for those attributes.
I have an class which decorates some methods using a decorator from another library. Specifically, the class subclasses flask-restful resources, decorates the http methods with httpauth.HTTPBasicAuth().login_required(), and does some sensible defaults on a model service.
On most subclasses I want the decorator applied; therefore I'd rather remove it than add it in the subclasses.
My thought is to have a private method which does the operations and a public method which is decorated. The effects of decoration can be avoided by overriding the public method to call the private one and not decorating this override. Mocked example below.
I am curious to know if there's a better way to do this. Is there a shortcut for 'cancelling decorators' in python that gives this effect?
Or can you recommend a better approach?
Some other questions have suitable answers for this, e.g. Is there a way to get the function a decorator has wrapped?. But my question is about broader design - i am interested in any pythonic way to run the operations in decorated methods without the effects of decoration. E.g. my example is one such way but there may be others.
def auth_required(fn):
def new_fn(*args, **kwargs):
print('Auth required for this resource...')
fn(*args, **kwargs)
return new_fn
class Resource:
name = None
#auth_required
def get(self):
self._get()
def _get(self):
print('Getting %s' %self.name)
class Eggs(Resource):
name = 'Eggs'
class Spam(Resource):
name = 'Spam'
def get(self):
self._get()
# super(Spam, self)._get()
eggs = Eggs()
spam = Spam()
eggs.get()
# Auth required for this resource...
# Getting Eggs
spam.get()
# Getting Spam
Flask-HTTPAuth uses functools.wraps in the login_required decorator:
def login_required(self, f):
#wraps(f)
def decorated(*args, **kwargs):
...
From Python 3.2, as this calls update_wrapper, you can access the original function via __wrapped__:
To allow access to the original function for introspection and other
purposes (e.g. bypassing a caching decorator such as lru_cache()),
this function automatically adds a __wrapped__ attribute to the
wrapper that refers to the function being wrapped.
If you're writing your own decorators, as in your example, you can also use #wraps to get the same functionality (as well as keeping the docstrings, etc.).
See also Is there a way to get the function a decorator has wrapped?
Another common option is to have the decorated function keep a copy of the original function that can be accessed:
def auth_required(fn):
def new_fn(*args, **kwargs):
print('Auth required for this resource...')
fn(*args, **kwargs)
new_fn.original_fn = fn
return new_fn
Now, for any function that has been decorated, you can access its original_fn attribute to get a handle to the original, un-decorated function.
In that case, you could define some type of dispatcher that either makes plain function calls (when you are happy with the decorator behavior) or makes calls to thing.original_fn when you prefer to avoid the decorator behavior.
Your proposed method is also a valid way to structure it, and whether my suggestion is "better" depends on the rest of the code you're dealing with, who needs to read it, and other kinds of trade-offs.
I am curious to know if there's a better way to do this. Is there a
shortcut for 'cancelling decorators' in python that gives this effect?
Use the undecorated library. It digs through all the decorators and returns just the original function. The docs should be self-explanatory, basically you just call: undecorated(your_decorated_function)
Is it good style to daisy-chain Python/Django custom decorators? And pass different arguments than received?
Many of my Django view functions start off with the exact same code:
#login_required
def myView(request, myObjectID):
try:
myObj = MyObject.objects.get(pk=myObjectID)
except:
return myErrorPage(request)
try:
requester = Profile.objects.get(user=request.user)
except:
return myErrorPage(request)
# Do Something interesting with requester and myObj here
FYI, this is what the corresponding entry in urls.py file looks like:
url(r'^object/(?P<myObjectID>\d+)/?$', views.myView, ),
Repeating the same code in many different view functions is not DRY at all. I would like to improve it by creating a decorator that would do this repetitive work for me and make the new view functions much cleaner and look like this:
#login_required
#my_decorator
def myView(request, requester, myObj):
# Do Something interesting with requester and myObj here
So here are my questions:
Is this a valid thing to do? Is it good style? Notice that I will be changing the signature of the myView() function. That feels a bit strange and risky to me. But I'm not sure why
If I make multiple such decorators that do some common function but each call the wrapped function with different arguments than the decorator received, is it OK if I daisy-chain them together?
If it is OK to #1 and #2 above, what is the best way to indicate to the users of this myView what the set of arguments are that they should pass in (because just looking at the parameters in the function definition is no longer really valid)
That's a very interesting question ! Another one has already been answered in depth on the basic usage of decorators. But it does not provide much insight on modifying arguments
Stackable decorators
You can find on that other question an example of stacked decorators with the following piece of explanation hidden in a very, very long and detailed answer :
Yes, that’s all, it’s that simple. #decorator is just a shortcut to:
another_stand_alone_function = my_shiny_new_decorator(another_stand_alone_function)
And that's the magic. As python documentation states : a decorator is a function returning another function.
That means you can do :
from functools import wraps
def decorator1(f):
#wraps(f)
def wrapper(*args, **kwargs):
do_something()
f(*args, **kwargs)
return wrapper
def decorator2(f):
#wraps(f)
def wrapper(*args, **kwargs):
do_something_else()
f(*args, **kwargs)
return wrapper
#decorator1
#decorator2
def myfunc(n):
print "."*n
#is equivalent to
def myfunc(n):
print "."*n
myfunc = decorator1(decorator2(myfunc))
Decorators are not Decorators
Python decorators might be puzzling for developpers who learned OOP with a language where GoF has already used half of the dictionary to name the patterns who fix the failures of the language is the de-facto design pattern shop.
GoF's decorators are subclasses of the component (interface) they're decorating, therefore sharing this interface with any other subclass of that component.
Python decorators are functions returning functions (or classes).
Functions all the way down
A python decorator is a function returning a function, any function.
Most decorators out there are designed to extend the decorated function without getting in the way of it's expected behavior. They are shaped after GoF's definition of the Decorator pattern, which describes a way to extend an object while keeping it's interface.
But GoF's Decorator is a pattern, whereas python's decorator is a feature.
Python decorators are functions, these functions are expected to return functions (when provided a function).
Adapters
Let's take another GoF pattern : Adapter
An adapter helps two incompatible interfaces to work together.
This is the real world definition for an adapter.
[An Object] adapter contains an instance of the class it wraps.
In this situation, the adapter makes calls to the instance of the wrapped
object.
Take for example an object — say a dispatcher, who would call a function which takes some defined parameters, and take a function who would do the job but provided another set of parameters. Parameters for the second function can be derived from those of the first.
A function (which is a first-class object in python) who would take the parameters of the first and derive them to call the second and return a value derived from its result would be an adapter.
A function returning an adapter for the function it is passed would be an adapter factory.
Python decorators are functions returning functions. Including adapters.
def my_adapter(f):
def wrapper(*args, **kwargs):
newargs, newkwargs = adapt(args, kwargs)
return f(*newargs, **newkwargs)
#my_adapter # This is the contract provider
def myfunc(*args, **kwargs):
return something()
Oooooh, I see what you did there… is it good style ?
I'd say, hell yeah, yet another built-in pattern ! But you'd have to forget about GoF Decorators and simply remember that python decorators are functions which return functions. Therefore, the interface you're dealing with is the one of the wrapper function, not the decorated one.
Once you decorate a function, the decorator defines the contract, either telling it's keeping the interface of the decorated function or abstracting it away. You don't call that decorated function anymore, it's even tricky to try it, you call the wrapper.
First of all, this block of code:
try:
myObj = MyObject.objects.get(pk=myObjectID)
except:
return myErrorPage(request)
can be replaced with:
from django.shortcuts import get_object_or_404
myObj = get_object_or_404(MyObject, pk=myObjectID)
The same applies with the second block of code you have.
That in and of itself makes this a lot more elegant.
If you'd like to go further and implement your own decorator, your best bet is to subclass #login_required. If you're passing different arguments or don't want to do that, then you can indeed make your own decorator and it wouldn't be wrong.
1) Yes, chaining decorators is valid as other answers have already pointed out. Good style is subjective, but personally I think it would make your code much harder to read for others. Someone familiar with Django but not your application would need to keep extra context in their head while working with your code. I think it's very important to stick to framework conventions to make your code as maintainable as possible.
2) The answer is yes, it is technically okay to pass in different arguments to the wrapped function, but consider a simple code example of how this would work:
def decorator1(func):
def wrapper1(a1):
a2 = "hello from decorator 1"
func(a1, a2)
return wrapper1
def decorator2(func):
def wrapper2(a1, a2):
a3 = "hello from decorator 2"
func(a1, a2, a3)
return wrapper2
#decorator1
#decorator2
def my_func(a1, a2, a3):
print a1, a2, a3
my_func("who's there?")
# Prints:
# who's there?
# hello from decorator 1
# hello from decorator2
In my opinion, any person reading this would need to be a mental gymnast to keep context of the method signatures at each level of the decorator stack.
3) I would use a class-based view and override the dispatch() method to set instance variables like this:
class MyView(View):
#method_decorator(login_required)
def dispatch(self, *args, **kwargs):
self.myObj = ...
self.requester = ...
return super(MyView, self).dispatch(*args, **kwargs)
The dispatch method is what calls your get()/post() methods. From the django docs:
The as_view entry point creates an instance of your class and calls its dispatch() method. dispatch looks at the request to determine whether it is a GET, POST, etc, and relays the request to a matching method if one is defined
Then you could access these instance variables in your get() and/or post() view methods. The advantage of this approach is that you could extract this out to a base class and use it in any number of View subclasses. It is also a lot more traceable in an IDE because this is standard inheritance.
An example of how a get() request would look like:
class MyView(View):
def get(self, request, id):
print 'requester is {}'.format(self.requester)
I am trying to build a decorator that can inspect the wrapped function's arguments, the context for the need is to allow easier validation of certain arguments before they hit the function, avoiding the boiler plate of checking in every place I need said check/validation.
This is not a difficult decorator to write though, and this gets the job done (to make sure a value is not None for example):
def check_arg(func):
def wrapped(value):
assert value is not None
func(value)
return wrapped
The above example is not production code, it is merely an example of something that will work, but the problem becomes apparent when the function to decorate can have other decorators being used.
In that case, the above decorator example will not work, because value can now be another decorator. I could add another nested function here but it looks like there should be a better way of dealing with it programmatically, plus, it would break whenever the number of decorators change.
How could I get to the arguments of the wrapped function regardless of the number of decorators used by it?
EDIT: I should've noted than I am not simply stacking decorators, I am using one decorator that decorates re-usable decorators. In the example above I would re-use that in this context:
#check_arg
def some_other_decorator(func):
def wrapped(*args, **kw):
... # some interaction here
return wrapped
As I was writing this edit, I realized that the check_arg decorator was in the wrong place, it should've been in the wrapped function call within the some_other_decorator.
Are you in control of the other decorators? If so, make sure all of them play nice by properly
wrapping the calling function:
import functools
def check_arg(func):
#functools.wraps(func)
def wrapped(value):
assert value is not None
func(value)
return wrapped