I was wondering if there is a standardized approach or best practice to scan/ autodiscover decorators like it is done here but also in several other libs like Django, Flask. Usually a decorator provides extra/ wrapped functionality right at the time the inner func is called.
In the example shown below but also in Flask/ Django (route decorators) the decorator is rather used to add overarching functionalities, e.g. spawning of a tcp client initially within the decorator logic and then call the inner func when there is a message received to process it.
Flask/ Django register for an url route where the inner func is only called later when the url is requested. All examples require an initial registration (scan/ discover) of decorator logic to also initially start the overarching functionality. To me this seems to be an alternative use of decorators and I would like to understand the best practice approach if there is any.
See Faust example below where decorator app.agent() automatically triggers a listening (kafka stream) client within asyncio event loop and incoming message is then processed by the inner function hello() later, only when there is a message received, requiring an initial check/ scan/ discovery of related decorator logic first at the start of the script.
import faust
class Greeting(faust.Record):
from_name: str
to_name: str
app = faust.App('hello-app', broker='kafka://localhost')
topic = app.topic('hello-topic', value_type=Greeting)
#app.agent(topic)
async def hello(greetings):
async for greeting in greetings:
print(f'Hello from {greeting.from_name} to {greeting.to_name}')
#app.timer(interval=1.0)
async def example_sender(app):
await hello.send(
value=Greeting(from_name='Faust', to_name='you'),
)
if __name__ == '__main__':
app.main()
Nothing is "discovered". When you import a module from a package, all of that code is executed. This is why we have if __name__ == '__main__' to stop certain code being executed on import. The decorators will be "discovered" when you run your code.
I think the Flask blueprint is a nice example. Here you can see how it registers the url endpoints when you import modules. All it's doing is appending to a list:
def route(self, rule, **options):
"""Like :meth:`Flask.route` but for a blueprint. The endpoint for the
:func:`url_for` function is prefixed with the name of the blueprint.
"""
def decorator(f):
endpoint = options.pop("endpoint", f.__name__)
self.add_url_rule(rule, endpoint, f, **options)
return f
return decorator
The code runs, the decorators are evaluated and they need only keep some internal list of all the functions they decorate. These are stored in the Blueprint object.
Related
I have a celery task:
#task
def foo():
part1()
part2()
part3()
...that I'm breaking up into a chain of subtasks
#task
def foo():
#task
def part1():
...
#task
def part2():
...
#task
def part3():
...
chain(part1.s(), part2.s(), part3.s()).delay()
The subtasks are inner funcs because I don't want them executed outside the context of the parent task. Issue is my worker does not detect and/or register the inner tasks (I am using autoregister to discover apps and task modules). If I move them out to the same level in the module as the parent task foo, it works fine.
Does celery support inner functions as tasks? If so, how do I get workers to register them?
The problem with your code is that you get new definitions of part1 every time you call foo(). Also note that not a single part1 function is created until you call foo so it is not possible for celery to register any one of part functions to be created when it initializes a worker.
I think the following code is the closest to what you want.
def make_maintask():
#task
def subtask1():
print("do subtask")
# ...
#task
def _maintask():
chain(subtask1.si(), subtask2.si(), subtask3.si()).delay()
return _maintask
maintask = make_maintask()
This way, each definition of subtask and such is not visible from outside.
some comments
If all you want to do is hiding subtask, please think twice. The designer of the python language didn't believe that one needs a access control such as public and private as in java. It is a functionality that severely complicates a language with a dubious advantage. I think well-organized packages and modules and good names (such as adding underscores in front) can solve all your problems.
If all _maintask does is delegating subtasks to other subprocesses, you don't really need to define it as a celery task. Don't make a celery task call another celery task unless you really need it.
This question already has answers here:
What does the "at" (#) symbol do in Python?
(14 answers)
Closed 3 years ago.
I'm new to Flask. When I came across basic Flask example(below code), I stuck with the uncertainty: what is the need to use # before variable app. I tried running app while removing #, but I can't. If it is some very basic thing that I'm asking about, please comment.
from flask import Flask, escape, request
app = Flask(__name__)
#app.route('/')
def hello():
name = request.args.get("name", "World")
return f'Hello, {escape(name)}!'
The # symbol is used for decorators.
A decorator is a function that takes another function as argument, and modifies its behavior.
In the case of Flask, app.route is a decorator that will "install" your function as the handler for a speficif route in your web app.
Doing this:
#app.route('/foo')
def hello():
return 'Hello'
is the same as doing:
def f():
return 'Hello'
decorator = app.route('/foo')
hello = decorator(f)
What the # symbol does is implicitly calling the result of app.route('/foo') with your function as argument. As you can see this makes the above code more convenient and easy to read.
If you look at the Flask source code, you'll see the definition of route() as a method of the class Flask:
class Flask:
#...
def route(self, rule, **options):
def decorator(f):
endpoint = options.pop("endpoint", None)
self.add_url_rule(rule, endpoint, f, **options)
return f
return decorator
That's a decorator. In short route is decorator that tell Flask what URL should trigger our function.
Python decorators are functions that are used to transform other functions. When a decorated function is called, the decorator is called instead. The decorator can then take action, modify the arguments, halt execution or call the original function. We can use decorators to wrap views with code we’d like to run before they are executed.
#decorator_function
def decorated():
pass
If you’ve gone through the Flask tutorial, the syntax in this code block might look familiar to you.
#app.route is a decorator used to match URLs to view functions in Flask apps.
For more detailed description, you can even refer to this documentation
Also, there is a similar question asked on the stackoverflow, you can refer to this as well.
I'm familiar with the basics of Python's decorators. However I don't understand how this specific decorator, as used for Flask routes, works.
Here is a code snippet from the Flask website:
from flask import Flask, escape, request
app = Flask(__name__)
#app.route('/')
def hello():
name = request.args.get("name", "World")
return f'Hello, {escape(name)}!'
.route('/') is the part which confuses me. The way I've seen decorators being used is that the decorator annoation is usually a function, which takes one argument, which is the decorated function itself, which can then be executed inside the decorator. But this seems to be different here.
Thank you.
Update: Thanks for the hint about self in my second code snippet. Being new to Python I totally forgot that something's first argument has to be self. At second glance, I don't think that the example helps to clarify the question and have therefore removed it.
It sounds like you have a couple misconceptions that we can hopefully clear up. Firstly, a decorator is just a term (and some language "sugar", the #decorator syntax) for a callable that accepts a callable and returns another callable. This callable can be a function (e.g., def foo(func)) or a method (e.g., obj.method(func)) -- it just needs to be something that accepts a callable and returns one.
In the case of Flask's #app.route decorator, you're passing your hello function to an object's (the flask.Flask object I believe) route method. In turn it adds your function and its associated configuration to its own "memory" for routing once the Flask application is told to begin serving requests.
Secondly, your example has def something(arg1, arg2): -- the first argument passed to an object's method when called is the object instance, usually called self
The Primer on Python Decorators article might be a good place to get started to read up on how/why they work the way they do.
A decorator is simply a function that accepts a function as a parameter and returns another function. A decorator that accepts arguments (like flask's route decorator) must return a function that can be used as a simple decorator.
For example, we could write that route decorator like this:
routes = {}
def route(path):
def inner(func):
global routes
routes[path] = func
return func
return inner
Calling route('/') returns a function that takes a function as an argument, which updates the global routes variable using the value of path passed to the outer function and the func value received by the inner function.
If we use that on a couple of functions:
#route('/')
def func1():
...
#route('/example')
def func2():
...
Then we can inspect the routes variable and see that it now maps paths to the appropriate functions. Running:
print(routes)
Produces something like:
{'/': <function func1 at 0x7f0c4aeac050>, '/example': <function func2 at 0x7f0c4aeb1e60>}
Edit 2: I misunderstood how decorators work. The decorator is run even if the decorated function is not called (even though you might not see it's effects). function = dec(function) would be an equivalent way of showing what a decorator does and obviously in that scenario the dec() function runs without any calls to function().
Edit: Why is my post being down voted with no explanation? How can I fix it? There are multiple answers, one which clearly answers the question. What is the issue?
I've been learning about decorators in python and I think I have a good grasp on them. However I am still slightly confused about how the app.route decorator works in flask. By my understanding the decorator changes the behavior of a function but does not run unless the function is called. So if I have:
#app.route("/")
def hello():
return "Hello world"
hello()
The hello function will get passed to app.route and whatever behavior the decorator dictates will execute. However in flask apps the function itself never appears to be run (in my above example it is). How is the route function/decorator executed if the function it decorates is never called? I know that app.route essentially stores the "/" along with its corresponding function in a dictionary but I don't understand how this code is ever executed without any calls of the decorated function. I assume it is somehow connected to the app.run at the end of flask apps but I am unclear on how app.run can call the functions you defined.
Edit: to add to what I've shown here. There is an example from this explanation: https://ains.co/blog/things-which-arent-magic-flask-part-1.html That raises the same questions. I would think that hello() needs to be called in order for the route function to do anything.
class NotFlask():
def __init__(self):
self.routes = {}
def route(self, route_str):
def decorator(f):
self.routes[route_str] = f
return f
return decorator
def serve(self, path):
view_function = self.routes.get(path)
if view_function:
return view_function()
else:
raise ValueError('Route "{}"" has not been
registered'.format(path))
app = NotFlask()
#app.route("/")
def hello():
return "Hello World!"
Python decorators like this:
#decorator
def func():
pass
Can be changed to look like this instead:
def func():
pass
decorator(func)
Or in other words, they're functions that take functions. In some circumstances, you might not see the effects of the decorator immediately, so it may seem like the decorator itself isn't used until the function it decorates is called, but that's not an actual restriction of Python decorators.
In your code, #app.route("/") is a decorator which adds an endpoint to the app object. It doesn't actually modify any behavior of the function, and is instead sugar to simplify the process. Without the route() decorator, this is how you'd do the equivalent route registration in Flask.
from flask import Flask
app = Flask(_name_)
def hello():
return "Hello world"
app.add_url_rule("/", "hello", hello)
And if you look at the implementation of the route decorator in Flask, you'll see that this is the equivalent.
def route(self, rule, **options):
"""A decorator that is used to register a view function for a
given URL rule. This does the same thing as :meth:`add_url_rule`
but is intended for decorator usage::
#app.route('/')
def index():
return 'Hello World'
For more information refer to :ref:`url-route-registrations`.
:param rule: the URL rule as string
:param endpoint: the endpoint for the registered URL rule. Flask
itself assumes the name of the view function as
endpoint
:param options: the options to be forwarded to the underlying
:class:`~werkzeug.routing.Rule` object. A change
to Werkzeug is handling of method options. methods
is a list of methods this rule should be limited
to (``GET``, ``POST`` etc.). By default a rule
just listens for ``GET`` (and implicitly ``HEAD``).
Starting with Flask 0.6, ``OPTIONS`` is implicitly
added and handled by the standard request handling.
"""
def decorator(f):
endpoint = options.pop('endpoint', None)
self.add_url_rule(rule, endpoint, f, **options)
return f
return decorator
So you can see that route adds it to the app's router, so once the Flask app gets a request, it decides how to execute code/view for the endpoint that's been requested.
Your code snippet is calling the hello function directly. That's why the function is running.
For Flask, you define the routes using decorator functions. When you navigate to a URL that matches the decorator, it will execute the decorated function. In this example, if I navigate to "/" on my webserver, the hello function will execute.
To actually run your server, you need to do the following from the command line (which can also be found in Flask's documentation.
$ export FLASK_APP=hello.py
$ flask run
Where hello.py is the name of the file containing your Python code. You should also remove the direct call to hello() in your file. Then open a browser and navigate to http://localhost:5000/ to see the result.
How do Decorators Work
Decorators are functions that wrap other functions in an attempt to alter the behavior of the sub-function. You can read the very detailed explanation in the Python wiki.
Decorators take in a function as their argument. Typically, decorators run some code before executing the sub-function. For example, if you want to add authentication to certain endpoints of your Flask app, you can use decorators. The decorators check to make sure a user is authenticated to use a resource before the actual code of that resource executes.
#authenticate
def hello():
return "Hello world"
The authenticate function will run first. And then if the user is authenticated, it will call hello to execute the rest, otherwise, it will send an error back to the user. In the case of Flask's decorators, they are checking to see if an incoming request matches the route you've specified. If it does, then it executes the function. Otherwise, it checks the next route.
The following article is my go to for learning about decorators: https://realpython.com/blog/python/primer-on-python-decorators/
I've decorated a method in Python. And when I import the module that contains the method, the decorator autoruns.
I realize that this is how decorators were made however Is there a way to have decorators NOT do this?
It sounds like what you want to do is to choose what decorator to apply at run time. Something like this might work:
to_decorate = []
def decorate_later(func):
to_decorate.append(func)
return func
#decorate_later
def do_stuff(*args, **kw):
print('I am doing stuff')
#decorate_later
def do_more_stuff(*args, **kw):
print('Even more stuff')
def apply_decorator(decorator):
for func in to_decorate:
globals()[func.func_name] = decorator(func)
Then you can import the module and all the functions will be defined as normal. decorate_later returns the original function unmodified. You can call apply_decorator() to apply a specified decorator to all of the functions in the module that were registered by #decorate_later
This is exactly what the venusian library does; you define your decorators according to their API, but the actual behavior isn't triggered until you do a "scan" of the containing module or package.
You don't even need to have a global app object to use venusian decorators; you can pass in the app object as part of the scan, and it'll get passed along to the decorator implementations. So, for example, the same functions can be shared among multiple owners with only a single decorator, just by doing more than one scan.
This is what the Pyramid web framework uses for e.g. event registration, so that merely importing a module doesn't expect to need an app instance. A good example is their event subscriber.
Use
if __name__ == "__main__":
#code
in the file, where code is all outside a method or class ( that runs when you import it).