How do I use asyncio and run the function forever. I know that there's run_until_complete(function_name) but how do I use run_forever how do I call the async function?
async def someFunction():
async with something as some_variable:
# do something
I'm not sure how to start the function.
run_forever doesn't mean that an async function will magically run forever, it means that the loop will run forever, or at least until someone calls loop.stop(). To literally run an async function forever, you need to create an async function that does that. For example:
async def some_function():
async with something as some_variable:
# do something
async def forever():
while True:
await some_function()
loop = asyncio.get_event_loop()
loop.run_until_complete(forever())
This is why run_forever() doesn't accept an argument, it doesn't care about any particular coroutine. The typical pattern is to add some coroutines using loop.create_task or equivalent before invoking run_forever(). But even an event loop that runs no tasks whatsoever and sits idly can be useful since another thread can call asyncio.run_coroutine_threadsafe and give it work.
I'm unsure as to exactly what you mean when you say I'm not sure how to start the function. If you're asking the question in the literal sense:
loop = asyncio.get_event_loop()
loop.run_forever()
If you wish to add a function to the loop before initialising the loop then the following line prior to loop.run_forever() will suffice:
asyncio.async(function())
To add a function to a loop that is already running you'll need ensure_future:
asyncio.ensure_future(function(), loop=loop)
In both cases the function you intend to call must be designated in some way as asynchronous, i.e. using the async function prefix or the #asyncio.coroutine decorator.
Related
I'm using a library that itself makes the call to asyncio.run(internal_function) so I can't control that at all. I do however have access to the event loop, it's something that I pass into this library.
Given that, is there some way I can set up an recurring async event that will execute every X seconds while the main library is running.
This doesn't exactly work, but maybe it's close?
import asyncio
from third_party import run
loop = asyncio.new_event_loop()
async def periodic():
while True:
print("doing a thing...")
await asyncio.sleep(30)
loop.create_task(periodic())
run(loop) # internally this will call asyncio.run() using the given loop
The problem here of course is that the task I've created is never awaited. But I can't just await it, because that would block.
Edit: Here's a working example of what I'm facing. When you run this code you will only ever see "third party code executing" and never see "doing my stuff...".
import asyncio
# I don't know how the loop argument is used
# by the third party's run() function,
def third_party_run(loop):
async def runner():
while True:
print("third party code executing")
await asyncio.sleep(5)
# but I do know that this third party eventually runs code
# that looks **exactly** like this.
try:
asyncio.run(runner())
except KeyboardInterrupt:
return
loop = asyncio.new_event_loop()
async def periodic():
while True:
print("doing my stuff...")
await asyncio.sleep(1)
loop.create_task(periodic())
third_party_run(loop)
If you run the above code you get:
third party code executing
third party code executing
third party code executing
^CTask was destroyed but it is pending!
task: <Task pending name='Task-1' coro=<periodic() running at example.py:22>>
/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:674: RuntimeWarning: coroutine 'periodic' was never awaited
You don't need to await on a created task.
It will run in the background as long as the event loop is active and is not stuck in a CPU bound operation.
According to your comment, you don't have an access to the event loop. In this case you don't have many options other than running in a different thread (which will have its own loop), or changing the loop creation policy in order to get the event loop, which is a very bad idea in most cases.
I found a way to make your test program run. However, it's a hack. It could fail, depending on the internal design of your third party library. From the information you provided, the library has been structured to be a black box. You can't interact with the event loop or schedule a callback. It seems like there might be a very good reason for this.
If I were you I would try to contact the library designer and let him know what your problem is. Perhaps there is a better solution. If this is a commercial project, I would make 100% certain that the team understands the issue, before attempting to use my below solution or anything like it.
The script below overrides one method (new_event_loop) in the DefaultEventLoopPolicy. When this method is called, I create a task in this loop to execute your periodic function. I don't know how often, or for what purpose, the library will call this function. Also, if the library internally overrides the EventLoopPolicy then this solution will not work. In both of these cases it may lead to unforeseeable consequences.
OK, enough disclaimers.
The only significant change to your test script was to replace the infinite loop in runner with a one that times out. This allowed me to verify that the program shuts down cleanly.
import asyncio
# I don't know how the loop argument is used
# by the third party's run() function,
def third_party_run():
async def runner():
for _ in range(4):
print("third party code executing")
await asyncio.sleep(5)
# but I do know that this third party eventually runs code
# that looks **exactly** like this.
try:
asyncio.run(runner())
except KeyboardInterrupt:
return
async def periodic():
while True:
print("doing my stuff...")
await asyncio.sleep(1)
class EventLoopPolicyHack(asyncio.DefaultEventLoopPolicy):
def __init__(self):
self.__running = None
super().__init__()
def new_event_loop(self):
# Override to create our periodic task in the new loop
# Get a loop from the superclass.
# This method must return that loop.
print("New event loop")
loop = super().new_event_loop()
if self.__running is not None:
self.__running.cancel() # I have no way to test this idea
self.__running = loop.create_task(periodic())
return loop
asyncio.set_event_loop_policy(EventLoopPolicyHack())
third_party_run()
I want to return the first element of async generator and handle the remainning values without return like fire and forget. How to make early return of coroutine in python?
After passing the iterator to asyncio.create_task, it doesn't print the remaining values.
import asyncio
import time
async def async_iter(num):
for i in range(num):
await asyncio.sleep(0.5)
yield i
async def handle_remains(it):
async for i in it:
print(i)
async def run() -> None:
it = async_iter(10)
async for i in it:
print(i)
break
# await handle_remains(it)
# want to make this `fire and forget`(no await), expecting just printing the remainning values.
asyncio.create_task(handle_remains(it))
return i
if __name__ == '__main__':
asyncio.run(run())
time.sleep(10)
You’re close with the code, but not quite there yet (see also my comments above). In short, creating the Task isn’t enough: the Task needs to run:
task = asyncio.create_task(handle_remains(it)) # Creates a Task in `pending` state.
await task # Run the task, i.e. execute the wrapped coroutine.
A Task, along with coroutines and Futures, is an “Awaitable”. In fact:
When a coroutine is wrapped into a Task with functions like asyncio.create_task() the coroutine is automatically scheduled to run soon.
Notice the “scheduled to run soon”, now you have to make sure to actually run the task by calling await, a keyword which…
is used to obtain a result of coroutine execution.
I'm trying to understand how asyncio works. As for I/O operation i got understand that when await was called, we register Future object in EventLoop, and then calling epoll for get sockets which belongs to Future objects, that ready for give us data. After we run registred callback and resume function execution.
But, the thing that i cant understant, what's happening if we use await not for I/O operation. How eventloop understands that task is complete? Is it create socket for that or use another kind of loop? Is it use epoll? Or doesnt it add to Loop and used it as generator?
There is an example:
import asyncio
async def test():
return 10
async def my_coro(delay):
loop = asyncio.get_running_loop()
end_time = loop.time() + delay
while True:
print("Blocking...")
await test()
if loop.time() > end_time:
print("Done.")
break
async def main():
await my_coro(3.0)
asyncio.run(main())
await doesn't automatically yield to the event loop, that happens only when an async function (anywhere in the chain of awaits) requests suspension, typically due to IO or timeout not being ready.
In your example the event loop is never returned to, which you can easily verify by moving the "Blocking" print before the while loop and changing main to await asyncio.gather(my_coro(3.0), my_coro(3.0)). What you'll observe is that the coroutines are executed in series ("blocking" followed by "done", all repeated twice), not in parallel ("blocking" followed by another "blocking" and then twice "done"). The reason for that was that there was simply no opportunity for a context switch - my_coro executed in one go as if they were an ordinary function because none of its awaits ever chose to suspend.
async def caller():
await bar()
print("finish")
async def bar():
// some code here
async def caller():
bar()
print("finish")
def bar():
//some code here
In above example. caller has to wait for the completion of bar() for both cases. Any difference for bar to be a normal / coroutine for this situation? If we want to "await" some functions, why not just use a normal function.
The difference is that in the second example bar() is a non-async function, so it itself cannot await anything. For example, if you wanted to access a web service from within bar(), it wouldn't be a problem in the first example, you'd just use aiohttp. In the second example it would be pretty much impossible, as async libraries require being used from async functions, and non-async libraries will block the whole event loop while waiting for response.
If we want to "await" some functions, why not just use a normal function.
If the function you await doesn't need to communicate with the outside world (e.g. if it just shuffles data in a dict or so), it can and should be a normal function. On the other hand, if it needs to do IO, it should be an async function.
If it's not an async function, then you don't need to await it obviously. Not every function you call inside an async function must be async nor must be awaited; you can call regular non-async functions from within an async function.
The entire asyncio model works around an event loop. Only one task can run at any one time, and the event loop coordinates what is currently running. An await inside a function suspends the execution of that function and allows another task to be run on the event loop. So, in this example:
async def caller():
await bar()
print('finish')
The execution goes like this:
caller() is called and scheduled on the event loop, which will execute it as soon as an availability exists.
It calls bar(), which schedules its execution on the event loop.
The await suspends the execution of caller.
The event loop executes bar; let's say it's making a network request, so nothing will happen until that response returns, the event loop is free to run any other scheduled async tasks…
The network response returns, the event loop resumes the execution of bar.
bar ends, the event loop resumes the execution of caller.
await exists to coordinate the sequence in which asynchronous tasks are run and what task depends on the result of what other task.
A coroutine can't run in the simply calling, it need to run in event loop.
The event loop will listen for the events that we add into the event pool and execute callback when the event fire
And when it execute the "await" part of the code, it probably means that there are some I/O bounds task, so that the event loop will go on next event, so that won't block the thread.
I was trying to explain an example of async programming in python but I failed.
Here is my code.
import asyncio
import time
async def asyncfoo(t):
time.sleep(t)
print("asyncFoo")
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncfoo(10)) # I think Here is the problem
print("Foo")
loop.close()
My expectation is that I would see:
Foo
asyncFoo
With a wait of 10s before asyncFoo was displayed.
But instead I got nothing for 10s, and then they both displayed.
What am I doing wrong, and how can I explain it?
run_until_complete will block until asyncfoo is done. Instead, you would need two coroutines executed in the loop. Use asyncio.gather to easily start more than one coroutine with run_until_complete.
Here is a an example:
import asyncio
async def async_foo():
print("asyncFoo1")
await asyncio.sleep(3)
print("asyncFoo2")
async def async_bar():
print("asyncBar1")
await asyncio.sleep(1)
print("asyncBar2")
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(async_foo(), async_bar()))
loop.close()
Your expectation would work in contexts where you run your coroutine as a Task independent of the flow of the code. Another situation where it would work is if you are running multiple coroutines side-by-side, in which case the event-loop will juggle the code execution from await to await statement.
Within the context of your example, you can achieve your anticipated behaviour by wrapping your coroutine in a Task object, which will continue-on in the background without holding up the remainder of the code in the code-block from whence it is called.
For example.
import asyncio
async def asyncfoo(t):
await asyncio.sleep(t)
print("asyncFoo")
async def my_app(t):
my_task = asyncio.ensure_future(asyncfoo(t))
print("Foo")
await asyncio.wait([my_task])
loop = asyncio.get_event_loop()
loop.run_until_complete(my_app(10))
loop.close()
Note that you should use asyncio.sleep() instead of the time module.
run_until_complete is blocking. So, even if it'll happen in 10 seconds, it will wait for it. After it's completed, the other print occurs.
You should launch your loop.run_until_complete(asyncfoo(10)) in a thread or a subprocess if you want the "Foo" to be print before.