from what I know about asyncio this should only print 0 to 4, but it goes through the full 10 digits.
Shouldn't the stop_loop coroutine stop awaiting the event and cancel the loop after i hits 5?
import asyncio
async def run():
for i in range(10):
if i == 5:
e.set()
print(i)
async def stop_loop():
await e.wait()
l.stop()
e = asyncio.Event()
l = asyncio.get_event_loop()
l.set_debug(True)
l.create_task(stop_loop())
l.create_task(run())
try:
l.run_forever()
finally:
l.close()
Output is
machine:programs user$ python3 conditional_stop.py
0
1
2
3
4
5
6
7
8
9
asyncio works by switching between tasks that are implemented as coroutines. A coroutine is a cooperative routine, in that coroutines voluntarily give up control once in a while, to let the asyncio event loop switch to another task. This is different from threading, where each task can and will be interrupted 'at will' by the scheduler.
And coroutines give up control every time they use await on another coroutine, usually at points where some I/O is involved. I/O is slow, and the asyncio event loop takes responsibility of monitoring for changes in I/O streams so it can know what tasks are ready to do more work again.
Your problem is that you have a coroutine that is not cooperating:
async def run():
for i in range(10):
if i == 5:
e.set()
print(i)
That routine has no await statements, so it never gives up control to the event loop. No other coroutines can be run.
You could await on a asyncio.sleep() call:
async def run():
for i in range(10):
if i == 5:
e.set()
print(i)
await asyncio.sleep(0.01) # wait 1/100th of a second
Another option would be to replace the print(i) call (which is an I/O operation) with one that uses a non-blocking output stream. If you are not on Windows, then you can create a StreamWriter asynchronous I/O wrapper for sys.stdout:
import os
import sys
async def run():
# create an async writer for sys.stdout
loop = asyncio.get_event_loop()
writer_transport, writer_protocol = await loop.connect_write_pipe(
asyncio.streams.FlowControlMixin, os.fdopen(sys.stdout.fileno(), 'wb'))
writer = asyncio.streams.StreamWriter(
writer_transport, writer_protocol, None, loop)
for i in range(10):
if i == 5:
e.set()
writer.write(b'%d\n' % i)
await writer.drain()
Unfortunately, there is no support yet for creating async I/O streams for the Windows console streams, see Pyton issue #26832, you'd have to use a threadpool executor instead.
Note that even with the latter coroutine, there is no guarantee that the stop coroutine will actually be run soon enough after e.set() is called to cancel run() before it reads 10! The loop is free to give control right back to the same coroutine after await writer.drain() has been handled. Writing short lines to the sys.stdout stream buffer is fast, and the only thing .drain() does is just give the writing coroutines time to flush the internal transport buffer; with direct non-blocking writes to sys.stdout succeeding most of the time that's not always enough room for stop_loop() to jump in and the run() coroutine will have written all its lines to the writer transport .
Related
My Source Code:
import asyncio
async def mycoro(number):
print(f'Starting {number}')
await asyncio.sleep(1)
print(f'Finishing {number}')
return str(number)
c = mycoro(3)
task = asyncio.create_task(c)
loop = asyncio.get_event_loop()
loop.run_until_complete(task)
loop.close()
Error:
RuntimeError: no running event loop
sys:1: RuntimeWarning: coroutine 'mycoro' was never awaited
I was watching a tutorial and according to my code it was never awaited when I did and it clearly does in the video I was watching.
Simply run the coroutine directly without creating a task for it:
import asyncio
async def mycoro(number):
print(f'Starting {number}')
await asyncio.sleep(1)
print(f'Finishing {number}')
return str(number)
c = mycoro(3)
loop = asyncio.get_event_loop()
loop.run_until_complete(c)
loop.close()
The purpose of asyncio.create_task is to create an additional task from inside a running task. Since it directly starts the new task, it must be used inside a running event loop – hence the error when using it outside.
Use loop.create_task(c) if a task must be created from outside a task.
In more recent version of asyncio, use asyncio.run to avoid having to handle the event loop explicitly:
c = mycoro(3)
asyncio.run(c)
In general, use asyncio.create_task only to increase concurrency. Avoid using it when another task would block immediately.
# bad task usage: concurrency stays the same due to blocking
async def bad_task():
task = asyncio.create_task(mycoro(0))
await task
# no task usage: concurrency stays the same due to stacking
async def no_task():
await mycoro(0)
# good task usage: concurrency is increased via multiple tasks
async def good_task():
tasks = [asyncio.create_task(mycoro(i)) for i in range(3)]
print('Starting done, sleeping now...')
await asyncio.sleep(1.5)
await asyncio.gather(*tasks) # ensure subtasks finished
Change the line
task = asyncio.Task(c)
I try to cancel process when timeout but asyncio.wait_for not working. How do i cancel process when reached time out. My code below:
import asyncio
async def process():
# do something take a long time like this
for i in range(0,10000000000,1):
for j in range(0,10000000000,1):
continue
print('done!')
async def main():
# I want to cancel process when reached timeout
try:
await asyncio.wait_for(process(), timeout=1.0)
except asyncio.TimeoutError:
print('timeout!')
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
This doesn't work because your process function is async in name only - it doesn't await anything. That means that it finishes in its entirety without giving the event loop a chance to interrupt it. Since asyncio is cooperative (as are other async/await based systems), such a function is not a correctly written async function and cannot be interrupted.
If you add an await asyncio.sleep(0.001) into the inner loop (or anything else that awaits something that actually suspends), your code will work fine.
I have a code like the foolowing:
def render():
loop = asyncio.get_event_loop()
async def test():
await asyncio.sleep(2)
print("hi")
return 200
if loop.is_running():
result = asyncio.ensure_future(test())
else:
result = loop.run_until_complete(test())
When the loop is not running is quite easy, just use loop.run_until_complete and it return the coro result but if the loop is already running (my blocking code running in app which is already running the loop) I cannot use loop.run_until_complete since it will raise an exception; when I call asyncio.ensure_future the task gets scheduled and run, but I want to wait there for the result, does anybody knows how to do this? Docs are not very clear how to do this.
I tried passing a concurrent.futures.Future calling set_result inside the coro and then calling Future.result() on my blocking code, but it doesn't work, it blocks there and do not let anything else to run. ANy help would be appreciated.
To implement runner with the proposed design, you would need a way to single-step the event loop from a callback running inside it. Asyncio explicitly forbids recursive event loops, so this approach is a dead end.
Given that constraint, you have two options:
make render() itself a coroutine;
execute render() (and its callers) in a thread different than the thread that runs the asyncio event loop.
Assuming #1 is out of the question, you can implement the #2 variant of render() like this:
def render():
loop = _event_loop # can't call get_event_loop()
async def test():
await asyncio.sleep(2)
print("hi")
return 200
future = asyncio.run_coroutine_threadsafe(test(), loop)
result = future.result()
Note that you cannot use asyncio.get_event_loop() in render because the event loop is not (and should not be) set for that thread. Instead, the code that spawns the runner thread must call asyncio.get_event_loop() and send it to the thread, or just leave it in a global variable or a shared structure.
Waiting Synchronously for an Asynchronous Coroutine
If an asyncio event loop is already running by calling loop.run_forever, it will block the executing thread until loop.stop is called [see the docs]. Therefore, the only way for a synchronous wait is to run the event loop on a dedicated thread, schedule the asynchronous function on the loop and wait for it synchronously from another thread.
For this I have composed my own minimal solution following the answer by user4815162342. I have also added the parts for cleaning up the loop when all work is finished [see loop.close].
The main function in the code below runs the event loop on a dedicated thread, schedules several tasks on the event loop, plus the task the result of which is to be awaited synchronously. The synchronous wait will block until the desired result is ready. Finally, the loop is closed and cleaned up gracefully along with its thread.
The dedicated thread and the functions stop_loop, run_forever_safe, and await_sync can be encapsulated in a module or a class.
For thread-safery considerations, see section “Concurrency and Multithreading” in asyncio docs.
import asyncio
import threading
#----------------------------------------
def stop_loop(loop):
''' stops an event loop '''
loop.stop()
print (".: LOOP STOPPED:", loop.is_running())
def run_forever_safe(loop):
''' run a loop for ever and clean up after being stopped '''
loop.run_forever()
# NOTE: loop.run_forever returns after calling loop.stop
#-- cancell all tasks and close the loop gracefully
print(".: CLOSING LOOP...")
# source: <https://xinhuang.github.io/posts/2017-07-31-common-mistakes-using-python3-asyncio.html>
loop_tasks_all = asyncio.Task.all_tasks(loop=loop)
for task in loop_tasks_all: task.cancel()
# NOTE: `cancel` does not guarantee that the Task will be cancelled
for task in loop_tasks_all:
if not (task.done() or task.cancelled()):
try:
# wait for task cancellations
loop.run_until_complete(task)
except asyncio.CancelledError: pass
#END for
print(".: ALL TASKS CANCELLED.")
loop.close()
print(".: LOOP CLOSED:", loop.is_closed())
def await_sync(task):
''' synchronously waits for a task '''
while not task.done(): pass
print(".: AWAITED TASK DONE")
return task.result()
#----------------------------------------
async def asyncTask(loop, k):
''' asynchronous task '''
print("--start async task %s" % k)
await asyncio.sleep(3, loop=loop)
print("--end async task %s." % k)
key = "KEY#%s" % k
return key
def main():
loop = asyncio.new_event_loop() # construct a new event loop
#-- closures for running and stopping the event-loop
run_loop_forever = lambda: run_forever_safe(loop)
close_loop_safe = lambda: loop.call_soon_threadsafe(stop_loop, loop)
#-- make dedicated thread for running the event loop
thread = threading.Thread(target=run_loop_forever)
#-- add some tasks along with my particular task
myTask = asyncio.run_coroutine_threadsafe(asyncTask(loop, 100200300), loop=loop)
otherTasks = [asyncio.run_coroutine_threadsafe(asyncTask(loop, i), loop=loop)
for i in range(1, 10)]
#-- begin the thread to run the event-loop
print(".: EVENT-LOOP THREAD START")
thread.start()
#-- _synchronously_ wait for the result of my task
result = await_sync(myTask) # blocks until task is done
print("* final result of my task:", result)
#... do lots of work ...
print("*** ALL WORK DONE ***")
#========================================
# close the loop gracefully when everything is finished
close_loop_safe()
thread.join()
#----------------------------------------
main()
here is my case, my whole programe is async, but call some sync lib, then callback to my async func.
follow the answer by user4815162342.
import asyncio
async def asyncTask(k):
''' asynchronous task '''
print("--start async task %s" % k)
# await asyncio.sleep(3, loop=loop)
await asyncio.sleep(3)
print("--end async task %s." % k)
key = "KEY#%s" % k
return key
def my_callback():
print("here i want to call my async func!")
future = asyncio.run_coroutine_threadsafe(asyncTask(1), LOOP)
return future.result()
def sync_third_lib(cb):
print("here will call back to your code...")
cb()
async def main():
print("main start...")
print("call sync third lib ...")
await asyncio.to_thread(sync_third_lib, my_callback)
# await loop.run_in_executor(None, func=sync_third_lib)
print("another work...keep async...")
await asyncio.sleep(2)
print("done!")
LOOP = asyncio.get_event_loop()
LOOP.run_until_complete(main())
I was wondering how concurrency works in python 3.6 with asyncio. My understanding is that when the interpreter executing await statement, it will leave it there until the awaiting process is complete and then move on to execute the other coroutine task. But what I see here in the code below is not like that. The program runs synchronously, executing task one by one.
What is wrong with my understanding and my impletementation code?
import asyncio
import time
async def myWorker(lock, i):
print("Attempting to attain lock {}".format(i))
# acquire lock
with await lock:
# run critical section of code
print("Currently Locked")
time.sleep(10)
# our worker releases lock at this point
print("Unlocked Critical Section")
async def main():
# instantiate our lock
lock = asyncio.Lock()
# await the execution of 2 myWorker coroutines
# each with our same lock instance passed in
# await asyncio.wait([myWorker(lock), myWorker(lock)])
tasks = []
for i in range(0, 100):
tasks.append(asyncio.ensure_future(myWorker(lock, i)))
await asyncio.wait(tasks)
# Start up a simple loop and run our main function
# until it is complete
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print("All Tasks Completed")
loop.close()
Invoking a blocking call such as time.sleep in an asyncio coroutine blocks the whole event loop, defeating the purpose of using asyncio.
Change time.sleep(10) to await asyncio.sleep(10), and the code will behave like you expect.
asyncio use a loop to run everything, await would yield back the control to the loop so it can arrange the next coroutine to run.
I'm trying to asynchronously download a file in Python, using wget in a subprocess. My code looks like this:
async def download(url, filename):
wget = await asyncio.create_subprocess_exec(
'wget', url,
'O', filename
)
await wget.wait()
def main(url):
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(download(url, 'test.zip'), loop=loop)
print("Downloading..")
time.sleep(15)
print("Still downloading...")
loop.run_until_complete(future)
loop.close()
What I'm trying to do is witness the printing of "Downloading.." then 15 seconds later "Still downloading...", all while the download of the file has started. What I'm actually seeing is that the download of the file only starts when the code hits loop.run_until_complete(future)
My understanding is that asyncio.ensure_future should start running the code of the download coroutine, but apparently I'm missing something.
When passed a coroutine, asyncio.ensure_future converts it to a task - a special kind of future that knows how to drive the coroutine - and enqueues it in the event loop. "Enqueue" means that the code inside the coroutine will be executed by a running event loop that schedules the coroutines. If the event loop is not running, then none of the coroutines will get a chance to run either. The loop is told to run by a call to loop.run_forever() or loop.run_until_complete(some_future). In the question the event loop is only started after the call to time.sleep(), so the beginning of the download is delayed by 15 seconds.
time.sleep should never be called in a thread that runs the asyncio event loop. The correct way to sleep is with asyncio.sleep, which yields the control to the event loop while waiting. asyncio.sleep returns a future that can be submitted to the event loop or awaited from a coroutine:
# ... definition of download omitted ...
async def report():
print("Downloading..")
await asyncio.sleep(15)
print("Still downloading...")
def main(url):
loop = asyncio.get_event_loop()
dltask = loop.create_task(download(url, 'test.zip'))
loop.create_task(report())
loop.run_until_complete(dltask)
loop.close()
The above code has a different problem. When the download is shorter than 15 seconds, it results in a Task was destroyed but it is pending! warning being printed. The problem is that the report task was never canceled when the download task finished and the loop closed, it was just abandoned. This happening often indicates a bug or a misunderstanding of how asyncio works, so asyncio flags it with a warning.
The obvious way to eliminate the warning is by explicitly canceling the task of the report coroutine, but the resulting code ends up being verbose and not very elegant. An simpler and shorter fix is to change report to await the download task, specifying a timeout for displaying the "Still downloading..." message:
async def dl_and_report(dltask):
print("Downloading..")
try:
await asyncio.wait_for(asyncio.shield(dltask), 15)
except asyncio.TimeoutError:
print("Still downloading...")
# assuming we want the download to continue; otherwise
# remove the shield(), and dltask will be canceled
await dltask
def main(url):
loop = asyncio.get_event_loop()
dltask = loop.create_task(download(url, 'test.zip'))
loop.run_until_complete(dl_and_report(dltask))
loop.close()