I created a MCVE example of a much larger code base that I am working on, so some things look to be a funky way of doing things, but they are just the minimal versions, so not everything makes sense as far as WHY I am doing it this way. I already know some work-arounds, and at this point am mostly just curious as to why I am seeing this behaviour.
I have the following script which has an async function that waits for a future. The script then intercepts a signal to set the future result:
import asyncio
import time
import signal
f = asyncio.Future()
loop = asyncio.get_event_loop()
async def wait_till_signal():
await f
def send_stop():
print('stopping', time.time())
f.set_result(True)
print('sent')
print(f.result())
def handle_signal(s, a):
print('sending stop', time.time())
send_stop()
signal.signal(signal.SIGINT, handle_signal)
loop.run_until_complete(wait_till_signal())
This script correctly gets the interrupt, and appears to set the future correctly, but for some reason the script never terminates.
To reproduce for you, just run the script, then hit ctrl+c. For some reason it never stops.
Now here is where it gets weird. If you add the following to the top of the script (after defining the loop), then the script stops just fine.
async def do_nothing_useful():
for i in range(30):
await asyncio.sleep(1)
loop.create_task(do_nothing_useful())
Why is the coroutine not getting the future in the first case, but it gets it correctly in the second case?
Also, another weird thing is that if you set the send_stop function to be async, and add it as a task, it never gets called. (This follows the same behaviour as above. If the do_nothing_useful() function is on the loop, everything works fine, but without it, it doesnt)
Here is the version where the send_stop is never called:
import asyncio
import time
import signal
f = asyncio.Future()
async def wait_till_signal():
await f
async def send_stop():
# this is async only because we are trying to try out crazy things
print('stopping', time.time())
f.set_result(True)
print('sent')
print(f.result())
def handle_signal(s, a):
print('sending stop', time.time())
loop.create_task(send_stop())
signal.signal(signal.SIGINT, handle_signal)
loop = asyncio.get_event_loop()
loop.run_until_complete(wait_till_signal())
print('done')
and the script never prints stopping.
I have tried this on python 3.5.3 and 3.6 on linux
The correct method to add a signal handler that wakes up a loop is by using loop.add_signal_handler. This will make sure select() wakes up to handle the signal.
Related
I'm using a library that itself makes the call to asyncio.run(internal_function) so I can't control that at all. I do however have access to the event loop, it's something that I pass into this library.
Given that, is there some way I can set up an recurring async event that will execute every X seconds while the main library is running.
This doesn't exactly work, but maybe it's close?
import asyncio
from third_party import run
loop = asyncio.new_event_loop()
async def periodic():
while True:
print("doing a thing...")
await asyncio.sleep(30)
loop.create_task(periodic())
run(loop) # internally this will call asyncio.run() using the given loop
The problem here of course is that the task I've created is never awaited. But I can't just await it, because that would block.
Edit: Here's a working example of what I'm facing. When you run this code you will only ever see "third party code executing" and never see "doing my stuff...".
import asyncio
# I don't know how the loop argument is used
# by the third party's run() function,
def third_party_run(loop):
async def runner():
while True:
print("third party code executing")
await asyncio.sleep(5)
# but I do know that this third party eventually runs code
# that looks **exactly** like this.
try:
asyncio.run(runner())
except KeyboardInterrupt:
return
loop = asyncio.new_event_loop()
async def periodic():
while True:
print("doing my stuff...")
await asyncio.sleep(1)
loop.create_task(periodic())
third_party_run(loop)
If you run the above code you get:
third party code executing
third party code executing
third party code executing
^CTask was destroyed but it is pending!
task: <Task pending name='Task-1' coro=<periodic() running at example.py:22>>
/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:674: RuntimeWarning: coroutine 'periodic' was never awaited
You don't need to await on a created task.
It will run in the background as long as the event loop is active and is not stuck in a CPU bound operation.
According to your comment, you don't have an access to the event loop. In this case you don't have many options other than running in a different thread (which will have its own loop), or changing the loop creation policy in order to get the event loop, which is a very bad idea in most cases.
I found a way to make your test program run. However, it's a hack. It could fail, depending on the internal design of your third party library. From the information you provided, the library has been structured to be a black box. You can't interact with the event loop or schedule a callback. It seems like there might be a very good reason for this.
If I were you I would try to contact the library designer and let him know what your problem is. Perhaps there is a better solution. If this is a commercial project, I would make 100% certain that the team understands the issue, before attempting to use my below solution or anything like it.
The script below overrides one method (new_event_loop) in the DefaultEventLoopPolicy. When this method is called, I create a task in this loop to execute your periodic function. I don't know how often, or for what purpose, the library will call this function. Also, if the library internally overrides the EventLoopPolicy then this solution will not work. In both of these cases it may lead to unforeseeable consequences.
OK, enough disclaimers.
The only significant change to your test script was to replace the infinite loop in runner with a one that times out. This allowed me to verify that the program shuts down cleanly.
import asyncio
# I don't know how the loop argument is used
# by the third party's run() function,
def third_party_run():
async def runner():
for _ in range(4):
print("third party code executing")
await asyncio.sleep(5)
# but I do know that this third party eventually runs code
# that looks **exactly** like this.
try:
asyncio.run(runner())
except KeyboardInterrupt:
return
async def periodic():
while True:
print("doing my stuff...")
await asyncio.sleep(1)
class EventLoopPolicyHack(asyncio.DefaultEventLoopPolicy):
def __init__(self):
self.__running = None
super().__init__()
def new_event_loop(self):
# Override to create our periodic task in the new loop
# Get a loop from the superclass.
# This method must return that loop.
print("New event loop")
loop = super().new_event_loop()
if self.__running is not None:
self.__running.cancel() # I have no way to test this idea
self.__running = loop.create_task(periodic())
return loop
asyncio.set_event_loop_policy(EventLoopPolicyHack())
third_party_run()
I'm trying to create a wrapper around an asyncio coroutine that allows the user to use it as a "normal" function.
To give a bit of context, this is a function inside a package that war originally not-async. For a series of reasons, I now need to have an async version of it. To avoid duplicating the whole code, I'm trying to create a wrapper to allow existing code (that doesn't use asyncio) to keep running without breaking back compatibility.
To make things more complicated, the majority of the users (it's a company code) use this code inside Spyder IDE.
To sort it, I did something like this
import asyncio
async def an_async_subfunction(t, tag):
print(f"I'm inside an_async_subfunction named {tag}")
await asyncio.sleep(t)
print(f"Leaving an_async_subfunction named {tag}")
async def an_async_function(n):
print(f"I'm inside an_async_function")
tasks = [an_async_subfunction(t, t) for t in range(n)]
await asyncio.gather(*tasks)
print(f"Leaving an_async_function")
async def main_async(n):
# the old main function, now become a corouting
await an_async_function(n)
return 'a result'
def main(*args):
# the wrapper exposed to the users
return asyncio.run(main_async(*args))
if __name__ == '__main__':
print('Normal version')
# The user can call the main function without bothering with asyncio
result = main(3)
print('Async version')
# ...or can use the async version of main if he wants
async def a_user_defined_async_function():
return await main_async(3)
result = asyncio.run(a_user_defined_async_function())
This works as expected, allowing the basic user to call main without bothering that it is a coroutine, while if a user wants to use main inside a custom-made async function, he can use main_async.
However, if you try to run this code in Spyder, you get the error:
RuntimeError: asyncio.run() cannot be called from a running event loop
This is caused by the fact that Spyder has its own event loop running as explained here.
I tried to fix it doing something like:
def main(*args):
if asyncio.get_event_loop().is_running():
return asyncio.create_task(main_async(*args)).result()
else:
return asyncio.run(main_async(*args))
This is now "Spyder-friendly" an it works inside Spyder without problems. The problem is that .result() is called before the Task created by asyncio.create_task is finished and an InvalidStateError exception is returned.
I can't put an await in front of create_task as main is not a coroutine, and I can't make main a coroutine, otherwise the whole thing would have been pointless.
Is there a solution to this mess?
Just can't wrap my head around solving this issue, so maybe someone here can enlighten me or maybe even tell me that what I want to achieve isn't possible. :)
Problem statement:
I have an asyncio event loop, on that loop I create a task by supplying my asynchronous coroutine work(). I could then go ahead and cancel the task by invoking its cancel() method - this works.
But in my very special case, the asynchronous task itself spawns another operation, which is an underlying blocking / synchronous function.
What happens now, if I decide to cancel the task, is that my asynchronous work() function will be cancelled appropriately, however, the synchronous function is still going to be executed as if nothing ever happened.
I tried to make an example as simple as possible to illustrate my problem:
import asyncio
import time
def sync_work():
time.sleep(10)
print("sync work completed")
return "sync_work_result"
async def work(loop):
result = await loop.run_in_executor(None, sync_work)
print(f"sync_work {result}")
print("work completed")
async def main(loop):
t1 = loop.create_task(work(loop))
await asyncio.sleep(4)
t1.cancel()
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(main(loop))
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print("loop closing")
loop.close()
This will print out sync work completed after about 10 seconds.
How would I invoke the synchronous function in a way, that would allow me to terminate it once my asynchronous task is cancelled? The tricky part is, that I would not have control over sync_work() as this comes from another external package.
I'm open to other approaches of calling my synchronous function from an asynchronous function that would allow it to be terminated properly in some kind of way.
I'm trying to understand how asyncio works. As for I/O operation i got understand that when await was called, we register Future object in EventLoop, and then calling epoll for get sockets which belongs to Future objects, that ready for give us data. After we run registred callback and resume function execution.
But, the thing that i cant understant, what's happening if we use await not for I/O operation. How eventloop understands that task is complete? Is it create socket for that or use another kind of loop? Is it use epoll? Or doesnt it add to Loop and used it as generator?
There is an example:
import asyncio
async def test():
return 10
async def my_coro(delay):
loop = asyncio.get_running_loop()
end_time = loop.time() + delay
while True:
print("Blocking...")
await test()
if loop.time() > end_time:
print("Done.")
break
async def main():
await my_coro(3.0)
asyncio.run(main())
await doesn't automatically yield to the event loop, that happens only when an async function (anywhere in the chain of awaits) requests suspension, typically due to IO or timeout not being ready.
In your example the event loop is never returned to, which you can easily verify by moving the "Blocking" print before the while loop and changing main to await asyncio.gather(my_coro(3.0), my_coro(3.0)). What you'll observe is that the coroutines are executed in series ("blocking" followed by "done", all repeated twice), not in parallel ("blocking" followed by another "blocking" and then twice "done"). The reason for that was that there was simply no opportunity for a context switch - my_coro executed in one go as if they were an ordinary function because none of its awaits ever chose to suspend.
I was trying to explain an example of async programming in python but I failed.
Here is my code.
import asyncio
import time
async def asyncfoo(t):
time.sleep(t)
print("asyncFoo")
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncfoo(10)) # I think Here is the problem
print("Foo")
loop.close()
My expectation is that I would see:
Foo
asyncFoo
With a wait of 10s before asyncFoo was displayed.
But instead I got nothing for 10s, and then they both displayed.
What am I doing wrong, and how can I explain it?
run_until_complete will block until asyncfoo is done. Instead, you would need two coroutines executed in the loop. Use asyncio.gather to easily start more than one coroutine with run_until_complete.
Here is a an example:
import asyncio
async def async_foo():
print("asyncFoo1")
await asyncio.sleep(3)
print("asyncFoo2")
async def async_bar():
print("asyncBar1")
await asyncio.sleep(1)
print("asyncBar2")
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(async_foo(), async_bar()))
loop.close()
Your expectation would work in contexts where you run your coroutine as a Task independent of the flow of the code. Another situation where it would work is if you are running multiple coroutines side-by-side, in which case the event-loop will juggle the code execution from await to await statement.
Within the context of your example, you can achieve your anticipated behaviour by wrapping your coroutine in a Task object, which will continue-on in the background without holding up the remainder of the code in the code-block from whence it is called.
For example.
import asyncio
async def asyncfoo(t):
await asyncio.sleep(t)
print("asyncFoo")
async def my_app(t):
my_task = asyncio.ensure_future(asyncfoo(t))
print("Foo")
await asyncio.wait([my_task])
loop = asyncio.get_event_loop()
loop.run_until_complete(my_app(10))
loop.close()
Note that you should use asyncio.sleep() instead of the time module.
run_until_complete is blocking. So, even if it'll happen in 10 seconds, it will wait for it. After it's completed, the other print occurs.
You should launch your loop.run_until_complete(asyncfoo(10)) in a thread or a subprocess if you want the "Foo" to be print before.