I'd like to define what essentially is an asynchronous __del__ that closes a resource. Here's an example.
import asyncio
class Async:
async def close(self):
print('closing')
return self
def __del__(self):
print('destructing')
asyncio.ensure_future(self.close())
async def amain():
Async()
if __name__ == '__main__':
asyncio.run(amain())
This works, printing destructing and closing as expected. However, if the resource is defined outside an asynchronous function, __del__ is called, but closing is never performed.
def main():
Async()
No warning is raised here, but the prints reveal that closing was not done. The warning is issued if an asynchronous function has been run, but any instance is created outside of it.
def main2():
Async()
asyncio.run(amain())
RuntimeWarning: coroutine 'Async.close' was never awaited
This has been the subject in 1 and 2, but neither quite had what I was looking for, or maybe I didn't know how to look. Particularly the first question was about deleting a resource, and its answer suggested using asyncio.ensure_future, which was tested above. Python documentation suggests using the newer asyncio.create_task, but it straight up raises an error in the non-async case, there being no current loop. My final, desperate attempt was to use asyncio.run, which worked for the non-async case, but not for the asynchronous one, as calling run is prohibited in a thread that already has a running loop. Additionally, the documentation states that it should only be called once in a program.
I'm still new to async things. How could this be achieved?
A word on the use case, since asynchronous context managers were mentioned as the preferred alternative in comments. I agree, using them for short-term resource management would be ideal. However, my use case is different for two reasons.
Users of the class are not necessarily aware of the underlying resources. It is better user experience to hide closing the resource from a user who doesn't fiddle with the resource itself.
The class needs to be instantiated (or for it to be possible to instantiate it) in a synchronous context, and it is often created just once. For example, in a web server context the class would be instantiated in the global scope, after which its async functions would be used in the endpoint definitions.
For example:
asc = Async()
server.route('/', 'GET')
async def root():
return await asc.do_something(), 200
I'm open to other suggestions of implementing such a feature, but at this point even my curiosity for the possibility that this can be done is enough for me to want an answer to this specific question, not just the general problem.
Only thing that comes to mind is to run cleanup after the server shutdown. It'll look something like this:
asc = Async()
try:
asyncio.run(run_server()) # You already do it now somewhere
finally:
asyncio.run(asc.close())
Since asyncio.run creates new event loop each time, you may want to go even deeper and reuse the same event loop:
loop = asyncio.get_event_loop()
asc = Async()
try:
loop.run_until_complete(run_server())
finally:
loop.run_until_complete(asc.close())
It's absolutely ok to call run_until_complete multiple times as long as you know what you're doing.
Full example with your snippet:
import asyncio
class Async:
async def close(self):
print('closing')
return self
async def cleanup(self):
print('destructing')
await self.close()
loop = asyncio.get_event_loop()
asc = Async()
async def amain():
await asyncio.sleep(1) # Do something
if __name__ == '__main__':
try:
loop.run_until_complete(amain())
finally:
loop.run_until_complete(asc.cleanup())
loop.close()
Related
I would like to know if the following is a safe use of futures to bridge
callback-based code to asynchronous code.
I hope the following illustrates what I mean.
I would like to communicate between two coroutines. One coroutine is mainly
running a library (not my code, represented by b method below), which provides
a synchronous callback which is called whenever an event happens. (Note that the library is using asyncio, but this particular API it provides uses a synchronous callback). I would like
to get data from that callback into a second coroutine (my code, represented by
a method below).
I don't care about missing messages (X objects). callback is called
frequently, much more often than we actually need X objects. It is no problem to
just wait for the next if we miss it. And we don't always need the latest X,
there are only certain situations when we need to refresh it (get a new one),
the rest of the time we just ignore them.
Can I share a future across coroutines like this, or do I need some kind of locking around setting/accessing self.fut? I can't actually use an asyncio.Lock in callback (a synchronous function) because I would need to use await lock.acquire() or async with lock: .... As mentioned above, I can't change this API to make callback an async function.
Also, are there any better alternative approaches? I could use an unlimited asyncio.Queue and use put_nowait synchronously from
the callback function. But it seems like overkill to use a Queue for the reasons
above (I only care about the latest, I don't care about missing messages, I
ignore most of them). The queue would (should) only ever have at most 1 item.
import asyncio
from typing import Optional
class C:
fut: Optional[asyncio.Future[X]]
def __init__(self):
self.fut = None
async def a(self):
while True:
...
# Some condition comes up where we need an X
# We are signalling to b() that we want an X object by setting
# self.fut to a Future (making it not None)
fut = asyncio.get_running_loop().create_future()
self.fut = fut
x = await fut
...
# (use the X object)
async def b(self):
while True:
...
# Calls synchronous callback when it has an X object ready
self.callback(x)
def callback(self, x: X):
...
# callback does other things as well as sending X objects over the Future
# But if the future is not None, it will use it send the X object
fut = self.fut
# Reset it to None because we only want one X object
self.fut = None
if fut is not None:
fut.set_result(x)
async def main():
c = C()
tasks = asyncio.create_task(c.a()), asyncio.create_task(c.b())
for coro in asyncio.as_completed(tasks)
await coro
if __name__ == '__main__':
asyncio.run(main())
I'm using a library that itself makes the call to asyncio.run(internal_function) so I can't control that at all. I do however have access to the event loop, it's something that I pass into this library.
Given that, is there some way I can set up an recurring async event that will execute every X seconds while the main library is running.
This doesn't exactly work, but maybe it's close?
import asyncio
from third_party import run
loop = asyncio.new_event_loop()
async def periodic():
while True:
print("doing a thing...")
await asyncio.sleep(30)
loop.create_task(periodic())
run(loop) # internally this will call asyncio.run() using the given loop
The problem here of course is that the task I've created is never awaited. But I can't just await it, because that would block.
Edit: Here's a working example of what I'm facing. When you run this code you will only ever see "third party code executing" and never see "doing my stuff...".
import asyncio
# I don't know how the loop argument is used
# by the third party's run() function,
def third_party_run(loop):
async def runner():
while True:
print("third party code executing")
await asyncio.sleep(5)
# but I do know that this third party eventually runs code
# that looks **exactly** like this.
try:
asyncio.run(runner())
except KeyboardInterrupt:
return
loop = asyncio.new_event_loop()
async def periodic():
while True:
print("doing my stuff...")
await asyncio.sleep(1)
loop.create_task(periodic())
third_party_run(loop)
If you run the above code you get:
third party code executing
third party code executing
third party code executing
^CTask was destroyed but it is pending!
task: <Task pending name='Task-1' coro=<periodic() running at example.py:22>>
/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:674: RuntimeWarning: coroutine 'periodic' was never awaited
You don't need to await on a created task.
It will run in the background as long as the event loop is active and is not stuck in a CPU bound operation.
According to your comment, you don't have an access to the event loop. In this case you don't have many options other than running in a different thread (which will have its own loop), or changing the loop creation policy in order to get the event loop, which is a very bad idea in most cases.
I found a way to make your test program run. However, it's a hack. It could fail, depending on the internal design of your third party library. From the information you provided, the library has been structured to be a black box. You can't interact with the event loop or schedule a callback. It seems like there might be a very good reason for this.
If I were you I would try to contact the library designer and let him know what your problem is. Perhaps there is a better solution. If this is a commercial project, I would make 100% certain that the team understands the issue, before attempting to use my below solution or anything like it.
The script below overrides one method (new_event_loop) in the DefaultEventLoopPolicy. When this method is called, I create a task in this loop to execute your periodic function. I don't know how often, or for what purpose, the library will call this function. Also, if the library internally overrides the EventLoopPolicy then this solution will not work. In both of these cases it may lead to unforeseeable consequences.
OK, enough disclaimers.
The only significant change to your test script was to replace the infinite loop in runner with a one that times out. This allowed me to verify that the program shuts down cleanly.
import asyncio
# I don't know how the loop argument is used
# by the third party's run() function,
def third_party_run():
async def runner():
for _ in range(4):
print("third party code executing")
await asyncio.sleep(5)
# but I do know that this third party eventually runs code
# that looks **exactly** like this.
try:
asyncio.run(runner())
except KeyboardInterrupt:
return
async def periodic():
while True:
print("doing my stuff...")
await asyncio.sleep(1)
class EventLoopPolicyHack(asyncio.DefaultEventLoopPolicy):
def __init__(self):
self.__running = None
super().__init__()
def new_event_loop(self):
# Override to create our periodic task in the new loop
# Get a loop from the superclass.
# This method must return that loop.
print("New event loop")
loop = super().new_event_loop()
if self.__running is not None:
self.__running.cancel() # I have no way to test this idea
self.__running = loop.create_task(periodic())
return loop
asyncio.set_event_loop_policy(EventLoopPolicyHack())
third_party_run()
I'm trying to run some IO blocking code, so I'm using to_thread to send the function to another thread. I tried several things, but in all cases, I seem to have to await the to_thread which just returns another coroutine (?!) that I then have to await again. For some reason this isn't quite clicking.
import asyncio
async def search(keyword_list):
coroutines = set()
for index, kw in enumerate(keyword_list):
coroutines.add(asyncio.to_thread(do_lookup, keyword, index))
for result in asyncio.as_completed(coroutines):
outcome, index = await result
# Do some magic with the outcome and index
# BUT, this doesn't work because `await result` apparently
# just returns ANOTHER coroutine!
async def do_lookup(keyword, index):
# Do long, blocking stuff here
print(f'running...{keyword} {index}')
return keyword, index
if __name__ == '__main__':
asyncio.run(search([1, 2, 3, 4]))
As I was copy/pasting and adapting my code to make a generic example, I discovered the problem here.
do_lookup is supposed to a synchronous function (because of the usage of to_thread), so by defining it async dev do_lookup I'm instead defining it as an asynchronous function, thereby causing the "double" await issue.
Simply redefining do_lookup without the async keyword did the trick!
I'm trying to create a wrapper around an asyncio coroutine that allows the user to use it as a "normal" function.
To give a bit of context, this is a function inside a package that war originally not-async. For a series of reasons, I now need to have an async version of it. To avoid duplicating the whole code, I'm trying to create a wrapper to allow existing code (that doesn't use asyncio) to keep running without breaking back compatibility.
To make things more complicated, the majority of the users (it's a company code) use this code inside Spyder IDE.
To sort it, I did something like this
import asyncio
async def an_async_subfunction(t, tag):
print(f"I'm inside an_async_subfunction named {tag}")
await asyncio.sleep(t)
print(f"Leaving an_async_subfunction named {tag}")
async def an_async_function(n):
print(f"I'm inside an_async_function")
tasks = [an_async_subfunction(t, t) for t in range(n)]
await asyncio.gather(*tasks)
print(f"Leaving an_async_function")
async def main_async(n):
# the old main function, now become a corouting
await an_async_function(n)
return 'a result'
def main(*args):
# the wrapper exposed to the users
return asyncio.run(main_async(*args))
if __name__ == '__main__':
print('Normal version')
# The user can call the main function without bothering with asyncio
result = main(3)
print('Async version')
# ...or can use the async version of main if he wants
async def a_user_defined_async_function():
return await main_async(3)
result = asyncio.run(a_user_defined_async_function())
This works as expected, allowing the basic user to call main without bothering that it is a coroutine, while if a user wants to use main inside a custom-made async function, he can use main_async.
However, if you try to run this code in Spyder, you get the error:
RuntimeError: asyncio.run() cannot be called from a running event loop
This is caused by the fact that Spyder has its own event loop running as explained here.
I tried to fix it doing something like:
def main(*args):
if asyncio.get_event_loop().is_running():
return asyncio.create_task(main_async(*args)).result()
else:
return asyncio.run(main_async(*args))
This is now "Spyder-friendly" an it works inside Spyder without problems. The problem is that .result() is called before the Task created by asyncio.create_task is finished and an InvalidStateError exception is returned.
I can't put an await in front of create_task as main is not a coroutine, and I can't make main a coroutine, otherwise the whole thing would have been pointless.
Is there a solution to this mess?
I'm running into some strange errors with initialising Locks and running asynchronous code. Suppose we had a class to use with some resource protected by a lock.
import asyncio
class C:
def __init__(self):
self.lock = asyncio.Lock()
async def foo(self):
async with self.lock:
return 'foo'
def async_foo():
c = C()
asyncio.run(c.foo())
if __name__ == '__main__':
async_foo()
async_foo()
This throws an error when run. It occurs on lock initialisation in init.
RuntimeError: There is no current event loop in thread 'MainThread'.
So duplicating the asyncio.run call in the function does not have this effect. It seems that the object needs to be initialised multiple times. It is also not enough to instantiate multiple locks in a single constructor. So perhaps it has something to do with the event loops state after asyncio.run is called.
What is going on? And how could I modify this code to work? Let me also clarify a bit, the instance is created outside asyncio.run and async functions for a reason. I'd like for it to be usable elsewhere too. If that makes a difference.
Alternatively, can threading.Lock be used for async things also? It would have the added benefit of being thread-safe, which asyncio.Lock reportedly is not.
What is going on?
When async object is created (asyncio.Lock()) it is attached to current event loop and can only be used with it
Main thread have some default current event loop (but other threads you create won't have default event loop)
asyncio.run() internally creates new event loop, set it current and close it after finished
So you're trying to use lock with event loop other than one it was attached to on creation. It leads to errors.
And how could I modify this code to work?
Ideal solution is following:
import asyncio
async def main():
# all your code is here
if __name__ == "__main__":
asyncio.run(main())
This will guarantee that every async object created is attached to proper event loop asyncio.run has created.
Running event loop (inside asyncio.run) is meant to be global "entry point" of your async program.
I'd like for it to be usable elsewhere too.
You're able to create an object outside asyncio.run, but then you should you should move creating async object from __init__ somewhere elsewhere so that asyncio.Lock() wouldn't be created until asyncio.run() is called.
Alternatively, can threading.Lock be used for async things also?
No, it is used to work with threads, while asyncio operates coroutines inside a single thread (usually).
It would have the added benefit of being thread-safe, which asyncio.Lock reportedly is not.
In asyncio you usually don't need threads other than main. There're still some reasons to do it, but thread-unsafety of asyncio.Lock shouldn't be an issue.
Consider reading following links. It may help to comprehend a situation better:
why we need asyncio/threads at all
When should I write asynchronous code instead of synchronous?