I have legacy python application which is synchronous.
I started to use async code inside this application in this way (simplified):
async def loader():
async with trio.open_nursery() as nursery:
# some async tasks started here
await trio.to_thread.run_sync(legacyCode)
if __name__ == '__main__':
trio.run(loader)
Inside legacyCode I can use trio.from_thread.run(asyncMethod) to run some async code from the legacy synchronous code.
It works well, but now I need to include new library (triopg) which use internally trio_asyncio.
So I need to modify the way how I start my application - I need to replace trio.run by trio_asyncio.run. That's easy but after the trio.to_thread -> trio.from_thread the async code does not work because trio_asyncio has no loop defined.
Here is a short demonstration:
import trio
import trio_asyncio
def main():
trio.from_thread.run(amain)
async def amain():
print(f"Loop in amain: {trio_asyncio.current_loop.get()}") # this print none
async def loader():
print(f"Loop in loader: {trio_asyncio.current_loop.get()}") # this print some loop
await trio.to_thread.run_sync(main)
if __name__ == '__main__':
trio_asyncio.run(loader)
How should I modify the example above so the trio_asyncio is able to found the loop inside amain() function?
Or is this approach completely wrong? If so, how can I use small pieces of async code inside huge synchronous application when libraries needs to use trio and trio_asyncio?
I use python 3.9.
Finally I found the solution and ... it seems to be easy :-)
The trio_asyncio loop needs to be opened manually when we call the async function from thread. So the only difference is to add open_loop() call in amain() function:
import trio
import trio_asyncio
def main():
trio.from_thread.run(amain)
async def amain():
async with trio_asyncio.open_loop():
print(f"Loop in amain: {trio_asyncio.current_loop.get()}") # this print another loop
async def loader():
print(f"Loop in loader: {trio_asyncio.current_loop.get()}") # this print one loop
await trio.to_thread.run_sync(main)
if __name__ == '__main__':
trio_asyncio.run(loader)
Related
Im trying to run a blocking task asynchronously with ProcessPoolExecutor (It works with ThreadPoolExecutor but I need ProcessPoolExecutor for CPU-bound task). Here is my code :
import asyncio
import time
from concurrent.futures import ProcessPoolExecutor
async def run_in_thread(task, *args):
with ProcessPoolExecutor() as process_pool:
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(process_pool, task, *args)
return result
async def main_task():
while True:
await asyncio.sleep(1)
print("ticker")
async def main():
asyncio.create_task(main_task())
global blocking_task
def blocking_task():
time.sleep(5)
print("blocking task done!")
await run_in_thread(blocking_task)
if __name__ == "__main__":
asyncio.run(main())
And I get this error :
result = await loop.run_in_executor(process_pool, task, *args)
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
I don't understand where is the issue, can someone please help me?
I'd also like to understand why it works with ThreadPoolExecutor but not ProcessPoolExecutor
I was expecting the code to print :
ticker
ticker
ticker
ticker
ticker
blocking task done!
Move the definition of blocking_task to the outer level of the module. As the script stands this function is invisible to other Processes. The code of the function isn't sent directly to the other Process, only its name. The other Process performs its own separate import of the script but the name isn't defined at the top level.
It's the same logic as if you tried to import this script into another script. Let's say this script is in a file named foo.py. After you do import foo, there is no function named foo.blocking_task so you would be unable to call it.
This is a little bit more clear if you looked at the whole traceback, instead of just the last line.
Incidentally, using the global statement in front of the function definition isn't the same thing as moving the definition to the top level. In your script the name blocking_task does not exist at module level until the main() function actually runs (which the secondary Process never does). In the working script below, the name blocking_task exists as soon as the module is imported.
import asyncio
import time
from concurrent.futures import ProcessPoolExecutor
async def run_in_thread(task, *args):
with ProcessPoolExecutor() as process_pool:
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(process_pool, task, *args)
return result
async def main_task():
while True:
await asyncio.sleep(1)
print("ticker")
def blocking_task():
time.sleep(5)
print("blocking task done!")
async def main():
asyncio.create_task(main_task())
await run_in_thread(blocking_task)
if __name__ == "__main__":
asyncio.run(main())
This prints exactly what you were expecting.
Can you help me see what I have understood wrong here please. I have two functions and I would like the second one to run regardless of the status of the first one (whether it is finished or not). Hence I was thinking to make the first function asynchronous. This is what I have done
import os
import asyncio
from datetime import datetime
async def do_some_iterations():
for i in range(10):
print(datetime.now().time())
await asyncio.sleep(1)
print('... Cool!')
async def main():
task = asyncio.create_task (do_some_iterations())
await task
def do_something():
print('something...')
if __name__ == "__main__":
asyncio.run(main())
do_something()
The output is:
00:46:00.145024
00:46:01.148533
00:46:02.159751
00:46:03.169868
00:46:04.179915
00:46:05.187242
00:46:06.196356
00:46:07.207614
00:46:08.215997
00:46:09.225066
Cool!
something...
which looks like the traditional way where one function has to finish and then move to the next call.
I was hoping instead to execute do_something() before the asynchronous function started generating the print statements (or at lease at the very top of those statements..)
What am I doing wrong please? How I should edit the script?
They both need to be part of the event loop the you created. asyncio.run() itself is not async, which means it will run until the loop ends. One easy way to do this is to use gather()
import asyncio
from datetime import datetime
async def do_some_iterations():
for i in range(10):
print(datetime.now().time())
await asyncio.sleep(1)
print('... Cool!')
async def do_something():
print('something...')
async def main():
await asyncio.gather(
do_some_iterations(),
do_something()
)
if __name__ == "__main__":
asyncio.run(main())
print("done")
This will print:
16:08:38.921879
something...
16:08:39.922565
16:08:40.923709
16:08:41.924823
16:08:42.926004
16:08:43.927044
16:08:44.927877
16:08:45.928724
16:08:46.929589
16:08:47.930453
... Cool!
done
You can also simply add another task:
async def main():
task = asyncio.create_task(do_some_iterations())
task2 = asyncio.create_task(do_something())
In both cases the function needs to be awaitable.
I'm new to coroutines in python but I know in lua you would do this to create one
coroutine.wrap(function()
while true do
end
end)()
But I dont know how I would do this python.
I tried using this code which included a function from the slp_coroutine library which didnt work
import slp_coroutine
def test():
while True:
print("hi")
time.sleep(3)
r = slp_coroutine.await_coroutine(test())
and I also tried this and it also didnt work.
async def test():
while True:
print("hi")
await asyncio.sleep(3)
asyncio.run(test())
In my simple asyncio Python program below, bar_loop is supposed to run continuously with a 1 second delay between loops.
Things run as expected when we have simply
async def bar_loop(self):
while True:
print('bar')
However, when we add a asyncio.sleep(1), the loop will end instead of looping.
async def bar_loop(self):
while True:
print('bar')
await asyncio.sleep(1)
Why does asyncio.sleep() cause bar_loop to exit immediately? How can we let it loop with a 1 sec delay?
Full Example:
import asyncio
from typing import Optional
class Foo:
def __init__(self):
self.bar_loop_task: Optional[asyncio.Task] = None
async def start(self):
self.bar_loop_task = asyncio.create_task(self.bar_loop())
async def stop(self):
if self.bar_loop_task is not None:
self.bar_loop_task.cancel()
async def bar_loop(self):
while True:
print('bar')
await asyncio.sleep(1)
if __name__ == '__main__':
try:
foo = Foo()
asyncio.run(foo.start())
except KeyboardInterrupt:
asyncio.run(foo.stop())
Using Python 3.9.5 on Ubuntu 20.04.
This behavior has nothing to do with calling asyncio.sleep, but with the expected behavior of creating a task and doing nothing else.
Tasks will run in parallel in the the asyncio loop, while other code that uses just coroutine and await expressions can be thought as if run in a linear pattern - however, as the are "out of the way" of the - let's call it "visible path of execution", they also won't prevent that flow.
In this case, your program simply reaches the end of the start method, with nothing left being "awaited", the asyncio loop simply finishes its execution.
If you have no explicit code to run in parallel to bar_loop, just await for the task. Change your start method to read:
async def start(self):
self.bar_loop_task = asyncio.create_task(self.bar_loop())
try:
await self.bar_loop_task
except XXX:
# handle excptions that might have taken place inside the task
I need to call an async function inside the on_next of a python rx subscription as this:
from rx.subject import Subject
import asyncio
async def asyncPrint(value: str):
print(f'async print: {value}')
async def main():
s = Subject()
s.subscribe(
on_error=lambda e: print(e),
on_next=lambda value: asyncPrint(value)
)
s.on_next('Im from the subject')
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
but I get the async error:
$ python test.py
rx\core\observer\autodetachobserver.py:26:
RuntimeWarning: coroutine 'asyncPrint' was never awaited
self._on_next(value)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
I really don't want to use asyncio.get_event_loop().run_until_complete(...) because I'm already running the main loop and I don't want to start a new one neither use nested loops.
I searched a about it and I saw that a lambda function can't be async. And I think maybe that's the problem here, because I really don't know how to get the on_next value without a lambda funtion using rx library.
I also searched about async-rx library, but this looks like the only different thing that can be done is an await subscribe(...), but that's not what I want. I want something like subscribe(on_next=await...).
Is that possible? from my javascript background its easy to start a async function inside a subscribe so it looks like a possible task for me. Hope someone could find a solution to that.
Thank's a lot!
finally I manage to make this work by using asyncio.run_task()
from rx.subject import Subject
import asyncio
async def asyncPrint(value: str):
print(f'async print: {value}')
async def main():
s = Subject()
s.subscribe(
on_error=lambda e: print(e),
on_next=lambda value: asyncio.create_task(asyncPrint(value))
)
s.on_next('Im from the subject')
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
output:
$ python teste.py
async print: Im from the subject