I'm trying to consume multiple queues concurrently using python, asyncio and asynqp.
I don't understand why my asyncio.sleep() function call does not have any effect. The code doesn't pause there. To be fair, I actually don't understand in which context the callback is executed, and whether I can yield control bavck to the event loop at all (so that the asyncio.sleep() call would make sense).
What If I had to use a aiohttp.ClientSession.get() function call in my process_msg callback function? I'm not able to do it since it's not a coroutine. There has to be a way which is beyond my current understanding of asyncio.
#!/usr/bin/env python3
import asyncio
import asynqp
USERS = {'betty', 'bob', 'luis', 'tony'}
def process_msg(msg):
asyncio.sleep(10)
print('>> {}'.format(msg.body))
msg.ack()
async def connect():
connection = await asynqp.connect(host='dev_queue', virtual_host='asynqp_test')
channel = await connection.open_channel()
exchange = await channel.declare_exchange('inboxes', 'direct')
# we have 10 users. Set up a queue for each of them
# use different channels to avoid any interference
# during message consumption, just in case.
for username in USERS:
user_channel = await connection.open_channel()
queue = await user_channel.declare_queue('Inbox_{}'.format(username))
await queue.bind(exchange, routing_key=username)
await queue.consume(process_msg)
# deliver 10 messages to each user
for username in USERS:
for msg_idx in range(10):
msg = asynqp.Message('Msg #{} for {}'.format(msg_idx, username))
exchange.publish(msg, routing_key=username)
loop = asyncio.get_event_loop()
loop.run_until_complete(connect())
loop.run_forever()
I don't understand why my asyncio.sleep() function call does not have
any effect.
Because asyncio.sleep() returns a future object that has to be used in combination with an event loop (or async/await semantics).
You can't use await in simple def declaration because the callback is called outside of async/await context which is attached to some event loop under the hood. In other words mixing callback style with async/await style is quite tricky.
The simple solution though is to schedule the work back to the event loop:
async def process_msg(msg):
await asyncio.sleep(10)
print('>> {}'.format(msg.body))
msg.ack()
def _process_msg(msg):
loop = asyncio.get_event_loop()
loop.create_task(process_msg(msg))
# or if loop is always the same one single line is enough
# asyncio.ensure_future(process_msg(msg))
# some code
await queue.consume(_process_msg)
Note that there is no recursion in _process_msg function, i.e. the body of process_msg is not executed while in _process_msg. The inner process_msg function will be called once the control goes back to the event loop.
This can be generalized with the following code:
def async_to_callback(coro):
def callback(*args, **kwargs):
asyncio.ensure_future(coro(*args, **kwargs))
return callback
async def process_msg(msg):
# the body
# some code
await queue.consume(async_to_callback(process_msg))
See Drizzt1991's response on github for a solution.
Related
In an API developed using FastAPI framework, I am using asyncio to make calls to solr collections, but when I am calling asyncio.run(man(a,b)) from a python file query.py then I am getting asyncio.run() cannot be called from a running event loop.
in controller.py
#router.api_route()
async def api_call(a,b):
#calling a function
resp = query(a,b)
return resp
in query.py
def query(a,b):
result = asyncio.run(man(a,b))
return result
in database_call.py
async def man(a,b)
async with aiohttp.ClientSession() as session:
url = ''
async with session.get(pokemon_url) as resp:
result = await resp.json()
return result
when I am calling asyncio.run(man(a,b)) from query then I am getting asyncio.run() cannot be called from a running event loop. Kindly help me resolve the issue.
I tried:
in query.py
def query(a,b):
loop = asyncio.get_event_loop
result = loop.create_task(man(a,b))
return result
then I am getting <coroutine object main at 0x0394999ejt>
The docs say that you should have only one call to asyncio.run in a program. "Should" doesn't mean the same thing as "must", so it's not a requirement. But it's a good guideline.
Solution 1: get rid of query entirely, and just await the coroutine man() directly.
#router.api_route()
async def api_call(a,b):
return await man(a, b)
Solution 2: declare query to be an async def function, and await it:
#router.api_route()
async def api_call(a,b):
#calling a function
return await query(a,b)
async def query(a,b):
return await man(a,b)
Solution 3: Do not declare query to be async def, but have it return an awaitable object. This is similar to what you tried in your last listing, but you need to await the result.
#router.api_route()
async def api_call(a,b):
#calling a function
return await query(a,b)
def query(a, b):
return asyncio.create_task(man(a, b))
Solution 4: run query in another thread using the asyncio.to_thread function.
Solution 5: run query in a ThreadPool or ProcessPool.
The whole idea of asyncio (or any other form of parallel processing) is to allow the use of a task or a coroutine that does not finish immediately. It runs at a later time, and its result is not available until enough time has passed. Whether you use threading, multiprocessing or asyncio the situation is the same. If you need the answer from the coroutine/task/function in order for your program to continue, you need to wait for it somehow.
AWS IoT python Sdk has a function publishAsync
the signature of the function is publishAsync(topic, payload, QoS, ackCallback=None)
I want to warp this to a coroutine and use asyncio.
what I want to do is this:
connect to AWS
publish 10 messages asynchronously
disconnect
I don't know how should I wrap that function to a async function.
async def asyncPublish(self,
msg:str,
topic:str,
QoS=1):
# the publishAckFn
def internalPubAckHandler(mid):
print(json.dumps({
'messageID':mid,
'acknowledged':True
}))
return True
pass
# publish to the topic asynchronously
messageID = self.awsIoTClient.publishAsync(topic,msg,QoS,ackCallback=internalPubAckHandler)
print(json.dumps({
'messageID':messageID,
'topic':topic,
'payload':msg,
'QoS':QoS,
'async':True
}))
pass
---- in my main file
tasks = []
for i in range(10):
tasks += asyncPublish('test','test')
pass
loop = asyncio.get_event_loop()
loop.set_debug(True) # debug mode
gw1.connect()
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
gw1.disconnect()
but when I do this, the disconnect will be called sooner than internalPubAckHandler
and I will not get any acknowledgment.
Finally, I find the solution. We can use futures.
Future objects are used to bridge low-level callback-based code with high-level async/await code. Ref
If the signature of the function is something like this:
def functionWithCallback(args,callback)
and we need to wrap this to a async function that waits for the callback, we can do something like the following:
async def asyncVersion(args):
# one way of creating a future
loop = asyncio.get_running_loop()
future = loop.create_future()
def futureCallback():
# do something here and then signal the future
...
loop.call_soon_threadsafe(future .set_result,time.time())
pass
# call your function
functionWithCallback(args,futureCallback)
# wait for the callback
await future
# do other stuff after the callback is done
.....
pass
I want to use both ThreadPoolExecutor from concurrent.futures and async functions.
My program repeatedly submits a function with different input values to a thread pool. The final sequence of tasks that are executed in that larger function can be in any order, and I don't care about the return value, just that they execute at some point in the future.
So I tried to do this
async def startLoop():
while 1:
for item in clients:
arrayOfFutures.append(await config.threadPool.submit(threadWork, obj))
wait(arrayOfFutures, timeout=None, return_when=ALL_COMPLETED)
where the function submitted is:
async def threadWork(obj):
bool = do_something() # needs to execute before next functions
if bool:
do_a() # can be executed at any time
do_b() # ^
where do_b and do_a are async functions.The problem with this is that I get the error: TypeError: object Future can't be used in 'await' expression and if I remove the await, I get another error saying I need to add await.
I guess I could make everything use threads, but I don't really want to do that.
I recommend a careful readthrough of Python 3's asyncio development guide, particularly the "Concurrency and Multithreading" section.
The main conceptual issue in your example that event loops are single-threaded, so it doesn't make sense to execute an async coroutine in a thread pool. There are a few ways for event loops and threads to interact:
Event loop per thread. For example:
async def threadWorkAsync(obj):
b = do_something()
if b:
# Run a and b as concurrent tasks
task_a = asyncio.create_task(do_a())
task_b = asyncio.create_task(do_b())
await task_a
await task_b
def threadWork(obj):
# Create run loop for this thread and block until completion
asyncio.run(threadWorkAsync())
def startLoop():
while 1:
arrayOfFutures = []
for item in clients:
arrayOfFutures.append(config.threadPool.submit(threadWork, item))
wait(arrayOfFutures, timeout=None, return_when=ALL_COMPLETED)
Execute blocking code in an executor. This allows you to use async futures instead of concurrent futures as above.
async def startLoop():
while 1:
arrayOfFutures = []
for item in clients:
arrayOfFutures.append(asyncio.run_in_executor(
config.threadPool, threadWork, item))
await asyncio.gather(*arrayOfFutures)
Use threadsafe functions to submit tasks to event loops across threads. For example, instead of creating a run loop for each thread you could run all async coroutines in the main thread's run loop:
def threadWork(obj, loop):
b = do_something()
if b:
future_a = asyncio.run_coroutine_threadsafe(do_a(), loop)
future_b = asyncio.run_coroutine_threadsafe(do_b(), loop)
concurrent.futures.wait([future_a, future_b])
async def startLoop():
loop = asyncio.get_running_loop()
while 1:
arrayOfFutures = []
for item in clients:
arrayOfFutures.append(asyncio.run_in_executor(
config.threadPool, threadWork, item, loop))
await asyncio.gather(*arrayOfFutures)
Note: This example should not be used literally as it will result in all coroutines executing in the main thread while the thread pool workers just block. This is just to show an example of the run_coroutine_threadsafe() method.
I am trying to create a periodic task for an asyncio event loop as shown below, however I am getting a "RuntimeError: cannot reuse already awaited coroutine" exception. Apparently, asyncio does not allow for the same awaitable function to be awaited as discussed in this bug thread. This is how I tried to implement it:
import asyncio
class AsyncEventLoop:
def __init__(self):
self._loop = asyncio.get_event_loop()
def add_periodic_task(self, async_func, interval):
async def wrapper(_async_func, _interval):
while True:
await _async_func # This is where it goes wrong
await asyncio.sleep(_interval)
self._loop.create_task(wrapper(async_func, interval))
return
def start(self):
self._loop.run_forever()
return
Because of my while loop, the same awaitable function (_async_func) would be executed with a sleep interval in between. I got my inspiration for the implementation of periodic tasks from How can I periodically execute a function with asyncio?
.
From the bug thread mentioned above, I infer that the idea behind the RuntimeError was so that developers wouldn't accidentally await the same coroutine twice or more, as the coroutine would be marked as done and yield None instead of the result. Is there a way I can await the same function more than once?
It seems you are confusing async functions (coroutine functions) with coroutines - values that these async functions produce.
Consider this async function:
async def sample():
await asyncio.sleep(3.14)
You are passing result of its call: add_periodic_task(sample(), 5).
Instead, you should pass async function object itself: add_periodic_task(sample, 5), and call it within your wrapper:
while True:
await _async_func()
await asyncio.sleep(_interval)
I am trying to translate this key "debouncing" logic from Javascript to Python.
function handle_key(key) {
if (this.state == null) {
this.state = ''
}
this.state += key
clearTimeout(this.timeout)
this.timeout = setTimeout(() => {
console.log(this.state)
}, 500)
}
handle_key('a')
handle_key('b')
The idea is that subsequent key presses extend the timeout. The Javascript version prints:
ab
I don't want to translate the JS timeout functions, I'd rather have idiomatic Python using asyncio. My attempt in Python (3.5) is below, but it doesn't work as global_state is not actually updated when I expect.
import asyncio
global_state = ''
#asyncio.coroutine
def handle_key(key):
global global_state
global_state += key
local_state = global_state
yield from asyncio.sleep(0.5)
#if another call hasn't modified global_state we print it
if local_state == global_state:
print(global_state)
#asyncio.coroutine
def main():
yield from handle_key('a')
yield from handle_key('b')
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(main())
It prints:
a
ab
I have looked into asyncio Event, Queue and Condition but it isn't clear to me how to use them for this. How would you implement the desired behavior using Python's asyncio?
EDIT
Some more details on how I'd like to use handle_keys. I have an async function that checks for key presses.
#asyncio.coroutine
def check_keys():
keys = driver.get_keys()
for key in keys:
yield from handle_key(key)
Which in turn is scheduled along with other program tasks
#asyncio.coroutine
def main():
while True:
yield from check_keys()
yield from do_other_stuff()
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(main())
Qeek's use of asyncio.create_task and asyncio.gather makes sense. But how would I use it within a loop like this? Or is there another way to schedule the async tasks that would allow handle_keys calls to "overlap"?
Actual code on GitHub if you are interested.
Why your code doesn't work now?
Both handle_key javascript functions don't block execution. Each just clear timeout callback and set new one. It happens immediately.
Coroutines work another way: using yield from or newer syntax await on coroutine means that we want to resume execution flow only after this coroutine if fully done:
async def a():
await asyncio.sleep(1)
async def main():
await a()
await b() # this line would be reached only after a() done - after 1 second delay
asyncio.sleep(0.5) in your code - is not setting callback by timeout, but code that should be done before handle_key finsihed.
Let's try to make code work
You can create task to start execution some coroutine "in background". You can also cancel task (just like you do with clearTimeout(this.timeout)) if you don't want it to be finished.
Python version that emulates your javascript snippet:
import asyncio
from contextlib import suppress
global_state = ''
timeout = None
async def handle_key(key):
global global_state, timeout
global_state += key
# cancel previous callback (clearTimeout(this.timeout))
if timeout:
timeout.cancel()
with suppress(asyncio.CancelledError):
await timeout
# set new callback (this.timeout = setTimeout ...)
async def callback():
await asyncio.sleep(0.5)
print(global_state)
timeout = asyncio.ensure_future(callback())
async def main():
await handle_key('a')
await handle_key('b')
# both handle_key functions done, but task isn't finished yet
# you need to await for task before exit main() coroutine and close loop
if timeout:
await timeout
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
finally:
loop.close()
Idiomatic?
While code above works, it is not how asyncio should be used. Your javascript code based on callbacks, while asyncio usually is about to avoid using of callbacks.
It's hard to demonstrate difference on your example since it's callback based by nature (key handling - is some sort of global callback) and doesn't have more async logic. But this understanding would be important later when you'll add more async operations.
Right now I advice you to read about async/await in modern javascript (it's similar to Python's async/await) and look at examples comparing it to callbacks/promises. This article looks good.
It'll help you understand how you can use coroutine-based approach in Python.
Upd:
Since buttons.check needs to periodically call driver.get_buttons() you'll have to use loop. But it can be done as task along with your event loop.
If you had some sort of button_handler(callback) (this is usually how different libs allow to handle user input) you could use it to set some asyncio.Future directly and avoid loop.
Consider possibility write some little gui app with asyncio from the beginning. I think it may help you to better understand how you can adapt your existing project.
Here's some pseudo-code that shows background task to handle
buttons and using asyncio to handle some simple UI events/states logic:
.
import asyncio
from contextlib import suppress
# GUI logic:
async def main():
while True:
print('We at main window, popup closed')
key = await key_pressed
if key == 'Enter':
print('Enter - open some popup')
await popup()
# this place wouldn't be reached until popup is not closed
print('Popup was closed')
elif key == 'Esc':
print('Esc - exit program')
return
async def popup():
while True:
key = await key_pressed
if key == 'Esc':
print('Esc inside popup, let us close it')
return
else:
print('Non escape key inside popup, play sound')
# Event loop logic:
async def button_check():
# Where 'key_pressed' is some global asyncio.Future
# that can be used by your coroutines to know some key is pressed
while True:
global key_pressed
for key in get_buttons():
key_pressed.set_result(key)
key_pressed = asyncio.Future()
await asyncio.sleep(0.01)
def run_my_loop(coro):
loop = asyncio.get_event_loop()
# Run background task to process input
buttons_task = asyncio.ensure_future(button_check())
try:
loop.run_until_complete(main())
finally:
# Shutdown task
buttons_task.cancel()
with suppress(asyncio.CancelledError):
loop.run_until_complete(buttons_task)
loop.close()
if __name__ == '__main__':
run_my_loop(main())
What's wrong
Basically the yield from xy() is very similar to the normal function call. The difference between function call and yield from is that the function call immediately start processing called function. The yield from statement insert called coroutine into queue inside event loop and give control to event loop and it decide which coroutine in it's queue will be processed.
Here is the explanation of what you code does:
It adds the main into event loop's queue.
The event loop start processing coroutine in the queue.
The queue contains only the main coroutine so it starts that.
The code hits the yield from handle_key('a').
It adds the handle_key('a') in the event loop's queue.
The event loop now contains the main and handle_key('a') but the main cannot be started because it is waiting for the result of the handle_key('a').
So the event loop starts the handle_key('a').
It will do some stuff until it hits the yield from asyncio.sleep(0.5).
Now the event loop contains main(), handle_key('a') and sleep(0.5).
The main() is waiting for result from handle_key('a').
The handle_key('a') is waiting for result from sleep(0.5).
The sleep has no dependency so it can be started.
The asyncio.sleep(0.5) returns None after 0.5 second.
The event loop takes the None and return it into the handle_key('a') coroutine.
The return value is ignored because it isn't assign into anything
The handle_key('a') prints the key (because nothing change the state)
The handle_key coroutine at the end return None (because there isn't return statement).
The None is returned to the main.
Again the return value is ignored.
The code hits the yield from handle_key('b') and start processing new key.
It run same steps from step 5 (but with the key b).
How to fix it
The main coroutinr replace with this:
#asyncio.coroutine
def main(loop=asyncio.get_event_loop()):
a_task = loop.create_task(handle_key('a'))
b_task = loop.create_task(handle_key('b'))
yield from asyncio.gather(a_task, b_task)
The loop.create_task adds into the event loop's queue the handle_key('a') and handle_key('b') and then the yield from asyncio.gather(a_task, b_task) give control to the event loop. The event loop from this point contains handle_key('a'), handle_key('b'), gather(...) and main().
The main() wiating for result from gather()
The gather() waiting until all tasks given as parameters are finished
The handle_key('a') and handle_key('b') has no dependencies so they can be started.
The event loop now contains 2 coroutine which can start but which one will it pick? Well... who knows it is implementation depended. So for better simulation of pressed keys this one replace should be a little better:
#asyncio.coroutine
def main(loop=asyncio.get_event_loop()):
a_task = loop.create_task(handle_key('a'))
yield from asyncio.sleep(0.1)
b_task = loop.create_task(handle_key('b'))
yield from asyncio.gather(a_task, b_task)
Python 3.5 bonus
From the documentation:
Coroutines used with asyncio may be implemented using the async def statement.
The async def type of coroutine was added in Python 3.5, and is recommended if there is no need to support older Python versions.
It means that you can replace:
#asyncio.coroutine
def main():
with newer statement
async def main():
If you start using the new syntax then you have to also replace yield from with await.