I'm currently trying to learn asyncio in Python. I know that the await keyword tells the loop that it can switch coroutines. However, when should I actually use it? Why not put it before everything?
Additionally, why is the await before 'response.text()', why not before the session.get(url)?
async def print_preview(url):
# connect to the server
async with aiohttp.ClientSession() as session:
# create get request
async with session.get(url) as response:
# wait for response
response = await response.text()
# print first 3 not empty lines
count = 0
lines = list(filter(lambda x: len(x) > 0, response.split('\n')))
print('-'*80)
for line in lines[:3]:
print(line)
print()
You use await with functions that are marked as coroutine in the documentation. For example, ClientResponse.text is marked as coroutine, while ClientResponse.close is not, which means you must await the former and must not await the latter. If you forget to await a coroutine, it simply won't execute and its return value will be a "coroutine object", which is useless (except for use with await).
session.get() returns an async context manager. When passed to async with, the coroutines it implements are awaited behind the scenes.
Also note that awaiting is not the only thing you can do with coroutines, the other is converting them into tasks, which allows them to run in parallel (without additional cost on the OS level). For more information, consult a tutorial on asyncio.
Related
In an API developed using FastAPI framework, I am using asyncio to make calls to solr collections, but when I am calling asyncio.run(man(a,b)) from a python file query.py then I am getting asyncio.run() cannot be called from a running event loop.
in controller.py
#router.api_route()
async def api_call(a,b):
#calling a function
resp = query(a,b)
return resp
in query.py
def query(a,b):
result = asyncio.run(man(a,b))
return result
in database_call.py
async def man(a,b)
async with aiohttp.ClientSession() as session:
url = ''
async with session.get(pokemon_url) as resp:
result = await resp.json()
return result
when I am calling asyncio.run(man(a,b)) from query then I am getting asyncio.run() cannot be called from a running event loop. Kindly help me resolve the issue.
I tried:
in query.py
def query(a,b):
loop = asyncio.get_event_loop
result = loop.create_task(man(a,b))
return result
then I am getting <coroutine object main at 0x0394999ejt>
The docs say that you should have only one call to asyncio.run in a program. "Should" doesn't mean the same thing as "must", so it's not a requirement. But it's a good guideline.
Solution 1: get rid of query entirely, and just await the coroutine man() directly.
#router.api_route()
async def api_call(a,b):
return await man(a, b)
Solution 2: declare query to be an async def function, and await it:
#router.api_route()
async def api_call(a,b):
#calling a function
return await query(a,b)
async def query(a,b):
return await man(a,b)
Solution 3: Do not declare query to be async def, but have it return an awaitable object. This is similar to what you tried in your last listing, but you need to await the result.
#router.api_route()
async def api_call(a,b):
#calling a function
return await query(a,b)
def query(a, b):
return asyncio.create_task(man(a, b))
Solution 4: run query in another thread using the asyncio.to_thread function.
Solution 5: run query in a ThreadPool or ProcessPool.
The whole idea of asyncio (or any other form of parallel processing) is to allow the use of a task or a coroutine that does not finish immediately. It runs at a later time, and its result is not available until enough time has passed. Whether you use threading, multiprocessing or asyncio the situation is the same. If you need the answer from the coroutine/task/function in order for your program to continue, you need to wait for it somehow.
I am trying to use use grequests to make a single HTTP call asynchronously.
All goes find until I try to call a function (handle_cars) to handle the response.
The problem is that the function is an async function and it I don't know how to await it while passing.
Is this even possible?
I need the function to be async because there is another async function I need to call from it. Another solution would be to be able to call and await the other async function (send_cars) from inside the sync function.
async def handle_cars(res):
print("GOT IT")
await send_cars()
async def get_cars():
req = grequests.get(URL, hooks=dict(response=handle_cars))
job = grequests.send(req, grequests.Pool(1))
How do I set the response argument to await the function? Or how do I await send_cars if I make handle_cars synchronous?
Thanks
According to the OP's comments, he wishes to create a request in the background, and call a callback when it finishes.
The way to do so can be implemented using asyncio.create_task() and using task.add_done_callback().
A simpler way however, will work by creating a new coroutine and running it in the background.
Demonstrated using aiohttp, it will work in the following way:
async def main_loop():
task = asyncio.create_task(handle_cars(URL))
while True:
...
async def handle_cars(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
await send_cars()
I am trying to learn async, and now I am trying to get whois information for a batch of domains. I found this lib aiowhois, but there are only a few strokes of information, not enough for such newbie as I am.
This code works without errors, but I don't know how to print data from parsed whois variable, which is coroutine object.
resolv = aiowhois.Whois(timeout=10)
async def coro(url, sem):
parsed_whois = await resolv.query(url)
async def main():
tasks = []
sem = asyncio.Semaphore(4)
for url in domains:
task = asyncio.Task(coro(url, sem))
tasks.append(task)
await asyncio.gather(*tasks)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
You can avoid using tasks. Just apply gather to the coroutine directly.
In case you are confused about the difference, this SO QA might help you (especially the second answer).
You can have each coroutine return its result, without resorting to global variables:
async def coro(url):
return await resolv.query(url)
async def main():
domains = ...
ops = [coro(url) for url in domains]
rets = await asyncio.gather(*ops)
print(rets)
Please see the official docs to learn more about how to use gather or wait or even more options
Note: if you are using the latest python versions, you can also simplify the loop running with just
asyncio.run(main())
Note 2: I have removed the semaphore from my code, as it's unclear why you need it and where.
all_parsed_whois = [] # make a global
async def coro(url, sem):
all_parsed_whois.append(await resolv.query(url))
If you want the data as soon as it is available you could task.add_done_callback()
python asyncio add_done_callback with async def
I want to be able to yield results from a set of tasks run by gather as they come in for further processing.
# Not real code, but an example
async for response in asyncio.gather(*[aiohttp.get(url) for url in ['https://www.google.com', 'https://www.amazon.com']]):
await process_response(response)
At present, I can use the gather method to run all get concurrently, but must wait until they're all complete to process them. I'm still new to Python async/await, so maybe there's some obvious way of doing this I'm missing.
# What I can do now
responses = await asyncio.gather(*[aiohttp.get(url) for url in ['https://www.google.com', 'https://www.amazon.com']])
await asyncio.gather(*[process_response(response) for response in responses])
Thanks!
gather as you already noted will wait until all coroutines are done, thus you need to find another way.
For example you can use function asyncio.as_completed that seems to do exactly what you want.
import asyncio
async def echo(t):
await asyncio.sleep(t)
return t
async def main():
coros = [
echo(3),
echo(2),
echo(1),
]
for first_completed in asyncio.as_completed(coros):
res = await first_completed
print(f'Done {res}')
asyncio.run(main())
Result:
Done 1
Done 2
Done 3
[Finished in 3 sec]
I'm trying to consume multiple queues concurrently using python, asyncio and asynqp.
I don't understand why my asyncio.sleep() function call does not have any effect. The code doesn't pause there. To be fair, I actually don't understand in which context the callback is executed, and whether I can yield control bavck to the event loop at all (so that the asyncio.sleep() call would make sense).
What If I had to use a aiohttp.ClientSession.get() function call in my process_msg callback function? I'm not able to do it since it's not a coroutine. There has to be a way which is beyond my current understanding of asyncio.
#!/usr/bin/env python3
import asyncio
import asynqp
USERS = {'betty', 'bob', 'luis', 'tony'}
def process_msg(msg):
asyncio.sleep(10)
print('>> {}'.format(msg.body))
msg.ack()
async def connect():
connection = await asynqp.connect(host='dev_queue', virtual_host='asynqp_test')
channel = await connection.open_channel()
exchange = await channel.declare_exchange('inboxes', 'direct')
# we have 10 users. Set up a queue for each of them
# use different channels to avoid any interference
# during message consumption, just in case.
for username in USERS:
user_channel = await connection.open_channel()
queue = await user_channel.declare_queue('Inbox_{}'.format(username))
await queue.bind(exchange, routing_key=username)
await queue.consume(process_msg)
# deliver 10 messages to each user
for username in USERS:
for msg_idx in range(10):
msg = asynqp.Message('Msg #{} for {}'.format(msg_idx, username))
exchange.publish(msg, routing_key=username)
loop = asyncio.get_event_loop()
loop.run_until_complete(connect())
loop.run_forever()
I don't understand why my asyncio.sleep() function call does not have
any effect.
Because asyncio.sleep() returns a future object that has to be used in combination with an event loop (or async/await semantics).
You can't use await in simple def declaration because the callback is called outside of async/await context which is attached to some event loop under the hood. In other words mixing callback style with async/await style is quite tricky.
The simple solution though is to schedule the work back to the event loop:
async def process_msg(msg):
await asyncio.sleep(10)
print('>> {}'.format(msg.body))
msg.ack()
def _process_msg(msg):
loop = asyncio.get_event_loop()
loop.create_task(process_msg(msg))
# or if loop is always the same one single line is enough
# asyncio.ensure_future(process_msg(msg))
# some code
await queue.consume(_process_msg)
Note that there is no recursion in _process_msg function, i.e. the body of process_msg is not executed while in _process_msg. The inner process_msg function will be called once the control goes back to the event loop.
This can be generalized with the following code:
def async_to_callback(coro):
def callback(*args, **kwargs):
asyncio.ensure_future(coro(*args, **kwargs))
return callback
async def process_msg(msg):
# the body
# some code
await queue.consume(async_to_callback(process_msg))
See Drizzt1991's response on github for a solution.