We Keep Coding

Chaining Requests using Python and Aiohttp

I am new to asynchronous programming in python, have been working on a script using aiohttp that fetches data from a get request and passes a specific variable from the response onto another post request. A sample of what I have tried is below:
async def fetch1(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp: # First hit to the url
data = resp.json() # Grab response
return await fetch2(data['uuid']) # Pass uuid to the second function for post request
async def fetch2(id):
url2 = "http://httpbin.org/post"
params = {'id': id}
async with aiohttp.ClientSession() as session:
async with session.post(url2,data=params) as resp:
return await resp.json()
async def main():
url = 'http://httpbin.org/uuid'
data = await fetch1(url)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
When I execute the script, I get the following error:
Traceback (most recent call last):
File ".\benchmark.py", line 27, in <module>
loop.run_until_complete(main())
File "C:\ProgramFiles\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2288.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File ".\benchmark.py", line 22, in main
data = await fetch1(url)
File ".\benchmark.py", line 10, in fetch1
return fetch2(data['uuid'])
TypeError: 'coroutine' object is not subscriptable
sys:1: RuntimeWarning: coroutine 'ClientResponse.json' was never awaited
I know that the coroutine is a generator, but how do I go ahead, any help will be appreciated.

The error says coroutine 'ClientResponse.json' was never awaited which means it must have an await before the json part. This is because you are using an async function.
async def fetch1(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp: # First hit to the url
data = await resp.json() # Grab response
return await fetch2(data['uuid']) # Pass uuid to the second function for post request
async def fetch2(id):
url2 = "http://httpbin.org/post"
params = {'id': id}
async with aiohttp.ClientSession() as session:
async with session.post(url2,data=params) as resp:
return await resp.json()
async def main():
url = 'http://httpbin.org/uuid'
data = await fetch1(url)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Async processing of function requests using asyncio

I am trying to achieve aiohttp async processing of requests that have been defined in my class as follows:
class Async():
async def get_service_1(self, zip_code, session):
url = SERVICE1_ENDPOINT.format(zip_code)
response = await session.request('GET', url)
return await response
async def get_service_2(self, zip_code, session):
url = SERVICE2_ENDPOINT.format(zip_code)
response = await session.request('GET', url)
return await response
async def gather(self, zip_code):
async with aiohttp.ClientSession() as session:
return await asyncio.gather(
self.get_service_1(zip_code, session),
self.get_service_2(zip_code, session)
)
def get_async_requests(self, zip_code):
asyncio.set_event_loop(asyncio.SelectorEventLoop())
loop = asyncio.get_event_loop()
results = loop.run_until_complete(self.gather(zip_code))
loop.close()
return results
When running to get the results from the get_async_requests function, i am getting the following error:
TypeError: object ClientResponse can't be used in 'await' expression
Where am i going wrong in the code? Thank you in advance

When you await something like session.response, the I/O starts, but aiohttp returns when it receives the headers; it doesn't want for the response to finish. (This would let you react to a status code without waiting for the entire body of the response.)
You need to await something that does that. If you're expecting a response that contains text, that would be response.text. If you're expecting JSON, that's response.json. This would look something like
response = await session.get(url)
return await response.text()

aiohttp how to save a persistent ClientSession in a class?

I'm writing a class that will do http requests using aiohttp. According to the docs I should not to create a ClientSession per request, so I want to reuse the same session.
code:
class TestApi:
def __init__(self):
self.session = aiohttp.ClientSession()
# async defs methods from here
When doing
TestApi()
I get the error: Unclosed client session.
What is the solution to persist the ClientSession object?

The expression TestApi() on a line by itself creates a TestApi object and immediately throws it away. aiohttp complaints that the session was never closed (either by leaving an async with block or with an explicit call to close()), but even without the warning it doesn't make sense not to assign the API object to a variable where it will be actually used.
To reuse the session, your code needs to have access to the session, or to an object that holds it:
async def fetch(url):
async with aiohttp.request('GET', url) as resp:
resp.raise_for_status()
return await resp.read()
async def main():
url1_data, url2_data = asyncio.gather(
fetch('http://url1'), fetch('http://url2'))
url3_data, url4_data = asyncio.gather(
fetch('http://url3'), fetch('http://url4'))
One option is to add a session parameter to fetch (and other functions) and consistently call it with a session created in main(). A better option is to create an API class and convert the global functions like fetch to methods:
class Http:
async def __aenter__(self):
self._session = aiohttp.ClientSession()
return self
async def __aexit__(self, *err):
await self._session.close()
self._session = None
async def fetch(self, url):
async with self._session.get(url) as resp:
resp.raise_for_status()
return await resp.read()
main() can still exist as a function, but it can consistently use the object that holds the session:
async def main():
async with Http() as http:
url1_data, url2_data = await asyncio.gather(
http.fetch('http://url1'), http.fetch('http://url2'))
url3_data, url4_data = await asyncio.gather(
http.fetch('http://url3'), http.fetch('http://url4'))
In the above code, the async with statement is used to ensure that the session is closed whenever the scope is left.

How to resolve an aiohttp json future?

import asyncio
import Response
import aiohttp
async def resolve_response_json(res):
new_res = Response()
async with res:
new_res.status = res.status
new_res.json = await res.json()
return new_res
class Client:
async def request(url):
async with aiohttp.ClientSession() as sess:
res = await sess.get(url=url)
return await resolve_response_json(res).json
client = Client()
loop = asyncio.get_event_loop()
value = loop.run_until_complete(client.request('https://example.com/api/v1/resource'))
Why does this piece of code give me:
> return await resolve_response_json(res).json
E AttributeError: 'coroutine' object has no attribute 'json'
I thought that the await keyword always returns an actual value. If it actually does, why is my code throwing this error?
Or am I just silly and probably forgot to put an await somewhere?

You are awaiting resolve_response_json(res).json, not resolve_response_json(res).
Change it to (await resolve_response_json(res)).json may work.

Python package - aiohttp has a warning message "Unclosed client session"

My code is as follows:
import asyncio
import aiohttp
urls = [
'http://www.163.com/',
'http://www.sina.com.cn/',
'https://www.hupu.com/',
'http://www.csdn.net/'
]
async def get_url_data(u):
"""
read url data
:param u:
:return:
"""
print('running ', u)
resp = await aiohttp.ClientSession().get(url=u)
headers = resp.headers
print(u, headers)
return headers
async def request_url(u):
"""
main func
:param u:
:return:
"""
res = await get_url_data(u)
return res
loop = asyncio.get_event_loop()
task_lists = asyncio.wait([request_url(u) for u in urls])
loop.run_until_complete(task_lists)
loop.close()
When i running my code, it's display a warning message:
Unclosed client session
Anybody can give me some solutions about that?
Thanks a lot

You should close the connection in the end.
You have 2 options:
You can close the connection manually:
import aiohttp
session = aiohttp.ClientSession()
# use the session here
session.close()
Or you can use it with a contex manager:
import aiohttp
import asyncio
async def fetch(client):
async with client.get('http://python.org') as resp:
assert resp.status == 200
return await resp.text()
async def main(loop):
async with aiohttp.ClientSession(loop=loop) as client:
html = await fetch(client)
print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
The client session supports the context manager protocol for self closing.

If you are not using context manager, the proper way to close it would also need an await. Many answers on the internet miss that part, and few people actually notice it, presumably because most people use the more convenient context manager. But the manual await session.close() is essential when/if you are closing a class-wide session inside the tearDownClass() when doing unittesting.
import aiohttp
session = aiohttp.ClientSession()
# use the session here
await session.close()

You should use ClientSession using async context manager for proper blocking/freeing resources:
async def get_url_data(u):
"""
read url data
:param u:
:return:
"""
print('running ', u)
async with aiohttp.ClientSession() as session:
resp = await session.get(url=u)
headers = resp.headers
print(u, headers)
return headers