I am currently looking into pulsar for an asynchronous HTTP client.
The following example is in the docs:
from pulsar.apps import http
async with http.HttpClient() as session:
response1 = await session.get('https://github.com/timeline.json')
response2 = await session.get('https://api.github.com/emojis.json')
but when I try to execute it I get
async with http.HttpClient() as session:
^ SyntaxError: invalid syntax
It looks like the async keyword is not recognized. I am using Python 3.5.
Working example:
import asyncio
from pulsar.apps.http import HttpClient
async def my_fun():
async with HttpClient() as session:
response1 = await session.get('https://github.com/timeline.json')
response2 = await session.get('https://api.github.com/emojis.json')
print(response1)
print(response2)
loop = asyncio.get_event_loop()
loop.run_until_complete(my_fun())
you can only use async with inside a coroutines, so you have to do this
from pulsar.apps.http import HttpClient
import pulsar
async def my_fun():
async with HttpClient() as session:
response1 = await session.get('https://github.com/timeline.json')
response2 = await session.get('https://api.github.com/emojis.json')
return response1, response2
loop = pulsar.get_event_loop()
res1, res2 = loop.run_until_complete(my_fun())
print(res1)
print(res2)
internally pulsar use asyncio, so you don't have to import it explicitly to use it, use it through pulsar
as a side note, if you upgrade to python 3.6 you can use async list/set/etc comprehension
from pulsar.apps.http import HttpClient
import pulsar
async def my_fun():
async with HttpClient() as session:
urls=['https://github.com/timeline.json','https://api.github.com/emojis.json']
return [ await session.get(url) for url in urls]
loop = pulsar.get_event_loop()
res1, res2 = loop.run_until_complete(my_fun())
print(res1)
print(res2)
Related
async def get_quote():
async with aiohttp.ClientSession() as session:
async with session.get("https://zenquotes.io/api/random") as response:
json_data = json.loads(response.text)
quote = json_data[0]["q"]
return (quote)
#client.command(aliases=["wow"])
async def inspire(ctx):
quote = await get_quote()
await ctx.send(quote)
I was told to use aiohttp instead of requests module, so I did. Then when I used aiohttp, it said module 'aiohttp' has no attribute 'get'. So this is my final code right now. Please help.
response.text is an async function, not an attribute, so it must be called and awaited. Try:
async def get_quote():
async with aiohttp.ClientSession() as session:
async with session.get("https://zenquotes.io/api/random") as response:
json_data = json.loads(await response.text())
quote = json_data[0]["q"]
return quote
I used requests.
import requests
#bot.command()
async def quote(ctx):
response = requests.get('https://api.quotable.io/random')
r = response.json()
await ctx.send(r["content"])
This works fine for me.
please install the package aiohttp and import that one
pip3 install aiohttp
import aiohttp
Please refer the first example from this link docs.aiohttp.org
install httpx
pip3 install httpx
for stater test this one
import asyncio
import httpx
async def test():
async with httpx.AsyncClient() as client:
resp = await client.get('https://dog.ceo/api/breeds/image/random')
print(resp)
asyncio.run(test())
I need to know how much Mbytes per second there are in the request with is sent with asyncio in python. I tried with resp.read() or with resp.content.read() but it isn't working.
import aiohttp
import asyncio
async def get_requests(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
result = resp
return result
big_urls = ['http://google.com', 'http://yahoo.com']
loop = asyncio.get_event_loop()
coroutines = [get_requests(url) for url in big_urls]
results = loop.run_until_complete(asyncio.gather(*coroutines))
print(results)
I have the following code
import asyncio
import aiohttp
urls = [
'http://54.224.27.241',
'http://54.224.27.241',
'http://54.224.27.241',
'http://54.224.27.241',
'http://54.224.27.241',
]
async def query(urls):
out = []
with aiohttp.ClientSession() as session:
for url in urls:
try:
async with session.get(url, timeout=5) as resp:
text = await resp.text()
out.append(resp.status)
except:
print('timeout')
return out
loop = asyncio.get_event_loop()
out = loop.run_until_complete(query(urls))
loop.close()
print(str(out))
The code is much slower than the one that uses a threadpool and keep increasing if you increase the number of urls (lets say 20, 50 etc.)
I have a feeling that when the initial connection establishment is not done in an async way.
(Note that I am connecting here to an non-existing server to deliberately produce a connection timeout).
Can someone point out what is wrong here?
Warning: I don't promise this code works, as I can't install aiohttp atm, but looking at the example in the docs
async def fetch(session, url):
async with async_timeout.timeout(10):
async with session.get(url) as response:
return await response.text()
async def main():
async with aiohttp.ClientSession() as session:
html = await fetch(session, 'http://python.org')
print(html)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Notice how they're calling the aiohttp.ClientSession() with the async keyword. Additionally, I was getting some error in your line data = await async with session.get(url) as resp:
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
async def main():
out = []
async with aiohttp.ClientSession() as session:
for url in urls:
data = await fetch(session, url)
out.append(data)
return out
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
My code is as follows:
import asyncio
import aiohttp
urls = [
'http://www.163.com/',
'http://www.sina.com.cn/',
'https://www.hupu.com/',
'http://www.csdn.net/'
]
async def get_url_data(u):
"""
read url data
:param u:
:return:
"""
print('running ', u)
resp = await aiohttp.ClientSession().get(url=u)
headers = resp.headers
print(u, headers)
return headers
async def request_url(u):
"""
main func
:param u:
:return:
"""
res = await get_url_data(u)
return res
loop = asyncio.get_event_loop()
task_lists = asyncio.wait([request_url(u) for u in urls])
loop.run_until_complete(task_lists)
loop.close()
When i running my code, it's display a warning message:
Unclosed client session
Anybody can give me some solutions about that?
Thanks a lot
You should close the connection in the end.
You have 2 options:
You can close the connection manually:
import aiohttp
session = aiohttp.ClientSession()
# use the session here
session.close()
Or you can use it with a contex manager:
import aiohttp
import asyncio
async def fetch(client):
async with client.get('http://python.org') as resp:
assert resp.status == 200
return await resp.text()
async def main(loop):
async with aiohttp.ClientSession(loop=loop) as client:
html = await fetch(client)
print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
The client session supports the context manager protocol for self closing.
If you are not using context manager, the proper way to close it would also need an await. Many answers on the internet miss that part, and few people actually notice it, presumably because most people use the more convenient context manager. But the manual await session.close() is essential when/if you are closing a class-wide session inside the tearDownClass() when doing unittesting.
import aiohttp
session = aiohttp.ClientSession()
# use the session here
await session.close()
You should use ClientSession using async context manager for proper blocking/freeing resources:
async def get_url_data(u):
"""
read url data
:param u:
:return:
"""
print('running ', u)
async with aiohttp.ClientSession() as session:
resp = await session.get(url=u)
headers = resp.headers
print(u, headers)
return headers
I am trying to teach myself Python's async functionality. To do so I have built an async web scraper. I would like to limit the total number of connections I have open at once to be a good citizen on servers. I know that semaphore's are a good solution, and the asyncio library has a semaphore class built in. My issue is that Python complains when using yield from in an async function as you are combining yield and await syntax. Below is the exact syntax I am using...
import asyncio
import aiohttp
sema = asyncio.BoundedSemaphore(5)
async def get_page_text(url):
with (yield from sema):
try:
resp = await aiohttp.request('GET', url)
if resp.status == 200:
ret_val = await resp.text()
except:
raise ValueError
finally:
await resp.release()
return ret_val
Raising this Exception:
File "<ipython-input-3-9b9bdb963407>", line 14
with (yield from sema):
^
SyntaxError: 'yield from' inside async function
Some possible solution I can think of...
Just use the #asyncio.coroutine decorator
Use threading.Semaphore? This seems like it may cause other issues
Try this in the beta of Python 3.6 for this reason.
I am very new to Python's async functionality so I could be missing something obvious.
You can use the async with statement to get an asynchronous context manager:
#!/usr/local/bin/python3.5
import asyncio
from aiohttp import ClientSession
sema = asyncio.BoundedSemaphore(5)
async def hello(url):
async with ClientSession() as session:
async with sema, session.get(url) as response:
response = await response.read()
print(response)
loop = asyncio.get_event_loop()
loop.run_until_complete(hello("http://httpbin.org/headers"))
Example taken from here. The page is also a good primer for asyncio and aiohttp in general.
OK, so this is really silly but I just replaces yield from with await in the semaphore context manager and it is working perfectly.
sema = asyncio.BoundedSemaphore(5)
async def get_page_text(url):
with (await sema):
try:
resp = await aiohttp.request('GET', url)
if resp.status == 200:
ret_val = await resp.text()
except:
raise ValueError
finally:
await resp.release()
return ret_val
For the semaphore only:
sem = asyncio.Semaphore(10)
# ... later
async with sem:
# work with shared resource
which is equivalent to:
sem = asyncio.Semaphore(10)
# ... later
await sem.acquire()
try:
# work with shared resource
finally:
sem.release()
ref:
https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore