We Keep Coding

Why playwright isn't modifying request body?

I need to modify the request body that is sent by the browser, i do it like this:
async def handle_uri_route(route: Route):
response = await route.fetch()
response_json = await response.json()
reg_notification_uri = response_json['confirmUri']
await page.route(reg_notification_uri, modify_notification_uri)
await route.continue_()
async def modify_notification_uri(route: Route):
post_data = route.request.post_data_json
post_data['notificationUi'] = 'https://httpbin.org/anything'
await route.continue_(post_data=urlencode(post_data))
pprint(route.request.post_data_json)
await page.route(re.compile(r'api\/common\/v1\/register_notification_uri\?'), handle_uri_route)
pprint displays the changed data, but if i open the devtools, then i see that the request hasn't really changed.
What am I doing wrong?

How can i send file with aiohttp?

this is my code:
payload = {'text': input_text,
'question_info': '',
'include_intonation': 1,
'stress_version': stress_version,
'include_fluency': 1,
'include_ielts_subscore': 1}
files = [
('user_audio_file', open(saved_file_path, 'rb'))
]
headers = {}
form = aiohttp.FormData()
for key, value in payload.items():
form.add_field(key, value)
form.add_field('user_audio_file', open(saved_file_path, 'rb'))
async with aiohttp.ClientSession() as session:
async with session.post(url,data=form) as response:
response_json = await response.json()
and I want to send file with aiohttp to URL but I got this exception
'Can not serialize value type: <class \'int\'> headers: {} value: 1'
I do that with requests library like this
response = request(
"POST", url, headers=headers, data=payload, files=files)
response_json = response.json()
but I decided to use aiohttp because it shoud be async
please help me for this decision
thanks

you need to serialize payload data using data= b'form'
e.g.
async with aiohttp.ClientSession() as session:
async with session.post(url,data=b'form') as response:
response_json = await response.json()
By default session uses python’s standard json module for serialization. But it is possible to use different serializer. ClientSession accepts json_serialize parameter. Then you dont need to explicitly serialize your payload.
import ujson
async with aiohttp.ClientSession(
json_serialize=ujson.dumps) as session:
await session.post(url,data=form) as response:
response_json = await response.json()
....
Warning: above code is not tested.
Update
I tried setting up a local http server and upload a json. I am getting past your error and able to upload data. Are your serializing form data using b'form'?
As per this GitHub issue discussion, we need asyncio to control async event loop and execute async/await through a function.
Here's relevant code.
async def uploadForm():
async with aiohttp.ClientSession() as session:
async with session.post(url,data=b'form') as response: #Converting form to binary payload using b'form'
response_json = await response.json(content_type='text/html')
print(response_json)
def main():
loop = asyncio.get_event_loop()
loop.run_until_complete(uploadForm())
loop.close()
if __name__ == '__main__':
main()
Hope this helps you.

how to schedule the execution of an async function in python and immediately return

I need to implement a proxy server in python that forwards client requests to some api if it doesn't have the data in it's cache. The requirement for when the data isn't present in the cache, is to not let the client wait at all but rather send back something like "you'll have your data soon" and in the meanwhile send a request to the api. It was my understanding I need to use async/await for this, but I could not make this work no matter what I tried. I am using asyncio and aiohttp libraries for this.
So let's say I have my function that sends a request to the api:
async def fetch(url, page_num):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
resp = await response.json()
cache[page_num] = (resp, datetime.now())
return resp
what I would like is the following behavior:
if not_in_cache(page_number):
fetch(url, page_number) #this needs to return immediately so the client won't wait!!!
return Response("we're working on it") #send back a response without data
So on the one hand I want the method to immediately return a response to the client, but in the background I want it to get the data and store it in the cache. How can you accomplish that with async/await?

Create a task. Instead of:
if not_in_cache(page_number):
await fetch(url, page_number)
return Response(...)
write:
if not_in_cache(page_number):
asyncio.create_task(fetch(url, page_number))
return Response(...)
Don't forget to read the asyncio docs: Coroutines and Tasks

Async processing of function requests using asyncio

I am trying to achieve aiohttp async processing of requests that have been defined in my class as follows:
class Async():
async def get_service_1(self, zip_code, session):
url = SERVICE1_ENDPOINT.format(zip_code)
response = await session.request('GET', url)
return await response
async def get_service_2(self, zip_code, session):
url = SERVICE2_ENDPOINT.format(zip_code)
response = await session.request('GET', url)
return await response
async def gather(self, zip_code):
async with aiohttp.ClientSession() as session:
return await asyncio.gather(
self.get_service_1(zip_code, session),
self.get_service_2(zip_code, session)
)
def get_async_requests(self, zip_code):
asyncio.set_event_loop(asyncio.SelectorEventLoop())
loop = asyncio.get_event_loop()
results = loop.run_until_complete(self.gather(zip_code))
loop.close()
return results
When running to get the results from the get_async_requests function, i am getting the following error:
TypeError: object ClientResponse can't be used in 'await' expression
Where am i going wrong in the code? Thank you in advance

When you await something like session.response, the I/O starts, but aiohttp returns when it receives the headers; it doesn't want for the response to finish. (This would let you react to a status code without waiting for the entire body of the response.)
You need to await something that does that. If you're expecting a response that contains text, that would be response.text. If you're expecting JSON, that's response.json. This would look something like
response = await session.get(url)
return await response.text()

Session reusing in aiohhttp

I try to reuse HTTP-session as aiohttp docs advice
Don’t create a session per request. Most likely you need a session per
application which performs all requests altogether.
But usual pattern which I use with requests lib doesn`t work:
def __init__(self):
self.session = aiohttp.ClientSession()
async def get_u(self, id):
async with self.session.get('url') as resp:
json_resp = await resp.json()
return json_resp.get('data', {})
Then I try to
await client.get_u(1)
I got error
RuntimeError: Timeout context manager should be used inside a task
Any workarounds with async_timeout didn't help.
Another way is working:
async def get_u(self, id):
async with aiohttp.ClientSession() as session:
with async_timeout.timeout(3):
async with session.get('url') as resp:
json_resp = await resp.json()
return json_resp.get('data', {})
But it seems like creating session per request.
So my question: how to properly reuse aiohttp-session?
UPD: minimal working example. Sanic application with following view
import aiohttp
from sanic.views import HTTPMethodView
class Client:
def __init__(self):
self.session = aiohttp.ClientSession()
self.url = 'https://jsonplaceholder.typicode.com/todos/1'
async def get(self):
async with self.session.get(self.url) as resp:
json_resp = await resp.json()
return json_resp
client = Client()
class ExView(HTTPMethodView):
async def get(self, request):
todo = await client.get()
print(todo)

I had the same error. The solution for me was initializing the client within an async function. EG:
class SearchClient(object):
def __init__(self, search_url: str, api_key: str):
self.search_url = search_url
self.api_key = api_key
self.session = None
async def _get(self, url, attempt=1):
if self.session is None:
self.session = aiohttp.ClientSession(raise_for_status=True)
headers = {
'Content-Type': 'application/json',
'api-key': self.api_key
}
logger.info("Running Search: {}".format(url))
try:
with timeout(60):
async with self.session.get(url, headers=headers) as response:
results = await response.json()
return results

For example you can create ClientSession on app start (using on_startup signal https://docs.aiohttp.org/en/stable/web_advanced.html#signals).
Store it to you app (aiohttp application has dict interface for such issues https://aiohttp.readthedocs.io/en/stable/faq.html#id4) and get access to your session through request.app['YOU_CLIENT_SESSION'] in request.