We Keep Coding

How to post Multipart Form Data through python aiohttp ClientSession

I am trying to asynchronously send some Multipart-Encoded Form Data as a post request, mainly a file and two other fields.
Before trying to use asyncio I was doing the process synchronously with requests-toolbelt MultipartEncoder (https://github.com/requests/toolbelt) which worked great for normal requests, but did not work when using aiohttp for async. aiohttp provides 2 multipart classes, a FormData() class and a MultipartWriter() class, neither of which have given me much success.
After some testing, it seems like the difference is that when I use the toolbelt MultipartEncoder() the request sends the data in the form section of the post request as it should. However, when using aiohttp the request is put into the body section of the request. Not sure why they are acting differently
def multipartencode() -> ClientResponse():
# Using MultipartEncoder
m = MultipartEncoder(
fields={'type': type_str,
'metadata': json.dumps(metadata),
'file': (filename, file, 'application/json')}
)
# Using FormData
data = FormData()
data.add_field('file', file, filename=filename,
content_type='multipart/form-data')
data.add_field('type', type_str, content_type='multipart/form-data')
data.add_field('metadata', json.dumps(metadata),
content_type='multipart/form-data')
# Using MultipartWriter
with MultipartWriter('multipart/form-data') as mpwriter:
part = mpwriter.append(
file, {'CONTENT-TYPE': 'multipart/form-data'})
part.set_content_disposition('form-data')
part = mpwriter.append_form([('type', type_str)])
part.set_content_disposition('form-data')
part = mpwriter.append_form([('metadata', json.dumps(metadata))])
part.set_content_disposition('form-data')
# send request with ClientSession()
resp = await session.post(url=url, data=data, headers=headers)
return resp
How can I properly format/build the multipart-encoded request to get it to send using aiohttp?

How to make client request to external server avoiding cache using aiohttp

We are using aiohttp to make multiple requests to various website vendors to grab their latest data.
Some of the content providers serve the data from a cache. Is it possible to request the data from the server directly? We have tried to pass in the headers parameter with no luck.
async def fetch(url):
global response
headers = {'Cache-Control': 'no-cache'}
async with ClientSession() as session:
async with session.get(url, headers=headers, proxy="OUR-PROXY") as response:
return await response.read()
The goal is to get the last-modified date header, which is not provided from the cache request.

Try to add some additional variable with dynamic value to URL (e.g. timestamp).
This will prevent caching on the server side even if it ignores Cache-Control.
Example:
from: https://example.com/test
to: https://example.com/test?timestamp=20180724181234

Python Tornado - CORS PUT

I am facing a problem with Tornado. I have an API endpoint for PUT HTTP Method in Tornado. I also have a web application that sends the request to this API with jQuery and AJAX, but always I get a 405 response because the request is going as HTTP Method OPTIONS.
I understand the way it works and I did configured my Tornado Server to allow it. But even so I having this situation.
Can someone help me?
There is my server code:
class BaseHandler(RequestHandler):
def __init__(self, *args, **kwargs):
super(BaseHandler, self).__init__(*args, **kwargs)
self.set_header('Cache-Control', 'no-store, no-cache, must- revalidate, max-age=0')
self.set_header("Access-Control-Allow-Origin", "*")
self.set_header("Access-Control-Allow-Headers", "Content-Type")
self.set_header('Access-Control-Allow-Methods', 'POST, GET, PUT, DELETE, OPTIONS')
Many thanks

You need to add an options handler that just sends the headers with no body:
def options(self):
# no body
self.set_status(204)
self.finish()
See Tornado server: enable CORS requests for a complete code snippet.
Or else just install the tornado-cors package:
pip install tornado-cors
That will add the necessary handlers for you, and ensure the right response headers get sent.

if you don't define put method return 405
class Handler(tornado.web.RequestHandler):
def put(self):
self.set_header('Cache-Control', 'no-store, no-cache, must-revalidate, max-age=0')
self.set_header("Access-Control-Allow-Origin", "*")
self.set_header("Access-Control-Allow-Headers", "Content-Type")
self.set_header('Access-Control-Allow-Methods', 'POST, GET, PUT, DELETE, OPTIONS')
[I 170205 04:56:35 web:1971] 200 PUT

Python tornado AsyncHTTPClient 599

I try to send async PUT request with AsyncHTTPClient, but tornado return HTTP 599 error before request_timeout.

I fix this problem by write body="" to request:
req = tornado.httpclient.HTTPRequest(url,
method="PUT",
request_timeout=10,
headers=headers,
**body=""**)
Tornado can't send put request with empty body.

mitmproxy, how to get the response time/duration of a call?

I am able to get the response code, body, url and querystring form a request and response but I am not able to fetch time taken by a call.
We have
flow.response.code
flow.response.content
flow.response.url
does mitm has anything to get duration for the call.

The mitmproxy datastructures are documented at http://docs.mitmproxy.org/en/latest/dev/models.html.
For example:
def request(context, flow):
print(flow.request.timestamp_start)