I've made an python server with swagger-codegen. I have one endpoint that receives an file with mutlipart/form-data
And also created an client with go-swagger for testing.
created an file to upload: $ echo "123file content321" > data
and used the client to upload the file to the server. The resulting HTTP request looks like this:
POST /api/order/1/attachment HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Accept: application/json
Content-Type: multipart/form-data; boundary=5f3f0ad86e6345b77c869cbe0a5e608f038354cf9ceab74ec2533d7555c0
Accept-Encoding: gzip
ff
--5f3f0ad86e6345b77c869cbe0a5e608f038354cf9ceab74ec2533d7555c0
Content-Disposition: form-data; name="file"; filename="data"
Content-Type: application/octet-stream
123file content321
--5f3f0ad86e6345b77c869cbe0a5e608f038354cf9ceab74ec2533d7555c0--
but the server doesn't accept it and responds:
HTTP/1.0 400 BAD REQUEST
Connection: close
Content-Length: 120
Content-Type: application/problem+json
Date: Fri, 19 May 2017 15:15:44 GMT
Server: Werkzeug/0.12.1 Python/3.6.1
{
"type": "about:blank",
"title": "Bad Request",
"detail": "Missing formdata parameter 'file'",
"status": 400
}
So the request isn't parsed properly. But when I use the swagger-ui, the file is uploaded correctly. Is there problem with the client's request, or the server has a problem?
EDIT: I think that there is missing Content-Lenght or the ff at the beginning of the BODY might not be there
EDIT2: the swagger-ui request:
POST /api/order/1/attachment HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 211
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarypzmNwrDR7zzpZ7SJ
Accept: application/json
X-Requested-With: XMLHttpRequest
DNT: 1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
------WebKitFormBoundarypzmNwrDR7zzpZ7SJ
Content-Disposition: form-data; name="file"; filename="data"
Content-Type: application/octet-stream
123file content321
------WebKitFormBoundarypzmNwrDR7zzpZ7SJ--
The first request you send is a HTTP/1.1 request using chunked transfer encoding. This means the body is consisting of multiple chunks where each chunk is prefixed by the size in hex followed by \r\n followed by the data and again \r\n. I'm not sure if the ff at the beginning of the body you show really specifies the size of the following data (i.e. 255 bytes). But, the last chunk with a size of 0 is missing, so this request is incomplete. But maybe you just omitted the missing part from this question only.
Apart from that the server is responding with version HTTP/1.0. Chunked transfer encoding is only defined for HTTP/1.1 which means that this request will not be understood by a HTTP/1.0 server. And not even all HTTP/1.1 server will understand chunked transfer encoding in the request even if they should.
The second request you show (created by Chrome) does not use chunked transfer encoding but specifies instead the length of the header using Content-length in the HTTP header. That's the way you should go since this works with all web servers, including HTTP/1.0 servers.
Based on the two requests you have posted I would attempt to set the Content-Length on your go request first and test that. I've run into issues before with the ArangoDB HTTP API not accepting requests without a correct content length value.
If the succeeds then yay.
Otherwise, that ff in your request is the next thing I'd look at getting rid of. But I'd focus on the Content-Length header first.
Related
I want to handle multipart/form-data with python.
The request will look like this.
POST /upload HTTP/1.1
Host: localhost:8000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:8000/uplaod.html
Content-Type: multipart/form-data; boundary=---------------------------13077140074516507283069689500
Content-Length: 20970
Origin: http://localhost:8000
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
-----------------------------13077140074516507283069689500
Content-Disposition: form-data; name="mylogo"; filename="logo.png"
Content-Type: image/png
image_byte_code
-----------------------------13077140074516507283069689500--
This will be received in bytes from client browser.
I need to decode it to string.
But when i decode it the image bytes get destroyed.
Which then when i save it it will not be an image.
I have so far been able to decode it and receive text files.
Using the email library that defaults with python can really help. you just split the header from the data then use this https://julien.danjou.info/handling-multipart-form-data-python/. Make sure you get the boundary data from the headers.
Normally, when sending a HTTP request, the actually traffic is like below:
GET /abc?hello HTTP/1.1
Host: localhost:8080
User-Agent: python-requests/2.7
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
However, I would like to send URLs without the leading slash, for example:
GET abc?hello HTTP/1.1
GET ftp://abc?hello HTTP/1.1
I understand that's not compliant with RFCs, but just need to send such request for testing purpose in Python.
Have checked requests, urllib, urllib2, urllib3, haven't figured out how to do it.
Anyone can help me out?
I'd like to call the Webiopi REST API from my angular application in a browser running on the Raspberry. As Webiopi HTTP server doesn't allow CORS request, I have created a proxy with apache that sends the Header add "Access-Control-Allow-Origin" "*" header.
This is working fine, however the call to the REST API throws many errors mainly because the browser sends an OPTIONS request to the server in case of a CORS request to check wether it is allowed or not. But the webiopi http handler doesn't handle the OPTIONS verb at all.
So I started to write it into the code myself with zero python experience. In the file python/webiopi/protocols/http.py I have added at the end:
def do_OPTIONS(self):
self.send_response(200,"ok")
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Methods", "POST,GET,OPTIONS")
self.send_header("Access-Control-Allow-Headers", "Authorization")
self.send_header("Access-Control-Allow-Headers", "Content-Type")
self.end_headers()
Now it doesn't throw any error but doesn't give me the proper response to my GET request. It just stops after the OPTIONS. The request and response looks like this:
Request headers:
OPTIONS /GPIO/1/value HTTP/1.1
Host: localhost:8000
Connection: keep-alive
Access-Control-Request-Method: GET
Origin: http://192.168.1.108:51443
User-Agent: Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Raspbian Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36
Access-Control-Request-Headers: authorization
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.9,en-US;q=0.8,en;q=0.7
Response headers:
HTTP/1.1 200 OK
Date: Fri, 23 Nov 2018 22:06:28 GMT
Server: WebIOPi/0.7.1/Python3.5
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST,GET,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,POST,OPTIONS
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
General (from chrome network tab):
Request URL: http://localhost:8000/GPIO/1/value
Request Method: OPTIONS
Status Code: 200 OK
Remote Address: [::1]:8000
Referrer Policy: no-referrer-when-downgrade
Where is my GET request? Why do I see only the OPTIONS which by the way I'm not initiating at all?
The request from angular:
this.http.get<number>(this.route+'GPIO/'+gpio+'/value').subscribe(result => {
resolve(result);
})
I had to enable all headers to the http server:
def do_OPTIONS(self):
self.send_response(200,"ok")
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Methods", "*")
self.send_header("Access-Control-Allow-Headers", "*")
self.end_headers()
I am having some problem with a POST request that is driving me crazy. I am trying to upload an image using a POST using python requests by replicating the original request
First of all, this is the original request I can see with Firebug that the request header is:
Accept application/json, text/javascript, */*; q=0.01
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.7,it;q=0.3
Cache-Control no-cache
Connection keep-alive
Content-Length 26051
Content-Type multipart/form-data; boundary=--------------------------- 37693668010849786771875799013
Cookie some cookie
DNT 1
Host my.host.com
Pragma no-cache
Referer https://my.host.com/postad/1dc185ff9e814068be23ed674956a190
User-Agent Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0
X-Requested-With XMLHttpRequest
and the Body of the POST request:
-----------------------------5697279162042627623887894974
Content-Disposition: form-data; name="image"; filename="clip.jpg"
Content-Type: image/jpeg
ÿØÿà�JFIF������ÿÛ�C� .....
-----------------------------5697279162042627623887894974
Content-Disposition: form-data; name="upload"
-----------------------------5697279162042627623887894974--
This is the code I am using to upload the image:
files = {'image': ('clip.jpg', open('clip.jpg', 'rb'))}
post_image_url = "https://my.host.com/postad/563a533d9105448dbaf853f7ca0265fa/images"
r = session.post(post_image_url, files=files)
but I always get a 404 error, supposing that the request is not correct. Think I have to adjust the request to reflect the origin request examined with Firebug. I believe the request its not complete, I probably am missing the last part of the request:
-----------------------------5697279162042627623887894974
Content-Disposition: form-data; name="upload"
-----------------------------5697279162042627623887894974--
which I cannot replicate in any way.
You can add additional fields with the data argument:
data = {'upload': ''}
r = session.post(post_image_url, files=files, data=data)
This is just the value of the submit button, an empty string in your case.
I need to log into a website using python but the login page requires a sessionID cookie in the request header. Using Google developer tools along with a webclient(hurl.it), I was able to determine the required format of the request header that is acceptable by the webserver:
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 85
Content-Type: application/x-www-form-urlencoded
Cookie: www_amsterdam-dance-event_nl_session=l9Abno8a1UyHPof%2BOyVqk8BxHjesGMi78z6Ot0ZXCCbI%2BxVKqjm30ALTfW%2FR7yKcDaqfEtFOyysTrjIeU8lU5ylv1TOlW6GLHY8jDfKKWSULKsUUJiTh92DbvkuYBuE6zt%2FeLs44lDna6Nz3uMCOaSARN7gCpoSz0TOcFaes8Hk9q6FikP1F9e%2B%2FsMwfUP0RTA0Rc5gJFyJPxHXNCdn%2BT49mhHYnzoIWVlxGHhlaEkZX1PPsYx1xq0BCgpb0WnPViuiZiBnQY2nz%2BBO4Uur0WPNfpSSWZg5Qxz79nYeChlRe16JhYjVOdaiUhnfEvp1jM7h%2BCdR6cUeatd7HGbftRCjINDrVuPeyB5ltVihStmzKEjOmWetI0xNuaNswsPIKKuo%2BV6JFNfdLcA6h3iy1K8o%2FA49tKGMP2rmGe4e5Jec%3Df395212364d1ffc80cf95ebf5abf3b40f9dc6441;
User-Agent: runscope/0.1
email=******%40beatswitch.com&login_token=545a46230b291&password=*****&submission=
I have produced the following request using Python requests module:
POST /my-ade/login/ HTTP/1.1
Host: www.amsterdam-dance-event.nl
Content-Length: 85
Accept-Encoding: gzip,deflate
Accept: */*
User-Agent: runscope/0.1
Connection: keep-alive
Cookie: www_amsterdam-dance-event_nl_session=l9Abno8a1UyHPof%2BOyVqk8BxHjesGMi78z6Ot0ZXCCbI%2BxVKqjm30ALTfW%2FR7yKcDaqfEtFOyysTrjIeU8lU5ylv1TOlW6GLHY8jDfKKWSULKsUUJiTh92DbvkuYBuE6zt%2FeLs44lDna6Nz3uMCOaSARN7gCpoSz0TOcFaes8Hk9q6FikP1F9e%2B%2FsMwfUP0RTA0Rc5gJFyJPxHXNCdn%2BT49mhHYnzoIWVlxGHhlaEkZX1PPsYx1xq0BCgpb0WnPViuiZiBnQY2nz%2BBO4Uur0WPNfpSSWZg5Qxz79nYeChlRe16JhYjVOdaiUhnfEvp1jM7h%2BCdR6cUeatd7HGbftRCjINDrVuPeyB5ltVihStmzKEjOmWetI0xNuaNswsPIKKuo%2BV6JFNfdLcA6h3iy1K8o%2FA49tKGMP2rmGe4e5Jec%3Df395212364d1ffc80cf95ebf5abf3b40f9dc6441;
Content-Type: application/x-www-form-urlencoded
login_token=545a46230b291&password=*****&email=******%40beatswitch.com&submission='
When I load the former request header with hurl.it, everything works perfectly and the webserver lets me log in but trying the almost-same request with the same parameters fails in python. While using python's request, the webserver presents an error page. Any help would be highly appreciated. I need a solution desperately.
EDIT:
Here is the code:
#Open the login page to get sessionID and login_token
loginURL = "https://www.amsterdam-dance-event.nl/my-ade/login/"
loginReq = session.get(loginURL)
loginSoup = BeautifulSoup(loginReq.text)
loginToken = loginSoup.find('input',attrs={'name':'login_token'})['value']
sessionID= loginReq.cookies['www_amsterdam-dance-event_nl_session']
cookie = 'www_amsterdam-dance-event_nl_session='+sessionID
#Construct the header and post it to the webserver
headers = {'Content-Length':'85','Accept':'*/*','User-Agent':' runscope/0.1','Content-Type':'application/x-www-form-urlencoded','Accept-Encoding':'gzip,deflate','Cookie':cookie}
payload = {'email':'*******#beatswitch.com','password':'********','login_token':loginToken,'submission':''}
loggedinReq = session.post(loginURL,headers=headers,data=payload)
I found the solution, thanks to Md. Mohsin. I was trying to handle the request headers and cookies manually while the requests module can handle them by itself. So I REMOVED the following line from the code and let requests take total control, and everything worked as intended:
headers = {'Content-Length':'85','Accept':'*/*','User-Agent':' runscope/0.1','Content-Type':'application/x-www-form-urlencoded','Accept-Encoding':'gzip,deflate','Cookie':cookie}