I'm trying to login into this IoT device using python-requests. But I noticed that response headers are empty. Further investigation and I think I found the reason: double headers on response.
Response captured with wireshark:
HTTP/1.0 200 OK
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
...
It seems that requests reads only the first "HTTP/1.0 200 OK" and nothing more as trying to run for example this:
import requests
r = requests.get('http://10.0.0.10/cgi-bin/index')
print(r.headers)
returns this:
{}
I need to receive "Set-cookie" header in order to manage the device. What should I do? Should I try to read raw http response and try to parse the "Set-cookie" header?
Related
I am getting http code 400 (bad request) when I am trying to post a file to server using python requests library.
Corresponding curl request which is successful:
curl -X POST -i https://de.api.labs.sophos.com/analysis/file/static/v1 \
-H 'Authorization: auth_string' \
-H 'Content-Type: multipart/form-data' \
-F "file=#filename"
API documentation: https://api.labs.sophos.com/doc/analysis/file/static.html
Can someone help me out what I might be doing wrong?
My code so far:
import requests
url = "https://de.api.labs.sophos.com/analysis/file/static/v1"
headers = {'content-type': 'multipart/form-data', 'Authorization': authorization}
with open(filepath, 'rb') as f:
files = {'file': f} # Even tried {'file': f.read()}
r = requests.post(url, files=files, headers=headers)
if r.status_code in [200, 202]:
return r.json()
else:
return r
TL;DR
Try to do it this way:
import requests
url = "https://de.api.labs.sophos.com/analysis/file/static/v1"
headers = {'Authorization': authorization} # no Content-Type here
r = requests.post(url, headers=headers, files={"file": open(filepath, "rb")})
print(r.status_code, r.text)
Why
You shouldn't set Content-Type header manually when posting files with requests.
There are 2 reasons why:
requests will set Content-Type to multipart/form-data implicitly before making an actual HTTP request (as it does for Content-Length for example)
When using Content-Type: multipart/form-data you should specify a boundary as well. If no boundary is set, server will not be able to read data from request body correctly. So boundary is a required part of Content-Type header, if you use multipart/form-data.
In your example you haven't set boundary for the request. The fact is that requests does not set it for you if you override Content-Type header (which you do). And then server is not able to read your file in request body. Therefore, it returns you 400 Bad Request.
You can check it by typing print(r.request.headers["Content-Type"]) after you've made your request. It will output this:
multipart/form-data
, but it must look like this instead:
multipart/form-data; boundary=6387a52fb4d1465310a2b63b2d1c6e70
On the other hand, curl adds boundary implicitly, so you everything is fine and you receive 200 OK.
You can check it as well:
curl -H 'Content-Type: multipart/form-data' -F "file=#123.txt" -v http://httpbin.org/post
Which outputs:
* Connected to httpbin.org (34.230.136.58) port 80 (#0)
> POST /post HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.47.0
> Content-Type: multipart/form-data; boundary=------------------------d257f5f4377a3997
...
I am using an API, which receives a pdf file and does some analysis, but I am receiving Response 500 always
Have initially tested using Postman and the request goes through, receiving response 200 with the corresponding JSON information. The SSL security should be turned off.
However, when I try to do request via Python, I always get Response 500
Python code written by me:
import requests
url = "https://{{BASE_URL}}/api/v1/documents"
fin = open('/home/train/aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf', 'rb')
files = {'file': fin}
r = requests.post(url, files=files, verify=False)
print (r)
#r.text is empty
Python code, produced by the Postman:
import requests
url = "https://{{BASE_URL}}/api/v1/documents"
payload = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"file\"; filename=\"aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf\"\r\nContent-Type: application/pdf\r\n\r\n\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"
headers = {
'content-type': "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW",
'Content-Type': "application/x-www-form-urlencoded",
'cache-control': "no-cache",
'Postman-Token': "65f888e2-c1e6-4108-ad76-f698aaf2b542"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
Have masked the API link as {{BASE_URL}} due to the confidentiality
Response by Postman:
{
"id": "5e69058e2690d5b0e519cf4006dfdbfeeb5261b935094a2173b2e79a58e80ab5",
"name": "aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf",
"fileIds": {
"original": "5e69058e2690d5b0e519cf4006dfdbfeeb5261b935094a2173b2e79a58e80ab5.pdf"
},
"creationDate": "2019-06-20T09:41:59.5930472+00:00"
}
Response by Python:
Response<500>
UPDATE:
Tried the GET request - works fine, as I receive the JSON response from it. I guess the problem is in posting pdf file. Is there any other options on how to post a file to an API?
Postman Response RAW:
POST /api/v1/documents
Content-Type: multipart/form-data; boundary=--------------------------375732980407830821611925
cache-control: no-cache
Postman-Token: 3e63d5a1-12cf-4f6b-8f16-3d41534549b9
User-Agent: PostmanRuntime/7.6.0
Accept: */*
Host: {{BASE_URL}}
cookie: c2b8faabe4d7f930c0f28c73aa7cafa9=736a1712f7a3dab03dd48a80403dd4ea
accept-encoding: gzip, deflate
content-length: 3123756
file=[object Object]
HTTP/1.1 200
status: 200
Date: Thu, 20 Jun 2019 10:59:55 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Location: /api/v1/files/95463e88527ecdc94393fde685ab1d05fa0ee0b924942f445b14b75e983c927e
api-supported-versions: 1.0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Referrer-Policy: strict-origin
{"id":"95463e88527ecdc94393fde685ab1d05fa0ee0b924942f445b14b75e983c927e","name":"aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf","fileIds":{"original":"95463e88527ecdc94393fde685ab1d05fa0ee0b924942f445b14b75e983c927e.pdf"},"creationDate":"2019-06-20T10:59:55.7038573+00:00"}
CORRECT REQUEST
So, eventually - the correct code is the following:
import requests
files = {
'file': open('/home/train/aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf', 'rb'),
}
response = requests.post('{{BASE_URL}}/api/v1/documents', files=files, verify=False)
print (response.text)
A 500 error indicates an internal server error, not an error with your script.
If you're receiving a 500 error (as opposed to a 400 error, which indicates a bad request), then theoretically your script is fine and it's the server-side code that needs to be adjusted.
In practice, it could still be due a bad request though.
If you're the one running the API, then you can check the error logs and debug the code line-by-line to figure out why the server is throwing an error.
In this case though, it sounds like it's a third-party API, correct? If so, I recommend looking through their documentation to find a working example or contacting them if you think it's an issue on their end (which is unlikely but possible).
I'm getting the following output
301 Moved Permanently --- when using http.client
200 --- when using requests
URL handling "http://i.imgur.com/fyxDric.jpg" passed as arg through command
What I expect is give me 200 status ok response.
This is the body
if scheme == 'http':
print('Ruuning in the http')
conn = http.client.HTTPConnection("www.i.imgur.com")
conn.request("GET", urlparse(url).path)
conn_resp = conn.getresponse()
body = conn_resp.read()
print(conn_resp.status, conn_resp.reason, body)
When using the requests
headers = {'User-Agent': 'Mozilla/5.0 Chrome/54.0.2840.71 Safari/537.36'}
response = requests.get(url, allow_redirects=False)
print(response.status_code)
You are trying to hit imgur over http, but imgur redirects all its request to process over https.
Due to this redirect the issue is occurring.
http module doesnt inherently handle the redirects you need to handle the redirects, where as requests module handles these redirects by itself.
The documentation on the http module includes in its first sentence "It is normally not used directly." Unlike requests it doesn't action the 301 response and follow the redirection in the headers. It instead returns the 301, which you would have to process yourself.
I am struggling from 2 days with a post request to be made only using urllib & urllib2. I have limitations in using curl or requests library, as the machine I would need to deploy my code doesn't support any of these.
The post call would be accompanied with a Header and json Body. I am able to make any get call, but POST with Data & Header throws 400 bad requests. Tried and applied all the options available in google/stackoverflow, but nothing solved!
Below is the sample code:--
import urllib
import urllib2
url = 'https://1.2.3.4/rest/v1/path'
headers = {'X-Auth-Token': '123456789sksksksk111',
'Content-Type': 'application/json'}
body = {'Action': 'myaction',
'PressType': 'Format1', 'Target': '/abc/def'}
data = urllib.urlencode(body)
request = urllib2.Request(url, data, headers)
response = urllib2.urlopen(request, data)
And on setting debug handler, below is the format of the request that can be traced:--
send: 'POST /rest/v1/path HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 52\r\nHost: 1.2.3.4\r\nUser-Agent: Python-urllib/2.7\r\nConnection: close\r\nX-Auth-Token: 123456789sksksksk111\r\nContent-Type: application/json\r\n\r\nAction=myaction&PressType=Format1&Target=%2Fabc%2Fdef'
reply: 'HTTP/1.1 400 Bad Request\r\n'
Please note, the same post request works perfectly fine with any REST client and with Requests library. In the debug handler output, if we see, the json structure is Content-Type: application/json\r\n\r\nAction=myaction&PressType=Format1&Target=%2Fabc%2Fdef, can that be a problem!
You can dump the json instead of encoding it. I was facing the same and got solved with it!
Remove data = urllib.urlencode(body) and use urllib2.urlopen(req, json.dumps(data))
That should solve.
I need to log into a website using python but the login page requires a sessionID cookie in the request header. Using Google developer tools along with a webclient(hurl.it), I was able to determine the required format of the request header that is acceptable by the webserver:
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 85
Content-Type: application/x-www-form-urlencoded
Cookie: www_amsterdam-dance-event_nl_session=l9Abno8a1UyHPof%2BOyVqk8BxHjesGMi78z6Ot0ZXCCbI%2BxVKqjm30ALTfW%2FR7yKcDaqfEtFOyysTrjIeU8lU5ylv1TOlW6GLHY8jDfKKWSULKsUUJiTh92DbvkuYBuE6zt%2FeLs44lDna6Nz3uMCOaSARN7gCpoSz0TOcFaes8Hk9q6FikP1F9e%2B%2FsMwfUP0RTA0Rc5gJFyJPxHXNCdn%2BT49mhHYnzoIWVlxGHhlaEkZX1PPsYx1xq0BCgpb0WnPViuiZiBnQY2nz%2BBO4Uur0WPNfpSSWZg5Qxz79nYeChlRe16JhYjVOdaiUhnfEvp1jM7h%2BCdR6cUeatd7HGbftRCjINDrVuPeyB5ltVihStmzKEjOmWetI0xNuaNswsPIKKuo%2BV6JFNfdLcA6h3iy1K8o%2FA49tKGMP2rmGe4e5Jec%3Df395212364d1ffc80cf95ebf5abf3b40f9dc6441;
User-Agent: runscope/0.1
email=******%40beatswitch.com&login_token=545a46230b291&password=*****&submission=
I have produced the following request using Python requests module:
POST /my-ade/login/ HTTP/1.1
Host: www.amsterdam-dance-event.nl
Content-Length: 85
Accept-Encoding: gzip,deflate
Accept: */*
User-Agent: runscope/0.1
Connection: keep-alive
Cookie: www_amsterdam-dance-event_nl_session=l9Abno8a1UyHPof%2BOyVqk8BxHjesGMi78z6Ot0ZXCCbI%2BxVKqjm30ALTfW%2FR7yKcDaqfEtFOyysTrjIeU8lU5ylv1TOlW6GLHY8jDfKKWSULKsUUJiTh92DbvkuYBuE6zt%2FeLs44lDna6Nz3uMCOaSARN7gCpoSz0TOcFaes8Hk9q6FikP1F9e%2B%2FsMwfUP0RTA0Rc5gJFyJPxHXNCdn%2BT49mhHYnzoIWVlxGHhlaEkZX1PPsYx1xq0BCgpb0WnPViuiZiBnQY2nz%2BBO4Uur0WPNfpSSWZg5Qxz79nYeChlRe16JhYjVOdaiUhnfEvp1jM7h%2BCdR6cUeatd7HGbftRCjINDrVuPeyB5ltVihStmzKEjOmWetI0xNuaNswsPIKKuo%2BV6JFNfdLcA6h3iy1K8o%2FA49tKGMP2rmGe4e5Jec%3Df395212364d1ffc80cf95ebf5abf3b40f9dc6441;
Content-Type: application/x-www-form-urlencoded
login_token=545a46230b291&password=*****&email=******%40beatswitch.com&submission='
When I load the former request header with hurl.it, everything works perfectly and the webserver lets me log in but trying the almost-same request with the same parameters fails in python. While using python's request, the webserver presents an error page. Any help would be highly appreciated. I need a solution desperately.
EDIT:
Here is the code:
#Open the login page to get sessionID and login_token
loginURL = "https://www.amsterdam-dance-event.nl/my-ade/login/"
loginReq = session.get(loginURL)
loginSoup = BeautifulSoup(loginReq.text)
loginToken = loginSoup.find('input',attrs={'name':'login_token'})['value']
sessionID= loginReq.cookies['www_amsterdam-dance-event_nl_session']
cookie = 'www_amsterdam-dance-event_nl_session='+sessionID
#Construct the header and post it to the webserver
headers = {'Content-Length':'85','Accept':'*/*','User-Agent':' runscope/0.1','Content-Type':'application/x-www-form-urlencoded','Accept-Encoding':'gzip,deflate','Cookie':cookie}
payload = {'email':'*******#beatswitch.com','password':'********','login_token':loginToken,'submission':''}
loggedinReq = session.post(loginURL,headers=headers,data=payload)
I found the solution, thanks to Md. Mohsin. I was trying to handle the request headers and cookies manually while the requests module can handle them by itself. So I REMOVED the following line from the code and let requests take total control, and everything worked as intended:
headers = {'Content-Length':'85','Accept':'*/*','User-Agent':' runscope/0.1','Content-Type':'application/x-www-form-urlencoded','Accept-Encoding':'gzip,deflate','Cookie':cookie}