I'm trying to simulate browser work. When I'm sending image from browser I get in request.FILES that
<MultiValueDict: {u'file': [<InMemoryUploadedFile: 1.jpg (image/jpeg)>]}>
If I send image from back end POST request:
import httplib
conn = httplib.HTTPConnection(HOST, PORT)
headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": 'text/plain',
"Cookie": cookies, 'X_REQUESTED_WITH': 'XMLHttpRequest'}
conn.request("POST", "/upload/%s" % storyId, open("pictures/1.jpg"), headers)
response = conn.getresponse()
I get:
<MultiValueDict: {}>
What's wrong?
When you send a file from broswer, the Content Type header is multipart/form-data; not application/x-www-form-urlencoded also there are others changes in the request when you send a file. Take a look, here is how a request headers looks like when you send a file from a browser.
POST /upload?upload_progress_id=12344 HTTP/1.1
Host: localhost:3000
Content-Length: 1325
Origin: http://localhost:3000
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryePkpFF7tjBAqx29L
<other headers>
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="MAX_FILE_SIZE"
100000
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="uploadedfile"; filename="hello.o"
Content-Type: application/x-object
<file data>
------WebKitFormBoundaryePkpFF7tjBAqx29L--
You can read more here in this stackoverflow answer
Said that, you aren't "simulating" a browser file request correctly.
Related
I am very new with API things.
I have to make a POST request to API with the following "format"
content-type: multipart/form-data
Content-Disposition: form-data; name=""; filename=""
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Form data:
file = file.xlsx
How can I perform the API request using Python?
Using requests library, can I perform it:
requests.post(
'api_url',
headers = {'Content-Type':'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'},
data = {"filename.xlsx": open(filepath, "rb")}
)
Thanks
I prefer pool manager as this can manage timeout, retry, etc. easily:
import urllib3
from urllib3.util import Retry, Timeout
http_client = urllib3.PoolManager(retries=Retry(connect=5, read=2, redirect=5),
timeout=Timeout(connect=5.0, read=10.0),
num_pools=2)
data = {'asd': 'asd'}
request = http_client.request('POST', "http://localhost:8081", fields=data, encode_multipart=True)
This will give you:
>nc -l 127.0.0.1 8081
POST / HTTP/1.1
Host: localhost:8081
Accept-Encoding: identity
Content-Length: 125
Content-Type: multipart/form-data; boundary=6ce0c07687204c761cc1e5a6d6f6046e
User-Agent: python-urllib3/1.26.4
--6ce0c07687204c761cc1e5a6d6f6046e
Content-Disposition: form-data; name="asd"
asd
--6ce0c07687204c761cc1e5a6d6f6046e--
I am trying to fill a form like that and submit it automaticly. To do that, I sniffed the packets while logging in.
POST /?pg=ogrgiris HTTP/1.1
Host: xxx.xxx.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Origin: http://xxx.xxx.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15
Referer: http://xxx.xxx.com/?pg=ogrgiris
Upgrade-Insecure-Requests: 1
DNT: 1
Content-Length: 60
Connection: close
seviye=700&ilkodu=34&kurumkodu=317381&ogrencino=40&isim=ahm
I repeated that packet by burp suite and saw works porperly. the response was the html of the member page.
Now I tried to do that on python. The code is below:
import requests
url = 'http://xxx.xxx.com/?pg=ogrgiris'
headers = {'Host':'xxx.xxx.com',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate',
'Content-Type':'application/x-www-form-urlencoded',
'Referer':'http://xxx.xxx.com/?pg=ogrgiris',
'Content-Lenght':'60','Connection':'close'}
credentials = {'seviye': '700','ilkodu': '34','kurumkodu': '317381','ogrecino': '40','isim': 'ahm'}
r = requests.post(url,headers=headers, data=credentials)
print(r.content)
the problem is, that code prints the html of the login page even I send all of the credentials enough to log in. How can I get the member page? thanks.
If the POST request displays a page with the content you want, then the problem is only that you are sending data as JSON, not in "form" data format (application/x-www-form-urlencoded).
If a session is created at the request base and you have to make another request for the requested data, then you have to deal with cookies.
Problem with data format:
r = requests.post(url, headers=headers, data=credentials)
Kwarg json = creates a request body as follows:
{"ogrecino": "40", "ilkodu": "34", "isim": "ahm", "kurumkodu": "317381", "seviye": "700"}
While data= creates a request body like this:
seviye=700&ilkodu=34&kurumkodu=317381&ogrencino=40&isim=ahm
You can try https://httpbin.org:
from requests import post
msg = {"a": 1, "b": True}
print(post("https://httpbin.org/post", data=msg).json()) # Data as Form data, look at key `form`, it's object in JSON because it's Form data format
print(post("https://httpbin.org/post", json=msg).json()) # Data as json, look at key `data`, it's string
If your goal is to replicate the sample request, you are missing a lot of the headers; this in particular is very important Content-Type: application/x-www-form-urlencoded because it will tell your HTTP client how to format/encode the payload.
Check the documentation for requests so see how these form posts can work.
I am using an API, which receives a pdf file and does some analysis, but I am receiving Response 500 always
Have initially tested using Postman and the request goes through, receiving response 200 with the corresponding JSON information. The SSL security should be turned off.
However, when I try to do request via Python, I always get Response 500
Python code written by me:
import requests
url = "https://{{BASE_URL}}/api/v1/documents"
fin = open('/home/train/aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf', 'rb')
files = {'file': fin}
r = requests.post(url, files=files, verify=False)
print (r)
#r.text is empty
Python code, produced by the Postman:
import requests
url = "https://{{BASE_URL}}/api/v1/documents"
payload = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"file\"; filename=\"aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf\"\r\nContent-Type: application/pdf\r\n\r\n\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"
headers = {
'content-type': "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW",
'Content-Type': "application/x-www-form-urlencoded",
'cache-control': "no-cache",
'Postman-Token': "65f888e2-c1e6-4108-ad76-f698aaf2b542"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
Have masked the API link as {{BASE_URL}} due to the confidentiality
Response by Postman:
{
"id": "5e69058e2690d5b0e519cf4006dfdbfeeb5261b935094a2173b2e79a58e80ab5",
"name": "aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf",
"fileIds": {
"original": "5e69058e2690d5b0e519cf4006dfdbfeeb5261b935094a2173b2e79a58e80ab5.pdf"
},
"creationDate": "2019-06-20T09:41:59.5930472+00:00"
}
Response by Python:
Response<500>
UPDATE:
Tried the GET request - works fine, as I receive the JSON response from it. I guess the problem is in posting pdf file. Is there any other options on how to post a file to an API?
Postman Response RAW:
POST /api/v1/documents
Content-Type: multipart/form-data; boundary=--------------------------375732980407830821611925
cache-control: no-cache
Postman-Token: 3e63d5a1-12cf-4f6b-8f16-3d41534549b9
User-Agent: PostmanRuntime/7.6.0
Accept: */*
Host: {{BASE_URL}}
cookie: c2b8faabe4d7f930c0f28c73aa7cafa9=736a1712f7a3dab03dd48a80403dd4ea
accept-encoding: gzip, deflate
content-length: 3123756
file=[object Object]
HTTP/1.1 200
status: 200
Date: Thu, 20 Jun 2019 10:59:55 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Location: /api/v1/files/95463e88527ecdc94393fde685ab1d05fa0ee0b924942f445b14b75e983c927e
api-supported-versions: 1.0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Referrer-Policy: strict-origin
{"id":"95463e88527ecdc94393fde685ab1d05fa0ee0b924942f445b14b75e983c927e","name":"aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf","fileIds":{"original":"95463e88527ecdc94393fde685ab1d05fa0ee0b924942f445b14b75e983c927e.pdf"},"creationDate":"2019-06-20T10:59:55.7038573+00:00"}
CORRECT REQUEST
So, eventually - the correct code is the following:
import requests
files = {
'file': open('/home/train/aab2wieuqcnvn3g6syadumik4bsg5.0062.pdf', 'rb'),
}
response = requests.post('{{BASE_URL}}/api/v1/documents', files=files, verify=False)
print (response.text)
A 500 error indicates an internal server error, not an error with your script.
If you're receiving a 500 error (as opposed to a 400 error, which indicates a bad request), then theoretically your script is fine and it's the server-side code that needs to be adjusted.
In practice, it could still be due a bad request though.
If you're the one running the API, then you can check the error logs and debug the code line-by-line to figure out why the server is throwing an error.
In this case though, it sounds like it's a third-party API, correct? If so, I recommend looking through their documentation to find a working example or contacting them if you think it's an issue on their end (which is unlikely but possible).
I'm trying to upload an image using requests on python.
This is what I send using browser
POST /upload-photo/{res1}/{res2}/{res3}/ HTTP/1.1
Host: tgt.tgdot.com
Connection: keep-alive
Content-Length: 280487
Authorization: Basic {value}=
Accept: */*
Origin: http://tgt.tgdot.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA8sGeB48ZZCvG127
Referer: http://tgt.tgdot.com/{res1}/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,es;q=0.6
Cookie: fttoken={cookie_value}
This is my code
with open(os.getcwd()+"/images/thee1.JPG", "rb") as image_file:
encoded_image = base64.b64encode(image_file.read())
headers = {"Content-Type":"multipart/form-data", "Authorization":"Basic " + authvalue}
cookie = {cookiename: token.value}
r = requests.post(url, headers =headers, cookies = cookie, params=encoded_image)
print r.request.headers
print r.status_code
print r.text
I keep getting 414 Request-URI Too Large
I'm not sure what's missing here. I would really appreciate help
You are encoding the whole image into the request parameters, effectively extending the URL by the length of the image.
If you already encoded the image data, use the data parameter:
r = requests.post(url, headers=headers, cookies=cookie, data=encoded_image)
Note that requests can encode multipart/form-data POST bodies directly, there is no need for you to encode it yourself. Use the files parameter in that case, passing in a dictionary or sequence of tuples. See the POST Multiple Multipart-Encoded Files section of the documentation.
The library can also handle a username and password pair to handle the Authorization header; simply pass in a (username, password) tuple for the auth keyword argument.
Encoding an image to Base64 is not sufficient however. Your content-type header and your POST payload are not matching. You'd instead post the file with a field name:
with open(os.getcwd()+"/images/thee1.JPG", "rb") as image_file:
files = {'field_name': image_file}
cookie = {cookiename: token.value}
r = requests.post(url, cookies = cookie, files=files, auth=(username, password)
I am having some problem with a POST request that is driving me crazy. I am trying to upload an image using a POST using python requests by replicating the original request
First of all, this is the original request I can see with Firebug that the request header is:
Accept application/json, text/javascript, */*; q=0.01
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.7,it;q=0.3
Cache-Control no-cache
Connection keep-alive
Content-Length 26051
Content-Type multipart/form-data; boundary=--------------------------- 37693668010849786771875799013
Cookie some cookie
DNT 1
Host my.host.com
Pragma no-cache
Referer https://my.host.com/postad/1dc185ff9e814068be23ed674956a190
User-Agent Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0
X-Requested-With XMLHttpRequest
and the Body of the POST request:
-----------------------------5697279162042627623887894974
Content-Disposition: form-data; name="image"; filename="clip.jpg"
Content-Type: image/jpeg
ÿØÿà�JFIF������ÿÛ�C� .....
-----------------------------5697279162042627623887894974
Content-Disposition: form-data; name="upload"
-----------------------------5697279162042627623887894974--
This is the code I am using to upload the image:
files = {'image': ('clip.jpg', open('clip.jpg', 'rb'))}
post_image_url = "https://my.host.com/postad/563a533d9105448dbaf853f7ca0265fa/images"
r = session.post(post_image_url, files=files)
but I always get a 404 error, supposing that the request is not correct. Think I have to adjust the request to reflect the origin request examined with Firebug. I believe the request its not complete, I probably am missing the last part of the request:
-----------------------------5697279162042627623887894974
Content-Disposition: form-data; name="upload"
-----------------------------5697279162042627623887894974--
which I cannot replicate in any way.
You can add additional fields with the data argument:
data = {'upload': ''}
r = session.post(post_image_url, files=files, data=data)
This is just the value of the submit button, an empty string in your case.