We Keep Coding

How to make a POST request using multi/form-data using Python?

I am very new with API things.
I have to make a POST request to API with the following "format"
content-type: multipart/form-data
Content-Disposition: form-data; name=""; filename=""
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Form data:
file = file.xlsx
How can I perform the API request using Python?
Using requests library, can I perform it:
requests.post(
'api_url',
headers = {'Content-Type':'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'},
data = {"filename.xlsx": open(filepath, "rb")}
)
Thanks

I prefer pool manager as this can manage timeout, retry, etc. easily:
import urllib3
from urllib3.util import Retry, Timeout
http_client = urllib3.PoolManager(retries=Retry(connect=5, read=2, redirect=5),
timeout=Timeout(connect=5.0, read=10.0),
num_pools=2)
data = {'asd': 'asd'}
request = http_client.request('POST', "http://localhost:8081", fields=data, encode_multipart=True)
This will give you:
>nc -l 127.0.0.1 8081
POST / HTTP/1.1
Host: localhost:8081
Accept-Encoding: identity
Content-Length: 125
Content-Type: multipart/form-data; boundary=6ce0c07687204c761cc1e5a6d6f6046e
User-Agent: python-urllib3/1.26.4
--6ce0c07687204c761cc1e5a6d6f6046e
Content-Disposition: form-data; name="asd"
asd
--6ce0c07687204c761cc1e5a6d6f6046e--

how to post form request on python

I am trying to fill a form like that and submit it automaticly. To do that, I sniffed the packets while logging in.
POST /?pg=ogrgiris HTTP/1.1
Host: xxx.xxx.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Origin: http://xxx.xxx.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15
Referer: http://xxx.xxx.com/?pg=ogrgiris
Upgrade-Insecure-Requests: 1
DNT: 1
Content-Length: 60
Connection: close
seviye=700&ilkodu=34&kurumkodu=317381&ogrencino=40&isim=ahm
I repeated that packet by burp suite and saw works porperly. the response was the html of the member page.
Now I tried to do that on python. The code is below:
import requests
url = 'http://xxx.xxx.com/?pg=ogrgiris'
headers = {'Host':'xxx.xxx.com',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate',
'Content-Type':'application/x-www-form-urlencoded',
'Referer':'http://xxx.xxx.com/?pg=ogrgiris',
'Content-Lenght':'60','Connection':'close'}
credentials = {'seviye': '700','ilkodu': '34','kurumkodu': '317381','ogrecino': '40','isim': 'ahm'}
r = requests.post(url,headers=headers, data=credentials)
print(r.content)
the problem is, that code prints the html of the login page even I send all of the credentials enough to log in. How can I get the member page? thanks.

If the POST request displays a page with the content you want, then the problem is only that you are sending data as JSON, not in "form" data format (application/x-www-form-urlencoded).
If a session is created at the request base and you have to make another request for the requested data, then you have to deal with cookies.
Problem with data format:
r = requests.post(url, headers=headers, data=credentials)
Kwarg json = creates a request body as follows:
{"ogrecino": "40", "ilkodu": "34", "isim": "ahm", "kurumkodu": "317381", "seviye": "700"}
While data= creates a request body like this:
seviye=700&ilkodu=34&kurumkodu=317381&ogrencino=40&isim=ahm
You can try https://httpbin.org:
from requests import post
msg = {"a": 1, "b": True}
print(post("https://httpbin.org/post", data=msg).json()) # Data as Form data, look at key `form`, it's object in JSON because it's Form data format
print(post("https://httpbin.org/post", json=msg).json()) # Data as json, look at key `data`, it's string

If your goal is to replicate the sample request, you are missing a lot of the headers; this in particular is very important Content-Type: application/x-www-form-urlencoded because it will tell your HTTP client how to format/encode the payload.
Check the documentation for requests so see how these form posts can work.

Is this HTTP request valid?

I've made an python server with swagger-codegen. I have one endpoint that receives an file with mutlipart/form-data
And also created an client with go-swagger for testing.
created an file to upload: $ echo "123file content321" > data
and used the client to upload the file to the server. The resulting HTTP request looks like this:
POST /api/order/1/attachment HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Accept: application/json
Content-Type: multipart/form-data; boundary=5f3f0ad86e6345b77c869cbe0a5e608f038354cf9ceab74ec2533d7555c0
Accept-Encoding: gzip
ff
--5f3f0ad86e6345b77c869cbe0a5e608f038354cf9ceab74ec2533d7555c0
Content-Disposition: form-data; name="file"; filename="data"
Content-Type: application/octet-stream
123file content321
--5f3f0ad86e6345b77c869cbe0a5e608f038354cf9ceab74ec2533d7555c0--
but the server doesn't accept it and responds:
HTTP/1.0 400 BAD REQUEST
Connection: close
Content-Length: 120
Content-Type: application/problem+json
Date: Fri, 19 May 2017 15:15:44 GMT
Server: Werkzeug/0.12.1 Python/3.6.1
{
"type": "about:blank",
"title": "Bad Request",
"detail": "Missing formdata parameter 'file'",
"status": 400
}
So the request isn't parsed properly. But when I use the swagger-ui, the file is uploaded correctly. Is there problem with the client's request, or the server has a problem?
EDIT: I think that there is missing Content-Lenght or the ff at the beginning of the BODY might not be there
EDIT2: the swagger-ui request:
POST /api/order/1/attachment HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 211
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarypzmNwrDR7zzpZ7SJ
Accept: application/json
X-Requested-With: XMLHttpRequest
DNT: 1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
------WebKitFormBoundarypzmNwrDR7zzpZ7SJ
Content-Disposition: form-data; name="file"; filename="data"
Content-Type: application/octet-stream
123file content321
------WebKitFormBoundarypzmNwrDR7zzpZ7SJ--

The first request you send is a HTTP/1.1 request using chunked transfer encoding. This means the body is consisting of multiple chunks where each chunk is prefixed by the size in hex followed by \r\n followed by the data and again \r\n. I'm not sure if the ff at the beginning of the body you show really specifies the size of the following data (i.e. 255 bytes). But, the last chunk with a size of 0 is missing, so this request is incomplete. But maybe you just omitted the missing part from this question only.
Apart from that the server is responding with version HTTP/1.0. Chunked transfer encoding is only defined for HTTP/1.1 which means that this request will not be understood by a HTTP/1.0 server. And not even all HTTP/1.1 server will understand chunked transfer encoding in the request even if they should.
The second request you show (created by Chrome) does not use chunked transfer encoding but specifies instead the length of the header using Content-length in the HTTP header. That's the way you should go since this works with all web servers, including HTTP/1.0 servers.

Based on the two requests you have posted I would attempt to set the Content-Length on your go request first and test that. I've run into issues before with the ArangoDB HTTP API not accepting requests without a correct content length value.
If the succeeds then yay.
Otherwise, that ff in your request is the next thing I'd look at getting rid of. But I'd focus on the Content-Length header first.

Posting image using requests on python

I'm trying to upload an image using requests on python.
This is what I send using browser
POST /upload-photo/{res1}/{res2}/{res3}/ HTTP/1.1
Host: tgt.tgdot.com
Connection: keep-alive
Content-Length: 280487
Authorization: Basic {value}=
Accept: */*
Origin: http://tgt.tgdot.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA8sGeB48ZZCvG127
Referer: http://tgt.tgdot.com/{res1}/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,es;q=0.6
Cookie: fttoken={cookie_value}
This is my code
with open(os.getcwd()+"/images/thee1.JPG", "rb") as image_file:
encoded_image = base64.b64encode(image_file.read())
headers = {"Content-Type":"multipart/form-data", "Authorization":"Basic " + authvalue}
cookie = {cookiename: token.value}
r = requests.post(url, headers =headers, cookies = cookie, params=encoded_image)
print r.request.headers
print r.status_code
print r.text
I keep getting 414 Request-URI Too Large
I'm not sure what's missing here. I would really appreciate help

You are encoding the whole image into the request parameters, effectively extending the URL by the length of the image.
If you already encoded the image data, use the data parameter:
r = requests.post(url, headers=headers, cookies=cookie, data=encoded_image)
Note that requests can encode multipart/form-data POST bodies directly, there is no need for you to encode it yourself. Use the files parameter in that case, passing in a dictionary or sequence of tuples. See the POST Multiple Multipart-Encoded Files section of the documentation.
The library can also handle a username and password pair to handle the Authorization header; simply pass in a (username, password) tuple for the auth keyword argument.
Encoding an image to Base64 is not sufficient however. Your content-type header and your POST payload are not matching. You'd instead post the file with a field name:
with open(os.getcwd()+"/images/thee1.JPG", "rb") as image_file:
files = {'field_name': image_file}
cookie = {cookiename: token.value}
r = requests.post(url, cookies = cookie, files=files, auth=(username, password)

MultiValueDict is empty when image has loaded

I'm trying to simulate browser work. When I'm sending image from browser I get in request.FILES that
<MultiValueDict: {u'file': [<InMemoryUploadedFile: 1.jpg (image/jpeg)>]}>
If I send image from back end POST request:
import httplib
conn = httplib.HTTPConnection(HOST, PORT)
headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": 'text/plain',
"Cookie": cookies, 'X_REQUESTED_WITH': 'XMLHttpRequest'}
conn.request("POST", "/upload/%s" % storyId, open("pictures/1.jpg"), headers)
response = conn.getresponse()
I get:
<MultiValueDict: {}>
What's wrong?

When you send a file from broswer, the Content Type header is multipart/form-data; not application/x-www-form-urlencoded also there are others changes in the request when you send a file. Take a look, here is how a request headers looks like when you send a file from a browser.
POST /upload?upload_progress_id=12344 HTTP/1.1
Host: localhost:3000
Content-Length: 1325
Origin: http://localhost:3000
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryePkpFF7tjBAqx29L
<other headers>
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="MAX_FILE_SIZE"
100000
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="uploadedfile"; filename="hello.o"
Content-Type: application/x-object
<file data>
------WebKitFormBoundaryePkpFF7tjBAqx29L--
You can read more here in this stackoverflow answer
Said that, you aren't "simulating" a browser file request correctly.