We Keep Coding

How to upload the file in POST requests (form-data)?

I use this GET request https://api.telegram.org/file/bot<token>/<file_path> which allow us to download the file from Telegram API.
Now I need to send this file in the second POST request (form-data).
Right now my code raises such error:
Exception: embedded null byte Traceback
The code snippet:
# Download the file via GET request of the Telegram API.
response = requests.get("{0}/file/bot{1}/{2}".format(TELEGRAM_API_URL, telegram_bot_token, file_path))
files = [
('file', (file_name, open(response.content, 'rb'), 'application/octet-stream')) # error
]
# Upload the file via POST request.
response = requests.post(FILE_STORAGE_SERVICE_API_URL, files=files)
Question:
How do I convert a file correctly so that a POST request can process it?

The data type of the response.content which I take from the first request is <class 'bytes'>.
The working code snippet:
response = requests.get("{0}/file/bot{1}/{2}".format(TELEGRAM_API_URL, telegram_bot_token, file_path))
files = {
"file": response.content
}
requests.post(FILE_STORAGE_SERVICE_API_URL, files=files)

Simulate multipart/form-data file upload with Falcon's Testing module

This simple Falcon API will take a HTTP POST with enctype=multipart/form-data and a file upload in the file parameter and print the file's content on the console:
# simple_api.py
import cgi
import falcon
class SomeTestApi(object):
def on_post(self, req, resp):
upload = cgi.FieldStorage(fp=req.stream, environ=req.env)
upload = upload['file'].file.read()
print(upload)
app = falcon.API()
app.add_route('/', SomeTestApi())
One might also use the falcon-multipart middleware to achieve the same goal.
To try it out, run it e.g. with gunicorn (pip install gunicorn),
gunicorn simple_api.py
then use cUrl (or any REST client of choice) to upload a text file:
# sample.txt
this is some sample text
curl -F "file=#sample.txt" localhost:8000
I would like to test this API now with Falcon's testing helpers by simulating a file upload. However, I do not understand yet how to do this (if it is possible at all?). The simulate_request method has a file_wrapper parameter which might be useful but from the documentation I do not understand how this is supposed to be filled.
Any suggestions?

This is what I came up with, which tries to simulate what my Chrome does.
Note that this simulates the case when you are uploading only one file, but you can simply modify this function to upload multiple files, each one separated by two new lines.
def create_multipart(data, fieldname, filename, content_type):
"""
Basic emulation of a browser's multipart file upload
"""
boundry = '----WebKitFormBoundary' + random_string(16)
buff = io.BytesIO()
buff.write(b'--')
buff.write(boundry.encode())
buff.write(b'\r\n')
buff.write(('Content-Disposition: form-data; name="%s"; filename="%s"' % \
(fieldname, filename)).encode())
buff.write(b'\r\n')
buff.write(('Content-Type: %s' % content_type).encode())
buff.write(b'\r\n')
buff.write(b'\r\n')
buff.write(data)
buff.write(b'\r\n')
buff.write(boundry.encode())
buff.write(b'--\r\n')
headers = {'Content-Type': 'multipart/form-data; boundary=%s' %boundry}
headers['Content-Length'] = str(buff.tell())
return buff.getvalue(), headers
You can then use this function like the following:
with open('test/resources/foo.pdf', 'rb') as f:
foodata = f.read()
# Create the multipart data
data, headers = create_multipart(foodata, fieldname='uploadFile',
filename='foo.pdf',
content_type='application/pdf')
# Post to endpoint
client.simulate_request(method='POST', path=url,
headers=headers, body=data)

You can craft a suitable request body and Content-Type using the encode_multipart_formdata function in urllib3, documented here. An example usage:
from falcon import testing
import pytest
import myapp
import urllib3
# Depending on your testing strategy and how your application
# manages state, you may be able to broaden the fixture scope
# beyond the default 'function' scope used in this example.
#pytest.fixture()
def client():
# Assume the hypothetical `myapp` package has a function called
# `create()` to initialize and return a `falcon.App` instance.
return testing.TestClient(myapp.create())
# a dictionary mapping the HTML form label to the file uploads
fields = {
'file_1_form_label': ( # label in HTML form object
'file1.txt', # filename
open('path/to/file1.txt').read(), # file contents
'text/plain' # MIME type
),
'file_2_form_label': (
'file2.json',
open('path/to/file2.json').read(),
'application/json'
)
}
# create the body and header
body, content_type_header = urllib3.encode_multipart_formdata(fields)
# NOTE: modify these headers to reflect those generated by your browser
# and/or required by the falcon application you're testing
headers = {
'Content-Type': content_type_header,
}
# craft the mock query using the falcon testing framework
response = client.simulate_request(
method="POST",
path='/app_path',
headers=headers,
body=body)
print(response.status_code)
Note the syntax of the fields object, which is used as input for the encode_multipart_formdata function.
See Tim Head's blog post for another example:
https://betatim.github.io/posts/python-create-multipart-formdata/
Falcon testing example copied from their docs:
https://falcon.readthedocs.io/en/stable/api/testing.html

How to post and retrieve blob with Django

I have a blob. It's an image that I resized using a <canvas>. I've verified that the data is correct by converting it to a url to test it as per the MDN guide. So far so good. Now, I'd like to post it to my Django server (along with some other inputs).
So I do this:
var fd = new FormData(form);
canvas.toBlob( function(blob) {
fd.set("image0", blob, "image0.jpg");
}, "image/jpeg", 0.7);
var xhr = new XMLHttpRequest();
xhr.open('POST', '/ajax-upload/', true);
xhr.setRequestHeader("X-CSRFToken", csrftoken);
xhr.send(fd);
I inspect the POST message with the network inspector console. My blob is confirmed as sent with the POST request and I can see the binary data send as the "image0" field.
-----------------------------1773139883502878911993383390
Content-Disposition: form-data; name="image0"; filename="blob"
Content-Type: image/png
So I handle the POST request with this view, accessible at url /ajax-upload/:
def ajax_upload(request):
if request.method == 'POST':
print(request.POST.urlencode())
This gives me nothing. Once I find out where my blob went, how can I turn it into an Image? Something like img = Image.open(request.POST["image0"])?

A blob is binary data, so you can find it in the request.body in Django. Its Bytes encoded (not Unicode).
HttpRequest.body
The raw HTTP request body as a bytestring. This is useful for processing data in different ways than conventional HTML forms: binary images, XML payload etc.

It is easy, you may just... read it:
#views.py
class Upload(View):
def get(self, request):
# Sanity check if your server is up and running, reply to HTTP request
print('Got get request...') # log it for debug purpose
return HttpResponse('<H1>POST endpoint.</H1>', status=200)
def post(self, request):
## here I used it with plupload.js for chunked upload, but it works anyway
... ## here you do something with headers
file_name = request.POST['name'] # or some 'file_name_with_path.bin'
chunk_num = int(request.POST['chunk']) # get it from the request
## Finally, you know this is multipart and headers are okay, let save it.
uploaded_file = request.FILES['file'] # data from request
if chunk_num == 0:
## The very first chunk — create/overwrite binary file
with open(file_name, 'wb+') as f:
f.write(uploaded_file.read()) # so, just read it from the request
else:
## Next chunks — append the file!
with open(file_name, 'ab') as f:
f.write(uploaded_file.read())
In your case:
def ajax_upload(request):
...
img = Image.open(request.POST["file"].read())
...
return HttpResponse('{"Status":"OK"}', status=200)
PS: I am aware, the question is a bit old, but I had the same issue and solved it in no time, but I struggled to find a solution by search initially.

s3 uploading file post request

I am trying to upload a file via a post request to amazon s3. Trouble is I don't know how to format the request to be a multipart form.
This is what I have right now:
content_type = "image/JPEG"
key = 'uploads/filename.jpg'
acl = "public-read"
bucket = None
params_raw = create_upload_data(content_type,key,acl,bucket)
params = { 'policy': params_raw['policy'],'acl':acl,'signature':params_raw['signature'],'key':params_raw['key'],'Content-Type':params_raw['Content-Type'],'AWSAccessKeyId':params_raw['AWSAccessKeyId'],'success_action_status':params_raw['success_action_status'],'binary': binary_data}
r = requests.post(params_raw['form_action'],data=params)
I think I am getting a bad response because it isn't a multipart form but here is what the response text looks like:
{"request": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidArgument</Code><Message>Conflicting query string parameters: acl, policy</Message><ArgumentName>ResourceType</ArgumentName><ArgumentValue>acl</ArgumentValue><RequestId>D558E016151E448F</RequestId><HostId>WT5aT0OOqJx3ziPgYFzjuTHJSERaCcuJG+y/acs6+l/mWVwO0MiH3lhWyBWIdhKr9BnhdIpkarw=</HostId></Error>"}
How do I structure the request with the file... it is a .jpg in base 64?

It is sufficient to change content-type
content-type = 'multipart/form-data'

Had a fair bit of pain around this but finally got it going. Very simple in the end!
url = "https://yourbucket.s3.amazonaws.com"
#complete_path is the local server path to the file
files = {'file':open(complete_path,'rb')}
r = requests.post(url, data=params, files=files)

OkHttp upload multiple files with names to Python Flask server

At Server end I use Python flask:
from flask import Flask, request
app = Flask(__name__)
#app.route("/upload", methods=["POST"])
def upload():
print request.files
print request.form
return "200 ok"
if __name__ == '__main__':
app.run(port=5000)
Java test code block:
public void test_Upload() throws Exception{
MediaType MEDIA_TYPE_PNG = MediaType.parse("image/png");
MediaType MEDIA_TYPE_XO = MediaType.parse("image/png");
RequestBody requestBody = new MultipartBuilder()
.type(MultipartBuilder.FORM)
.addPart(
Headers.of("Content-Disposition", "form-data; name=\"title\""),
RequestBody.create(null, "Square Logo"))
.addPart(
Headers.of("Content-Disposition", "form-data; name=\"image\""),
RequestBody.create(MEDIA_TYPE_PNG, new File("/Users/lollipop/Downloads/ic_launch.png")))
.addPart(
Headers.of("Content-Disposition", "form-data; name=\"google\""),
RequestBody.create(MEDIA_TYPE_XO, new File("/Users/lollipop/Downloads/google-logo.png")))
.build();
Request request = new Request.Builder()
.url("http://localhost:5000/upload")
.post(requestBody)
.build();
Response resp = httpClient.newCall(request).execute();
System.out.println(resp.body().string());
}
And I run the test. server end cannot read the file from request.forms
output on server:
ImmutableMultiDict([])
ImmutableMultiDict([('image', u'5 ...many data ... fffd'), ('google', u'5i\u ...many data ... fffd'),('title', u'Square Logo')])
Why my files upload to request.form not in request.files. And all binary file data is parsed to unicode string.
Next, I test in Python requests. follows codes:
resp = requests.post("http://localhost:5000/upload",
files={
"image": open("/Users/lollipop/Downloads/ic_launch.png", "rb"),
"title": open("/Users/lollipop/Downloads/ic_launch.png", "rb"),
"google": open("/Users/lollipop/Downloads/google-logo.png", "rb")
})
And the server end output is reversed:
ImmutableMultiDict([('image', <FileStorage: u'ic_launch.png' (None)>), ('google', <FileStorage: u'google-logo.png' (None)>), ('title', <FileStorage: u'ic_launch.png' (None)>)])
ImmutableMultiDict([])
the upload files are in request.files not in request.form, this is my expected result.
So. how can I use OkHttp to upload files to flask server, and use request.files to retrive.
Update
the request is Flask requst.
from flask import request
the requests is a Python http client library.

This bug is not to Flask. I get misguided by this answer
At the OkHttp recipe in the documents. Described the post a file use the following code
public static final MediaType MEDIA_TYPE_MARKDOWN
= MediaType.parse("text/x-markdown; charset=utf-8");
private final OkHttpClient client = new OkHttpClient();
public void run() throws Exception {
File file = new File("README.md");
Request request = new Request.Builder()
.url("https://api.github.com/markdown/raw")
.post(RequestBody.create(MEDIA_TYPE_MARKDOWN, file))
.build();
Response response = client.newCall(request).execute();
if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);
System.out.println(response.body().string());
}
Actually, this manner is not error, it post a file you don't need to provide a filename and other form data.
But this is not a usual usage.
We often need to upload a file with specified name and some key-value form data. We even need to upload more than one file.
RequestBody requestBody = new MultipartBuilder()
.type(MultipartBuilder.FORM)
.addFormDataPart("username", username)
.addFormDataPart("password", password)
.addFormDataPart("avatar", "avatar.png", RequestBody.create(MEDIA_TYPE_PNG, sourceFile))
.addFormDataPart("introduce_image", "introduce_image.png", RequestBody.create(MEDIA_TYPE_PNG, sourceFile))
.build();