I have a NestJS api (uses express) that gets a file (pdf or image) and sends it to a Flask api to further analize.
When sent through Postman, the files are available in Flask through request.files but I've found no way to not send it to request.form when sending it from the node server:
formData.append(
"file",
file.buffer,
)
formData.append(
"filename",
file.originalname
)
const res = await fetch('http://flask_api', {
method: 'POST',
body: formData.getBuffer(),
headers: formData.getHeaders()
});
Then in the Flask api, the value is available through: request.form["file"] but it is a str and therefore can't be used as a file to be saved, etc.
Will somehow sending files to request.files in Flask from Node fix this? If so, can this be done all from memory i.e. not saving a temp file when I get it in the Node server?
Thank you!
Related
I have code that uses python requests to send a multipart request to a server.
the call looks like so:
files = { 'complex_data': (tuple with binary data)}
data = { 'simple_dict': value }
self.session.request(method='POST',full_uri, timeout=self.timeout, data=data,files=files)
On the server side, it's being read using request.multipart.file and request.multipart.field (for the 'files' and 'data' part respectively).
I want to update the client call to aiohttp, but I'm having difficulties getting the same effect. I cannot change the server handling.
I have tried:
with aiohttp.MultipartWriter("form-data") as mpwriter:
mpwriter.append_form(files)
mpwriter.append_json(data)
response = await self.session.request(method='POST',
full_uri,
timeout=self.timeout,
data=mpwriter)
This code doesn't seem to pass the data to the server correctly.
When trying to use append only (instead of append_form), I receive a parsing error on the client side.
I know there's some 'payload' option, but I couldn't find any information on how to use it correctly.
I am trying to understand more precisely how Http connections work with Flask, so I tried writing a very simple app and another simple connection with requests:
app = Flask('file-streamer')
#app.route("/uploadDumb", methods=["POST"])
def upload_dumb():
print("Hello")
return Response(status=200)
Then I am sending a big file (1.5 GB) to the endpoint with requests :
url = "http://localhost:5000/uploadDumb"
values = {"file": open(file, "rb")}
r = requests.post(url=url, data={}, files=values)
I expected Flask to wait for the whole file to be sent, even if the file is useless. However this is not what's happening, instead Flask returns a 200 response very quickly, which causes a BrokenPipeError with my second script.
Can someone explain to me what is happening here ?
I suppose that happens because Flask body-parsing is lazy (which is a good thing, actually). So when request comes, Flask only reads headers until body is accessed somehow (through request.data, request.files, request.json(), etc.). So to trigger a full body (and file) upload, try accessing request body to make Flask parse a file, like:
_ = request.data
What i am trying to ask is, i have a website which takes the user profile and generates some kind of user specific reports in pdf. Pdf is generated at back end. So when user clicks on "generate" button, it should show blocking message and once the file is generated blocking message is removed and file is displayed with tag.
I am able to do this with help of socket.io and to be very Frank I copied this solution from one blog without understanding much. So my stack is Python and socket.io is on node. While it worked perfectly on my dev machine but on server its keep on throwing connection error message in JavaScript and post message fails on back end. So I am currently having issues with front and back end. And thanks to my blindly copy paste thing I am kind of clueless about the fix.
Is there any alternative to this approach? I can think of using Python based socket io (this time I will read and implement ) but something similar can be achieved by any other approach? I am using this to only send the generated pad back to client there is no other communication b/w client and server.
Adding code
It works like, client opens the download/generate page and gets the ID assigned, once generate button is clicked ID is sent to back-end, now celery generates the PDF (converts to base64) and then posts that on nodejs/notify with ID. Now the socket.io is already listening on /notify url, so once the data is posted its displayed in browser.
Browser -> sock (register) -> AJAX -> Celery (task) -> on nodejs/notify -> browser sockio client displays the data
Here is the code,
Client / Browser code
// func called on generate btn press
$client.on('notify', function(result) {
$.unblockUI();
// alert('In Receiving function...');
var $obj = $('#dynamic-pdf');
$obj.hide();
$obj.attr('data', result);
$obj.show();
$('button#view-again').removeClass('disabled');
$('#previe').modal('show');
});
blockWithMsg("Generating PDF, Please Wait...");
$.ajax({
url: $url,
data: {clientId: $clientId},
method: 'POST',
cache: false,
headers: {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8'},
});
// func finished
<!-- register client on page load -->
<script src="http://localhost:3000/socket.io/socket.io.js"></script>
<script>
var $client = io('http://localhost:3000'),
$clientId = null;
$client.on('register', function(id) {
$clientId = id;
});
</script>
Node js code
var app = require('express')(),
server = require('http').Server(app),
io = require('socket.io')(server),
bodyParser = require('body-parser');
// Accept URL-encoded body in POST request.
app.use(bodyParser.json({limit: '50mb'}));
app.use(bodyParser.urlencoded({ limit: '50mb', extended: true, parameterLimit: 50000}));
app.use(require('morgan')('dev'));
// Echo the client's ID back to them when they connect.
io.on('connection', function(client) {
client.emit('register', client.id);
});
// Forward task results to the clients who initiated them.
app.post('/notify', function(request, response) {
var client = io.sockets.connected[request.body.clientId];
client.emit('notify', request.body.result);
response.type('text/plain');
response.send('Result broadcast to client.');
});
server.listen(3000);
I am getting emit on undefined value error
(Posted on behalf of the OP).
The error was I started site on different domain than one mentioned in the socket.io connection (which was still localhost) in browser JavaScript.
At Server end I use Python flask:
from flask import Flask, request
app = Flask(__name__)
#app.route("/upload", methods=["POST"])
def upload():
print request.files
print request.form
return "200 ok"
if __name__ == '__main__':
app.run(port=5000)
Java test code block:
public void test_Upload() throws Exception{
MediaType MEDIA_TYPE_PNG = MediaType.parse("image/png");
MediaType MEDIA_TYPE_XO = MediaType.parse("image/png");
RequestBody requestBody = new MultipartBuilder()
.type(MultipartBuilder.FORM)
.addPart(
Headers.of("Content-Disposition", "form-data; name=\"title\""),
RequestBody.create(null, "Square Logo"))
.addPart(
Headers.of("Content-Disposition", "form-data; name=\"image\""),
RequestBody.create(MEDIA_TYPE_PNG, new File("/Users/lollipop/Downloads/ic_launch.png")))
.addPart(
Headers.of("Content-Disposition", "form-data; name=\"google\""),
RequestBody.create(MEDIA_TYPE_XO, new File("/Users/lollipop/Downloads/google-logo.png")))
.build();
Request request = new Request.Builder()
.url("http://localhost:5000/upload")
.post(requestBody)
.build();
Response resp = httpClient.newCall(request).execute();
System.out.println(resp.body().string());
}
And I run the test. server end cannot read the file from request.forms
output on server:
ImmutableMultiDict([])
ImmutableMultiDict([('image', u'5 ...many data ... fffd'), ('google', u'5i\u ...many data ... fffd'),('title', u'Square Logo')])
Why my files upload to request.form not in request.files. And all binary file data is parsed to unicode string.
Next, I test in Python requests. follows codes:
resp = requests.post("http://localhost:5000/upload",
files={
"image": open("/Users/lollipop/Downloads/ic_launch.png", "rb"),
"title": open("/Users/lollipop/Downloads/ic_launch.png", "rb"),
"google": open("/Users/lollipop/Downloads/google-logo.png", "rb")
})
And the server end output is reversed:
ImmutableMultiDict([('image', <FileStorage: u'ic_launch.png' (None)>), ('google', <FileStorage: u'google-logo.png' (None)>), ('title', <FileStorage: u'ic_launch.png' (None)>)])
ImmutableMultiDict([])
the upload files are in request.files not in request.form, this is my expected result.
So. how can I use OkHttp to upload files to flask server, and use request.files to retrive.
Update
the request is Flask requst.
from flask import request
the requests is a Python http client library.
This bug is not to Flask. I get misguided by this answer
At the OkHttp recipe in the documents. Described the post a file use the following code
public static final MediaType MEDIA_TYPE_MARKDOWN
= MediaType.parse("text/x-markdown; charset=utf-8");
private final OkHttpClient client = new OkHttpClient();
public void run() throws Exception {
File file = new File("README.md");
Request request = new Request.Builder()
.url("https://api.github.com/markdown/raw")
.post(RequestBody.create(MEDIA_TYPE_MARKDOWN, file))
.build();
Response response = client.newCall(request).execute();
if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);
System.out.println(response.body().string());
}
Actually, this manner is not error, it post a file you don't need to provide a filename and other form data.
But this is not a usual usage.
We often need to upload a file with specified name and some key-value form data. We even need to upload more than one file.
RequestBody requestBody = new MultipartBuilder()
.type(MultipartBuilder.FORM)
.addFormDataPart("username", username)
.addFormDataPart("password", password)
.addFormDataPart("avatar", "avatar.png", RequestBody.create(MEDIA_TYPE_PNG, sourceFile))
.addFormDataPart("introduce_image", "introduce_image.png", RequestBody.create(MEDIA_TYPE_PNG, sourceFile))
.build();
I am trying to upload a file to s3 from a form via ajax. I am using fineuploader http://fineuploader.com/ on the client side and webapp2 on the server side. it sends the parameter as qqfile in the request and I can see the image data in the request headers but I have no idea how to get it back in browsers that do not use the multipart encoding.
This is how I was doing it in the standard html form post with multipart.
image = self.request.POST["image"]
this gives me the image name and the image file
currently I have only been able to get the image filename back not the data with
image = self.request.get_all('image')
[u'image_name.png']
when using the POST I get a warning about the content headers being application/octet-stream
<NoVars: Not an HTML form submission (Content-Type: application/octet-stream)>
How do I implement a BlobStoreHandler in webapp2 outside of GAE?
Your question code is not very clear to me. But you can use an example.
Have a look at this article from Nick Johnson. He implements a dropbox service using app engine blobstore and Plupload in the client : http://blog.notdot.net/2010/04/Implementing-a-dropbox-service-with-the-Blobstore-API-part-2
I ended up using fineuploader http://fineuploader.com/ which sends a multipart encoded form to my handler endpoint.
inside the handler I could simply just reference the POST and then read the FieldStorage Object into a cStringIO object.
image = self.request.POST["qqfile"]
imgObj = cStringIO.StringIO(image.file.read())
# Connect to S3...
# create s3 Key
key = bucket.new_key("%s" % uuid.uuid4());
# guess mimetype for headers
file_mime_type = mimetypes.guess_type(image.filename)
if file_mime_type[0]:
file_headers = {"Content-Type": "%s" % file_mime_type[0]}
else:
file_headers = None
key.set_contents_from_string(imgObj.getvalue(), headers=file_headers)
key_str = key.key
#return JSON response with key and append to s3 url on front end.
note: qqfile is the parameter fineuploader uses.
I am faking the progress but that is ok for my use case no need for BlobStoreUploadHandler.