I have an HTML page that lets you upload a picture and displays it (very simple). But how do I send that particular loaded image into my flask app so that I could, say, perform some image processing on it using python's openCV or PIL packages?
Disclaimer
I've done similar thing recently, It may not be a 100% solution for you, but using parts of this code will propably solve all of your problems.
Flask Code
This part gets all files uploaded from browser and saves them with their respected extensions.
if request.method=="PUT":
for f in request.files.getlist("Files"): #<---- note 1
f.save(os.path.join(Path_To_Folder, f.filename))
return jsonify({"result": "res"}), 200 #status code is only thing important here
HTML
This just allows you to select file(s) from your device
<input type="file" id="file_loader" multiple/>
JavaScript
This code
const file_loader = document.getElementById("file_loader");
file_loader.onchange = async function(e){
alert("Upload started")
let sending = new FormData(); //creates form data object
//this for loop adds all files you selected to a form object
for (let i = 0; i < file_loader.files.length; i++) {
sending.append("Files", file_loader.files[i]); //<---- note 1
}
//this part just sends all the files to server
const podaci = await fetch(url_filesistem, {
method: "PUT",
body: sending,
})
.then((response) => response.json())
.then((pod) => {return pod;});
//removing all selected files from an input, so every time you want to select files
//its just those files, not ones previously selected
while (file_loader.length > 0) {
file_loader.pop();
}
alert("Finnished uploading")
}
Note 1
String "Files" mentioned in both lines needs to be the same in order for this method to work.
Advice
First save all the files on the server and then do the processing. I don't know what "f" object from for loop in python contains, therefore I don't know whether you can process them immediately.
Feel free to ask me anything regarding my code!
Best regards!
Related
We have a c#/.net Windows service that parses big log files for us and updates a meta table when it does so. The problem is whenever you need to stop the service or (services, we have multiple of them running), one must manually delete the files that are in the process of being parsed in the local folder and also update the queue DB table where it tracks files to process.
I want to automate this. I am much more familiar with python so ideally, it would be a python script as opposed to .net. Is it possible to have a script that will trigger when the service is stopped? How would one do this?
I have tried doing this internally in the .net service but since it's multithreaded, files don't get cleaned up neatly. There's always a "can't stop service because another process is using it". It is like the service gets stuck trying to delete files when the Onstop() method is called. This was how I had tried to do it internally within the service:
protected override void OnStop()
{
ProducerConsumerQueue.Dispose();
Logger.Info($"{ProducerConsumerQueue.Count()} logs will be canceled");
CancellationTokenSource.Cancel();
FileUtil.DeleteFilesInProgress(Constants.ODFS_STAGING);
MetadataDbContext.UpdateServiceEntriesOnServiceReset();
//look into some staging directory, delete all files.
Logger.Info($"{ProducerConsumerQueue.Count()} logs canceled");
}
public static void DeleteFilesInProgress(string directory)
{
var constantsutil = new ConstantsUtil();
constantsutil.InitializeConfiguration();
try
{
System.IO.DirectoryInfo di = new DirectoryInfo(directory);
foreach (FileInfo file in di.GetFiles())
{
file.Delete();
}
}
catch(Exception ex)
{
Logger.Error(ex.Message);
string subject = Constants.GENERAL_EMAIL_SUBJECT;
string body = "The following error occured in Client.Util.ConstantsUtil:";
string error = ex.ToString(); //ex.ToString makes it more verbose so you can trace it.
var result = EmailUtil.Emailalert(subject, body, error);
}
}
public static int UpdateServiceEntriesOnServiceReset()
{
int rowsAffected = 0;
try
{
string connectionString = GetConnectionString();
using (SqlConnection connection = new SqlConnection())
{
connection.ConnectionString = connectionString;
SqlCommand cmd = new SqlCommand();
cmd.CommandType = CommandType.Text;
cmd.CommandText = $"UPDATE {Constants.SERVICE_LOG_TBL} SET STATUS = '0'";
cmd.Connection = connection;
connection.Open();
rowsAffected = cmd.ExecuteNonQuery();
}
}
catch (Exception ex)
{
Logger.Error($"{ex.Message.ToString()}");
string subject = Constants.GENERAL_EMAIL_SUBJECT;
string body = "The following error occured in Client.MetadatDbContext while Parser was processing:";
string error = ex.ToString(); //ex.ToString makes it more verbose so you can trace it.
var result = EmailUtil.Emailalert(subject, body, error);
}
return rowsAffected;
}
You can run your script from OnStop:
using System.Diagnostics;
Process.Start("python yourscript.py");
// or whatever the command for executing your python script is on your system.
And then use something like pywin32's win32service to find out the status of the service that launched the script, and then wait for it to die and release its hold on the files.
Then wipe them.
I am very new to Django...
Using submit button i want to run a python file in background and display content on next page...
But my python file takes some time to take out the result so in between i wanted to put a loading html page in between....
I have written some code which correctlty runs the python file but i am not able to incorporate the loading page in between...
Take a look at my function in views.py
def submit(request):
info = request.POST['info']
print('value is ', info)
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE )
return render_to_response("loading.html")
run(['python', filename, info], stdout= PIPE )
return render(request, 'python_output.html', context)
ACTUAL RESULT:
return render_to_response("loading.html")
works but then the control does not shifts to run command...
I want to run the loading html page and run the python file in background and when python file run finishes it should move to python _output.html page where output is displayed...
Expected:
Loading page should work and after that control should shift to run command and then it should go to python_output.html page.../
The return statement will terminate the execution of the function so anything below it will never be reached.
You can use Javascript to show the loading icon and then use JQuery to run a GET request in the background where you call a custom view from Django that will output the result of the command. When data is received you can then remove the icon and process the data as you want.
Basic Example :
Django
------
url(r'^command/', views.command, name='command'),
def command(request):
info = request.POST['info']
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE
return result
Javascript
----------
<img id="loading-icon" src="loading.gif">
$.get("/command", function(text)
{
$("#loading-icon").remove();
process(text);
});
You need to understand the basic flow in Django
You can only add one return in your view. after the execution of the first return it goes to the response middleware so all other returns below are ignored.The loading can be done in javascript in frontend
What you want involves a bit more work than you might have expected:
Install a background task framework like celery (and a queue like Redis or RabbitMQ to store tasks) that will fetch tasks from the queue and process them.
Your initial view needs to start a background task.
You want to keep track of the task_id for this task, either by returning it in your response to the user or saving it in the session of the user. Your view responds with the HTML page saying "please have some patience..." and javascript to handle what follows (see below).
Then you need a view that can check the status of the task (based on either the task_id passed in the query or saved in the current session). It responds with JSON, either by saying "status = processing..." or "status = done" with the result of the task.
In the HTML page, you need javascript that queries that view at regular intervals, until the status is "done" and then processes the result of the task to update the HTML of the page.
Search for "django celery tutorial" and you'll find plenty of examples.
Try to use an ajax request when you load your first page (loading.html) to run the python file in background and when it is done, display the result via output.html.
Using JQuery, in the template file, you must call a function like this :
<script>
var url_file_to_run = "{% url "your_app:file_to_run_adress" 0 %}";
var url = "{% url "your_app:python_output" 0 %}";
$.ajax({
url: url_file_to_run,
}
}).done(function(data) {
$( location ).attr("href", url);
});
</script>
I hope i understand your problem.
Summary
I am trying to set my FormData properly using javascript.
I need to be able to upload jpg/png, but I might need to upload some other file types pdf/csv in the future using fetch.
Expected
I expect it to append the data to the form
Error
Working
This snippet is working fine:
const formData = new FormData(document.querySelector('form'));
formData.append("extraField", "This is some extra data, testing");
return fetch('http://localhost:8080/api/upload/multi', {
method: 'POST',
body: formData,
});
Not working
const formData = new FormData();
const input = document.querySelector('input[type="file"]');
formData.append('files', input.files);
Question
Does fetch support multiple file upload natively?
If you want multiples file, you can use this
var input = document.querySelector('input[type="file"]')
var data = new FormData()
for (const file of input.files) {
data.append('files',file,file.name)
}
fetch('http://localhost:8080/api/upload/multi', {
method: 'POST',
body: data
})
The issue with your code is in the lineformData.append('files', input.files);
Instead of that, you should upload each file running a loop with unique keys, like this
const fileList = document.querySelector('input[type="file"]').files;
for(var i=0;i<fileList.length;i++) {
formData.append('file'+i, fileList.item(i));
}
I have created a simple error fiddle here with your code. You can check its' submitted post data here, where you can see that no file has been uploaded.
At the bottom of the page you can find
.
I have corrected the fiddle here with the fix. You can check its'post data from the server, where it shows the details of the two files that I uploaded.
I mentioned this on a similar question: I had the same problem, but with a PHP backend. The unique formData keys work, but I found that the classic HTML notation worked just fine and simply results in an array on the server.
formData.append('file[]', data[i]);
I like that a lot better, since I can use the same methods to process this as with a classic <input type="file" multiple />.
I am attempting to implement a flask application for uploading files. This file could be very large. For example, almost 2G in size.
I have finished the server side process function like this:
#app.route("/upload/<filename>", methods=["POST", "PUT"])
def upload_process(filename):
filename = secure_filename(filename)
fileFullPath = os.path.join(application.config['UPLOAD_FOLDER'], filename)
with open(fileFullPath, "wb") as f:
chunk_size = 4096
while True:
chunk = flask.request.stream.read(chunk_size)
if len(chunk) == 0:
return
f.write(chunk)
return jsonify({'filename': filename})
As for browser side, I should give users a from to submit the file. One file at a time. Show progressbar to indicate the uploading process.
But I have no idea about the browser side code. How can I use javascript code to start the uploading and show it status?
This will be a difficult task for your to figure out on your own. I would suggest a plugin like https://blueimp.github.io/jQuery-File-Upload/
You can see from this projects source code they use a method name which essentially looks at how large the file is and how much data has been transferred thus far and how much remains to show a percentage complete div.
code example from this project
progressall: function (e, data) {
var $this = $(this);
$this.find('.fileupload-progress')
.find('.progress').progressbar(
'option',
'value',
parseInt(data.loaded / data.total * 100, 10)
).end()
.find('.progress-extended').each(function () {
$(this).html(
($this.data('blueimp-fileupload') ||
$this.data('fileupload'))
._renderExtendedProgress(data)
);
});
}
https://github.com/blueimp/jQuery-File-Upload/blob/master/js/jquery.fileupload-jquery-ui.js
So if you do want to come up with your own solution, I would suggest you start by building a UI div rectangle which has a dynamic width which updates according to your percentage calculation based upon the file upload size and data uploaded... or just go with an already established solution.
How to restrict the size of file being uploaded.
I am using django 1.1 with apache.
Can I use apache for this and show some html error page if say size is bigger then 100MB.
Thanks.
I mean before uploading the file
On client side it isn't possible...
I suggest to write a custom upload handlers and to override receive_data_chunk.
Example: QuotaUploadHandler
You can do this in javascript in most recent browsers, using the File API: http://www.w3.org/TR/FileAPI/
For example (using jquery):
var TYPES = ['image/jpeg', 'image/jpg', 'image.png'];
var file = $('#my_file_input')[0].files[0];
var size = file.size || file.fileSize;
var type = file.type;
if (size > MAX_BYTES) {
alert('Error: file too large');
} else if (TYPES.indexOf(type) < 0) {
alert('Error: file not a JPG or PNG');
} else {
// proceed with file upload
}
No need for Java or Flash. Of course, you'll still need some sort of checking on the server for users who disable javascript.
apache has a server setting for max file size..(also dont forget max post size). I do not believe apache can show an error page on its own, you can probably use python for that.
unfortunetly I know nothing obout python (yet) so I can't really help you beyond that.
I know php can do that easily so I'm sure there is a method for python.
If you want to get the file size before uploading begins you will need to use Flash or a Java applet.
Writing a custom upload handler is the best approach. I think something like the following would work (untested). It terminates the upload as early as possible.
from django.conf import settings
from django.core.files.uploadhandler import FileUploadHandler, StopUpload
class MaxSizeUploadHandler(FileUploadHandler):
"""
This test upload handler terminates the connection for
files bigger than settings.MAX_UPLOAD_SIZE
"""
def __init__(self, request=None):
super(MaxSizeUploadHandler, self).__init__(request)
def handle_raw_input(self, input_data, META, content_length, boundary, encoding=None):
if content_length > settings.MAX_UPLOAD_SIZE:
raise StopUpload(connection_reset=True)