I developed a flask app which runs a mathematical optimization script (PuLP+ Python) for user provided data input. Since the computation takes quite a while, I want to display the optimizer's print statements constantly (without refreshing) on my webpage.
Based on this solution I managed to run a subprocesswith my .py script. Within this script, the print() statement does what it should, however I can only see the output in the command line. By using flash() I managed to display the output, yet it gets rendered only once the computation has finished and the page is reloaded. I try to output the print() statements in realtime within my HTML. Is there a way to achieve this?
Thanks!
Excerpt from routes.py:
#app.route('/optimize', methods=['GET', 'POST'])
def solve():
path = os.path.join(app.root_path, 'optimization', 'assets')
file = "test.py"
execute(file, path)
return redirect(url_for('home'))
The execute(file,path) function:
import os
import subprocess as sub
def execute(command, directory):
# change directory to execute the command
os.chdir(directory)
proc = sub.Popen(['python','-u', command], stdout=sub.PIPE, universal_newlines=True) # Python 2: bufsize=1
for stdout_line in proc.stdout:
print(stdout_line.rstrip() + '\n')
flash(stdout_line.rstrip(), 'python_print')
proc.stdout.close()
return_code = proc.wait()
if return_code:
raise sub.CalledProcessError(return_code, command)
And finally my HTML which is rendered in the home route:
{% with prints = get_flashed_messages(category_filter=["python_print"]) %}
{% if prints %}
<div class="content-section">
<h3>Solver Status</h3>
<ul>
{%- for message in prints %}
<li>{{ message }}</li>
{% endfor -%}
</ul>
</div>
{% endif %}
{% endwith %}
As far as I know this is how message flashing is supposed to work, i.e. it will display all the messages for a particular end point all at once.
I am not sure about the exact implementation since I have not looked at the source code for flash() and get_flash_messages() but this is my understanding of what might be happening in the background, every message you flash with flash function in your python code it gets appended to an iterable, and when you call the get_flashed_messages() it returns that iterable back, which you can loop over to get the messages.
Though your execute() function is adding messages to that iterable before the process has been completed, but notice the execute function is being called inside the view solve and the actual redirect can happen only after the execute function has completed its execution, and that's when the template home will gets the iterable having all the flashed messages.
Hope that makes sense.
Based on the suggestions provided I changed my approach and got it running. Instead of using subprocess, the solver is imported as module. I'm generating a .txt using the python logging module rather than trying to catch the print statements. This file is streamed constantly using flask response + yield and parsed via AJAX. The getter is called in jQuery via setInterval (here: 1Hz) and added to my HTML. I will post some snippets below for anyone dealing with the same issue.
Thanks for your support!
Flask optimize route:
import supplierAllocation.optimization.optimizer as optimizer
#app.route('/optimize')
def optimize():
if session.get('input_file') is not None:
# Get input_file
input_file = session.get('input_file')
# Path to logfile
log_file = os.path.join(
app.config['LOG_FOLDER'], session.get('log_file'))
# Assign random unique hex file name from session to output file
out_file_fn = session.get('session_ID') + '.xlsx'
output_file = os.path.join(app.config['DOWNLOAD_FOLDER'], out_file_fn)
# Search for outdated output; if present: delete
if session.get('output_file') is None:
session['output_file'] = out_file_fn
else:
session_output = os.path.join(
app.config['DOWNLOAD_FOLDER'], session.get('output_file'))
silent_remove(session_output)
# Pass input and output parameter to solver
optimizer.launch_solver(input_file, output_file, log_file)
else:
flash("Please upload your data before launching the solver!", 'warning')
return redirect(url_for('home'))
Flask streaming route:
#app.route('/streamLog')
def stream_logfile():
if session.get('log_file') is None:
return Response(status=204)
else:
session_log = os.path.join(
app.config['LOG_FOLDER'], session.get('log_file'))
def stream():
with open(session_log, "rb") as file:
yield file.read()
return Response(stream(), mimetype='text/plain')
Call of AJAX request on button click:
// Print solver output with 1Hz
var session_log = setInterval(get_Log, 1000);
AJAX request in jQuery:
// AJAX Request to download solutions
function get_Log() {
console.log('Sending AJAX request...');
$.ajax({
url: "streamLog",
dataType: "text",
success: function(result) {
$("#code-wrapper").html(result);
},
error: function() {
console.log("ERROR! Logfile could not be retrieved.");
}
});
}
Related
In my django web app, I create a .csv file from a list in my views.py and I want the user to be able to download it when the button is pressed. However, when this is run, I get the error that 'headers' is an unexpected argument.
This is the code in my views.py:
output = ['a', 'b', 'c']
response = HttpResponse(content_type='text/csv', headers={'Content-Disposition': 'attachment; filename="myfile.csv"'})
writer = csv.writer(response)
writer.writerows(output)
return response
I have no idea why this doesn't work, since I have used identical code elsewhere with success in the past. Does anyone know if this could be a browser support issue?
Another way to set your header for HttpResponse is like this:
response['Content-Disposition'] = 'attachment; filename="myfile.csv"'
For the error you are getting, can you double check that your django version is indeed 3.2?
I was able to make it work (but not really). I pass the output to the same webpage in a template in JSON format. Then I download the contents of that template using JavaScript. It works, but it feels janky and there are two fairly significant drawbacks to this approach:
All of the form fields on that webpage are reset
Commas in strings become a new cell in the .csv file, shifting everything after to the right.
views.py
output = json.dumps(output)
form = MyForm(initial={'data':my_data})
return render(request, 'wireCalculator/chassis.html', {'base':base, 'download':output, 'form':form})
html page addition:
<div style="display: none;">
<p id="download">{{ download }}</p>
</div>
Javascript addition:
function download(filename, text) {
// creates a link, downloads it, then deletes the link
var element = document.createElement('a')
element.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(text))
element.setAttribute('download', filename)
element.style.display = 'none'
document.body.appendChild(element)
element.click()
document.body.removeChild(element)
}
const download = JSON.parse(document.getElementById("download").innerHTML)
if (download != "") {
var downloadText = ""
for (i=0; i < download.length; i++){
downloadText = downloadText.concat(download[i])
downloadText = downloadText.concat("\n")
}
download("filename.csv", downloadText)
}
I am very new to Django...
Using submit button i want to run a python file in background and display content on next page...
But my python file takes some time to take out the result so in between i wanted to put a loading html page in between....
I have written some code which correctlty runs the python file but i am not able to incorporate the loading page in between...
Take a look at my function in views.py
def submit(request):
info = request.POST['info']
print('value is ', info)
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE )
return render_to_response("loading.html")
run(['python', filename, info], stdout= PIPE )
return render(request, 'python_output.html', context)
ACTUAL RESULT:
return render_to_response("loading.html")
works but then the control does not shifts to run command...
I want to run the loading html page and run the python file in background and when python file run finishes it should move to python _output.html page where output is displayed...
Expected:
Loading page should work and after that control should shift to run command and then it should go to python_output.html page.../
The return statement will terminate the execution of the function so anything below it will never be reached.
You can use Javascript to show the loading icon and then use JQuery to run a GET request in the background where you call a custom view from Django that will output the result of the command. When data is received you can then remove the icon and process the data as you want.
Basic Example :
Django
------
url(r'^command/', views.command, name='command'),
def command(request):
info = request.POST['info']
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE
return result
Javascript
----------
<img id="loading-icon" src="loading.gif">
$.get("/command", function(text)
{
$("#loading-icon").remove();
process(text);
});
You need to understand the basic flow in Django
You can only add one return in your view. after the execution of the first return it goes to the response middleware so all other returns below are ignored.The loading can be done in javascript in frontend
What you want involves a bit more work than you might have expected:
Install a background task framework like celery (and a queue like Redis or RabbitMQ to store tasks) that will fetch tasks from the queue and process them.
Your initial view needs to start a background task.
You want to keep track of the task_id for this task, either by returning it in your response to the user or saving it in the session of the user. Your view responds with the HTML page saying "please have some patience..." and javascript to handle what follows (see below).
Then you need a view that can check the status of the task (based on either the task_id passed in the query or saved in the current session). It responds with JSON, either by saying "status = processing..." or "status = done" with the result of the task.
In the HTML page, you need javascript that queries that view at regular intervals, until the status is "done" and then processes the result of the task to update the HTML of the page.
Search for "django celery tutorial" and you'll find plenty of examples.
Try to use an ajax request when you load your first page (loading.html) to run the python file in background and when it is done, display the result via output.html.
Using JQuery, in the template file, you must call a function like this :
<script>
var url_file_to_run = "{% url "your_app:file_to_run_adress" 0 %}";
var url = "{% url "your_app:python_output" 0 %}";
$.ajax({
url: url_file_to_run,
}
}).done(function(data) {
$( location ).attr("href", url);
});
</script>
I hope i understand your problem.
I have a Python flask application which takes input id's and dynamically generates data into a html file. Below is my app.py file.
#app.route('/execute', methods=['GET', 'POST'])
def execute():
if request.method == 'POST':
id = request.form['item_ids']
list = [id]
script_output = subprocess.Popen(["python", "Search_Script.py"] + list)
# script_output = subprocess.call("python Search_Script.py "+id, shell=True)
# render_template('running.html')
script_output.communicate()
#driver = webdriver.Chrome()
#driver.get("home.html")
#driver.execute_script("document.getElementById('Executed').style.display = '';")
return render_template('execute.html')
#app.route('/output')
def output():
return render_template('output.html')
output.html file has below code at the bottom.
<div class="container" style="text-align: center;">
{% include 'itemSearchDetails.html' %}
</div>
itemSearchDetails.html is generated every time dynamically based on the input. I check for different inputs and it is generating perfectly. When I run it with some input(assume 2) values for the first time, it runs perfectly and shows the output correctly. But, when I run for different values(assume 4) for the next time, the file 'itemSearchDetails.html' is generated for those 4 values but the browser only shows output for the first 2 values. No matter how many times I run it, browser shows only output with the first run values.
So, every time only the first inputted values are shown no matter how many times I run. I am not sure if it is browser cache issue since I tried "disabling cache" in chrome. Still it didn't work. Please let me know if there is something I am missing.
Try solution from this answer:
Parameter TEMPLATES_AUTO_RELOAD
Whether to check for modifications of the template source and reload
it automatically. By default the value is None which means that Flask
checks original file only in debug mode.
Original documentation could be found here.
Looks like Jinja is caching the included template.
If you don't need to interpret the HTML as a Jinja template, but instead just include its contents as-is, read the file first and pass the contents into the template:
with open('itemSearchDetails.html', 'r') as infp:
data = infp.read()
return render_template('execute.html', data=data)
...
{{ data|safe }}
(If you do need to interpret the HTML page as Jinja (as include will), you can parse a Jinja Template out of data, then use the include tag with that dynamically compiled template.)
I've been reading the book 'Head First Python' where the writer talks about creating dynamic webpages using a module he created called 'yate', an HTML template engine (which I renamed to site_yate in the code below). The example he works through is a hypothetical coach wanting his athletes to be able to check their times online. The design is as follows: first you enter the homepage which has a link to run a script which generates a webpage where you can select the athlete whose times you want to view. Then when you select your athlete and click submit the form calls another script called "site_generate_timing_data.py" where you can views the athlete's top times. So I decided to take it further and add functionality to add a time for the athlete, using this extra line of code in my python script.
print(site_yate.do_form("addtime.py", [athlete_id]))
The HTML this will generate will be this:
<form action="addtime.py" method="POST">
<h1>Want to add a time?</h1>
<input type="Text" name="1" size=40> //"1" is the athlete's id in this example
<input type="Submit" value="Submit">
</form>
As you can see this code calls the script 'addtime.py' which has the following code:
import cgi
import sqlite3
data = cgi.FieldStorage().value[0] #this attribute will be in the form MininFieldStorage(name, value)
id = data.name #this attribute is the input's name i.e. athlete's id
time = data.value #this attribute is the input's value i.e. the time
connection = sqlite3.connect("NUACDB.sqlite") #my DB's name
cursor = connection.cursor()
cursor.execute("""INSERT INTO timing_data (athlete_id, time)
VALUES (?, ?)""",
(id, time)) #just SQL stuff
connection.commit()
connection.close()
Which works fine, however I want to change a few thing about this, since it leaves the user on a blank page. I could generate some HTML code to provide links to the homepage etc. or even JavaScript code to redirect the user automatically, but I want to keep this script HTML-free so that I can also use it elsewhere.
What I want to do instead is make the script execute on the same page. Not only that, but I would also prefer if I could put the addtime.py code as a function in another module called 'athletemodel.py' and call it form there, i.e. athletemodel.addtime() (or I could do from athletemodel import addtime so I can call the function directly). How can I call a python function using HTML code? I'm aware of the onsubmit="" form attribute but apparently that is for JavaScript functions. Another thing I'm unsure about is whether the data submitted in the form will still be accessible through CGI FieldStorage and hence whether my addtime.py code will still work as it is.
This stuff is so confusing! All help is appreciated.
Not sure if you already had it in mind, but I would use ajax (remember to include the jQuery library). Here's a rough example to get you started if this is what you want. It'll keep them on the same page:
JavaScript file:
$('#submitButtonId').click(function (event) {
event.preventDefault();
$('#submitButtonId').hide();
$('#thinking').show(); //some div with a nice ajax loader gif...
$.ajax({
type: 'POST',
data: $('#formId').serialize(),
url: '/URL_path_to_function',
success: function (data) {
$('#loading').hide();
var response = data.result //now do stuff with your response
}
error: function(error){
console.log('Error')}
});
Python view/function:
import jsonify
if request.method == 'POST':
value = request.form['input value'] #flask...
#Do stuff
return jsonify(result='Some response')
I'm new to web servers, so I need some guidance as to how to achieve the following:
User that's on the same network as server opens a page on the server (eg 10.42.68.130:8080/cycle).
User is presented with a form, that asks him to enter a number of cycles.
User clicks submit. The submission activates another function that executes a for-loop the specified number of times (I simplified all of this, so it's really supposed to fire GPIO pins on a beaglebone black)
The user receives progress feedback, so a counter that shows how many cycles have been completed.
Any help would be much appreciated.
I'm going off the tutorials in learnpythonthehardway.com lessons 50 and 51. This is what I managed to get going.
bin/app.py
import web
import time
urls = ('/cycle', 'Index')
app = web.application(urls, globals())
render = web.template.render('templates/', base="layout")
class Index(object):
def GET(self):
return render.hello_form2()
def POST(self):
form = web.input(cycles=0)
cycleparams = "%s" % (form.cycles)
cycleGPIO(form.cycles)
return render.index2(cycleparams = cycleparams)
if __name__ == "__main__":
app.run()
def cycleGPIO(cycles):
for i in range(int(cycles)):
print "cycle " + str(i+1)
time.sleep(1)
templates/layout.html
$def with (content)
<html>
<head>
<title>Test Page</title>
</head>
<body>
$:content
</body>
</html>
templates/index2.html
$def with (cycleparams)
$if cycleparams:
Test completed with the following parameters:
<br>
$cycleparams
$else:
No parameters specified
<br><br>
Return to form
templates/hello_form2.html
<fieldset>
<legend>Output Cycling</legend>
<form action="/cycle" method="POST">
<br>Number of cycles:
<br><input type="number" name="cycles">
<input type="submit">
</form>
</fieldset>
The best way to approach this issue is to separate your web and GPIO cycle processes. After this you can use one of the many interprocess communication mechanisms available in python. A good page to read about this is Interprocess Communication and Networking.
For now, I'll just pick an easiest way to communicate between two python processes, a plain text file. We'll have two files. One that your web service process will use to send your form input to GPIO process, and another one that GPIO process will use to send feedback to the web service.
Please remember that this is only an example, and there are many many better ways to solve interprocess communication problem. This is just here to give you a rough idea, and not something that you should use in any production systems.
This is how code could look like.
web.py service changes
...
urls = ('/cycle-status', 'StatusIndex')
...
class StatusIndex(object):
def GET(self):
# Read feedback params from a file
cycleparams = read_gpio_feedback_file()
return render.index2(cycleparams = cycleparams)
class Index(object):
...
def POST(self):
form = web.input(cycles = 0)
cycleparams = "%s" % (form.cycles)
# Write cycle params to a file
write_cycle_params(form.cycles)
# This call probably won't produce any results
# because your GPIO process couldn't react so fast.
return render.index2(cycleparams = {})
def write_cycle_params(cycles):
f = open('/path/to/cycles/file.txt', 'w')
for i in range(int(cycles)):
f.write("cycle " + str(i + 1))
f.close()
def read_gpio_feedback():
cycleparams = {}
f = open('/path/to/feedback/file.txt', 'r')
for line in f:
cycleparams['param1'] = line
f.close()
return cycleparams
GPIO process
#!/usr/bin/env python
import sys
if __name__ == '__main__':
cycleparams = ""
f = open('/path/to/cycles/file.txt', 'r')
for line in f:
cycleparams = cycleparams + line
f.close()
# Do something with cycleparams related to GPIO
# Get your cycle params from GPIO pins
...
cycleparams = ""
f = open('/path/to/feedback/file.txt','w')
f.write(cycleparams + '\n')
f.close()
sys.exit(0)
Your GPIO process will have to run periodically to read info from the web service generated file, and also to write feedback that web service will parse and display output. You can achieve this by adding GPIO process into crontab (or any other process scheduling mechanism will work fine).