I am very new to Django...
Using submit button i want to run a python file in background and display content on next page...
But my python file takes some time to take out the result so in between i wanted to put a loading html page in between....
I have written some code which correctlty runs the python file but i am not able to incorporate the loading page in between...
Take a look at my function in views.py
def submit(request):
info = request.POST['info']
print('value is ', info)
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE )
return render_to_response("loading.html")
run(['python', filename, info], stdout= PIPE )
return render(request, 'python_output.html', context)
ACTUAL RESULT:
return render_to_response("loading.html")
works but then the control does not shifts to run command...
I want to run the loading html page and run the python file in background and when python file run finishes it should move to python _output.html page where output is displayed...
Expected:
Loading page should work and after that control should shift to run command and then it should go to python_output.html page.../
The return statement will terminate the execution of the function so anything below it will never be reached.
You can use Javascript to show the loading icon and then use JQuery to run a GET request in the background where you call a custom view from Django that will output the result of the command. When data is received you can then remove the icon and process the data as you want.
Basic Example :
Django
------
url(r'^command/', views.command, name='command'),
def command(request):
info = request.POST['info']
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE
return result
Javascript
----------
<img id="loading-icon" src="loading.gif">
$.get("/command", function(text)
{
$("#loading-icon").remove();
process(text);
});
You need to understand the basic flow in Django
You can only add one return in your view. after the execution of the first return it goes to the response middleware so all other returns below are ignored.The loading can be done in javascript in frontend
What you want involves a bit more work than you might have expected:
Install a background task framework like celery (and a queue like Redis or RabbitMQ to store tasks) that will fetch tasks from the queue and process them.
Your initial view needs to start a background task.
You want to keep track of the task_id for this task, either by returning it in your response to the user or saving it in the session of the user. Your view responds with the HTML page saying "please have some patience..." and javascript to handle what follows (see below).
Then you need a view that can check the status of the task (based on either the task_id passed in the query or saved in the current session). It responds with JSON, either by saying "status = processing..." or "status = done" with the result of the task.
In the HTML page, you need javascript that queries that view at regular intervals, until the status is "done" and then processes the result of the task to update the HTML of the page.
Search for "django celery tutorial" and you'll find plenty of examples.
Try to use an ajax request when you load your first page (loading.html) to run the python file in background and when it is done, display the result via output.html.
Using JQuery, in the template file, you must call a function like this :
<script>
var url_file_to_run = "{% url "your_app:file_to_run_adress" 0 %}";
var url = "{% url "your_app:python_output" 0 %}";
$.ajax({
url: url_file_to_run,
}
}).done(function(data) {
$( location ).attr("href", url);
});
</script>
I hope i understand your problem.
Related
I have a django project which contains a long running process. I have used django-background-tasks library for this. It works but I want to create a pending page for users and display the status of the task. I should refresh that page every 60 seconds and update the status. How can I do that?
Thank you.
Hope you know about Ajax.
How to use Ajax with Django: https://simpleisbetterthancomplex.com/tutorial/2016/08/29/how-to-work-with-ajax-request-with-django.html
How to run Ajax code every n seconds: https://thisinterestsme.com/ajax-request-every-10-seconds/
If you want to load a page partially, then you should split the page into two parts. One should contain the particular div which you want to refresh or reload, consider the page name as partial_load.html. And other page can have remaining codes consider the file name as full_page.html. The partial_load.html can be included inside the full_page.html by using include tag ({% include "partial_load.html" %})
def refresh_page(request):
if request.is_ajax():
now = timezone.now()
pending, running = get_process_status()
context = {
"pending": count_pending,
"running": count_running
}
return render(request, "partial_load.html", context)
full_page.html
<html>
<div> Something </div>
.....
<div id="status">
{% include "partial_load.html" %}
</div>
......
<div> some more stuff </div>
</html>
If you migrated the database changes after the installation of the background_task package (Then only the database have background_task package's tables).
You can get the status of the processes running in background by simply querying the background_task models like querying other user defined models.
from background_task.models import Task, CompletedTask
from django.utils import timezone
def get_process_status(parameters):
now = timezone.now()
# pending tasks will have `run_at` column greater than current time.
# Similar for running tasks, it shall be
# greater than or equal to `locked_at` column.
# Running tasks won't work with SQLite DB,
# because of concurrency issues in SQLite.
# If your task still not started running and waiting in the queue, then you can find it in pending_tasks
pending_tasks = Task.objects.filter(run_at__gt=now)
# If your your task is in running state, you can find it in running_tasks
running_tasks = Task.objects.filter(locked_at__gte=now)
# Completed tasks goes in `CompletedTask` model.
# I have picked all, you can choose to filter based on what you want.
# If your task completed you can find it in Completed task model.
completed_tasks = CompletedTask.objects.all()
# If you want the result in json format just add .values() at the end of the
# ORM query like "completed_tasks = CompletedTask.objects.all().values()"
print(completed_tasks, running_tasks, pending_tasks)
......
......
return process_status
If you want to run the function for every 60 seconds, schedule the task using background_task.
Sample code:
#background(schedule=60)
def get_process_status(parameters):
.....
code
.....
return process_status
Hope it will help you.
I developed a flask app which runs a mathematical optimization script (PuLP+ Python) for user provided data input. Since the computation takes quite a while, I want to display the optimizer's print statements constantly (without refreshing) on my webpage.
Based on this solution I managed to run a subprocesswith my .py script. Within this script, the print() statement does what it should, however I can only see the output in the command line. By using flash() I managed to display the output, yet it gets rendered only once the computation has finished and the page is reloaded. I try to output the print() statements in realtime within my HTML. Is there a way to achieve this?
Thanks!
Excerpt from routes.py:
#app.route('/optimize', methods=['GET', 'POST'])
def solve():
path = os.path.join(app.root_path, 'optimization', 'assets')
file = "test.py"
execute(file, path)
return redirect(url_for('home'))
The execute(file,path) function:
import os
import subprocess as sub
def execute(command, directory):
# change directory to execute the command
os.chdir(directory)
proc = sub.Popen(['python','-u', command], stdout=sub.PIPE, universal_newlines=True) # Python 2: bufsize=1
for stdout_line in proc.stdout:
print(stdout_line.rstrip() + '\n')
flash(stdout_line.rstrip(), 'python_print')
proc.stdout.close()
return_code = proc.wait()
if return_code:
raise sub.CalledProcessError(return_code, command)
And finally my HTML which is rendered in the home route:
{% with prints = get_flashed_messages(category_filter=["python_print"]) %}
{% if prints %}
<div class="content-section">
<h3>Solver Status</h3>
<ul>
{%- for message in prints %}
<li>{{ message }}</li>
{% endfor -%}
</ul>
</div>
{% endif %}
{% endwith %}
As far as I know this is how message flashing is supposed to work, i.e. it will display all the messages for a particular end point all at once.
I am not sure about the exact implementation since I have not looked at the source code for flash() and get_flash_messages() but this is my understanding of what might be happening in the background, every message you flash with flash function in your python code it gets appended to an iterable, and when you call the get_flashed_messages() it returns that iterable back, which you can loop over to get the messages.
Though your execute() function is adding messages to that iterable before the process has been completed, but notice the execute function is being called inside the view solve and the actual redirect can happen only after the execute function has completed its execution, and that's when the template home will gets the iterable having all the flashed messages.
Hope that makes sense.
Based on the suggestions provided I changed my approach and got it running. Instead of using subprocess, the solver is imported as module. I'm generating a .txt using the python logging module rather than trying to catch the print statements. This file is streamed constantly using flask response + yield and parsed via AJAX. The getter is called in jQuery via setInterval (here: 1Hz) and added to my HTML. I will post some snippets below for anyone dealing with the same issue.
Thanks for your support!
Flask optimize route:
import supplierAllocation.optimization.optimizer as optimizer
#app.route('/optimize')
def optimize():
if session.get('input_file') is not None:
# Get input_file
input_file = session.get('input_file')
# Path to logfile
log_file = os.path.join(
app.config['LOG_FOLDER'], session.get('log_file'))
# Assign random unique hex file name from session to output file
out_file_fn = session.get('session_ID') + '.xlsx'
output_file = os.path.join(app.config['DOWNLOAD_FOLDER'], out_file_fn)
# Search for outdated output; if present: delete
if session.get('output_file') is None:
session['output_file'] = out_file_fn
else:
session_output = os.path.join(
app.config['DOWNLOAD_FOLDER'], session.get('output_file'))
silent_remove(session_output)
# Pass input and output parameter to solver
optimizer.launch_solver(input_file, output_file, log_file)
else:
flash("Please upload your data before launching the solver!", 'warning')
return redirect(url_for('home'))
Flask streaming route:
#app.route('/streamLog')
def stream_logfile():
if session.get('log_file') is None:
return Response(status=204)
else:
session_log = os.path.join(
app.config['LOG_FOLDER'], session.get('log_file'))
def stream():
with open(session_log, "rb") as file:
yield file.read()
return Response(stream(), mimetype='text/plain')
Call of AJAX request on button click:
// Print solver output with 1Hz
var session_log = setInterval(get_Log, 1000);
AJAX request in jQuery:
// AJAX Request to download solutions
function get_Log() {
console.log('Sending AJAX request...');
$.ajax({
url: "streamLog",
dataType: "text",
success: function(result) {
$("#code-wrapper").html(result);
},
error: function() {
console.log("ERROR! Logfile could not be retrieved.");
}
});
}
I've been reading the book 'Head First Python' where the writer talks about creating dynamic webpages using a module he created called 'yate', an HTML template engine (which I renamed to site_yate in the code below). The example he works through is a hypothetical coach wanting his athletes to be able to check their times online. The design is as follows: first you enter the homepage which has a link to run a script which generates a webpage where you can select the athlete whose times you want to view. Then when you select your athlete and click submit the form calls another script called "site_generate_timing_data.py" where you can views the athlete's top times. So I decided to take it further and add functionality to add a time for the athlete, using this extra line of code in my python script.
print(site_yate.do_form("addtime.py", [athlete_id]))
The HTML this will generate will be this:
<form action="addtime.py" method="POST">
<h1>Want to add a time?</h1>
<input type="Text" name="1" size=40> //"1" is the athlete's id in this example
<input type="Submit" value="Submit">
</form>
As you can see this code calls the script 'addtime.py' which has the following code:
import cgi
import sqlite3
data = cgi.FieldStorage().value[0] #this attribute will be in the form MininFieldStorage(name, value)
id = data.name #this attribute is the input's name i.e. athlete's id
time = data.value #this attribute is the input's value i.e. the time
connection = sqlite3.connect("NUACDB.sqlite") #my DB's name
cursor = connection.cursor()
cursor.execute("""INSERT INTO timing_data (athlete_id, time)
VALUES (?, ?)""",
(id, time)) #just SQL stuff
connection.commit()
connection.close()
Which works fine, however I want to change a few thing about this, since it leaves the user on a blank page. I could generate some HTML code to provide links to the homepage etc. or even JavaScript code to redirect the user automatically, but I want to keep this script HTML-free so that I can also use it elsewhere.
What I want to do instead is make the script execute on the same page. Not only that, but I would also prefer if I could put the addtime.py code as a function in another module called 'athletemodel.py' and call it form there, i.e. athletemodel.addtime() (or I could do from athletemodel import addtime so I can call the function directly). How can I call a python function using HTML code? I'm aware of the onsubmit="" form attribute but apparently that is for JavaScript functions. Another thing I'm unsure about is whether the data submitted in the form will still be accessible through CGI FieldStorage and hence whether my addtime.py code will still work as it is.
This stuff is so confusing! All help is appreciated.
Not sure if you already had it in mind, but I would use ajax (remember to include the jQuery library). Here's a rough example to get you started if this is what you want. It'll keep them on the same page:
JavaScript file:
$('#submitButtonId').click(function (event) {
event.preventDefault();
$('#submitButtonId').hide();
$('#thinking').show(); //some div with a nice ajax loader gif...
$.ajax({
type: 'POST',
data: $('#formId').serialize(),
url: '/URL_path_to_function',
success: function (data) {
$('#loading').hide();
var response = data.result //now do stuff with your response
}
error: function(error){
console.log('Error')}
});
Python view/function:
import jsonify
if request.method == 'POST':
value = request.form['input value'] #flask...
#Do stuff
return jsonify(result='Some response')
So, I'm trying to make a simple call using jQuery .getJSON to my local web server using python/django to serve up its requests. The address being used is:
http://localhost:8000/api/0.1/tonight-mobile.json?callback=jsonp1290277462296
I'm trying to write a simple web view that can access this url and return a JSON packet as the result (worried about actual element values/layout later).
Here's my simple attempt at just alerting/returning the data:
$.getJSON("http://localhost:8000/api/0.1/tonight-mobile.json&callback=?",
function(json){
alert(json);
<!--$.each(json.items, function(i,item){
});-->
});
I am able to access this URL directly, either at http://localhost:8000/api/0.1/tonight-mobile.json or http://localhost:8000/api/0.1/tonight-mobile.json&callback=jsonp1290277462296 and get back a valid JSON packet... So I'm assuming it's in my noob javascript:)
My views.py function that is generating this response looks as follows:
def tonight_mobile(request):
callback = request.GET.get('callback=?', '')
def with_rank(rank, place):
return (rank > 0)
place_data = dict(
Places = [make_mobile_place_dict(request, p) for p in Place.objects.all()]
)
xml_bytes = json.dumps(place_data)
xml_bytes = callback + '(' + xml_bytes + ');'
return HttpResponse(xml_bytes, mimetype="application/json")
With corresponding urls.py configuration:
(r'^tonight-mobile.json','iphone_api.views.tonight_mobile'),
I am still somewhat confused on how to use callbacks, so maybe that is where my issue lies. Note I am able to call directly a 'blah.json' file that is giving me a response, but not through a wired URL. Could someone assist me with some direction?
First, callback = request.GET.get('callback=?', '') won't get you the value of callback.
callback = request.GET.get( 'callback', None )
Works much better.
To debug this kind of thing. You might want to include print statements in your Django view function so you can see what's going on. For example: print repr(request.GET) is a helpful thing to put in a view function so that you can see the GET dictionary.
I have some JavaScript from a 3rd party vendor that is initiating an image request. I would like to figure out the URI of this image request.
I can load the page in my browser, and then monitor "Live HTTP Headers" or "Tamper Data" in order to figure out the image request URI, but I would prefer to create a command line process to do this.
My intuition is that it might be possible using python + qtwebkit, but perhaps there is a better way.
To clarify: I might have this (overly simplified code).
<script>
suffix = magicNumberFunctionIDontHaveAccessTo();
url = "http://foobar.com/function?parameter=" + suffix
img = document.createElement('img'); img.src=url; document.all.body.appendChild(img);
</script>
Then once the page is loaded, I can go figure out the url by sniffing the packets. But I can't just figure it out from the source, because I can't predict the outcome of magicNumberFunction...().
Any help would be muchly appreciated!
Thank you.
The simplest thing to do might be to use something like HtmlUnit and skip a real browser entirely. By using Rhino, it can evaluate JavaScript and likely be used to extract that URL out.
That said, if you can't get that working, try out Selenium RC and use the captureNetworkTraffic command (which requires the Selenium instant be started with an option of captureNetworkTraffic=true). This will launch Firefox with a proxy configured and then let you pull the request info back out as JSON/XML/plain text. Then you can parse that content and get what you want.
Try out the instant test tool that my company offers. If the data you're looking for is in our results (after you click View Details), you'll be able to get it from Selenium. I know, since I wrote the captureNetworkTraffic API for Selenium for my company, BrowserMob.
I would pick any one of the many http proxy servers written in Python -- probably one of the simplest ones at the very top of the list -- and tweak it to record all URLs requested (as well as proxy-serve them) e.g. appending them to a text file -- without loss of generality, call that text file 'XXX.txt'.
Now all you need is a script that: starts the proxy server in question; starts Firefox (or whatever) on your main desired URL with the proxy in question set as your proxy (see e.g. this SO question for how), though I'm sure other browsers would work just as well; waits a bit (e.g. until the proxy's XXX.txt file has not been altered for more than N seconds); reads XXX.txt to extract only the URLs you care about and record them wherever you wish; turns down the proxy and Firefox processes.
I think this will be much faster to put in place and make work correctly, for your specific requirements, than any more general solution based on qtwebkit, selenium, or other "automation kits".
Use Firebug Firefox plugin. It will show you all requests in real time and you can even debug the JS in your Browser or run it step-by-step.
Ultimately, I did it in python, using Selenium-RC. This solution requires the python files for selenium-rc, and you need to start the java server ("java -jar selenium-server.jar")
from selenium import selenium
import unittest
import lxml.html
class TestMyDomain(unittest.TestCase):
def setUp(self):
self.selenium = selenium("localhost", \
4444, "*firefox", "http://www.MyDomain.com")
self.selenium.start()
def test_mydomain(self):
htmldoc = open('site-list.html').read()
url_list = [link for (element, attribute,link,pos) in lxml.html.iterlinks(htmldoc)]
for url in url_list:
try:
sel = self.selenium
sel.open(url)
sel.select_window("null")
js_code = '''
myDomainWindow = this.browserbot.getUserWindow();
for(obj in myDomainWindow) {
/* This code grabs the OMNITURE tracking pixel img */
if ((obj.substring(0,4) == 's_i_') && (myDomainWindow[obj].src)) {
var ret = myDomainWindow[obj].src;
}
}
ret;
'''
omniture_url = sel.get_eval(js_code) #parse&process this however you want
except Exception, e:
print 'We ran into an error: %s' % (e,)
self.assertEqual("expectedValue", observedValue)
def tearDown(self):
self.selenium.stop()
if __name__ == "__main__":
unittest.main()
Why can't you just read suffix, or url for that matter? Is the image loaded in an iframe or in your page?
If it is loaded in your page, then this may be a dirty hack (substitute document.body for whatever element is considered):
var ac = document.body.appendChild;
var sources = [];
document.body.appendChild = function(child) {
if (/^img$/i.test(child.tagName)) {
sources.push(child.getAttribute('src'));
}
ac(child);
}