We Keep Coding

Run Python script in Django template through a button

I have a very small project in Django where I get fx_rates through a Python script (details below). These are used to convert amounts in different currencies to GBP. I would like to know how I can just have a button on the website that allows to refresh this query instead of having to run it manually in the IDE. Can I create a view for this? How would I show this in a template? Thanks
fx_rates.py
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE','fx_rates_project.settings')
import django
django.setup()
from fx_rates_app.models import fx_table
import pandas_datareader.data as web
from datetime import datetime
os.environ["ALPHAVANTAGE_API_KEY"] = '#########'
fx_gbp_to_eur = web.DataReader("GBP/EUR","av-forex")
eur = float(fx_gbp_to_eur[4:5].values[0][0])
fx_gbp_to_aud = web.DataReader("GBP/AUD","av-forex")
aud = float(fx_gbp_to_aud[4:5].values[0][0])
fx_gbp_to_usd = web.DataReader("GBP/USD","av-forex")
usd = float(fx_gbp_to_usd[4:5].values[0][0])
from datetime import datetime
now = datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
webpg1 = fx_table.objects.get_or_create(eur_to_gbp=eur,aud_to_gbp=aud,usd_to_gbp=usd,date_time=dt_string)[0]
In the template I included:
<form method="post">
{% csrf_token %}
<button type="submit" name="run_script">Refresh</button>
</form>
However, I don't think the best way to do this would be to copy the all script in views with request.POST. Is there another way to leave the script in another file and just create a view to run it (e.g. if request.method == 'POST' run fx_rates.py)?

This can be definitely done by creating a function update_fx to wrap your script and then call that function in your view.
Now for the tricky part.
Do you need the values to render them or just to update them?
Rendering will be easy, just return them from your function and you will
be able to use them in your template.
This though includes the waiting time to fetch these values. When your view will be executed you will have to wait for the function to fetch the values.
The alternative is to "trigger" a second job that updates the values from your
view. This pattern is called a task queue and there are many solutions that you can leverage already, depending on how complicated things should be (what happens if the job has an error? how many times should we retry? ...).
This also has the unfortunate effect that now you run a second python script somewhere that listens for new tasks and executes them.
As always your choice is based on tradeoffs, personally if the server has good internet connection and the request is completed fast I would not mind putting it
in my view as is.
Hope this is helpful.
Let's see an example
# update.py
from fx_rates_app.models import fx_table
import pandas_datareader.data as web
from django.conf import settings
from datetime import datetime
import os
def get_price():
os.environ["ALPHAVANTAGE_API_KEY"] = '#########'
fx_gbp_to_eur = web.DataReader("GBP/EUR","av-forex") # here we are reading stuff from the network I guess
eur = float(fx_gbp_to_eur[4:5].values[0][0])
fx_gbp_to_aud = web.DataReader("GBP/AUD","av-forex")
aud = float(fx_gbp_to_aud[4:5].values[0][0])
fx_gbp_to_usd = web.DataReader("GBP/USD","av-forex")
usd = float(fx_gbp_to_usd[4:5].values[0][0])
now = datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
# here you have two choices,
# either you return the raw values
# or you put them into a model
return {
'timestamp': dt_string,
'value': 1,
'currency': 'gbp',
'eur': eur,
'aud': aud,
'us': us,
}
webpg1 = fx_table.objects.get_or_create(eur_to_gbp=eur,aud_to_gbp=aud,usd_to_gbp=usd,date_time=dt_string)[0]
return webpg1
# views.py
from django.http import HttpResponse
from update import get_price
def refresh_from_values(request):
data = get_price() # here we don't have a model, just data
webpg1 = fx_table.objects.get_or_create(
eur_to_gbp=data['eur'],
aud_to_gbp=data['aud'],
usd_to_gbp=data['usd'],
date_time=data['timestamp'],
)[0]
# here your render the model using the template
def refresh_from_model(request):
fx = get_price() # here we get back the model instead
# you can render again the model as you can refresh the page
In the future you might be interested in running this thing continuously
and then you would have to move to an event driven app which is more compilicated but can be very satisfying to get right. In django everything
would start with signals and more likely you would have to write some javascript to
update part of the page without refreshing it.

Multiple render in 1 function in views.py Django?

I am very new to Django...
Using submit button i want to run a python file in background and display content on next page...
But my python file takes some time to take out the result so in between i wanted to put a loading html page in between....
I have written some code which correctlty runs the python file but i am not able to incorporate the loading page in between...
Take a look at my function in views.py
def submit(request):
info = request.POST['info']
print('value is ', info)
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE )
return render_to_response("loading.html")
run(['python', filename, info], stdout= PIPE )
return render(request, 'python_output.html', context)
ACTUAL RESULT:
return render_to_response("loading.html")
works but then the control does not shifts to run command...
I want to run the loading html page and run the python file in background and when python file run finishes it should move to python _output.html page where output is displayed...
Expected:
Loading page should work and after that control should shift to run command and then it should go to python_output.html page.../

The return statement will terminate the execution of the function so anything below it will never be reached.
You can use Javascript to show the loading icon and then use JQuery to run a GET request in the background where you call a custom view from Django that will output the result of the command. When data is received you can then remove the icon and process the data as you want.
Basic Example :
Django
------
url(r'^command/', views.command, name='command'),
def command(request):
info = request.POST['info']
filename = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
result = run(['python', filename, info], stdout= PIPE
return result
Javascript
----------
<img id="loading-icon" src="loading.gif">
$.get("/command", function(text)
{
$("#loading-icon").remove();
process(text);
});

You need to understand the basic flow in Django
You can only add one return in your view. after the execution of the first return it goes to the response middleware so all other returns below are ignored.The loading can be done in javascript in frontend

What you want involves a bit more work than you might have expected:
Install a background task framework like celery (and a queue like Redis or RabbitMQ to store tasks) that will fetch tasks from the queue and process them.
Your initial view needs to start a background task.
You want to keep track of the task_id for this task, either by returning it in your response to the user or saving it in the session of the user. Your view responds with the HTML page saying "please have some patience..." and javascript to handle what follows (see below).
Then you need a view that can check the status of the task (based on either the task_id passed in the query or saved in the current session). It responds with JSON, either by saying "status = processing..." or "status = done" with the result of the task.
In the HTML page, you need javascript that queries that view at regular intervals, until the status is "done" and then processes the result of the task to update the HTML of the page.
Search for "django celery tutorial" and you'll find plenty of examples.

Try to use an ajax request when you load your first page (loading.html) to run the python file in background and when it is done, display the result via output.html.
Using JQuery, in the template file, you must call a function like this :
<script>
var url_file_to_run = "{% url "your_app:file_to_run_adress" 0 %}";
var url = "{% url "your_app:python_output" 0 %}";
$.ajax({
url: url_file_to_run,
}
}).done(function(data) {
$( location ).attr("href", url);
});
</script>
I hope i understand your problem.

Python flask application not displaying generated html file for second time

I have a Python flask application which takes input id's and dynamically generates data into a html file. Below is my app.py file.
#app.route('/execute', methods=['GET', 'POST'])
def execute():
if request.method == 'POST':
id = request.form['item_ids']
list = [id]
script_output = subprocess.Popen(["python", "Search_Script.py"] + list)
# script_output = subprocess.call("python Search_Script.py "+id, shell=True)
# render_template('running.html')
script_output.communicate()
#driver = webdriver.Chrome()
#driver.get("home.html")
#driver.execute_script("document.getElementById('Executed').style.display = '';")
return render_template('execute.html')
#app.route('/output')
def output():
return render_template('output.html')
output.html file has below code at the bottom.
<div class="container" style="text-align: center;">
{% include 'itemSearchDetails.html' %}
</div>
itemSearchDetails.html is generated every time dynamically based on the input. I check for different inputs and it is generating perfectly. When I run it with some input(assume 2) values for the first time, it runs perfectly and shows the output correctly. But, when I run for different values(assume 4) for the next time, the file 'itemSearchDetails.html' is generated for those 4 values but the browser only shows output for the first 2 values. No matter how many times I run it, browser shows only output with the first run values.
So, every time only the first inputted values are shown no matter how many times I run. I am not sure if it is browser cache issue since I tried "disabling cache" in chrome. Still it didn't work. Please let me know if there is something I am missing.

Try solution from this answer:
Parameter TEMPLATES_AUTO_RELOAD
Whether to check for modifications of the template source and reload
it automatically. By default the value is None which means that Flask
checks original file only in debug mode.
Original documentation could be found here.

Looks like Jinja is caching the included template.
If you don't need to interpret the HTML as a Jinja template, but instead just include its contents as-is, read the file first and pass the contents into the template:
with open('itemSearchDetails.html', 'r') as infp:
data = infp.read()
return render_template('execute.html', data=data)
...
{{ data|safe }}
(If you do need to interpret the HTML page as Jinja (as include will), you can parse a Jinja Template out of data, then use the include tag with that dynamically compiled template.)

How do you call a python function instead of a script using a form?

I've been reading the book 'Head First Python' where the writer talks about creating dynamic webpages using a module he created called 'yate', an HTML template engine (which I renamed to site_yate in the code below). The example he works through is a hypothetical coach wanting his athletes to be able to check their times online. The design is as follows: first you enter the homepage which has a link to run a script which generates a webpage where you can select the athlete whose times you want to view. Then when you select your athlete and click submit the form calls another script called "site_generate_timing_data.py" where you can views the athlete's top times. So I decided to take it further and add functionality to add a time for the athlete, using this extra line of code in my python script.
print(site_yate.do_form("addtime.py", [athlete_id]))
The HTML this will generate will be this:
<form action="addtime.py" method="POST">
<h1>Want to add a time?</h1>
<input type="Text" name="1" size=40> //"1" is the athlete's id in this example
<input type="Submit" value="Submit">
</form>
As you can see this code calls the script 'addtime.py' which has the following code:
import cgi
import sqlite3
data = cgi.FieldStorage().value[0] #this attribute will be in the form MininFieldStorage(name, value)
id = data.name #this attribute is the input's name i.e. athlete's id
time = data.value #this attribute is the input's value i.e. the time
connection = sqlite3.connect("NUACDB.sqlite") #my DB's name
cursor = connection.cursor()
cursor.execute("""INSERT INTO timing_data (athlete_id, time)
VALUES (?, ?)""",
(id, time)) #just SQL stuff
connection.commit()
connection.close()
Which works fine, however I want to change a few thing about this, since it leaves the user on a blank page. I could generate some HTML code to provide links to the homepage etc. or even JavaScript code to redirect the user automatically, but I want to keep this script HTML-free so that I can also use it elsewhere.
What I want to do instead is make the script execute on the same page. Not only that, but I would also prefer if I could put the addtime.py code as a function in another module called 'athletemodel.py' and call it form there, i.e. athletemodel.addtime() (or I could do from athletemodel import addtime so I can call the function directly). How can I call a python function using HTML code? I'm aware of the onsubmit="" form attribute but apparently that is for JavaScript functions. Another thing I'm unsure about is whether the data submitted in the form will still be accessible through CGI FieldStorage and hence whether my addtime.py code will still work as it is.
This stuff is so confusing! All help is appreciated.

Not sure if you already had it in mind, but I would use ajax (remember to include the jQuery library). Here's a rough example to get you started if this is what you want. It'll keep them on the same page:
JavaScript file:
$('#submitButtonId').click(function (event) {
event.preventDefault();
$('#submitButtonId').hide();
$('#thinking').show(); //some div with a nice ajax loader gif...
$.ajax({
type: 'POST',
data: $('#formId').serialize(),
url: '/URL_path_to_function',
success: function (data) {
$('#loading').hide();
var response = data.result //now do stuff with your response
}
error: function(error){
console.log('Error')}
});
Python view/function:
import jsonify
if request.method == 'POST':
value = request.form['input value'] #flask...
#Do stuff
return jsonify(result='Some response')

Django: display time it took to load a page on every page

In Django, how can I return the time it took to load a page (not the date) in every page of the site, without having to write in every views.py a code similar to the following one?
start = time.time()
#model operations
loadingpagetime = time.time() - start
If using a TEMPLATE_CONTEXT_PROCESSOR is the best option.
How would I get the whole page loading time from there, instead of just getting the template loading time?
UPDATE:
As the initial question doesn't seem to be clear enough, here is an approach of what would be the Python version of what I want to do.
#!/usr/bin/env python
import cgitb; cgitb.enable()
import time
print 'Content-type: text/html\n\n'
start = time.time()
print '<html>'
print '<head>'
print '</head>'
print '<body>'
print '<div>HEADER</div>'
print '<div>'
print '<p>Welcome to my Django Webpage!</p>'
print '<p>Welcome to my Django Webpage!</p>'
print '<p>Welcome to my Django Webpage!</p>'
print '</div>'
time.sleep(3)
loadingtime = time.time() - start
print '<div>It took ',loadingtime,' seconds to load the page</div>'
print '</body>'
print '</html>'

You can create a custom middleware to log this. Here is how I create a middleware to achieve this purpose base on http://djangosnippets.org/snippets/358/ (I modified the code a bit).
Firstly, assuming your project has a name: test_project, create a file name middlewares.py, I place it in the same folder as settings.py:
from django.db import connection
from time import time
from operator import add
import re
class StatsMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
'''
In your base template, put this:
<div id="stats">
<!-- STATS: Total: %(total_time).2fs Python: %(python_time).2fs DB: %(db_time).2fs Queries: %(db_queries)d ENDSTATS -->
</div>
'''
# Uncomment the following if you want to get stats on DEBUG=True only
#if not settings.DEBUG:
# return None
# get number of db queries before we do anything
n = len(connection.queries)
# time the view
start = time()
response = view_func(request, *view_args, **view_kwargs)
total_time = time() - start
# compute the db time for the queries just run
db_queries = len(connection.queries) - n
if db_queries:
db_time = reduce(add, [float(q['time'])
for q in connection.queries[n:]])
else:
db_time = 0.0
# and backout python time
python_time = total_time - db_time
stats = {
'total_time': total_time,
'python_time': python_time,
'db_time': db_time,
'db_queries': db_queries,
}
# replace the comment if found
if response and response.content:
s = response.content
regexp = re.compile(r'(?P<cmt><!--\s*STATS:(?P<fmt>.*?)ENDSTATS\s*-->)')
match = regexp.search(s)
if match:
s = (s[:match.start('cmt')] +
match.group('fmt') % stats +
s[match.end('cmt'):])
response.content = s
return response
Secondly, modify settings.py to add your middleware:
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
# ... your existing middlewares ...
# your custom middleware here
'test_project.middlewares.StatsMiddleware',
)
Note: you have to add the full path to your middleware class like above, the format is:
<project_name>.<middleware_file_name>.<middleware_class_name>
A second note is I added this middleware to the end of the list because I just want to log the template load time alone. If you want to log the load time of templates + all middlewares, please put it in the beginning of MIDDLEWARE_CLASSES list (credits to #Symmitchry).
Back to the main topic, the next step is to modify your base.html or whatever pages you want to log load time, add this:
<div id="stats">
<!-- STATS: Total: %(total_time).2fs Python: %(python_time).2fs DB: %(db_time).2fs Queries: %(db_queries)d ENDSTATS -->
</div>
Note: you can name the <div id="stats"> and use CSS for that div however you want, but DON'T change the comment <!-- STATS: .... -->. If you want to change it, be sure that you test it against the regex pattern in the created middlewares.py.
Voila, enjoy the statistics.
EDIT:
For those who use CBVs (Class Based Views) a lot, you might have encountered the error ContentNotRenderedError with above solution. Have no fear, here is the fix in middlewares.py:
# replace the comment if found
if response:
try:
# detects TemplateResponse which are not yet rendered
if response.is_rendered:
rendered_content = response.content
else:
rendered_content = response.rendered_content
except AttributeError: # django < 1.5
rendered_content = response.content
if rendered_content:
s = rendered_content
regexp = re.compile(
r'(?P<cmt><!--\s*STATS:(?P<fmt>.*?)ENDSTATS\s*-->)'
)
match = regexp.search(s)
if match:
s = (s[:match.start('cmt')] +
match.group('fmt') % stats +
s[match.end('cmt'):])
response.content = s
return response
I got it working with Django 1.6.x, if you have problem with other version of Django, please ping me in comment section.

Geordi gives you an awesome breakdown of everything that happens in the request cycle. It's a middleware that generates a full call-tree to show you exactly what's going on and how long is spent in each function.
It looks like this:
I highly recommend it :)
Image credit: http://evzijst.bitbucket.org/pycon.in

I can be wrong but the first thing I remembered was document.ready and ajax...
I don't think it is the best solution, but it should be quite simple to implement not only in django but anywhere else.
When you process the request in a view you start timer, and when document.ready is triggered you make an ajax call to stop the timer.
But I should admit that #Hieu Nguyen is quite brilliant)

If you want to show page loading info to your visitors, the answer from Hieu Nguyen will be a great option. But let me also recommend you some good tools.
So if you need this data just for development purposes, take a look at django-debug-toolbar. It will attach a nice toolbar with useful statistics for every page, including time it took for DB queries, template rendering and so on.
Finally, if you need professional profiling tools for Djagno, New Relic might be a very good choice.

I've upgraded middleware.py. In few words, we add some unique text in base template, then replace it in Middleware process_response. This method looks little tricky, but we have generation time closer to true, and we have not problem with javascript. Javascript on client cannot use page headers on firs loaded page, only if create XHR object, then get page.
https://stackoverflow.com/a/35237632/2837890