I have a very small project in Django where I get fx_rates through a Python script (details below). These are used to convert amounts in different currencies to GBP. I would like to know how I can just have a button on the website that allows to refresh this query instead of having to run it manually in the IDE. Can I create a view for this? How would I show this in a template? Thanks
fx_rates.py
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE','fx_rates_project.settings')
import django
django.setup()
from fx_rates_app.models import fx_table
import pandas_datareader.data as web
from datetime import datetime
os.environ["ALPHAVANTAGE_API_KEY"] = '#########'
fx_gbp_to_eur = web.DataReader("GBP/EUR","av-forex")
eur = float(fx_gbp_to_eur[4:5].values[0][0])
fx_gbp_to_aud = web.DataReader("GBP/AUD","av-forex")
aud = float(fx_gbp_to_aud[4:5].values[0][0])
fx_gbp_to_usd = web.DataReader("GBP/USD","av-forex")
usd = float(fx_gbp_to_usd[4:5].values[0][0])
from datetime import datetime
now = datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
webpg1 = fx_table.objects.get_or_create(eur_to_gbp=eur,aud_to_gbp=aud,usd_to_gbp=usd,date_time=dt_string)[0]
In the template I included:
<form method="post">
{% csrf_token %}
<button type="submit" name="run_script">Refresh</button>
</form>
However, I don't think the best way to do this would be to copy the all script in views with request.POST. Is there another way to leave the script in another file and just create a view to run it (e.g. if request.method == 'POST' run fx_rates.py)?
This can be definitely done by creating a function update_fx to wrap your script and then call that function in your view.
Now for the tricky part.
Do you need the values to render them or just to update them?
Rendering will be easy, just return them from your function and you will
be able to use them in your template.
This though includes the waiting time to fetch these values. When your view will be executed you will have to wait for the function to fetch the values.
The alternative is to "trigger" a second job that updates the values from your
view. This pattern is called a task queue and there are many solutions that you can leverage already, depending on how complicated things should be (what happens if the job has an error? how many times should we retry? ...).
This also has the unfortunate effect that now you run a second python script somewhere that listens for new tasks and executes them.
As always your choice is based on tradeoffs, personally if the server has good internet connection and the request is completed fast I would not mind putting it
in my view as is.
Hope this is helpful.
Let's see an example
# update.py
from fx_rates_app.models import fx_table
import pandas_datareader.data as web
from django.conf import settings
from datetime import datetime
import os
def get_price():
os.environ["ALPHAVANTAGE_API_KEY"] = '#########'
fx_gbp_to_eur = web.DataReader("GBP/EUR","av-forex") # here we are reading stuff from the network I guess
eur = float(fx_gbp_to_eur[4:5].values[0][0])
fx_gbp_to_aud = web.DataReader("GBP/AUD","av-forex")
aud = float(fx_gbp_to_aud[4:5].values[0][0])
fx_gbp_to_usd = web.DataReader("GBP/USD","av-forex")
usd = float(fx_gbp_to_usd[4:5].values[0][0])
now = datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
# here you have two choices,
# either you return the raw values
# or you put them into a model
return {
'timestamp': dt_string,
'value': 1,
'currency': 'gbp',
'eur': eur,
'aud': aud,
'us': us,
}
webpg1 = fx_table.objects.get_or_create(eur_to_gbp=eur,aud_to_gbp=aud,usd_to_gbp=usd,date_time=dt_string)[0]
return webpg1
# views.py
from django.http import HttpResponse
from update import get_price
def refresh_from_values(request):
data = get_price() # here we don't have a model, just data
webpg1 = fx_table.objects.get_or_create(
eur_to_gbp=data['eur'],
aud_to_gbp=data['aud'],
usd_to_gbp=data['usd'],
date_time=data['timestamp'],
)[0]
# here your render the model using the template
def refresh_from_model(request):
fx = get_price() # here we get back the model instead
# you can render again the model as you can refresh the page
In the future you might be interested in running this thing continuously
and then you would have to move to an event driven app which is more compilicated but can be very satisfying to get right. In django everything
would start with signals and more likely you would have to write some javascript to
update part of the page without refreshing it.
Related
I have recently started to learn and work on Django i came across this error multiple times sometimes it was due to repetitive use of a name in an html and sometimes it was due the part for example "variable_name=request.POST['variable_name']"
I visited various sites regarding this error and was able to find a solution i.e, once i have typed in "variable_name=request.POST['variable_name']" in a function in views.py
the next time i tried to POST the data again "variable_name=request.POST['variable_name']" I had to do something like this "variable_name=request.POST.get('variable-name', 'blank_or_whatever_canBeFalseAlso')
The Above statement helped me solve that error I got this solution from A Thread in StackOverflow itself Quite a useful site.
Now for the Question am just Curious to know whats the difference between
Variable_Name=request.POST['Variable_Name']
Variable_Name=request.POST.get('Variable_Name','Whatever')
Code (views.py) Is given Below:
from django.shortcuts import render
from . import views
from datetime import date
from .models import Batch, Deleted_Batch
from django.utils.datastructures import MultiValueDictKeyError
# Create your views here.
today = date.today()
Date_Batch = str(today.day)+'/'+str(today.month)+'/'+str(today.year)
def Reg_Batch(request):
return render(request, 'Reg_Batch.html',{'Date_Batch':Date_Batch})
def Reg_Batch_Receive(request):
Region_Code=request.POST['Region_Code']
Branch_Code=request.POST['Branch_Code']
Batch_Number=request.POST['Batch_Number']
Farm_Code=request.POST['Farm_Code']
Farm_Status=request.POST['Farm_Status']
Farmer_Name=request.POST['Farmer_Name']
Farmer_Address=request.POST['Farmer_Address']
Farmer_Phone_Number=request.POST['Farmer_Phone_Number']
Farmer_Email=request.POST['Farmer_Email']
return render(request, 'Reg_Batch_Receive.html')
def Reg_Batch_View(request):
Region_Code=request.POST.get('Region_Code', 'Blank')
Branch_Code=request.POST.get('Branch_Code', 'Blank')
Farm_Code=request.POST.get('Farm_Code', 'Blank')
Farm_Status=request.POST.get('Farm_Status', 'Blank')
return render(request, 'Reg_Batch_View.html',{'Region_Code':Region_Code,'Branch_Code':Branch_Code,'Farm_Code':Farm_Code,'Farm_Status':Farm_Status})
def Reg_Batch_View_Receive(request):
Date_Batch= '20/08/2000'
Region_Code='586001'
Branch_Code='586001'
Batch_Number='586001'
Farm_Code='5484712'
Farm_Status='Active'
Farmer_Name='Joel Liao'
Farmer_Address='jojojojojojojo'
Farmer_Phone_Number='8088078086'
Farmer_Email='kuroneko#gmail.com'
return render(request,'Reg_Batch_View_Receive.html',{'Date_Batch':Date_Batch,'Region_Code':Region_Code,'Branch_Code':Branch_Code,'Batch_Number':Batch_Number,'Farm_Code':Farm_Code,'Farm_Status':Farm_Status,'Farmer_Name':Farmer_Name,'Farmer_Address':Farmer_Address,'Farmer_Phone_Number':Farmer_Phone_Number,'Farmer_Email':Farmer_Email})
def Reg_Batch_Delete(request):
Region_Code='586001'
Branch_Code='586001'
Batch_Number='586001'
Farm_Code='5484712'
Farm_Status='Active'
Farmer_Name='Joel Liao'
Farmer_Address='jojojojojojojo'
Farmer_Phone_Number='8088078086'
Farmer_Email='kuroneko#gmail.com'
Deleted_Batch_DataBase=Deleted_Batch({'Region_Code':Region_Code,'Branch_Code':Branch_Code,'Batch_Number':Batch_Number,'Farm_Code':Farm_Code,'Farm_Status':Farm_Status,'Farmer_Name':Farmer_Name,'Farmer_Address':Farmer_Address,'Farmer_Phone_Number':Farmer_Phone_Number,'Farmer_Email':Farmer_Email}),
return render(request,'Reg_Batch_Delete.html',{'Batch_Number':Batch_Number})
def Reg_Batch_Delete_Receive(request):
Batch_Number='586001'
return render(request,'Reg_Batch_Delete_Receive.html',{'Batch_Number':Batch_Number})
request.POST is a dictionary-like object (see docs).
So if you do
request.POST['foobar']
you can access the posted variable foobar. But if this is not being posted and therefore not set inside POST, you will get the MultiValueDictKeyError.
This is similar to access a key inside a default dictionary which is not set, e.g.
d = {}
d['foobar'] # raises Exception
In order to avoid this error, you can then do
request.POST.get('foobar', 'default-value')
in order to get the posted field and a default if it is not present.
So if foobar isn't set inside your POST-data, the received value is default-value.
This is similar to the default dictionary handling:
d = {}
d.get('foobar', 'random') # returns 'random'
I have a django project which contains a long running process. I have used django-background-tasks library for this. It works but I want to create a pending page for users and display the status of the task. I should refresh that page every 60 seconds and update the status. How can I do that?
Thank you.
Hope you know about Ajax.
How to use Ajax with Django: https://simpleisbetterthancomplex.com/tutorial/2016/08/29/how-to-work-with-ajax-request-with-django.html
How to run Ajax code every n seconds: https://thisinterestsme.com/ajax-request-every-10-seconds/
If you want to load a page partially, then you should split the page into two parts. One should contain the particular div which you want to refresh or reload, consider the page name as partial_load.html. And other page can have remaining codes consider the file name as full_page.html. The partial_load.html can be included inside the full_page.html by using include tag ({% include "partial_load.html" %})
def refresh_page(request):
if request.is_ajax():
now = timezone.now()
pending, running = get_process_status()
context = {
"pending": count_pending,
"running": count_running
}
return render(request, "partial_load.html", context)
full_page.html
<html>
<div> Something </div>
.....
<div id="status">
{% include "partial_load.html" %}
</div>
......
<div> some more stuff </div>
</html>
If you migrated the database changes after the installation of the background_task package (Then only the database have background_task package's tables).
You can get the status of the processes running in background by simply querying the background_task models like querying other user defined models.
from background_task.models import Task, CompletedTask
from django.utils import timezone
def get_process_status(parameters):
now = timezone.now()
# pending tasks will have `run_at` column greater than current time.
# Similar for running tasks, it shall be
# greater than or equal to `locked_at` column.
# Running tasks won't work with SQLite DB,
# because of concurrency issues in SQLite.
# If your task still not started running and waiting in the queue, then you can find it in pending_tasks
pending_tasks = Task.objects.filter(run_at__gt=now)
# If your your task is in running state, you can find it in running_tasks
running_tasks = Task.objects.filter(locked_at__gte=now)
# Completed tasks goes in `CompletedTask` model.
# I have picked all, you can choose to filter based on what you want.
# If your task completed you can find it in Completed task model.
completed_tasks = CompletedTask.objects.all()
# If you want the result in json format just add .values() at the end of the
# ORM query like "completed_tasks = CompletedTask.objects.all().values()"
print(completed_tasks, running_tasks, pending_tasks)
......
......
return process_status
If you want to run the function for every 60 seconds, schedule the task using background_task.
Sample code:
#background(schedule=60)
def get_process_status(parameters):
.....
code
.....
return process_status
Hope it will help you.
I have a flask application, which I want to use to do some data visualization of a very large dataset that I have in a postgresql database.
I have sqlalchemy, plotly, google maps set up, but my current problem is that I want to fetch a rather large dataset from my database and pass that to a template that will take care of doing some different visualizations. I only want to load the data in once, when the site is started, and then avoid running it later on when navigating the other pages.
which methods can I use to achieve this? And how can I access the data that I get in other pages (templates)?
I read somewhere about session and I tried to implement such a functionality, but then I got a "not serializable error", which I fixed by parsing my object to JSON, but then I got a "too large header" error. I feel like I'm sort of out of possiblities..
Just to show some code, here is what I'm currently messing around with.
Models.py:
from sqlalchemy import BigInteger, Column, JSON, Text
from app import db
class Table910(db.Model):
__tablename__ = 'table_910'
id = db.Column(BigInteger, primary_key=True)
did = db.Column(Text)
timestamp = db.Column(BigInteger)
data = db.Column(JSON)
db_timestamp = db.Column(BigInteger)
def items(self):
return {'id': self.id, 'did': self.did, 'timestamp': self.timestamp, 'data': self.data, 'db_timestamp': self.db_timestamp}
def __repr__(self):
return '<1_Data %r, %r>' % (self.did, self.id)
Views.py
from flask import render_template, session
from app import app, db, models
from datetime import datetime
import matplotlib.dates as mdates
import matplotlib.pyplot as plt
import numpy as np
import json
import plotly
#app.route('/')
#app.route('/index')
def index():
# fetching from the database (takes ~50 seconds to run)
rows = models.Table910.query \
.filter_by(did='357139052424715') \
.filter((models.Table910.timestamp > 1466846920000) & (models.Table910.timestamp < 1467017760000)) \
.order_by(models.Table910.timestamp.asc())
#
# managing and drawing data on plots
#
return render_template('index.html',
ids=ids,
graphJSON=graphJSON,
rows=rows,
routeCoordinates=routeCoordinates))
EDIT:
I might have explained my problem a little too vaguely, therefore to elaborate a little I want to use a specific page to load the data, and then have another page take care of drawing the graphs. The best procedure I can think of currently would be to be able to press a button to get the data and once data is loaded press another button to draw the graphs.. How can I achieve this?
I've been reading the book 'Head First Python' where the writer talks about creating dynamic webpages using a module he created called 'yate', an HTML template engine (which I renamed to site_yate in the code below). The example he works through is a hypothetical coach wanting his athletes to be able to check their times online. The design is as follows: first you enter the homepage which has a link to run a script which generates a webpage where you can select the athlete whose times you want to view. Then when you select your athlete and click submit the form calls another script called "site_generate_timing_data.py" where you can views the athlete's top times. So I decided to take it further and add functionality to add a time for the athlete, using this extra line of code in my python script.
print(site_yate.do_form("addtime.py", [athlete_id]))
The HTML this will generate will be this:
<form action="addtime.py" method="POST">
<h1>Want to add a time?</h1>
<input type="Text" name="1" size=40> //"1" is the athlete's id in this example
<input type="Submit" value="Submit">
</form>
As you can see this code calls the script 'addtime.py' which has the following code:
import cgi
import sqlite3
data = cgi.FieldStorage().value[0] #this attribute will be in the form MininFieldStorage(name, value)
id = data.name #this attribute is the input's name i.e. athlete's id
time = data.value #this attribute is the input's value i.e. the time
connection = sqlite3.connect("NUACDB.sqlite") #my DB's name
cursor = connection.cursor()
cursor.execute("""INSERT INTO timing_data (athlete_id, time)
VALUES (?, ?)""",
(id, time)) #just SQL stuff
connection.commit()
connection.close()
Which works fine, however I want to change a few thing about this, since it leaves the user on a blank page. I could generate some HTML code to provide links to the homepage etc. or even JavaScript code to redirect the user automatically, but I want to keep this script HTML-free so that I can also use it elsewhere.
What I want to do instead is make the script execute on the same page. Not only that, but I would also prefer if I could put the addtime.py code as a function in another module called 'athletemodel.py' and call it form there, i.e. athletemodel.addtime() (or I could do from athletemodel import addtime so I can call the function directly). How can I call a python function using HTML code? I'm aware of the onsubmit="" form attribute but apparently that is for JavaScript functions. Another thing I'm unsure about is whether the data submitted in the form will still be accessible through CGI FieldStorage and hence whether my addtime.py code will still work as it is.
This stuff is so confusing! All help is appreciated.
Not sure if you already had it in mind, but I would use ajax (remember to include the jQuery library). Here's a rough example to get you started if this is what you want. It'll keep them on the same page:
JavaScript file:
$('#submitButtonId').click(function (event) {
event.preventDefault();
$('#submitButtonId').hide();
$('#thinking').show(); //some div with a nice ajax loader gif...
$.ajax({
type: 'POST',
data: $('#formId').serialize(),
url: '/URL_path_to_function',
success: function (data) {
$('#loading').hide();
var response = data.result //now do stuff with your response
}
error: function(error){
console.log('Error')}
});
Python view/function:
import jsonify
if request.method == 'POST':
value = request.form['input value'] #flask...
#Do stuff
return jsonify(result='Some response')
I want to generate for my users temporary download link.
Is that ok if i use django to generate link using url patterns?
Could it be correct way to do that. Because can happen that I don't understand some processes how it works. And it will overflow my memory or something else. Some kind of example or tools will be appreciated. Some nginx, apache modules probably?
So, what i wanna to achieve is to make url pattern which depend on user and time. Decript it end return in view a file.
A simple scheme might be to use a hash digest of username and timestamp:
from datetime import datetime
from hashlib import sha1
user = 'bob'
time = datetime.now().isoformat()
plain = user + '\0' + time
token = sha1(plain)
print token.hexdigest()
"1e2c5078bd0de12a79d1a49255a9bff9737aa4a4"
Next you store that token in a memcache with an expiration time. This way any of your webservers can reach it and the token will auto-expire. Finally add a Django url handler for '^download/.+' where the controller just looks up that token in the memcache to determine if the token is valid. You can even store the filename to be downloaded as the token's value in memcache.
Yes it would be ok to allow django to generate the urls. This being exclusive from handling the urls, with urls.py. Typically you don't want django to handle the serving of files see the static file docs[1] about this, so get the notion of using url patterns out of your head.
What you might want to do is generate a random key using a hash, like md5/sha1. Store the file and the key, datetime it's added in the database, create the download directory in a root directory that's available from your webserver like apache or nginx... suggest nginx), Since it's temporary, you'll want to add a cron job that checks if the time since the url was generated has expired, cleans up the file and removes the db entry. This should be a django command for manage.py
Please note this is example code written just for this and not tested! It may not work the way you were planning on achieving this goal, but it works. If you want the dl to be pw protected also, then look into httpbasic auth. you can generate and remove entries on the fly in a httpd.auth file using htpasswd and the subprocess module when you create the link or at registration time.
import hashlib, random, datetime, os, shutil
# model to hold link info. has these fields: key (charfield), filepath (filepathfield)
# datetime (datetimefield), url (charfield), orgpath (filepathfield of the orignal path
# or a foreignkey to the files model.
from models import MyDlLink
# settings.py for the app
from myapp import settings as myapp_settings
# full path and name of file to dl.
def genUrl(filepath):
# create a onetime salt for randomness
salt = ''.join(['{0}'.format(random.randrange(10) for i in range(10)])
key = hashlib('{0}{1}'.format(salt, filepath).hexdigest()
newpath = os.path.join(myapp_settings.DL_ROOT, key)
shutil.copy2(fname, newpath)
newlink = MyDlink()
newlink.key = key
newlink.date = datetime.datetime.now()
newlink.orgpath = filepath
newlink.newpath = newpath
newlink.url = "{0}/{1}/{2}".format(myapp_settings.DL_URL, key, os.path.basename(fname))
newlink.save()
return newlink
# in commands
def check_url_expired():
maxage = datetime.timedelta(days=7)
now = datetime.datetime.now()
for link in MyDlink.objects.all():
if(now - link.date) > maxage:
os.path.remove(link.newpath)
link.delete()
[1] http://docs.djangoproject.com/en/1.2/howto/static-files/
It sounds like you are suggesting using some kind of dynamic url conf.
Why not forget your concerns by simplifying and setting up a single url that captures a large encoded string that depends on user/time?
(r'^download/(?P<encrypted_id>(.*)/$', 'download_file'), # use your own regexp
def download_file(request, encrypted_id):
decrypted = decrypt(encrypted_id)
_file = get_file(decrypted)
return _file
A lot of sites just use a get param too.
www.example.com/download_file/?09248903483o8a908423028a0df8032
If you are concerned about performance, look at the answers in this post: Having Django serve downloadable files
Where the use of the apache x-sendfile module is highlighted.
Another alternative is to simply redirect to the static file served by whatever means from django.