I am trying to get a webpage to update from an API endpoint whenever that endpoint updates. Right now my webpage works if I refresh it but I want it to update automatically.
import json
import requests
from flask import Flask, render_template
app = Flask(__name__)
s = requests.session()
# API Key
s.headers.update({'Authorization': 'apikeygoeshere'})
#app.route("/")
def index():
alarms = s.get('https://site/api/alarms')
alarmData = json.loads(alarms.text)
if next in alarmData:
while alarmData['next']:
nextUrl = alarmData['next']['href']
print(nextUrl)
nextUrlGet = s.get(nextUrl)
nextData = json.loads(nextUrlGet.text)
alarmData = nextData
if not alarmData['next']:
break
else:
print('No Next -> Update Ready')
alarmUpdateUrl = alarmData['updates']['href']
update = s.get(alarmUpdateUrl)
updateData = json.loads(update.text)
if not updateData['updates']:
updateName = "Nothing"
else:
updateName = updateData['updates'][0]['source']['name']
alarmUpdateUrl = updateData['next']['href']
print(alarmUpdateUrl)
s.get(alarmUpdateUrl)
return render_template('index.html', data=updateName, url=alarmUpdateUrl)
if __name__ == '__main__':
app.run(debug=True, threaded=True)
I tried putting the code in a while loop but it didn't seem to restart. Esentially the endpoint may not have an updates url on page one so i have to follow the next link until there is an update href then request that to initiate a long poll.
When you call an endpoint, Flask figures out which method to run, runs this method, and returns a response, in your case HTML. The connection between client (your browser) and server is then terminated and flask stops doing anything (more or less) until a new request comes in. If you want a website to refresh automatically, you have a few options, but all of them have to be implemented on the client side:
Add <meta http-equiv="refresh" content="30">
to your html head in index html. This will cause your browser to reload the page every 30 seconds.
Add some javascript to your index.html which will poll your API endpoint and update your page when new information is available.
Use websockets (this is similar to the pure javascript approach but potentially with less polling).
Related
I have a flask app that gets it's data from am oracle database using a SQL request. That database is updated very infrequently, so my idea was to get those data from the database and perform many manipulations of those data when the application loads. Then the web pages are very fast because I do not have make a SQL request again, and I also have no fear of SQL injection. However, I do need to update the data about once a day.
Below is a minimum verifiable example of the code which does not use SQL, but whichshould still work to demonstrate the principle.
In my real code df come from the database, but I create multiple variables from that that get passed into the web page using flask. How can I refresh these?
I tried this
How to re-run my python flask application every night?
but the server never refreshed, even when I put debug mode = False. The reason it doesn't work is that once the flask app is running it's like it's in it's own while loop serving the app. The first time I press ctrl+c it exits the app server and starts the while loop again, restarting the server, but that doesn't help.
from flask import Flask, render_template, redirect, url_for, Response
import pandas as pd
import datetime as dt
import flask
#####################################################################
#start of part that I need to refresh
df= pd.DataFrame(range(5),columns=['Column1'])
df['Column2'] = df['Column1']*2
df['Column3'] =dt.datetime.now()
df['Column4'] = df['Column3'] + pd.to_timedelta(df['Column2'],'d')
var1=min(df['Column4'])
var2=max(df['Column4'])
#End of Refresh
####################################################################
app = flask.Flask(__name__, static_url_path='',
static_folder='static',
template_folder='template')
app.config["DEBUG"] = True
#app.route("/home")
def home():
return render_template("home.html",
var1=str(var1),
var2=str(var2))
if __name__ == '__main__':
app.run(host="localhost", port=5000, debug=True)
HOME HTML
'''
<HTML>
<body>
<div>This is Var1: {{var1}} </div>
<div>This is Var2: {{var2}} </div>
</body>
</HTML>
'''
I would like to make a working example that refreshed every 5 minutes as a proof of concept.
Based on the code you provided, the part you want to refresh will only be run once in the startup of your main process. You could have a cron job outside of your flask server that restarts it but this will cause a downtime if someone tries to query while the server is being restarted.
A better solution to this, is to add this query and the manipulation of the data in a function and call it every time someone tries to access the page. This way, you can set a cache which will query the data once and it will save them in memory. You also have the option to specify how long you want the data to be cached, and it will automatically drop the data until the next time someone asks for them.
from cachetools import TTLCache
# Cache the results with a time to live cache. A cached entry will get deleted
# after 300s
TTLCache(maxsize=200, ttl=300)
def get_data():
#####################################################################
#start of part that I need to refresh
df= pd.DataFrame(range(5),columns=['Column1'])
df['Column2'] = df['Column1']*2
df['Column3'] =dt.datetime.now()
df['Column4'] = df['Column3'] + pd.to_timedelta(df['Column2'],'d')
var1=min(df['Column4'])
var2=max(df['Column4'])
#End of Refresh
####################################################################
return var1, var2
Now you can call this function but it will only get executed if the result is not already saved in cache.
#app.route("/home")
def home():
var1, var2 = get_data()
return render_template("home.html",
var1=str(var1),
var2=str(var2))
Another advantage of this solution is that you can always clear the cache completely when you have updated the data.
today I try to make a "waiting page" using Flask.
I mean a client makes a request, I want to show him a page like "wait the process can take a few minutes", and when the process ends on the server display the result.I want to display "wait" before my function manageBill.teste but redirect work only when it returned right?
#application.route('/teste', methods=['POST', 'GET'])
def test_conf():
if request.method == 'POST':
if request.form.get('confList') != None:
conf_file = request.form.get('confList')
username = request.form.get('username')
password = request.form.get('password')
date = request.form.get('date')
if date == '' or conf_file == '' or username == '' or password == '':
return "You forget to provide information"
newpid = os.fork()
if newpid == 0: # in child procces
print('A new child ', os.getpid())
error = manageBill.teste(conf_file, username, password, date)
print ("Error :" + error)
return redirect('/tmp/' + error)
else: # in parent procces
return redirect('/tmp/wait')
return error
return manageBill.manageTest()`
My /tmp route:
#application.route('/tmp/<wait>')
def wait_teste(wait):
return "The procces can take few minute, you will be redirected when the teste is done.<br>" + wait
If you are using the WSGI server (the default), requests are handled by threads. This is likely incompatible with forking.
But even if it wasn't, you have another fundamental issue. A single request can only produce a single response. Once you return redirect('/tmp/wait') that request is done. Over. You can't send anything else.
To support such a feature you have a few choices:
The most common approach is to have AJAX make the request to start a long running process. Then setup an /is_done flask endpoint that you can check (via AJAX) periodically (this is called polling). Once your endpoint returns that the work is done, you can update the page (either with JS or by redirecting to a new page).
Have /is_done be a page instead of an API endpoint that is queried from JS. Set an HTTP refresh on it (with some short timeout like 10 seconds). Then your server can send a redirect for the /is_done endpoint to the results page once the task finishes.
Generally you should strive to serve web requests as quickly as possible. You shouldn't leave connections open (to wait for a long task to finish) and you should offload these long running tasks to a queue system running separately from the web process. In this way, you can scale your ability to handle web requests and background processes separately (and one failing does not bring the other down).
I have a scikit-learn classifier running as a Dockerised Flask app, launched with gunicorn. It receives input data in JSON format as a POST request, and responds with a JSON object of results.
When the app is first launched with gunicorn, a large model (serialised with joblib) is read from a database, and loaded into memory before the app is ready for requests. This can take 10-15 minutes.
A reproducible example isn't feasible, but the basic structure is illustrated below:
from flask import Flask, jsonify, request, Response
import joblib
import json
def classifier_app(model_name):
# Line below takes 10-15 mins to complete
classifier = _load_model(model_name)
app = Flask(__name__)
#app.route('/classify_invoice', methods=['POST'])
def apicall():
query = request.get_json()
results = _build_results(query['data'])
return Response(response=results,
status=200,
mimetype='application/json')
print('App loaded!')
return app
How do I configure Flask or gunicorn to return a 'still loading' response (or suitable error message) to any incoming http requests while _load_model is still running?
Basically, you want to return two responses for one request. So there are two different possibilities.
First one is to run time-consuming task in background and ping server with simple ajax requests every two seconds to check if task is completed or not. If task is completed, return result, if not, return "Please standby" string or something.
Second one is to use websockets and flask-socketio extension.
Basic server code would be something like this:
from threading import Thread
from flask import Flask
app = Flask(__name__)
socketio = SocketIO(app)
def do_work():
result = your_heavy_function()
socketio.emit("result", {"result": result}, namespace="/test/")
#app.route("/api/", methods=["POST"])
def start():
socketio.start_background_task(target=do_work)
# return intermediate response
return Response()
On the client side you should do something like this
var socket = io.connect('http://' + document.domain + ':' + location.port + '/test/');
socket.on('result', function(msg) {
// Process your request here
});
For further details, visit this blog post, flask-socketio documentation for server-side reference and socketio documentation for client-side reference.
PS Using web-sockets this you can make progress-bar too.
The app is loading the "links_submit.html" which has a field where you can write a link, like for example (www.google.com) and you submit it, then the app is receiving this URL as HTTP Post and redirecting to another page "post_response.html" which contains a simple html for feedback with the word "Ok". Then I want to do a process with this link (crawl google and search for a specific thing) and after finish this process, automatically redirect from the "post_reponse.html" to another page to show the results I have extracted from google. Now I'm not sure how say to my app on flask: "Ok now lets use a normal function (not route) like for example:
def loadpage(link_sent_by_the_http post):
res = requests.get('www.google.com')
Imagine that after load the page I also extract some html tag on google and after finish this process I want to redirect the page "post_respose.html" with the "ok" to a new html page which contains the html tag extracted from google.
Please note I know how to load the page google and extract what I want, but I don't know how to insert this function/process in the middle of Flask and then redirect from a normal html with "ok" for a new route with the results I have extracted.
import requests
from flask import Flask, render_template, request, url_for
app = Flask(__name__)
#app.route('/test')
def form():
return render_template('links_submit.html')
#app.route('/links/', methods=['POST'])
def links():
links=request.form['links']
return render_template('post_response.html')
Intern Process (Load the received link > Extract what I want)
and then redirect the "post_response.html" to another "html" which will
contain the results that I have extracted)
if __name__ == '__main__':
app.run(debug=True)
Two ways to do it-
Create a python file say webfunctions.py and put your function in this file.
e.g. -
def inc(x):
return int(x) + 1
Now in your flask app file, you can import the whole file or just the function-
from webfunctions import inc
#app.route('/whatsnext/', methods=['POST'])
def waiting():
curVal=request.form['x']
nextVal = inc(curVal)
return render_template('post_response.html', nextVal=nextVal)
or else, you may declare your definitions at the top of your flask app file. Like below -
import requests
from flask import Flask, render_template, request, url_for
def inc(x):
return int(x) + 1
#app.route('/whatsnext/', methods=['POST'])
def waiting():
curVal=request.form['x']
nextVal = inc(curVal)
return render_template('post_response.html', nextVal=nextVal)
I am working on a small project in Python. It is divided into two parts.
First part is responsible to crawl the web and extract some infromation and insert them into a database.
Second part is resposible for presenting those information with use of the database.
Both parts share the database. In the second part I am using Flask framework to display information as html with some formatting, styling and etc. to make it look cleaner.
Source files of both parts are in the same package, but to run this program properly user has to run crawler and results presenter separately like this :
python crawler.py
and then
python presenter.py
Everything is allright just except one thing. What I what presenter to do is to create result in html format and open the page with results in user's default browser, but it is always opened twice, probably due to the presence of run() method, which starts Flask in a new thread and things get cloudy for me. I don't know what I should do to be able to make my presenter.py to open only one tab/window after running it.
Here is the snippet of my code :
from flask import Flask, render_template
import os
import sqlite3
# configuration
DEBUG = True
DATABASE = os.getcwd() + '/database/database.db'
app = Flask(__name__)
app.config.from_object(__name__)
app.config.from_envvar('CRAWLER_SETTINGS', silent=True)
def connect_db():
"""Returns a new connection to the database."""
try:
conn = sqlite3.connect(app.config['DATABASE'])
return conn
except sqlite3.Error:
print 'Unable to connect to the database'
return False
#app.route('/')
def show_entries():
u"""Loads pages information and emails from the database and
inserts results into show_entires template. If there is a database
problem returns error page.
"""
conn = connect_db()
if conn:
try:
cur = connect_db().cursor()
results = cur.execute('SELECT url, title, doctype, pagesize FROM pages')
pages = [dict(url=row[0], title=row[1].encode('utf-8'), pageType=row[2], pageSize=row[3]) for row in results.fetchall()]
results = cur.execute('SELECT url, email from emails')
emails = {}
for row in results.fetchall():
emails.setdefault(row[0], []).append(row[1])
return render_template('show_entries.html', pages=pages, emails=emails)
except sqlite3.Error, e:
print ' Exception message %s ' % e
print 'Could not load data from the database!'
return render_template('show_error_page.html')
else:
return render_template('show_error_page.html')
if __name__ == '__main__':
url = 'http://127.0.0.1:5000'
webbrowser.open_new(url)
app.run()
I use similar code on Mac OS X (with Safari, Firefox, and Chrome browsers) all the time, and it runs fine. Guessing you may be running into Flask's auto-reload feature. Set debug=False and it will not try to auto-reload.
Other suggestions, based on my experience:
Consider randomizing the port you use, as quick edit-run-test loops sometimes find the OS thinking port 5000 is still in use. (Or, if you run the code several times simultaneously, say by accident, the port truly is still in use.)
Give the app a short while to spin up before you start the browser request. I do that through invoking threading.Timer.
Here's my code:
import random, threading, webbrowser
port = 5000 + random.randint(0, 999)
url = "http://127.0.0.1:{0}".format(port)
threading.Timer(1.25, lambda: webbrowser.open(url) ).start()
app.run(port=port, debug=False)
(This is all under the if __name__ == '__main__':, or in a separate "start app" function if you like.)
So this may or may not help. But my issue was with flask opening in microsoft edge when executing my app.py script... NOOB solution. Go to settings and default apps... And then change microsoft edge to chrome... And now it opens flask in chrome everytime. I still have the same issue where things just load though