I'm creating a live data table page using Flask. The data shown is coming from a PostgreSQL database.
I am passing the new data being inserted into the database using the pubsub mechanism, and the operation is then transmitted using SSE to the front end Flask application.
Everything is working fine when I am opening one window but when I'm opening another window (to simulate multiple users), only the most recently loaded window receives the SSE. What am I doing wrong?
I tried multiple solutions, from using Flask in threaded mode to using gevent or gunicorn
Here is my code:
main.py
from gevent import monkey
monkey.patch_all()
import psycopg2, pgpubsub
from flask import Flask, render_template, Response
from gevent.pywsgi import WSGIServer
import time
app = Flask(__name__)
pubsub = pgpubsub.connect(database="postgres"....)
def get_connection():
try:
return psycopg2.connect(database="postgres.....")
except Exception as e:
return f"Error connecting to DB: {e}"
#app.route('/')
def home():
return render_template('index.html')
#app.route('/events')
def events():
def update_pusher():
print('Started listening')
pubsub.listen('data_changed')
while True:
for event in pubsub.events(yield_timeouts=True):
if event is None:
pass
else:
yield f"data: {event.payload}\nevent: online\n\n"
time.sleep(0.01)
return Response(
response=update_pusher(),
mimetype='text/event-stream'
)
if __name__ == "__main__":
http_server = WSGIServer(("localhost", 5003), app)
http_server.serve_forever()
index.html
<p>This list is populated by server side events.</p>
<ul id="list"></ul>
<script>
var eventSource = new EventSource("/events")
eventSource.addEventListener("online", function(e) {
// console.log(e.data.color)
data = JSON.parse(e.data)
const li = document.createElement('li')
li.innerText = data
list.appendChild(li)
}, false)
Related
The following scenario:
I have a Raspberry Pi running as a server. Currently I am using a Python script with Flask and I can also access the Raspberry Pi from my PC. (The flask server runs an react app.)
But the function should be extended. It should look like the following:
2nd Python script is running all the time. This Python script fetches data from an external API every second and processes it. If certain conditions are met, the data should be processed and then the data should be communicated to the Python Flask server. And the Flask server then forwards the data to the website running on the computer.
How or which method is best to program this "interprocess communication". Are there any libraries? I tried Celery, but then it throws up my second Python script whenever I want to access the external API, so I don't know if this is the right choice.
What else would be the best approach? Threading? Direct interprocess communication?
If important, this is how my server application looks so far:
from gevent import monkey
from flask import Flask, render_template
from flask_socketio import SocketIO
monkey.patch_all()
app = Flask(__name__, template_folder='./build', static_folder='./build/static')
socket_io = SocketIO(app)
#app.route('/')
def main():
return render_template('index.html')
#socket_io.on('fromFrontend')
def handleInput(input):
print('Input from Frontend: ' + input)
send_time()
#socket_io.on('time')
def send_time():
socket_io.emit('time', {'returnTime': "some time"})
if __name__ == '__main__':
socket_io.run(app, host='0.0.0.0', port=5000, debug=True)
Well i found a solution for my specific problem i implemented it with a thread as follows:
import gevent.monkey
gevent.monkey.patch_all()
from flask import Flask, render_template
from flask_socketio import SocketIO
import time
import requests
from threading import Thread
app = Flask(__name__, template_folder='./build', static_folder='./build/static')
socket_io = SocketIO(app)
#app.route('/')
def main():
thread = Thread(target=backgroundTask)
thread.daemon = True
thread.start()
return render_template('index.html')
#socket_io.on('fromFrontend')
def handleInput(input):
print('Input from Frontend: ' + input)
#socket_io.on('time')
def send_time():
socket_io.emit('time', {'returnTime': 'hi frontend'})
def backgroundTask():
# do something here
# access socket to push some data
socket_io.emit('time', {'returnTime': "some time"})
if __name__ == '__main__':
socket_io.run(app, host='0.0.0.0', port=5000, debug=True)
I'm using the Flask-SocketIO library which works fine but I need to send a notification with emit to the outside of a socket.io decorator and it's a real pain. Looking at the solutions, many people use rabbitmq or redis but I don't know how to use them.
Here's my code :
from flask import Flask, render_template
from flaskwebgui import FlaskUI
from flask_socketio import SocketIO, emit
app = Flask(__name__)
async_mode = None
app.config['SECRET_KEY'] = 'hello'
socketio = SocketIO(app, async_mode=async_mode, message_queue='amqp:///socketio')
def run_sock():
socketio.run(app, debug=True)
ui = FlaskUI(app, fullscreen=True, server=run_sock,)
#app.route("/")
def index():
return render_template('index.html')
#socketio.on('test', namespace='/test')
def test():
print("test")
if __name__ == "__main__":
ui.run()
io = SocketIO(message_queue='amqp:///socketio')
io.emit('test_emit', {'data': 'toto'}, namespace='/test')
My JS front-end never gets the test_emit message, how do I do?
The problem with your emit is that it appears below the ui.run() call, which does not return until you close the application. Move the emit to any function in your application that executes while the server is running (such as a Flask view function) and it should work just fine.
Why do you have two SocketIO objects in the same process? The socketio instance that you defined near the top of the script can be used anywhere within the process, no need to create a second instance. You do not need to use a message queue for this problem, since you have all the usages of Socket.IO within a single process.
I have a scikit-learn classifier running as a Dockerised Flask app, launched with gunicorn. It receives input data in JSON format as a POST request, and responds with a JSON object of results.
When the app is first launched with gunicorn, a large model (serialised with joblib) is read from a database, and loaded into memory before the app is ready for requests. This can take 10-15 minutes.
A reproducible example isn't feasible, but the basic structure is illustrated below:
from flask import Flask, jsonify, request, Response
import joblib
import json
def classifier_app(model_name):
# Line below takes 10-15 mins to complete
classifier = _load_model(model_name)
app = Flask(__name__)
#app.route('/classify_invoice', methods=['POST'])
def apicall():
query = request.get_json()
results = _build_results(query['data'])
return Response(response=results,
status=200,
mimetype='application/json')
print('App loaded!')
return app
How do I configure Flask or gunicorn to return a 'still loading' response (or suitable error message) to any incoming http requests while _load_model is still running?
Basically, you want to return two responses for one request. So there are two different possibilities.
First one is to run time-consuming task in background and ping server with simple ajax requests every two seconds to check if task is completed or not. If task is completed, return result, if not, return "Please standby" string or something.
Second one is to use websockets and flask-socketio extension.
Basic server code would be something like this:
from threading import Thread
from flask import Flask
app = Flask(__name__)
socketio = SocketIO(app)
def do_work():
result = your_heavy_function()
socketio.emit("result", {"result": result}, namespace="/test/")
#app.route("/api/", methods=["POST"])
def start():
socketio.start_background_task(target=do_work)
# return intermediate response
return Response()
On the client side you should do something like this
var socket = io.connect('http://' + document.domain + ':' + location.port + '/test/');
socket.on('result', function(msg) {
// Process your request here
});
For further details, visit this blog post, flask-socketio documentation for server-side reference and socketio documentation for client-side reference.
PS Using web-sockets this you can make progress-bar too.
I want to build a Webapp with Flask where some data is printed on a dynamic page in real time.
The data is taken from a Python script which connects to a Websocket, then it's printed on the frontend with Flask.
I have two problems:
1) I can't run both the scripts together
2) I don't know how to call parsed from test to yield
Here is the code:
from time import sleep
from flask import Flask, render_template
import websocket
from bitmex_websocket import Instrument
from bitmex_websocket.constants import InstrumentChannels
from bitmex_websocket.constants import Channels
import json
from threading import Thread, Event
app = Flask(__name__)
websocket.enableTrace(True)
channels = [
InstrumentChannels.trade,
]
XBTUSD = Instrument(symbol='XBTUSD',
channels=channels)
XBTUSD.on('action', lambda msg: test(msg))
def test(msg):
parsed = json.loads(json.dumps(msg))
print(parsed)
#app.route('/')
def index():
# render the template (below) that will use JavaScript to read the stream
return render_template('index.html')
#app.route('/stream_sqrt')
def stream():
def generate():
yield '{}\n'.format('test')
return app.response_class(generate(), mimetype='text/plain')
if __name__ == '__main__':
XBTUSD.run_forever()
app.run()
If i put XBTUSD.run_forever() before app.run() i will start the part supposed to retrieve the data but the Flask app won't start. If i do the opposite, the Flask app will run but not the other part. How can i run together the whole app? How could i "share" variables between test and generate?
An easier way to go, please use flask-socketio instead flask.
https://flask-socketio.readthedocs.io/en/latest/
Sample for sending messages using flask-socketio
https://flask-socketio.readthedocs.io/en/latest/#sending-messages
I am receiving some socket.io event. After the event is handled I would like to redirect the user to some page. However, the redirection doesn't work. I don't know what is wrong. Below is my code:
from flask import Flask, render_template, send_from_directory, redirect, url_for
from flask_socketio import SocketIO, emit
import base64
import os
import random, string
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app)
#app.route('/')
def index():
try:
image_names = os.listdir('./images')
print image_names
return render_template("gallery.html", image_names=image_names)
except Exception as ex:
print ex
#socketio.on('takephoto')
def takePhoto(*args):
try:
decoded = base64.b64decode(args[0])
filename = ''.join(random.choice(string.lowercase) for x in range(6)) + '.jpg'
with open("./images/" + filename, "wb") as fh:
fh.write(args[0].decode('base64'))
except Exception, ex:
print ex
redirect(url_for('index')) #This doesnt work
#how can i go to index from this point?
if __name__ == '__main__':
from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler
app.debug = True
server = pywsgi.WSGIServer(('', 5001), app, handler_class=WebSocketHandler)
server.serve_forever()
I think you are missing the return statement.
return redirect(url_for('index'))
A little late to the party, but I just ran into this. Emitting a 'redirect' event and sending url_for('index') all via sockets is your best bet.
Server side:
emit('redirect', url_for('auth.login'), namespace=request.namespace, room=[request.sid])
Client side:
socket.on('redirect', (dest) => {
window.location.href = dest;
});