I have a scikit-learn classifier running as a Dockerised Flask app, launched with gunicorn. It receives input data in JSON format as a POST request, and responds with a JSON object of results.
When the app is first launched with gunicorn, a large model (serialised with joblib) is read from a database, and loaded into memory before the app is ready for requests. This can take 10-15 minutes.
A reproducible example isn't feasible, but the basic structure is illustrated below:
from flask import Flask, jsonify, request, Response
import joblib
import json
def classifier_app(model_name):
# Line below takes 10-15 mins to complete
classifier = _load_model(model_name)
app = Flask(__name__)
#app.route('/classify_invoice', methods=['POST'])
def apicall():
query = request.get_json()
results = _build_results(query['data'])
return Response(response=results,
status=200,
mimetype='application/json')
print('App loaded!')
return app
How do I configure Flask or gunicorn to return a 'still loading' response (or suitable error message) to any incoming http requests while _load_model is still running?
Basically, you want to return two responses for one request. So there are two different possibilities.
First one is to run time-consuming task in background and ping server with simple ajax requests every two seconds to check if task is completed or not. If task is completed, return result, if not, return "Please standby" string or something.
Second one is to use websockets and flask-socketio extension.
Basic server code would be something like this:
from threading import Thread
from flask import Flask
app = Flask(__name__)
socketio = SocketIO(app)
def do_work():
result = your_heavy_function()
socketio.emit("result", {"result": result}, namespace="/test/")
#app.route("/api/", methods=["POST"])
def start():
socketio.start_background_task(target=do_work)
# return intermediate response
return Response()
On the client side you should do something like this
var socket = io.connect('http://' + document.domain + ':' + location.port + '/test/');
socket.on('result', function(msg) {
// Process your request here
});
For further details, visit this blog post, flask-socketio documentation for server-side reference and socketio documentation for client-side reference.
PS Using web-sockets this you can make progress-bar too.
Related
I am building a rest api with fastapi. I implemented the data layer separately from the fastapi application meaning I do not have direct access to the database session in my fastapi application.
I have access to the storage object which have method like close_session which allow me to close the current session.
Is there a equivalent of flask teardown_request in fastapi?
Flask Implementation
from models import storage
.....
.....
#app.teardown_request
def close_session(exception=None):
storage.close_session()
I have looked at fastapi on_event('shutdown') and on_event('startup'). These two only runs when the application is shutting down or starting up.
We can do this by using dependency.
credit to williamjemir: Click here to read the github discussion
from fastapi import FastAPI, Depends
from models import storage
async def close_session() -> None:
"""Close current after every request."""
print('Closing current session')
yield
storage.close()
print('db session closed.')
app = FastAPI(dependencies=[Depends(close_session)])
#app.get('/')
def home():
return "Hello World"
if __name__ == '__main__':
import uvicorn
uvicorn.run(app)
use fastapi middleware
A "middleware" is a function that works with every request before it is processed by any specific path operation. And also with every response before returning it.
It takes each request that comes to your application.
It can then do something to that request or run any needed code.
Then it passes the request to be processed by the rest of the application (by some path operation).
It then takes the response generated by the application (by some path operation).
It can do something to that response or run any needed code.
Then it returns the response.
Example:
import time
from fastapi import FastAPI, Request
app = FastAPI()
#app.middleware("http")
async def add_process_time_header(request: Request, call_next):
# do things before the request
response = await call_next(request)
# do things after the response
return response
references:
https://fastapi.tiangolo.com/tutorial/middleware/
I have below flask application that should stream JSON response
Directory structure:
server -
api.py
resources -
stream.py
api.py
from flask import Flask
from flask_restful import Api
from resources.stream import Stream
from flask_compress import Compress
compress = Compress()
app = Flask(__name__)
compress.init_app(app)
api = Api(app)
api.add_resource(Stream, '/stream')
if __name__ == '__main__':
app.run(debug=False,host='0.0.0.0')
stream.py in resources directory
from flask import Response, request, stream_with_context
from flask_restful import Resource
class Stream(Resource):
def get(self):
def generator():
yield '{"data": ['
#creating large number of entries for data array with yield
yield ']}'
resp = Response(stream_with_context(generator()), status=200, content_type='application/json')
return resp
I started the flask app with python3.9 api.py
I am able to get response when I hit url http://127.0.0.1:5000/stream but in network tab I can see some issues:
If the response was streamed it should not have content-length
The streamed response should not be in Waiting for server response state, somehow its waiting to finish whole response and then starts downloading.
And below is the output that I am trying to achieve. A streamed output which would start Content Downloading and not be stuck in Waiting for server response
I found the solution for the problem I faced. I was using flask_compress which was causing this issue. Removing it would stream the response as expected. Thank you for the help though.
I am trying to use Flask to send a stream of events to a front-end client as documented in this question. This works fine if I don't access anything in the request context, but fails as soon as I do.
Here's an example to demonstrate.
from time import sleep
from flask import Flask, request, Response
app = Flask(__name__)
#app.route('/events')
def events():
return Response(_events(), mimetype="text/event-stream")
def _events():
while True:
# yield "Test" # Works fine
yield request.args[0] # Throws RuntimeError: Working outside of request context
sleep(1)
Is there a way to access the request context for server-sent events?
You can use the #copy_current_request_context decorator to make a copy of the request context that your event stream function can use:
from time import sleep
from flask import Flask, request, Response, copy_current_request_context
app = Flask(__name__)
#app.route('/events')
def events():
#copy_current_request_context
def _events():
while True:
# yield "Test" # Works fine
yield request.args[0]
sleep(1)
return Response(_events(), mimetype="text/event-stream")
Note that to be able to use this decorator the target function must be moved inside the view function that has the source request.
I'm trying to better understand Google App Engine, specifically the request processing time limits. On the documentation it states:
A request handler has a limited amount of time to generate and return
a response to a request, typically around 60 seconds. Once the
deadline has been reached, the request handler is interrupted.
I understood that to mean that a call to the sleep function below (via the [project-id].appspot.com/sleep url) would result in an error, but that is not the case. I've ran it for 600 seconds and it still returned the expected response.
import time
from flask import Flask
# If `entrypoint` is not defined in app.yaml, App Engine will look for an app
# called `app` in `main.py`.
app = Flask(__name__)
#app.route("/")
def hello():
"""Return a friendly HTTP greeting."""
return "Hello World!"
#app.route("/sleep")
def sleep():
sleep_time = request.args.get("time", default=120, type=int)
time.sleep(sleep_time)
return f"slept {sleep_time} seconds"
Two things are happening: App Engine is being generous here, and you haven't quite exceeded the limit. I tried the following app:
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
"""Return a friendly HTTP greeting."""
time.sleep(1000000)
return "Hello World!"
And it timed out after 601.9 seconds. I'm sure you'll get similar behavior if you try longer times, but generally you shouldn't depend on the extra runtime being available.
I currently have a flask app that makes a call to S3 as well as an external API with the following structure before rendering the data in javascript:
from flask import Flask, render_template,make_response
from flask import request
import requests
import requests_cache
import redis
from boto3.session import Session
import json
app = Flask(__name__)
#app.route('/test')
def test1():
bucket_root = 'testbucket'
session = Session(
aws_access_key_id='s3_key',
aws_secret_access_key='s3_secret_key')
s3 = session.resource('s3')
bucket = s3.Bucket(bucket_root)
testvalues = json.dumps(s3.Object(bucket_root,'all1.json').get()['Body'].read())
r = requests.get(api_link)
return render_template('test_html.html',json_s3_test_response=r.content,
limit=limit, testvalues=testvalues)
#app.route('/test2')
def test2():
bucket_root = 'testbucket'
session = Session(
aws_access_key_id='s3_key',
aws_secret_access_key='s3_secret_key')
s3 = session.resource('s3')
bucket = s3.Bucket(bucket_root)
testvalues = json.dumps(s3.Object(bucket_root,'all2.json').get()['Body'].read())
r = requests.get(api_link)
return render_template('test_html.html',json_s3_test_response=r.content,
limit=limit, testvalues=testvalues)
#app.errorhandler(500)
def internal_error(error):
return "500 error"
#app.errorhandler(404)
def not_found(error):
return "404 error",404
#app.errorhandler(400)
def custom400(error):
return "400 error",400
//catch all?
#app.errorhandler(Exception)
def all_exception_handler(error):
return 'error', 500
Obviously I have a lot of inefficiencies here, but my main question is:
To me it seems like I'm calling S3 and the external API for each client, every time they refresh the page. This increases the chance for the app to crash due to timeouts (and my poor error handling) and diminishes performance. I would like to resolve this by periodically caching the S3 results (say every 10 mins) into a local redis server (already set up and running) as well as just pinging the external API just once from the server every few seconds before passing it onto ALL clients.
I have code that can store the data into redis every 10 mins in a regular python script, however, I'm not sure where to place this within the flask server? Do I put it as it's own function or keep the call to redis in the #app.route()?
Thank you everyone for your time and effort. Any help would be appreciated! I'm new to flask so some of this has been confusing.