I'm deploying a sample fast api app to the cloud with google standard app engine model. The app is served with gunicorn this way:
gunicorn main:app --workers 4 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:80
This command spawns 4 worker proccesses of my app.
I've read that in fast api you can either create sync or async endpoints. If an endpoint is async all requests run on a single thread with the event loop. If the endpoint is sync, it runs the function on another thread to prevent it from blocking the server.
I have sync blocking endpoints, so fastapi should run them on threads, but also i have gunicorn spawning worker proccesess.
Given that python only executes one thread at a time, but also the standard app engine is also limited CPU wise on multiple proccessing, i'm confused on the best configuration for a fastapi application on the cloud.
Should i let gunicorn or fastapi handle the concurrency?
The number of workers you specify should match the instance class of your App Engine app; and since you're using 4 workers in your app, it has an equivalence of 4 instance classes. Here's an example that shows an App Engine deployment that uses 4 gunicorn workers for serving apps: entrypoint: gunicorn -b :8080 -w 4 main:app. The examples I've provided was stated in the entrypoint best practices.
Just a note, the gunicorn uses sync workers by default so that worker class is compatible with all web applications, but each worker can only handle one request at a time.
Lastly if using Google App Engine Flex, kindly check the recommended gunicorn configurations for further guide in your app.
Related
I am working on a flask backend and one of its features require regular calls to an external API and store some results in the database. I used APScheduler to do so.
Not having full access to the upstream server (docker container within google cloud platforms), we managed to serve the backend using apache's mod_wsgi.
Running the backend from my debugger on my PC, the scheduled tasks seem to work as my database is populated.
But the server does not seem to run those tasks at all at when I query the database which should be populated, the tables are empty.
My usage of APScheduler is as follows in the __init__.py :
from apscheduler.schedulers.background import BackgroundScheduler
# Some unrelated code here
scheduler = BackgroundScheduler()
import module as service
scheduler.add_job(service.job1, 'interval', minutes=10)
scheduler.add_job(service.job2, 'interval', minutes=5)
scheduler.add_job(service.job3, 'interval', minutes=1)
scheduler.start()
I'm asking if there are additional steps that I need to do in order for those tasks to be run on the upstream server.
In development, flask-socketio (4.1.0) with uwsgi is working nicely with just 1 worker and standard initialization.
Now I'm preparing for production and want to make it work with multiple workers.
I've done the following:
Added redis message_queue in init_app:
socketio = SocketIO()
socketio.init_app(app,async_mode='gevent_uwsgi', message_queue=app.config['SOCKETIO_MESSAGE_QUEUE'])
(Sidenote: we are using redis in the app itself as well)
gevent monkey patching at top of the file that we run with uwsgi
from gevent import monkey
monkey.patch_all()
run uwsgi with:
uwsgi --http 0.0.0.0:63000 --gevent 1000 --http-websockets --master --wsgi-file rest.py --callable application --py-autoreload 1 --gevent-monkey-patch --workers 4 --threads 1
This doesn't seem to work. The connection starts rapidly alternating between a connection and 400 Bad request responses. I suspect these correspond to the ' Invalid session ....' errors I see when I enable SocketIO logging.
Initially it was not using redis at all,
redis-cli > PUBSUB CHANNELS *
resulted in an empty result even with workers=1.
it seemed the following (taken from another SO answer) fixed that:
# https://stackoverflow.com/a/19117266/492148
import gevent
import redis.connection
redis.connection.socket = gevent.socket
after doing so I got a "flask-socketio" pubsub channel with updating data.
but after returning to multiple workers, the issue returned. Given that changing the redis socket did seem to bring things in the right direction I feel like the monkeypatching isn't working properly yet, but the code I used seems to match all examples I can find and is at the very top of the file that is loaded by uwsgi.
You can run as many workers as you like, but only if you run each worker as a standalone single-worker uwsgi process. Once you have all those workers running each on its own port, you can put nginx in front to load balance using sticky sessions. And of course you also need the message queue for the workers to use when coordinating broadcasts.
Eventually found https://github.com/miguelgrinberg/Flask-SocketIO/issues/535
so it seems you can't have multiple workers with uwsgi either as it needs sticky sessions. Documentation mentions that for gunicorn, but I did not interpret that to extend to uwsgi.
I'm trying to deploy a flask app on heroku. I've gotten to the point where the app builds and deploys, but when I try to go to the URL, the app times out with the following error.
Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
I think the problem is with my procfile. It has one line.
web: python add_entry3.py
Other people have procfiles that look like this:
web: gunicorn app:app
This is just a toy app and I don't care about performance so I don't think I need to use gunicorn for the web server. Should I be putting a colon and command after my app's file name (add_entry3.py)?
Most likely your flask app isn't answering on the port and interface the Heroku expects. By default, Flask only listens on 127.0.0.1, and I think on port 5000. Heroku passes your app a PORT environment variable and you'd need to tell Flask to listen on all interfaces.
But there are reasons other than performance you want to avoid Flask's default debug server for production code. It's got memory leaks, there are security implications, and really ... just don't do it. Add gunicorn to your requirements.txt and use that.
But if you must use the Flask test/debug server, change your app.run() call to something like this:
app.run(host='0.0.0.0', port=int(os.environ.get("PORT", 5000)))
I have web application written in Flask. As suggested by everyone, I can't use Flask in production. So I thought of Gunicorn with Flask.
In Flask application I am loading some Machine Learning models. These are of size 8GB collectively. Concurrency of my web application can go upto 1000 requests. And the RAM of machine is 15GB.
So what is the best way to run this application?
You can start your app with multiple workers or async workers with Gunicorn.
Flask server.py
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run()
Gunicorn with gevent async worker
gunicorn server:app -k gevent --worker-connections 1000
Gunicorn 1 worker 12 threads:
gunicorn server:app -w 1 --threads 12
Gunicorn with 4 workers (multiprocessing):
gunicorn server:app -w 4
More information on Flask concurrency in this post: How many concurrent requests does a single Flask process receive?.
The best thing to do is to use pre-fork mode (preload_app=True). This will initialize your code in a "master" process and then simply fork off worker processes to handle requests. If you are running on linux and assuming your model is read-only, the OS is smart enough to reuse the physical memory amongst all the processes.
I found this 0 dependency python websocket server from SO: https://gist.github.com/jkp/3136208
I am using gunicorn for my flask app and I wanted to run this websocket server using gunicorn also. In the last few lines of the code it runs the server with:
if __name__ == "__main__":
server = SocketServer.TCPServer(
("localhost", 9999), WebSocketsHandler)
server.serve_forever()
I cannot figure out how to get this websocketserver.py running in gunicorn. This is because one would think you would want gunicorn to run server_forever() as well as the SocketServer.TCPServer(....
Is this possible?
GUnicorn expects a WSGI application (PEP 333) not just a function. Your app has to accept an environ variable and a start_response callback and return an iterator of data (roughly speaking). All the machinery encapsuled by SocketServer.StreamRequestHandler is on gunicorn side. I imagine this is a lot of work to modify this gist to become a WSGI application (But that'll be fun!).
OR, maybe this library will get the job done for you: https://github.com/CMGS/gunicorn-websocket
If you use Flask-Sockets extension, you have a websocket implementation for gunicorn directly in the extension which make it possible to start with the following command line :
gunicorn -k flask_sockets.worker app:app
Though I don't know if that's what you want to do.