When my Flask app is launched, it starts a RabbitMQ consumer on a new thread. The consumer should keep running regardless of requests/responses and wait for incoming messages. Everything works fine when running locally and also when I deploy the app on linux.
Unfortunatly, due to some third-party dependencies I must deploy the app on a Windows server. I am able to configure IIS to serve the app using wfastcgi, but it seems that after some time with no activity the app gets shutdown until the next HTTP request, when it is automatically initalized again. This behaviour would not bother me normally, but it in this case the RabbitMQ consumer is also closed every time and then the incoming messages from the queue are not recieved.
Is there a way to keep the app running infinitly?
I saw some settings in IIS like Activity Timeout, Idle Timeout, etc. but I don't know which one is relevant. Also, what if I want the app to keep running and consuming messages from RabbitMQ even if there are no requests for a very long time, e.g. months or years?
Related
I have a WSGI application (it's a Flask app, but that should be irrelevant, I think) running under a Gunicorn server at port 9077. The app has a /status endpoint, which is supposed to report 'OK' if the app is running. If it fails to report OK within a reasonable time, the whole container gets killed (by Kubernetes).
The problem is this: when the app is under very heavy load (which does happen occasionally), the /status endpoint can take a while to respond and the container sometimes gets killed prematurely. Is there a way to configure Gunicorn to always serve the /status endpoint in a separate thread? Perhaps even on a different port? I would appreciate any hints or ideas for dealing with this situation.
never worked with Gunicorn, and im not sure if it supports this feature.
But with uWSGI, when i know that the app is going to be under a heavy load,
i run uwsgi with --processes (can also run in multithread mode or both)
uWSGI just spins up multiple instances of the flask app and act as a load balancer, no need for different ports, uwsgi takes care of everything.
You are not bound by GIL anymore and your app uses all the resources available on the machine.
documentation about uWSGI concurrency
a quick tutorial on how to setup a flask app, uWSGI and nginx (you can skip the nginx part)
here is an example of the config file i provide.
[uwsgi]
module = WSGI:app
master = true
processes = 16
die-on-term = true
socket = 0.0.0.0:8808
protocol = http
uwsgi --daemonize --ini my_uwsgi_conf.ini
I can easily achieve 1000 calls/sec when its running that way.
hope that helps.
ps: Another solution for you, just spin up more containers that are running your app.
And put them behind nginx to load-balance
I occasionally have a problem with flask socket-io freezing, and I have no clue how to fix it.
My client connects to my socket-io server and performs some chat sessions. It works nicely. But for some reason, sometimes from the client side, there is some call that blocks the whole server (The server is stuck in the process, and all other calls are frozen). What is strange is that the server can be blocked as long as the client side app is not totally shutdown.This is an ios-app / web page, and I must totally close the app or the safari page. Closing the socket itself, and even deallocating it doesn't resolve the problem. When the app is in the background, the sockets are closed and deallocated but the problem persists.
This is a small server, and it deals with both html pages and the socket-server so I have no idea if it is the socket itself or the html that blocks the process. But each time the server was freezing, the log showed some socket calls.
Here is how I configured my server:
socketio = SocketIO(app, ping_timeout=5)
socketio.run(app, host='0.0.0.0', port=5001, debug=True, ssl_context=context)
So my question is:
What can freeze the server (this seems to happen when I leave the app or web-site open for a long time while doing nothing). If I use the services normally the server never freezes. And how I can prevent it from happening (Even if I don't know what causing this, is there a way to blindly stop my server from being stuck at a call?
Thanks you for the answers
According to your comment above, you are using the Flask development web server, without the help of an asynchronous framework such as eventlet or gevent. Besides this option being highly inefficient, you should know that this web server is not battle tested, it is meant for short lived tests during development. I'm not sure it is able to run for very long, specially under the unusual conditions Flask-SocketIO puts it through, which regular Flask apps do not exercise. I think it is quite possible that you are hitting some obscure bug in Werkzeug that causes it to hang.
My recommendation is that you install eventlet and try again. All you need to do is pip install eventlet, and assuming you are not passing an explicit async_mode argument, then just by installing this package Flask-SocketIO should configure itself to use it.
I would also remove the explicit timeout setting. In almost all cases, the defaults are sufficient to maintain a healthy connection.
I have a python web application running on apache2 deployed with mod_wsgi. The application has a thread continuously running. This thread is a ZeroMQ thread and listening to a port in loop. The application is not maintaining session. Now if I open the browser and sends a request to the apache server the data is accepted for the first time. Now when second time I send the request It shows Internal server error. When I checked the error log file for traceback, It shows the ZMQError:- The address already in use.
Does apache reloads the application on each request sent from the browser since so that the ZeroMQ thread is being created everytime and being assigned the port but since the port has already been assigned it shows error....
It looks like your application is using zmq to bind to some port.
As you have suspected already, each request can be run as independent process, thus competing in access to the port to bind to.
There can be so called workers, each running one process processing http/wsgi requests, and each trying to bind.
You shall redesign your app not to use bind, but connect, this will probably require having another process with zeromq serving something you do with that (but this last line is dependent on what you do in your app).
So, probably a dumb question, but I am beginning to learn all this so your feedback will be valuable for me.
The question is: In flask documentation it says start the flask server by entering the command 'python hello.py' and I do it successfully to see the output on localhost:5000. Now, I have a shared hosting plan and if I upload this file over there will i need to initiate the server over there as well like this? If so, when I close the terminal over there, will the flask server shut down (because when I close the terminal on my computer it shuts down the flask server and the results are no more available on localhost:5000)?.. It basically suggests me that I have to keep running the terminal all the time..please tell me what is the basic idea here? Thanks.
What you're asking is how you deploy your app. There are many options, that will depend on your needs, your hosting service, etc.
You should check the flask docs for the options. http://flask.pocoo.org/docs/deploying/
In essence, you'll have your flask app running as a local service on the server, so it's not shut down when you close the terminal, and an HTTP server that somehow proxies requests to that service. I guess the most popular is uWSGI with nginx.
When you upload your code to a remote host, you will need to provide a way to start the server and get things running. How this works is host- and software-dependent. As an example, here is some documentation for how you would fire Flask up on Heroku.
I have a wsgi app with a celery component. Basically, when certain requests come in they can hand off relatively time-consuming tasks to celery. I have a working version of this product on a server I set up myself, but our client recently asked me to deploy it to Cloud Foundry. Since Celery is not available as a service on Cloud Foundry, we (me and the client's deployment team) decided to deploy the app twice – once as a wsgi app and once as a standalone celery app, sharing a rabbitmq service.
The code between the apps is identical. The wsgi app responds correctly, returning the expected web pages. vmc logs celeryapp shows that celery is to be up-and-running, but when I send requests to wsgi that should become celery tasks, they disappear as soon as they get to a .delay() statement. They neither appear in the celery logs nor do they appear as an error.
Attempts to debug:
I can't use celery.contrib.rdb in Cloud Foundry (to supply a telnet interface to pdb), as each app is sandboxed and port-restricted.
I don't know how to find the specific rabbitmq instance these apps are supposed to share, so I can see what messages it's passing.
Update: to corroborate the above statement about finding rabbitmq, here's what happens when I try to access the node that should be sharing celery tasks:
root#cf:~# export RABBITMQ_NODENAME=eecef185-e1ae-4e08-91af-47f590304ecc
root#cf:~# export RABBITMQ_NODE_PORT=57390
root#cf:~# ~/cloudfoundry/.deployments/devbox/deploy/rabbitmq/sbin/rabbitmqctl list_queues
Listing queues ...
=ERROR REPORT==== 18-Jun-2012::11:31:35 ===
Error in process <0.36.0> on node 'rabbitmqctl17951#cf' with exit value: {badarg,[{erlang,list_to_existing_atom,["eecef185-e1ae-4e08-91af-47f590304ecc#localhost"]},{dist_util,recv_challenge,1},{dist_util,handshake_we_started,1}]}
Error: unable to connect to node 'eecef185-e1ae-4e08-91af-47f590304ecc#cf': nodedown
diagnostics:
- nodes and their ports on cf: [{'eecef185-e1ae-4e08-91af-47f590304ecc',57390},
{rabbitmqctl17951,36032}]
- current node: rabbitmqctl17951#cf
- current node home dir: /home/cf
- current node cookie hash: 1igde7WRgkhAea8fCwKncQ==
How can I debug this and/or why are my tasks vanishing?
Apparently the problem was caused by a deadlock between the broker and the celery worker, such that the worker would never acknowledge the task as complete, and never accept a new task, but never crashed or failed either. The tasks weren't vanishing; they were simply staying in queue forever.
Update: The deadlock was caused by the fact that we were running celeryd inside a wrapper script that installed dependencies. (Literally pip install -r requirements.txt && ./celeryd -lINFO). Because of how Cloud Foundry manages process trees, Cloud Foundry would try to kill the parent process (bash), which would HUP celeryd, but ultimately lots of child processes would never die.