I've been working on a Flask app which handles SMS messages using Twilio, stores them in a database, and provides access to a frontend via JSONP GET requests. I've daemonized it using supervisord, which seems to be working pretty well, but every few days it starts to hang (i.e. all requests pend forever or time out) and I have to restart the process. (I've also tried simply running it with nohup, but same problem.) I was suspicious that sqlite3 was somehow blocking occasionally, but my most recent test was to write a request method which didn't involve database access, and that's timing out too. I'm incredibly puzzled -- hopefully you've seen something similar or know what might be causing this.
The relevant code can be found here, and it's currently running (and stalled, as of this post) on my VPS at mattnichols.net:6288
Thanks!
Update: do you think this could be an issue with Flask's dev server? I'd like to believe that wrapping my app with Tornado (or something similar) could solve the problem, but I've also run other things for much longer without problems using the dev server.
For the record, this seems to have been solved by running my app using Tornado instead of the Flask dev server. Wrapping my Flask code into a Tornado server was super easy once I decided to do so: consult http://flask.pocoo.org/docs/deploying/wsgi-standalone/#tornado if you find yourself in my same situation.
Related
I realize similar questions have been asked however they have all been about a sepficic problem whereas I don't even now how I would go about doing what I need to.
That is: From my Django webapp I need to scrape a website periodically while my webapp runs on a server. The first options that I found were "django-background-tasks" (which doesn't seem to work the way I want it to) and 'celery-beat' which recommends getting another server if i understood correctly.
I figured just running a seperate thread would work but I can't seem to make that work without it interrupting the server and vice-versa and it's not the "correct" way of doing it.
Is there a way to run a task periodically without the need for a seperate server and a request to be made to an app in Django?
'celery-beat' which recommends getting another server if i understood correctly.
You can host celery (and any other needed components) on the same server as your Django app. They would be separate processes entirely.
It's not an uncommon setup to have a Django app + celery worker(s) + message queue all bundled into the same server deployment. Deploying on separate servers may be ideal, just as it would be ideal to distribute your Django app across many servers, but is by no means necessary.
I'm not sure if this is the "correct" way but it was a cheap and easy way for me to do it. I just created custom Django Management Commands and have them run via a scheduler such as CRON or in my case I just utilized Heroku Scheduler for my app.
I've been trying websocket-client and socketio-client with no luck so far. The broad picture of what I want to accomplish is this:
Currently, I have a Flask Rest API that has both a web front-end and a command line interface, and it handles several different sets of file uploads/downloads. Both communicate with the server using HTTP requests, the web one from JQuery AJAX and the CLI uses python requests. I would like to switch to using sockets so that database changes from one client appear on all of them. I have been able to get Flask-SocketIO working between my JQuery and Flask server, but I'm struggling with getting any client libraries working from the CLI portion. Is there an easy to use python library for sockets similar to requests I should be using for this transition, or am I going in a totally wrong direction with making this switch?
Another option, unsure of the viability, would be to try and keep both the REST API for the CLI and have sockets for the web interface. Sounds very messy though.
After doing a lot of searching and messing around with various libraries, the one that was easiest to get up and running connecting a command line tool with a Flask-SocketIO webapp was socketIO-client.
This repository came in handy for the issues where I was struggling to understand how to correctly use the waits to receive info on the client side.
Once I've finished the project in a few weeks, I will come back and add more details so people finding this in the future can have an easier time getting this set up.
I have an existing app running uwsgi/flask behind nginx. I want to integrate websockets into the same app. Flask sockets looks promising, so I'm trying that.
Flask sockets briefly mentions running gevent-websocket, but no sample code is provided to integrate an existing app into this setup. I've also tried several combinations of gevent-websocket with my existing flask app, but I still keep getting 500s. Additionally, it would be nice if I could find out the nature of the 500s, but the errors don't show up in the uwsgi log, as they do for the existing flask app. So, if you have any debugging dips to show a tracedump, that'd be great.
Can anyone help me get a simple echo route working with this setup? Thanks in advance.
FYI, I'm currently using:
uWSGI 1.0.3-debian
python 2.7.3
Flask 0.10.1
nginx 1.4.3
Flask Sockets 0.1
If you upgrade to uWSGI 2.0 you get websockets support for free without the need of additional frameworks and with gevent compatibility. More important you get a really fast implementation (suitable for gaming):
http://uwsgi-docs.readthedocs.org/en/latest/WebSockets.html
Although #roberto's solution is likely viable, it's a little too soon from the last major release for me to push it out to production.
I just went back to using socket.io and node in the interim. As time passes, maybe his suggestion will be more viable for me.
I've already looked at other threads on this, but most don't go into enough setup detail which is where I need help.
I have an Ubuntu based VPS running with nginx, serving PHP sites through php-cgi on port 9000.
I'd like to start doing more with Python, so I've written a deployment script which I essentially want to use as a post-receive hook on my local GitLab server as my first python script. I can run this script successfully by running python script.py on the command line but in order to use this as a post-receive hook I need it be able to access it via http.
I looked at this guide on the nginx wiki but partway down is says to:
And start the django fastcgi process:
python ./manage.py runfcgi host=127.0.0.1 port=8080
Now, like I said I am pretty new to python, and I have never used the Django framework. Can anyone assit on how I am supposed to start the fastcgi server? Do I replace ./manage.py with the name of my script? Any help would be appriciated as everything I've found online refers to working with Django.
Do I replace ./manage.py with the name of my script?
No. It's highly unlikely your script is a FastCGI server, or that it can accept HTTP requests of any kind since you mention running it over the command line. (From what little I know of FastCGI, an app supporting it has to be able to handle a stream of requests coming in over stdin in a specific format, so there's definitely some plumbing involved.)
I'd say the easiest approach would be to use some web framework just to act as HTTP/FastCGI middleware. For your use a "microframework" like Flask (or even Paste but I found the documentation inscrutable) sounds like it'd work fine. The idea would be to have two interfaces to your main code, one that can handle command line arguments, and one that can handle a HTTP request, ultimately both would just call one function that actually does the work. (If you want to keep the command-line version of the app.)
The Flask documentation also mentions using uWSGI or standalone workers as deployment options. I'm not familiar with the former; the latter I wouldn't recommend for a simple, low-traffic app for the same reasons as the approach in the next paragraph.
Considering you use a VPS, you might even be able to just run the app as a standalone server process using the http.server module, but I'm not sure that's the better choice unless you absolutely want to avoid using any sort of framework. You'd have to make sure the app starts up if the server is rebooted or that it restarts when it crashes and it seems easier to just have nginx do the job of the supervisor.
UPDATE: Scratch that, it seems that nginx won't handle supervising a FastCGI worker process for you, which would've been the main advantage of the approach. In light of that it doesn't matter which of the three approaches you use since you'll have to set up a service supervisor one way or the other. I'd say go with uWSGI since flup (which is needed for Flask+FastCGI) seems abandoned since 2011, and the uWSGI protocol is apparently supported in nginx natively. Otherwise you'd need to use a different webserver than nginx, one that will manage a FastCGI worker for you. If this is an option, I'd consider Cherokee, which can be configured using a web GUI.
tl;dr: you need to write a (very simple) webapp. While it is feasible to do this without a web framework of any kind, in my opinion using one is easier, since you some (nontrivial) plumbing for free and there's a lot of guidance available on how to deploy them.
I'm working with web2py and for some reason web2py seems to fail to notice when code has changed in certain cases. I can't really narrow it down, but from time to time changes in the code are not reflected, web2py obviously has the old version cached somewhere.
The only thing that helps is quitting web2py and restarting it (i'm using the internal server).
Any hints ? Thank you !
web2py does cache your code, except for Google App Engine (for speed). That is not the problem. If you you edit code in models, views or controllers, you see the effect immediately.
The problem may be modules; if you edit code in modules you will not see the effect immediately, unless you import them with local_import('module', reload=True), or by restarting web2py.
Is that is also not your problem, then your browser is caching something. Please bring up this question to the web2py mailing list as we can help more.
P.S. If you are using the latest web2py it no longer comes with cherrypy. The built-in web server is called Rocket.
web2py itself shouldn't "cache" your code, but whatever app server you're using it on surely might. But web2py can be deployed on such a huge variety of app servers that it's impossible to give completely general suggestions.
If you're using the popular cherrypy WSGI server that I believe comes bundled with web2py, for example, see, in cherrypy's own docs, the AutoReload feature. Such features are not recommended in a production deployment (they can require very significant resources), but they sure come in handy when you're just developing!-)