I'm facing an inexplicable problem with uwsgi: Crashs irregular happens when upload a big file size. Scenario:
Context
For a simple wsgi application, here a python Flask application in /home/bastien/Projects/test_upload/wsgi.py:
# -*- coding: utf-8 -*-
from flask import Flask, request
app = Flask(__name__)
#app.route('/', methods=['POST'])
def hello_world():
f = request.files['file'].read()
return 'Hello, World! {}'.format(len(f))
application = app
No crash when:
Use this configuration uwsgi file, /etc/uwsgi/apps-available/test_upload.ini:
[uwsgi]
plugins = python3
chdir = /home/bastien/Projects/test_upload/tracim
home = /home/bastien/Projects/test_upload/venv3.4
module = wsgi
callable = application
enable-threads = true
env = PYTHON_EGG_CACHE=/tmp
limit-post = 0
chmod-socket = 660
vacuum = true
Run uwsgi with:
uwsgi -i /etc/uwsgi/apps-available/test_upload.ini --http-socket :6543
And send file (~262Mo) with /httpie:
http -h -f POST :6543 'file#/home/bastien/Téléchargements/pycharm-professional-2017.2.3.tar.gz'
HTTP request can be repeated, no crash.
Crash when:
Use this configuration uwsgi file, /etc/uwsgi/apps-available/test_upload.ini with symbolic link into /etc/uwsgi/apps-enabled:
[uwsgi]
plugins = python3
chdir = /home/bastien/Projects/test_upload/tracim
home = /home/bastien/Projects/test_upload/venv3.4
module = wsgi
callable = application
http-socket = :4321
enable-threads = true
env = PYTHON_EGG_CACHE=/tmp
limit-post = 0
chmod-socket = 660
vacuum = true
Note: Only difference is http-socket = :4321
Run uwsgi with service uwsgi start (on debian 8.9) and send file with:
http -h -f POST :4321 'file#/home/bastien/Téléchargements/pycharm-professional-2017.2.3.tar.gz'
This request will work one time, sometimes two times:
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 5188
Content-Type: text/html; charset=utf-8
Pragma: no-cache
But finally crash with:
http: error: ConnectionError: ('Connection aborted.', BadStatusLine("''",)) while doing POST request to URL: http://localhost:4321/
Note: Any wsgi application can be used for reproduce
Note: No log is produced by uwsgi or application about this "error"
Summary:
The error is not conistent and only difference is uwsgi usage as service with:
Debian 8.9
uwsgi 2.0.7-1+deb8u1 apt installed
Question
How can produce this difefrence ? Where can i search to know how is started uwsgi by service command ?
Problem solved when use Debian 9 with uwsgi in apt version 2.0.14+20161117-3.
Related
Do we need to use runserver command in production to start our project? If yes, then how to do it, and If No then how does out project server starts?
The most common way starting Django in production for now is using UWSGI (https://uwsgi-docs.readthedocs.io/en/latest/) (or an asynchronous version ASGI)
You need config like this:
[uwsgi]
socket = 0.0.0.0:3300
chdir = /app/src
pythonpath=/app/src
wsgi-file = /app/src/django_app/wsgi.py
env = DJANGO_SETTINGS_MODULE=django_app.settings
module = django.core.wsgi:get_wsgi_application()
max-requests = 1000
harakiri = 60
buffer-size = 65535
no-orphans = true
touch-reload = /var/run/uwsgi-touch-reload
uid = www-data
gid = www-data
master = 1
workers = 8
And then you start your application with:
#!/usr/bin/env bash
uwsgi --ini /app/uwsgi.ini
And then your HTTP server, for example nginx, sends requests to uwsgi application. Nginx config my be like this:
upstream django {
server django:3300;
}
location / {
include uwsgi_params;
default_type text/html;
uwsgi_pass django;
uwsgi_param HTTP_X_FORWARDED_PROTOCOL 'https';
}
I'm desperately trying to figure out the issue of why I cannot deploy my flask application with uWSGI nginx and Ubuntu server.
Here is my error in nginx error log:
2019/05/12 09:00:19 [error] 10154#10154: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 24.27.57.18, server: itinareport.tk, request: "POST /login HTTP/1.1", upstream: "uwsgi://unix:/home/pinchrep2/itinarep/itinarep.sock", host: "www.itinareport.tk", referrer: "http://www.itinareport.tk/login"
Upon having everything setup with flask uWSGI and nginx I'm able to put my domain name in and it takes me to my web page (Login Page). Upon attempting to login the page pauses for some time and then I receive the 504 Gateway Time-Out
Please advise me on what I should do. This is my first time ever deploying an application and it's been taking me about a week of trying different configurations. I should have you know that I have my flask application connecting to a database through ssh tunnel to another server. Not sure if that would cause this issue as well. Please help if possible or point me in the right direction.
Here is my server block configuration file
server{
listen 80;
server_name itinareport.tk www.itinareport.tk;
location / {
include uwsgi_params;
uwsgi_pass unix:/home/pinchrep2/itinarep/itinarep.sock;
}
Here is my service file:
[Unit]
Description=uWSGI instance to serve itinarep
After=network.target
[Service]
User=pinchrep2
Group=www-data
WorkingDirectory=/home/pinchrep2/itinarep
Environment="PATH=/home/pinchrep2/itinarep/it_venv/bin"
ExecStart=/home/pinchrep2/itinarep/it_venv/bin/uwsgi --ini itinarep.ini
[Install]
WantedBy=multi-user.target
Here is my ini file
[uwsgi]
module = wsgi:app
master = true
processes = 5
socket = itinarep.sock
chmod-socket = 660
vacuum = true
die-on-term=true
Sometimes when deploying a flask app with library like Keras and others they have some issue with threading in uWSGI.
You can change a few things in myproject.ini to make it work:
add these setting in myproject.ini
master = false <-------- this
processes = 1 <--------- and this
add cheaper = 0 to your myproject.ini
add lazy-apps = true to your myproject.ini
So now your myproject.ini file may look like:
[uwsgi]
module = wsgi:app
master = false <-------- this
processes = 1 <--------- and this
socket = myproject.sock
chmod-socket = 660
vacuum = true
die-on-term = true
cheaper = 0 <----------- also this, if option 1 is not enough or doesn't work
lazy-apps = true <------ in some case, this might help
I have a Flask python web app on uWSGI/nginx that works fine, except when I use pymongo, specifically when I initialize the MongoClient class. I get the following nginx error when I try to access the app while using pymongo:
019/02/19 21:58:13 [error] 16699#0: *5 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: example.com, request: "GET /api/test HTTP/1.1", upstream: "uwsgi://unix:/var/www/html/myapp/myapp.sock:”, host: “example.com”
My small test app:
from flask import Flask
from flask_cors import CORS
from bson.json_util import dumps
import pymongo
DEBUG = True
app = Flask(__name__)
app.config.from_object(__name__)
CORS(app)
client = pymongo.MongoClient() # This line
db = client.myapp
#app.route('/api/test')
def test():
item = db.items.find_one()
return item['name']
def create_app(app_name='MYAPP'):
return app
# if __name__ == '__main__':
# app.run(debug=True, threaded=True, host='0.0.0.0')
If I run this app from the command line (python app.py) it works fine accessing 0.0.0.0:5000/api/test, so I'm pretty sure it's just a uWSGI configuration issue. My first thought was to increase the uwsgi_read_timeout parameter in my nginx config file:
uwsgi_read_timeout 3600
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name example.com www.example.com;
location /api {
include uwsgi_params;
uwsgi_read_timeout 3600;
uwsgi_pass unix:/var/www/html/myapp/myapp.sock;
}
location / {
root /var/www/html/myapp;
try_files $uri $uri/ /index.html;
}
#return 301 https://$server_name$request_uri;
}
But it had no apparent effect. My uWSGI app is running as a service, using the following config (myapp.ini):
[uwsgi]
module = wsgi:app
master = true
processes = 4
enable-threads = True
socket = /var/www/html/myapp/myapp.sock
chmod-socket = 660
vacuum = true
die-on-term = true
Again, everything seems to work fine except for when I try to initialize pymongo. Finally, my app's service file:
[Unit]
Description=uWSGI Python container server
After=network.target
[Service]
User=pi
Group=www-data
WorkingDirectory=/var/www/html/myapp
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/apps-available/myapp.ini
[Install]
WantedBy=multi-user.target
I believe the issue is that you're forking and this causes issues with PyMongo.
PyMongo is thread safe but not Fork safe. Once you run the app in daemon mode you are forking the process. You'll have to create a MongoClient inside the app so that your threads can see it after the process has started.
You can try this(I didn't try this out, I normally wrap stuff like this in a class and do this in the init method):
def create_app(app_name='MYAPP'):
app.client = pymongo.MongoClient(connect=False) # this will prevent connecting until you need it.
app.db = app.client.myapp
return app
Read this: http://api.mongodb.com/python/current/faq.html#id3
I have my Python Flask web app hosted on nginx. While trying to execute a request it shows a timeout error in the nginx error log as shown below :
[error] 2084#0: *1 upstream timed out (110: Connection timed out)
while reading response header from upstream, client:
192.168.2.224, server: 192.168.2.131, request: "POST /execute HTTP/1.1", upstream: "uwsgi://unix:/hom
e/jay/PythonFlaskApp/app.sock", host: "192.168.2.131:9000", referrer:
"http://192.168.2.131:9000/"
If I try to run the app locally it works fine and responds fine.
Any one have any idea what might be wrong ?
the error found in browser console is :
Gateway Time-out
Here is the nginx config file:
server {
listen 9000;
server_name 192.168.2.131;
location / {
include uwsgi_params;
proxy_read_timeout 300;
uwsgi_pass unix:/home/jay/PythonFlaskApp/app.sock;
}
}
And here is the Python Fabric code that i trying to execute. i'm not sure if this is causing the issue, but any waz here is the code :
from fabric.api import *
#application.route("/execute",methods=['POST'])
def execute():
try:
machineInfo = request.json['info']
ip = machineInfo['ip']
username = machineInfo['username']
password = machineInfo['password']
command = machineInfo['command']
isRoot = machineInfo['isRoot']
env.host_string = username + '#' + ip
env.password = password
resp = ''
with settings(warn_only=True):
if isRoot:
resp = sudo(command)
else:
resp = run(command)
return jsonify(status='OK',message=resp)
except Exception, e:
print 'Error is ' + str(e)
return jsonify(status='ERROR',message=str(e))
I have a uWSGi config file for the web app and started it using an upstart script. Here is uwSGi conf file :
[uwsgi]
module = wsgi
master = true
processes = 5
socket = app.sock
chmod-socket = 660
vacuum = true
die-on-term = true
and here is upstart script
description "uWSGI server instance configured to serve Python Flask App"
start on runlevel [2345]
stop on runlevel [!2345]
setuid jay
setgid www-data
chdir /home/jay/PythonFlaskApp
exec uwsgi --ini app.ini
I have followed the below tutorial on running flask app on nginx
This is likely a problem with the Fabric task, not with Flask. Have you tried isolating / removing Fabric from the application, just for troubleshooting purposes? You could try stubbing out a value for resp, rather than actually executing the run/sudo commands in your function. I would bet that the app works just fine if you do that.
And so that would mean that you've got a problem with Fabric executing the command in question. First thing you should do is verify this by mocking up an example Fabfile on the production server using the info you're expecting in one of your requests, and then running it with fab -f <mock_fabfile.py>.
It's also worth noting that using with settings(warn_only=True): can result in suppression of error messages. I think that you should remove this, since you are in a troubleshooting scenario. From the docs on Managing Output:
warnings: Warning messages. These are often turned off when one expects a given operation to fail, such as when using grep to test existence of text in a file. If paired with setting env.warn_only to True, this can result in fully silent warnings when remote programs fail. As with aborts, this setting does not control actual warning behavior, only whether warning messages are printed or hidden.
As a third suggestion, you can get more info out of Fabric by using the show('debug') context manager, as well as enabling Paramiko's logging:
from fabric.api import env
from fabric.context_managers import show
# You can also enable Paramiko's logging like so:
import logging
logging.basicConfig(level=logging.DEBUG)
def my_task():
with show('debug'):
run('my command...')
The Fabric docs have some additional suggestions for troubleshooting: http://docs.fabfile.org/en/1.6/troubleshooting.html. (1.6 is an older/outdated version, but the concepts still apply.)
I am trying to run cherrypy with cherrypy.engine.start instead of cherrypy.quickstart. That's because I want to run cherrypy in non blocking state to start and stop a web server within my functional tests with py.test.
This works fine:
cherrypy.quickstart(WebServerTest(None), config=testconf)
The response to a curl is:
curl --head http://127.0.0.1:1026/index HTTP/1.1 200 OK
Date: Thu, 08 Aug 2013 12:54:37 GMT
Content-Length: 0
Content-Type: text/html;charset=utf-8
Server: CherryPy/3.2.2
But it's blocking the rest of the script to execute.
However this does not work:
testconf = path.join(path.dirname(__file__), 'webservertest.conf')
web_server = WebServerTest(None)
cherrypy.tree.mount(web_server, "", config=testconf)
cherrypy.engine.start()
time.sleep(60)
cherrypy.engine.stop()
The response to a curl is:
curl --head http://127.0.0.1:1026/index
curl: (7) couldn't connect to host
Adding cherrypy.engine.block() aftet cherrypy.engine.start does not solve the problem.
So how can I make it work with cherrypy.engine.start()?
The webservertest.conf config file is:
[global]
server.socket_host = "127.0.0.1"
server.socket_port = 1026
server.thread_pool = 10
You also need to pass the conf to cherrypy.config.update(conf). This is for global config (including your server host and port), whereas the tree.mount call only sets config for that particular app. Read the source code of quickstart to see all the gory details.