We Keep Coding

uwsgi + nginx + flask + emperor: (104: Connection reset by peer) while reading response header from upstream

I am running a flask application using uwsgi and using nginx to serve the content.
vassal ini file:
[uwsgi]
chdir = /var/www/html/PyNLP/
;emperor = /var/www/html/PyNLP/
module = nltk_endpoint:app
master = true
processes = 5
socket = /tmp/PyNLP.sock
virtualenv = /var/www/html/PyNLP/nltkenv
mount = /PyNLP=nltk_endpoint:app
chmod-socket = 666
uid = www-data
gid = www-data
vacuum = true
die-on-term = true
enable-threads = true
;harakiri = 300
;buffer-size=65535
The project is contained in /var/www/html/PyNLP and contains the file nltk_endpoint.py along with the above .ini file
Now, when I run it without emperor:
uwsgi --ini PyNLP.ini --manage-script-name
It seems to be working fine when I try to access this flask API via a postman POST request. The URL looks like:
http://myIP:81/PyNLP/spellCheck
The first part /PyNLP is the mount point for the flask application and the last part /spellCheck is an app route in the project:
#app.route("/spellCheck", methods=["POST"])
def spellCheck():
req = request.get_json()
resp = jsonify(spell_check(req['text']))
return resp
The nginx config looks like:
location = /PyNLP { rewrite ^ /PyNLP/; }
location /PyNLP { try_files $uri #PyNLP; }
location #PyNLP {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
#try_files $uri $uri/ =404;
include uwsgi_params;
uwsgi_pass unix:/tmp/PyNLP.sock;
}
Without emperor this works just fine. But when I try to use:
uwsgi --emperor emperor/emperor.ini --manage-script-name
I get a 404 error for the url path on postman
Emperor ini file:
[uwsgi]
;emperor-tyrant = true
;emperor = %dvassals
;emperor-pidfile = %demperor.pid
;emperor-stats = %demperor.stat.sock
;vassals-include = %dvassals-default.ini
;touch-logrotate = %p
;touch-reload = %p
;touch-reload = %dvassals-default.ini
;log-date = true
;log-truncate = true
;logto =
emperor = /var/www/html/PyNLP/
;chdir = /var/www/html/PyNLP/
;mount = /PyNLP=nltk_endpoint:app
;buffer-size=65535
uid = www-data
gid = www-data
vacuum = true
die-on-term = true
uwsgi log for this request was :
[pid: 9812|app: 0|req: 1/1] 111.125.198.234 () {48 vars in 841 bytes} [Tue Jan 23 10:49:17 2018] POST /PyNLP/spellCheck => generated 233 bytes in 5 msecs (HTTP/1.1 404) 2 headers in 72 bytes (1 switches on core 0)announcing my loyalty to the Emperor...
and nginx/error.log as :
*43 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 111.125.198.234, server: _, request: "POST /PyNLP/spellCheck HTTP/1.1", upstream: "uwsgi://unix:/tmp/PyNLP.sock:", host: "XX.XX.XX.XX:81"
Similar questions on stackoverflow suggested either to change buffer-size in ini files, which did not help as well as to verify if post body is handled correctly, which I think is fine as well, since it worked with POST request just fine without emperor. I suspect it may be a mount or url path issue or a permission issue, but cant figure out why this wont work with emperor. Any ideas? Thanks in advance.

POST to Flask from espduino timing out

I'm playing with an esp8266 providing WiFi to an Arduino with this library. I have it set up properly to POST to Pushover as well as requestbin, and between the debug output and the interface from these tools, I can verify that the request data is being POSTed properly, and the esp8266 / arduino unit's response status code properly shows 200.
I wanted to test the reliability of the setup, so I figured I'd turn to Flask. I put in a for loop to POST 100 requests from the espduino to a Flask app running at 0.0.0.0:5000. The POST contains a test string (trying to roughly gauge simple data integrity, make sure the string comes through uncorrupted) as well as the number of the loop that is being sent (e.g. 0 on the first loop ... 99 on the last). Flask keeps track of these requests and updates its output to show which requests came through properly and which did not.
The setup is working great except for one thing: the module is timing out after each request. Each POST appears to work, Flask updates the output appropriately, but the espduino takes its full 5 seconds (default timeout) for each request and says it got a response code of 0.
So to reiterate the situation: my espduino setup properly POSTs and gets a 200 response from requestb.in and pushover.net, but with my local Flask server POSTs and then times out.
WHYT:
Make sure all form data is read by Flask -> no difference
Serve with gunicorn instead of the built-in Flask server -> no difference
Change response content type explicitly to "text/html" -> no difference
Change response content type to "application/json" -> no difference
Add ; charset=utf-8 to content type -> no difference
Change Flask api endpoint from index / to /api -> no difference
Change port -> no difference
Verify Firewall is off (OS X 10.10.5 hosting the Flask app) -> no difference
nc -l 5000 and check the incoming post from the module (looks good)
Post from Postman and curl to the Flask app to inspect response (and compare with responses from Requestb.in / sites that work)
Removed special chars from the test_string
Change WSGIRequestHandler.protocol_version to HTTP/1.1 -> no difference
I've spent a couple days working on this and not made much progress. The biggest breakthrough I had today was that I can successfully post and get 200 responses if I put my gunicorn / Flask setup behind nginx, with no changes to the espduino code, so I'm sure there's something that Flask is or isn't doing (I was concerned it might be the espduino's processing of an IP address vs domain name, but I think this rules that out).
Summary of the setups I've tried:
POST to requestb.in -> POST works, 200 response, no timeout
POST to pushover.net -> POST works, 200 response, no timeout
POST to local Flask server -> POST works, no response, times out
GET to local Flask server -> no response, times out (have not tested if GETs data)
POST to local gunicorn serving Flask app -> POST works, no response, times out
POST to local nginx serving gunicorn / Flask -> POST works, 200 response, no timeout
POST to local gunicorn serving httpbin -> no response, times out
POST to local cherrypy server -> no response, times out
Current code:
from flask import Flask, request, make_response
from datetime import datetime
app = Flask(__name__)
ESPDUINO_IP = 'XXX.XXX.XXX.XXX'
INCOMING_TEST_STRING = 'This is my espduino test string!'
incoming_requests = []
with open('results.txt', 'w') as f:
f.write("Test run start: {}\n".format(datetime.now()))
#app.route('/api', methods=['GET', 'POST'])
def count_requests():
if request.method == 'POST':
form = request.form
incoming_ip = request.remote_addr
if incoming_ip == ESPDUINO_IP and form['test_string'] == INCOMING_TEST_STRING:
test_num = int(form['test_num'])
incoming_requests.append(test_num)
msg = "All is peachy!"
with open('results.txt', 'a') as f:
f.write("{:02d}: {}\n".format(test_num, datetime.now()))
else:
msg = "Hey, you're not the espduino!<br>"
msg += str(len(incoming_requests)) + " requests so far.<br>"
missing = set(range(100)) - set(incoming_requests)
msg += "Missing: {}<br>".format(', '.join(map(str, missing)) if missing else "None!")
msg += '<br>'.join(map(str, incoming_requests))
resp = make_response('{"this": "that"}')
resp.headers['Content-Type'] = "application/json"
return resp
# return "<html><body>{}</body></html>".format(msg)
if __name__ == '__main__':
app.run(host='0.0.0.0', debug=True)
Here's what the POST from the espduino looks like:
$ nc -l 5000
POST /api HTTP/1.1
Host: XXX.XXX.XXX.XXX
Content-Length: 55
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: ESPDRUINO#tuanpm
test_string=This is my espduino test string!&test_num=0
As compared to curl -X POST -d 'test_string=This is my espduino test string!&test_num=0' localhost:5000/api:
$ nc -l 5000
POST /api HTTP/1.1
Host: localhost:5000
User-Agent: curl/7.43.0
Accept: */*
Content-Length: 55
Content-Type: application/x-www-form-urlencoded
test_string=This is my espduino test string!&test_num=0
Would love to hear any ideas on what may be going on. I'm wondering if this might be a WSGI problem?
Update Aug 31, 2015:
I still haven't figured this out, but it's definitely not a problem specific to Flask. As I noted above, I've also replicated the timeout with CherryPy, as well as with python3 -m http.server --bind 0.0.0.0 5000 (and changing the espduino code to GET /), as well as with ruby -run -e httpd. I still don't understand why nginx, requestbin, etc. serve it with no problems.
In response to #Miguel's comment about the HOST header not having the port, I'm working on forking and building the firmware to change this, but in the meantime I hard-coded the client host and port into a little HTTP server script with no luck.
from http.server import BaseHTTPRequestHandler, HTTPServer
class MyServer(BaseHTTPRequestHandler):
# protocol_version = 'HTTP/1.1'
# close_connection = True
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
def do_GET(self):
self._set_headers()
self.wfile.write(b"<html><body><h1>hi!</h1></body></html>")
def do_HEAD(self):
self._set_headers()
def do_POST(self):
self.client_address = ("192.168.0.4", 5000)
self._set_headers()
self.wfile.write(b"<html><body><h1>POST!</h1></body></html>")
# import pdb; pdb.set_trace()
def run(server_class=HTTPServer, handler_class=MyServer, port=5000):
server_address = ('0.0.0.0', port)
httpd = server_class(server_address, handler_class)
print('Starting httpd...')
httpd.serve_forever()
if __name__ == "__main__":
run()
Looking through tcpdump to see if I can find any difference between the working (nginx) and nonworking network data. Haven't found anything so far, but I'm also new to the tool.
Update Sep 08, 2015
Still haven't figured this out, but it looks like the tcpdump is significantly different between the nginx and Python servers. Here is an example POST and response -- I have replaced the IPs with ESPDUINO_IP and OSX_IP for clarity, and cleaned up the surrounding ACK calls and such. I need to look into why the Python response is getting interrupted by that odd line -- I examined 10+ consecutive POST / Response pairs, and every one of the Python responses was interrupted like that (in exactly the same place), and none of the nginx responses were, so I wonder if that might be the problem. (Also, as you can see, during this round of testing I had changed the response body to text instead of JSON -- no change in results.)
nginx (works)
POST /api HTTP/1.1
Host: OSX_IP
Content-Length: 29
Connection: close
Content-Type: application/x-www-form-urlencoded; charset=utf-8
User-Agent: espduino#n8henrie
test_string=simple&test_num=0
09:16:04.079291 IP OSX_IP.commplex-main > ESPDUINO_IP.49146: Flags [P.], seq 1:183, ack 211, win 65535, length 182
HTTP/1.1 200 OK
Server: nginx/1.8.0
Date: Mon, 31 Aug 2015 15:16:04 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 26
Connection: close
<html><body></body></html>
Flask (times out)
POST /api HTTP/1.1
Host: OSX_IP
Content-Length: 29
Connection: close
Content-Type: application/x-www-form-urlencoded; charset=utf-8
User-Agent: espduino#n8henrie
test_string=simple&test_num=3
09:00:19.424086 IP OSX_IP.commplex-main > ESPDUINO_IP.48931: Flags [P.], seq 1:18, ack 211, win 65535, length 17
HTTP/1.0 200 OK
09:00:36.382125 IP OSX_IP.commplex-main > ESPDUINO_IP.48931: Flags [FP.], seq 18:181, ack 211, win 65535, length 163
E....F#.#..,...e.......#...k..S.P.......Content-Type: text/html; charset=utf-8
Content-Length: 26
Server: Werkzeug/0.10.4 Python/3.4.3
Date: Mon, 31 Aug 2015 15:00:36 GMT
<html><body></body></html>
It looks to me like Python is breaking the response into two halves for some reason, e.g. one part of length 17 and another of length 163, as compared to nginx's single response of length 182.
Update Sep 10, 2015
Interestingly, everything works as expected if I run it through mitmproxy -- even directly to the Flask app without nginx or gunicorn. As soon as I remove mitmproxy, it's back to the timeouts as per above.

Still haven't fixed the problem, but I think I may have figured out what's causing it. Not a Flask problem after all.
Unfortunately this seems to be a bug with the esp_bridge library whose firmware espduino uses in the esp8266. Pardon the likely incorrect terminology, but from what I can tell for some reason it doesn't seem to be joining TCP packets. The servers that produce an HTTP response that is split into separate TCP packets (e.g. Flask) are failing, while tcpdump can verify that nginx and mitmproxy are joining the split TCP packets and returning a response in a single packet, which is why they are working.
https://github.com/tuanpmt/esp_bridge/issues/10
Update 20160128
I revisited this issue today and found a workaround. While the ideal solution would be to fix esp_bridge to reassemble multi-packet reponses, as long as the response is quite small one can force Flask to write the responses in a single packet.
from werkzeug.serving import WSGIRequestHandler
# Your Flask code here...
if __name__ == "__main__":
WSGIRequestHandler.wbufsize = -1
app.run()

Flask change the server header

I've made a simple flask application:
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
host:google.be
HTTP/1.0 404 NOT FOUND
Content-Type: text/html
Content-Length: 233
Server: Werkzeug/0.9.6 Python/2.7.6
Date: Mon, 08 Dec 2014 19:15:43 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
Connection closed by foreign host.
One of the things I would like the change is the server header which at the moment is set as Werkzeug/0.9.6 Python/2.7.6 to something my own chosing. But I can't seem to find anything in the documentation on how to do this.

You can use Flask's make_response method to add or modify headers.
from flask import make_response
#app.route('/index')
def index():
resp = make_response("Hello, World!")
resp.headers['server'] = 'ASD'
return resp

#bcarroll's answer works but it will bypass other processes defined in original process_response method such as set session cookie.
To avoid the above:
class localFlask(Flask):
def process_response(self, response):
#Every response will be processed here first
response.headers['server'] = SERVER_NAME
super(localFlask, self).process_response(response)
return(response)

You can change the Server header for every response by overriding the Flask.process_response() method.
from flask import Flask
from flask import Response
SERVER_NAME = 'Custom Flask Web Server v0.1.0'
class localFlask(Flask):
def process_response(self, response):
#Every response will be processed here first
response.headers['server'] = SERVER_NAME
return(response)
app = localFlask(__name__)
#app.route('/')
def index():
return('<h2>INDEX</h2>')
#app.route('/test')
def test():
return('<h2>This is a test</h2>')
http://flask.pocoo.org/docs/0.12/api/#flask.Flask.process_response

Overriding Server header in code does not work if You use production server like gunicorn. The better way is to use proxy server behind gunicorn and there change Server header.

TL;DR - overwrite /python3.8/http/server.py send_response method. Comment the server header addition line.
Why?
Adding/Manipulating headers in flask (in any way that mentioned above) will fire the response with the configured headers from flask to the web server but the WSGI logic (which happens independently, after & before flask logic) will be the last one to modify those values if any.
In your case(Werkzeug) some headers are hard-coded in python http module which werkzeug depending on. The server header is one of them.

Easy way:
#app.after_request
def changeserver(response):
response.headers['server'] = SERVER_NAME
return response

certain page of Flask web app goes down after a couple hours

I am hosting a Flask app with nginx and uWSGI on a CentOS box. After a few hours, one of my pages goes down with a 500 internal server error Do you see anything jump out at you from my configs below?
My ers_portal_nginx.conf:
server {
listen 80;
server_name www.mydomain.com;
location / { try_files $uri #app; }
location #app {
include uwsgi_params;
uwsgi_pass unix:/home/metheuser/webapps/ers_portal/run_web_uwsgi.sock;
}
}
My ers_portal_uwsgi.ini:
[uwsgi]
#user info
uid = metheuser
gid = ers_group
#application's base folder
base = /home/metheuser/webapps/ers_portal
#python module to import
app = run_web
module = %(app)
home = %(base)/ers_portal_venv
pythonpath = %(base)
#socket file's location
socket = /home/metheuser/webapps/ers_portal/%n.sock
#permissions for the socket file
chmod-socket = 666
#uwsgi varible only, does not relate to your flask application
callable = app
#location of log files
logto = /home/metheuser/webapps/ers_portal/logs/%n.log
Relevant parts of my views.py
data_modification_time = None
data = None
def reload_data():
global data_modification_time, data, sites, column_names
filename = '/home/metheuser/webapps/ers_portal/app/static/' + ec.dd_filename
mtime = os.stat(filename).st_mtime
if data_modification_time != mtime:
data_modification_time = mtime
with open(filename) as f:
data = pickle.load(f)
return data
#a bunch of authentication stuff...
#app.route('/')
#app.route('/index')
def index():
return render_template("index.html",
title = 'Main',)
#app.route('/login', methods = ['GET', 'POST'])
def login():
login stuff...
#app.route('/my_table')
#login_required
def my_table():
print 'trying to access data table...'
data = reload_data()
return render_template("my_table.html",
title = "Rundata Viewer",
sts = sites,
cn = column_names,
data = data) # dictionary of data
I installed nginx via yum (yesterday)
I am using uWSGI installed in my venv via pip
I am on CentOS 6
I have everything set up very similar to here, meaning all the configs are in my app folder and /etc/nginx/nginx.conf is symlinked to.
My uwsgi log shows:
Wed Jun 11 17:20:01 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET /whm-server-status (127.0.0.1)
IOError: write error
[pid: 9586|app: 0|req: 135/135] 127.0.0.1 () {24 vars in 292 bytes} [Wed Jun 11 17:20:01 2014] GET /whm-server-status => generated 0 bytes in 3 msecs (HTTP/1.0 404) 2 headers in 0 bytes (0 switches on core 0)
When its working, the print statement in the views "my_table" route prints into the log file. But not once it stops working.
Any ideas?

logging errors with flask

I'm trying to log an error in a decorator function using app.logger.error(''), but it just doesn't work. In addition I cant debug this well and I can only see the response from the http client:
(I'm using nginx+uwsgi+flask)
HTTP/1.1 502 Bad Gateway
Server: nginx
Date: Sun, 12 Aug 2012 15:45:09 GMT
Content-Type: text/html
Content-Length: 14
Connection: keep-alive
Everything works great with out the line: app.logger.error('panic !!!')
def mydecorator():
def decorator(f):
def wrapped_function(*args, **kwargs):
try:
ip = Mytable.query.filter_by(ip=request.remote_addr).first()
except:
app.logger.error('panic !!!')
else:
dootherthing()
resp = make_response(f(*args, **kwargs))
h = resp.headers
h['add-this-header'] = ":)"
return resp
return update_wrapper(wrapped_function, f)
return decorator
It seems that it is out of context or something.

in fact, the decorator wasnt able to detect the app instance out of context, i solve this using current_app:
1st. Import the method: from flask import current_app
2nd. append any app class to current_app: current_app.logger.error('panic !!!')
info # http://flask.pocoo.org/docs/api/#flask.current_app
"Points to the application handling the request. This is useful for
extensions that want to support multiple applications running side by
side. This is powered by the application context and not by the
request context, so you can change the value of this proxy by using
the app_context() method."

Is app defined anywhere in the script that you've posted?
Also, to help with debugging, when testing you should consider using the run() method with debug mode.
app.run(debug=True)