We Keep Coding

Django + Nginx + UWSGI resource not available error - after using max-request = 500 in uwsgi.ini

To overcome the memory usage of the server, we have introduced max-requests = 500 to the uswgi.ini file. We face side effects like below
10.115.21.68 - - [01/Jan/2020:04:09:50 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:09:50 [error] 30522#0: *22690 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:09:48 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:09:48 [error] 30523#0: *22688 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.21.68 - - [01/Jan/2020:04:09:06 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:09:06 [error] 30522#0: *22682 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:08:49 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:08:49 [error] 30523#0: *22680 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.21.68 - - [01/Jan/2020:04:08:23 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:08:23 [error] 30522#0: *22676 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:07:51 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:07:51 [error] 30522#0: *22672 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.21.68 - - [01/Jan/2020:04:07:49 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:07:49 [error] 30522#0: *22670 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:07:07 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:07:07 [error] 30522#0: *22664 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
Following is the current configuration file that I am using on my server (basically configured in AWS Cloudformation template).
files:
/home/uwsgi/.aws/config:
mode: "000444"
/srv/uwsgi.ini:
content: !Sub |
[uwsgi]
socket = /tmp/uwsgi.sock
chmod-socket=777
chdir = /srv/app/${AppDirectory}
pythonpath = .:..:../..
module = ${AppName}.wsgi
env = PYTHONPATH=.:..:../..
enable-threads = true
single-interpreter = true
max-requests = 500
processes = 4
threads = 2
uid = uwsgi
gid = uwsgi
harikari = 7200
master = true
harikari-verbose = true
buffer-size = 8192
logformat = "uwsgi:request %(addr) | %(method) %(uri) | %(status) | %(msecs) | %(ctime) | %(vszM)"
daemonize = /var/log/uwsgi/uwsgi.log
die-on-term = true
for-readline = /srv/env_vars
env = %(_)
endfor =
/etc/init.d/uwsgi:
mode: '000555'
content: |
#!/usr/bin/env bash
# uwsgi - this script starts and stops the uwsgi daemon
#
# chkconfig: - 85 15
# description: uwsgi serves python web apps
# processname: uwsgi
# pidfile: /var/run/uwsgi/uwsgi.pid
#
set -e
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/uwsgi
RUN_DIR=/var/run/uwsgi
LOG_DIR=/var/log/uwsgi
# VASSALS_DIR=/etc/uwsgi/vassals
NAME=uwsgi
DESC=uwsgi
OWNER=uwsgi
GROUP=uwsgi
[[ -x $DAEMON ]] || exit 0
[[ -d $RUN_DIR ]] || mkdir $RUN_DIR && chown $OWNER:$GROUP $RUN_DIR
[[ -d $LOG_DIR ]] || mkdir $LOG_DIR && chown $OWNER:$GROUP $LOG_DIR
do_pid_check()
{
local PIDFILE=$1
[[ -f $PIDFILE ]] || return 0
local PID=$(cat $PIDFILE)
for p in $(pgrep $NAME); do
[[ $p == $PID ]] && return 1
done
return 0
}
do_start()
{
local PIDFILE=$RUN_DIR/$NAME.pid
if do_pid_check $PIDFILE; then
$DAEMON /srv/uwsgi.ini
else
echo "Already running!"
fi
}
send_sig()
{
local PIDFILE=$RUN_DIR/$NAME.pid
set +e
[[ -f $PIDFILE ]] && kill $1 $(cat $PIDFILE) > /dev/null 2>&1
set -e
}
wait_and_clean_pidfile()
{
local PIDFILE=$RUN_DIR/$NAME.pid
until do_pid_check $PIDFILE; do
echo -n "";
done
rm -f $PIDFILE
}
do_stop()
{
send_sig -3
wait_and_clean_pidfile
}
do_reload()
{
send_sig -1
}
do_force_reload()
{
send_sig -15
}
get_status()
{
send_sig -10
}
case "$1" in
start)
echo "Starting $DESC: "
do_start
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
do_stop
echo "$NAME."
;;
reload)
echo -n "Reloading $DESC: "
do_reload
echo "$NAME."
;;
force-reload)
echo -n "Force-reloading $DESC: "
do_force_reload
echo "$NAME."
;;
restart)
echo "Restarting $DESC: "
do_stop
sleep 1
do_start
echo "$NAME."
;;
status)
get_status
;;
*)
N=/etc/init.d/$NAME
echo "Usage: $N {start|stop|restart|reload|force-reload|status}">&2
exit 1
;;
esac
exit 0
services:
sysvinit:
uwsgi:
enabled: "true"
ensureRunning: "true"
files:
- "/srv/uwsgi.ini"
/etc/nginx/conf.d/django.conf:
upstream django {
server unix:///tmp/uwsgi.sock;
}
server {
listen *:443 ssl;
ssl on;
ssl_certificate /srv/certs/main.crt;
ssl_certificate_key /srv/certs/main.key;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
# the domain name it will serve for
charset utf-8;
#Max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /srv/www/media; # your Django project's media files
}
location /static {
alias /var/www/static; # your Django project's static files
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
uwsgi_read_timeout 7201;
client_max_body_size 200M;
# uwsgi_buffering off;
include /etc/nginx/uwsgi_params;
}
}
Questions are
1. what makes this error unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream?
2. What is the workaround to fix the error to avoid the memory consumption of Django App?
Open thoughts.

in nginx.conf
upstream django {
server unix:///tmp/uwsgi.sock;
}
server {
listen *:443 ssl;
ssl on;
ssl_certificate /srv/certs/main.crt;
ssl_certificate_key /srv/certs/main.key;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 10m;
ssl_prefer_server_ciphers on;
# The domain name it will serve for
charset utf-8;
#Max upload size
client_max_body_size 75M; # adjust to taste
# Timeout should be equal to ngix.conf -> time keepalive_timeout 75;
client_header_timeout 75s;
# Django media
location /media {
alias /var/www/media; # your Django project's media files
}
location /static {
alias /var/www/static; # your Django project's static files
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
# when a client closes the connection then keep the channel to uwsgi open. Otherwise uwsgi throws an IOError
uwsgi_ignore_client_abort on;
uwsgi_read_timeout 120s;
uwsgi_send_timeout 120s;
client_max_body_size 200M;
# uwsgi_buffering off;
include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
}
}

Python2 webserver: Do not log request from localhost

The following Python2 webserver will log every single request including the one from localhost (127.0.0.1).
webserver.py
import SimpleHTTPServer, SocketServer, sys
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
port = 80
httpd = SocketServer.TCPServer(("", port), Handler)
sys.stderr = open('/home/user/log.txt', 'w', 1)
httpd.serve_forever()
As example; curl localhost (from the same machine) will produce the following log.
10.0.0.1 - - [10/Jan/2019 00:00:00] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:00:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:01:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:02:02] "GET / HTTP/1.1" 200 -
My question: Would it be possible to make an exception for local request? I don't want to log any request from localhost/127.0.0.1.
I'm thinking something like this but not really sure how to implement it in Python2 yet.
webserver_v2_do_not_log_localhost.py
import webserver.py # webserver code above, or simply paste everything in here.
if SourceIPAddress == 127.0.0.1:
print('DO NOT log request from localhost/127.0.0.1')
# Script here
else:
print('Log everything')
# Script here
Any idea on the scripts would be highly appreciated. Thanks
Desired Output when performing tail -F log.txt (external IP only, not localhost)
10.0.0.1 - - [10/Jan/2019 00:00:00] "GET / HTTP/1.1" 200 -

You can use logging.Filterclass.
When you declare your logger, do something like that:
import logging
logging.basicConfig(filename='myapp.log', level=logging.INFO)
class Global:
SourceIPAddress = ''
class IpFilter(logging.Filter):
def filter(self, rec):#the rec is part of the function signature.
return not Global.SourceIPAddress == '127.0.0.1'
def main():
log = logging.getLogger('myLogger')
log.addFilter(IpFilter())
log.info("log")
Global.SourceIPAddress = '127.0.0.1'
log.info("Don't log")
if __name__ == '__main__':
main()
Of course I implemented it in a very simple way and you should save the IP in a better place(:
I would also check this links for more info:
https://docs.python.org/3/howto/logging-cookbook.html
https://www.programcreek.com/python/example/3364/logging.Filter

Flask-SocketIO socket fails to connect and continuously re-attempts

I have an app that's outputting a JSON through a socket which was working fine in the past, however recently the socket doesn't seem to be establishing and it continuously POST & GET's with the lines:
<user_ip>,<client_ip> - - [17/Jul/2018 12:48:17]"GET /socket.io/?EIO=3&transport=polling......HTTP/1.1" 200 221 0.000000
<user_ip>,<client_ip> - - [17/Jul/2018 12:48:17]"POST /socket.io/?EIO=3&transport=polling......HTTP/1.1" 200 243 0.517600
Additionally, when the WebSocket first tries, I can see from Chromes console the message:
WebSocket connection to ..... failed: Establishing a tunnel via proxy server failed.
I've also had a look through what I think is a similar issue based on socketIO-client but wasn't able to resolve my problem.
Can anyone help with overcoming this connection issue?
Included a full log on running the app below:
Server initialized for eventlet.
* Debugger is active!
* Debugger PIN: 129-744-633
(7616) wsgi starting up on http://0.0.0.0:6328
(7616) accepted (<client_ip>, 50548)
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /<app_url> HTTP/1.1" 200 1664 0.015000
3602c46fa50247eb9d397fda82f3eae8: Sending packet OPEN data {'sid': '3602c46fa50247eb9d397fda82f3eae8', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 0
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /socket.io/?EIO=3&transport=polling&t=1531829903367-0 HTTP/1.1" 200 381 0.000000
(7616) accepted (<client_ip>, 50560)
3602c46fa50247eb9d397fda82f3eae8: Received packet MESSAGE data 0/testnamespace
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 0/testnamespace
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "POST /socket.io/?EIO=3&transport=polling&t=1531829903377-1&sid=3602c46fa50247eb9d397fda82f3eae8 HTTP/1.1" 200 221 0.000000
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /socket.io/?EIO=3&transport=polling&t=1531829903379-2&sid=3602c46fa50247eb9d397fda82f3eae8 HTTP/1.1" 200 226 0.000000
(7616) accepted (<client_ip>, 50562)
3602c46fa50247eb9d397fda82f3eae8: Received packet MESSAGE data 2/testnamespace,["event_1",{"data":"Web app connection successful."}]
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 2/testnamespace,["client_event",{"json":[{"Date&Time":"........}]

Gevent, Flask app stuck while streaming video until client disconnects

I'm using gevent and flask to create a streaming app. I've embedded an img element inside the html page and set its src=/video_feed
The corresponding flask code is
def gen():
global vc
"""Video streaming generator function."""
if vc is None:
vc = cv2.VideoCapture(0)
_, frame = vc.read()
while True:
rval, frame = vc.read()
r, frame = cv2.imencode('.jpg', frame)
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n\r\n' + frame.tobytes() + b'\r\n')
#app.route('/video_feed')
def video_feed():
"""Video streaming route. Put this in the src attribute of an img tag."""
return Response(gen(),
mimetype='multipart/x-mixed-replace; boundary=frame')
The problem: The streaming works fine but the problem is Flask blocks and refuses to accept new requests until the client disconnects!
This is the code I use to start it with gevent
def start_gevent(app_port):
http_server = WSGIServer(('', app_port), app)
http_server.serve_forever()
Here is my app log.
::1 - - [2018-04-27 18:02:56] "GET / HTTP/1.1" 200 1827 0.008464
::1 - - [2018-04-27 18:02:59] "GET /video.html HTTP/1.1" 200 1850 <--streaming starts here
0.001877 [wxpython.py] OnClose called <-- client disconnect
::1 - - [2018-04-27 18:03:01] "GET /video_feed HTTP/1.1" 200 2951956 2.790426 <-- flask log of the request comes late
::1 - - [2018-04-27 18:03:01] "GET /database.html HTTP/1.1" 200 118 0.003394 <-- client made these requests concurrently, but flask didnt respond, it logs them now
::1 - - [2018-04-27 18:03:01] "GET /database.html HTTP/1.1" 200 118 0.000549
Can anyone advice?

Flask's built-in server always 404 with SERVER_NAME set

Here is a minimal example:
from flask import Flask
app = Flask(__name__)
app.config['DEBUG'] = True
app.config['SERVER_NAME'] = 'myapp.dev:5000'
#app.route('/')
def hello_world():
return 'Hello World!'
#app.errorhandler(404)
def not_found(error):
print(str(error))
return '404', 404
if __name__ == '__main__':
app.run(debug=True)
If I set SERVER_NAME, Flask would response every URL with a 404 error, and when I comment out that line, it functions correctly again.
/Users/sunqingyao/Envs/flask/bin/python3.6 /Users/sunqingyao/Projects/play-ground/python-playground/foo/foo.py
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 422-505-438
127.0.0.1 - - [30/Oct/2017 07:19:55] "GET / HTTP/1.1" 404 -
404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
Please note that this is not a duplicate of Flask 404 when using SERVER_NAME, since I'm not using Apache or any production web server. I'm just dealing with Flask's built-in development server.
I'm using Python 3.6.2, Flask 0.12.2, Werkzeug 0.12.2, PyCharm 2017.2.3 on macOS High Sierra, if it's relevant.

When set SERVER_NAME, you should make HTTP request header 'Host' the same with it:
# curl http://127.0.0.1:5000/ -sv -H 'Host: myapp.dev:5000'
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> Host: myapp.dev:5000
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 13
< Server: Werkzeug/0.14.1 Python/3.6.5
< Date: Thu, 14 Jun 2018 09:34:31 GMT
<
* Closing connection 0
Hello, World!
if you use web explorer， you should access it use http://myapp.dev:5000/ and set /etc/hosts file.
It is like the nginx vhost, use Host header to do routing.
I think The SERVER_NAME is mainly used for route map.
you should set host and ip by hand
app.run(host="0.0.0.0",port=5000)
if you not set host/ip but set SERVER_NAME and found it seems to work,because the app.run() have this logic:
def run(self, host=None, port=None, debug=None,
load_dotenv=True, **options):
...
_host = '127.0.0.1'
_port = 5000
server_name = self.config.get('SERVER_NAME')
sn_host, sn_port = None, None
if server_name:
sn_host, _, sn_port = server_name.partition(':')
host = host or sn_host or _host
port = int(port or sn_port or _port)
...
try:
run_simple(host, port, self, **options)
finally:
self._got_first_request = False
At last, don't use SERVER_NAME to set host,ip app.run() used, unless you know its impact on the route map.

Sometimes I find Flask's docs to be confusing (see the quotes above by #dm295 - the meaning of the implications surrounding 'SERVER_NAME' is hard to parse). But an alternative setup to (and inspired by) #Dancer Phd's answer is to specify the 'HOST' and 'PORT' parameters in a config file instead of 'SERVER_NAME'.
For example, let's say you use this config strategy proposed in the Flask docs, add the host & port number like so:
class Config(object):
DEBUG = False
TESTING = False
DATABASE_URI = 'sqlite://:memory:'
HOST = 'http://localhost' #
PORT = '5000'
class ProductionConfig(Config):
DATABASE_URI = 'mysql://user#localhost/foo'
class DevelopmentConfig(Config):
DEBUG = True
class TestingConfig(Config):
TESTING = True

From Flask docs:
the name and port number of the server. Required for subdomain support
(e.g.: 'myapp.dev:5000') Note that localhost does not support
subdomains so setting this to “localhost” does not help. Setting a
SERVER_NAME also by default enables URL generation without a request
context but with an application context.
and
More on SERVER_NAME
The SERVER_NAME key is used for the subdomain
support. Because Flask cannot guess the subdomain part without the
knowledge of the actual server name, this is required if you want to
work with subdomains. This is also used for the session cookie.
Please keep in mind that not only Flask has the problem of not knowing
what subdomains are, your web browser does as well. Most modern web
browsers will not allow cross-subdomain cookies to be set on a server
name without dots in it. So if your server name is 'localhost' you
will not be able to set a cookie for 'localhost' and every subdomain
of it. Please choose a different server name in that case, like
'myapplication.local' and add this name + the subdomains you want to
use into your host config or setup a local bind.
It looks like there's no point to setting it to localhost. As suggested in the docs, try something like myapp.dev:5000.

You can also use just port number and host inside of the app.run like:
app.run(debug=True, port=5000, host="localhost")
Delete:
app.config['DEBUG'] = True
app.config['SERVER_NAME'] = 'myapp.dev:5000'

Using debug=True worked for me:
from flask import Flask
app = Flask(__name__)
app.config['SERVER_NAME'] = 'localhost:5000'
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
app.run(debug=True)