We Keep Coding

I am working on a web app using Django, running on Nginx, and I am getting 111 connection refused, and I cant work out why

I have previously configured an app in the exact same way but when I updated Ubuntu, I broke my python install. I fixed python, and tried to set up another app in the exact same way, using the same tutorials, but I am getting a 111 error when I try to connect. I can connect if I run the app using gunicorn (which I only tried because it wasn't working using nginx), but I want to find the problem and fix it. As far as I can see everything is configured according to the documentation I went through.
Here is the uwsgi config file
# glossbox_uwsgi.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = /usr/local/apps/glossbox/glossbox
# Django's wsgi file
wsgi-file = /usr/local/apps/glossbox/glossbox/glossbox/wsgi.py
# the virtualenv (full path)
home = /usr/local/apps/glossbox/venv
plugins = python
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe
socket = server 127.0.0.1:8001
# ... with appropriate permissions - may be needed
chmod-socket = 664
clear environment on exit
vacuum = true
here is the nginx conf file:
# glossbox_nginx.conf
# the upstream component nginx needs to connect to
upstream django {
# server unix:///path/to/your/mysite/mysite.sock; # for a file socket
server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}
# configuration of the server
server {
# the port your site will be served on
listen 80;
# the domain name it will serve for
server_name vps746196.ovh.net; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /usr/local/apps/glossbox/glossbox/media; # your Django project's media files - amend as required
}
location /static {
alias /usr/local/apps/glossbox/glossbox/static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
include /usr/local/apps/glossbox/glossbox/uwsgi_params; # the uwsgi_params file you installed
}
}
the app runs using the command 'uwsgi --ini glossbox_uwsgi.ini' but I get 111 connection refused when I try to connect.
I tried to run using 'gunicorn glossbox.wsgi:application --bind 0.0.0.0:8000' just to see if it worked, and it did, although site couldn't find the static files for the admin site, I assume that is down to me having no configuration set up for gunicorn. Either way, I need to find out why it is refusing the connection.
Any help will be appreciated :-)

Not able to check the health of the flask application which inside the dockerised NGINX

I have the python flask application which i am using inside the NGINX and i have dockerised this image. By default flask application runs in 5000 port and NGINX in 80 port. If I run image in the container all the services are working fine. I am able to access the services from NGINX port 80 which internally mapped to flask 5000 port.
Now I want to add the health check for this image. So i am using the py-healthcheck module in the flask application like this.
health = HealthCheck()
def redis_available():
return True, "UP"
health.add_check(redis_available)
app.add_url_rule("/health", "healthcheck", view_func=lambda: health.run())
Now If i run only the flask application(without NGINX in my local system) using the URL
http://localhost:5000/health
I am getting the proper response saying the applicarion is up.
In order to add the healthcheck for image i have adde this command in Dockerfile
HEALTHCHECK --interval=30s --timeout=120s --retries=3 CMD wget --no-check-certificate --quiet --tries=1 --spider https://localhost:80/health || exit 1
Here i am assuming that i am trying to access the healthcheck endpoint from NGINX thats why i am using localhost:80. But if i run the conainer the container is always unhealthy but all the end points are working fine. Whether i have to do some configuration in NGINX conf file in order access the healthcheck endpoint of flask from NGINX?
Here is the nginx config:
# based on default config of nginx 1.12.1
# Define the user that will own and run the Nginx server
user nginx;
# Define the number of worker processes; recommended value is the number of
# cores that are being used by your server
# auto will default to number of vcpus/cores
worker_processes auto;
# altering default pid file location
pid /tmp/nginx.pid;
# turn off daemon mode to be watched by supervisord
daemon off;
# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;
# events block defines the parameters that affect connection processing.
events {
# Define the maximum number of simultaneous connections that can be opened by a worker process
worker_connections 1024;
}
# http block defines the parameters for how NGINX should handle HTTP web traffic
http {
# Include the file defining the list of file types that are supported by NGINX
include /opt/conda/envs/analytics_service/etc/nginx/mime.types;
# Define the default file type that is returned to the user
default_type text/html;
# Don't tell nginx version to clients.
server_tokens off;
# Specifies the maximum accepted body size of a client request, as
# indicated by the request header Content-Length. If the stated content
# length is greater than this size, then the client receives the HTTP
# error code 413. Set to 0 to disable.
client_max_body_size 0;
# Define the format of log messages.
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
# Define the location of the log of access attempts to NGINX
access_log /opt/conda/envs/analytics_service/etc/nginx/access.log main;
# Define the location on the file system of the error log, plus the minimum
# severity to log messages for
error_log /opt/conda/envs/analytics_service/etc/nginx/error.log warn;
# Define the parameters to optimize the delivery of static content
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# Define the timeout value for keep-alive connections with the client
keepalive_timeout 65;
# Define the usage of the gzip compression algorithm to reduce the amount of data to transmit
#gzip on;
# Include additional parameters for virtual host(s)/server(s)
include /opt/conda/envs/analytics_service/etc/nginx/conf.d/*.conf;
}

502 bad gateway error with upstream prematurely closed connection?

I am trying to run an machine learning inference server on docker container with AWS sagemaker , Flask, Nginx and Gunicorn. I have tried running with a c5.xlarge instance and c5.4xlarge instance on AWS sagemaker and it always breaks when run on a c5.xlarge instance.
When the request comes to check health of the application by loading the ML model which is around 300 mb. When inference endpoint is called, it checks if the model is up and running in the worker and if not get the ML model up and then run the prediction with data. I usually call the model with <=5MB data .
Nginx Config:
worker_processes auto;
daemon off; # Prevent forking
pid /tmp/nginx.pid;
error_log /var/log/nginx/error.log;
events {
# defaults
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log combined;
upstream gunicorn {
server unix:/tmp/gunicorn.sock;
}
server {
listen 8080 deferred;
client_max_body_size 5m;
keepalive_timeout 10000;
location ~ ^/(ping|invocations) {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://gunicorn;
}
location / {
return 404 "{}";
}
}
}
gunicorn :
subprocess.Popen(['gunicorn',
'--timeout', str(model_server_timeout),
'-k', 'gevent',
'-b', 'unix:/tmp/gunicorn.sock',
'-w', str(model_server_workers),
'--error-logfile', '-',
'--access-logfile', '-',
'--preload',
'wsgi:app'])
I have looked at the timeout (it is already set at 60 secs for gunicorn), tried preloading the app, and the logs thrown to stdout have only upstream prematurely closed connection while reading response in the error.

How long does your container usually respond to requests? If you use the container in a hosted Endpoint, the container has to respond to requests within 60 seconds. It might be helpful to set the gunicorn timeout to be a little lower than 60 seconds.
https://docs.aws.amazon.com/sagemaker/latest/dg/API_runtime_InvokeEndpoint.html
It looks like the response time depends on the instance type. If this is the case, and you do want to use c5.xlarge instance type for example, you can try to create a batch transform job instead of using a real time inference endpoint. Batch transform job does allow >60 seconds response time for each request.
https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTransformJob.html
Hope this helps!
-Han

Empty REMOTE_ADDR value in Django application, when using nginx as reverse proxy with gunicorn

I have nginx set up as reverse proxy, with gunicorn in the background. This web-server set up feeds my Django app, that has a postgresql backend. The whole set up is hosted on two Ubuntu machines (one's the application machine, other's the DB machine).
I tested this set up solely via gunicorn, without nginx. Worked perfectly. Next to get it up for production, I added the nginx reverse proxy in front of gunicorn. Immediately I ran into a debilitating error: invalid input syntax for type inet: "" (comes when a user tries to log into my Django app)
The IPs of users who log into my app are saved in a session table; Django does that on its own. Now it's a known fact that Postgresql requires all client IPs to be of the INET sort (some other DBs allow string IPs too, but not postgres). INET type doesn't allow "" (i.e. empty) values, and instead throws an error invalid input syntax for type inet: "".
In other words, my nginx reverse proxy is not sending the value for REMOTE_ADDR to the Django app. Solely using gunicorn correctly sets that value (and hence everything works). How do I get nginx to pass an $remote_addr value to REMOTE_ADDR in Django's request.META?
I've tried including proto_set_header REMOTE_ADDR $remote_addr; in the location block in my /etc/nginx/sites-avaialble/myproject file. It does NOT work - I can see a HTTP_REMOTE_ADDR value in request.META in the aftermath, but REMOTE_ADDR is still ' '.
So how do I set REMOTE_ADDR (i.e. the client's IP address) field in Django's request.META? Maybe I can pass it explicitly via gunicorn? Someone mentioned I should handle it at the DB end - I'm not sure how I can do that? Should I edit pg_hba.conf or postgresql.conf or something? I've looked into those files, there's no option to 'allow null values for IPs' to be logged. Moreover, I'd rather pass whatever value resides in $remote_addr to Django, instead of letting all logged in users' IPs be null.
And let's not forget that if I use solely gunicorn, REMOTE_ADDR in Django's request.META gets correctly set; so my guess is the problem lies with how I'm passing it via nginx.
Please help! And feel free to ask for more information if you feel you need it.
/etc/nginx/sites-available/myproject:
server {
listen 80;
server_name example.cloudapp.net;
charset utf-8;
underscores_in_headers on;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/mhb11/folder/myproject;
}
location / {
proxy_pass_request_headers on;
include proxy_params;
proxy_pass http://unix:/home/mhb11/folder/myproject/myproject.sock;
}
error_page 500 502 503 504 /500.html;
location = /500.html {
root /home/mhb11/folder/myproject/templates/;
}
}
/etc/nginx/proxy_params:
proxy_set_header Host $host;
proxy_set_header User-Agent $http_user_agent;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE_ADDR $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

This is a known issue. Check out this discussion around REMOTE_ADDR being invalid when running a domain socket server: https://github.com/python-web-sig/wsgi-ng/issues/11
The one thing you can do to ensure all dependencies remain intact is write middleware that takes care of your problem at the Django project level.
For instance, something like this:
class XForwardedForMiddleware():
def process_request(self, request):
if "HTTP_X_FORWARDED_FOR" in request.META:
request.META["HTTP_X_PROXY_REMOTE_ADDR"] = request.META["REMOTE_ADDR"]
parts = request.META["HTTP_X_FORWARDED_FOR"].split(",", 1)
request.META["REMOTE_ADDR"] = parts[0]
(source)
Try it, it's surely going to solve your problem.

I was having some minor issues with the provided solution. In some cases, for some bots, the value actually contained 'unknown' as the first entry. So I had to tweak it to look at all the values, taking the first valid one as the REMOTE_ADDR.
class XForwardedForMiddleware:
"""
Set REMOTE_ADDR if it's missing because of a reverse proxy (nginx + gunicorn) deployment.
https://stackoverflow.com/questions/34251298/empty-remote-addr-value-in-django-application-when-using-nginx-as-reverse-proxy
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if 'HTTP_X_FORWARDED_FOR' in request.META:
remote_addrs = request.META['HTTP_X_FORWARDED_FOR'].split(',')
remote_addr = None
# for some bots, 'unknown' was prepended as the first value: `unknown, ***.***.***.***`
# in which case the second value actually is the correct one
for ip in remote_addrs:
ip = self._validated_ip(ip)
if ip is not None:
remote_addr = ip
break
if remote_addr is None:
raise SuspiciousOperation('Malformed X-Forwarded-For.')
request.META['HTTP_X_PROXY_REMOTE_ADDR'] = request.META['REMOTE_ADDR']
request.META['REMOTE_ADDR'] = remote_addr
return self.get_response(request)
def _validated_ip(self, ip):
ip = ip.strip()
try:
validate_ipv46_address(ip)
except ValidationError:
return None
return ip

ip_address = request.META.get("HTTP_X_REAL_IP")
It works with Django versions greater than 3.

I fixed it with the following code。
from django.utils.deprecation import MiddlewareMixin
class XForwardedForMiddleware(MiddlewareMixin):
def process_request(self, request):
if "HTTP_X_FORWARDED_FOR" in request.META:
request.META["REMOTE_ADDR"] = request.META["HTTP_X_FORWARDED_FOR"]

nginx + uwsgi for multiple sites using multiple ports

I would like to host 2 sites in one IP address 1.2.3.4 for example. I want to visit them by using different ports. For example, I would like to have 1.2.3.4:8000 for siteA, while 1.2.3.4:9000 to point to siteB. I am using nginx + uwsgi.
Here is the example to configure one of sites.
For NGINX, I had:
server {
listen 8000; ## listen for ipv4; this line is default and implied
location / {
uwsgi_pass unix:///tmp/uwsgi.sock;
include uwsgi_params;
uwsgi_read_timeout 1800;
}
}
For UWSGI, I had:
[uwsgi]
socket = /tmp/uwsgi.sock
master = true
harakiri = 60
listen = 5000
limit-as = 512
reload-on-as = 500
reload-on-rss = 500
pidfile = /tmp/uwsgi.pid
daemonize = /tmp/uwsgi.log
**chdir = /home/siteA**
module = wsgi_app
plugins = python
To visit siteA, I simple go to 1.2.3.4:8000.
I have no problem with configuration of one site, but I have no idea to make it working with two sites.
Please note that I didnot bind the site with the server name. Does it matter?
Thanks in advance.
P.S. The following is the way I launch NGINX and UWSGI.
I first put the nginx conf file (for siteA, I called it as siteA_for_ngxing.conf) in the /etc/nginx/sites-available/ directory.
I then use uwsgi --ini uwsgi.ini to start uwsgi. (the file of uwsgi.ini contains the above [uwsgi])...
Any help?

The following example might be useless for you, because it seems you installed uWSGI manually, instead of using system repository. But I think, you can easly find how uWSGI is configured on Ubuntu and make the same configuration on your system.
Here how I have done it on Ubuntu. I installed both uWSGI and nginx from Ubuntu repo, so I got the following dirs:
/etc/nginx/sites-available
/etc/nginx/sites-enabled
/etc/uwsgi/apps-available
/etc/uwsgi/apps-enabled
On /etc/uwsgi/apps-available I placed two files: app_a.ini and app_b.ini. There is no option socket (as well as pid and daemonize) in these files. uWSGI will detect socket, log, and pid file names using ini-file name. Then I created symlink to these files in /etc/uwsgi/apps-enabled to enable apps.
For nginx I used /etc/nginx/sites-available/default config file (it already symlinked to enabled dir).
upstream app_a {
server unix:///run/uwsgi/app/app_a/socket;
}
upstream app_b {
server unix:///run/uwsgi/app/app_b/socket;
}
server {
listen 8000;
location / {
uwsgi_pass app_a;
include uwsgi_params;
}
}
server {
listen 9000;
location / {
uwsgi_pass app_b;
include uwsgi_params;
}
}