I'm using base64 to upload an image to a Django server. When the image size is larger than 2M, the server can't get the image.
I set up the uwsgi and nginx configuration, and made the upload size 75M, but it did not work.
Client:
image1 = base64.b64encode(open(file_path, 'rb').read())
r = requests.post(url, data={"image": image1})
Server:
result = request.POST.get("image")
Nginx:
```bash
server {
# the port your site will be served on
listen ****;
# the domain name it will serve for
#server_name .example.com; # substitute your machine's IP address or FQDN
server_name ****;
charset utf-8;
# max upload size
client_max_body_size 75M;
}
```
uwsgi:
```bash
# ocr.ini file
# Django-related settings
# the base directory (full path)
chdir= /root/ubuntu
# Django's wsgi file
module= mysite.wsgi
# the virtualenv (full path)
home= /root/ubuntu
# process-related settings
# master
master= true
# maximum number of worker processes
processes= 32
max-requests= 10000
daemonize= /tmp/a.log
pidfile= /tmp/a.pid
#reload-on-as = 126
#reload-on-rss = 126
#enable-threads= true
# the socket (use the full path to be safe
socket= /root/ubuntu/mysite.sock
# ... with appropriate permissions - may be needed
chmod-socket= 666
# clear environment on exit
vacuum= true
limit-post= 20000000
harakiri=30
post-buffering=20000000
py-autoreload = 1
```
Error:
You're probably hitting a hard limit set somewhere in the server's library. If you are trying to upload such a big file, then using a POST passing everything in a data parameter is a bad idea (and also really slow).
You should use the files keyword argument for sending with requests, and the FILES property for receiving with Django.
Client:
r = requests.post(files={'image': open(file_path,'rb')})
Server:
img = request.FILES.get("xxx") # <-- file name here
with open('path/to/destination', 'wb') as dest:
for chunk in img.chunks():
dest.write(chunk)
Check out the File Uploads page on the Django documentetion.
Related
I have previously configured an app in the exact same way but when I updated Ubuntu, I broke my python install. I fixed python, and tried to set up another app in the exact same way, using the same tutorials, but I am getting a 111 error when I try to connect. I can connect if I run the app using gunicorn (which I only tried because it wasn't working using nginx), but I want to find the problem and fix it. As far as I can see everything is configured according to the documentation I went through.
Here is the uwsgi config file
# glossbox_uwsgi.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = /usr/local/apps/glossbox/glossbox
# Django's wsgi file
wsgi-file = /usr/local/apps/glossbox/glossbox/glossbox/wsgi.py
# the virtualenv (full path)
home = /usr/local/apps/glossbox/venv
plugins = python
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe
socket = server 127.0.0.1:8001
# ... with appropriate permissions - may be needed
chmod-socket = 664
clear environment on exit
vacuum = true
here is the nginx conf file:
# glossbox_nginx.conf
# the upstream component nginx needs to connect to
upstream django {
# server unix:///path/to/your/mysite/mysite.sock; # for a file socket
server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}
# configuration of the server
server {
# the port your site will be served on
listen 80;
# the domain name it will serve for
server_name vps746196.ovh.net; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /usr/local/apps/glossbox/glossbox/media; # your Django project's media files - amend as required
}
location /static {
alias /usr/local/apps/glossbox/glossbox/static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
include /usr/local/apps/glossbox/glossbox/uwsgi_params; # the uwsgi_params file you installed
}
}
the app runs using the command 'uwsgi --ini glossbox_uwsgi.ini' but I get 111 connection refused when I try to connect.
I tried to run using 'gunicorn glossbox.wsgi:application --bind 0.0.0.0:8000' just to see if it worked, and it did, although site couldn't find the static files for the admin site, I assume that is down to me having no configuration set up for gunicorn. Either way, I need to find out why it is refusing the connection.
Any help will be appreciated :-)
I have the python flask application which i am using inside the NGINX and i have dockerised this image. By default flask application runs in 5000 port and NGINX in 80 port. If I run image in the container all the services are working fine. I am able to access the services from NGINX port 80 which internally mapped to flask 5000 port.
Now I want to add the health check for this image. So i am using the py-healthcheck module in the flask application like this.
health = HealthCheck()
def redis_available():
return True, "UP"
health.add_check(redis_available)
app.add_url_rule("/health", "healthcheck", view_func=lambda: health.run())
Now If i run only the flask application(without NGINX in my local system) using the URL
http://localhost:5000/health
I am getting the proper response saying the applicarion is up.
In order to add the healthcheck for image i have adde this command in Dockerfile
HEALTHCHECK --interval=30s --timeout=120s --retries=3 CMD wget --no-check-certificate --quiet --tries=1 --spider https://localhost:80/health || exit 1
Here i am assuming that i am trying to access the healthcheck endpoint from NGINX thats why i am using localhost:80. But if i run the conainer the container is always unhealthy but all the end points are working fine. Whether i have to do some configuration in NGINX conf file in order access the healthcheck endpoint of flask from NGINX?
Here is the nginx config:
# based on default config of nginx 1.12.1
# Define the user that will own and run the Nginx server
user nginx;
# Define the number of worker processes; recommended value is the number of
# cores that are being used by your server
# auto will default to number of vcpus/cores
worker_processes auto;
# altering default pid file location
pid /tmp/nginx.pid;
# turn off daemon mode to be watched by supervisord
daemon off;
# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;
# events block defines the parameters that affect connection processing.
events {
# Define the maximum number of simultaneous connections that can be opened by a worker process
worker_connections 1024;
}
# http block defines the parameters for how NGINX should handle HTTP web traffic
http {
# Include the file defining the list of file types that are supported by NGINX
include /opt/conda/envs/analytics_service/etc/nginx/mime.types;
# Define the default file type that is returned to the user
default_type text/html;
# Don't tell nginx version to clients.
server_tokens off;
# Specifies the maximum accepted body size of a client request, as
# indicated by the request header Content-Length. If the stated content
# length is greater than this size, then the client receives the HTTP
# error code 413. Set to 0 to disable.
client_max_body_size 0;
# Define the format of log messages.
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
# Define the location of the log of access attempts to NGINX
access_log /opt/conda/envs/analytics_service/etc/nginx/access.log main;
# Define the location on the file system of the error log, plus the minimum
# severity to log messages for
error_log /opt/conda/envs/analytics_service/etc/nginx/error.log warn;
# Define the parameters to optimize the delivery of static content
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# Define the timeout value for keep-alive connections with the client
keepalive_timeout 65;
# Define the usage of the gzip compression algorithm to reduce the amount of data to transmit
#gzip on;
# Include additional parameters for virtual host(s)/server(s)
include /opt/conda/envs/analytics_service/etc/nginx/conf.d/*.conf;
}
I have a Flask app that redirects requests that should get served static files to NGINX through x-accel-redirect. On occasion, those downloads will get cut off before being finished. For example, through cURL, I'd see:
curl http://my_server/some_static_file.tar > temp.tar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
77 14.4G 77 11.2G 0 0 55.8M 0 0:04:24 0:03:25 0:00:59 58.9M
curl: (18) transfer closed with 3449105332 bytes remaining to read
This seems to happen more often with very big files (10gb+), but I've seen it also happen on smaller files of ~90mb. Nginx access logs show requests coming through and being served different, incomplete amounts of data:
1.2.3.4 - - [18/Apr/2017:01:16:26 +0000] "GET /some/flask/static/file/path HTTP/1.1" 200 15146008576 "-" "curl/7.38.0" "5.6.7.8"
1.2.3.5 - - [18/Apr/2017:01:16:29 +0000] "GET /some/flask/static/file/path HTTP/1.1" 200 15441739776 "-" "curl/7.38.0" "6.7.8.9"
errors.log has nothing useful.
My relevant flask config is as follows:
response = make_response('')
response.headers.set('X-Accel-Redirect', '/_special_nginx_path/' + file_name)
response.headers.set('Content-Disposition', 'attachment',
filename=file_name)
# have tried both with and without setting content-length
response.headers.set('Content-Length', os.path.getsize(file_path))
try:
response.mimetype = mimetypes.guess_type(file_name)[0]
if not response.mimetype:
response.mimetype = 'application/octet-stream'
except AttributeError:
response.mimetype = 'application/octet-stream'
return response
My relevant NGINX config is as follows (where a uWSGI server running my flask app is running at 127.0.0.1:1234):
location / {
proxy_pass http://127.0.0.1:1234;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /_special_nginx_path {
internal;
alias /path/to/static/files;
}
Please check your disk usage, it may happen due to this, check the nginx error logs first, the error log may have logs like:
2018/10/28 14:20:24 [crit] 5432#5432: *75 pwritev() "/var/lib/nginx/uwsgi/1/00/0000000001" failed (28: No space left on device) while reading upstream,
First, identify which partition doesn’t have free space. You can do so by typing the following command in the terminal:
df -h
You’ll now see the following details on the screen:
File system. Size. Used. Available. Used. Mounted on.
Go through the partition details and check whether any partition’s disk space usage has reached up to 100%.
Once you find the partition, open it and delete useless files so as to free up the disk space and fix the problem.
In case the partition is mounted on the system memory (indicated by the TMPFS directory), run the below command to unmount it.
Umount path_to_the_directory.
Now, restart Nginx. The error will now disappear from the file.
To prevent the no space left on device error in future, edit the Nginx configuration file (or your website’s config file) and increase the value of the key zone.
Users face the problem because they configure the OS to serve cache files from RAM. Although this can boost the performance of your site quickly, it reduces the amount of RAM available for other applications running on the server and leads to out of memory error.
If your server uses SSD instead of HDD, you don’t have to mount the partition into the system memory.
Thanks to the blog which helped me...
This is really weird situation.
I'm trying to create an web framework using Python from scratch.
And I'm trying to get the current working directory (technically the full path of the current .py file) of a file using below code.
os.getcwd()
The problem is..
If I type above code and restart uwsgi server then print it.
At the first time, it always prints like below.
/var/www/html/mydocroot
Then if I refresh the page, 'mydocroot' directory disappears automatically like below.
/var/www/html
And it creates same output after refreshing the page as above.
I spent almost whole day to figure out that issue.
Any idea or same experiences?
and also I tried,
os.path.dirname(os.path.abspath(__file__))
this one as well.
For example,
/var/www/html/mydocroot/Core/Settings
is the result when I restart uwsgi server,
/var/www/html/Core/Settings
When I refresh the page, 'mydocroot' disappears by itself as above.
server{
server_name example.com;
rewrite ^(.*) http://www.example.com permanent;
}
server {
listen 80;
server_name example.com *.example.com;
location / {
include uwsgi_params;
uwsgi_pass unix:/var/www/html/example/myapp.sock;
}
}
Above is my nginx server config. and below is wsgi configuration.
[uwsgi]
module = wsgi:application
master = true
processes = 1
touch-reload = /var/www/html/ggr/wsgi.py
socket = myapp.sock
chmod-socket = 664
vacuum = true
max-requests = 1
die-on-term = true
logger = file:/var/log/ggr.log
catch-exceptions = true
Files don't have a "current working directory". There's just one global current working directory for a given process. The current working directory can change
os.chdir('/new/working/directory')
If you want the directory of a specific file, you can do this
os.path.dirname(os.path.abspath(__file__))
I am using Django with uwsgi and nginx but everytime I am getting bad gateway 502. Please help me out. If you need any more details let me know
I am using emperor for uwsgi here is my ini file
Emporer.ini
# /etc/uwsgi/emperor.ini
[uwsgi]
# This is the directory the emperor will watch for new .ini files
emperor = /etc/uwsgi/apps-enabled
# Set the uid and gid. This could be any user you wish
uid = www-data
gid = www-data
# Set the log file path.
# Make sure the log file has the required write
# permissions for the emperor process.
logto = /var/log/uwsgi/emperor.log
ss.ini(django app)
# /etc/uwsgi/apps-available/grub_club.ini
[uwsgi]
# This should point to your Django project directory
chdir = /home/ss1/ss/src/
# This should point to your Django wsgi file.
# This usually lives alongside your settings.py file
module = ss.wsgi:application
# This should point to your python environment
# If you have a virtual environment you like to use,
# specify that here
home = /home/ss1/pro/
master = true
# This is the number of cores on your server
processes = 1
# You can either connect using HTTP OR
# you can use unix sockets (recommended)
socket = /tmp/ss.sock
chmod-socket = 664
# Removes all generated sockets and pids when closed
vacuum = true
# Log file for this Project.
# Make sure the log file has the required write
# permissions for the emperor process.
logto = /var/log/uwsgi/app/ss.log
Here is my nginx location app
nginx/sites-available/ss
# /etc/nginx/sites-available/ss
upstream ss_cluster{
# This will contain one or more endpoints that uWSGI is listening to
server unix:///tmp/ss.sock;
}
server {
server_name domain.in;
charset utf-8;
#server_name domain.in;
#return 301 $scheme://domain.in$request_uri;
# Configuration to deliver static files
location ^~ /static/ {
# This will point to the folder where all the static files reside
# Same location as Django's STATIC_ROOT
alias /home/ss1/src/ss/static/;
# Sets the HTTP headers for expiration time
location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff)$ {
expires 30d;
}
}
# Configuration to deliver media files
location ^~ /media/ {
# This will point to the folder where all the media files reside
# Same location as Django's MEDIA_ROOT
alias /home/ss1/src/ss/media/;
# Sets the HTTP headers for expiration time
location ~* \.(jpg|jpeg|png|gif){
expires 365d;
}
}
# All other requests are sent to the Django project
location / {
include /etc/nginx/uwsgi_params;
uwsgi_pass ss_cluster;
}
But I am getting 502 Bad Gateway?
Thanks in Advance