We Keep Coding

uWSGI will not run in mixed Python environment in order to operate correctly with nginx and run Django app

Environment: Ubuntu 16.04 (with system Python at 2.7.12) running in Vagrant/Virtualbox on Windows 10 host
Python Setup: System python verified by doing python -V with no virtualenv's activated. Python 3.5 is also installed, and I've done pipenv --three to create the virtualenv for this project. Doing python -V within the activated virtualenv (pipenv shell to activate) shows Python 3.5.2.
Additional Background: I'm developing a Wagtail 2 app. Wagtail 2 requires Django 2 which, of course, requires Python 3. I have other Django apps on this machine that were developed in Django 1.11/Python 2.7 and are served by Apache. We are moving to Django 2/Python 3 for development going forward and are moving to nginx/uWSGI for serving the apps.
I have gone through many tutorials/many iterations. All Vagrant port mapping is set up fine with nginx serving media/static files and passing requests upstream to the Django app on a unix socket, but this is giving a 502 Gateway not found error because uWSGI will not run correctly. Therefore, right now I'm simply running the following from the command line to try to get uWSGI to run: uwsgi --ini /etc/uwsgi/sites/my_site.com.ini. This file contains:
[uwsgi]
uid = www-data
gid = www-data
plugin = python35
# Django-related settings
# the base directory (full path)
chdir=/var/sites/my_site
# Django's wsgi file
wsgi-file = my_site.wsgi
# the virtualenv (full path)
virtualenv=/root/.local/share/virtualenvs/my_site-gmmiTMID
# process-related settings
# master
master = True
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe)
socket = /tmp/my_site.sock
# clear environment on exit
vacuum = True
I've tried installing uWSGI in the following ways:
system-wide with pip install uwsgi as well as pip3 install uwsgi
using apt-get install uwsgi uwsgi-plugin-python3
I've ensured that only one install is in place at a time by uninstalling any previous uwsgi installs. The latter install method places uwsgi-core in usr/bin and also places in usr/bin shortcuts to uwsgi, uwsgi_python3, and uwsgi_python35.
In the .ini file I've also tried plugin = python3. I've also tried from the command line:
uwsgi_python3 --ini /etc/uwsgi/sites/my_site.com.ini
uwsgi_python35 --ini /etc/uwsgi/sites/my_site.com.ini
I've tried executing the uwsgi ... .ini commands from both within the activated virtual environment and with the virtualenv deactivated. Each of the three command line uwsgi ... .ini executions (uwsgi ..., uwsgi_python3 ... and uwsgi_python35 ...) DO cause the .ini file to be executed, but each time I'm getting the following error (the last two lines of the following statements):
[uwsgi] implicit plugin requested python35
[uWSGI] getting INI configuration from /etc/uwsgi/sites/my_site.com.ini
*** Starting uWSGI 2.0.12-debian (64bit) on [Wed Mar 7 03:54:44 2018] ***
compiled with version: 5.4.0 20160609 on 31 August 2017 21:02:04
os: Linux-4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018
nodename: vagrant-ubuntu-trusty-64
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /vagrant/my_site
detected binary path: /usr/bin/uwsgi-core
setgid() to 33
setuid() to 33
chdir() to /var/sites/my_site
your processes number limit is 7743
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/my_site.sock fd 3
Python version: 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
Set PythonHome to /root/.local/share/virtualenvs/my_site-gmmiTMID
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
If I go into the Python command line within the activated virtualenv and do import encodings, it imports fine (no message - just comes back to command line). A search for this particular error has turned up nothing of use. Any idea why the ImportError: No module named 'encodings' is coming up?
UPDATE - PROBLEM STILL OCCURRING
I'm using pipenv, and it stores individual virtualenvs in the /home/username/.local/share/virtualenvs folder. Though I was able to start uWSGI from the command line by executing the uWSGI config file as the vagrant user (see comment below), I have still not been able start the service with /home/vagrant/.local/share/virtualenvs/my_venv in the uWSGI config file. I tried adding the vagrant user to the www-data group and adding the www-data user to the vagrant group. I also put both read and execute permission for the world on the whole path (including the individual venv), but the uWSGI service will still not start.
A HACKISH WORKAROUND
I did finally get the uWSGI service to start by copying the venv to /opt/virtualenvs/my_venv. I was then able to start the service with sudo service uwsgi start. Ownership of that whole path is root:root.
STILL LOOKING FOR A SOLUTION...
This solution is not optimal since I am now executing from a virtualenv that will have to be updated when the default virtualenv is updated since this location is not the default for pipenv, so I'm still looking for answers. Perhaps it is a Ubuntu permissions error, but I just can't find the problem.

It might be the problem with your virtual environment. Try the following
rm -rf /root/.local/share/virtualenvs/my_site-gmmiTMID
virtualenv -p python3 /root/.local/share/virtualenvs/my_site-gmmiTMID
source /root/.local/share/virtualenvs/my_site-gmmiTMID/bin/activate
pip install -r requirements.txt
and in uwsgi conf try to change
virtualenv=/root/.local/share/virtualenvs/my_site-gmmiTMID
to
home=/root/.local/share/virtualenvs/my_site-gmmiTMID
Reference:
http://uwsgi-docs.readthedocs.io/en/latest/Options.html#virtualenv

Django doesn't see environment variables when deployed to Elastic Beanstalk

I'm trying to set up a Django/DRF application on Elastic Beanstalk, and for whatever reason, Django just can't see the desired environment variables. When I log in, I can see them just fine, by using
$ eb ssh
$ cat /opt/python/current/env
I can also see them, except relatively sensitive ones involving RDS, simply using $eb printenv.
All that appears to be set up and working properly. However, Django likes to read the environment immediately on starting up, and it appears that the environment variables just aren't set yet. I've experimented with simply inserting print(os.environ) in settings.py, and when I do that, I discover a whole bunch of environment variables which I don't need (i.e. 'SUPERVISOR_GROUP_NAME': 'httpd'), and none of the ones I've set myself, like DJ_SECRET_KEY.
I've since changed the code to report the absence of specific environment variables when it loads the settings, and from a recent run, it generated this:
[Wed Nov 23 15:56:38.164153 2016] [:error] [pid 15708] DJ_SECRET_KEY not in environment; falling back to hardcoded value.
[Wed Nov 23 15:56:38.189717 2016] [:error] [pid 15708] RDS_DB_NAME not in environment; falling back to sqlite
[Wed Nov 23 15:56:38.189751 2016] [:error] [pid 15708] AWS_STORAGE_BUCKET_NAME not in environment; falling back to local static storage.
Again, those variables are set in the settings, and they show up with any other reporting tool EB gives me. They just aren't set in time for Django to read them when it launches and reads settings.py.
This looks pretty close to this issue, but it's not really the same: I know how to see / load the environment variables into the shell when ssh-ing into the eb instance; they're just not showing up when I need them to for the actual project.
This is almost exactly the issue I'm having, but the accepted-correct answer makes no sense to me, and the top-voted answer doesn't apply; those files are already in git.
How should I configure things so that Django can see the environment variables?

Given that the EB stores all these environment variables in a canonical location as a bash script, I ended up simply having bash execute the script, and updating the environment from the parsed results.
I created the file get_eb_env.py in parallel to my settings.py. Its main contents:
import os
import subprocess
ENV_PATH = '/opt/python/current/env'
def patch_environment(path=ENV_PATH):
"Patch the current environment, os.environ, with the contents of the specified environment file."
# mostly pulled from a very useful snippet: http://stackoverflow.com/a/3505826/504550
command = ['bash', '-c', 'source {path} && env'.format(path=path)]
proc = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
proc_stdout, _ = proc.communicate(timeout=5)
# proc_stdout is just a big string, not a file-like object
# we can't iterate directly over its lines.
for line in proc_stdout.splitlines():
(key, _, value) = line.partition("=")
os.environ[key] = value
Then, I just import and call patch_environment() near the head of my settings.py.

I solved the problem modifying the key "DJANGO_AWS_SECRET_ACCESS_KEY" because DJANGO generated a key with quotation marks (`) and interpreted it as a command.

This is not exactly what you are asking for, but I hope this solves your issue anyway.
I am using a lot of environment variables in settings. I think Elastic Beanstalk sets everything up once before it runs container commands and such and the variables are not available then and that is why your logging is showing them empty. But do you really need them the variables at that point?
You can't put whatever local development settings you need in local_settings.py and keep that out of version control.
We use them like this.
if 'DB_HOST' in os.environ:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'ebdb',
'USER': 'ebroot',
'PASSWORD': 'ebpassword',
'HOST': os.environ['DB_HOST'],
'PORT': '3306',
}
}
try:
from local_settings import *
except ImportError:
pass
They are also available when you run container_commands:
container_commands:
01_do_something:
command: "do_something_script_{$PARAM1}.sh"

Apache2 not serving django content

I am taking over a django project which another developer maintained. The service is run on an Ubuntu machine. ZEO is used for content caching. Ajax/Dajax is used for asynchronious content. Celery is used for task management and Django is used for the project itself.
The service is usually reached via a specific IP address which limits access to specific URLs. http://my_server_ip. Without knowingly changing anything, this started to not work. Instead of taking me to the splash page, entering the IP would hang, unsuccesfully connecting. I don't get a 404, 500 or some other error, it just sits and continually tries to load as if waiting to connect or to receive content.
I attempted to restart the service in the hopes that this would solve the problem, it did not. I performed a system reboot and followed the following commands, as per the prior developer's documentation, to reboot the server.
From within the django project:
runzeo -a localhost:8090 -f /path/to/operations_cache.fs
su project_owner
python manage.py celery worker -n multiprocessing_worker --loglevel=debug -Q multiprocessing_queue
python manage.py celery worker --concurrency=500 --pool=eventlet --loglevel=debug -Q celery -n eventlet_worker
The two celery commands had to be run as the owner of the project directory.
Finally I ran sudo service apache2 restart. Upon completion, I tried to navigate to the webpage but received the same response: hung on connecting. The trac pages do work at http://my_server_ip/trac.
The following is all I have found in the apache log files.
error.log
[Fri Feb 06 16:01:11 2015] [error] /usr/lib/python2.7/dist-packages/configobj.py:145: DeprecationWarning: The compiler package is deprecated and removed in Python 3.x.
[Fri Feb 06 16:01:11 2015] [error] import compiler
[Fri Feb 06 16:01:11 2015] [error]
access.log
my.ip - user [06/Feb/2015:15:55:40 -0500] "GET / HTTP/1.1" 500 632 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:35.0 Gecko/20100101 Firefox/35.0"
I have tried looking into the django logs, but nothing appears there. Perhaps I am not finding the correct logs.
As a good start, where can I find out what the system is hanging on. How do I determine if it is an issue with django, apache, the interaction between the two, etc. This would help me zero in on specifically what is happening.
Edit
I was able to resolve my problem, though I cannot say for sure what resolved the issue. I suspect that the issue had to do with permissions on the static files folder. I serve my content through the www-user. After going through the steps described above I then ran yes yes | python manage.py collectstatic as the user www-data. The server was able restart and I was able to access the trac content as well as the django content.
I am hesitant to post this as an answer because I do not know with certainty whether my solution described here is the step which solved the problem.

mod_wsgi - mod_wsgi-express error AH00526: Syntax error on line 16

This question is related with my question.
Actually I use the Apacha Version over Ubuntu 14.04 LTS 64bits:
Server version: Apache/2.4.7 (Ubuntu)
Server built: Jul 22 2014 14:36:38
When I type command mod_wsgi-express start-server to check if it runs. It seems not running and I have an error:
root#website:/home/website# mod_wsgi-express start-server
Server URL : http://localhost:8000/
Server Root : /tmp/mod_wsgi-localhost:8000:0
Server Conf : /tmp/mod_wsgi-localhost:8000:0/httpd.conf
Error Log File : /tmp/mod_wsgi-localhost:8000:0/error_log
AH00526: Syntax error on line 16 of /tmp/mod_wsgi-localhost:8000:0/httpd.conf:
Error:\tApache has not been designed to serve pages while\n\trunning as root. There are known race conditions that\n\twill allow any local user to read any file on the system.\n\tIf you still desire to serve pages as root then\n\tadd -DBIG_SECURITY_HOLE to the CFLAGS env variable\n\tand then rebuild the server.\n\tIt is strongly suggested that you instead modify the User\n\tdirective in your httpd.conf file to list a non-root\n\tuser.\n
I have the same problem during installation to doom. Any suggest is welcome!
More Information:
Modules used in Apache:
Compiled in modules:
core.c
mod_so.c
mod_watchdog.c
http_core.c
mod_log_config.c
mod_logio.c
mod_version.c
mod_unixd.c
Python Version:
>>> print (sys.version)
2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2]

I had a similar issue trying to spin up a TurboGears python application. I was finally able to get it started by executing the start-server command with the user option included. I set the user option equal to the user running apache. In my case the user is "apache". See below.
[root#localhost ~]# mod_wsgi-express start-server --user=apache <path_to_my_wsgi_file>

Python web script doesn't run on apache

I am trying getting starting with python web programming but I have a problem. I want to work with CGI
I installed Ampps web-server on pc (Linux). According to this instruction Apache Tutorial: Dynamic Content with CGI. I put my file index.pl in the folder '/usr/local/ampps/www/cgi-bin' and tried to accesses it
http://localhost/cgi-bin/index.py
Here is code of my file:
#!/usr/local/ampps/python/bin/python
print "Content-type: text/html\n\n"
print "Hello World from PythonStandard Hello World from a Python"
But it gives me mistake Sorry, the page you are looking for is currently unavailable.
Here is error log of Apache:
[Tue Jul 22 20:05:45.019799 2014] [cgi:error] [pid 11952] [client 127.0.0.1:38359] AH01215: (13)Permission denied: exec of '/usr/local/ampps/www/cgi-bin/index.py' failed
[Tue Jul 22 20:05:45.020262 2014] [cgi:error] [pid 11952] [client 127.0.0.1:38359] End of script output before headers: index.py
I corrected conf of Apache
AddHandler cgi-script .cgi .pl .py
it looked ok too
LoadModule cgi_module modules/mod_cgi.so
ScriptAlias /cgi-bin/ "{$path}/www/cgi-bin/"
What can be wrong else?

Your script needs to have the executable bit set (using chmod) and the Apache user or group (usually named 'Apache' or 'www-data' depending on your distro - you can check /etc/passwd to find out) must have the 'execute' permission on it.

I was getting similar errors. It seemed that the AMPPS installation of Python was missing one of the DLLs. I downloaded the 2.7 .msi package from ActiveState and installed it.
A couple of things before you start: the location, and the permissions.
I may be wrong on this, but Apache will be looking for the Python executable in the directory that was created for it in the AMPPS installation -- c:\whatever\whatever\ampps\python. I tried installing it in c:\whatever\whatever\ampps\python2.7, but it didn't work. I suppose I could have edited the httpd.conf file, but since the original installation wasn't working anyway, I overwrote the c:\w\w\ampps\python files.
I also ran the .msi as an administrator. This was tricky, because my company-issued machine doesn't let me simply right-click on an .msi and select "run as administrator." I finally went into the task manager (option 8 here: http://www.eightforums.com/tutorials/9564-run-administrator-windows-8-a.html), and put the full path to the downloaded file in the entry. And voila, "hello world."