We Keep Coding

'socket' object has no attribute 'sendfile' while sending a file in flask + gunicorn + nginx + supervisor setup

Using flask, I'm trying to send a file to the user on clicking a button in UI using send_from_directory function. It used to work fine. I wanted to change the repo and since changing it, I'm no more able to download the file. On looking at the supervisor log, I see this:
[9617] [ERROR] Error handling request
Traceback (most recent call last):
File "path_to_file/venv/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 182, in handle_request
resp.write_file(respiter)
File "path_to_file/venv/lib/python3.4/site-packages/gunicorn/http/wsgi.py", line 385, in write_file
if not self.sendfile(respiter):
File "path_to_file/venv/lib/python3.4/site-packages/gunicorn/http/wsgi.py", line 375, in sendfile
self.sock.sendfile(respiter.filelike, count=nbytes)
AttributeError: 'socket' object has no attribute 'sendfile'
In the same repo, this works fine locally. But when trying in remote server using the gunicorn + supervisor + nginx setup, I get the above error message. I do get 200 Ok response in the application log file. Spent a lot of time trying to fix but without success.
Also, the notable difference between the working app between the previous repo and the non-working current repo is the python version. Previous: python2.7, Current: python3.4

For me, usually when a script works locally but not when hosted its one (or more) of these possibilities:
Location / path to the files is different
older version of python
older version of the library

Pointing the virtual env to Python 3.6 and upgrading all the relevant libraries including Flask resolved the issue.

Unable to import sendgrid into GAE application

I have a GAE application that I want to integrate with Sendgrid. I've followed the instructions (https://cloud.google.com/appengine/docs/python/mail/sendgrid) on how to install Sendgrid and everything works fine in my local dev environment.
However, when I push my application to GAE and run it, I immediately receive the following 500 Server Error:
Error: Server Error
The server encountered an error and could not complete your request.
Please try again in 30 seconds.
Even with debug on, that's all I get. But digging into the logs at GAE I can see the source of the problem:
Traceback (most recent call last): File
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py",
line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) File
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py",
line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py",
line 85, in LoadObject
obj = ____import____(path[0]) File "/base/data/home/apps/....wsgi_app.py",
line 16, in
import sendgrid File "/base/data/home/apps/..../sendgrid/____init____.py",
line 7, in
from .client import SendGridAPIClient File "/base/data/home/apps/..../sendgrid/client.py",
line 1, in
import python_http_client ImportError: No module named python_http_client
So I went into sendgrid/client.py and commented out the following line of code...
import python_http_client
Once I do that, I can run my app without receiving the 500 Server Error but the test email I tried to send wasn't delivered (although I didn't receive any error messages when trying to initiate it).
It doesn't seem right that I need to comment out a line of the Sendgrid code to make the import work and I can't figure out why others that are running Sendgrid with Python and GAE aren't having the same problem. Any thoughts would be appreciated. Thanks.

sendgrid does need python_http_client, which Sendgrid itself maintains at https://github.com/sendgrid/python-http-client -- just copy the few files in directory https://github.com/sendgrid/python-http-client/tree/master/python_http_client to a directory named python_http_client, making the latter a sibling of the sendgrid directory. I'm not sure why the online docs don't mention that -- I'll work to get it fixed, but meanwhile I hope this workaround lets you get started.

Django populate() isn't reentrant

I keep getting this when I try to load my Django application on production . I tried all the stackoverflow answers but nothing has fixed it. Any other ideas. (I'm using Django 1.5.2 and Apache)
Traceback (most recent call last):
File "/var/www/thehomeboard/wwwhome/wsgi.py", line 37, in <module>
application = get_wsgi_application()
File "/usr/local/lib/python2.7/dist-packages/django/core/wsgi.py", line 14, in get_wsgi_application
django.setup()
File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 78, in populate
raise RuntimeError("populate() isn't reentrant")
RuntimeError: populate() isn't reentrant

This RuntimeError first occured for me after upgrading to Django 1.7 (and still is present with Django 1.8). It is usually caused by an Django application which raises an error, but that error is swallowed somehow.
Here's a workaround which works for me. Add it to your wsgi.py and the real error should be logged:
import os
import time
import traceback
import signal
import sys
from django.core.wsgi import get_wsgi_application
try:
application = get_wsgi_application()
print 'WSGI without exception'
except Exception:
print 'handling WSGI exception'
# Error loading applications
if 'mod_wsgi' in sys.modules:
traceback.print_exc()
os.kill(os.getpid(), signal.SIGINT)
time.sleep(2.5)
See this thread on modwsgi for more details.

In the end the problem that I had was that I tried to run a second Django app and did not have the following defined in my apache config:
WSGIDaemonProcess ...
WSGIProcessGroup ...
Just learned that you can run a single django app without defining them but when its two it produces a conflict.

There will be many reason to causes to populate() isn't reentrant error. If you look at the registry.py in your in django application probably inside this directory /python2.7/site-packages/django/apps
# app_config should be pristine, otherwise the code below won't
# guarantee that the order matches the order in INSTALLED_APPS.
if self.app_configs:
raise RuntimeError("populate() isn't reentrant")
As you see in the comment it says app_config should be pristine. Which means if one of configuration is not correct or required library missing it will rise this populate error. I got this error because I have missed sqlite installation. Even as you see there is no mentioning possible causes in the exception. I installed sqlite by this command on debian
pip install pysqlite
It solved my problem. My exception because of missing pysqlite.Your maybe having missing of another required packages or errors in your settings.py

For those using AWS Lambda (and optionally using zappa) this can happen when the size of code & dependencies zipped into a deployment package exceeds 250MB after decompression.
Typically the zip may only be 50 MB but may decompress to over 250MB so you may need to manually unzip the deployment package to check it isn't too large when uncompressed.
https://docs.aws.amazon.com/lambda/latest/dg/limits.html

Full disclosure - "populate() isn't reentrant" errors can have multiple causes, and checking any recent config or program changes is a very good idea.
However, this error can also occur when Apache is updated, and a module is no longer valid/is corrupted/needs refreshing. This occurred to us on Webfaction after an Apache update (but may happen on any host).
Using the Apache restart script will NOT help this, because the modules remain loaded on the restart. Depending on your system, and whether the mods are cached even when Apache is shut down, this may help.
Fully stop Apache. On Webfaction, that is:
~/webapps/<YOUR WEB APP>/apache2/bin/stop
Wait a few seconds, and then...
~/webapps/<YOUR WEB APP>/apache2/bin/start
That should correct the issue. If your system caches the mods, you may have to flush the cache before starting.
Hope this helps!
Here is the link they gave me (I know the error is different, but we had this happen for the same reason with the populate error as well):
https://statusblog.webfaction.com/2018/05/16/regarding-glibc_private-errors-in-your-python-application/

For me this error was caused, because I hadn't correctly split my INSTALLED_APPS for local and production. Meaning in local I was using django-cors-headers and in production I was not. But I accidentally had left django-cors-headers even though I removed it from my production requirements.txt. After delete cors-headers from the installed apps in production the error went away.

django.core.exceptions.ImproperlyConfigured: Requested setting CACHES, but settings are not configured. You must either define the environment varia

I've tried everything that I could find to fix this issue, and I'm starting to tear my hair out a little.
I'm getting this error:
django.core.exceptions.ImproperlyConfigured: Requested setting CACHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
My scripts run fine when I do:
python3 ./manage.py runserver
However, whenever I try to run tests, I get the above error...
I use a VirtualEnv, that inherits nothing globally, everything is installed (with the correct version), and within my manage.py I have set:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<appname>.settings")
I'm using PyCharm Proffesional to develop, and I've tried running the tests in both the IDE, and in the shell.
Within the shell I'm using :
python3 manage.py test
The shell is finding no tests. The test is basic, and I'm not really all that bothered about the content of it currently as it's the environment I'm struggling with. UPDATE: I have solved the issue with the shell. Tests must be defined by:
def test_<name>():
However this hasn't solved my issue with PyCharm.
I have also called:
settings.configure()
Which told me that it was already configured.
Please note that I am not using any database with Django, and I have commented the appropriate things out of the settings.
The full error is:
Traceback (most recent call last):
File "/root/kiloenv/lib/python3.4/site-packages/django/conf/__init__.py", line 38, in _setup
settings_module = os.environ[ENVIRONMENT_VARIABLE]
File "/usr/lib/python3.4/os.py", line 631, in __getitem__
raise KeyError(key) from None
KeyError: 'DJANGO_SETTINGS_MODULE'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/<username>/Documents/<dirname>/<appname>/tests.py", line 1, in <module>
from django.test.utils import setup_test_environment
File "/root/virtualenv/lib/python3.4/site-packages/django/test/__init__.py", line 5, in <module>
from django.test.client import Client, RequestFactory
File "/root/virtualenv/lib/python3.4/site-packages/django/test/client.py", line 11, in <module>
from django.contrib.auth import authenticate, login, logout, get_user_model
File "/root/virtualenv/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 6, in <module>
from django.middleware.csrf import rotate_token
File "/root/virtualenv/lib/python3.4/site-packages/django/middleware/csrf.py", line 14, in <module>
from django.utils.cache import patch_vary_headers
File "/root/virtualenv/lib/python3.4/site-packages/django/utils/cache.py", line 26, in <module>
from django.core.cache import get_cache
File "/root/virtualenv/lib/python3.4/site-packages/django/core/cache/__init__.py", line 69, in <module>
if DEFAULT_CACHE_ALIAS not in settings.CACHES:
File "/root/virtualenv/lib/python3.4/site-packages/django/conf/__init__.py", line 54, in __getattr__
self._setup(name)
File "/root/virtualenv/lib/python3.4/site-packages/django/conf/__init__.py", line 47, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting CACHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
I have ran PyCharm under sudo to ensure that the issue wasn't permissions as I'm storing the env under root.
EDIT: I've just discovered that my tests which do not use Django are running fine, but Pycharm is still throwing several failures. These failures aren't the individual tests, they're just the error which I have mentioned here (there's 341 tests that aren't Django related). I only have one test which uses Django, which will not get past initializing and throwing the mentioned error.
Hope I've been explanitory

Use this
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
instead of
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<appname>.settings")

In your python script, you are trying to access Django models before setting the environment try it in this order :
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<appname>.`settings`")
from <appname>.models import Class1, Class2,...

If you are using PyCharm Pro, you can either test your app by action 'Run Django Console...'. After you click 'test', it will prompt you for the app you want to test.
or
Create a Django test in Run/Debug configuration.

since, I don't want to add this in every file, I created
a file called usercustomize.py and added this content:
import os
os.environ['DJANGO_SETTINGS_MODULE']='foosite.settings'
The file path in my case:
src/foosite/usercustomize.py
I installed foosite with "pip install -e". I am unsure if this works for normal installs.

In my case I needed to use
python3 manage.py check --deploy
instead of
django-admin check --deploy

fill "working directory" correctly with your working path
then write the copy down in it's place

I got this error when I imported TestCase from unittest2. I fixed it by importing TestCase by:
from django.test import TestCase
My suspicions were raised when the test was labelled "Unittest" rather than "Test". I had to run the whole suite before Pycharm realised I'd made this change...

If you're using PyCharm you may have forgotten to choose your debug configuration here:

Settings module not found deploying django on a shared server

I'm trying to deploy my django project on a shared hosting as describe here
I have my project on /home/user/www/testa
I'm using this script
#!/usr/bin/python
import sys, os
sys.path.append("/home/user/bin/python")
sys.path.append('/home/user/www/testa')
os.chdir("/home/user/www/testa")
os.environ['DJANGO_SETTINGS_MODULE'] = "settings.py"
from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")
And here's the error I get when trying to run it from shell:
WSGIServer: missing FastCGI param REQUEST_METHOD required by WSGI!
WSGIServer: missing FastCGI param SERVER_NAME required by WSGI!
WSGIServer: missing FastCGI param SERVER_PORT required by WSGI!
WSGIServer: missing FastCGI param SERVER_PROTOCOL required by WSGI!
Traceback (most recent call last):
File "build/bdist.linux-i686/egg/flup/server/fcgi_base.py", line 558, in run
File "build/bdist.linux-i686/egg/flup/server/fcgi_base.py", line 1118, in handler
File "/home/user/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 230, in __call__
self.load_middleware()
File "/home/user/lib/python2.4/site-packages/django/core/handlers/base.py", line 33, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/home/user/lib/python2.4/site-packages/django/utils/functional.py", line 269, in __getattr__
self._setup()
File "/home/usr/lib/python2.4/site-packages/django/conf/__init__.py", line 40, in _setup
self._wrapped = Settings(settings_module)
File "/home/user/lib/python2.4/site-packages/django/conf/__init__.py", line 75, in __init__
raise ImportError, "Could not import settings '%s' (Is it on sys.path? Does it have syntax errors?): %s" % (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'settings.py' (Is it on sys.path? Does it have syntax errors?): No module named settings.py
Content-Type: text/html
Unhandled Exception
Unhandled Exception
An unhandled exception was thrown by the application.
What am I doing wrong?
Running the script from the browser just gives me an internal server error.

The line
os.environ['DJANGO_SETTINGS_MODULE'] = "settings.py"
should be more like
os.environ['DJANGO_SETTINGS_MODULE'] = "settings"
based on how you're setting up sys.path. That environment variable is supposed to contain the path to the module as it should be imported by Python, not the actual filename of the module.
Basically, the way you've got it now is making Django do something like this internally:
import settings.py
I.e., it's trying to import a py module from inside a settings module.