We Keep Coding

Dockerizing Django App

I have a Django application that uses Python 2.7.10 and Django 1.10.3. My Dockerfile contains the following:
FROM python:2.7.10
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/
My requirements.txt contains the following:
Django>=1.10,<2.0
Mysql-python
My docker-compose.yml contains the following:
version: '2'
services:
localhost:
image: mysql
restart: always
environment:
MYSQL_DATABASE: 'test'
MYSQL_USER: 'test'
MYSQL_PASSWORD: 'password'
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- localhost
settings.py contains the following for the database:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'test',
'USER': 'test',
'PASSWORD': 'password',
'HOST': 'localhost',
'PORT': '3306',
}
I first do a "% docker-compose build", then I do a "% docker-compose up" and get the following error:
web_1 | django.db.utils.OperationalError: (2002, "Can't connect
to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
I've looked and tried the solutions available on Stackoverflow with no luck. Any help would be much appreciated.

Any help would be much appreciated.
In regards to MySql and Django App relation you have three possible cases:
MySql is residing on same container as Django App (localhost)
MySql is residing on separate container from Django App (hostname is different, can be handled on different topology - same subnet or different etc)
MySql is residing outside of docker ecosystem (bare metal, cloud, whatever...)
Judging from your settings.py and docker-compose.yml you are mixing first two approaches. Now, since you can reference external MySql server from settings.py I'll consider that case trivial, next, cramming MySql on same container as Django App is usually bad idea (if you ever want to scale them separately) so this answer will focus on second case (MySql in separate container) and, for simplicity, on same docker-compose file (you can separate them as well).
Important excerpt from docker-compose.yml is:
version: '2'
services:
# The Database
database:
image: mysql:5.6
volumes:
- dbdata:/var/lib/mysql
environment:
- "MYSQL_DATABASE=test_db"
- "MYSQL_USER=test_user"
- "MYSQL_PASSWORD=some_password"
- "MYSQL_ROOT_PASSWORD=some_root_password"
ports:
- "33061:3306"
container_name: "mysql_database"
# Django App (taken from your example, added env vars as hint)
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
environment:
- "DB_PORT=3306"
- "DB_HOST=database"
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- database
volumes:
dbdata:
With such docker-compose you would have following setting.py config (note that HOST is taking value of docker-compose service entry for database container, port is internal to container not external to docker-host):
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'test_db',
'USER': 'test_user',
'PASSWORD': 'some_password',
'HOST': 'database',
'PORT': '3306',
}
As a sidenote, we usually run database with separate docker-compose.yml from Django App file since in usual dev cycle it is far more frequent to tweak/restart/crash something on Django App container than on database one and we can handle it separately that way, plus such a separately defined database can handle several different DjangoApps each with its own container.

Cannot connect to remote PostgreSQL instance from Django app running within Docker container

I am trying to connect a Django app running inside of a docker container to a PostgreSQL database running on another computer (lets say with an IP address of 192.168.1.22). Thus far in my experimentation, I always get the following error when attempting to connect to the PostgreSQL instance from a docker container (my code works fine outside of the docker container):
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
I've made sure that I allow all inbound connections to the PostgreSQL database (by changing the config file for my PostgreSQL server as recommended here.
Here is my settings code in my Django app for the database connection:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'xxxxx',
'USER': 'xxxxxxx',
'PASSWORD': 'xxxxxxxxxxx',
'HOST': '192.168.1.22',
'PORT': '5432',
}
}
Here is my Dockerfile:
FROM python:3.6
ENV PYTHONUNBUFFERED 1
# Install dependencies
RUN mkdir /config
ADD /config/requirements.pip /config/
RUN pip install -r /config/requirements.pip
# Install source code
RUN mkdir /src
WORKDIR /src
COPY ./src/ /src/
ENV UWSGI_WSGI_FILE=/src/xxxxxxxxx/wsgi.py UWSGI_HTTP=:8000 UWSGI_MASTER=1 UWSGI_WORKERS=2 UWSGI_THREADS=8 UWSGI_UID=1000 UWSGI_GID=2000 UWSGI_LAZY_APPS=1 UWSGI_WSGI_ENV_BEHAVIOR=holy
RUN python /src/manage.py collectstatic --noinput
EXPOSE 8000
CMD ["uwsgi", "--http-auto-chunked", "--http-keepalive"]
Here is the script I use to build (& run) my docker container:
#!/bin/bash
docker build .
docker run -i -t -p 8000:8000 xxxx/xxxxxxx
I've also tried using the add-host option for Docker to add the database as a host, then reference the host name from within the settings file of my project. In all scenarios I end up with the same error above.
Thank you for the help in advance!

I fix this problem using extra_hosts and Docker-compose
For example
web:
image: my-web-app
build: .
command: bash -c "uwsgi --http-auto-chunked --http-keepalive"
ports:
- "8000:8000"
extra_hosts:
- "postgresql:192.168.1.22"
In your Django configuration
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'xxxxx',
'USER': 'xxxxxxx',
'PASSWORD': 'xxxxxxxxxxx',
'HOST': 'postgresql',
'PORT': '5432',
}
To run your app use docker-compose up

How to connect MySQL Database in django 1.10.6

import MySQLDB as Database
File "c:\pythonprojects\env\lib\site-packages\mysql_python-1.2.2-py3.6-win32.egg\mysqldb__init__py", line 22
raise ImportError, "This is MySQL Version %s, But _mysql is version %r"
SyntaxError: invalid syntax

Make sure you have pymysql installed. Then in settings.py do this
try:
import pymysql
pymysql.install_as_MySQLdb()
except:
pass
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'db-name',
'USER': 'username',
'PASSWORD': 'password',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '3306',
}
}

i) First you have to install the mysql server in your machine, if you already have mysql server up and running then jump to next step.
ii) Create a user in mysql (you can use the root user, but I would suggest create a separate one) you can refer mysql documentation for creating a new user. Let, you have created a user with:- username = testuser , password = testpass#123
iii) Create a database in mysql , let named it testDB
iv) Now you have to install mysqlclient-python, to install it from PyPI you can do so by typing pip install mysqlclient (by default pip the python package manager points python2 series if you are setting up your django project on python3 then you have to use pip3) for more information on how to install mysqlclient-python on your particular OS you can refer this link: https://github.com/PyMySQL/mysqlclient-python
v) Now In settings.py under key DATABASES make these changes:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # for mysql engine
'NAME': 'testDB',
'USER': 'testuser',
'PASSWORD': 'testpass#123',
'HOST': 'localhost', # if you want to run your project on your localmachine
}
}
vi) Now change your working directory to your django project directory: " cd yourDjangoProject" and run these django commands
python manage.py makemigrations
python manage.py migrate
Now I hope that above django commands will run successfully and you would be able to use to your mysql database in your django project.

Django + MySQL on Elastic Beanstalk - Error When Querying MySQL

I'm getting an error when I query MySQL on my Django app hosted on elastic beanstalk. The error says:
OperationalError at /admin/login
(1045, "Access denied for user 'adminDB'#'172.30.23.5' (using password: YES)")
Here is my .config file:
container_commands:
01_migrate:
command: "source /opt/python/run/venv/bin/activate && python manage.py migrate --noinput"
leader_only: true
option_settings:
"aws:elasticbeanstalk:application:environment":
DJANGO_SETTINGS_MODULE: "mysite.settings"
"PYTHONPATH": "/opt/python/current/app/mysite:$PYTHONPATH"
"ALLOWED_HOSTS": ".elasticbeanstalk.com"
"aws:elasticbeanstalk:container:python":
WSGIPath: mysite/wsgi.py
NumProcesses: 3
NumThreads: 20
"aws:elasticbeanstalk:container:python:staticfiles":
"/static/": "www/static/"
Here is my the databases section on settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
I created the eb environment through the eb create command and then proceeded to create an RDS database on the eb console on their website. Is there anything else I need to do to connect Django to MySQL? Something to do with security groups or something?
Thanks!

Thanks for your response #DrewPierce, but the problem was a simple one, and it's also extremely silly. Turns out the dollar sign in my RDS password was causing a problem. Changed it to a simpler password and I successfully logged in.
Hope this helps anyone else with a similar issue!

How to set up a PostgreSQL database in Django

I'm new to Python and Django.
I'm configuring a Django project using a PostgreSQL database engine backend, But I'm getting errors on each database operation. For example when I run manage.py syncdb, I'm getting:
C:\xampp\htdocs\djangodir>python manage.py syncdb
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line
438, in execute_manager
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line
379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line
261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line
67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "C:\Python27\lib\site-packages\django\utils\importlib.py", line 35, in im
port_module
__import__(name)
File "C:\Python27\lib\site-packages\django\core\management\commands\syncdb.py"
, line 7, in <module>
from django.core.management.sql import custom_sql_for_model, emit_post_sync_
signal
File "C:\Python27\lib\site-packages\django\core\management\sql.py", line 6, in
<module>
from django.db import models
File "C:\Python27\lib\site-packages\django\db\__init__.py", line 77, in <modul
e>
connection = connections[DEFAULT_DB_ALIAS]
File "C:\Python27\lib\site-packages\django\db\utils.py", line 92, in __getitem
__
backend = load_backend(db['ENGINE'])
File "C:\Python27\lib\site-packages\django\db\utils.py", line 33, in load_back
end
return import_module('.base', backend_name)
File "C:\Python27\lib\site-packages\django\utils\importlib.py", line 35, in im
port_module
__import__(name)
File "C:\Python27\lib\site-packages\django\db\backends\postgresql\base.py", li
ne 23, in <module>
raise ImproperlyConfigured("Error loading psycopg module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg module: No mo
dule named psycopg
Can someone give me a clue on what is going on?

You need to install psycopg2 Python library.
Installation
Download http://initd.org/psycopg/, then install it under Python PATH
After downloading, easily extract the tarball and:
$ python setup.py install
Or if you wish, install it by either easy_install or pip.
(I prefer to use pip over easy_install for no reason.)
$ easy_install psycopg2
$ pip install psycopg2
Configuration
in settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'db_name',
'USER': 'db_user',
'PASSWORD': 'db_user_password',
'HOST': '',
'PORT': 'db_port_number',
}
}
- Other installation instructions can be found at download page and install page.

Also make sure you have the PostgreSQL development package installed.
On Ubuntu you need to do something like this:
$ sudo apt-get install libpq-dev

Step by step that I use:
- sudo apt-get install python-dev
- sudo apt-get install postgresql-server-dev-9.1
- sudo apt-get install python-psycopg2 - Or sudo pip install psycopg2
You may want to install a graphic tool to manage your databases, for that you can do:
sudo apt-get install postgresql pgadmin4
After, you must change Postgre user password, then do:
- sudo su
- su postgres -c psql postgres
- ALTER USER postgres WITH PASSWORD 'YourPassWordHere';
- \q
On your settings.py file you do:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'dbname',
'USER': 'postgres',
'PASSWORD': 'postgres',
'HOST': '',
'PORT': '',
}
}
Extra:
If you want to create the db using the command line you can just do:
- sudo su
- su postgres -c psql postgres
- CREATE DATABASE dbname;
- CREATE USER djangouser WITH ENCRYPTED PASSWORD 'myPasswordHere';
- GRANT ALL PRIVILEGES ON DATABASE dbname TO djangouser;
On your settings.py file you do:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'dbname',
'USER': 'djangouser',
'PASSWORD': 'myPasswordHere',
'HOST': '',
'PORT': '',
}
}

You can install "psycopg" with the following command:
# sudo easy_install psycopg2
Alternatively, you can use pip :
# pip install psycopg2
easy_install and pip are included with ActivePython, or manually installed from the respective project sites.
Or, simply get the pre-built Windows installer.

This may seem a bit lengthy, but it worked for me without any error.
At first, Install phppgadmin from Ubuntu Software Center.
Then run these steps in terminal.
sudo apt-get install libpq-dev python-dev
pip install psycopg2
sudo apt-get install postgresql postgresql-contrib phppgadmin
Start the apache server
sudo service apache2 start
Now run this too in terminal, to edit the apache file.
sudo gedit /etc/apache2/apache2.conf
Add the following line to the opened file:
Include /etc/apache2/conf.d/phppgadmin
Now reload apache. Use terminal.
sudo /etc/init.d/apache2 reload
Now you will have to create a new database. Login as 'postgres' user. Continue in terminal.
sudo su - postgres
In case you have trouble with the password of 'postgres', you can change it using the answer here https://stackoverflow.com/a/12721020/1990793 and continue with the steps.
Now create a database
createdb <db_name>
Now create a new user to login to phppgadmin later, providing a new password.
createuser -P <new_user>
Now your postgressql has been setup, and you can go to:
http://localhost/phppgadmin/
and login using the new user you've created, in order to view the database.

The immediate problem seems to be that you're missing the psycopg2 module.

If you are using Fedora 20, Django 1.6.5, postgresql 9.3.* and you need the psycopg2 module, do this:
yum install postgresql-devel
easy_install psycopg2
If you are like me, you may have trouble finding the well documented libpq-dev rpm... The above worked for me just now.

I was having the same Issue on Mac.
The solution was to use only PIP to install everything, and touch some things.
First install PIP from: https://pip.pypa.io/en/latest/
Then you want to make sure if path to pg_config is in your PATH (echo $PATH), if not you can edit your bash_profile:
vi /Users/<user>/.bash_profile
and add this line:
export PATH=$PATH:/path/to/pg_config/bin
If you don't know where pg_config is you can use the "locate" tool, but be sure your locate.db is up to date (i was using an old locate.db and using paths that does not exists).
sudo /usr/libexec/locate.updatedb
locate pg_config
Then install Django (if needed) and psycopg2.
sudo pip install Django
sudo pip install psycopg2
And then in settings.py (localhost:defaultport)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'dbname',
'USER': 'postgres',
'PASSWORD': 'postgres',
'HOST': '',
'PORT': '',
}
}
Greets!

$ sudo apt-get install libpq-dev
Year, this solve my problem.
After execute this, do:
pip install psycopg2

Steps to create a Django app with PostgreSQL database from scratch.
Check if PostgreSQL is installed in your system. In your bash shell enter
psql --version
If PostgreSQL is installed jump to step 6.
To install PostgreSQL -
sudo apt-get install python3-dev libpq-dev
sudo apt-get install postgresql postgresql-contrib
By default PostgreSQL installation creates a special user named "postgres" that has all rights. To login to PostgreSQL
sudo -u postgres psql
To exit PostgreSQL session
\q
Now set password for "postgres" user.
Login to PostgreSQL -
sudo -u postgres psql
In PostgreSQL interactive session enter -
\password postgres
Create PostgreSQL database.
To create directly from bash shell -
sudo -u postgres createdb your_db_name
To create from PostgreSQL interactive session -
createdb your_db_name
To check if the database is created, list all databases using PostgreSQL command
\l
If you already have an existing Django project then jump to step 11. Otherwise create a virtual environment. I use virtualenv wrapper.
mkvirtualenv your_project_name
Create Django project.
django-admin startproject your_project_name
Cd to the project directory and create your app.
python manage.py startapp your_app_name
Install Django.
pip install django
Install Psycopg2. Psycopg2 is a PostgreSQL database adapter for Python. This is required for Django to connect with PostgreSQL.
pip install psycopg2
Update settings.py as follows
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'your_db_name',
'USER': 'postgres',
'PASSWORD': 'xxxxx',
'HOST': 'localhost',
'PORT': '',
}
}
Run migration.
python3 manage.py migrate
Run Django server and visit localhost:8000
python3 manage.py runserver 0.0.0.0:8000
If required you can install pgadmin - a graphical client to view and manipulate PostgreSQL database schema and data.

Please note that installation of psycopg2 via pip or setup.py requires to have Visual Studio 2008 (more precisely executable file vcvarsall.bat). If you don't have admin rights to install it or set the appropriate PATH variable on Windows, you can download already compiled library from here.

This is one of the very good and step by step process to set up PostgreSQL in ubuntu server. I have tried it with Ubuntu 16.04 and its working.
https://www.digitalocean.com/community/tutorials/how-to-use-postgresql-with-your-django-application-on-ubuntu-14-04

integrating postgresql in django
Arch: Ubuntu and Nginx
sudo apt install libpq-dev postgresql postgresql-contrib
systemctl status postgresql.service is usually active by now if not check back installation
By default, Postgres uses an authentication scheme called “peer authentication” for local connections. Basically, this means that if the user’s operating system username john#example matches a valid Postgres username john, that user can login with no further authentication. During the Postgres installation, an operating system user named postgres was created to correspond to the postgres administrative user. We use this user to perform administrative tasks.
sudo -u postgres psql
postgres=# create database example;
CREATE DATABASE
postgres=# create user john with password 'abcxyz';
CREATE ROLE
postgres=# ALTER ROLE john SET client_encoding TO 'utf8';
postgres=# ALTER ROLE john SET default_transaction_isolation TO 'read committed';
postgres=# ALTER ROLE john SET timezone TO 'UTC';
postgres=# GRANT ALL PRIVILEGES ON DATABASE example TO john;
Postgres is now set up so that Django can connect to and manage its database information.
pip install psycopg2-binary is not recommended for production as psycopg2 has unfixed issues.
settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'example',
'USER': 'john',
'PASSWORD': 'abcxyz',
'HOST': 'localhost',
'PORT': '5432', can be left empty
}
}
At the end, use these basic django migration commands to complete the integration.
python manage.py makemigrations
python manage.py migrate
Extra TODO Tips
It is highly recommended to use better methods to set critical values like NAME, USER and PASSWORD rather than using plain text like shown above in database code snippet.
One method could be using a json file to replace those contents from outside the scope of project root directories. So even in unforeseeable cases of attempted source code hijacking your critical auth details could be out of reach.
A new config.json file with corresponding key-value pairs is assumed to be created.
{
"POSTGRES_DB": "example",
"POSTGRES_USER": "john",
"POSTGRES_PASSWORD": "abcxyz",
"POSTGRES_HOST": "localhost",
"POSTGRES_PORT": ""
}
Placing this code in settings.py to load and use config.json.
import os
import json
with open('/path to your config.json file/') as config_file:
config = json.load(config_file)
Update database config to load the params from config.json file:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': config['POSTGRES_DB'],
'USER': config['POSTGRES_USER'],
'PASSWORD': config['POSTGRES_PASSWORD'],
'HOST': config['POSTGRES_HOST'],
'PORT': config['POSTGRES_PORT'],
}
}

postgres runs on port 5432
specify this in settings.py
https://www.postgresql.org/docs/14/app-postgres.html