We Keep Coding

Hosting multiple Django instances on a VPS

I'm moving away from WordPress and into bespoke Python apps.
I've settled on Django as my Python framework, my only problems at the moment are concerning hosting. My current shared hosting environment is great for WordPress (WHM on CloudLinux), but serving Django on Apache/cPanel appears to be hit and miss, although I haven't tried it as yet with my new hosting company. - who have Python enabled in cPanel.
What is the easiest way for me to set up a VPS to run a hosting environment for say, twenty websites? I develop everything in a virtualenv, but I have no experience in running Django in a production environment as yet. I would assume that venv isn't secure enough or has scalability issues? I've read some things about people using Docker to set up separate Django instances on a VPS, but I'm not sure whether they wrote their own management system.
It's my understanding that each instance Python/Django needs uWSGI and Nginx residing within that virtual container? I'm looking for a simple and robust solution to host 20 Django sites on a VPS - is there an out of the box solution? I'm also happy to develop one and set up a VPS if I'm pointed in the right direction.
Any wisdom would be gratefully accepted.
Andy :)

Traditional approach
Virtualenv is good enough and perfectly ready for production use. You can have multiple virtualenv for multiple projects on the same VM.
If you have multiple database engines for multiple projects. Like, MySQL for one, PostgreSQL for another something like this then you just need to set up each individually.
Install Nginx and configure each according to project.
Install supervisor to manage(restart/start/stop) each project individually.
Anything that required by the project.
Here it has a huge drawback. Because you can't use different versions on your database engine for a different project in an easy way. So, containerization is highly recommended.
For simple and robust solution,
Use Docker(docker-compose) for local and production deployment.
Configure uWsgi with Nginx(Available on docker.)
Create a CI/CD pipeline with any tool like Jenkins.
Monitor your projects using any good tool like Raygun.
That's it.

I created a bash script that deploys as many websites as you want on your server. It automatically installs all dependencies on your server, creates a virtual environment, configure Gunicorn, Nginx, and a database for Django, etc. Check it out:
https://github.com/jdbit/django-auto-deploy

Deploy Django project using wsgi and virtualenv on shared webhosting server without root access

I have a Django project which I would like to run on my shared webspace (1und1 Webspace) running on linux. I don't have root access and therefore can not edit apache's httpd.conf or install software system wide.
What I did so far:
installed squlite locally since it is not available on the server
installed Python 3.5.1 in ~/.localpython
installed virtualenv for my local python
created a virtual environment in ~/ve_tc_lb
installed Django and Pillow in my virtual environment
cloned my django project from git server
After these steps, I'm able to run python manage.py runserver in my project directory and it seems to be running (I can access the login screen using lynx on my local machine).
I read many postings on how to configure fastCGI environments, but since I'm using Django 1.9.1, I'm depening on wsgi. I saw a lot about configuring django for wsgi and virtualenv, but all examples required access to httpd.conf.
The shared web server is apache.
I can create a new directory in my home with a sample hello.py and it is working when I enter the url, but it is (of course) using the python provided by the server and not my local installation.
When I change the first line indicating which python version to use to my virtual environment (#!/path/to/home/ve_tc_lb/bin/python), it seems to use the correct version in the virtual environment. Since I'm using different systems for developing and deployment, I'm not sure whether it is a good idea to e.g. add such a line in my djangoproject/wsgi.py.
Update 2016-06-02
A few more things I tried:
I learned that I don't have access to the apache error logs
read a lot about mod_wsgi and django in various sources which I just want to share here in case someone needs them in the future:
modwsgi - IntegrationWithDjango.wiki
debug mod_wsgi installation (only applicable if you are root)
mod_wsgi configuration guide
I followed the wsgi test script installation here - but the wsgi-file is just displayed in my browser instead of beeing executed.
All in all it seems like my provider 1und1 did not install wsgi extensions (even though the support told me a week ago it would be installed)
Update 2016-06-12: I got a reply from support (after a week or so :-S ) confirming that they dont have mod_wsgi but wsgiref...
So I'm a bit stuck here - which steps should I do next?
I'll update the question regularly based on comments and remarks. Any help is appreciated.

Since your apache is shared, I don't expect you can change the httpd.conf but use instead your solution. My suggestion is:
If you have multiple servers you will deploy your project (e.g. testing, staging, production), then do the following steps for each deploy target.
In each server, create a true wsgi.py file which you will never put in versioning systems. Pretty much like you would do with a local_settings.py file. This file will be named wsgy.py since most likely you cannot edit the apache settings (since it is shared) and that name will be expected for your wsgi file.
The content for the file will be:
#!/path/to/your/virtualenv/python
from my_true_wsgi import *
Which will be different for each deploy server, but the difference will be, most likely, in the shebang line to locate the proper python interpreter.
You will have a file named my_true_wsgi to have it matching the import in the former code. That file will be in the versioning systems, unlike the wsgi.py file. The contents of such file is the usual contents of the wsgi.py on any regular django project, just that you are not using that name directly.
With this solution you can have several different wsgi files with no conflict on shebangs.

You'll have to use a webhost that supports Django. See https://code.djangoproject.com/wiki/DjangoFriendlyWebHosts. Personally, I've used WebFaction and was quite happy with it, their support was great and customer service very responsive.

Can I use Heroku as a Python server?

My web host does not have python and I am trying to build a machine learning application. I know that heroku lets you use python. I was wondering if I could use heroku as a python server? As in I would let heroku do all of the python processing for me and use my regular domain for everything else.

Yes, and it may be a pain at first but once it is set I would say Heroku is the easiest platform to continually deploy to. However, it is not intuitive - don't try and just 'take a stab' at it; follow a tutorial and try and understand why Heroku works the way it does.
Following the docs is a good bet; Heroku has great documentation for the most part.
Here's the generalized workflow for deploying to Heroku:
Locally, create your project and use virtualenv to install/manage
libraries.
Initialize a git repository in the base dir for your
Python project; create a heroku remote (heroku create)
Create a
procfile for Heroku to use when starting gunicorn (or see
the options for using waitress/etc); this is used by Heroku to start your process
cd to your base dir; freeze
your virtualenv (pip freeze > requirements.txt) and add/commit
requirements.txt. This tells Heroku what packages need to be installed, a requirement for your deployment to work. If you are trying to run a Python project and there are required packages missing, the app will be unable to start and Heroku will display an Internal Server Error.
Whenever changes are made, git commit your changes and git push heroku master to push all commits to Heroku. This will cause Heroku to restart the server application with your updated deployment. If there's a failure, you can use heroku rollback to just return to your last deployment.
In reality, it's not a pain in the ass, just particular. Knowing the rules of Heroku, you are able to manage your deployment with command-line git commands with ease.
One caveat - If deploying Django, Flask applications etc there are peculiarities to account for; specifically, non-project files (including assets) should NOT be stored on Heroku as Heroku periodically restarts your 'dyno' (server instance(s)), loading the whole project from the latest push to Heroku. With Django and Flask, this typically means serving assets/static/media files from an Amazon S3 bucket.
That being said, if you use virtualenv properly, provision your databases, and follow Heroku practices for serving files and commiting updates, it is (imho) the absolute best platform out there for ease of use, reliable uptime, and well-oiled rolling deployments.
One last tip - if you are creating a Django app, I'd suggest starting your project out of this boilerplate. I have a custom one I use for new projects and can start and publish a project in minutes.

Yes, you can use Heroku as a python server. I put a Python Flask server on Heroku but it was a pain: Heroku seemed to have some difficulties, and there were lots of conflicting advice on getting around those. I eventually got it working, can't remember what web page had the ultimate answer but you might look at this one: http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xviii-deployment-on-the-heroku-cloud

Have you done your Python Server on Heroku by using twisted?
I don't know if this can help you.
I see the doc 'Getting Started on Heroku with Python' is about the Django.
It is sure that Heroku can use Twisted from docs
Pure Python applications, such as headless processes and evented web frameworks like Twisted, are fully supported.
django-twisted-server has twisted in django but it isn't on Heroku.

Installing a my Django app on ec2

Im in the process of launching a Django app on ec2, but have hit a wall trying to install my code on my AMI instance. This is my situation: I have a bitnami AMI up and running that has Django, apache, Postgresql, and nearly all my dependancies pre installed, and I have my fully functional Django app running on my local machine that I have been testing thus far with the Django Dev server. After quite a bit of googling, the most common methods of installing an app to an ec2 instance seem either using ssh/sftp/scp to drop a tarball in the instance, or creating a repository and importing code from there. If anyone can tell me the method they prefer, and guide me through the process, or provide a link to a good tutorial, it would be hugely appreciated!

tar -pczf yourfile.tar.gz MyProject
scp -i /home/user/.cert/yourcert.pem yourfile.tar.gz user#serveripaddress:/home/user
tar -xvf /home/user/yourfile.tar

I usually simply scp -R my whole site directory into /home/bitnami of my AMI. I'm using Apache/NGINX/Django with mod_wsgi. So the directory (for example /home/bitnami/djangosites/) gets referred to based on my mod_wsgi path in my apache cfg file.
In other words, why not just move the whole directory recursively (scp -R) instead of making a tarball etc?

Directly copy the folder where your project resides may work. However you mention that you are using a BitNami image, so it is likely that you are using the BitNami Django Stack Amazon image. BitNami also provides a native version of the BitNami Django Stack so I would suggest that you first try to deploy your application on top of the native installer and see what exact steps you need to follow. For instance you may need to install python dependencies or if you plan to use Apache on production instead of the Django development server you will need to configure Apache to serve your project. I'm a BitNami developer and I mention this because make easier the deployment in different platforms (including ec2) is one of the goal of BitNami and as you are already using it you can take advantage of this.

How can Django projects be deployed with minimal installation work?

To deploy a site with Python/Django/MySQL I had to do these on the server (RedHat Linux):
Install MySQLPython
Install ModPython
Install Django (using python setup.py install)
Add some directives on httpd.conf file (or use .htaccess)
But, when I deployed another site with PHP (using CodeIgniter) I had to do nothing. I faced some problems while deploying a Django project on a shared server. Now, my questions are:
Can the deployment process of Django project be made easier?
Am I doing too much?
Can some of the steps be omitted?
What is the best way to deploy django site on a shared server?

To enable easy Django deployement I would to the following:
Fisrt-time server configuration
Install mod_wsgi which allow you to run in embedded mode OR in daemon mode.
Install python and virtualenv
In your development environment
Use virtualenv. Take a look at mod_wsgi and virtualenv configuration
Install Django your django version (using python setup.py install)
Install your python libs
Develop your project
Every time you want to deploy
Copy your virtual environment to the production server
Just add an Include directive in your httpd.conf file (or use .htaccess) to your project's apache configuration. As stated in mod_wsgi integration with django documentation, one example of how Apache included file could be configured would be:
Alias /media/ /usr/local/django/mysite/media/
<Directory /usr/local/django/mysite/media>
Order deny,allow
Allow from all
</Directory>
WSGIScriptAlias / /usr/local/django/mysite/apache/django.wsgi
<Directory /usr/local/django/mysite/apache>
Order deny,allow
Allow from all
</Directory>
Automating deployement
I would consider using Fabric to automate deployement

Can the deployment process of django project be made easier?
No. You can script some of this, if you want. However, you're never going to install MySQL, MySQLPuthon, mod_wsgi (or mod_python), or Django again.
You will, however, tweak your application all the time.
Am I doing too much?
No. Python (and Django) are not part of Apache. PHP is embedded in Apache. PHP is exactly like mod_python (or mod_wsgi). Just one piece of the pie. (Apparently, some hosts handle the PHP installation for you, but don't handle the mod_wsgi or mod_python installation.)
Can some of the steps be omitted?
No. However, you only do it once.
What is the best way to deploy django site on a shared server?
You're doing it correctly.
When I deployed another site with php (using CodeIgniter) I had to do nothing
Certainly an unfair comparison. Apparently, they already installed PHP and the database for you. Nice of them.
Also, PHP is not Python. PHP is a plug-in to Apache. Python is "just" a programming language, that requires a separate plug-in to Apache (i.e., mod_python or mod_wsgi).
See How nicely does Python 'flow' with HTML as compared to PHP?

Django hosting support is not as widespread as for PHP, but there are some good options. I can recommend WebFaction - they provide an easy-to-use control panel which offers various combinations of Django versions, Python versions, mod_python, mod_wsgi, MySQL, PostgreSQL etc. They're cost-effective, too. If you use their setup, you get SSH access but just about all of the setting up can be done via their control panel, apart from the actual uploading of your project folder.
Disclaimer: apart from being a happy customer I have no other connection with them.

You didn't have to do anything when deploying a PHP site because your hosting provider had already installed it. Web hosts which support Django typically install and configure it for you.

You just install this already made solution if your allowed to run an image on a virtual machine. I can imagine installations will be done this way in future as complicated security configuration can be done automatically.

Most shared hosting sites run the LAMP (Linux, Apache, MySQL, PHP) stack so deployment is just a matter of copying some files over. If you were using one of the PHP frameworks like CakePHP or something the service hasn't installed (like an imaging library) you'd be going through extra deployment steps as well.
With Django (or Rails, or any other complex framework) you have to set up the stack yourself that one time, then you're good to go.
However, you'll also want to think about post-deployment updating. If it's something you're going to do often you may also want to look into Fabric or Capistrano to help automate that.
P.S. I'll second that WebFaction recommendation. It's as close to one-button installation as I've seen. Pretty happy customer although I mostly use them for test-sites and prototyping.

You can use Python virtualenv and pip (see also "Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip"). I developed my Django project in the virtual environment. I copy the virtual environment file to the production machine when I deploy my application. I use mod_wsgi. You must write that in the mod_wsig file:
import site
site.addsitedir('C:\PythonVirtualEnv\IntegralEnv\Lib\site-packages')