We Keep Coding

Do I need to install django separately for different python version?

I had python version 2.7.3 and i wanted to learn django so i installed django version 1.8.2 on my ubuntu 12.0.4 .
invivtus#invictus:~/bin$ python
Python 2.7.3 (default, Sep 26 2013, 20:08:41)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.VERSION
(1, 8, 2, 'final', 0)
Then i read that best way yo work with django is to work on python version 3.3 so i installed python version 3.3.6 on my system where py is symbolic link pointing to /opt/python3.3/bin/python3.3
invictus#invictus:~/bin$ py
Python 3.3.6 (default, Jun 21 2015, 16:13:35)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
when I try and import django here i get error
>>> import django
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'django
I see django got installed my python 2.7 directory.
>>> import django
>>> django
<module 'django' from '/usr/local/lib/python2.7/dist-packages/django/__init__.pyc'>
How can i use this django with my 3.3.6 version. My default python version is 2.7.3
What are possible workout here?

Yes, each Python version has its own folder with installed packages. You'll have to install Django separately for Python 3.3. The same is true for every package that is not available by default.
(If you're using Python 3, why not go for the latest and greatest, 3.4?)

As some of the comments said, you should be using a virtualenv to isolate your environments. You would do it like this:
1) Ensure you have virtualenv installed. On Ubuntu for instance, that would be package virtualenv.
2) Create a new, empty environment. You choose which python version it will be like this:
virtualenv -p /usr/bin/python3.4 env
3) That created an env folder. Activate the newly created environment:
. env/bin/activate
This updates your paths so now, when you run python or pip from this shell, they will execute in the context of your virtualenv.
4) Update the virtualenv (optional)
pip install -U pip
5) Install whatever packages you need. The recommended way is to have a requirements.txt file at the root of your project. You would pull them this way:
pip install -r myproject/requirements.txt
That's it. Use the pip command as usual. As long as you're working with the virtualenv active, your python command will only see the modules you explicitly install in it.
6) Don't forget to re-run . env/bin/activate in every new shell. If you think you'll probably forget, you can add this to your manage.py:
import sys
if __name__ == "__main__":
if not hasattr(sys, 'real_prefix'):
sys.stderr.write('Running outside of any virtualenv - did you forget to activate one?\n')
What are the benefits?
You have an isolated environment for every project (no conflicts).
You may use different versions of the same module in different projects.
System updates will not break your project.
You are not polluting your system with unmanaged files.
You never run stuff as root, which means both added isolation, and the possibility of running your project without having root access to the system.
As long as you keep your requirements.txt up to date (using pip freeze), you can rebuild the virtualenv on another system and it will work.
[edit: using requirements.txt]
That's just a file that has pip install specifications, one by line. It allows to rebuild the virtualenv from scratch easily. You can generate it from your current virtualenv using:
pip freeze > requirements.txt
So the idea is just to remember to re-run this command everytime you change your environment (installing, removing or upgrading some package).

Django, Pyenv and Git file structure on Ubuntu

I am working to set up a django project on ec2 with an Ubuntu 14.4 LTS instance. I want to write my code using python 3 and django. I've been advised that the best way to do this is to use a virtualenv. Following
https://robinwinslow.co.uk/2013/12/26/python-3-4-virtual-environment/
I tried:
~$ pyvenv-3.4 djenv
Which appears to create a virtualenv (please see screenshot). Now I have 2 questions:
1) What folder should I place my django project. - I'm thinking within the djenv folder. In other words I'd run:
/home/ubuntu/djenv$ django-admin.py startproject testproject.
2) init a git repository. I'm assuming I'd to it it in the same location, i.e.
/home/ubuntu/djenv$ git init
from within
Does this seem correct or is there a better way to do this?

Your project source code should be entirely separate from your virtual env in the file system. If they are in the same place, as you suggest, then you will end up checking libraries into your git repository needlessly and that will take up extra space end up causing problems.
Once you have activated a virtualenv you can run Python and use all the libraries in it. You don't need any connection in the file system.
You should store a PIP file in your git repo somewhere that describes how to install the relevant dependencies into your virtualenv so you can re-create it on another machine.
On my machine my projects are in /home/me/projects/«project» and my virtualenvs are in /home/me/envs/«envname». I use virtualenvwrapper which makes things easy.
Create an environment
$ mkvirtualenv test
New python executable in test/bin/python
Installing Setuptools......done.
Installing Pip.........done.
Activate it
$ workon test
Python now refers to the one in my environment. It has its own site-packages etc.
$ which python
/Users/joe/Envs/test/bin/python
If we run it and look at the paths, they point to the virtualenv. This is where it looks for packages (lots removed from my path for simplicity).
$ python
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/joe/Envs/test/lib/python27.zip', '/Users/joe/Envs/test/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/Users/joe/Envs/test/lib/python2.7/site-packages']
>>>

Python3 pip installing globally when in a virtualenv

Attempting to finally make the jump to Python 3, but am running into some issues with virtualenvwrapper. I start out by creating the virtual environment like so:
mkvirtualenv -p /usr/local/bin/python3 projectname
which yields:
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3'
New python executable in projectname/bin/python3.3
Also creating executable in projectname/bin/python
Installing setuptools, pip...done.
So far, so good. I check the python console to make sure that the environment is looking at the correct interpreter and all that and it is. Here's where sadness happens (while the virtualenv is active):
pip install flask claims to be successful, but alas:
Python 3.3.3 (default, Jan 2 2014, 13:26:32)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'flask'
Here's the issue:
$ pip show flask
---
Name: Flask
Version: 0.10.1
Location: /usr/local/lib/python3.3/site-packages
Requires: Werkzeug, Jinja2, itsdangerous
Unless I'm completely misunderstanding virtualenv/wrapper and their respective magics (which I very well could be), it seems like pip install is installing Flask globally rather than to the site-packages within my virtualenv, and thus the virtualenv is ignoring it.
Any clues what's going on here/how to fix? Am I wrong in assuming that virtualenvwrapper is ready for primetime with python3? Pretty solutions where I don't have to mangle my .bashrc or manually set environment variables are preferable. I'm hoping there's a way to do this through the api's provided by virtualenv and virtualenvwrapper.
Thanks!

I had problems with pip installing packages globally instead of in the activated virtualenv too. Have a look at pip installing in global site-packages instead of virtualenv for the question (and the answer).
Basically, the solution consisted of modifying the shebang of the pip scripts within the virtualenv as they pointed to the wrong python installation (global instead of in the virtualenv). Just change the shebang to point to the correct location and you're set.
Note: credit should go to Chase Ries who came up with the solution.

I had the same issue. It appears to be resolved as of Virtualenv 1.11.4.

WARNING: IPython History requires SQLite, your history will not be saved

Hi I'm using Ubuntu release 12.10 (quantal) 32-bit with Linux Kernel 3.5.0-21-generic. I'm trying to get IPython's History to work. I've set it up using pythonbrew and a virtual environment. In there I use pip to install IPython. Currently, when I start up IPython in a terminal I get:
WARNING: IPython History requires SQLite, your history will not be saved
Python 2.7.3 (default, Nov 8 2012, 18:25:10)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
Searching on the warning in the first line, I found this issue report, so I went back and installed the following:
sudo apt-get install libsqlite0 libsqlite0-dev libsqlite3-0 libsqlite3-dev
and then removed and reinstalled pysqlite using pip
pip uninstall pysqlite
pip install pysqlite
After that I thought I would check the installation by importing the module:
Python 2.7.3 (default, Nov 8 2012, 18:25:10)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named _sqlite3
So now it seems the file _sqlite3.so can't be found. That's when I found this SO question. Either it doesn't exist or it's not in my PYTHONPATH environment variable. Searching for the file, I get:
$ locate _sqlite3.so
/home/me/Desktop/.dropbox-dist/_sqlite3.so
/home/me/epd/lib/python2.7/lib-dynload/_sqlite3.so
/usr/lib/python2.7/lib-dynload/_sqlite3.so
So the file is there, but when I looked in my python path:
import sys
for p in sys.path:
print p
none of the above paths that contain _sqlite3.so were contained in my PYTHONPATH. For giggles, I added the path /usr/lib/python2.7/lib-dynload to my PYTHONPATH in a terminal and then tried to import sqlite3 again:
Python 2.7.3 (default, Nov 8 2012, 18:25:10)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/usr/lib/python2.7/lib-dynload")
>>> import sqlite3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: /usr/lib/python2.7/lib-dynload/_sqlite3.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
Uh oh. Now I'm completely stuck. Can anyone help me out? I've also read in a few places that I may have to rebuild Python. I have no idea how to do this in pythonbrew. Can anyone point me in the right direction?

I've also read in a few places that I may have to rebuild Python.
This is correct. SQLite is part of the standard library,
and is built when you compile Python. There are a few 'optional' parts
of the standard library, which Python will simply skip (with minimal warning, unfortunately)
if the dependencies are missing at build time, and sqlite is one of these.
You should be able to just install libsqlite3-dev,
then rebuild Python and you should be set.
Keep an eye on the build messages,
as they do report which modules they are skipping due to missing dependencies.

Thanks to minrk for pointing me in the right direction. All I had to do was rebuild python. I've outlined the steps below for those that are using pythonbrew. Notice that I already installed the libsqlite3-dev package up in the question section.
First, with the proper version of python and virtual environment loaded up run the command:
$ pip freeze -l > requirements.txt
This gives us a text file list of all of the pip packages that have been installed in the virtual environment for this particular python release in pythonbrew. Then we remove the version of python from pythonbrew and reinstall it (this is the "rebuild python" step):
$ pythonbrew uninstall 2.7.3
$ pythonbrew install 2.7.3
After that, we switch over to the newly installed python version 2.7.3 and create a new virtual environment (which I've called "sci"):
$ pythonbrew switch 2.7.3
$ pythonbrew venv create sci
$ pythonbrew venv use sci
Ideally you should be able to run the command:
$ pip install -r requirements.txt
and according to this pip should reinstall all the modules that you had in the virtual environment before we clobbered that version of python (2.7.3). It didn't work for me for whatever reason so I manually installed all of the modules using pip individuality.
$ ipython --pylab
Python 2.7.3 (default, Jan 5 2013, 18:48:27)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
and IPython history works!

What worked for me (using osx + homebrew + brewed python):
# Reinstall Python 2.7 with sqlite
brew remove python
brew install readline sqlite gdbm --universal
brew install python --universal --framework
# Reinstall iPython with correct bindings
pip uninstall ipython
pip install ipython
And you should be good to go.

You should rebuild your python with sqlite support
sudo apt-get install libsqlite3-dev
wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz
tar -xvf Python-2.7.15.tgz
cd Python-2.7.15
./configure
make
sudo make install
Recreate your virtual environment and you should be good to go
rmvirtualenv venv
mkvirtualenv -p python2 venv
workon venv
pip install -r requirements.txt
# or
pip install ipython

This warning appears on macOS when python is installed with pyenv. By default it installs python without sqlite. These commands reinstall python with sqlite support:
pyenv uninstall 3.7
CFLAGS="-I$(xcrun --show-sdk-path)/usr/include" pyenv install 3.7

Too many different Python versions on my system and causing problems

During the past years, I have installed many Python libraries with various Python versions. To make them ready to work immediately, I installed them blindly without control. Currently they're causing problems when I tried to install pynest which invokes numpy, scipy and matplotlib. After struggling, I am going to clean and reinstall Python and the libraries.
After investigation, I found Python 2.5/2.6/2.7/3.2 on my system, and each of them has some copies or other things at: (my OS == Mac OS X 10.7.5 Lion)
/Library/Frameworks/
/opt/local/Library/Frameworks/
/opt/local/bin/
/Applications/
/usr/local/bin/
/usr/bin/
/System/Library/Frameworks/
I know I'm crazy to have these. Now I have removed all these except the things in /System/Libarary/Frameworks (I never remove any thing from /System/Library/). After the clean work, which python now gives /usr/bin/python which links to /System/Library/Frameworks.
Now, is it a clear environment for me to reinstall python? How to double check that there's no other versions existing? How should I reinstall them to guarantee that they and their libraries won't be everywhere and have many copies again?
I want to install a clean Python 2.7 onto a proper location, and make my system know exactly where it is and never install any libraries somewhere else. Please give me some advice that how to manage it like in a professional way.
For your information, here is my current $PATH, I think it should be modified:
/opt/local/bin:/opt/local/sbin:/opt/nest/lib/python2.7/site-packages:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/django/bin:/usr/X11/bin:/opt/local/bin:/opt/local/sbin:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/3.2/bin
Please let me know If you need more information. Thank you!
UPDATE:
I'm rethinking profoudly why it becomes so crazy. I believe it's because I installed things via:
easy_install / macports / homebrew / fink / pip sometimes;
.dmg sometimes;
.pkg sometimes;
compile source code sometimes;
and they made things at different locations. I wonder what's the mechanism behind these ways? How do they choose target location? How to prevent them from messing things up?

Why did it get messed up?
There're a couples of different way to install Python, as the update of OP says, and they locate files in different locations. For example, macports puts things into /opt/local/, while homebrew puts things into /usr/local/. Also, Mac OS X brings a few python versions with itself. So, if you install python many times via different ways, you will get many python versions existing independently on your system.
What problem does it cause?
I don't know exactly. I guess the problem is that if you have many versions of python, then which one to use and where to find packages will be determined by the path order in your system PATH and the PYTHONPATH respectively. So you may lose control of where to install python modules. Consider that if you run sudo python setup.py install to install a module (it finds python by the root's PATH) and then try to import the module by python -c "import it" (this time it finds python by your PATH), maybe something will go wrong. This is my guess, I didn't validate it. But in my own case, something did go wrong.
How to avoid this?
I think the principle would be that be aware of that different ways and tools install things independently to different locations, so use them mindfully.
Unless you intend to, don't install the same thing twice via different
ways. (If you intend to do it for python, you might want to check out virtualenv)
Keep an eye on the path order in your PATH and consider if it's
correct.
When installing modules, be clear which python (or pip) is
running and where the module is installed.
So, how did I solve my own case?
Since it had been messing up already and seemed to be very hard to cure, so finally I solved this question by a full OS re-installation, and started to follow the DOs-and-DONTs above. For the installation of the scientific environment with python (numpy/scipy/matplotlib, which had shown problems to make me ask this question), I found this tutorial was extremely helpful. So, problem solved finally.

Here is what was confusing me and how I solved it.
$ which python
/usr/bin/python
$ which python3
/usr/local/bin/python3
$ ls /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory
So notice I didn't have a HomeBrew installation of python2.7, but did have the python3 installation. The version under /usr/bin/python is using the system default. You can tell based on the module search path:
$ /usr/bin/python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
`enter code here`Type "help", "copyright", "credits" or "license" for
more information.
>>> import sys
>>> sys.path
['', '/Library/Python/2.7/...
Notice the '/Library/Python'... that's Mac OS's version of python. But I want to stay strictly on a user installed version (i.e. HomeBrew).
So here's what I did to fix this:
$ brew install python
...
Warning: python 2.7.13 is already installed, it's just not linked.
You can use `brew link python` to link this version.
$ brew link --overwrite python
$ which python
/usr/local/bin/python
$ python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python/2.7.13...
Its no longer /Library/.. but /usr/local.
Now its finding all of my pip installed modules! Problem solved!
UPDATE:
After updating brew to version 1.5.4, it seems the symbolic links were removed. And now you have to add this to your path:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Read the Caveats section in 'brew info python':
==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.bash_profile:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Pip and setuptools have been installed. To update them
pip2 install --upgrade pip setuptools
You can install Python packages with
pip2 install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html

In order to install a python distributions into specific folder, you can use the --prefix scheme during python installation. Using the prefix scheme, you can for example install Python 2.7 into the folder /opt/py27. Now, in order to use the new installed Python distribution you have to: cleanup you PATH and LD_LIBRARY_PATH:
Remove all 'old' Python paths and
configure (according to my example) the environment variables like this:
PATH: Add /opt/py27/bin
LD_LIBRARY_PATH: Add /opt/py27/lib
That's it.
(In case you need multiple environments of Python installed at the same time, I'd suggest to have a look at virtualenv)

tl;dr
brew install python
Symptoms
I had similar issues with python programs not finding dependencies.
My python3 version was a broken symlink.
My pip was pointing to a python 3.8
And my pip3 was pointing to 3.9
python -V was outputting some python 2.7 version
python3 -V was outputting some python3.8 version
Solution
I ran brew install python and it fixed all my problems.