We Keep Coding

How to install python-distutils for old python versions

I am running Ubuntu 20.04 with python 3.6, 3.7, and 3.8 installed.
I am trying to install some packages using pip on 3.6 and 3.7 versions using 'python3.7 -m pip install package' but, I am getting this error:
ModuleNotFoundError: No module named 'distutils.util
I already have python3-distutils and python3-distutils-extra installed but pip only works for python 3.8.
How can I make pip work for installing packages on python 3.6 and 3.7?

What about things like deadsnakes or pyenv, would those help? Otherwise you might need to add the apt repositories for older Ubuntu versions (say 18.04), but I am not sure what the side effects might be. This is a question that I would rather ask on Ask Ubuntu or Super User.

(from bounty description)
I want a definitive solution that does NOT involve someone telling me to install:
python-distutils
python3-distutils
python3-distutils-extra
I will be handing out FREE DOWNVOTES to anyone who mentions installing these.
Well, I have some bad news for you. There is no "official" way to get distutils on the Debian-provided python* packages without installing these packages, not unless you go outside of Debian's default package repositories.
This is a consequence of Debian's decision to break up various parts of Python's standard library into separate packages, on the basis of their policy to split things into runtime packages and development packages. They consider distutils to fall in the "development" part, and thus, don't distribute it as part of the standard python package.
Basically, your options are:
install a python*-distutils package.
This is what your OS maintainers recommend doing in your situation.
You've not explained why you'll "be handing out free downvotes" for anyone recommending this, but this is genuinely the "most correct" solution.
install Python, from somewhere that does not break up the standard library.
This would mean using a package source other than the official debian repositories.
The easiest source would be the deadsnakes ppa.
compile your own Python!
This isn't actually that difficult, and tools like pyenv make it easy to compile and manage multiple python versions.
hack around the problem!
You could... download the sources from CPython's repo, and place the Lib/distutils folder somewhere on the import path for pythonX.Y? This is a 100% hack though and I strongly recommend not doing this. If something bad happens, because you did this, I'm not responsible.

I've been looking for an answer to this question for almost as long as you, and finally found a solution.
Be warned, I may have screwed up my python3.8 installation as a side-effect, and it's possible the package will be uninstalled again and break things as soon as you run apt-update.
Also, it may be possible to reduce the number of commands, but here's what I ended up doing:
Run apt list to see a list of available packages:
$ apt list -a python3-distutils
Listing... Done
python3-distutils/focal-updates,focal-updates,focal-security,focal-security,now 3.8.5-1~20.04.1 all [installed]
python3-distutils/focal,focal 3.8.2-1ubuntu1 all
python3-distutils/bionic-updates,bionic-updates 3.6.9-1~18.04 all
python3-distutils/bionic,bionic 3.6.5-3 all
Then download the "bionic" deb package,
because it actually contains multiple versions of distutils,
for Pythons 3.6, 3.7 & 3.8:
$ apt download python3-distutils/bionic
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-distutils all 3.6.9-1~18.04 [144 kB]
Fetched 144 kB in 0s (661 kB/s)
Finally, install the deb package:
$ sudo dpkg --install python3-distutils_3.6.9-1~18.04_all.deb
dpkg: warning: downgrading python3-distutils from 3.8.5-1~20.04.1 to 3.6.9-1~18.04
(Reading database ... 193375 files and directories currently installed.)
Preparing to unpack python3-distutils_3.6.9-1~18.04_all.deb ...
Unpacking python3-distutils (3.6.9-1~18.04) over (3.8.5-1~20.04.1) ...
Setting up python3-distutils (3.6.9-1~18.04) ...
At this point, I was finally able to use pip with python3.6.
Note-1: The printouts of apt list -a above depend on whether you run an Ubuntu or a Debian distribution, and which apt-repos you have activated in /etc/apt/sources.list*.
Note-2: As seen in the case of python3-distutils/bionic, above, the name of the package does not always coincide with its contents.
To view them, you have to download it and inspect it with dpkg -C <pafkage-file>, or from remote with apt-file list python3-distutils.
Note-3: In case you cannot find the exact distutils version for a specific Python release, you may install an earlier version and symlink to it.
For example in Debian-unstable("sid") and "bullseye" as of January 2020, the python3-distutils package contained in their apt-repos has files just for Python3.9, and the previous apt-repos from "buster" contain files for Python3.7 only.
So if you wish to install distutils for Python3.8, you have to download the "buster" package and execute:
$ sudo dpkg --unpack python3-distutils_3.7.3-1_all.deb
$ sudo rm /usr/lib/python3.8/distutils/distutils/
$ sudo ln -w /usr/lib/python3.{7,8}/distutils/distutils/

The instructions here worked for me: https://www.linuxcapable.com/how-to-install-python-3-7-on-ubuntu-20-04-lts/
Namely this one sudo apt install python3.7-distutils
Was trying to install checkov and getting the ModuleNotFound error.
After the above, it installed just fine following these instructions

Here is what has worked for me on Ubuntu 22.04 for Python 3.6.
pip won't install as distutils is not available.
install python3.6 via apt
check the exact version with python3.6 --version
find a matching tag here: https://github.com/python/cpython.git, i.e. 3.6.9
adapt and call git clone --depth=1 -b v3.6.9 https://github.com/python/cpython.git
export PYTHONPATH=`pwd`/cpython/Lib
curl https://bootstrap.pypa.io/pip/3.6/get-pip.py | python3.6 - --user
python3.6 -m pip list
If the old version is not available, cd into the cpython source directory and use e.g. ./configure --enable-optimizations --prefix=$HOME/.local and make -j4 and make altinstall (set some PREFIX to install to when calling configure, so it won't clobber the system, but note that the install path has to be in your search paths)

How to get "python -m venv" to directly install latest pip version

As part of the compilation step for a new python version, I fetch and run get-pip.py, to have the latest pip installed next to the python executable:
$ /opt/python/3.7.0/bin/python --version
Python 3.7.0
$ /opt/python/3.7.0/bin/pip --version
pip 18.0 from /opt/python/3.7.0/lib/python3.7/site-packages/pip (python 3.7)
I have 25 such versions under /opt/python, although I mostly use the five latest versions of each major.minor version that is not EOL. To setup an invironment I used to run virtualenv or my virtualenvutils with the -p /opt/python/X.Y.Z/bin/python option to get a virtual environment with a specific version.
With Python 3.7 this gives the imp module deprecation warning:
$ virtualenv -p /opt/python/3.7.0/bin/python /tmp/py37virtualenv
Running virtualenv with interpreter /opt/python/3.7.0/bin/python
Using base prefix '/opt/python/3.7.0'
/opt/util/virtualenvutils/lib/python3.6/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
New python executable in /tmp/py37virtualenv/bin/python
Installing setuptools, pip, wheel...done.
I have little hope this will be solved in virtualenv, as this has had a PendingDeprecationWarning at least since 2014 (as can be seen from the output in this question)
While investigating replacing virtualenv with python -m venv in virtualenvutils, I first created a new venv based virtual environment by hand:
$ /opt/python/3.7.0/bin/python -m venv /tmp/py37venv
$ /tmp/py37venv/bin/pip --version
pip 10.0.1 from /tmp/py37venv/lib/python3.7/site-packages/pip (python 3.7)
That has an old pip version! If you use it, you'll get:
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command
In the virtual environment created with virtualenv you immediately get the latest version:
$ /tmp/py37virtualenv/bin/pip --version
pip 18.0 from /tmp/py37virtualenv/lib/python3.7/site-packages/pip (python 3.7)
I can run a post-creation step:
/tmp/py37venv/bin/pip install -U --disable-pip-version-check pip
which will take extra time. And if there was a some security update for pip, this would imply running the non-secure version to get a secure version, an ideal point of attack.
From virtualenvutils it is trivial to do the multiple steps to create a pip-less virtualenv and then add pip using get-pip.py. From the command-line this is not so simple:
$ /opt/python/3.7.0/bin/python -m venv --without-pip /tmp/py37venvnopip
$ /tmp/py37venvnopip/bin/python -c "from urllib.request import urlopen; response = urlopen('https://bootstrap.pypa.io/get-pip'); open('/tmp/tmp_get_pip.py', 'w').write(response.read())"
$ /opt/python/3.7.0/bin/python /tmp/tmp_get_pip.py
......
$ /opt/python/3.7.0/bin/pip --version
pip 18.0 from /opt/python/3.7.0/lib/python3.7/site-packages/pip (python 3.7)
What is causing /opt/python/3.7.0/bin/python -m venv to take that old pip version? Is that the version available when 3.7.0 was released?
How can I update my install under /opt/python/3.7.0 in some way so that using /opt/python/3.7.0/bin/python -m venv creates a virtualenv with the latest pip version without reverting to scripts, aliases or using multiple commands? Having the latest pip installed under /opt/python/3.7.0 obviously is not enough.
There are two bundled wheels:
/opt/python/3.7.0/lib/python3.7/ensurepip/_bundled/setuptools-39.0.1-py2.py3-none-any.whl
/opt/python/3.7.0/lib/python3.7/ensurepip/_bundled/pip-10.0.1-py2.py3-none-any.whl
I suspect I need to update those. Is there a better way than updating those by hand? Some option for /some/python -m venv would be nice.
(And running /some/python -m ensurepip --upgrade doesn't do the trick)
Running the deprecated /opt/python/3.7.0/bin/pyvenv has the same old pip version problem.

The trick is not to install the bundled version of pip (which will almost always be out of date), but to use it to install the most current version from the internet.
Standard library venv offers a --without-pip flag that can help here. After creating the virtual environment without pip, you can then you can "execute" ensurepip's wheel directly thanks to Python's zip importer. This is both faster and less hacky than installing pip and then immediately using that same pip installation to uninstall itself and upgrade.
Code speaks louder than words, so here's an example bash function for the process I've described:
# in ~/.bashrc or wherever
function ve() {
local py="python3"
if [ ! -d ./.venv ]; then
echo "creating venv..."
if ! $py -m venv .venv --prompt=$(basename $PWD) --without-pip; then
echo "ERROR: Problem creating venv" >&2
return 1
else
local whl=$($py -c "import pathlib, ensurepip; whl = list(pathlib.Path(ensurepip.__path__[0]).glob('_bundled/pip*.whl'))[0]; print(whl)")
echo "boostrapping pip using $whl"
.venv/bin/python $whl/pip install --upgrade pip setuptools wheel
source .venv/bin/activate
fi
else
source .venv/bin/activate
fi
}
If you prefer the older project virtualenv, it also offers --no-pip, --no-setuptools, and --no-wheel flags to achieve the same on Python 2.7.
Note: Python 3.9+ venv has an --upgrade-deps option to immediately upgrade the pip/setuptools versions after creating an environment, see https://bugs.python.org/issue34556 for more info about that. I don't use this option because it still goes through an unnecessary install/uninstall of the vendored versions, which is inferior to the method of creating an environment with the latest versions directly as shown above.

I use upgrade-ensurepip to update those pip and setuptools wheel files that are part of the ensurepip package. It's not as elegant as being able to upgrade ensurepip via pip, but it's still preferable to doing it manually.
https://pypi.org/project/upgrade-ensurepip/

It is an expected behavior. python -m venv calls python -m ensurepip to install pip and This answer shows that ensurepip would only install the bundled version even with --upgrade option. There isn't any official option to update the bundled pip and setuptools.
Well I have also no good idea to fix this problem as it just is the designed behavior. I would like to give two suggestions:
Use pipenv. It is really good! And it will be the next-generation official package manager in the future(Although there is a big problem related to current Pypi's structure. In short, a package manager can only decide the dependencies with downloading the whole package. This gives a huge difficulty to building dependencies graph.).
Implement your custom EnvBuilder, actually there is an official example about this. And in the example, it also use get-pip.py to install the latest pip.

Tox installs the wrong version of pip to it's virtual env

I am using tox to manage some testing environments. I have a dependency (backports.ssl-match-hostname) that I cannot download using the latest version of pip, so I need to revert back to pip 8.0.3 to allow the install to work.
I have included the 8.0.3 version of pip inside my tox.ini file for dependencies.
deps=
pip==8.0.3
However, when I run
source .tox/py27/bin/activate
and enter the virtual testing environment, and then run
pip --version
I end up with
8.1.2
However, outside of my tox environment, when I run the same command, I get
8.0.3
Is there anything special that tox does when grabbing pip? Why am I not able to specify the version of pip that I want to use as a dependency?
EDIT : to add to this, it seems as though I am able to grab the dependency pip==8.0.3, but for the other dependencies, they are still running from the command launched with pip==8.1.2
So, I need to be able to grab pip==8.0.3 first, and then once installed, grab everything else. Still unsure why tox is starting with pip==8.1.2

This was apparently the result of the "virtualenvs" python package containing a pre-selected group of python packages that it refers to, one of which was the latest and greatest pip.
I don't know if this is the preferred way of doing this, but I found success by running
pip uninstall virtualenv
And then reinstalling with the version that worked
pip install virtualenv==15.0.1
With the "correct" version of virtualenv in place, I was able to run my tox command
source .tox/py27/bin/activate
and see the desired version of pip
pip --version
pip 8.0.3

A workaround for this is here: https://github.com/pypa/pip/issues/3666
Although to make it work I had to write "pip install pip==8.1.1" in my script. So to recap:
Add a pip.sh script to your project:
#!/bin/bash
pip install pip==8.1.1
pip install "$#"
Add to your tox.ini:
install_command = {toxinidir}/pip.sh {opts} {packages}

I've recently hit this problem. I've had it for a while but it just didn't register because I had such occasional failures with Python 2/3 code. Another way that this can happen is, if like me, you change the virtualenv between different Python versions and don't clean up.
Check /bin or /Scripts to see whether python2 points to python. If the virtualenv is Python 3 then this will mean that python2 actually calls Python 3. Vice versa, of course, if you the virtualenv is Python 2 and you want to test Python 3 code.

New versions of virtualenv reach out to download the latest pip, setuptools, and wheel -- you can disable this behavior when running through tox with the tox-virtualenv-no-download package See: https://github.com/asottile/tox-virtualenv-no-download#wait-why

apt-get installing older version of packages (Ubuntu)

I'm trying to install pip and virtualenv on a server (running Ubuntu 12.04.4 LTS) on which I have access, but I can only do it with sudo apt-get install (school politics). The problem is that althought I have run the sudo apt-get update command to update the packages list, I think it keeps installing old ones. After doing sudo apt-get install python-pip python-virtualenv, I do pip --version on which I get the 1.0, and virtualenv --version on which I get 1.7.1.2. These two version are quite old (pip is already in 1.5.5 and virtualenv in 1.11.5). I read that the problem is that the packages list is not up-to-date, but the command sudo apt-get update should solve this, but I guess no. How can I solve this? Thanks a lot!

apt-get update updates packages from Ubuntu package catalog, which has nothing to do with mainstream versions.
LTS in Ubuntu stands for Long Term Support. Which means that after a certain period in time they will only release security-related bugfixes to the packages. In general, major version of packages will not change inside of a major Ubuntu release, to make sure backwards-compatibility is kept.
So if then only thing you can do is apt-get update, you have 2 options:
find a PPA that provides fresher versions of packages that you need, add it and repeat the update/install exercise
find those packages elsewhere, download them in .deb format and install.

If you really need to use the latest stable versions of Python packages, then do not use apt-get for installing Python packages and use pip instead. If you would use apt-get and later install the same packages by means of pip or (better not) easy_install or setup.py, you are likely to run into version conflicts wondering, why your python based commands are of unexpected versions, or even worse, why they do not work at all.
I try to follow this pattern:
1. system wide pip installation first
Using instructions from here: http://pip.readthedocs.org/en/latest/installing.html find get-pip.py script, download it and run as python script.
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python get-pip.py
$ rm get-pip.py
2. use pip to install virtualenv system wide
this shall be as easy as:
$ sudo pip install virtualenv
3. (optional) install virtualenvwrapper - system wide or to user profile
$ sudo pip install virtualenvwrapper
and follow instructions for configuring it.
4. Since now, install inside your virtualenv environments
This shall prevent conflicts between various versions of packages.
You are free to update particular virtualenvs as you need one by one independently.
5. (optional) Configure installation cache directories for installation speed
There are method how to speed up repeated installation of packages, what comes handy if you get used using virtualenv often. For details see my answer: https://stackoverflow.com/a/18520729/346478

apt-get install for different python versions

I have ubuntu 10.04 with python2.6 by default. I have installed python2.7.
When I want to install python packages with
apt-get python-<package>
it gets installed to python2.6. How can I make it to install the package to python2.7? Is there any option?
I have looked at this, but I could not find such directories in my OS. I have considered using easy_install-2.7, but not all packages are supported. For example python-torctl.
I am more interested in binding python2.7 with apt-get install.

Python has got its own package managing facilities, in parallel to the one sets by the Linux distributions (including Ubuntu). The repository is the Pypi - Python Package Index, and packages are installed with pip or the easy_install script, which is part of Python's setuptools package.
As a rule of thumb, you should not use both the packages installed via pip/setuptools, and packages available to your distro (via apt-get, yum, urpmi, etc...) as they might conflict.
So, one of the less error prone way to deal with it is to have separate Python installs in your system - leave the python that came with the system for system scripts and such - on this python, make use of packages installed by your package manager only. And install other versions of Python (or even the same), to be run with "virtualenv"s - on these other install you install things with pip/setuptools only.
(And even if one opt to live boldly and not use virtualenvs, installing another python version on the same prefix (/usr, and even /usr/local) than your system's Python is a source to confusing errors and conflicts).
Note that the Debian - and Ubuntu - systems devised a way to run parallel official Python's in /usr, and to have apt-get to install Python packages to both Python versions at once. This mostly works, but they mess with Python's default directory hierarchy, and some applications fail to use Python in this way. (It is also a mess to find the module files themselves in a Debian or Ubuntu). So the above method apply as a recommendation even if your system do have more than one version of Python available on apt-get.
In short, once you have compiled your desired version of Python, do this:
use your system's package manager to install "python-setuptools" and "python-virtualenv" (not sure if these are the actual package names).
Use virtualenv to create an environment from which you will use your different Python version
Activate your virtualenv, and install Python packages using pip on it.
Virtualenv does feature a "--help" switch to help you, but you basically do:
$ virtualenv -p <path-to-python-interpreter> <environment-dir>
$ source <environment-dir>/bin/activate
And there you are - all things using Python will "see" the interpreter in the virtualenv, due to environment variables set.

ubuntu 10.04 doesn't have a python2.7 package. You have to build 2.7 yourself. I did read an article about ubuntu releasing a python2.7 package when 12.04 came out but i'm not sure what the repository location is.
http://eli.thegreenplace.net/2011/10/10/installing-python-2-7-on-ubuntu/
or:
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python2.7
https://askubuntu.com/questions/101591/install-python-2-7-2-on-ubuntu-10-04-64-bit
this question has lots of answers online.

pyenv
https://github.com/pyenv/pyenv
Pyenv allows you to manage multiple Python versions without sudo for a single user, much like Node.js NVM and Ruby RVM.
Install Pyenv:
curl https://pyenv.run | bash
Then add to your .bashrc:
export PATH="${HOME}/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Find Python version to install:
pyenv install --list
Install the python version you want:
# Increase the chances that the build will have all dependencies.
# https://github.com/pyenv/pyenv/wiki/Common-build-problems
sudo apt build-dep python3
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
# Build and install a Python version from source.
pyenv install 3.8.0
List available Python versions:
pyenv versions
We now have:
* system (set by /home/cirsan01/.pyenv/version)
3.8.0
Select a different python version:
pyenv global 3.8.0
python --version
python3 --version
Both output:
Python 3.8.0
We can now proceed to install and use packages normally:
pip install cowsay
python -c 'import cowsay; cowsay.tux("Python is fun")'
cowsay 'hello'
We can confirm that everything is locally installed in our clean environemnt with:
python -c 'import cowsay; print(cowsay.__file__)'
gives:
/home/ciro/.pyenv/versions/3.8.0/lib/python3.8/site-packages/cowsay/__init__.py
and:
which cowsay
gives:
/home/ciro/.pyenv/shims/cowsay
and:
which python
gives:
/home/ciro/.pyenv/shims/python
Per project usage
In the previous section, we saw how to use pyenv in a global setup.
However, what you usually want is to set a specific python and package version on a per-project basis. This is how to do it.
First install your desired Python version as before.
Then, from inside your project directory, set the desired python version with:
pyenv local 3.8.0
which creates a file .python-version containing the version string.
And now let's install a package locally just for our project: TODO: there is no nice way it seems: Pyenv choose virtualenv directory
Now, when someone wants to use your project, they will do:
pyenv local
which sets the Python version to the correct one.
Related threads:
https://askubuntu.com/questions/682869/how-do-i-install-a-different-python-version-using-apt-get
https://unix.stackexchange.com/questions/9711/what-is-the-proper-way-to-manage-multiple-python-versions
apt-get install for different python versions
Tested on Ubuntu 18.04, pyenv 1.2.15.