We Keep Coding

How to install a specific version of python with pip, venv, and distutils on ubuntu

I've recently had to debug a cython library for a specific version of python on ubuntu and I needed python, venv, distutils, cython, pip, a compiler, and a text editor. I had to go fishing around the web for instructions on how to do this, so I'm asking this question to answer with what I did.
I googled it and found instructions in one place for pip, another place for venv, another place for compilers.

I figured this out on ubuntu 20 in docker (I was running as root). If you are not running as root - this answer won't help you.
# update the package manager
apt-get update
# install git, C/C++ compiler and a text editor (I prefer vim)
apt install -y git software-properties-common curl build-essential vim
# add package source for python distributions
add-apt-repository ppa:deadsnakes/ppa
# install specific version of python with venv and distutils
apt install -y python3.9 python3.9-distutils python3.9-venv
# get pip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.9 get-pip.py

You have to install the version of python that you want, i recommend use dead-sneak, https://www.codegrepper.com/code-examples/whatever/install+python+3.7+from+source+in+ubuntu+linux.
Later set your python version in the venv, something like "virtualenv venv --python=python{python version}" or "python{python version} -m venv venv"

Bazel cross compile of tensorflow for ARM fails

I am trying to build tensorflow to run on a Zynq, specifically, the Z7020. I have petalinux running on the board, and python 3.4.9. When trying to build tensorflow following the instructions found here:[https://www.tensorflow.org/install/install_raspbian#cross-compiling_from_sources]
Note that both petalinux and raspbian are both Debian derivatives and the Z7020 has the same CortexA9 cores as the raspberry-pi 0 and 1 series boards.
I am trying to build on an Ubuntu 16.04 host. The command I am using to build is:
sudo CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3 -e CROSSTOOL_PYTHON_INCLUDE=/home/rklein/Python-3.4.9/Include" tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 tensorflow/tools/ci_build/pi/build_raspberry_pi.sh PI_ONE
Bazel churns for about 2 hours and comes back with the following error message:
/home/rklein/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0--lots of hex digits--85e8/external/arm_compiler/bin/arm-linux-gnueablhf-gcc --lots of options
In file included from /usr/include/python2.7/Python.h:8:0, from ./tensorflow/python/lib/core/bfloat16.h:19,
from tensorflow/python/lib/core/bfloat16.h:18:
from /usr/include/python2.7/pyconfig.h:13:54:
fatal error: arm-linux-gnueabihf/python2.7/pyconfig.h: No such file or directory
#include <arm-linux-gnueabihf/python2.7/pyconfig.h>
^
compilation terminated.
What settings are needed to tell Bazel to use python3? Note that there is no /usr/include/python2.7 directory on the host machine, so I suspect that Basel is doing some voodoo behind the scenes. The command
find ~ -name python2.7
comes up empty.
I have tried to read up as much as I can on Bazel, but the documentation seems pretty lean - any good references would be appreciated.

I can't help you with your error message (or Bazel altogether). However I installed TensorFlow on an Xilinx Zynq Ultrascale+ with a Petalinux kernel and an Ubuntu (arm64) root filesystem. It's not the same exact chip (but the installation process should be similar). I didn't build TensorFlow myself, instead I used the packages provided by the tensorflow-on-arm project. Maybe my experience will be useful for other people to get TensorFlow running:
You need a working OS (Xilinx has documentation for that). Depending on your chip you need either a 32 (armhf) or 64 Bit (arm64) rootfs. I used an Ubuntu rootfs, so I could use apt-install.
You need to install some dependencies. I followed the instructions from the tensorflow-on-arm project.
apt-get install openjdk-8-jdk automake autoconf curl zip unzip libtool swig libpng12-dev zlib1g-dev pkg-config git g++ wget xz-utils
You also need Python (be sure to install Python v3.5 - not Python v3.6, etc.).
apt-get install python3-numpy python3-dev python3-pip python3-mock
I also needed to install two not listed packages.
apt-get install cython3 libhdf5-dev
Install some pip3 packages (you might want to install those in a virtual-environment and also update pip3).
pip3 install -U --user keras_applications==1.0.5 --no-deps
pip3 install -U --user keras_preprocessing==1.0.3 --no-deps
pip3 install -U --user numpy grpcio h5py
Now you should download the TensorFlow pip package. The different packages are listed under Releases. I chose TensorFlow v.1.12 for Python v3.5 and arm64 / aarch64.
wget https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.12.0/tensorflow-1.12.0-cp35-none-linux_aarch64.whl
Now you can install the package with pip3.
pip3 install -U --user tensorflow-1.12.0*
I hope it worked for you!

Difficulties in installation of pysparse and superlu

I want to run a python2.7 program (this one). I'm having a lot of trouble (I spend my whole aftenoon on this), because of the installation of the python 2.7 dependencies.
Config
I am running an Ubuntu 16.04 64bits ([Mint XFCE 18), based on Debian. My computer is a Dell Inspiron N5110 from 2011, with dual boot W7/U16. The keyboard-to-screen interface is really new in this world and perhaps need to learn a lot more about it to solve this alone.
Proceeds
I started by installing various programs with apt:
sudo apt-get install -y git python-dev libpython-dev libevent-dev libsuperlu-dev libblas-dev liblapack-dev
After git cloning the program I wanted, I installed the dependencies. I don't know why, but sudo pip install pysparse didn't worked. Instead, sudo pip install csc-pysparse worked fine.
Issue
When I run my program, it tells me from pysparse import superlu, itsolvers, precon and then ImportError: cannot import name superlu.
Why ? Isn't the pip resolving the dependencies problems it could have and install superlu ? Do I need to install superlu manually or to install the pysparse instead of csc-pysparse ?
(and please be indulgent, It's my really first post on stackoverflow, as thoses were my really firsts posts on github)

I did get an answer, thanks to William Hunter.
The procedure to install it is the following :
sudo apt-get install -y python-dev python-tk libpython-dev libevent-dev libsuperlu-dev libblas-dev liblapack-dev libatlas3-base libatlas-dev
sudo pip install matplotlib setuptools SymPy pysparse pyvtk

pip error while installing Python: "Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS"

I downloaded the Python 3.5 source code and ran the following:
$ tar -xf Python-3.5.2.tar.xz
$ ./configure --with-ensurepip=upgrade
$ make
$ sudo make altinstall
It proceeded well until make. When sudo make altinstall ran, it printed:
Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS
What went wrong?

You are most likely not compiling Python with SSL/TLS support - this is likely because you don't have the SSL development dependencies installed on your system.
Install the following dependency, and then re-configure and re-compile Python 3.5.
Ubuntu
apt-get install libssl-dev
In addition it is recommended to install the following.
apt-get install make build-essential libssl-dev zlib1g-dev libbz2-dev libsqlite3-dev
CentOS
yum install openssl-devel
In addition it is recommended to install the following.
yum install zlib-devel bzip2-devel sqlite sqlite-devel openssl-devel

Please note that besides that as of version OpenSSL 1.1.0 (included from Ubuntu 18.04 repositories, I am not sure for other Linux distributions) there are python vesions that may have compilation issues (see this question and my answer), in short:
Python <3.4.5 and Python <3.5.3 have not implemented this newest version of OpenSSL, most likely other major Python versions suffer the same issue. The fix is to download the minor version that implements the fix (or newer). Check the full changelogs to check which minor version that is (changelog for 3.4 and 3.5)

On Ubuntu 18.04 only this worked for me
sudo apt-get install libssl1.0

CentOS:
If you didn't have openssl-devel at the time of installing python, pip 8.1.1 won't install.
After installing openssl-devel, reinstall python.
This time it should install pip just file.

I tried to install Python 3.4.10 using asdf but got following error:
$ asdf install python 3.4.10
python-build 3.4.10 /home/br0ke/.asdf/installs/python/3.4.10
Downloading Python-3.4.10.tar.xz...
-> https://www.python.org/ftp/python/3.4.10/Python-3.4.10.tar.xz
Installing Python-3.4.10...
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?
Please consult to the Wiki page to fix the problem.
https://github.com/pyenv/pyenv/wiki/Common-build-problems
BUILD FAILED (Fedora 30 using python-build 1.2.15-2-g22c02022)
Inspect or clean up the working tree at /tmp/python-build.20191121110112.9959
Results logged to /tmp/python-build.20191121110112.9959.log
Last 10 log lines:
(cd /home/br0ke/.asdf/installs/python/3.4.10/share/man/man1; ln -s python3.4.1 python3.1)
if test "xupgrade" != "xno" ; then \
case upgrade in \
upgrade) ensurepip="--upgrade" ;; \
install|*) ensurepip="" ;; \
esac; \
./python -E -m ensurepip \
$ensurepip --root=/ ; \
fi
Ignoring ensurepip failure: pip 9.0.1 requires SSL/TLS
It may mean that I have too fresh OpenSSL version installed in my system and this version is not supported by Python I'm trying to build. I managed to install Python 3.4.10 on my Fedora 30 machine (has OpenSSL 1.1+ by default) using the following steps:
Temporary remove openssl-devel if it is installed as it will conflict with older version:
$ sudo dnf remove openssl-devel
Install compat OpenSSL 1.0 packages:
$ sudo dnf install compat-openssl10 compat-openssl10-devel
Build Python 3.4.10 (I use asdf but it should work with pyenv as well):
$ asdf install python 3.4.10
Remove compat library and install modern instead to be able to build something more fresh next time:
$ sudo dnf remove compat-openssl10-devel
$ sudo dnf install openssl-devel
Inspired by this workaround with help from Fedora community people.

If you use Manjaro Linux, you can use this:
sudo pacman -S openssl , check

If you are using RedHat Enterprise Linux, you can use this:
yum install openssl-devel -y

CentOS or RHEL machine Python3(3.7.0) ssl not supported as of now.
pip installation supported with Python <=3.6

building Python from source with zlib support

When building Python 3.2.3 from source on Ubuntu 12.04, the zlib module is not available.
I downloaded the official source distribution from python.org, and attempted to build and install it with the following commands.
tar xfa Python3.2.3.tar.bz2
cd Python-3.2.3
./configure --prefix=/opt/python3.2
make
sudo make install
The make command output includes the following.
Python build finished, but the necessary bits to build these modules were not found:
_curses _curses_panel _dbm
_gdbm _sqlite3 _ssl
_tkinter bz2 readline
zlib
After running make install and starting the interpreter, the zlib module cannot be imported.
I confirmed that the zlib1g-dev package is installed on my system.
I also found this similar question, which suggests adding the --with-zlib flag to the ./configure command. However, that returns an error that it is an unrecognized option and has no effect.

I had a similar problem on CentOS 6.3 and python 3.2.3
I solved it by:
Edit /Modules/Setup and uncomment the line:
zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
change to directory /Modules/zlib:
./configure
make
sudo make install
then compiled my python3.2 source.
and was then able to test import zlib and it all worked fine :)

I am using CentOS 6.6 and was recieving zlib errors. None of the other answers proposed here worked for me (including the fix for CentOS 6.3 of uncommenting a line in Modules/Setup). I have fixed it using the following commands.
yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
Then configuring and installing python as follows:
./configure --prefix=/usr/local LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall
I can now import zlib in /usr/local/bin/python2.7 with no problems.
These instructions are slightly modified from an article found here.

The solution is to install the Ubuntu package dpkg-dev.
sudo apt-get install dpkg-dev
The reason is explained here.
In short, recent versions of Ubuntu don't store libz.so in the standard /usr/lib location, but rather in a platform specific location. For example, on my system is is in /usr/lib/x86_64-linux-gnu. This prevents Python's build system from finding it.
The dpkg-dev package installs the dpkg-architecture executable, which enables Python to find the necessary libraries.
The original question was about Python 3.2.3. I also downloaded Python 2.7.3 and confirmed that the same problem exists, and this solution is applicable to it as well.

For anyone who's trying to use a non-system / non-standard zlib (e.g. building your own from source), make sure to pass both CPPFLAGS (not CFLAGS!) and LDFLAGS to ./configure. For example, if your zlib is in /opt/zlib:
./configure CPPFLAGS='-I/opt/zlib/include' LDFLAGS='-L/opt/zlib/lib'
make
sudo make install
I ended up going down the rabbit hole trying to figure out why our Python wasn't building with zlib support and found out that the CPython setup.py does not look at CFLAGS for include dirs, only CPPFLAGS:
https://github.com/python/cpython/blob/master/setup.py#L562

The only solution that helped me with installing python 3.5.1 was to apt-get zlib1g-dev (and other packages such as python-setuptools and python-pip) and then rebuild python 3.5.1 from source.
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get install build-essential python-dev python-setuptools python-pip python-smbus
sudo apt-get install build-essential libncursesw5-dev libgdbm-dev libc6-dev
sudo apt-get install zlib1g-dev libsqlite3-dev tk-dev
sudo apt-get install libssl-dev openssl
cd ~
mkdir build
cd build
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
tar -zxvf Python-3.5.1.tgz
cd Python-3.5.1
./configure
make
sudo make install
Taken from: https://github.com/MrYsLab/xideco/wiki/Installing-Python-3.5
As I undestand new build of python is made with inclusion of previously apt-getted related packages.
So when you browse the content of new Python-3.5.1/lib/site-packages there will be pip and setuptools. More importantly, they will be copied to any virtualenv you make using Python-3.5.1 AND this virtualenv will use THEM insted of system-default. This is very, very important to rememmber when installing new python version. Otherwise one might get into a black hole of errors such as:
zlib not installed;
"pip install ..." executed from virtualenv that installs package to system-default python instead of virtualenv.

I was having the same error while working on MAC
My MAC OS version
$ uname -v
Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64
python3.4 is used here
Issue(s)
zlib not available while using python3.4
$ python3.4 get-pip.py
Traceback (most recent call last):
File "get-pip.py", line 20204, in
main()
File "get-pip.py", line 152, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 82, in bootstrap
import pip
zipimport.ZipImportError: can't decompress data; zlib not available
Rebuilding Python fails
./configure --with-zlib-dir=/usr/local/lib
...
configure: WARNING: unrecognized options: --with-zlib-dir
...
Solution
Ensure zlib is installed .
By default it will be installed in /usr/lib
ls /usr/lib/libz.*
If not installed,
a. download and install
i)from zlib.net site
or
ii) from a git repo like the below
git clone https://github.com/madler/zlib.git
or
iii). Use the zlib source in the python source directory
Modules/zlib
b. Install zlib
./configure --prefix=/usr/local
make
sudo make install
2.Edit /Module/Setup by uncommenting the line below
"#zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz "
3.Rebuild the Python3.4 from source again
cd ${PYTHON_SRC_CODE_DIR}
./configure --prefix=${PYTHON_HOME_DIR}
make
sudo make install
4.Confirm installation
Please note gzip depends on zlib.
nbr_repeation=100
f=open("some_file.txt","at")
for line in range(nbr_repeation):
print('[{}] This file will be compressed using python zlib/gzipmodule'.format(line),file=f)
f.close()
f=open("some_file.txt","rt")
import gzip
gz=gzip.open('some_file.gz', 'wt')
for line in f : gz.write(line)
gz.close() # Like to be clean exit
f.close() # Like a clean exit
"""confirm the creation of the compressed gzip files"""
import os
print([ (file,os.stat(file)[6],"bytes") for file in os.listdir(".") if file.startswith("some")])

sudo apt-get install build-essential python-dev
Even though python-dev is for python2.7 it will still bring in all the necessary dependencies.
You will then need to do:
./configure
make
sudo make install
To rebuild python3

The easiest solution I found, is on python.org:
sudo apt-get build-dep python3.6
If that package is not available for your system, try reducing the minor version until you find a package that is available in your system’s package manager.
If you see something like this: E: You must put some ‘source’ URIs in your sources.list, Open Software & Updates and enable Source code.
I tried explaining details, on a blog post.

sudo apt-get install zlib1g-dev
is what worked for me.

For anyone having the same error on macOS Mojave, this is the easiest solution for installing/linking the header files:
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
Then just build Python again as usual (also works with pyenv builds).

This is how I've built Python 3.7 on a CentOS 7 machine without devel libraries and installed it into user's ~/.local without sudo. zlib, OpenSSL and readline modules are built.
Download Python source in .tgz format from https://www.python.org/downloads. Download zlib-devel-1.2.7-18.el7.x86_64.rpm, openssl-devel-1.0.2k-19.el7.x86_64.rpm, krb5-devel-1.15.1-50.el7.x86_64.rpm, libcom_err-devel-1.42.9-19.el7.x86_64.rpm, readline-devel-6.2-11.el7.x86_64.rpm from https://centos.pkgs.org for SSL, zlib and readline modules. Put all files into $DIST_PATH.
cd ~
DIST_PATH=<path to downloaded files>
for f in $DIST_PATH/*.rpm; do rpm2cpio $f | cpio -idmv; done
mkdir ~/usr/lib
# symlinks in ~/usr/lib64 are broken, so create new links to system libraries in ~/usr/lib and pass this folder to ./configure
for f in ~/usr/lib64/*.so; do ln -s /lib64/`readlink $f` ~/usr/lib/`basename $f`; done
tar -xzf $DIST_PATH/Python-3.7.13.tgz && cd Python-3.7.13
# That machine has devtoolset-7 with newer version GCC
scl enable devtoolset-7 bash
# curly brackets are important here, otherwise LDFLAGS is -LOME/usr/lib
./configure --enable-optimizations --prefix=$HOME/.local --with-openssl=$HOME/usr CPPFLAGS='-I${HOME}/usr/include' LDFLAGS='-L${HOME}/usr/lib'
make
# (!) altinstall is used, use python3.7 command to access newly built Python
make altinstall
rm -rf ~/usr
Links:
Unpacking RPM packages
krb5.h error
com_err.h
CPPFLAGS, LDFLAGS