building Python from source with zlib support - python

When building Python 3.2.3 from source on Ubuntu 12.04, the zlib module is not available.
I downloaded the official source distribution from python.org, and attempted to build and install it with the following commands.
tar xfa Python3.2.3.tar.bz2
cd Python-3.2.3
./configure --prefix=/opt/python3.2
make
sudo make install
The make command output includes the following.
Python build finished, but the necessary bits to build these modules were not found:
_curses _curses_panel _dbm
_gdbm _sqlite3 _ssl
_tkinter bz2 readline
zlib
After running make install and starting the interpreter, the zlib module cannot be imported.
I confirmed that the zlib1g-dev package is installed on my system.
I also found this similar question, which suggests adding the --with-zlib flag to the ./configure command. However, that returns an error that it is an unrecognized option and has no effect.

I had a similar problem on CentOS 6.3 and python 3.2.3
I solved it by:
Edit /Modules/Setup and uncomment the line:
zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
change to directory /Modules/zlib:
./configure
make
sudo make install
then compiled my python3.2 source.
and was then able to test import zlib and it all worked fine :)

I am using CentOS 6.6 and was recieving zlib errors. None of the other answers proposed here worked for me (including the fix for CentOS 6.3 of uncommenting a line in Modules/Setup). I have fixed it using the following commands.
yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
Then configuring and installing python as follows:
./configure --prefix=/usr/local LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall
I can now import zlib in /usr/local/bin/python2.7 with no problems.
These instructions are slightly modified from an article found here.

The solution is to install the Ubuntu package dpkg-dev.
sudo apt-get install dpkg-dev
The reason is explained here.
In short, recent versions of Ubuntu don't store libz.so in the standard /usr/lib location, but rather in a platform specific location. For example, on my system is is in /usr/lib/x86_64-linux-gnu. This prevents Python's build system from finding it.
The dpkg-dev package installs the dpkg-architecture executable, which enables Python to find the necessary libraries.
The original question was about Python 3.2.3. I also downloaded Python 2.7.3 and confirmed that the same problem exists, and this solution is applicable to it as well.

For anyone who's trying to use a non-system / non-standard zlib (e.g. building your own from source), make sure to pass both CPPFLAGS (not CFLAGS!) and LDFLAGS to ./configure. For example, if your zlib is in /opt/zlib:
./configure CPPFLAGS='-I/opt/zlib/include' LDFLAGS='-L/opt/zlib/lib'
make
sudo make install
I ended up going down the rabbit hole trying to figure out why our Python wasn't building with zlib support and found out that the CPython setup.py does not look at CFLAGS for include dirs, only CPPFLAGS:
https://github.com/python/cpython/blob/master/setup.py#L562

The only solution that helped me with installing python 3.5.1 was to apt-get zlib1g-dev (and other packages such as python-setuptools and python-pip) and then rebuild python 3.5.1 from source.
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get install build-essential python-dev python-setuptools python-pip python-smbus
sudo apt-get install build-essential libncursesw5-dev libgdbm-dev libc6-dev
sudo apt-get install zlib1g-dev libsqlite3-dev tk-dev
sudo apt-get install libssl-dev openssl
cd ~
mkdir build
cd build
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
tar -zxvf Python-3.5.1.tgz
cd Python-3.5.1
./configure
make
sudo make install
Taken from: https://github.com/MrYsLab/xideco/wiki/Installing-Python-3.5
As I undestand new build of python is made with inclusion of previously apt-getted related packages.
So when you browse the content of new Python-3.5.1/lib/site-packages there will be pip and setuptools. More importantly, they will be copied to any virtualenv you make using Python-3.5.1 AND this virtualenv will use THEM insted of system-default. This is very, very important to rememmber when installing new python version. Otherwise one might get into a black hole of errors such as:
zlib not installed;
"pip install ..." executed from virtualenv that installs package to system-default python instead of virtualenv.

I was having the same error while working on MAC
My MAC OS version
$ uname -v
Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64
python3.4 is used here
Issue(s)
zlib not available while using python3.4
$ python3.4 get-pip.py
Traceback (most recent call last):
File "get-pip.py", line 20204, in
main()
File "get-pip.py", line 152, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 82, in bootstrap
import pip
zipimport.ZipImportError: can't decompress data; zlib not available
Rebuilding Python fails
./configure --with-zlib-dir=/usr/local/lib
...
configure: WARNING: unrecognized options: --with-zlib-dir
...
Solution
Ensure zlib is installed .
By default it will be installed in /usr/lib
ls /usr/lib/libz.*
If not installed,
a. download and install
i)from zlib.net site
or
ii) from a git repo like the below
git clone https://github.com/madler/zlib.git
or
iii). Use the zlib source in the python source directory
Modules/zlib
b. Install zlib
./configure --prefix=/usr/local
make
sudo make install
2.Edit /Module/Setup by uncommenting the line below
"#zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz "
3.Rebuild the Python3.4 from source again
cd ${PYTHON_SRC_CODE_DIR}
./configure --prefix=${PYTHON_HOME_DIR}
make
sudo make install
4.Confirm installation
Please note gzip depends on zlib.
nbr_repeation=100
f=open("some_file.txt","at")
for line in range(nbr_repeation):
print('[{}] This file will be compressed using python zlib/gzipmodule'.format(line),file=f)
f.close()
f=open("some_file.txt","rt")
import gzip
gz=gzip.open('some_file.gz', 'wt')
for line in f : gz.write(line)
gz.close() # Like to be clean exit
f.close() # Like a clean exit
"""confirm the creation of the compressed gzip files"""
import os
print([ (file,os.stat(file)[6],"bytes") for file in os.listdir(".") if file.startswith("some")])

sudo apt-get install build-essential python-dev
Even though python-dev is for python2.7 it will still bring in all the necessary dependencies.
You will then need to do:
./configure
make
sudo make install
To rebuild python3

The easiest solution I found, is on python.org:
sudo apt-get build-dep python3.6
If that package is not available for your system, try reducing the minor version until you find a package that is available in your system’s package manager.
If you see something like this: E: You must put some ‘source’ URIs in your sources.list, Open Software & Updates and enable Source code.
I tried explaining details, on a blog post.

sudo apt-get install zlib1g-dev
is what worked for me.

For anyone having the same error on macOS Mojave, this is the easiest solution for installing/linking the header files:
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
Then just build Python again as usual (also works with pyenv builds).

This is how I've built Python 3.7 on a CentOS 7 machine without devel libraries and installed it into user's ~/.local without sudo. zlib, OpenSSL and readline modules are built.
Download Python source in .tgz format from https://www.python.org/downloads. Download zlib-devel-1.2.7-18.el7.x86_64.rpm, openssl-devel-1.0.2k-19.el7.x86_64.rpm, krb5-devel-1.15.1-50.el7.x86_64.rpm, libcom_err-devel-1.42.9-19.el7.x86_64.rpm, readline-devel-6.2-11.el7.x86_64.rpm from https://centos.pkgs.org for SSL, zlib and readline modules. Put all files into $DIST_PATH.
cd ~
DIST_PATH=<path to downloaded files>
for f in $DIST_PATH/*.rpm; do rpm2cpio $f | cpio -idmv; done
mkdir ~/usr/lib
# symlinks in ~/usr/lib64 are broken, so create new links to system libraries in ~/usr/lib and pass this folder to ./configure
for f in ~/usr/lib64/*.so; do ln -s /lib64/`readlink $f` ~/usr/lib/`basename $f`; done
tar -xzf $DIST_PATH/Python-3.7.13.tgz && cd Python-3.7.13
# That machine has devtoolset-7 with newer version GCC
scl enable devtoolset-7 bash
# curly brackets are important here, otherwise LDFLAGS is -LOME/usr/lib
./configure --enable-optimizations --prefix=$HOME/.local --with-openssl=$HOME/usr CPPFLAGS='-I${HOME}/usr/include' LDFLAGS='-L${HOME}/usr/lib'
make
# (!) altinstall is used, use python3.7 command to access newly built Python
make altinstall
rm -rf ~/usr
Links:
Unpacking RPM packages
krb5.h error
com_err.h
CPPFLAGS, LDFLAGS

Related

Python build from source have error "no module named math", "no module named setuptools" etc

i just build python 3.11 (its no matter which version because i have the same problem with other versions) on cygwin and catch this erorr. dyn-load are in PATH - thats ok. may be i need some env to setting up?
I configure it by
./configure --prefix="/usr" --enable-optimizations
You are probably lacking system libraries needed to build Python from source
Necessary packages
sudo apt-cyg install python-dev # for math
sudo apt-cyg install python3-setuptools # for setup tools
sudo apt-cyg install build-essential # you probably need this too

How to compile and install python3.9.6 on unbuntu

I'm trying to install python3.9.6 on ubuntu
apt only had python3.8
so I tried this https://tecadmin.net/how-to-install-python-3-9-on-ubuntu-18-04/
but it installed python3.9.5,
next, I tried to compile and build python but it didn't install pip so I had to install zlib and spend like 5 days trying to make it work, and it did work and I was able to install both python2.7.18 and 3.9.6 with pip but it didn't install the SSL module so I had to install that and bla bla...
it worked fine after installing openssl but when I tried to install scapy it showed an error message, after some research I found out that the error was caused by outdated SSL module
I figured that compiling and building python had too many problems it didn't installed all the packages for tools like pip.
if I spend some more time I think could fix this but I'm worried that this kind of problem
could happen again in the future,
I'm really desperate, so if you got any ideas please let me know.
1. Update your local repositories
sudo apt update
2. Install supporting software (installing from source requires additional tools)
sudo apt install build-essential zlib1g-dev libncurses5-dev \
libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev wget
3. Download the latest version of Python Source Code
You might want to do this in a separate directory (like /tmp)
wget https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz
4. Extract downloaded files
tar -xf Python-3.9.6.tgz
5. Test system and optimize python
cd Python-3.9.6
./configure --enable-optimizations
This might take a bit of time to complete
6a. Install a second instance of Python (highly recommended)
sudo make altinstall
It is recommended that you use the altinstall method. Your Ubuntu system may have software packages dependent on Python2.x/3.x.
6b. Overwrite default python installation (not recommended!!!)
sudo make install
7. Verify Python installation
python3 --version
# or
python3.6 --version

Building Python 3.7.1 - SSL module failed

Building Python 3.7 from source runs into following error:
Failed to build these modules:
_hashlib _ssl
Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381
I tried so many workarounds from other stackoverflow-questions, but it doesnt work. I build newest OpenSSL and LibreSSL from source. OpenSSL path is: "/usr/local/ssl" with version OpenSSL 1.0.2p.
./configure --with-openssl=/usr/local/ssl/
(./configure CPPFLAGS="-I/usr/local/ssl/include" LDFLAGS="-L/usr/local/ssl/lib")
make
make altinstall
My system:
Ubuntu 12.04.5 LTS
Any ideas?
I solved it after 3 days only because of this blog. with python 3.7.4 openssl 1.1.0 centOS 6.
here is the summary :
First, some prerequisites:
sudo apt-get install build-essential checkinstall libreadline-gplv2-dev libncursesw5-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
use yum instead of apt-get if using centos linux.
Install ssl 1.0.2 or higher.
cd /usr/src
curl https://www.openssl.org/source/openssl-1.0.2o.tar.gz | tar xz
cd openssl-1.0.2o
./config shared --prefix=/usr/local/
sudo make
sudo make install
We will need to pass /usr/src/openssl-1.0.2o into the Python configure script.
mkdir lib
cp ./*.{so,so.1.0.0,a,pc} ./lib
Now proceed with installing Python:
cd /usr/src
sudo wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
sudo tar xzf Python-3.7.0.tgz
cd Python-3.7.0
./configure --with-openssl=/usr/src/openssl-1.0.2o --enable-optimizations
sudo make
sudo make altinstall
To test it out, run python3.7 and input:
import ssl
ssl.OPENSSL_VERSION
Hope it helps!
Compiling openssl
Download your openssl tarball, unzip, and then ensure that the install directory is named openssl.
I placed mine in /usr/local/openssl, so I'll use that in my example.
sudo mv openssl-1.0.2u /usr/local/openssl && cd /usr/local/openssl
sudo make distclean
sudo ./config -fPIC -shared
sudo make && sudo install
Now, add the openssl shared library to your PATH.
vim ~/.profile
Go
export LD_LIBRARY_PATH="/usr/local/openssl/lib:$LD_LIBRARY_PATH"
:wq
Compiling Python3
The key here is understanding that the path you define with --with-openssl= is where Python looks for /openssl/lib. You need to give Python the parent directory of the openssl directory.
That means that if you set --with-openssl=/usr/local/openssl your make install will fail even though the make logs show that openssl is fine!
--enable-optimizations is irrelevant but recommended - longer make for 10% faster Python code is a good tradeoff.
--prefix= is merely where I'd like python3 to install, if you didn't know.
sudo make distclean
Edit your python setup file
vim /{yourpythonsource}/Modules/Setup
Uncomment out the following lines and ensure that your SSL variable points to your openssl directory. In mine, it was looking for the directory 'ssl' instead of 'openssl.'
<pre><code># Socket module helper for SSL support; you must comment out the other </code>
<pre><code># socket line above, and possibly edit the SSL variable: </code>
<code>SSL=/usr/local/openssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto</code>
sudo ./configure --with-openssl=/usr/local --prefix=/opt/python-3.7.1
sudo make && sudo make install
While this might not be the best answer, I will share how I solved this problem.
First of all, in my case, OpenSSL did not build correctly, as make test did return errors (and consequently Python gave this error). This was solved by installing a newer version of Perl and then installing OpenSSL again (configure, make, etc).
Use this command before using ./configure
export LD_LIBRARY_PATH=/path/to/openssl/lib:$LD_LIBRARY_PATH
At the configure command, include the library:
LDFLAGS="-L/path/to/openssl/lib" ./configure (all your preferred options) --with-openssl=/path/to/openssl
as apparently the option for configure does not convey the message to the C compiler which needs it.
Am not sure whether option 2 and 3 are needed simultaneously, but I did so and it worked.
Edit setup.py
Find the following lines:
system_lib_dirs = ['/lib64', '/usr/lib64', '/lib', '/usr/lib']
system_include_dirs = ['/usr/include']
...and place each folder at the beginning of its respective list.
In my case I had to add: /usr/local/lib and /usr/local/include:
system_lib_dirs = ['/usr/local/lib', '/lib64', '/usr/lib64', '/lib', '/usr/lib']
system_include_dirs = ['/usr/local/include', '/usr/include']
Finally: make distclean && ./configure
You may want to ensure that export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH (or what have you) is added to the very end of /etc/profile and reboot, as well.
On CentOS / Linux 2 if you install openssl using
sudo yum install -y openssl-devel
then the library is installed to /usr/local/lib64, and you can configure Python as follows:
./configure --enable-shared --with-openssl=/usr/local/lib64
there are step-by-step instructions here: How to Install Latest (2020) Django to AWS EC2 Linux 2 Instance and Serve w/ Apache Hello World
I ran into this problem with LMDE 5 (running Debian Bullseye) compiling Python 3.10.4. It was fixed by doing:
sudo apt-get install libssl-dev
There was NO need to edit Modules/Setup file built python with customed openssl.
I have built python 3.11.0-rc2 under Debian 9 stretch follow the official document:
https://docs.python.org/3/using/unix.html?highlight=openssl#custom-openssl
To use your vendor’s OpenSSL configuration and system trust store, locate the directory with openssl.cnf file or symlink in /etc. On most distribution the file is either in /etc/ssl or /etc/pki/tls. The directory should also contain a cert.pem file and/or a certs directory.
$ find /etc/ -name openssl.cnf -printf "%h\n"
/etc/ssl
Download, build, and install OpenSSL. Make sure you use install_sw and NOT install. The install_sw target does NOT override openssl.cnf.
$ curl -O https://www.openssl.org/source/openssl-VERSION.tar.gz
$ tar xzf openssl-VERSION
$ pushd openssl-VERSION
$ ./config \
--prefix=/usr/local/custom-openssl \
--libdir=lib \
--openssldir=/etc/ssl
$ make -j1 depend
$ make -j8
$ make install_sw
$ popd
Build Python with custom OpenSSL (see the configure --with-openssl and --with-openssl-rpath options)
$ pushd python-3.x.x
$ ./configure -C \
--with-openssl=/usr/local/custom-openssl \
--with-openssl-rpath=auto \
--prefix=/usr/local/python-3.x.x
$ make -j8
$ make altinstall
ssl module check after installed :
# /usr/local/python-3.11.0-rc2/bin/python3.11 -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.1.1q 5 Jul 2022
Met same issue, looks configure of Python3 can't work well.
If you have installed the latest openssl, make sure the path of OPENSSL_LDFLAGS is correct in Makefile, below is my env case
OPENSSL_LDFLAGS=-L/usr/local/lib64
Execute till download python (3.10.4 is what i tried) from the link below
https://computingforgeeks.com/install-latest-python-on-centos-linux/
Upgrade openssl as documented in https://cloudwafer.com/blog/installing-openssl-on-centos-7/
modify $python_home/Modules/Setup
Update the OPENSSL location and uncomment the below lines
--------------------------------------------
OPENSSL=/usr/local/ssl
_ssl _ssl.c \
-I$(OPENSSL)/include -L$(OPENSSL)/lib \
-lssl -lcrypto
--------------------------------------------
Continue the installation steps from https://computingforgeeks.com/install-latest-python-on-centos-linux/
Hope it helps somebody..
fyi: I was installing this on a centos7 ec2 instance as a part of installing ansible.
Here is a solution on Mac OS X / Homebrew:
brew reinstall openssl
brew unlink openssl && brew link openssl --force # careful!
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile
Then download your python tarball and do this:
tar xvf Python-3.7.2.tar
cd Python-3.7.2
./configure CPPFLAGS="-I/usr/local/opt/openssl/include" LDFLAGS="-L/usr/local/opt/openssl/lib" --prefix=$PWD/Python-3.7.2/mybuild --enable-optimizations
More detai:
https://devguide.python.org/setup/#macos-and-os-x

Python 3 No module named '_ssl'

The Problem
While I run you python3 application, it shows
File "/usr/local/lib/python3.6/ssl.py", line 101, in <module>
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
What I've tried
install the dependencies
yum install openssl-devel
I also edited the setup.py file and recomplie python3
# Detect SSL support for the socket module (via _ssl)
search_for_ssl_incs_in = [
'/usr/local/ssl/include',
'/usr/local/include/openssl', #I've added this line
'/usr/contrib/ssl/include/'
]
I've complied the openssl with the path configuration
#tar -xzvf openssl-***.tar.gz
#./config --prefix=/usr/local --openssldir=/usr/local/openssl
#make & make install
CentOS 7
Python 3.6
I found some solution:
if you use centos,try:
s1
yum install openssl-devel -y
then when you compile, append --with-ssl,just like this
./configure prefix=/usr/local/share/python3 --with-ssl
s2
-- install depend library, make share compile is fluent
yum install -y zlib zlib-devel openssl-devel sqlite-devel bzip2-devel libffi libffi-devel gcc gcc-c++
(ubuntu)sudo apt-get install libz-dev
wget --no-check-certificate http://www.openssl.org/source/openssl-1.1.1.tar.gz
tar -zxvf openssl-1.1.1.tar.gz
cd openssl-1.1.1
./config --prefix=$HOME/openssl shared zlib
make && make install
-- configure shared ld library path so that compile can find it
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/openssl/lib" >> $HOME/.bash_profile
source $HOME/.bash_profile
(zsh user has some different with .zsh_profile)
-- compile with openssl path
./configure prefix=/usr/local/share/python3 --with-openssl=$HOME/openssl
I faced the same issue, I installed python from source and didn't enabled ssl option while compiling. So I find the solution in the following article. You need to find ssl section Modules/Setup.dist and uncomment that section. Hope this will help someone.

Installing distribute in Python 3.3 Ubuntu

I am running Ubuntu 12.04 and I have a distribution of Python 3.3.1 installed. I want to install some packages, so I first sought to install distribute-0.6.38. During the "install" phase, I am encountering the following runtime error ($HOME is the location of my Python3.3 installation):
File "$HOME/Python-3.3.1/Lib/zipfile.py", line 583, in _check_compression
"Compression requires the (missing) zlib module"
RuntimeError: Compression requires the (missing) zlib module
I tracked back through the files and function calls, but cannot tell why the creation of the zipfile (I assume this is the root of the error) failed.
Is there something missing from the package? Or is there an issue with the fact that this is a secondary installation of Python?
It is an issue with the fact you installed Python from source.
You need to install the zlib1g-dev package to provide the headers to Python to be able to compile in zlib support:
sudo apt-get install zlib1g-dev
You may be missing other dependencies; here is a list of packages I'd install if I were to compile Python on an Ubuntu machine:
build-essential
libbz2-dev
libncursesw5-dev
libreadline5-dev
libssl-dev
libgdbm-dev
libc6-dev
libsqlite3-dev
tk-dev
You need to have the zlib1g-dev library installed.
Try
sudo apt-get install zlib1g-dev
and reinstall.
If that doesn't work you may need to specify where zlib installs like so;
python setup.py install

Categories