Coredump when compiling python with a custom openssl version - python
When compiling python-3.4.0rc3 with a local openssl-1.0.1f shared install, make prints no error but then I get the following core dump on make install or make test:
Program terminated with signal 11, Segmentation fault.
(gdb) bt
#0 0x00007f131dd10510 in EVP_PKEY_CTX_dup () from /data2/soft/openssl/lib/libcrypto.so.1.0.0
#1 0x00007f131dd0284f in EVP_MD_CTX_copy_ex () from /data2/soft/openssl/lib/libcrypto.so.1.0.0
#2 0x00007f131e256ab5 in EVPnew (name_obj=0x7f131e46a500, digest=0x0, initial_ctx=0x7f131e459a40, cp=0x0, len=0) at /data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.c:410
#3 0x00007f131e25726e in EVP_new_md5 (self=<value optimized out>, args=<value optimized out>) at /data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.c:799
#4 0x00000000004c7eef in ?? ()
Here is the complete list of commands used
tar -axf Python-3.4.0rc3.tgz
cd Python-3*
# For lzma and a pip-compatible openssl
export CFLAGS='-I/data2/soft/openssl/include/openssl -I/data2/local/include/'
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/:$LD_LIBRARY_PATH"
# Ready !
./configure --prefix=/data2/soft/python3
make
make install
Notes:
OS is SUSE Linux Enterprise Server 11 (x86_64)
pointing python to a custom location for the lzma lib located in
openssl was built with ./config shared --openssldir=/data2/soft/openssl
openssl make test prints ALL TESTS SUCCESSFUL.
without the custom openssl *FLAGS, make install is successful and I get these results for make test:
71 tests OK.
tests failed:
test_cmd_line test_gdb test_smtpnet test_ssl
How can I fix this, or at least investigate what is going on ?
Edit 1--5:
The shared libs were generated correctly:
> ls /data2/soft/openssl/lib
drwxr-xr-x engines
-rw-r--r-- libcrypto.a
lrwxrwxrwx libcrypto.so -> libcrypto.so.1.0.0
-r-xr-xr-x libcrypto.so.1.0.0
-rw-r--r-- libssl.a
lrwxrwxrwx libssl.so -> libssl.so.1.0.0
-r-xr-xr-x libssl.so.1.0.0
drwxr-xr-x pkgconfig
So I changed this in Setup :
SSL=/data2/soft/openssl/
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
$(SSL)/lib/libssl.a $(SSL)/lib/libcrypto.a -ldl
And I changed back LDFLAGS/CFLAGS accordingly. But there still is a -lssl when I run make clean && make, because of the _hashopen module:
gcc -pthread -shared -L/data2/local/lib/ -L/data2/local/lib/ -L/data2/local/lib/ -I/data2/local/include/ build/temp.linux-x86_64-3.4/data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.o -L/data2/local/lib/ -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
I guess it is the one causing the cores, because they are still there... I tried adding similar stuff to Setup file, but there is no commented item for this one and creating it causes another more cryptic failure:
gcc -pthread -Xlinker -export-dynamic -o python Modules/python.o libpython3.4m.a -lpthread -ldl -lutil /data2/eoubrayrie/soft/openssl/lib/libssl.a /data2/eoubrayrie/soft/openssl/lib/libcrypto.a -ldl /data2/eoubrayrie/soft/openssl/lib/libssl.a /data2/eoubrayrie/soft/openssl/lib/libcrypto.a -ldl -lm
libpython3.4m.a(config.o):(.data+0x158): undefined reference to `PyInit__hashopenssl'
collect2: ld returned 1 exit status
Edit 6:
I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together
But I can get it linking dynamically to my own openssl through good old -I/-L:
ldd build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
libssl.so.1.0.0 => /data2/soft/openssl/lib/libssl.so.1.0.0 (0x00007f5605799000)
libcrypto.so.1.0.0 => /data2/soft/openssl/lib/libcrypto.so.1.0.0 (0x00007f56053bd000)
Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?
From To Syms Read Shared Object Library
0x00007ffff5465930 0x00007ffff5466e98 Yes /data2/soft/python3/Python-3.4.0rc3/build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
0x00007ffff5321220 0x00007ffff5351878 Yes /opt/python-2.6-64/lib/libssl.so.1.0.0
0x00007ffff50d3100 0x00007ffff519b118 Yes /opt/python-2.6-64/lib/libcrypto.so.1.0.0
env | grep -F 'python-2.6-64' -> shows nothing !
grep -RF 'python-2.6-64' /etc/ld.so.* -> idem
gcc -print-search-dirs | sed 's/:/\n/g' | grep python -> idem
find . -name '*.so*' | xargs ldd | grep ssl -> only gives me the good ones
Level 1 dependencies don't require any wrong ssl version either. This was checked with: find . -name '*.so*' | xargs ldd | awk '/\t+[[:alnum:].]+ => [[:alnum:]./]+ \(/ {print $3}' | sort | uniq | xargs ldd | grep ssl
strace ./python ./Tools/scripts/run_tests.py 2>&1 | grep python-2.6-64 -> shows nothing
So how does ld picks this wrong library if he cannot konow about it ?? It's not in any standard location (if it were in /lib I could understand...)
Solution:
Found how to statically link _hashlib thanks to this OpenOffice bug: though the -Wl,--exclude-libs=ALL" option didn't work either, it pointed me to the right lines in setup.py.
TL;DR Here is the patch to setup.py I applied.
And finally... it works !
#noloader I'm accepting your most complete answer as you help was invaluable, though the "exact" answer for anyone encountering this problem is to compile with the patch above.
How can I fix this, or at least investigate what is going on ?
...
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/
I run into these problems a lot too because I avoid the crippled versions of OpenSSL shipped by Debian, Ubuntu, Fedora, et al. For example, Ubuntu ships an OpenSSL that disables TLSv1.1 and TLS v1.2 (cf., Ubuntu 12.04 LTS: OpenSSL downlevel version and does not support TLS 1.2).
You're probably loading the wrong version of the OpenSSL library. If you can get the misbehaving program under the debugger, issue info shared to see which libcrypto and libssl you are actually loading.
ldd might help too. Run it on the Pyhton executable: ldd /data2/soft/python3/python. I can only say it "may" help because the OpenSSL's are binary compatible, so you might only see a dependency on, for example, libcrypto.so.1.0.0 (use otool -L on Mac OS X). Below I used an rpath to force linking against the libraries in /usr/local/ssl/lib/.
$ ldd my-test.exe
linux-vdso.so.1 => (0x00007fffd61ff000)
libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00007f151528e000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007f1514e74000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1514c42000)
...
As a fix, you might try adding an rpath:
LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/ -Wl,-rpath,/data2/soft/openssl/lib'
A better fix would be to link to the static version of the OpenSSL library to avoid these problems all together. I believe you can do that with: -Bstatic -lssl -lcrypto -Bdynamic -ldl.
Personally, I don't even use -Bstatic because of different one-off problems. I open the Makefiles, remove all instances of -lssl -lcrypto, and add a full path to the archive to remove all ambiguity. For example, /data2/soft/openssl/lib/libssl.a and /data2/soft/openssl/lib/libcrypto.a.
Note well: an rpath is not honored on Mac OS X. More extreme measures are needed for Mac OS X because the linker does not honor -Bstatic either. You'll have to use the full path trick.
openssl was built with ./config shared --openssldir=/data2/soft/openssl
Another thing... On Fedora, its not enough to specify shared. You need to add the following, too:
export CFLAGS="-fPIC"
Otherwise, the shared object is not built. If the 1.0.1f shared object is not built, you're probably getting a downlevel version supplied by the distro. You can check what was built before install with:
./config shared --openssldir=/data2/soft/openssl
make all
# Verify artifacts
find . -iname -libcrypto.*
find . -iname -libssl.*
# Proceed if OK
sudo make install
Finally, make sure that all of Python's dependencies are also using your version of OpenSSL and not the system's version of OpenSSL.
I recently suffered that problem when my program used my OpenSSL; but my program also used libevent and libevent used the system's version of OpenSSL. I fixed it by rebuilding libevent and forcing it to statically link to my version of OpenSSL.
How should I go about changing it ? Will adding a _ssl.c: gcc ... line anywhere override the default behaviour ? My Makefile skills are rusted and this beast is 1600+ lines long !!
Here are the files you want to look at:
$ cd Python-3.4.0rc3
$ grep -R -- '-lcrypto' *
Modules/Setup.dist:# -L$(SSL)/lib -lssl -lcrypto
$ grep -R -- '-lssl' *
Modules/Setup.dist:# -L$(SSL)/lib -lssl -lcrypto
Here are the lines of interest in Setup.dist:
# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
#_ssl _ssl.c \
# -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
# -L$(SSL)/lib -lssl -lcrypto
Uncomment the lines and change Setup.dist to the following. I keep my OpenSSL in /usr/local/ssl, so that's how mine is setup below (you should use /data2/soft/openssl/lib/...):
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
/usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Use the full path to the archive, and don't use -l. Be sure to add -ldl because OpenSSL needs it in this configuration.
Once you change Setup.dist, re-run ./configure to effect the changes.
After changing the above line to, here's what I look like after a configure:
$ grep -R "libssl.a" *
Makefile:LOCALMODLIBS= /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Makefile:Modules/_ssl$(SO): Modules/_ssl.o; $(BLDSHARED) Modules/_ssl.o /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl -o Modules/_ssl$(SO)
Modules/Setup: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Modules/Setup.dist: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
You can test the static linking by using ldd or otool -L. You will not see an OpenSSL dependency. After make'ing, here's what I got:
$ find . -iname python
./python
$ ldd ./python
linux-vdso.so.1 => (0x00007fff67709000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3aed8e1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3aed6dd000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3aed4d9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3aed257000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3aececc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3aedb14000)
No libssl or libcrypto dependencies to go wrong :) And make test ran fine (actually, one failed test due to a Python bug: Issue 20896):
======================================================================
ERROR: test_get_server_certificate (test.test_ssl.NetworkedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1373, in test_get_server_certificate
_test_get_server_certificate('svn.python.org', 443, SVN_PYTHON_ORG_ROOT_CERT)
File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1354, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port))
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 902, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 344, in wrap_socket
_context=self)
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 540, in __init__
self.do_handshake()
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 767, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:598)
----------------------------------------------------------------------
Ran 96 tests in 8.610s
FAILED (errors=1, skipped=3)
test test_ssl failed
make: *** [test] Error 1
This isn't an answer, just an observation after running though this with you:
$ make
...
gcc -pthread -c -Wno-unused-result -Werror=declaration-after-statement
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include
-DPy_BUILD_CORE -o Objects/capsule.o Objects/capsule.c
-fwrapv is really bad. Its used to make illegal programs work. Its better to fix the broken program and drop the -fwrapv. See Ian Lance Taylor's blog on Signed Overflow.
I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together
grep is your friend ;)
$ grep -R _hashlib * | grep ssl
Lib/hashlib.py: f = getattr(_hashlib, 'openssl_' + name)
Lib/hashlib.py: _hashlib.openssl_md_meth_names)
Lib/test/test_hashlib.py: self.assertTrue(hasattr(_hashlib, 'openssl_md5'))
Lib/test/test_hashlib.py: self.assertTrue(hasattr(_hashlib, 'openssl_sha1'))
Lib/test/test_hashlib.py: constructor = getattr(_hashlib, 'openssl_'+algorithm, None)
Lib/test/ssltests.py:TESTS = ['test_asyncio', 'test_ftplib', 'test_hashlib', 'test_httplib',
Lib/test/time_hashlib.py: print(" '_hashlib' 'openssl_hName' 'fast' tests the builtin _hashlib")
Modules/_hashopenssl.c: "_hashlib.HASH", /*tp_name*/
Modules/_hashopenssl.c:static struct PyModuleDef _hashlibmodule = {
Modules/_hashopenssl.c: "_hashlib",
Modules/_hashopenssl.c:PyInit__hashlib(void)
Modules/_hashopenssl.c: m = PyModule_Create(&_hashlibmodule);
PCbuild/build_ssl.py:# Script for building the _ssl and _hashlib modules for Windows.
PCbuild/build_ssl.py:# for the actual _ssl.pyd and _hashlib.pyd DLLs.
PCbuild/build_ssl.py:# it should configure and build SSL, then build the _ssl and _hashlib
setup.py: exts.append( Extension('_hashlib', ['_hashopenssl.c'],
setup.py: print("warning: openssl 0x%08x is too old for _hashlib" %
Tools/ssl/test_multiple_versions.py: "test_asyncio", "test_ftplib", "test_hashlib", "test_httplib",
Tools/ssl/test_multiple_versions.py:MINIMAL_TESTS = ["test_ssl", "test_hashlib"]
Another partial answer...
But I can get it linking dynamically to my own openssl through good old -I/-L:
...
Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?
That's your good old friends -l and -L. Don't use them because they do this sort of thing all the time (take it from a guy who has suffered it in the past). Instead, specify the full path to libssl and libcrypto. E.g., use /data2/soft/openssl/lib/libssl.a.
We had a similar problem. We are using apache httpd + mod_wsgi + python + django and our c++ module for the apache httpd which also uses openssl. Now everything is loaded within one httpd process, correct version of openssl shared lib was loaded (1.0.0l) with our c++ module. But as soon as we access the web, python loads the hashlib and exactly the same problem appears - segfault in openssl called from python.
Normally python compiles with whatever openssl is available at the default location and there is no way how to specify it without fiddling with setup.py or makefiles. Python developers should add configure setting --with_ssl=path.
We installed new openssl libs and rebuild python and other binaries but with no success. We mapped default libssl.so and libcrypto.so to the new openssl binaries with no success. Finally after reading this thread I realized that probably wrong headers are being used while compiling python. And that was the problem. Follow the step to workaround the problem:
there must not be incorrect version of openssl headers in /usr/include and default locations /usr/local/ssl, /usr/contrib/ssl (if you installed openssl-devel then uninstall it, or simply erase/rename the directory)
yum remove openssl-devel
make symbolic link to your openssl installation from /usr/local/ssl
ln -s /opt/openssl-1.0.1l /usr/local/ssl
ensure that the new openssl libs are accessible from /usr/lib (make symbolic links if it is not installed here)
ln -s /opt/openssl-1.0.1l/lib/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.1.0.0
...
now configure and clean build the python
I am able to run python 2.7.11 with non-default SSL after patching fix mentioned here https://gist.github.com/eddy-geek/9604982
However, with this it's not building _socket module which is required by many other modules. for example, easy_install / pip started failing with error Importerr: no module named _socket
In Module/Setup.dist, am i suppose to uncomment or comment the line
_socket socketmodule.o
?
I see socketmodule.o and timemodule.o getting generated. but not _socket.so
Am i missing something ?
Related
Unable to cross-compile Python-2.7.18 for x86,uclibc
I'm trying to cross-compile Python 2.7.18 for an x86,uclibc machine using a crosstool-ng example toolchain. The commands used are the following: CONFIG_SITE=config.site CC=/home/msainz/x-tools/x86_64-unknown-linux-uclibc/bin/x86_64-unknown-linux-uclibc-gcc CXX=/home/msainz/x-tools/x86_64-unknown-linux-uclibc/bin/x86_64-unknown-linux-uclibc-g++ AR=/home/msainz/x-tools/x86_64-unknown-linux-uclibc/bin/x86_64-unknown-linux-uclibc-ar RANLIB=/home/msainz/x-tools/x86_64-unknown-linux-uclibc/bin/x86_64-unknown-linux-uclibc-ranlib READELF=/home/msainz/x-tools/x86_64-unknown-linux-uclibc/bin/x86_64-unknown-linux-uclibc-readelf LDFLAGS="-L/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/lib -L/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/usr/lib" CFLAGS="-I/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/usr/include -I/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/include" CPPFLAGS="-I/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/usr/include -I/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/include" ./configure --enable-shared --host=x86_64-unknown-linux-uclibc --build=x86_64 --disable-ipv6 --prefix=/home/msainz/Projects/python2_top_uclibc/ followed by PATH=$PATH:/home/msainz/Projects/python2_top_glibc/bin/ make and PATH=$PATH:/home/msainz/Projects/python2_top_glibc/bin/ make install Execution ends with the following error: fi /home/msainz/x-tools/x86_64-unknown-linux-uclibc/bin/x86_64-unknown-linux-uclibc-gcc -L/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/lib -L/home/msainz/Projects/Selene/WP3/local/uclibc/base_rootfs/usr/lib -Xlinker -export-dynamic -o python \ Modules/python.o \ -L. -lpython2.7 -ldl -lpthread -lm _PYTHON_PROJECT_BASE=/home/msainz/Projects/Python-2.7.18 _PYTHON_HOST_PLATFORM=linux2-x86_64 PYTHONPATH=./Lib:./Lib/plat-linux2 python -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi python: error while loading shared libraries: libc.so.0: cannot open shared object file: No such file or directory generate-posix-vars failed make: *** [Makefile:523: pybuilddir.txt] Error 1 python2_top_glibc dir contains a previous Python-2.7.18 installation but for native glibc which was compiled perfectly. libc.so.0 is in fact in the base_rootfs of target system, which is being linked in ./configure stage. I'm stuck at this at the moment. Any clue will be appreciated. Any additional info will be supplied on demand. Thanks in advance.
python: cannot open shared object file: No such file or directory This is a run-time loader error. You are trying to run a python executable that is linked against that libc.so.0. If this executable can actually run in your host environment, you can enable it by adding your base_rootfs library to LD_LIBRARY_PATH. Otherwise, you need to use your host python executable in this step of the build process, or disable it altogether.
how to succesfully compile python 3.x
Upon attempting to compile python 3.7 I hit Could not import runpy module: jeremyr#b88:$ wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz .... jeremyr#b88:~/Python-3.7.3$ ./configure --enable-optimizations jeremyr#b88:~/Python-3.7.3$ make clean jeremyr#b88:~/Python-3.7.3$ make -j32 .... gcc -pthread -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o libpython3.7m.a -lcrypt -lpthread -ldl -lutil -lm ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi Could not import runpy module Traceback (most recent call last): File "/home/jeremyr/Python-3.7.3/Lib/runpy.py", line 15, in <module> import importlib.util File "/home/jeremyr/Python-3.7.3/Lib/importlib/util.py", line 14, in <module> from contextlib import contextmanager File "/home/jeremyr/Python-3.7.3/Lib/contextlib.py", line 4, in <module> import _collections_abc SystemError: <built-in function compile> returned NULL without setting an error generate-posix-vars failed Makefile:603: recipe for target 'pybuilddir.txt' failed make[1]: *** [pybuilddir.txt] Error 1 make[1]: Leaving directory '/home/jeremyr/Python-3.7.3' Makefile:531: recipe for target 'profile-opt' failed make: *** [profile-opt] Error 2 jeremyr#88:~/Python-3.7.3$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.11 (jessie) Release: 8.11 Codename: jessie jeremyr#88:~/Python-3.7.3$ gcc --version gcc (Debian 4.9.2-10+deb8u2) 4.9.2 Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. jeremyr#88:~/Python-3.7.3$ sudo apt upgrade gcc Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... gcc is already the newest version. jeremyr#b88:~/Python-3.7.3$ echo $PYTHONPATH Any advice on how to overcome this and install python3.7 appreciated. Edit - the solution listed below seems to work for various other python versions, so I changed title to python 3.x from 3.7
It seems the enable-optimizations was the problem, jeremyr#b88:~/Python-3.7.3$ ./configure jeremyr#b88:~/Python-3.7.3$ make clean takes care of it in my case.
In case others come across this question: I encountered the same problem on Centos 7. I also had --enable-optimizations but didn't want to remove that flag. Updating my build dependencies and then re-running solved the problem. To do that I ran: sudo yum groupinstall "Development Tools" -y In case the yum group is not available, you can also install the pacakges individually using: sudo yum install bison byacc cscope ctags cvs diffstat doxygen flex gcc gcc-c++ gcc-gfortran gettext git indent intltool libtool patch patchutils rcs redhat-rpm-config rpm-build subversion swig systemtap
For whomever MicGer's answer didn't work and would like to retain --enable-optimizations, check your gcc version. The error was solved for me on gcc 8.3.0.
For me, CentOS 7, compiling Python3.9, the fix was updating the gcc and related packages.
Removing --enable-optimizations didn't solved the issue here, neither upgrading GCC from gcc-7.5.0 to gcc-8.2.1 trying building Python 3.6.13 on opensuse-15.2. Here for some reason the build is picking a local python installation, so before calling make I have updated the PATH env to include the build directory (were python file is generated), for instance: export PATH=$(pwd):$PATH
Compiling vim with statically linked python support in a non-standard path configuration
I'm currently trying to compile vim on a Fedora 20 machine with a particularly exotic setup:
No root access
python2 and python3 manually compiled and installed in ~/.local correctly working (after exporting the PATH and LD_LIBRARY_PATH).
zsh as shell
gcc version 4.8.3 20140911 (Red Hat 4.8.3-7) (GCC)
If I configure the compiling process as:
./configure --with-features=huge --enable-pythoninterp --enable-python3interp --prefix=$HOME/.local
and then make && make install, vim is correctly compiled with +python/dyn +python3/dyn.
vim --version | grep python
+cryptv +linebreak +python/dyn +vreplace
+cscope +lispindent +python3/dyn +wildignore
but, inside vim, :echo has('python') returns a 0 (and the MatchTagAlways complains about that in facts...).
So I told to myself, let's try to force the statically linked installation:
export LDFLAGS=-static
./configure --with-features=huge --enable-pythoninterp --enable-python3interp --prefix=$HOME/.local
ends just a little bit after the configuring command:
configure: creating cache auto/config.cache
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/students/rm_16_17/dibattista/build/vim/src':
configure: error: C compiler cannot create executables
See `config.log' for more details
Here the full configure.log. The relevant line should be:
configure:3027: gcc -static conftest.c >&5
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
that I cannot really decrypt. It seems that gcc does not has the -static flag. Is that the issue?
At the end what I was missing was a flag when compiling Python to compile it as a shared library --enable-shared. As a reminder: always read all the INSTALL instructions :)
cross compiling gdb for arm with python failed
I want to debug ARM application on devices like Android machine, i prefer to use gdb(ARM version) than gdb with gdbserver to debug, because there is a dashboard , a visual interface for GDB in Python. It must cooperation with gdb(ARM version) on devices,so i need to cross compiling a ARM version of gdb with python, the command used shows below: ./configure --host=arm-linux-gnueabi --target=arm-linux-gnueabi --with-python=/usr/bin But finally a error message appeared: configure:8096: checking whether to use python configure:8098: result: /usr/bin/ configure:8316: checking for python2.7 configure:8334: arm-linux-gnueabi-gcc -o conftest -g -O2 -I/usr/include/python2.7 -I/usr/include/python2.7 conftest.c -ldl -ltermcap -lm -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions >&5 In file included from /usr/include/python2.7/Python.h:8:0, from conftest.c:50: /usr/include/python2.7/pyconfig.h:15:52: fatal error: arm-linux-gnueabi/python2.7/pyconfig.h: No such file or directory compilation terminated. Then I find the line 15 in /usr/include/python2.7/pyconfig.h, as below: # elif defined(__ARM_EABI__) && !defined(__ARM_PCS_VFP) #include <arm-linux-gnueabi/python2.7/pyconfig.h> Here is the point, I only have x86_64-linux-gnu/python2.7/pyconfig.h in /usr/include, how can I get the arm-linux-gnueabi/python2.7/pyconfig.h? I already apt-get install python2.7-dev.
I ran into this problem when attempting to cross-compile a python wrapper module built with SWIG, but it looks to me like it will happen to anyone cross compiling python-linked C code on a Debian system. Apparently the Debian python-dev packages are not set up with the header files to facilitate a cross compile, but it is possible to go get them manually. I'm not sure if this is a python bug or a Debian package bug, and I haven't researched if it applies to other distros. pyconfig.h sets preprocessor defines to tell the python source code about platform-depended things like endianness and data type sizes, so the proper pyconfig.h is definitely needed to correctly compile python source. Fortunately, the pyconfig.h file should be the only file you need to grab separately, and it is available from the Debian python-dev package for your target platform. you can download the package file for armeabi or any other architecture from https://packages.debian.org/jessie/libpython2.7-dev and extract the include directory yourself, or you can use the following commands to download the package and copy the proper files for armeabi to /usr/local/include wget http://security.debian.org/debian-security/pool/updates/main/p/python2.7/libpython2.7-dev_2.7.9-2+deb8u2_armel.deb dpkg -x libpython2.7-dev_2.7.9-2_armel.deb libpython2.7-dev_2.7.9-2_armel_extracted sudo cp -r libpython2.7-dev_2.7.9-2_armel_extracted/usr/include/arm-linux-gnueabi/ /usr/local/include/ rm -r libpython2.7-dev_2.7.9-2_armel* Note that on some cross compilers you will have to add -I /usr/local/include to the compiler options if it does not search this location by default, but to me this is better than modifying /usr/include and risking your changes being wiped out by the OS
Building Python with SSL support in non-standard location
I need to install several Python modules on a RHEL where I don't have root access. At least one of the modules also needs access to Python.h.
In this case I find that the best thing is to install python and it dependencies in ~/local. It usually just works, but this time Python fails to build the SSL module (see details below). Here's the trace of what I'm doing.
So I downloaded python 6 source and off I went:
./configure --prefix=/home/fds/rms/local
make >& make.log
An inspection to log reveals that ssl module has not been compiled, but there is no mention of the cause (no other occurence of ssl in make or configure):
Failed to find the necessary bits to build these modules:
_bsddb _curses _curses_panel
_hashlib _sqlite3 _ssl <----------
So I figure, python is not finding any ssl library at all (which is strange, but hey...). So I download openssl-0.9.8r and
./config --prefix=/home/fds/rms/local shared
make
make install
Now back to Python, I ./configure and make again. It fails, but this time it's different:
Failed to build these modules:
_hashlib _ssl
A closer inspection to the log file reveals this:
gcc -pthread -shared build/temp.linux-x86_64-2.6/home/fds/rms/installers/Python-2.6.6/Modules/_ssl.o -L/home/fds/rms/local/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-2.6/_ssl.so
*** WARNING: renaming "_ssl" since importing it failed: libssl.so.0.9.8: cannot open shared object file: No such file or directory
So now it's picking up the library but not quite getting it right (the file is there where is should be):
$ find /home/fds/rms/local -iname libssl.so.0.9.8
/home/fds/rms/local/lib/libssl.so.0.9.8
Next thing is tracing make and see where is it looking for the file:
$ strace -f make 2>&1 | grep libssl.so.0.9.8
[pid 5584] open("/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/lib64/tls/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/lib64/tls/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/lib64/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib64/tls/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib64/tls/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib64/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] write(1, "*** WARNING: renaming \"_ssl\" sin"..., 131*** WARNING: renaming "_ssl" since importing it failed: libssl.so.0.9.8: cannot open shared object file: No such file or directory
[pid 5584] open("/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib64/tls/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] open("/usr/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 5584] write(1, "*** WARNING: renaming \"_hashlib\""..., 135*** WARNING: renaming "_hashlib" since importing it failed: libssl.so.0.9.8: cannot open shared object file: No such file or directory
Mhhh, it's looking in all the wrong places. I try to give a hint:
CPPFLAGS="-I/home/fds/rms/local/include -I/home/fds/rms/local/include/openssl" LDFLAGS="-L/home/fds/rms/local/lib" ./configure --prefix=/home/fds/rms/local
But nothing changes, and make does not seem to try /home/fds/rms/local/lib at all.
I haven't done this in years, so maybe I'm overlooking something. Can anyone help with the problem?
You need to edit Modules/Setup.dist to specify the location of OpenSSL if it is not in the standard location. From Getting SSL Support in Python 2.5.1: If you find yourself on a linux box needing ssl support in python (to use a client in things like httplib.HTTPSConnection or imaplib.IMAP4_SSL), then let me save you a couple of hours of hunting around the web (of course if you have found this then that means you've done some level hunting already!). You'll know if you need ssl support compiled into your python installation if you get the following exception message: AttributeError: 'module' object has no attribute 'ssl' In order to make that go away so you can continue happily slinging python code, you'll need to first make sure you have OpenSSL installed. By default it is installed from source at: /usr/local/ssl If that directory doesn't exist, then grab the source package. Do the standard: tar zxf openssl-0.9.8g.tar.gz cd openssl-0.9.8g ./config make make install Then grab the python sources for 2.5.1 and: tar zxf Python-2.5.1.tgz && cd Python-2.5.1 Then you need to edit the Modules/Setup.dist: 204:# Socket module helper for SSL support; you must comment out the other 205:# socket line above, and possibly edit the SSL variable: 206:SSL=/usr/local/ssl 207:_ssl _ssl.c \ 208: -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ 209: -L$(SSL)/lib -lssl -lcrypto If you installed OpenSSL in the default locations you can just uncomment lines 206-209, then: ./configure make make install Then verify your installation with: python /usr/local/lib/python2.5/test/test_socket_ssl.py test_rude_shutdown ... test_basic ... test_timeout ... Make sure the changes to Modules/Setup.dist get picked up by cleaning the source root (e.g. make distclean) and run configure and make again.
On Linux Red Hat 7.7 x86_64, the following worked to install openssl-1.1.1d and Python-3.8.1 in my home directory (/home/unix/vangalen):
Install OpenSSL
source1 source2
cd ~
wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz
tar -xzf openssl-1.1.1d.tar.gz
cd /home/unix/vangalen/openssl-1.1.1d
./config --prefix=/home/unix/vangalen/openssl --openssldir=/home/unix/vangalen/openssl
make
make test
make install
Install Python
source2 source3 source4
cd ~
wget https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz
tar xf Python-3.8.1.tgz
Modify Python-3.8.1/Modules/Setup in a text editor. If this file doesn't exist you may need to go through a failed run first. Uncomment lines and adjust the alias for SSL in lines 206 to 213::
_socket socketmodule.c
# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
SSL=/home/unix/vangalen/openssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
cd ~/Python-3.8.1
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/unix/vangalen/openssl/lib
./configure --prefix=/home/unix/vangalen/py-381 --with-openssl=/home/unix/vangalen/openssl
make
make test
make install
#PeterVanGalen has a good method, and is relevant as of 2021, but I disagree with some details.
His method of modifying the Modules/Setup file results in the libssl.so and libcrypto.so libraries becoming dynamic dependencies of the python binary itself. This is not as intended -- those are supposed to be dependencies of python importable .so's, like _ssl.cpython-39-x86_64-linux-gnu.so.
His method also doesn't include a solution for how libssl.so and libcrypto.so will be found at runtime, rather than build time. This matters if they are in some unusual, custom path, or else python will either not run at all or be unable to import ssl. You could solve it in lots of ways (LD_LIBRARY_PATH and ld.so.conf come to mind) but I opted to use rpath, so that the .so files can always be found in their custom location when I use this python, but will otherwise stay out of the way.
This is my modification of #PeterValGalen's approach:
# First we need openssl installed in a custom location...
tar zxf openssl-1.1.1j.tar.gz
cd openssl-1.1.1j
./config --prefix=/my/weird/path --openssldir=/my/weird/path/ssl
make
make install # Add `sudo` if needed for permission to /my/weird/path.
cd ..
# Now the python part...
wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz
tar xf Python-3.9.2.tgz
cd Python-3.9.2
LDFLAGS="${LDFLAGS} -Wl,-rpath=/my/weird/path/lib" ./configure --with-openssl=/my/weird/path
make
make install # Add `sudo` if needed for permissions.
When done this way, the openssl libs are not dependencies of the python binary, but are dependencies of the _ssl*.so
$ ldd ./python | grep weird
# [Nothing]
$ ldd build/lib.linux-x86_64-3.9/_ssl.cpython-39-x86_64-linux-gnu.so | grep weird
libssl.so.1.1 => /my/weird/path/lib/libssl.so.1.1 (0x00007f733ee73000)
libcrypto.so.1.1 => /my/weird/path/lib/libcrypto.so.1.1 (0x00007f733e9ab000)
And it works:
$ ./python -c "import ssl; print('yay')"
yay
$ ./python Lib/test/test_ssl.py
# ... lots of good stuff...
OK (skipped=10)
in the Bourne shell (/bin/sh or /bin/bash): $ LD_LIBRARY_PATH=/usr/local/lib $ export LD_LIBRARY_PATH $ make in the C-shell (/bin/csh or /bin/tcsh): % setenv LD_LIBRARY_PATH /usr/local/lib % make
For me editing Modules/Setup was not enough as _hashlib module still ended up using the wrong OpenSSL version; and LD_LIBRARY_PATH was not taken into account at runtime on my SLES system. What worked was to statically link the local OpenSSL to both _ssl and _hashlib by editing setup.py as per GitHub patch: eddy-geek/ python_custom_openssl.diff, and then make clean && make. More details on why I used static links on Stack Overflow at Coredump when compiling python with a custom openssl version.
Here is the complete process I used with Python 2.7.11. In top level Python2.7.11 source dir: Change Modules/Setup.dist, Modules/Setup : Uncomment _ssl section, comment out _socket (no-op if it's already commented out), uncomment and set SSL appropriately (path to your new ssl lib/includes etc.) Note: Modules/Setup file does not exist initially but after first run it get's content from Modules/Setup.dist i believe. Make sure changes are reflected in here before every run. Apply patch: http://gist.github.com/eddy-geek/9604982 (make distclean if previously ran make) ./configure --prefix=/root/.local/aks/py-ssl/python2.7 --enable-shared # modify: Makefile -> set svnversion to "" make make altinstall
I was getting the same result until I got back to the logs for openssl. There I saw that you need to use -fPIC when building openssl: building '_ssl' extension: gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/ssl/include -I. -IInclude -I./Include -I/usr/local/include -I/home/feramos/Python-2.7.3/Include -I/home/feramos/Python-2.7.3 -c /home/feramos/Python-2.7.3/Modules/_ssl.c -o build/temp.linux-x86_64-2.7/home/feramos/Python-2.7.3/Modules/_ssl.o gcc -pthread -shared build/temp.linux-x86_64-2.7/home/feramos/Python-2.7.3/Modules/_ssl.o -L/usr/local/ssl/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-2.7/_ssl.so /usr/bin/ld: /usr/local/ssl/lib/libcrypto.a(x86_64cpuid.o): relocation R_X86_64_PC32 against `OPENSSL_cpuid_setup' can not be used when making a shared object; recompile with -fPIC openssl-0.9.8g]# .config -fPIC then, make, make install for openssl and then build Python again.
I have a set of a couple of patches for static openssl and static libintl for 2 and 3 below. For openssl (first patch) you have to set the env var OPENSSL_ROOT. This is based on the patch from http://gist.github.com/eddy-geek/9604982 . For Python 2.7.14: https://gist.github.com/rkitover/2d9e5baff1f1cc4f2618dee53083bd35 https://gist.github.com/rkitover/afab7ed3ac7ce1860c43a258571c8ae1 For Python 3.6.3: https://gist.github.com/rkitover/93d89a679705875c59275fb0a8f22b45 https://gist.github.com/rkitover/b18f19eafda3775a9652cc9cdf3ec914
I'm building Python2.7.13 and I see this same issue. For 2.7.13 you have to use "openssl1.0.0e" or above to make it work. I tried openssl-0.9.8g and it doesn't work. And somehow I can't make it work just modifying Modules/Setup.dist so I have to manually compile that _ssl.o. I guess this is because openssl-0.9.8g I provided is not working and it searched for system default libssl.so.10 which doesn't work either.
For MAC OS HIGH Sierra, and Python-3.5.6 In the above answer the openssl installation is done using source,but if you install using brew it tells where the installed package is, so if you install openssl using brew brew install openssl This will install openssl at /usr/local/Cellar/openssl/1.0.2o_2/ this path needs to be updated in Modules/Setup.dist at Follow this answer which is mentioned above where the location of openssl installation is not mentioned to update the Modules/Setup.dist In the above lines update SSL value to SSL=/usr/local/Cellar/openssl/1.0.2o_2/ and uncomment the lines, do a CMM and your python will get compiled with openssl.
OpenSSL
cd /opt
sudo wget https://www.openssl.org/source/openssl-1.1.1q.tar.gz --no-check-certificate
sudo mkdir /opt/openssl
sudo tar xfvz openssl-1.1.1q.tar.gz --directory /opt/openssl
export LD_LIBRARY_PATH=/opt/openssl/lib
cd /opt/openssl/openssl-1.1.1q/
sudo ./config --prefix=/opt/openssl --openssldir=/opt/openssl/ssl
sudo make
sudo make test
sudo make install
sudo ln -s /usr/local/bin/openssl /usr/bin/openssl
Python 3.9.13
cd /usr/src
sudo wget https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tgz
sudo tar xzf Python-3.9.13.tgz
cd /usr/src/Python-3.9.13
sudo CFLAGS="-I/opt/openssl/include/" LDFLAGS="${LDFLAGS} -Wl,-rpath=$LD_LIBRARY_PATH" ./configure --enable-optimizations --with-openssl=/opt/openssl
sudo make altinstall -j6
sudo ln -s /usr/local/bin/python3.9 /usr/bin/python3
pip3
sudo ln -s /usr/local/bin/pip3.9 /usr/bin/pip3
pip3 install --upgrade pip
Tested on Ubuntu 14.04.06
Try adding -Wl,-rpath,/home/fds/rms/local/lib to LDPATH.