cross compiling gdb for arm with python failed - python

I want to debug ARM application on devices like Android machine, i prefer to use gdb(ARM version) than gdb with gdbserver to debug, because there is a dashboard , a visual interface for GDB in Python.
It must cooperation with gdb(ARM version) on devices,so i need to cross compiling a ARM version of gdb with python, the command used shows below:
./configure --host=arm-linux-gnueabi --target=arm-linux-gnueabi --with-python=/usr/bin
But finally a error message appeared:
configure:8096: checking whether to use python
configure:8098: result: /usr/bin/
configure:8316: checking for python2.7
configure:8334: arm-linux-gnueabi-gcc -o conftest -g -O2 -I/usr/include/python2.7 -I/usr/include/python2.7 conftest.c -ldl -ltermcap -lm -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions >&5
In file included from /usr/include/python2.7/Python.h:8:0,
from conftest.c:50:
/usr/include/python2.7/pyconfig.h:15:52: fatal error: arm-linux-gnueabi/python2.7/pyconfig.h: No such file or directory
compilation terminated.
Then I find the line 15 in /usr/include/python2.7/pyconfig.h, as below:
# elif defined(__ARM_EABI__) && !defined(__ARM_PCS_VFP)
#include <arm-linux-gnueabi/python2.7/pyconfig.h>
Here is the point, I only have x86_64-linux-gnu/python2.7/pyconfig.h in /usr/include, how can I get the arm-linux-gnueabi/python2.7/pyconfig.h? I already apt-get install python2.7-dev.

I ran into this problem when attempting to cross-compile a python wrapper module built with SWIG, but it looks to me like it will happen to anyone cross compiling python-linked C code on a Debian system.
Apparently the Debian python-dev packages are not set up with the header files to facilitate a cross compile, but it is possible to go get them manually. I'm not sure if this is a python bug or a Debian package bug, and I haven't researched if it applies to other distros.
pyconfig.h sets preprocessor defines to tell the python source code about platform-depended things like endianness and data type sizes, so the proper pyconfig.h is definitely needed to correctly compile python source. Fortunately, the pyconfig.h file should be the only file you need to grab separately, and it is available from the Debian python-dev package for your target platform.
you can download the package file for armeabi or any other architecture from https://packages.debian.org/jessie/libpython2.7-dev and extract the include directory yourself, or you can use the following commands to download the package and copy the proper files for armeabi to /usr/local/include
wget http://security.debian.org/debian-security/pool/updates/main/p/python2.7/libpython2.7-dev_2.7.9-2+deb8u2_armel.deb
dpkg -x libpython2.7-dev_2.7.9-2_armel.deb libpython2.7-dev_2.7.9-2_armel_extracted
sudo cp -r libpython2.7-dev_2.7.9-2_armel_extracted/usr/include/arm-linux-gnueabi/ /usr/local/include/
rm -r libpython2.7-dev_2.7.9-2_armel*
Note that on some cross compilers you will have to add -I /usr/local/include to the compiler options if it does not search this location by default, but to me this is better than modifying /usr/include and risking your changes being wiped out by the OS

Related

Python3, Boost-Python and Cpp linker errors

So I'm about to throw my laptop through the window, out the window, and go and burn Apple HQ.
See Updates Below:
I can't get python3, boost-python and clang to work with each other. The error I'm stuck at is running:
clang++ <FLAGS/INCLUDES> -o hello.so hello.cpp
Invokes the response:
Undefined symbols for architecture x86_64:
"__Py_NoneStruct", referenced from:
boost::python::api::object::object() in hello-0c512e.o
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [hello] Error 1
Any help would be greatly, greatly appreciated. I think I have included everything necessary. Let me know if you require extra information.
The setup:
OSX 10.11.6 (El Capi-s##%)
Must use Xcode 7.3 (and appropriate CLT): Requirement of NVIDIA for CUDA programming (installed).
Must use Python3 (Homebrew installed)
brew install python3
which python -> /usr/bin/python
/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
I set up an alias for python3.5 (see below).
Using Boost-Python (Homebrew installed)
brew install boost
brew install boost-python --with-python3 --without-python
/usr/local/Cellar/boost-python/1.62.0/
Using LLVM 3.9.0 (Homebrew installed)
brew install llvm --universal
Now a few helpful things that you may ask for:
Clang++:
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
FLAGS and INCLUDES in makefile:
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
LDHEADERS = -I/usr/local/opt/llvm/include
LDLIBS = -L/usr/local/opt/llvm/lib
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.62.0/include/boost
BOOSTLIBS = -L/usr/local/Cellar/boost-python/1.62.0/lib
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
PYTHONLIBS = -L/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib
And finally, the code I am trying to compile:
hello.cpp
#include <boost/python.hpp>
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
using namespace boost::python;
BOOST_PYTHON_MODULE(hello)
{
class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
;
};
After much heartache and pain, I HAVE THE ANSWER!
To all those using OSX and homebrew, here's how you do it.
brew install python3 Python3 has UCS4 native (Unicode) which is an essential part of this process. If you need Python2, then make sure it is configured for UCS4 as it is natively UCS2.
brew install boost Install general boost first.
brew install boost-python --with-python3 --without-python This installs boost-python for Python3 WITHOUT Python2. You can change these options if you need Python2.
brew install llvm --universal Make sure you have llvm installed, this should have clang++ included in it which is the compiler we will use (not the Xcode one).
Create a makefile (see below) that includes the directories for all your python and boost headers/libraries, AND includes the libraries you want to use. (This is what tripped me up, I had the directories but did not specify which library in that directory the compiler should use).
My makefile:
# compiler flags:
# -g adds debugging information to the executable file
# -Wall turns on most, but not all, compiler warnings
COMPILER = /usr/local/Cellar/llvm/3.9.0/bin/clang++
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
# Python and BoostPython links.
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.62.0/include/boost
BOOSTLIBRARIES = -L/usr/local/Cellar/boost-python/1.62.0/lib/
# This is the boost library we want to use, there are also libraries for multithreading etc.
# All we do is find the file libboost_python3.a and link it by taking away the 'lib' and '.a'.
BOOSTLIB = -lboost_python3
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
PYTHONLIBRARIES = -L/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib
# Link to python3.5 library, the same as we did for boost.
PYTHONLIB = -lpython3.5
# Collect links.
LIBRARIES = $(BOOSTLIBRARIES) $(PYTHONLIBRARIES) $(PYTHONLIB) $(BOOSTLIB)
HEADERS = $(BOOSTHEADERS) $(PYTHONHEADERS)
# Build target.
TARGET = hello
# BEGIN MAKE
all: $(TARGET)
$(TARGET): $(TARGET).cpp
# Note that '-shared' creates a library that is accessible.
$(COMPILER) -shared $(LIBRARIES) $(HEADERS) $(TARGET).cpp -o $(TARGET).so
clean:
$(RM) $(TARGET)
Then all you need to do is run your makefile with all the inclusions and everything should be sweet :) I hope this helps someone and removes the pain I had of trying to get insertProfanity boost working.
Not enough points to comment so unfortunately I'll have to post as an answer since not too long ago I was going through this same linker error stuff with Boost.
My guess is that this appears to be a Python linker problem, as opposed to a Boost linker problem. I see that you've added the Python includes to your CXX_INCLUDE_PATH, but what about your Python library?
If the path you've mentioned above (that long path in your CXX_INCLUDE_PATH variable) is where your includes are then [that_long_path]/Versions/3.5/lib should be where your Python libraries are. Include this library when you run your build command by using
clang++ -g -v -std=c++11 -stdlib=libc++ -L/[that_long_path]/Versions/3.5/lib -lpython3.5m hello.cpp -o hello.so
The -L flag tells the compiler to include that directory, while the -l flag tells the compiler to include the following library. Alternatively you can just append this lib path to your current CXX_INCLUDE PATH, which would then look like this:
export CXX_INCLUDE_PATH="/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m:/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib"
Also make sure you are including any relevant boost include paths and library paths when you run the build command to avoid any more linker errors (if boost is not installed in a default location like usr/local/[some_place].
Then run this new build command (may be able to do without the -std and -stdlib flags:
clang++ -g -v -std=c++11 -stdlib=lib++ -lpython3.5m -o hello.so hello.cpp
In summary:
1) Include your python header files (which you seem to have done)
2) Include your python library file(s) (which you seem to be missing)
3) Include any relevant boost libraries and the boost include directory (which is a bit out of the scope of this question but still worth noting)
Hope this helps.

Compiling vim with statically linked python support in a non-standard path configuration

I'm currently trying to compile vim on a Fedora 20 machine with a particularly exotic setup:
No root access
python2 and python3 manually compiled and installed in ~/.local correctly working (after exporting the PATH and LD_LIBRARY_PATH).
zsh as shell
gcc version 4.8.3 20140911 (Red Hat 4.8.3-7) (GCC)
If I configure the compiling process as:
./configure --with-features=huge --enable-pythoninterp --enable-python3interp --prefix=$HOME/.local
and then make && make install, vim is correctly compiled with +python/dyn +python3/dyn.
vim --version | grep python
+cryptv +linebreak +python/dyn +vreplace
+cscope +lispindent +python3/dyn +wildignore
but, inside vim, :echo has('python') returns a 0 (and the MatchTagAlways complains about that in facts...).
So I told to myself, let's try to force the statically linked installation:
export LDFLAGS=-static
./configure --with-features=huge --enable-pythoninterp --enable-python3interp --prefix=$HOME/.local
ends just a little bit after the configuring command:
configure: creating cache auto/config.cache
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/students/rm_16_17/dibattista/build/vim/src':
configure: error: C compiler cannot create executables
See `config.log' for more details
Here the full configure.log. The relevant line should be:
configure:3027: gcc -static conftest.c >&5
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
that I cannot really decrypt. It seems that gcc does not has the -static flag. Is that the issue?
At the end what I was missing was a flag when compiling Python to compile it as a shared library --enable-shared.
As a reminder: always read all the INSTALL instructions :)

Building graph-tool on OSX with Python 3.4

I tried to install graph-tool on Mac OSX 10.10 using homebrew. The brew build process works fine, but when I try to import graph-tool I get the error described in this question.
Another problem with homebrew is that I always builds graph-tool for python2.7 and it installs the packages in the Python 2.7 sit-packages folder. But I want to use it with Python 3.4. These are the reasons why I tried to build graph-tool from source.
The ./configure command automatically uses Python 2.7, too. So I passed it the desired Python version with ./configure PYTHON=python3.4
It then detects the correct version as well as the related paths but crash with the following error:
configure: error:
Could not link test program to Python. Maybe the
main Python library has been installed in some non-standard library
path. If so, pass it to configure, via the LDFLAGS environment
variable.
Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
======================================================================
ERROR!
You probably have to install the development version of the
Python package for your distribution. The exact name of this package varies
among them.
======================================================================
The error occurs with and without PYTHON variable set.
From the output of ./configure I can see that everything works fine except for the last line, which says:
checking consistency of all components of python development
environment... no
Whats does the above line mean and how do I properly install graph-tool on my maschine?
The error message is explaining exactly what needs to be done. Since python was installed in a non-standard path, you need to pass the flag LDFLAGS="-L/usr/non-standard-path/python/lib" pointing to the directory where the python libraries are located. This is most likely "/usr/local/lib", if you are using homebrew.
I was getting this error when I was trying to install graph-tool using outdated an autoconf / automake / pkg-config combination (installed using yum in CentOS 5.10). Installing those packages from source fixed the problem... although I'm not sure how this related to my python installation....
It worked for me by passing the variable PYTHON_EXTRA_LDFLAGS="-Wl,-stack_size,1000000 -F/usr/local/Cellar/python3/3.6.3/Frameworks -framework CoreFoundation".
In your case, it would be the path to the homebrew installation of python3.4.
The way I found out is that in the config.log, the error message shows the following:
configure:19023: checking python extra libraries
configure:19030: result: -ldl -framework CoreFoundation
configure:19037: checking python extra linking flags
configure:19044: result: -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python
configure:19051: checking consistency of all components of python development environment
configure:19079: gcc -o conftest -g -O2 -DNDEBUG -I/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/include/python3.6m -F/usr/local/Cellar/python3/3.6.3/Frameworks/ -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python conftest.c -L/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib -lpython3.6m -ldl -framework CoreFoundation -ldl -framework CoreFoundation >&5
clang: error: no such file or directory: 'Python.framework/Versions/3.6/Python'
The error seems to be path 'Python.framework/Versions/3.6/Python', that in a homebrew installation does not exist. I search for the same path in the config.log and I found this line:
PYTHON_EXTRA_LDFLAGS="-Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python"
So, the solution for me was to pass this variable with the right path.

Libpython error while building YouCompleteMe

I am building YouCompleteMe plugin of vim, following this document. When I run make I get the following error.
Linking CXX shared library /home/sagar/.vim/bundle/YouCompleteMe/python/ycm_core.so
/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
What is this error?
I have installed pyenv to manage python versions. Is it causing problem?
Make the linker point to the .so (shared object) file and not the .a (static lib) file.
You can do this specifying the flag when running cmake:
cmake -G "Unix Makefiles" -DPYTHON_LIBRARY=/usr/local/lib/libpython2.7.so . ~/.vim/bundle/YouCompleteme/cpp
Do mind that even though you're using pyenv, YouCompleteMe build may point to an undesired
python build as they are not correctly auto-detected right now.
If you're having this problem, you should probably also specify the Python header files correctly:
cmake -G "Unix Makefiles" -DPYTHON_LIBRARY=/usr/local/lib/libpython2.7.so -DPYTHON_INCLUDE_DIR=/usr/local/include/python . ~/.vim/bundle/YouCompleteme/cpp
PS=(I'm assuming your headers are in that path, do check before)
Since some paths were different on my system from the accepted answer (both the CMake and the python lib ones) I'm posting an alternate solution for the above problem:
Make sure to have a shared library version of libpython2.7.so
$ locate libpython
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1
Either create a symlink to it from where CMake expects it to be
sudo ln -s "/usr/lib/x86_64-linux-gnu/libpython2.7.so.1" "/usr/lib/libpython2.7.so"
or alternatively, as written in YCM's build script code, you could add additional CMake options to ensure the .so library is properly found
export EXTRA_CMAKE_ARGS="-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so.1"

Coredump when compiling python with a custom openssl version

When compiling python-3.4.0rc3 with a local openssl-1.0.1f shared install, make prints no error but then I get the following core dump on make install or make test:
Program terminated with signal 11, Segmentation fault.
(gdb) bt
#0 0x00007f131dd10510 in EVP_PKEY_CTX_dup () from /data2/soft/openssl/lib/libcrypto.so.1.0.0
#1 0x00007f131dd0284f in EVP_MD_CTX_copy_ex () from /data2/soft/openssl/lib/libcrypto.so.1.0.0
#2 0x00007f131e256ab5 in EVPnew (name_obj=0x7f131e46a500, digest=0x0, initial_ctx=0x7f131e459a40, cp=0x0, len=0) at /data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.c:410
#3 0x00007f131e25726e in EVP_new_md5 (self=<value optimized out>, args=<value optimized out>) at /data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.c:799
#4 0x00000000004c7eef in ?? ()
Here is the complete list of commands used
tar -axf Python-3.4.0rc3.tgz
cd Python-3*
# For lzma and a pip-compatible openssl
export CFLAGS='-I/data2/soft/openssl/include/openssl -I/data2/local/include/'
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/:$LD_LIBRARY_PATH"
# Ready !
./configure --prefix=/data2/soft/python3
make
make install
Notes:
OS is SUSE Linux Enterprise Server 11 (x86_64)
pointing python to a custom location for the lzma lib located in
openssl was built with ./config shared --openssldir=/data2/soft/openssl
openssl make test prints ALL TESTS SUCCESSFUL.
without the custom openssl *FLAGS, make install is successful and I get these results for make test:
71 tests OK.
tests failed:
test_cmd_line test_gdb test_smtpnet test_ssl
How can I fix this, or at least investigate what is going on ?
Edit 1--5:
The shared libs were generated correctly:
> ls /data2/soft/openssl/lib
drwxr-xr-x engines
-rw-r--r-- libcrypto.a
lrwxrwxrwx libcrypto.so -> libcrypto.so.1.0.0
-r-xr-xr-x libcrypto.so.1.0.0
-rw-r--r-- libssl.a
lrwxrwxrwx libssl.so -> libssl.so.1.0.0
-r-xr-xr-x libssl.so.1.0.0
drwxr-xr-x pkgconfig
So I changed this in Setup :
SSL=/data2/soft/openssl/
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
$(SSL)/lib/libssl.a $(SSL)/lib/libcrypto.a -ldl
And I changed back LDFLAGS/CFLAGS accordingly. But there still is a -lssl when I run make clean && make, because of the _hashopen module:
gcc -pthread -shared -L/data2/local/lib/ -L/data2/local/lib/ -L/data2/local/lib/ -I/data2/local/include/ build/temp.linux-x86_64-3.4/data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.o -L/data2/local/lib/ -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
I guess it is the one causing the cores, because they are still there... I tried adding similar stuff to Setup file, but there is no commented item for this one and creating it causes another more cryptic failure:
gcc -pthread -Xlinker -export-dynamic -o python Modules/python.o libpython3.4m.a -lpthread -ldl -lutil /data2/eoubrayrie/soft/openssl/lib/libssl.a /data2/eoubrayrie/soft/openssl/lib/libcrypto.a -ldl /data2/eoubrayrie/soft/openssl/lib/libssl.a /data2/eoubrayrie/soft/openssl/lib/libcrypto.a -ldl -lm
libpython3.4m.a(config.o):(.data+0x158): undefined reference to `PyInit__hashopenssl'
collect2: ld returned 1 exit status
Edit 6:
I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together
But I can get it linking dynamically to my own openssl through good old -I/-L:
ldd build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
libssl.so.1.0.0 => /data2/soft/openssl/lib/libssl.so.1.0.0 (0x00007f5605799000)
libcrypto.so.1.0.0 => /data2/soft/openssl/lib/libcrypto.so.1.0.0 (0x00007f56053bd000)
Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?
From To Syms Read Shared Object Library
0x00007ffff5465930 0x00007ffff5466e98 Yes /data2/soft/python3/Python-3.4.0rc3/build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
0x00007ffff5321220 0x00007ffff5351878 Yes /opt/python-2.6-64/lib/libssl.so.1.0.0
0x00007ffff50d3100 0x00007ffff519b118 Yes /opt/python-2.6-64/lib/libcrypto.so.1.0.0
env | grep -F 'python-2.6-64' -> shows nothing !
grep -RF 'python-2.6-64' /etc/ld.so.* -> idem
gcc -print-search-dirs | sed 's/:/\n/g' | grep python -> idem
find . -name '*.so*' | xargs ldd | grep ssl -> only gives me the good ones
Level 1 dependencies don't require any wrong ssl version either. This was checked with: find . -name '*.so*' | xargs ldd | awk '/\t+[[:alnum:].]+ => [[:alnum:]./]+ \(/ {print $3}' | sort | uniq | xargs ldd | grep ssl
strace ./python ./Tools/scripts/run_tests.py 2>&1 | grep python-2.6-64 -> shows nothing
So how does ld picks this wrong library if he cannot konow about it ?? It's not in any standard location (if it were in /lib I could understand...)
Solution:
Found how to statically link _hashlib thanks to this OpenOffice bug: though the -Wl,--exclude-libs=ALL" option didn't work either, it pointed me to the right lines in setup.py.
TL;DR Here is the patch to setup.py I applied.
And finally... it works !
#noloader I'm accepting your most complete answer as you help was invaluable, though the "exact" answer for anyone encountering this problem is to compile with the patch above.
How can I fix this, or at least investigate what is going on ?
...
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/
I run into these problems a lot too because I avoid the crippled versions of OpenSSL shipped by Debian, Ubuntu, Fedora, et al. For example, Ubuntu ships an OpenSSL that disables TLSv1.1 and TLS v1.2 (cf., Ubuntu 12.04 LTS: OpenSSL downlevel version and does not support TLS 1.2).
You're probably loading the wrong version of the OpenSSL library. If you can get the misbehaving program under the debugger, issue info shared to see which libcrypto and libssl you are actually loading.
ldd might help too. Run it on the Pyhton executable: ldd /data2/soft/python3/python. I can only say it "may" help because the OpenSSL's are binary compatible, so you might only see a dependency on, for example, libcrypto.so.1.0.0 (use otool -L on Mac OS X). Below I used an rpath to force linking against the libraries in /usr/local/ssl/lib/.
$ ldd my-test.exe
linux-vdso.so.1 => (0x00007fffd61ff000)
libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00007f151528e000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007f1514e74000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1514c42000)
...
As a fix, you might try adding an rpath:
LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/ -Wl,-rpath,/data2/soft/openssl/lib'
A better fix would be to link to the static version of the OpenSSL library to avoid these problems all together. I believe you can do that with: -Bstatic -lssl -lcrypto -Bdynamic -ldl.
Personally, I don't even use -Bstatic because of different one-off problems. I open the Makefiles, remove all instances of -lssl -lcrypto, and add a full path to the archive to remove all ambiguity. For example, /data2/soft/openssl/lib/libssl.a and /data2/soft/openssl/lib/libcrypto.a.
Note well: an rpath is not honored on Mac OS X. More extreme measures are needed for Mac OS X because the linker does not honor -Bstatic either. You'll have to use the full path trick.
openssl was built with ./config shared --openssldir=/data2/soft/openssl
Another thing... On Fedora, its not enough to specify shared. You need to add the following, too:
export CFLAGS="-fPIC"
Otherwise, the shared object is not built. If the 1.0.1f shared object is not built, you're probably getting a downlevel version supplied by the distro. You can check what was built before install with:
./config shared --openssldir=/data2/soft/openssl
make all
# Verify artifacts
find . -iname -libcrypto.*
find . -iname -libssl.*
# Proceed if OK
sudo make install
Finally, make sure that all of Python's dependencies are also using your version of OpenSSL and not the system's version of OpenSSL.
I recently suffered that problem when my program used my OpenSSL; but my program also used libevent and libevent used the system's version of OpenSSL. I fixed it by rebuilding libevent and forcing it to statically link to my version of OpenSSL.
How should I go about changing it ? Will adding a _ssl.c: gcc ... line anywhere override the default behaviour ? My Makefile skills are rusted and this beast is 1600+ lines long !!
Here are the files you want to look at:
$ cd Python-3.4.0rc3
$ grep -R -- '-lcrypto' *
Modules/Setup.dist:# -L$(SSL)/lib -lssl -lcrypto
$ grep -R -- '-lssl' *
Modules/Setup.dist:# -L$(SSL)/lib -lssl -lcrypto
Here are the lines of interest in Setup.dist:
# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
#_ssl _ssl.c \
# -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
# -L$(SSL)/lib -lssl -lcrypto
Uncomment the lines and change Setup.dist to the following. I keep my OpenSSL in /usr/local/ssl, so that's how mine is setup below (you should use /data2/soft/openssl/lib/...):
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
/usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Use the full path to the archive, and don't use -l. Be sure to add -ldl because OpenSSL needs it in this configuration.
Once you change Setup.dist, re-run ./configure to effect the changes.
After changing the above line to, here's what I look like after a configure:
$ grep -R "libssl.a" *
Makefile:LOCALMODLIBS= /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Makefile:Modules/_ssl$(SO): Modules/_ssl.o; $(BLDSHARED) Modules/_ssl.o /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl -o Modules/_ssl$(SO)
Modules/Setup: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Modules/Setup.dist: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
You can test the static linking by using ldd or otool -L. You will not see an OpenSSL dependency. After make'ing, here's what I got:
$ find . -iname python
./python
$ ldd ./python
linux-vdso.so.1 => (0x00007fff67709000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3aed8e1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3aed6dd000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3aed4d9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3aed257000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3aececc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3aedb14000)
No libssl or libcrypto dependencies to go wrong :) And make test ran fine (actually, one failed test due to a Python bug: Issue 20896):
======================================================================
ERROR: test_get_server_certificate (test.test_ssl.NetworkedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1373, in test_get_server_certificate
_test_get_server_certificate('svn.python.org', 443, SVN_PYTHON_ORG_ROOT_CERT)
File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1354, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port))
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 902, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 344, in wrap_socket
_context=self)
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 540, in __init__
self.do_handshake()
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 767, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:598)
----------------------------------------------------------------------
Ran 96 tests in 8.610s
FAILED (errors=1, skipped=3)
test test_ssl failed
make: *** [test] Error 1
This isn't an answer, just an observation after running though this with you:
$ make
...
gcc -pthread -c -Wno-unused-result -Werror=declaration-after-statement
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include
-DPy_BUILD_CORE -o Objects/capsule.o Objects/capsule.c
-fwrapv is really bad. Its used to make illegal programs work. Its better to fix the broken program and drop the -fwrapv. See Ian Lance Taylor's blog on Signed Overflow.
I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together
grep is your friend ;)
$ grep -R _hashlib * | grep ssl
Lib/hashlib.py: f = getattr(_hashlib, 'openssl_' + name)
Lib/hashlib.py: _hashlib.openssl_md_meth_names)
Lib/test/test_hashlib.py: self.assertTrue(hasattr(_hashlib, 'openssl_md5'))
Lib/test/test_hashlib.py: self.assertTrue(hasattr(_hashlib, 'openssl_sha1'))
Lib/test/test_hashlib.py: constructor = getattr(_hashlib, 'openssl_'+algorithm, None)
Lib/test/ssltests.py:TESTS = ['test_asyncio', 'test_ftplib', 'test_hashlib', 'test_httplib',
Lib/test/time_hashlib.py: print(" '_hashlib' 'openssl_hName' 'fast' tests the builtin _hashlib")
Modules/_hashopenssl.c: "_hashlib.HASH", /*tp_name*/
Modules/_hashopenssl.c:static struct PyModuleDef _hashlibmodule = {
Modules/_hashopenssl.c: "_hashlib",
Modules/_hashopenssl.c:PyInit__hashlib(void)
Modules/_hashopenssl.c: m = PyModule_Create(&_hashlibmodule);
PCbuild/build_ssl.py:# Script for building the _ssl and _hashlib modules for Windows.
PCbuild/build_ssl.py:# for the actual _ssl.pyd and _hashlib.pyd DLLs.
PCbuild/build_ssl.py:# it should configure and build SSL, then build the _ssl and _hashlib
setup.py: exts.append( Extension('_hashlib', ['_hashopenssl.c'],
setup.py: print("warning: openssl 0x%08x is too old for _hashlib" %
Tools/ssl/test_multiple_versions.py: "test_asyncio", "test_ftplib", "test_hashlib", "test_httplib",
Tools/ssl/test_multiple_versions.py:MINIMAL_TESTS = ["test_ssl", "test_hashlib"]
Another partial answer...
But I can get it linking dynamically to my own openssl through good old -I/-L:
...
Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?
That's your good old friends -l and -L. Don't use them because they do this sort of thing all the time (take it from a guy who has suffered it in the past). Instead, specify the full path to libssl and libcrypto. E.g., use /data2/soft/openssl/lib/libssl.a.
We had a similar problem. We are using apache httpd + mod_wsgi + python + django and our c++ module for the apache httpd which also uses openssl. Now everything is loaded within one httpd process, correct version of openssl shared lib was loaded (1.0.0l) with our c++ module. But as soon as we access the web, python loads the hashlib and exactly the same problem appears - segfault in openssl called from python.
Normally python compiles with whatever openssl is available at the default location and there is no way how to specify it without fiddling with setup.py or makefiles. Python developers should add configure setting --with_ssl=path.
We installed new openssl libs and rebuild python and other binaries but with no success. We mapped default libssl.so and libcrypto.so to the new openssl binaries with no success. Finally after reading this thread I realized that probably wrong headers are being used while compiling python. And that was the problem. Follow the step to workaround the problem:
there must not be incorrect version of openssl headers in /usr/include and default locations /usr/local/ssl, /usr/contrib/ssl (if you installed openssl-devel then uninstall it, or simply erase/rename the directory)
yum remove openssl-devel
make symbolic link to your openssl installation from /usr/local/ssl
ln -s /opt/openssl-1.0.1l /usr/local/ssl
ensure that the new openssl libs are accessible from /usr/lib (make symbolic links if it is not installed here)
ln -s /opt/openssl-1.0.1l/lib/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.1.0.0
...
now configure and clean build the python
I am able to run python 2.7.11 with non-default SSL after patching fix mentioned here https://gist.github.com/eddy-geek/9604982
However, with this it's not building _socket module which is required by many other modules. for example, easy_install / pip started failing with error Importerr: no module named _socket
In Module/Setup.dist, am i suppose to uncomment or comment the line
_socket socketmodule.o
?
I see socketmodule.o and timemodule.o getting generated. but not _socket.so
Am i missing something ?

Categories