I am trying to install ViZDoom on Ubuntu system. I am following the steps given the following link https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md.
As shown below:
sudo apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \
nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \
libopenal-dev timidity libwildmidi-dev unzip
sudo apt-get install libboost-all-dev
sudo apt-get install python3-dev python3-pip
pip3 install numpy
sudo apt-get install julia
julia
julia> using Pkg
julia> Pkg.add("CxxWrap")
pip install git+https://github.com/mwydmuch/ViZDoom.git
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON=ON -DBUILD_JAVA=ON -DBUILD_LUA=ON -DBUILD_JULIA=ON
make
However, when I do this
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON3=ON -DBUILD_JAVA=ON -DBUILD_LUA=ON -DBUILD_JULIA=ON
I am getting the following error:
-- Boost version: 1.65.1
-- Found the following Boost libraries:
-- filesystem
-- thread
-- system
-- date_time
-- chrono
-- regex
-- iostreams
-- atomic
CMake Deprecation Warning at cmake_modules/CreateLaunchers.cmake:48 (cmake_policy):
The OLD behavior for policy CMP0026 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
src/vizdoom/CMakeLists.txt:13 (include)
-- Could NOT find GME (missing: GME_INCLUDE_DIR)
-- Using system zlib
-- Using system jpeg library
-- Using system bzip2 library
-- Using internal gme library
-- /usr/include
-- /usr/include
-- /home/tie/ViZDoom/src/vizdoom/lzma/C
-- Could not find FMOD include files
-- Could NOT find SndFile (missing: SNDFILE_LIBRARY SNDFILE_INCLUDE_DIR)
-- Could NOT find MPG123 (missing: MPG123_LIBRARIES MPG123_INCLUDE_DIR)
-- Selected assembler: /usr/bin/as
-- Boost version: 1.65.1
-- Found the following Boost libraries:
-- thread
-- system
-- date_time
-- chrono
-- atomic
-- Fluid synth libs: /usr/lib/x86_64-linux-gnu/libfluidsynth.so
-- pybind11 v2.8.1
CMake Error at src/lib_julia/CMakeLists.txt:1 (find_package):
By not providing "FindJlCxx.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "JlCxx", but
CMake did not find one.
Could not find a package configuration file provided by "JlCxx" with any of
the following names:
JlCxxConfig.cmake
jlcxx-config.cmake
Add the installation prefix of "JlCxx" to CMAKE_PREFIX_PATH or set
"JlCxx_DIR" to a directory containing one of the above files. If "JlCxx"
provides a separate development package or SDK, be sure it has been
installed.
-- Configuring incomplete, errors occurred!
See also "/home/tie/ViZDoom/build/CMakeFiles/CMakeOutput.log".
See also "/home/tie/ViZDoom/build/CMakeFiles/CMakeError.log".
I also install Julia successfully：
I think that there is CMake error or some version mismatch error but being new to it, I am unable to resolve it.
I'm trying to install openCV in python
I have the same problem as mentioned in those several github and stackoverflow pages but I'm still stuck:
github openCV issue #8382
github openCV issue #6027
github openCV issue #6066
Here is an ouput of the terminal:
mymacs-MacBook-Pro:build mymac$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
> -D CMAKE_INSTALL_PREFIX=/usr/local \
> -D INSTALL_C_EXAMPLES=OFF \
> -D INSTALL_PYTHON_EXAMPLES=ON \
> -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
> -D BUILD_EXAMPLES=ON ..
CMake Deprecation Warning at CMakeLists.txt:72 (cmake_policy):
The OLD behavior for policy CMP0022 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
CMake Deprecation Warning at CMakeLists.txt:77 (cmake_policy):
The OLD behavior for policy CMP0026 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
CMake Deprecation Warning at CMakeLists.txt:82 (cmake_policy):
The OLD behavior for policy CMP0042 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- Could NOT find GDAL (missing: GDAL_LIBRARY GDAL_INCLUDE_DIR)
-- Checking for module 'gstreamer-base-1.0'
-- No package 'gstreamer-base-1.0' found
-- Checking for module 'gstreamer-video-1.0'
-- No package 'gstreamer-video-1.0' found
-- Checking for module 'gstreamer-app-1.0'
-- No package 'gstreamer-app-1.0' found
-- Checking for module 'gstreamer-riff-1.0'
-- No package 'gstreamer-riff-1.0' found
-- Checking for module 'gstreamer-pbutils-1.0'
-- No package 'gstreamer-pbutils-1.0' found
-- Checking for module 'gstreamer-base-0.10'
-- No package 'gstreamer-base-0.10' found
-- Checking for module 'gstreamer-video-0.10'
-- No package 'gstreamer-video-0.10' found
-- Checking for module 'gstreamer-app-0.10'
-- No package 'gstreamer-app-0.10' found
-- Checking for module 'gstreamer-riff-0.10'
-- No package 'gstreamer-riff-0.10' found
-- Checking for module 'gstreamer-pbutils-0.10'
-- No package 'gstreamer-pbutils-0.10' found
-- Checking for module 'libv4l1'
-- No package 'libv4l1' found
-- Checking for module 'libv4l2'
-- No package 'libv4l2' found
-- Looking for linux/videodev.h
-- Looking for linux/videodev.h - not found
-- Looking for linux/videodev2.h
-- Looking for linux/videodev2.h - not found
-- Looking for sys/videoio.h
-- Looking for sys/videoio.h - not found
-- Checking for module 'libgphoto2'
-- No package 'libgphoto2' found
-- Found PythonInterp: /usr/local/bin/python2.7 (found suitable version "2.7.14", minimum required is "2.7")
-- Found PythonLibs: /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/bin (found suitable exact version "2.7.14")
-- Found PythonInterp: /Users/mymac/.virtualenvs/cv/bin/python (found suitable version "3.5.4", minimum required is "3.4")
-- Found PythonLibs: /usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/config-3.5m/libpython3.5.dylib (Required is exact version "3.5.4")
-- Could NOT find Matlab (missing: MATLAB_MEX_SCRIPT MATLAB_INCLUDE_DIRS MATLAB_ROOT_DIR MATLAB_LIBRARIES MATLAB_LIBRARY_DIRS MATLAB_MEXEXT MATLAB_ARCH MATLAB_BIN)
-- VTK is not found. Please set -DVTK_DIR in CMake to VTK build directory, or to VTK install subdirectory with VTKConfig.cmake file
-- Caffe: NO
-- Protobuf: NO
-- Glog: NO
CMake Error at /Users/mymac/opencv_contrib/modules/datasets/CMakeLists.txt:7 (ocv_append_source_files_cxx_compiler_options):
Unknown CMake command "ocv_append_source_files_cxx_compiler_options".
-- Configuring incomplete, errors occurred!
See also "/Users/mymac/opencv/build/CMakeFiles/CMakeOutput.log".
See also "/Users/mymac/opencv/build/CMakeFiles/CMakeError.log".
mymacs-MacBook-Pro:build mymac$
As mentioned on this here I tried checkout the same version for opencv_contrib that I did for opencv by running the following command:
cd ~
cd opencv
git checkout 3.0.0
cd ~
cd opencv_contrib
git checkout 3.0.0
It worked
I then ran the command make -j4
but the two following error messages appeared:
In file included from /Users/mymac/opencv_contrib/modules/cvv/src/gui/main_call_window.cpp:1:
In file included from /Users/mymac/opencv_contrib/modules/cvv/src/gui/main_call_window.hpp:9:
In file included from /Users/mymac/opencv_contrib/modules/cvv/src/gui/overview_panel.hpp:12:/Users/mymac/opencv_contrib/modules/cvv/src/gui/../stfl/stfl_engine.hpp:1056:30: error: use 'template' keyword to treat
'value' as a dependent template name
return settings.value(key).value<QStringList>();
^
template
In file included from /Users/mymac/opencv_contrib/modules/cvv/src/controller/view_controller.cpp:12:
In file included from /Users/mymac/opencv_contrib/modules/cvv/src/controller/../gui/overview_panel.hpp:12: /Users/mymac/opencv_contrib/modules/cvv/src/controller/../gui/../stfl/stfl_engine.hpp:1056:30: error: use 'template'
keyword to treat 'value' as a dependent template name
return settings.value(key).value<QStringList>();
^
template
For this i created a new question here
I am following this tutorial to install OpenCV 3.0 and Python 2.7 but however, I am getting Cmake errors at step 8.
-- Could NOT find PythonLibs: Found unsuitable version "2.7.11", but required is exact version "2.7.13" (found
/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin)
-- Could NOT find PythonInterp: Found unsuitable version "2.7.13", but required is at least "3.4" (found
/Users/SabrinaZuraimi/.virtualenvs/cv/bin/python)
-- Could NOT find PythonInterp: Found unsuitable version "2.7.13", but required is at least "3.2" (found
/Users/SabrinaZuraimi/.virtualenvs/cv/bin/python)
There is 2 versions of Python(2.7.11 and 2.7.13) in the usr/local/Cellar/Python but I don't understand why there is an error saying that i can't find 2.7.13
The command that I typed into the terminal was
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local \
-D PYTHON2_PACKAGES_PATH=~/.virtualenvs/cv/lib/python2.7/site-packages \
-D PYTHON2_LIBRARY=/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin \
-D PYTHON2_INCLUDE_DIR=/usr/local/Frameworks/Python.framework/Headers \
-D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON \
-D BUILD_EXAMPLES=ON \
-D INSTALL_C_EXAMPLES=OFF\
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules ..
I am not very familiar with cmake, and I appreciate any help that I can get.
that tutorial is out-of-date. you can try this new tutorial
To build OpenCV, I ran this in the terminal in my ~/opencv/build directory :
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/path/to/opencv-3.0.0/build -D PYTHON2_LIBRARY=/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/bin
-D PYTHON2_INCLUDE_DIR=/usr/local/Frameworks/Python.framework/Headers -D PYTHON2_PACKAGES_PATH=/usr/local/lib/python2.7/site-packages -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=/path/to/opencv_contrib-3.0.0/modules ../
But no matter what happens, I see this error in the Traceback : (Could NOT find PythonInterp:)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named numpy.distutils
-- Could NOT find PythonInterp: Found unsuitable version "2.7.11",
but required is at least "3.4" (found /usr/local/bin/python)
-- Could NOT find PythonInterp: Found unsuitable version "2.7.11",
but required is at least "3.2" (found /usr/local/bin/python)
Further down the line, this shows up
--
-- Python 2:
-- Interpreter: /usr/local/bin/python2.7 (ver 2.7.11)
--
-- Python 3:
-- Interpreter: NO
--
-- Python (for build): /usr/local/bin/python2.7
--
-- Java:
-- ant: NO
-- JNI:
The correct output should look somewhat like this:
correct Python Interp output
I have tried everything, but this keeps coming up!
In essence, I cannot build OpenCV properly, and hence cannot install it properly.
Your error talks about not being able to find numpy, so install numpy:
pip install numpy
In addition, change your include directory to make sure you are actually including it. For the PYTHON2_INCLUDE_DIR:
-D PYTHON2_INCLUDE_DIR=/usr/local/Frameworks/Python.framework/Headers
Then, run make clean, and delete your /build directory, and run your cmake, then make again.
You can specify the path to each version of Python manually.
When compiling python-3.4.0rc3 with a local openssl-1.0.1f shared install, make prints no error but then I get the following core dump on make install or make test:
Program terminated with signal 11, Segmentation fault.
(gdb) bt
#0 0x00007f131dd10510 in EVP_PKEY_CTX_dup () from /data2/soft/openssl/lib/libcrypto.so.1.0.0
#1 0x00007f131dd0284f in EVP_MD_CTX_copy_ex () from /data2/soft/openssl/lib/libcrypto.so.1.0.0
#2 0x00007f131e256ab5 in EVPnew (name_obj=0x7f131e46a500, digest=0x0, initial_ctx=0x7f131e459a40, cp=0x0, len=0) at /data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.c:410
#3 0x00007f131e25726e in EVP_new_md5 (self=<value optimized out>, args=<value optimized out>) at /data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.c:799
#4 0x00000000004c7eef in ?? ()
Here is the complete list of commands used
tar -axf Python-3.4.0rc3.tgz
cd Python-3*
# For lzma and a pip-compatible openssl
export CFLAGS='-I/data2/soft/openssl/include/openssl -I/data2/local/include/'
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/:$LD_LIBRARY_PATH"
# Ready !
./configure --prefix=/data2/soft/python3
make
make install
Notes:
OS is SUSE Linux Enterprise Server 11 (x86_64)
pointing python to a custom location for the lzma lib located in
openssl was built with ./config shared --openssldir=/data2/soft/openssl
openssl make test prints ALL TESTS SUCCESSFUL.
without the custom openssl *FLAGS, make install is successful and I get these results for make test:
71 tests OK.
tests failed:
test_cmd_line test_gdb test_smtpnet test_ssl
How can I fix this, or at least investigate what is going on ?
Edit 1--5:
The shared libs were generated correctly:
> ls /data2/soft/openssl/lib
drwxr-xr-x engines
-rw-r--r-- libcrypto.a
lrwxrwxrwx libcrypto.so -> libcrypto.so.1.0.0
-r-xr-xr-x libcrypto.so.1.0.0
-rw-r--r-- libssl.a
lrwxrwxrwx libssl.so -> libssl.so.1.0.0
-r-xr-xr-x libssl.so.1.0.0
drwxr-xr-x pkgconfig
So I changed this in Setup :
SSL=/data2/soft/openssl/
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
$(SSL)/lib/libssl.a $(SSL)/lib/libcrypto.a -ldl
And I changed back LDFLAGS/CFLAGS accordingly. But there still is a -lssl when I run make clean && make, because of the _hashopen module:
gcc -pthread -shared -L/data2/local/lib/ -L/data2/local/lib/ -L/data2/local/lib/ -I/data2/local/include/ build/temp.linux-x86_64-3.4/data2/soft/python3/Python-3.4.0rc3/Modules/_hashopenssl.o -L/data2/local/lib/ -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
I guess it is the one causing the cores, because they are still there... I tried adding similar stuff to Setup file, but there is no commented item for this one and creating it causes another more cryptic failure:
gcc -pthread -Xlinker -export-dynamic -o python Modules/python.o libpython3.4m.a -lpthread -ldl -lutil /data2/eoubrayrie/soft/openssl/lib/libssl.a /data2/eoubrayrie/soft/openssl/lib/libcrypto.a -ldl /data2/eoubrayrie/soft/openssl/lib/libssl.a /data2/eoubrayrie/soft/openssl/lib/libcrypto.a -ldl -lm
libpython3.4m.a(config.o):(.data+0x158): undefined reference to `PyInit__hashopenssl'
collect2: ld returned 1 exit status
Edit 6:
I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together
But I can get it linking dynamically to my own openssl through good old -I/-L:
ldd build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
libssl.so.1.0.0 => /data2/soft/openssl/lib/libssl.so.1.0.0 (0x00007f5605799000)
libcrypto.so.1.0.0 => /data2/soft/openssl/lib/libcrypto.so.1.0.0 (0x00007f56053bd000)
Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?
From To Syms Read Shared Object Library
0x00007ffff5465930 0x00007ffff5466e98 Yes /data2/soft/python3/Python-3.4.0rc3/build/lib.linux-x86_64-3.4/_hashlib.cpython-34m.so
0x00007ffff5321220 0x00007ffff5351878 Yes /opt/python-2.6-64/lib/libssl.so.1.0.0
0x00007ffff50d3100 0x00007ffff519b118 Yes /opt/python-2.6-64/lib/libcrypto.so.1.0.0
env | grep -F 'python-2.6-64' -> shows nothing !
grep -RF 'python-2.6-64' /etc/ld.so.* -> idem
gcc -print-search-dirs | sed 's/:/\n/g' | grep python -> idem
find . -name '*.so*' | xargs ldd | grep ssl -> only gives me the good ones
Level 1 dependencies don't require any wrong ssl version either. This was checked with: find . -name '*.so*' | xargs ldd | awk '/\t+[[:alnum:].]+ => [[:alnum:]./]+ \(/ {print $3}' | sort | uniq | xargs ldd | grep ssl
strace ./python ./Tools/scripts/run_tests.py 2>&1 | grep python-2.6-64 -> shows nothing
So how does ld picks this wrong library if he cannot konow about it ?? It's not in any standard location (if it were in /lib I could understand...)
Solution:
Found how to statically link _hashlib thanks to this OpenOffice bug: though the -Wl,--exclude-libs=ALL" option didn't work either, it pointed me to the right lines in setup.py.
TL;DR Here is the patch to setup.py I applied.
And finally... it works !
#noloader I'm accepting your most complete answer as you help was invaluable, though the "exact" answer for anyone encountering this problem is to compile with the patch above.
How can I fix this, or at least investigate what is going on ?
...
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/
I run into these problems a lot too because I avoid the crippled versions of OpenSSL shipped by Debian, Ubuntu, Fedora, et al. For example, Ubuntu ships an OpenSSL that disables TLSv1.1 and TLS v1.2 (cf., Ubuntu 12.04 LTS: OpenSSL downlevel version and does not support TLS 1.2).
You're probably loading the wrong version of the OpenSSL library. If you can get the misbehaving program under the debugger, issue info shared to see which libcrypto and libssl you are actually loading.
ldd might help too. Run it on the Pyhton executable: ldd /data2/soft/python3/python. I can only say it "may" help because the OpenSSL's are binary compatible, so you might only see a dependency on, for example, libcrypto.so.1.0.0 (use otool -L on Mac OS X). Below I used an rpath to force linking against the libraries in /usr/local/ssl/lib/.
$ ldd my-test.exe
linux-vdso.so.1 => (0x00007fffd61ff000)
libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00007f151528e000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007f1514e74000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1514c42000)
...
As a fix, you might try adding an rpath:
LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/ -Wl,-rpath,/data2/soft/openssl/lib'
A better fix would be to link to the static version of the OpenSSL library to avoid these problems all together. I believe you can do that with: -Bstatic -lssl -lcrypto -Bdynamic -ldl.
Personally, I don't even use -Bstatic because of different one-off problems. I open the Makefiles, remove all instances of -lssl -lcrypto, and add a full path to the archive to remove all ambiguity. For example, /data2/soft/openssl/lib/libssl.a and /data2/soft/openssl/lib/libcrypto.a.
Note well: an rpath is not honored on Mac OS X. More extreme measures are needed for Mac OS X because the linker does not honor -Bstatic either. You'll have to use the full path trick.
openssl was built with ./config shared --openssldir=/data2/soft/openssl
Another thing... On Fedora, its not enough to specify shared. You need to add the following, too:
export CFLAGS="-fPIC"
Otherwise, the shared object is not built. If the 1.0.1f shared object is not built, you're probably getting a downlevel version supplied by the distro. You can check what was built before install with:
./config shared --openssldir=/data2/soft/openssl
make all
# Verify artifacts
find . -iname -libcrypto.*
find . -iname -libssl.*
# Proceed if OK
sudo make install
Finally, make sure that all of Python's dependencies are also using your version of OpenSSL and not the system's version of OpenSSL.
I recently suffered that problem when my program used my OpenSSL; but my program also used libevent and libevent used the system's version of OpenSSL. I fixed it by rebuilding libevent and forcing it to statically link to my version of OpenSSL.
How should I go about changing it ? Will adding a _ssl.c: gcc ... line anywhere override the default behaviour ? My Makefile skills are rusted and this beast is 1600+ lines long !!
Here are the files you want to look at:
$ cd Python-3.4.0rc3
$ grep -R -- '-lcrypto' *
Modules/Setup.dist:# -L$(SSL)/lib -lssl -lcrypto
$ grep -R -- '-lssl' *
Modules/Setup.dist:# -L$(SSL)/lib -lssl -lcrypto
Here are the lines of interest in Setup.dist:
# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
#_ssl _ssl.c \
# -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
# -L$(SSL)/lib -lssl -lcrypto
Uncomment the lines and change Setup.dist to the following. I keep my OpenSSL in /usr/local/ssl, so that's how mine is setup below (you should use /data2/soft/openssl/lib/...):
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
/usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Use the full path to the archive, and don't use -l. Be sure to add -ldl because OpenSSL needs it in this configuration.
Once you change Setup.dist, re-run ./configure to effect the changes.
After changing the above line to, here's what I look like after a configure:
$ grep -R "libssl.a" *
Makefile:LOCALMODLIBS= /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Makefile:Modules/_ssl$(SO): Modules/_ssl.o; $(BLDSHARED) Modules/_ssl.o /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl -o Modules/_ssl$(SO)
Modules/Setup: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
Modules/Setup.dist: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl
You can test the static linking by using ldd or otool -L. You will not see an OpenSSL dependency. After make'ing, here's what I got:
$ find . -iname python
./python
$ ldd ./python
linux-vdso.so.1 => (0x00007fff67709000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3aed8e1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3aed6dd000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3aed4d9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3aed257000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3aececc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3aedb14000)
No libssl or libcrypto dependencies to go wrong :) And make test ran fine (actually, one failed test due to a Python bug: Issue 20896):
======================================================================
ERROR: test_get_server_certificate (test.test_ssl.NetworkedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1373, in test_get_server_certificate
_test_get_server_certificate('svn.python.org', 443, SVN_PYTHON_ORG_ROOT_CERT)
File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1354, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port))
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 902, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 344, in wrap_socket
_context=self)
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 540, in __init__
self.do_handshake()
File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 767, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:598)
----------------------------------------------------------------------
Ran 96 tests in 8.610s
FAILED (errors=1, skipped=3)
test test_ssl failed
make: *** [test] Error 1
This isn't an answer, just an observation after running though this with you:
$ make
...
gcc -pthread -c -Wno-unused-result -Werror=declaration-after-statement
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include
-DPy_BUILD_CORE -o Objects/capsule.o Objects/capsule.c
-fwrapv is really bad. Its used to make illegal programs work. Its better to fix the broken program and drop the -fwrapv. See Ian Lance Taylor's blog on Signed Overflow.
I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together
grep is your friend ;)
$ grep -R _hashlib * | grep ssl
Lib/hashlib.py: f = getattr(_hashlib, 'openssl_' + name)
Lib/hashlib.py: _hashlib.openssl_md_meth_names)
Lib/test/test_hashlib.py: self.assertTrue(hasattr(_hashlib, 'openssl_md5'))
Lib/test/test_hashlib.py: self.assertTrue(hasattr(_hashlib, 'openssl_sha1'))
Lib/test/test_hashlib.py: constructor = getattr(_hashlib, 'openssl_'+algorithm, None)
Lib/test/ssltests.py:TESTS = ['test_asyncio', 'test_ftplib', 'test_hashlib', 'test_httplib',
Lib/test/time_hashlib.py: print(" '_hashlib' 'openssl_hName' 'fast' tests the builtin _hashlib")
Modules/_hashopenssl.c: "_hashlib.HASH", /*tp_name*/
Modules/_hashopenssl.c:static struct PyModuleDef _hashlibmodule = {
Modules/_hashopenssl.c: "_hashlib",
Modules/_hashopenssl.c:PyInit__hashlib(void)
Modules/_hashopenssl.c: m = PyModule_Create(&_hashlibmodule);
PCbuild/build_ssl.py:# Script for building the _ssl and _hashlib modules for Windows.
PCbuild/build_ssl.py:# for the actual _ssl.pyd and _hashlib.pyd DLLs.
PCbuild/build_ssl.py:# it should configure and build SSL, then build the _ssl and _hashlib
setup.py: exts.append( Extension('_hashlib', ['_hashopenssl.c'],
setup.py: print("warning: openssl 0x%08x is too old for _hashlib" %
Tools/ssl/test_multiple_versions.py: "test_asyncio", "test_ftplib", "test_hashlib", "test_httplib",
Tools/ssl/test_multiple_versions.py:MINIMAL_TESTS = ["test_ssl", "test_hashlib"]
Another partial answer...
But I can get it linking dynamically to my own openssl through good old -I/-L:
...
Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?
That's your good old friends -l and -L. Don't use them because they do this sort of thing all the time (take it from a guy who has suffered it in the past). Instead, specify the full path to libssl and libcrypto. E.g., use /data2/soft/openssl/lib/libssl.a.
We had a similar problem. We are using apache httpd + mod_wsgi + python + django and our c++ module for the apache httpd which also uses openssl. Now everything is loaded within one httpd process, correct version of openssl shared lib was loaded (1.0.0l) with our c++ module. But as soon as we access the web, python loads the hashlib and exactly the same problem appears - segfault in openssl called from python.
Normally python compiles with whatever openssl is available at the default location and there is no way how to specify it without fiddling with setup.py or makefiles. Python developers should add configure setting --with_ssl=path.
We installed new openssl libs and rebuild python and other binaries but with no success. We mapped default libssl.so and libcrypto.so to the new openssl binaries with no success. Finally after reading this thread I realized that probably wrong headers are being used while compiling python. And that was the problem. Follow the step to workaround the problem:
there must not be incorrect version of openssl headers in /usr/include and default locations /usr/local/ssl, /usr/contrib/ssl (if you installed openssl-devel then uninstall it, or simply erase/rename the directory)
yum remove openssl-devel
make symbolic link to your openssl installation from /usr/local/ssl
ln -s /opt/openssl-1.0.1l /usr/local/ssl
ensure that the new openssl libs are accessible from /usr/lib (make symbolic links if it is not installed here)
ln -s /opt/openssl-1.0.1l/lib/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.1.0.0
...
now configure and clean build the python
I am able to run python 2.7.11 with non-default SSL after patching fix mentioned here https://gist.github.com/eddy-geek/9604982
However, with this it's not building _socket module which is required by many other modules. for example, easy_install / pip started failing with error Importerr: no module named _socket
In Module/Setup.dist, am i suppose to uncomment or comment the line
_socket socketmodule.o
?
I see socketmodule.o and timemodule.o getting generated. but not _socket.so
Am i missing something ?