We Keep Coding

how to succesfully compile python 3.x

Upon attempting to compile python 3.7 I hit Could not import runpy module:
jeremyr#b88:$ wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
....
jeremyr#b88:~/Python-3.7.3$ ./configure --enable-optimizations
jeremyr#b88:~/Python-3.7.3$ make clean
jeremyr#b88:~/Python-3.7.3$ make -j32
....
gcc -pthread -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o libpython3.7m.a -lcrypt -lpthread -ldl -lutil -lm
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Could not import runpy module
Traceback (most recent call last):
File "/home/jeremyr/Python-3.7.3/Lib/runpy.py", line 15, in <module>
import importlib.util
File "/home/jeremyr/Python-3.7.3/Lib/importlib/util.py", line 14, in <module>
from contextlib import contextmanager
File "/home/jeremyr/Python-3.7.3/Lib/contextlib.py", line 4, in <module>
import _collections_abc
SystemError: <built-in function compile> returned NULL without setting an error
generate-posix-vars failed
Makefile:603: recipe for target 'pybuilddir.txt' failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory '/home/jeremyr/Python-3.7.3'
Makefile:531: recipe for target 'profile-opt' failed
make: *** [profile-opt] Error 2
jeremyr#88:~/Python-3.7.3$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.11 (jessie)
Release: 8.11
Codename: jessie
jeremyr#88:~/Python-3.7.3$ gcc --version
gcc (Debian 4.9.2-10+deb8u2) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
jeremyr#88:~/Python-3.7.3$ sudo apt upgrade gcc
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... gcc is already the newest version.
jeremyr#b88:~/Python-3.7.3$ echo $PYTHONPATH
Any advice on how to overcome this and install python3.7 appreciated.
Edit - the solution listed below seems to work for various other python versions, so I changed title to python 3.x from 3.7

It seems the enable-optimizations was the problem,
jeremyr#b88:~/Python-3.7.3$ ./configure
jeremyr#b88:~/Python-3.7.3$ make clean
takes care of it in my case.

In case others come across this question: I encountered the same problem on Centos 7. I also had --enable-optimizations but didn't want to remove that flag. Updating my build dependencies and then re-running solved the problem. To do that I ran:
sudo yum groupinstall "Development Tools" -y
In case the yum group is not available, you can also install the pacakges individually using:
sudo yum install bison byacc cscope ctags cvs diffstat doxygen flex gcc gcc-c++ gcc-gfortran gettext git indent intltool libtool patch patchutils rcs redhat-rpm-config rpm-build subversion swig systemtap

For whomever MicGer's answer didn't work and would like to retain --enable-optimizations, check your gcc version. The error was solved for me on gcc 8.3.0.

For me, CentOS 7, compiling Python3.9, the fix was updating the gcc and related packages.

Removing --enable-optimizations didn't solved the issue here, neither upgrading GCC from gcc-7.5.0 to gcc-8.2.1 trying building Python 3.6.13 on opensuse-15.2.
Here for some reason the build is picking a local python installation, so before calling make I have updated the PATH env to include the build directory (were python file is generated), for instance: export PATH=$(pwd):$PATH

How to install libcurl with nss backend in aws ec2? (Python 3.6 64bit Amazon Linux)

I have an ec2 instance in AWS running Python3.6 (Amazon Linux/2.8.3) where I need to install pycurl with NSS ssl backend.
First I tried it by adding pycurl==7.43.0 --global-option="--with-nss" to my requirement.txt file but I was getting errors installation errors. So I ended up doing it by adding a .config file in .ebextensions (that runs during deployment):
container_commands:
09_pycurl_reinstall:
# the upgrade option is because it will run after PIP installs the requirements.txt file.
# and it needs to be done with the virtual-env activated
command: 'source /opt/python/run/venv/bin/activate && PYCURL_SSL_LIBRARY=nss pip3 install pycurl --global-option="--with-nss" --upgrade --no-cache-dir --compile --ignore-installed'
Pycurl seems to be correctly installed, but the celery worker is not running. The celery worker logs show:
__main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory
If I ssh connect to the instance and run python 3.6 -c 'import pycurl' I get a more detailed error:
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (nss)
So I guess that my problem is that I had previously installed libcurl with openSSL instead of NSS, and hence the mismatch between libcurl and pycurl.
According to another stackoverflow question, for libcurl to be installed with NSS backend I should have installed it with:
sudo apt libcurl4-nss-dev
But since the server is running Amazon Linux I can't use the apt command. So I did instead:
yum install libcurl-devel
And I guess this is the problem: this installs libcurl with OpenSSL support when I need it with NSS support.
How can I install libcurl with NSS in Amazon Linux?? (I need NSS because I'm running a Django app with celery using SQS as the broker, and SQS requires NSS).
Thank you very much!

I just ran into the same issue and did manage to fix it :)
Retrieve Amazon Linux configure options for libcurl:
curl-config --configure
All options referring to paths can be ignored, but other must be kept if you want the same features than system libcurl. Of course, --with-ssl will be replaced with --without-ssl --with-nss.
1.1 Install prerequisites:
sudo yum install libssh2-devel nss-devel
(of course you should rather add then to the packages > yum section of your ebextensions)
Compile libcurl from source (i chose 7.61.1 to match the one used by Amazon Linux 2018.03):
wget https://curl.haxx.se/download/curl-7.61.1.tar.gz
tar xf curl-7.61.1.tar.gz
cd curl-7.61.1
./configure '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-amazon-linux-gnu' '--program-prefix=' '--cache-file=../config.cache' '--disable-static' '--enable-symbol-hiding' '--enable-ipv6' '--enable-threaded-resolver' '--with-gssapi' '--with-nghttp2' '--without-ssl' '--with-nss' '--with-ca-bundle=/etc/pki/tls/certs/ca-bundle.crt' '--enable-ldap' '--enable-ldaps' '--enable-manual' '--with-libidn2' '--with-libpsl' '--with-libssh2' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-amazon-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
make
sudo make install
(of course this should be deployed properly as a Shell script through the files section of your ebextensions).
Now you should see libcurl.so.4 created in /usr/local/lib.
Define the following envvars for pycurl to be compiled using your custom libcurl:
LD_LIBRARY_PATH=/usr/local/lib
PYCURL_CURL_CONFIG=/usr/local/bin/curl-config
PYCURL_SSL_LIBRARY=nss
Run your pip install
You can check pycurl linked to the right libcurl:
ldd /opt/python/run/venv/local/lib64/python3.6/site-packages/pycurl.cpython-36m-x86_64-linux-gnu.so
should show you libcurl.so.4 => /usr/local/lib64/libcurl.so.4
And of course python 3.6 -c 'import pycurl' should work.
That's it! You should be able to run Celery with SQS.

I had loads of issues getting pycurl installed on my Python ec2 instance. First up, pip wouldn't install pycurl at all. I had to ssh and run
yum install libcurl-devel
After this, I added the following line to my requirements.txt file so that it included the relevant ssl option (--with-openssl) as an install option:
pycurl==7.43.0.5 --install-option="--with-openssl"
Once I deployed a new version with this change in requirements.txt, pycurl worked fine, which I tested with:
python 'import pycurl'
I knew I needed to use openssl from the error message
libcurl link-time ssl backend (openssl)

How to use transform_graph to optimize Tensorflow model

I used to use the optimize_for_inference library in optimizing frozen Tensorflow models. However, I have read from different sources that Tensorflow no longer supports it.
I came across transform_graph, and its documentation is found here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms#strip_unused_nodes
At first, I ran into errors and found out that I need to install Tensorflow from source (https://www.tensorflow.org/install/install_sources#install_the_pip_package) instead of using PIP.
I already re-installed Tensorflow from source, and ran this code in bash (/tensorflor/tensorflow dir):
bazel build tensorflow/tools/graph_transforms:transform_graph
bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph=tensorflow_inception_graph.pb \
--out_graph=optimized_inception_graph.pb \
--inputs='Mul' \
--outputs='softmax' \
--transforms='
strip_unused_nodes(type=float, shape="1,299,299,3")
fold_constants(ignore_errors=true)
fold_batch_norms
fold_old_batch_norms
round_weights(num_steps=256)'
And ran to again this error:
-bash: bazel-bin/tensorflow/tools/graph_transforms/transform_graph: No such file or directory
What seems to be the problem?

That is weird.
The code I write below is to install and using the transform_graph in CentOS7.
yum install epel-release
yum update
yum install patch
curl https://copr.fedorainfracloud.org/coprs/vbatts/bazel/repo/epel-7/vbatts-bazel-epel-7.repo -o /etc/yum.repos.d/vbatts-bazel-epel-7.repo
yum install bazel
curl -L -O https://github.com/tensorflow/tensorflow/archive/v1.8.0.tar.gz
cd tensorflow-1.8.0
./configure # interactive!
bazel build tensorflow/tools/graph_transforms:summarize_graph
bazel-bin/tensorflow/tools/graph_transforms/summarize_graph
After you install the Tensorflow by source code and finish the configure, the bazel codes should be working.
The error from you environment is occurred when you didn't finish the install the Tensorflow by source code, or you ran the script in wrong path.
Please check the configure step, path of the Tensorflow root.

how to build opencv for python3 when both python2 and python3 are installed

I was trying to build opencv for python3. However, cmake always sets python build option to be python2.7.11 even after I manually specified include and lib option for python3:
-- Python 2:
-- Interpreter: /home/ryu/anaconda2/bin/python2.7 (ver 2.7.11)
-- Python 3:
-- Interpreter: /usr/bin/python3 (ver 3.4.3)
-- Libraries: /usr/lib/x86_64-linux-gnu/libpython3.4m (ver 3.4.3)
-- numpy: /home/ryu/.local/lib/python3.4/site-packages/numpy/core/include (ver 1.11.0)
-- packages path: lib/python3.4/dist-packages
--
-- **Python (for build): /home/ryu/anaconda2/bin/python2.7**
Did I miss some cmake option?
OS: Ubuntu 14,04
thanks

You can override the python executable to build to by appending the argument PYTHON_DEFAULT_EXECUTABLE with the python executable URI during the cmake invokation.
cmake {...} -DPYTHON_DEFAULT_EXECUTABLE=$(which python3) ..

I was struggling with this one for some hours and the answers mentioned above didn't solve the problem straightaway.
Adding to Ivan's answer, I had to include these flags in cmake to make this work:
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_opencv_python3=ON \
-D HAVE_opencv_python3=ON \
-D PYTHON_DEFAULT_EXECUTABLE=<path_to_python3>
I leave that here, so it is maybe useful for someone else in the future.

it was take some hours for me. I built Dockerfile with opencv for python3
the key string is
pip install numpy
Full Docker file:
FROM python:3.8
RUN apt-get update && apt-get -y install \
cmake \
qtbase5-dev \
libdc1394-22-dev \
libavcodec-dev \
libavformat-dev \
libswscale-dev
RUN cd /lib \
&& git clone --branch 4.1.1 --depth 1 https://github.com/opencv/opencv.git \
&& git clone --branch 4.1.1 --depth 1 https://github.com/opencv/opencv_contrib.git
RUN pip install numpy \
&& mkdir /lib/opencv/build \
&& cd /lib/opencv/build \
&& cmake -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local -DWITH_TBB=ON -DWITH_V4L=ON -DWITH_QT=ON -DWITH_OPENGL=ON -DWITH_FFMPEG=ON -DOPENCV_ENABLE_NONFREE=ON -DOPENCV_EXTRA_MODULES_PATH=/lib/opencv_contrib/modules .. \
&& make -j8 \
&& make install
CMD ["bash"]
The main point is to force compiler to build cv2 module for python
To make it we need python3 should be included in line To be built in CMakeCache.txt file in build folder of opencv
ref https://breakthrough.github.io/Installing-OpenCV/
If there are any errors, ensure that you downloaded all the required packages - the output should help track down what is missing. To ensure the Python module will be built, you should see python2 in the list of configured modules after running cmake
(in my case python3)

I've been trying to install opencv on a Pi3 and this solution didn't work for me as python (for build) was always set to Python2.7 but I found that by changing the order of an elseif statement at the bottom of 'OpenCVDetectPython.cmake' fixed the problem. For me, this file is located at '~/opencv-3.3.1/cmake'.
The original code segment:
if(PYTHON_DEFAULT_EXECUTABLE)
set(PYTHON_DEFAULT_AVAILABLE "TRUE")
elseif(PYTHON2INTERP_FOUND) # Use Python 2 as default Python interpreter
set(PYTHON_DEFAULT_AVAILABLE "TRUE")
set(PYTHON_DEFAULT_EXECUTABLE "${PYTHON2_EXECUTABLE}")
elseif(PYTHON3INTERP_FOUND) # Use Python 3 as fallback Python interpreter (if there is no Python 2)
set(PYTHON_DEFAULT_AVAILABLE "TRUE")
set(PYTHON_DEFAULT_EXECUTABLE "${PYTHON3_EXECUTABLE}")
endif()
My re-ordered code segment:
if(PYTHON_DEFAULT_EXECUTABLE)
set(PYTHON_DEFAULT_AVAILABLE "TRUE")
elseif(PYTHON3INTERP_FOUND) # Use Python 3 as fallback Python interpreter (if there is no Python 2)
set(PYTHON_DEFAULT_AVAILABLE "TRUE")
set(PYTHON_DEFAULT_EXECUTABLE "${PYTHON3_EXECUTABLE}")
elseif(PYTHON2INTERP_FOUND) # Use Python 2 as default Python interpreter
set(PYTHON_DEFAULT_AVAILABLE "TRUE")
set(PYTHON_DEFAULT_EXECUTABLE "${PYTHON2_EXECUTABLE}")
endif()
I don't know the reasoning behind it, but cmake is set to default to python2 if python2 exists, swapping the order of these elseif statements switches it to default to python3 if it exists
** Disclaimer **
I was using the script found at https://gist.github.com/willprice/c216fcbeba8d14ad1138 to download, install and build everything
(script was modified to not create a virtual environment as I didn't want one and with j1 not j4 as it failed around 85% when running with multiple cores).
I don't think the relevant file exists until you have attempted a build.

Changing the options in cmake did nothing for me no matter what options I modified. The simpliest (hacky) solution for me was to
sudo mv /usr/bin/python2.7 /usr/bin/pythonNO-temp
Then you build and install opencv
then
sudo mv /usr/bin/pythonNO-temp /usr/bin/python2.7

subversion 1.6.x(1.7.x) python 2.6+ bindings for Trac

I have Trac 0.12rc1(customized by somebody) it needs python subversion bindings to work with svn repos. But all of my attempts to compile the libraries ended with:
Last command in make:
/bin/sh /usr/local/src/subversion-1.6.20/libtool --tag=CC --silent --mode=compile gcc -pthread -fPIC -g -O2 -pthread -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -I/usr/local
/src/subversion-1.6.20/subversion -I/usr/local/src/subversion-1.6.20/subversion/include
-I/usr/local/src/subversion-1.6.20/subversion/bindings/swig -I/usr/local/src/subversion-1.6.20/subversion/bindings/swig/include
-I/usr/local/src/subversion-1.6.20/subversion/bindings/swig/proxy
-I/usr/local/src/subversion-1.6.20/subversion/bindings/swig/proxy
-I/usr/include/apr-1 -I/usr/include/apr-1 -I/usr/include/python2.6
-I/usr/local/src/subversion-1.6.20/subversion/bindings/swig/python/libsvn_swig_py
-prefer-pic -c -o subversion/bindings/swig/python/svn_client.lo subversion/bindings
/swig/python/svn_client.c
Last part of it's output(all the time looks the same):
subversion/bindings/swig/python/svn_client.c:23637: error: expected ‘)’ before ‘*’ token
subversion/bindings/swig/python/svn_client.c: In function ‘init_client’:
subversion/bindings/swig/python/svn_client.c:23690: error: ‘PyObject’ undeclared (first use in this function)
subversion/bindings/swig/python/svn_client.c:23690: error: ‘m’ undeclared (first use in this function)
subversion/bindings/swig/python/svn_client.c:23690: error: ‘d’ undeclared (first use in this function)
subversion/bindings/swig/python/svn_client.c:23693: error: ‘SwigMethods’ undeclared (first use in this function)
What I tried:
Python:
2.4 (work, but our Trac doesn't work properly with it)
2.6 "make swig-py" fails
2.7 "make swig-py" fails
Subversion:
1.6.17
1.6.20
1.7.8
SWIG:
2.0.9
1.3.29
Does anyone had success with building subversion bindings for python2.6+ ?
Is it posible at all?
Is it possible to find any pre-built binaries for RHEL\OEL\CentOS 5.x ?

Use this: http://egao1980.blogspot.com/2011/03/installing-trac-and-subversion-with.html
Below copy paste from site:
Install RPMForge repo rpm -ihv http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
Good to have
yum install bash-completion
Black list subversion in [base] and [updates] repos
vim /etc/yum.repos.d/CentOS-Base.repo
[base]
exclude=subversion
exclude=subversion-devel
...
[updates]
exclude=subversion
exclude=subversion-devel
...
Get 1.6.6 Subversion install script from www.wandisco.com and follow instructions to install it.
Install MySQL.
yum install mysql mysql-devel
Install Python 2.5, Trac and rebuild mod_python accordig to Installing-python-25-on-centos-5 and Installing-trac-on-centos-5.
Build Subversion bindings:
get Sqlite:
wget http://www.sqlite.org/sqlite-autoconf-3070500.tar.gz
tar xzvf sqlite-autoconf-3070500.tar.gz && cd sqlite-autoconf-3070500 && make && make install
get and build SWIG
wget http://downloads.sourceforge.net/project/swig/swig/swig-2.0.2/swig-2.0.2.tar.gz
tar xzvf swig-2.0.2.tar.gz && ./configure --with-python=/usr/bin/python25 --prefix=/usr && make && make install
build Python 2.5 subversion bindings
wget http://subversion.tigris.org/downloads/subversion-1.6.16.tar.bz2
tar xjvf subversion-1.6.16.tar.bz2 && cd subversion-1.6.16
./configure PYTHON=/usr/bin/python25 --with-sqlite=/usr/local && make && make swig-py && make install-swig-py
echo /usr/local/lib/svn-python > /usr/lib/python2.5/site-packages/svn.pth
At this point you should have Subversion 1.6.6, Trac 0.12, MySQLDb 1.2.2 and mod_python configured to run with Python 2.5.

I've contacted to the Trac-users maillist and they provided a solution. It was a patch for spec file for rpm building. After patch was applied to the package everything went fine.
https://groups.google.com/d/topic/trac-users/BVVnh9I17Po/discussion