We Keep Coding

Problems building a Python/C++ pybind11 package with `CBUILDWHEEL` using Github actions?

I'm currently developing a C++/Python package using pybind11 for the Python bindings. This project is mixed: it has parts which are written in Python and other parts which are written in C++ and are compiled as an external module.
The project uses cmake>1.7. Roughly this is the directory structure of the project:
PythonProject
-> python_sources
-> include [cpp headers]
-> src [cpp sources]
--> module.cpp [pybind11 bindings declaration]
--> cpp_sources [where the implementation of the headers goes]
The project also depends on Eigen3.
I have the project setup, so that when building the wheels the cpp module is compiled first and then the *.so (if macOS) is copied to python_sources.
In my machine (macOS 11.2 with XCODE 12) I can generate the wheels with
python -m build or python -m build --sdist(depending if I am only building source distributions)
This works well and it copies the cpp built module correctly to PythonProject/python_sources so that the module can be imported as import python_sources.module.
Also, I'm able to test the installation with pip install -e ./ and all tests pass.
The issue is that when running cbuildwheel on with Github actions, it seems that the c++ module is never built.
I have tried to force building the module and copying it to python_sources, but this also does not work, and the tests do not pass.
This is my cbuildwheel configuration:
- uses: joerick/cibuildwheel#v1.10.0
env:
# Python 2.7 on Windows with workaround not supported by scikit-build
CIBW_SKIP: cp27-win*
CIBW_BUILD: cp38-${{ matrix.cibw-arch }}
CIBW_ENVIRONMENT: PYTHONPROJECT_EXTRA_CMAKE_ARGS="-DPREBUILT_DEPENDENCIES=$(python -c 'import os; print(os.getcwd().replace(os.path.sep, '/'))')/build_dependencies"
CMAKE_BUILD_PARALLEL_LEVEL=2
CMAKE_OSX_ARCHITECTURES="arm64"
CIBW_BEFORE_ALL_MACOS: |
brew install libomp && brew install eigen && brew install openblas
CIBW_BEFORE_ALL_LINUX: |
yum install eigen3-devel
CIBW_BEFORE_BUILD: cmake -S . -B build_dependencies $CMAKE_ARCH && cmake --build build_dependencies --target numerical -j 2 && cp build_dependencies/src/*.so python_sources/
CIBW_BEFORE_BUILD_MACOS: cmake -S . -B build_dependencies -DCMAKE_OSX_ARCHITECTURES="arm64" $CMAKE_ARCH && cmake --build build_dependencies --target module -j 2 && cp build_dependencies/src/*.so python_sources/
CIBW_BEFORE_TEST: pip install --upgrade -r tests/requirements.txt
CIBW_TEST_COMMAND: pytest {project}/tests
I want to be able to build these wheels on GitHub actions (not necessarily with cibuildwheels, although that is my preferred option). But I am quite lost here and would really appreciate some help on how to setup the CI configuration.

Error installing PyMVPA2

I'm getting the following error when I try to install using pip install for PyMVPA2 for Python. I have installed other libraries without any problems before. I would appreciate if anything could take a look at the errors:
C:\Users\usr>pip install pymvpa2
Collecting pymvpa2
Using cached pymvpa2-2.4.2.tar.gz
Complete output from command python setup.py egg_info:
running egg_info
running build_src
build_src
building extension "mvpa2.clfs.libsmlrc.smlrc" sources
building extension "mvpa2.clfs.libsvmc._svmc" sources
creating build
creating build\src.win32-2.7
creating build\src.win32-2.7\mvpa2
creating build\src.win32-2.7\mvpa2\clfs
creating build\src.win32-2.7\mvpa2\clfs\libsvmc
swig.exe++: mvpa2\clfs\libsvmc\svmc.i
swig.exe -python -I3rd\libsvm -c++ -I3rd\libsvm -o build\src.win32- 2.7\mvpa2\clfs\libsvmc\svmc_wrap.cpp -outdir build\src.win32-2.7\mvpa2\clfs\libsvmc mvpa2\clfs\libsvmc\svmc.i
error: command 'swig.exe' failed: No such file or directory
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\usr\appdata\local\temp\pip-build-v_zzkd\pymvpa2\

As the error states you either don't have swig installed or it is not on your path. You are obviously on Windows so you need to download the swig executable from the swig website - download the latest windows package and either unzip it to a directory on your existing path or to a new directory and add it to your path. Easiest, of course, is to unzip it to a new directory and then run your pip command in that directory.
What is swig anyway? Swig parses the interface definitions of code written in C/C++ and can output 'glue code' to allow code written in Python & a pile, currently 22, of other programming or scripting languages to use the C/C++ code transparently. Swig is free, open source and compatible with both open source and commercial use.
Should I keep it to hand? In most cases that I know of in python swig is only invoked during the installation or setup phase but since that setup phase will be used every time that any package built with swig is installed or upgraded there is no reason to get rid of it and quite a lot of reasons to have it somewhere on your path.

sudo apt-get remove swig
sudo apt-get install swig3.0
sudo ln -s /usr/bin/swig3.0 /usr/bin/swig
and then
pip install pymvpa2

Install m2crypto on a virtualenv without system packages

I have created a virtual environment without the system packages with python's virtualenv in Ubuntu and installed m2crypto, but when I execute a shell and I try to import M2Crypto i get the following error:
ImportError: /home/imediava/.virtualenvs/myenv/local/lib/python2.7/site-
packages/M2Crypto/__m2crypto.so: undefined symbol: SSLv2_method
From outside the environment I run into the same problem unless from ubuntu I install python-m2crypto with apt-get. I know that I could create the environment with the system packages but I would prefer not to do it.
Is there anyway that I could create a virtual environment without the system packages and then install m2crypto with pip without running into the SSLv2_method?

There seems to be a regression bug from an earlier version of M2Crypto.
After placing M2Crypto's source in your virtualenv, you can try to patch it with the diff code below.
You do this by downloading the source code, untar it via:
tar -xzf M2Crypto-0.21.1.tar.gz
This should create the directory M2Crypto-0.21.1 which will contain the SWIG directory
In SWIG you'll find _ssl.i, which is the file to be patched. In the same directory create a file called _ssl.i.patch for example using the nano editor and paste the complete diff code listed below into it.
Next issue the patch _ssl.i _ssl.i.patch command to merge the patch into the code. (Afterwards you may remove the patch file if you want.)
Finally issue the commands:
python setup.py build
followed by:
python setup.py install
to install manually.
diff code:
--- SWIG/_ssl.i 2011-01-15 20:10:06.000000000 +0100
+++ SWIG/_ssl.i 2012-06-17 17:39:05.292769292 +0200
## -48,8 +48,10 ##
%rename(ssl_get_alert_desc_v) SSL_alert_desc_string_long;
extern const char *SSL_alert_desc_string_long(int);
+#ifndef OPENSSL_NO_SSL2
%rename(sslv2_method) SSLv2_method;
extern SSL_METHOD *SSLv2_method(void);
+#endif
%rename(sslv3_method) SSLv3_method;
extern SSL_METHOD *SSLv3_method(void);
%rename(sslv23_method) SSLv23_method;

You can install this lib in your global environment and then just copy from your global site-packages to virtualenv.

M2Crypto 0.22.3 (the current version in pypi) fixes this problem, so the simplest solution is now simply:
pip install --upgrade M2Crypto
M2Crypto 0.22.3 has been released from martinpaljak's github repository, rather than from the original M2Crypto repository.

I had the same problem with the current release (M2Crypto-0.22.5). The latest RC build worked for me.
pip install M2Crypto==0.22.6rc4

There is a patch slated for 0.26.1.
In the meantime, you can clone the repo, apply the patch, and install from source.
git clone https://gitlab.com/m2crypto/m2crypto.git
(
cd m2crypto
git checkout 0.25.1
curl 'https://gitlab.com/m2crypto/m2crypto/merge_requests/117.diff' | git apply -v
)
pip install -U m2crypto

How to build pgmagick under pythonbrew on OS X?

I'm not having much success when attempting building pgmagick on OS X Lion with XCode 4.3.1.
I've installed both ImageMagick and GraphicsMagick, along side boost, using the following commands (via homebrew):
$ brew install graphicsmagick --with-magick-plus-plus
$ brew install imagemagick --with-magick-plus-plus
$ brew install boost --with-thread-unsafe
then I'm cloning the repo at https://bitbucket.org/hhatto/pgmagick:
$ hg clone https://bitbucket.org/hhatto/pgmagick/src
$ cd pgmagick
$ python setup.py build
However I always receive the following error:
ld: library not found for -lboost_python
collect2: ld returned 1 exit status
Based on the output on stdout, setup is looking in the right place for the boost (/usr/local/lib).
I've also tried easy_install and pip but with no luck. I'm using Pythonbrew but have also disabled this and tried using the stock python install -- still no success.
Any suggestions on how I can fix the problem, or further diagnose the issue?

According to my own reproduction of this issue in brew 0.9 and OSX 10.6.8, the problem is --with-thread-unsafe isn't being honored by the current brew formula file. You can verify this by checking the formula with brew edit boost and seeing if the option appears within the contents of the formula.
Because of this, libboost_python-mt.a and libboost_python-mt.dylib are being built instead of libboost_python.a and libboost_python.dylib.
The easiest ways to fix this are to edit your pgmagick setup.py to replace boost_lib="boost_python" with boost_lib="boost_python-mt" (as pointed out here) or to follow the instructions and patch here. It's otherwise a known issue.

The boost_python lib inside /usr/local/lib/ is named after libboost_python-mt.a and libboost_python-mt.dylib, since the default compiling is w/ multi-threads supporting enabled.
Grep boost_lib="boost_python" under ELSE condition in setup.py and replace it w/ boost_lib="boost_python-mt", will fix the "not found" issue.
Also it's OK to ln "-mt" version to libboost_python.a: as described here for linux boost which no longer appends '-mt' suffix since 1.42.
Ignore this line or you could "with-boost-python=boost_python-mt python setup.py install".
You could probably append '--with-boost-python=boost_python-mt' to extra_compile_args inside setup.py, to achieve the same goal.
Furthermore, you could install pgmagick through pip in managed envs. Refs http://rohanradio.com/blog/2011/12/02/installing-pgmagick-on-os-x/

Note that as of July 2014 the boost Python library is a separate homebrew package called boost-python.
5254f8f510fb30484f8fac8be3d38e388a4392e2
Author: Tim D. Smith <git#tim-smith.us>
Date: Sat Jul 19 15:37:25 2014 -0700
Split out Boost.Python
You need to install it separately to get the libboost_python shared library.

Does setting DYLD_FALLBACK_LIBRARY_PATH=/usr/local/lib in the environment help before the build
e.g
$ export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/lib
$ hg clone https://bitbucket.org/hhatto/pgmagick/src
$ cd pgmagick
$ python setup.py build

I've submitted a pull request to homebrew to build Boost with both mt and non mt (threaded and thread unsafe) binaries which are required to build pgmagick.
Turns out this is a rather common problem, until the patch is accepted, you can check out or use my formula for Boost to build pgmagick.

installing/building pymssql on Mac OS 10.6 (python 2.6)

As instructed in the README and here http://code.google.com/p/pymssql/wiki/Compilation, I've installed Cython (v0.14.1), FreeTDS (v0.82 using MacPorts), and I already have XCode installed with gcc.
I've run into build errors during python setup.py install:
a lot of undeclared name not builtin: with things like strlen, PyMem_Malloc, PyMem_Free, PY_LONG_LONG as well as a few ___ is deprecated, use 'cpython'
Google found me this thread http://groups.google.com/group/cython-users/browse_thread/thread/468bb80480ede699/dc8267a4274c2413 where someone addresses the first error (changed to from libc.string cimport strlen, strcpy from from stdlib cimport strlen, strcpy.)
Now I'm getting: _mssql.pyx:650:34: Cannot convert Python object to 'const_char *'
Here is the file that these issues are coming from http://code.google.com/p/pymssql/source/browse/tags/1.9.908/_mssql.pyx
My question:
If I have not prepared my system correctly for pymssql what have I done wrong?
or
If the _mssql.pyx file is using deprecated imports how do I fix them?

Have you tried building 1.9.909 from the trunk? It builds cleanly for me, where .908 did not.
Unfortunately I get import errors even though there is a successful build and install: Building pymssql on OS X

I have pip installed so I obtained Cython using:
pip install Cython
I went to www.freetds.org and got the cvs command to download the latest source (User Guide then "What to build: Packages, Tarballs, and the CVS repository"). To build this, ignore the instructions on the web site and look at the file INSTALL.CVS in the root of the repository you just downloaded. When you run ./autogen.sh use:
./autogen.sh --with-tdsver=7.0
the autogen.sh will create a configure script then the configure script will get the switch. You need to be using 7.0 for python to work with TDS and MS SQL. After the autogen.sh you do a 'make' then a 'sudo make install'.
Now you need the pymssql tarball. Unpack it and run:
python setup.py config
python setup.py build
sudo python setup.py install
that should do it.

This fork compiles like a charm:
https://github.com/blackbass1988/pymssql-macos-lion