We Keep Coding

Boost/Py-Boost not being found in Conda Environment while Installing ParallelFDTD

I'm trying to follow the instructions here to install ParallelFDTD's python bindings on a windows machine. I've barely worked with C++ before, and certainly never with Boost! Mostly have experience with Python.
So far, I have done the following:
installed the Windows SDK
installed the CUDA toolkit
I have successfully created a conda environment:
conda create -n PFDTD -c conda-forge boost py-boost cmake numpy scipy
conda activate PFDTD
Running conda list shows that boost 1.73 is installed, along with py-boost 1.73 and libboost 1.73 as well as boost-cpp 1.68.
So far so good.
When I try to install and build the library as per the instructions with the following, I get an error
pip install git+https://github.com/AaltoRSE/ParallelFDTD.git
produces:
Building wheels for collected packages: pyParallelFDTD
Running command python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-38
creating build\lib.win-amd64-cpython-38\pyParallelFDTD
copying dist\libPyFDTD\__init__.py -> build\lib.win-amd64-cpython-38\pyParallelFDTD
running build_ext
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22000.
-- The CXX compiler identification is MSVC 19.33.31630.0
-- The CUDA compiler identification is NVIDIA 11.8.89
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin/nvcc.exe - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- CONDA_PREFIX C:\ProgramData\Anaconda3\envs\opti-acoustics
-- Found Python: C:/ProgramData/Anaconda3/envs/opti-acoustics/python.exe (found version "3.8.13") found components: Interpreter
-- Found PythonLibs: C:/Python310/libs/python310.lib (found version "3.10.0")
-- CONDA_PREFIX C:\ProgramData\Anaconda3\envs\opti-acoustics
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
CMake Error at C:/ProgramData/Anaconda3/envs/opti-acoustics/Library/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Boost (missing: Boost_INCLUDE_DIR system thread date_time
unit_test_framework python38 chrono numpy38) (Required is at least version
"1.41")
Call Stack (most recent call first):
C:/ProgramData/Anaconda3/envs/opti-acoustics/Library/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/ProgramData/Anaconda3/envs/opti-acoustics/Library/share/cmake-3.24/Modules/FindBoost.cmake:2376 (find_package_handle_standard_args)
CMakeLists.txt:86 (find_package)
-- Configuring incomplete, errors occurred!
See also "C:/Users/../AppData/Local/Temp/pip-req-build-3mu877y3/build/temp.win-amd64-cpython-38/Release/CMakeFiles/CMakeOutput.log".
See also "C:/Users/../AppData/Local/Temp/pip-req-build-3mu877y3/build/temp.win-amd64-cpython-38/Release/CMakeFiles/CMakeError.log".
The key part of the error message seems to be:
Could NOT find Boost (missing: Boost_INCLUDE_DIR system thread date_time
unit_test_framework python38 chrono numpy38) (Required is at least version
"1.41")
The instructions suggest that the installation with conda should be sufficient, but it seems some environment variables are missing. In ParallelFDTD's CMakeLists.txt, you can see how it tries to handle boost installation in a conda env here.
It appears that conda install -c conda-forge boost py-boost installed successfully, for instance there are a lot of .hpp files and a /python folder as well as a bunch of other directories in C:\ProgramData\Anaconda3\envs\acoustics_env\library\include\boost. There are also a lot of boost_xxx.dll files in C:\ProgramData\Anaconda3\envs\acoustics_env\library\bin.
So far, I've forked the ParallelFDTD library so that I can make changes to the CMakeLists.txt, including pointing the BOOST_ROOT, Boost_INCLUDE_DIR and/or Boost_LIBRARY_DIR explicitly to folders in the anaconda env if need be, but I haven't been able to figure out the correct paths to use... I can also try installing boost manually, but just not sure what the best approach is.
Desperate for help! I'm sure it's a simple step I am missing...
Found lots of other threads that were somewhat related, but none of them seemed to be dealing with this kind of context of working in a conda environment with boost/py-boost.

pyarrow==0.17.1 installation ends with fatal error: 'arrow/python/config.h' file not found (Apple M1 chip)

I'm trying to install the pyarrow==0.17.1 package on my MacBook pro machine (Apple M1 chip) and I'm getting the following error
-- Found PkgConfig: /opt/homebrew/bin/pkg-config (found version "0.29.2")
-- Found Arrow: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/include (found version "7.0.0")
-- Arrow version: 7.0.0 (HOME: /opt/homebrew/Cellar/apache-arrow/7.0.0_3)
-- Arrow SO and ABI version: 700
-- Arrow full SO version: 700.0.0
-- Found the Arrow core shared library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow.dylib
-- Found the Arrow core import library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow.dylib
-- Found the Arrow core static library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow.a
-- Found ArrowPython: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/include (found version "7.0.0")
-- Found the Arrow Python by HOME: /opt/homebrew/Cellar/apache-arrow/7.0.0_3
-- Found the Arrow Python shared library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow_python.dylib
-- Found the Arrow Python import library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow_python.dylib
-- Found the Arrow Python static library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow_python.a
-- Configuring done
-- Generating done
-- Build files have been written to: /private/var/folders/6m/ychzdp6s7l390zfm77myvxzr0000gn/T/pip-install-lv5qrivl/pyarrow_12e0171ed6fa433da71d5392c3e10c3a/build/temp.macosx-12.0-arm64-3.7
-- Finished cmake for pyarrow
-- Running cmake --build for pyarrow
cmake --build . --config release --
[ 6%] Compiling Cython CXX source for lib...
[ 6%] Built target lib_pyx
[ 13%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
/private/var/folders/6m/ychzdp6s7l390zfm77myvxzr0000gn/T/pip-install-lv5qrivl/pyarrow_12e0171ed6fa433da71d5392c3e10c3a/build/temp.macosx-12.0-arm64-3.7/lib.cpp:774:10: fatal error: 'arrow/python/config.h' file not found
#include "arrow/python/config.h"
^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/lib.dir/lib.cpp.o] Error 1
make[1]: *** [CMakeFiles/lib.dir/all] Error 2
make: *** [all] Error 2
error: command 'cmake' failed with exit status 2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
I couldn't find a proven solution for this problem, the only reference I did review is the following - https://github.com/apache/arrow/issues/2281, as mentioned, no proven solution there.
Versions I'm using are:
python 3.7.12
pip 22.0.4
On my machine, I have the following installed - apache-arrow, apache-arrow-glib, cmake.
I've also tried to install the package with the --no-use-pep517 flag as the following- pip install --no-use-pep517 pyarrow==0.17.1 - no success there.
I took a peek inside arrow/python/ and config.h is obviously missing, I also look inside arrow/ which does contains a config.h, however, it's not the config.h that was required.
Does anyone have an idea how to solve this issue?

pyarrow==0.17.1 predates M1 chips (May 2020 vs November 2020).
Any particular reason you wouldn't want to use pyarrow==7.0.0? Your brew installed arrow seems to be 7.0.0 too.

Libpython error while building YouCompleteMe

I am building YouCompleteMe plugin of vim, following this document. When I run make I get the following error.
Linking CXX shared library /home/sagar/.vim/bundle/YouCompleteMe/python/ycm_core.so
/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
What is this error?
I have installed pyenv to manage python versions. Is it causing problem?

Make the linker point to the .so (shared object) file and not the .a (static lib) file.
You can do this specifying the flag when running cmake:
cmake -G "Unix Makefiles" -DPYTHON_LIBRARY=/usr/local/lib/libpython2.7.so . ~/.vim/bundle/YouCompleteme/cpp
Do mind that even though you're using pyenv, YouCompleteMe build may point to an undesired
python build as they are not correctly auto-detected right now.
If you're having this problem, you should probably also specify the Python header files correctly:
cmake -G "Unix Makefiles" -DPYTHON_LIBRARY=/usr/local/lib/libpython2.7.so -DPYTHON_INCLUDE_DIR=/usr/local/include/python . ~/.vim/bundle/YouCompleteme/cpp
PS=(I'm assuming your headers are in that path, do check before)

Since some paths were different on my system from the accepted answer (both the CMake and the python lib ones) I'm posting an alternate solution for the above problem:
Make sure to have a shared library version of libpython2.7.so
$ locate libpython
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1
Either create a symlink to it from where CMake expects it to be
sudo ln -s "/usr/lib/x86_64-linux-gnu/libpython2.7.so.1" "/usr/lib/libpython2.7.so"
or alternatively, as written in YCM's build script code, you could add additional CMake options to ensure the .so library is properly found
export EXTRA_CMAKE_ARGS="-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so.1"

Mesos ExamplesTest.PythonFramework check fails on OSX

After successfully compiling Mesos 0.16.0, running the tests fails when checking the PythonFramework. All other tests pass successfully.
Steps I used for building:
./bootstrap
mkdir build
cd build
../configure CXX=g++4.7 CC=gcc-4.7
make
Then, when running the tests;
make check
The results look like this:
[...]
[ RUN ] ExamplesTest.PythonFramework
../../src/tests/script.cpp:78: Failure
Failed
python_framework_test.sh exited with status 1
[ FAILED ] ExamplesTest.PythonFramework (201 ms)
[...]
Environment:
OS X 10.9.1 (Mavericks)
Python 2.7.5 (default, Aug 25 2013, 00:04:04) [GCC 4.2.1 Compatible
Apple LLVM 5.0 (clang-500.0.68)] on darwin
gcc-4.7 (GCC) 4.7.3 Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There
is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
How do I build proper Mesos Python bindings that pass the tests within this environment?

Update:
My former answer has mostly become obsolete as of Mesos 0.17.0 due to the fact that this version does fully handle clang compilage (Yay!). So there is no need to compile it using gcc anymore - just go ahead and use Xcode's clang (Xcode commandline utilities).
In case you still get into trouble getting the Python bindings to work, please add a comment or new question here on StackOverflow or post into the Mesos Mailing list.
Mesos version 0.16.0 or lower:
How to fix the Python bindings of Mesos on OS X (10.9).
Install Python 2.7.3 via homebrew
Find out which versions are available
brew versions python
2.7.6 git checkout 3c86d2b /usr/local/Library/Formula/python.rb
2.7.5 git checkout a04b443 /usr/local/Library/Formula/python.rb
2.7.3 git checkout 865f763 /usr/local/Library/Formula/python.rb
2.7.4 git checkout 280581d /usr/local/Library/Formula/python.rb
[...]
Select Python 2.7.3
cd /usr/local/Library/Formula/
git checkout 865f763 /usr/local/Library/Formula/python.rb
brew install python
Make sure you do not force installing a universal build (32 + 64bit) as that would again cause the same issue explained below. The default is 64bit only and that is just fine.
Rebuild Mesos in connection with your custom Python installation
rm -rf build
rm -rf ~/.python-eggs
mkdir build
cd build
../configure CXX=g++-4.7 CC=gcc-4.7 PYTHON=/usr/local/bin/python
make
make check
You should now see a properly functioning test, hence a perfectly fine Mesos Python binding:
[ RUN ] ExamplesTest.PythonFramework
[ OK ] ExamplesTest.PythonFramework (1682 ms)
As asking users to install a custom Python version often is just wrong but in this case appears to be inevitable, let me draft an explanation of the issue. And maybe one of the readers knows a better workaround.
Manually executing that test using the verbosive output setting does help identifying the exact problem.
bin/mesos-tests.sh --gtest_filter="*.PythonFramework" --verbose
Traceback (most recent call last): File
"/Users/till/Documents/Development/github/mesos-master/build/../src/examples/python/test_framework.py",
line 23, in
import mesos File "build/bdist.macosx-10.9-intel/egg/mesos.py", line 26, in File
"build/bdist.macosx-10.9-intel/egg/_mesos.py", line 7, in
File "build/bdist.macosx-10.9-intel/egg/_mesos.py", line 6, in
bootstrap ImportError: dlopen(/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so,
2): Symbol not found: __ZNSoD0Ev Referenced from:
/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so
Expected in: flat namespace in
/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so
The important detail is the fact that the dynamic linkage of that native Python egg failed.
The reasoning is to be found within the distutils build step of this module when building mesos 0.16. The Python distutils derive their build-settings directly from python-config. As your Python was built using clang, the distutils will try to build your native egg using clang as well.
Issues:
Mesos' autoconf phase did not propagate the compiler settings into the distutils build phase. So even though Mesos itself is being built using gcc-4.7 in the above description, the egg is being built using clang. The result is a mishmash of libc++ and stdlibc++ which are not ABI compatible.
that part is being fixed right now, Mesos will use the same compiler in the distutils build phase as well (see MESOS-798 and MESOS-799). Chances are pretty high that when you read this answer, that particular issue was already taken care of.
The default OS X Python distutils do enforce building universal binaries (i386 + x86_64) using parameters that only the gcc-frontend of clang supports. There appears to be no workaround, hence all dynamically linked dependencies of that egg will have to be built for both architectures as well (this appears to be a leftover from OS X 10.6).
Mesos itself is linked statically into that egg, hence it does not have to be built as a universal binary for the egg to build and function on a 64bit platform. It will however fail to execute on a 32bit platform.
As long as Mesos does not support clang compilage (hence being linked against libc++), the only proper workaround seems to be to install a differently compiled Python. A quick and easy solution is using homebrew to install Python 2.7.3. Note: do not install Python 2.7.6 (the current default of homebrew) as that one has issues in connection with its autoconf developer macro (see MESOS-617)

cross compiling opencv 2.4.5 for arm board (beaglebone black) on ubuntu- Linking CXX shared library error at 92%

I'm trying to cross compile opencv for my beaglebone black. All seemed well until I added python support to the cmake flags, since im going to be needing the python wrappers for my project.
I have numpy and python-dev installed, and I'm currently running python 2.7.3.
I installed both with:
sudo apt-get install numpy
sudo apt-get install python-dev
This is what I'm getting:
Scanning dependencies of target opencv_python
[ 92%] Building CXX object modules/python/CMakeFiles/opencv_python.dir/src2/cv2.cpp.o
**Linking CXX shared library** ../../lib/cv2.so
/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpython2.7.so when searching for -lpython2.7
/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpython2.7.a when searching for -lpython2.7
/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status
make[2]: *** [lib/cv2.so] Error 1
make[1]: *** [modules/python/CMakeFiles/opencv_python.dir/all] Error 2
make: *** [all] Error 2
I'm using the available toolchain provided in the linux dist of opencv for arm devices. I really didn't change anything else aside from that.
Is there something wrong with my python dependencies?
cmake flags used:
cmake -DSOFTFP=ON  -DENABLE_NEON=ON –D BUILD_ZLIB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -DCMAKE_TOOLCHAIN_FILE=../opencv-2.4.5/platforms/linux/arm-gnueabi.toolchain.cmake ../
I appreciate any help

As your CMake reports, it finds only x86 variant of Python. So you'll need some embedded Linux distro like Buildroot or OpenEmbedded to avoid dealing with dependency hell, i.e. you'll need to cross-compile Python, numpy and all its dependencies and embedded Linux distro would take this job from you.
But there are more issues. One of the biggest problems lies in numpy dependencies like LAPACK, BLAS etc. These libraries are not very good suited for cross compilation.
OpenEmbedded seems to have recipes for OpenCV with Python support. In Buildroot it is still a work in progress.