We Keep Coding

Python3, Boost-Python and Cpp linker errors

So I'm about to throw my laptop through the window, out the window, and go and burn Apple HQ.
See Updates Below:
I can't get python3, boost-python and clang to work with each other. The error I'm stuck at is running:
clang++ <FLAGS/INCLUDES> -o hello.so hello.cpp
Invokes the response:
Undefined symbols for architecture x86_64:
"__Py_NoneStruct", referenced from:
boost::python::api::object::object() in hello-0c512e.o
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [hello] Error 1
Any help would be greatly, greatly appreciated. I think I have included everything necessary. Let me know if you require extra information.
The setup:
OSX 10.11.6 (El Capi-s##%)
Must use Xcode 7.3 (and appropriate CLT): Requirement of NVIDIA for CUDA programming (installed).
Must use Python3 (Homebrew installed)
brew install python3
which python -> /usr/bin/python
/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
I set up an alias for python3.5 (see below).
Using Boost-Python (Homebrew installed)
brew install boost
brew install boost-python --with-python3 --without-python
/usr/local/Cellar/boost-python/1.62.0/
Using LLVM 3.9.0 (Homebrew installed)
brew install llvm --universal
Now a few helpful things that you may ask for:
Clang++:
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
FLAGS and INCLUDES in makefile:
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
LDHEADERS = -I/usr/local/opt/llvm/include
LDLIBS = -L/usr/local/opt/llvm/lib
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.62.0/include/boost
BOOSTLIBS = -L/usr/local/Cellar/boost-python/1.62.0/lib
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
PYTHONLIBS = -L/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib
And finally, the code I am trying to compile:
hello.cpp
#include <boost/python.hpp>
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
using namespace boost::python;
BOOST_PYTHON_MODULE(hello)
{
class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
;
};

After much heartache and pain, I HAVE THE ANSWER!
To all those using OSX and homebrew, here's how you do it.
brew install python3 Python3 has UCS4 native (Unicode) which is an essential part of this process. If you need Python2, then make sure it is configured for UCS4 as it is natively UCS2.
brew install boost Install general boost first.
brew install boost-python --with-python3 --without-python This installs boost-python for Python3 WITHOUT Python2. You can change these options if you need Python2.
brew install llvm --universal Make sure you have llvm installed, this should have clang++ included in it which is the compiler we will use (not the Xcode one).
Create a makefile (see below) that includes the directories for all your python and boost headers/libraries, AND includes the libraries you want to use. (This is what tripped me up, I had the directories but did not specify which library in that directory the compiler should use).
My makefile:
# compiler flags:
# -g adds debugging information to the executable file
# -Wall turns on most, but not all, compiler warnings
COMPILER = /usr/local/Cellar/llvm/3.9.0/bin/clang++
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
# Python and BoostPython links.
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.62.0/include/boost
BOOSTLIBRARIES = -L/usr/local/Cellar/boost-python/1.62.0/lib/
# This is the boost library we want to use, there are also libraries for multithreading etc.
# All we do is find the file libboost_python3.a and link it by taking away the 'lib' and '.a'.
BOOSTLIB = -lboost_python3
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
PYTHONLIBRARIES = -L/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib
# Link to python3.5 library, the same as we did for boost.
PYTHONLIB = -lpython3.5
# Collect links.
LIBRARIES = $(BOOSTLIBRARIES) $(PYTHONLIBRARIES) $(PYTHONLIB) $(BOOSTLIB)
HEADERS = $(BOOSTHEADERS) $(PYTHONHEADERS)
# Build target.
TARGET = hello
# BEGIN MAKE
all: $(TARGET)
$(TARGET): $(TARGET).cpp
# Note that '-shared' creates a library that is accessible.
$(COMPILER) -shared $(LIBRARIES) $(HEADERS) $(TARGET).cpp -o $(TARGET).so
clean:
$(RM) $(TARGET)
Then all you need to do is run your makefile with all the inclusions and everything should be sweet :) I hope this helps someone and removes the pain I had of trying to get insertProfanity boost working.

Not enough points to comment so unfortunately I'll have to post as an answer since not too long ago I was going through this same linker error stuff with Boost.
My guess is that this appears to be a Python linker problem, as opposed to a Boost linker problem. I see that you've added the Python includes to your CXX_INCLUDE_PATH, but what about your Python library?
If the path you've mentioned above (that long path in your CXX_INCLUDE_PATH variable) is where your includes are then [that_long_path]/Versions/3.5/lib should be where your Python libraries are. Include this library when you run your build command by using
clang++ -g -v -std=c++11 -stdlib=libc++ -L/[that_long_path]/Versions/3.5/lib -lpython3.5m hello.cpp -o hello.so
The -L flag tells the compiler to include that directory, while the -l flag tells the compiler to include the following library. Alternatively you can just append this lib path to your current CXX_INCLUDE PATH, which would then look like this:
export CXX_INCLUDE_PATH="/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m:/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib"
Also make sure you are including any relevant boost include paths and library paths when you run the build command to avoid any more linker errors (if boost is not installed in a default location like usr/local/[some_place].
Then run this new build command (may be able to do without the -std and -stdlib flags:
clang++ -g -v -std=c++11 -stdlib=lib++ -lpython3.5m -o hello.so hello.cpp
In summary:
1) Include your python header files (which you seem to have done)
2) Include your python library file(s) (which you seem to be missing)
3) Include any relevant boost libraries and the boost include directory (which is a bit out of the scope of this question but still worth noting)
Hope this helps.

Building graph-tool on OSX with Python 3.4

I tried to install graph-tool on Mac OSX 10.10 using homebrew. The brew build process works fine, but when I try to import graph-tool I get the error described in this question.
Another problem with homebrew is that I always builds graph-tool for python2.7 and it installs the packages in the Python 2.7 sit-packages folder. But I want to use it with Python 3.4. These are the reasons why I tried to build graph-tool from source.
The ./configure command automatically uses Python 2.7, too. So I passed it the desired Python version with ./configure PYTHON=python3.4
It then detects the correct version as well as the related paths but crash with the following error:
configure: error:
Could not link test program to Python. Maybe the
main Python library has been installed in some non-standard library
path. If so, pass it to configure, via the LDFLAGS environment
variable.
Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
======================================================================
ERROR!
You probably have to install the development version of the
Python package for your distribution. The exact name of this package varies
among them.
======================================================================
The error occurs with and without PYTHON variable set.
From the output of ./configure I can see that everything works fine except for the last line, which says:
checking consistency of all components of python development
environment... no
Whats does the above line mean and how do I properly install graph-tool on my maschine?

The error message is explaining exactly what needs to be done. Since python was installed in a non-standard path, you need to pass the flag LDFLAGS="-L/usr/non-standard-path/python/lib" pointing to the directory where the python libraries are located. This is most likely "/usr/local/lib", if you are using homebrew.

I was getting this error when I was trying to install graph-tool using outdated an autoconf / automake / pkg-config combination (installed using yum in CentOS 5.10). Installing those packages from source fixed the problem... although I'm not sure how this related to my python installation....

It worked for me by passing the variable PYTHON_EXTRA_LDFLAGS="-Wl,-stack_size,1000000 -F/usr/local/Cellar/python3/3.6.3/Frameworks -framework CoreFoundation".
In your case, it would be the path to the homebrew installation of python3.4.
The way I found out is that in the config.log, the error message shows the following:
configure:19023: checking python extra libraries
configure:19030: result: -ldl -framework CoreFoundation
configure:19037: checking python extra linking flags
configure:19044: result: -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python
configure:19051: checking consistency of all components of python development environment
configure:19079: gcc -o conftest -g -O2 -DNDEBUG -I/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/include/python3.6m -F/usr/local/Cellar/python3/3.6.3/Frameworks/ -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python conftest.c -L/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib -lpython3.6m -ldl -framework CoreFoundation -ldl -framework CoreFoundation >&5
clang: error: no such file or directory: 'Python.framework/Versions/3.6/Python'
The error seems to be path 'Python.framework/Versions/3.6/Python', that in a homebrew installation does not exist. I search for the same path in the config.log and I found this line:
PYTHON_EXTRA_LDFLAGS="-Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python"
So, the solution for me was to pass this variable with the right path.

How to install Boost.Python on Windows 7 in order to install a python package?

I want to install the pyvlfeat package. It requires Boost.Python.
When I run the command
python.exe setup.py build
I receive the following message:
C:\Users\alex\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Users\A
lexkow\AppData\Roaming\Python\Python27\site-packages\numpy\core\include -Ivlfeat
/ -IC:\Users\alex\Anaconda\include -IC:\Users\alex\Anaconda\PC -c vlfeat/m
ser/vl_erfill.cpp -o build\temp.win-amd64-2.7\Release\vlfeat\mser\vl_erfill.o -m
sse2 -O2 -fPIC -w
In file included from vlfeat/mser/vl_erfill.cpp:7:0:
vlfeat/mser/../py_vlfeat.h:18:28: fatal error: boost/python.hpp: No such file or
directory
Which tells me Boost.Python is not installed correctly on my computer, or that I don't launch the python install command correctly.
The package INSTALL instructions are :
Building the Module on a Unix System --
The C++ wrappers require Boost.Python to be installed:
$ sudo apt-get install boost-python1.35-dev
pyvlfeat uses distutils, so to build the library:
$ python setup.py build
As I am on Windows I can't sudo apt-get, so I downloaded boost 1.57.0 and extracted it into
C:\Program Files\boost\boost_1_57_0
It did not change the result. And now I don't know what I should do:
When I read the documentation
The section 3: "No-Install Quick Start" explains how to build an extension module called extending and test it by running a Python script called test_extending.py. I don't think that is what I want to achieve and it seems outdated because it talks about the bjam build driver.
The section 4: "Installing Boost.Python on your system" looks more interresting, but it says the information is in the Getting Started Guide, and it is not.
How to install Boost.Python on Windows 7 in order to install a python package ? ?

I am looking at the setup.py file. It looks like you can set BOOST_PATH at the top. You should set that to the correct path to the Boost you downloaded, I think.

How do I package for distribution a python module that uses a shared library?

I'm writing some bindings for a C library and am not sure how to configure all this for distribution so it is possible to pip install my package.
Let's say I have the following files:
library.c
library.h
wrapper.py
In order for my wrapper library to work it is necessary to:
compile library.c and create a shared library
run ctypesgen on library.h to generate the ctypes code
Here are the commands:
gcc -Wall -fPIC -c library.c
gcc -shared -Wl,-soname,liblibrary.so.1 -o liblibrary.so.1.0 library.o
ctypesgen.py library.h -L ./ -l library -o _library.py
Running setup.py will also depend on the user having installed ctypesgen.
I have no idea how to get this all set up so that someone interested in the library can simply pip install library and have all this happen automagically. Anyone able to help?

The Extensions capability of setuptools/distutils is what you need.
Documentation has more information, in short an example setup.py to do the above would look like the below
from setuptools import setup, find_packages, Extension
extensions = [Extension("my_package.ext_library",
["src/library.c"],
depends=["src/library.h"],
include_dirs=["src"],
),
]
setup(<..>,
ext_modules=extensions,
)
The .so is generated automatically by setup.py when the module is built. If it needs to link to other libraries can supply a libraries argument list to the extension. See docs(1) for more info.
Since this is built in functionality of setuptools, it works fine with pip and can be distributed (as source code only) on pypi. All source files referenced by the Extension must be present in the distributed pypi archive.
If you want to build distributable binary wheels including native code see manylinux.

Libpython error while building YouCompleteMe

I am building YouCompleteMe plugin of vim, following this document. When I run make I get the following error.
Linking CXX shared library /home/sagar/.vim/bundle/YouCompleteMe/python/ycm_core.so
/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
What is this error?
I have installed pyenv to manage python versions. Is it causing problem?

Make the linker point to the .so (shared object) file and not the .a (static lib) file.
You can do this specifying the flag when running cmake:
cmake -G "Unix Makefiles" -DPYTHON_LIBRARY=/usr/local/lib/libpython2.7.so . ~/.vim/bundle/YouCompleteme/cpp
Do mind that even though you're using pyenv, YouCompleteMe build may point to an undesired
python build as they are not correctly auto-detected right now.
If you're having this problem, you should probably also specify the Python header files correctly:
cmake -G "Unix Makefiles" -DPYTHON_LIBRARY=/usr/local/lib/libpython2.7.so -DPYTHON_INCLUDE_DIR=/usr/local/include/python . ~/.vim/bundle/YouCompleteme/cpp
PS=(I'm assuming your headers are in that path, do check before)

Since some paths were different on my system from the accepted answer (both the CMake and the python lib ones) I'm posting an alternate solution for the above problem:
Make sure to have a shared library version of libpython2.7.so
$ locate libpython
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1
Either create a symlink to it from where CMake expects it to be
sudo ln -s "/usr/lib/x86_64-linux-gnu/libpython2.7.so.1" "/usr/lib/libpython2.7.so"
or alternatively, as written in YCM's build script code, you could add additional CMake options to ensure the .so library is properly found
export EXTRA_CMAKE_ARGS="-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so.1"