So I'm about to throw my laptop through the window, out the window, and go and burn Apple HQ.
See Updates Below:
I can't get python3, boost-python and clang to work with each other. The error I'm stuck at is running:
clang++ <FLAGS/INCLUDES> -o hello.so hello.cpp
Invokes the response:
Undefined symbols for architecture x86_64:
"__Py_NoneStruct", referenced from:
boost::python::api::object::object() in hello-0c512e.o
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [hello] Error 1
Any help would be greatly, greatly appreciated. I think I have included everything necessary. Let me know if you require extra information.
The setup:
OSX 10.11.6 (El Capi-s##%)
Must use Xcode 7.3 (and appropriate CLT): Requirement of NVIDIA for CUDA programming (installed).
Must use Python3 (Homebrew installed)
brew install python3
which python -> /usr/bin/python
/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
I set up an alias for python3.5 (see below).
Using Boost-Python (Homebrew installed)
brew install boost
brew install boost-python --with-python3 --without-python
/usr/local/Cellar/boost-python/1.62.0/
Using LLVM 3.9.0 (Homebrew installed)
brew install llvm --universal
Now a few helpful things that you may ask for:
Clang++:
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
FLAGS and INCLUDES in makefile:
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
LDHEADERS = -I/usr/local/opt/llvm/include
LDLIBS = -L/usr/local/opt/llvm/lib
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.62.0/include/boost
BOOSTLIBS = -L/usr/local/Cellar/boost-python/1.62.0/lib
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
PYTHONLIBS = -L/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib
And finally, the code I am trying to compile:
hello.cpp
#include <boost/python.hpp>
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
using namespace boost::python;
BOOST_PYTHON_MODULE(hello)
{
class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
;
};
After much heartache and pain, I HAVE THE ANSWER!
To all those using OSX and homebrew, here's how you do it.
brew install python3 Python3 has UCS4 native (Unicode) which is an essential part of this process. If you need Python2, then make sure it is configured for UCS4 as it is natively UCS2.
brew install boost Install general boost first.
brew install boost-python --with-python3 --without-python This installs boost-python for Python3 WITHOUT Python2. You can change these options if you need Python2.
brew install llvm --universal Make sure you have llvm installed, this should have clang++ included in it which is the compiler we will use (not the Xcode one).
Create a makefile (see below) that includes the directories for all your python and boost headers/libraries, AND includes the libraries you want to use. (This is what tripped me up, I had the directories but did not specify which library in that directory the compiler should use).
My makefile:
# compiler flags:
# -g adds debugging information to the executable file
# -Wall turns on most, but not all, compiler warnings
COMPILER = /usr/local/Cellar/llvm/3.9.0/bin/clang++
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
# Python and BoostPython links.
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.62.0/include/boost
BOOSTLIBRARIES = -L/usr/local/Cellar/boost-python/1.62.0/lib/
# This is the boost library we want to use, there are also libraries for multithreading etc.
# All we do is find the file libboost_python3.a and link it by taking away the 'lib' and '.a'.
BOOSTLIB = -lboost_python3
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
PYTHONLIBRARIES = -L/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib
# Link to python3.5 library, the same as we did for boost.
PYTHONLIB = -lpython3.5
# Collect links.
LIBRARIES = $(BOOSTLIBRARIES) $(PYTHONLIBRARIES) $(PYTHONLIB) $(BOOSTLIB)
HEADERS = $(BOOSTHEADERS) $(PYTHONHEADERS)
# Build target.
TARGET = hello
# BEGIN MAKE
all: $(TARGET)
$(TARGET): $(TARGET).cpp
# Note that '-shared' creates a library that is accessible.
$(COMPILER) -shared $(LIBRARIES) $(HEADERS) $(TARGET).cpp -o $(TARGET).so
clean:
$(RM) $(TARGET)
Then all you need to do is run your makefile with all the inclusions and everything should be sweet :) I hope this helps someone and removes the pain I had of trying to get insertProfanity boost working.
Not enough points to comment so unfortunately I'll have to post as an answer since not too long ago I was going through this same linker error stuff with Boost.
My guess is that this appears to be a Python linker problem, as opposed to a Boost linker problem. I see that you've added the Python includes to your CXX_INCLUDE_PATH, but what about your Python library?
If the path you've mentioned above (that long path in your CXX_INCLUDE_PATH variable) is where your includes are then [that_long_path]/Versions/3.5/lib should be where your Python libraries are. Include this library when you run your build command by using
clang++ -g -v -std=c++11 -stdlib=libc++ -L/[that_long_path]/Versions/3.5/lib -lpython3.5m hello.cpp -o hello.so
The -L flag tells the compiler to include that directory, while the -l flag tells the compiler to include the following library. Alternatively you can just append this lib path to your current CXX_INCLUDE PATH, which would then look like this:
export CXX_INCLUDE_PATH="/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m:/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib"
Also make sure you are including any relevant boost include paths and library paths when you run the build command to avoid any more linker errors (if boost is not installed in a default location like usr/local/[some_place].
Then run this new build command (may be able to do without the -std and -stdlib flags:
clang++ -g -v -std=c++11 -stdlib=lib++ -lpython3.5m -o hello.so hello.cpp
In summary:
1) Include your python header files (which you seem to have done)
2) Include your python library file(s) (which you seem to be missing)
3) Include any relevant boost libraries and the boost include directory (which is a bit out of the scope of this question but still worth noting)
Hope this helps.
Related
Update
I'm not going to add this as an answer, since I still haven't technically solved the problem. But since I've now spent 2.5 days trying to get things to work with boost-python3, I've lost the will to live with it.
I've just come across pybind11 (how my previous lengthy searches for python binding tools didn't turn it up, I don't know) and am using that. 2.5 days of misery compares to <20 minutes installing and building their cmake example... and all the specific-python-version-dependency-hell is gone.
It's syntactically similar to boost-python but much easier to manage, quicker, is header-only and is more feature rich.
Yay!
Original question
I'm using boost::python to bind a class in python 3.7.2.
The class import successfully but instantiating it gives the following error:
<my-terminal>$ python
Python 3.7.2 (default, Feb 14 2019, 17:36:47)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import classes
>>> t = classes.World()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() should return None, not 'NoneType'
>>>
Here is classes.cpp:
#include <boost/python.hpp>
#include <boost/python/list.hpp>
#include <boost/python/extract.hpp>
#include <string>
#include <sstream>
#include <vector>
struct World
{
void set(std::string msg) { mMsg = msg; }
void many(boost::python::list msgs) {
long l = len(msgs);
std::stringstream ss;
for (long i = 0; i<l; ++i) {
if (i>0) ss << ", ";
std::string s = boost::python::extract<std::string>(msgs[i]);
ss << s;
}
mMsg = ss.str();
}
std::string greet() { return mMsg; }
std::string mMsg;
};
using namespace boost::python;
BOOST_PYTHON_MODULE(classes)
{
class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
.def("many", &World::many)
;
};
Hypothesis
This question, almost identical was solved because of a python 2/3 issue (linking against python 3 instead of python 2 libraries). So I suspected a library linking issue.
Checking the hypothesis
I can't get bjam to work, and wouldn't be able to switch all our build systems over for one module anyway... so am building with cmake, which compiles successfully to classes.so with output as follows, suggesting I'm finding all the correct includes, libraries and executables:
-- Found PythonInterp: /Users/me/.pyenv/versions/boost37/bin/python3 (found suitable version "3.7.2", minimum required is "3")
PYTHON_VERSION_SUFFIX
-- Boost version: 1.68.0
-- Found the following Boost libraries:
-- python37
-- Found PythonLibs: /usr/local/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib (found suitable version "3.7.2", minimum required is "3")
-- PYTHON_LIBRARIES = /usr/local/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib
-- PYTHON_EXECUTABLE = /Users/thc29/.pyenv/versions/boost37/bin/python3
-- PYTHON_INCLUDE_DIRS = /usr/local/Frameworks/Python.framework/Versions/3.7/include/python3.7m
-- Boost_LIBRARIES = /usr/local/lib/libboost_python37-mt.dylib
Boost-python3 library directory contents:
ls /usr/local/Cellar/boost-python3/1.68.0/lib
libboost_numpy37-mt.a libboost_numpy37.dylib libboost_python37.a
libboost_numpy37-mt.dylib libboost_python37-mt.a libboost_python37.dylib
libboost_numpy37.a libboost_python37-mt.dylib
I used brew install boost, and brew install boost-python3 --build-from-source with my python 3.7 virtualenv activated, to ensure boost-python3 is linked against the correct version of python.
Checking libraries...
otool -L classes.so gives:
classes.so:
/usr/l/opt/boost-python3/lib/libboost_python37-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python (compatibility version 3.7.0, current version 3.7.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
otool -L /usr/local/opt/boost-python3/lib/libboost_python37-mt.dylib gives:
/usr/local/opt/boost-python3/lib/libboost_python37-mt.dylib:
/usr/local/opt/boost-python3/lib/libboost_python37-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
In the related question, that showed their problem. But here it appears fine!
No progress yet...
After the painful process of getting this all compiling properly and checking the linking, I can't spot any flaws. Is this a different problem? Or is there a linking issue that I haven't spotted?
Thanks for any help!
Adding an answer here for those using the Anaconda or Conda-Forge Distribution:
The python interpreter statically links in the libpythonXY library. Which is why it makes the python binary different compared to other distributions.
The fix for the problem reported by OP is to use:
-undefined dynamic_lookup
Instead of:
-lpythonXY
You are creating a Python C/C++ extension, and not embedding the python interpreter. So you shouldn't be linking to the python library. Pybind11 handles this correctly.
See the following for more information:
https://gitlab.kitware.com/cmake/cmake/issues/18100
https://github.com/ContinuumIO/anaconda-issues/issues/9078
One a side note, python 3.8 has added an additional flag: --embed and only then it adds -lpythonXY in the output:
$ python3.8-config --libs
-ldl -framework CoreFoundation
$ python3.8-config --libs --embed
-lpython3.8 -ldl -framework CoreFoundation
I am following a similar example and I adopt the Makefile from here. I have installed python 3.7.4 and boost-python via brew on macOS. To fix the NoneType issue, I follow the procedure below:
1. Check the Python Path
To check the python path, use
which python
If the output does not look like the following one (brew's python installation path)
/usr/local/opt/python/libexec/bin/python
set the PATH variable as
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Check if the Python path looks like the one above again.
2. Check the Compilation Flag
Below is the adopted Makefile. Note the LIB variable. If the boost-python flag is -lboost_python, change it to -lboost_python37.
CPP = clang++
PYLIBPATH = $(shell python-config --exec-prefix)/lib
# LIB = -L$(PYLIBPATH) $(shell python-config --libs) -lboost_python
LIB = -L$(PYLIBPATH) $(shell python-config --libs) -lboost_python37
OPTS = $(shell python-config --include) -O2
default: hello.so
hello.so: hello.o
$(CPP) $(LIB) -Wl,-rpath,$(PYLIBPATH) -shared $< -o $#
hello.o: hello.cpp Makefile
$(CPP) $(OPTS) -c $< -o $#
clean:
rm -rf *.so *.o
.PHONY: default clean
Recompile the C++ code and run the python script. The NoneType issue should disappear.
Hope this helps.
Note
If you are using anaconda and want to restore the PATH variable after the above changes, try
export PATH="~/anaconda3/bin:$PATH"
Your anaconda's path may be different.
Credit
1. George's comment in How do I use brew installed Python as the default Python?
2. leiyc's comment in ld: library not found for -lboost_python on MacOS
Following up on Nehal's answer for Anaconda based builds. Rereading the FindPython documentation for cmake shows the Python::Module target was added in cmake 3.15 for creating modules. That means the CMakeLists.txt should be:
set(Python3_FIND_VIRTUALENV FIRST)
find_package(Python3 REQUIRED Development)
find_package(Boost REQUIRED python3)
add_library(classes MODULE classes.cpp)
target_link_libraries(classes PRIVATE Python3::Module Boost::python3)
Apparently, the Python3::Python is for embedding Python in another application.
It seems that there're some conflicts on CUDA 9.0 and GCC 6.x, (discussed here). So I decide to use gcc 5.5 to setup python package.
According to the answer https://stackoverflow.com/a/25595274/5634636, and https://stackoverflow.com/a/16737696/5634636, I tried to run setup.py as follow:
CC=gcc-5 CXX=g++-5 python setup.py install --user
which raise the error (I only paste part of it because it's too long):
/usr/include/c++/6/tuple: In instantiation of ‘static constexpr bool
std::_TC<, _Elements>::_MoveConstructibleTuple() [with
_UElements = {std::tuple};
It seems that python is still using g++ 6 (/usr/include/c++/6/tuple) library on compiling. How to avoid using g++ 6?
It seems that there are two places where gcc is used. One is python setup itself, the other is nvcc. The environment variable CC and CXX only specific the gcc version for python, but it didn't change the gcc version for nvcc.
To change the gcc version cuda used by default, do as follows:
sudo unlink /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/gcc-5 /usr/local/cuda/bin/gcc
There /usr/bin/gcc-5 can be any path to the gcc you want to use.
I'm currently trying to compile vim on a Fedora 20 machine with a particularly exotic setup:
No root access
python2 and python3 manually compiled and installed in ~/.local correctly working (after exporting the PATH and LD_LIBRARY_PATH).
zsh as shell
gcc version 4.8.3 20140911 (Red Hat 4.8.3-7) (GCC)
If I configure the compiling process as:
./configure --with-features=huge --enable-pythoninterp --enable-python3interp --prefix=$HOME/.local
and then make && make install, vim is correctly compiled with +python/dyn +python3/dyn.
vim --version | grep python
+cryptv +linebreak +python/dyn +vreplace
+cscope +lispindent +python3/dyn +wildignore
but, inside vim, :echo has('python') returns a 0 (and the MatchTagAlways complains about that in facts...).
So I told to myself, let's try to force the statically linked installation:
export LDFLAGS=-static
./configure --with-features=huge --enable-pythoninterp --enable-python3interp --prefix=$HOME/.local
ends just a little bit after the configuring command:
configure: creating cache auto/config.cache
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/students/rm_16_17/dibattista/build/vim/src':
configure: error: C compiler cannot create executables
See `config.log' for more details
Here the full configure.log. The relevant line should be:
configure:3027: gcc -static conftest.c >&5
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
that I cannot really decrypt. It seems that gcc does not has the -static flag. Is that the issue?
At the end what I was missing was a flag when compiling Python to compile it as a shared library --enable-shared.
As a reminder: always read all the INSTALL instructions :)
I want to debug ARM application on devices like Android machine, i prefer to use gdb(ARM version) than gdb with gdbserver to debug, because there is a dashboard , a visual interface for GDB in Python.
It must cooperation with gdb(ARM version) on devices,so i need to cross compiling a ARM version of gdb with python, the command used shows below:
./configure --host=arm-linux-gnueabi --target=arm-linux-gnueabi --with-python=/usr/bin
But finally a error message appeared:
configure:8096: checking whether to use python
configure:8098: result: /usr/bin/
configure:8316: checking for python2.7
configure:8334: arm-linux-gnueabi-gcc -o conftest -g -O2 -I/usr/include/python2.7 -I/usr/include/python2.7 conftest.c -ldl -ltermcap -lm -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions >&5
In file included from /usr/include/python2.7/Python.h:8:0,
from conftest.c:50:
/usr/include/python2.7/pyconfig.h:15:52: fatal error: arm-linux-gnueabi/python2.7/pyconfig.h: No such file or directory
compilation terminated.
Then I find the line 15 in /usr/include/python2.7/pyconfig.h, as below:
# elif defined(__ARM_EABI__) && !defined(__ARM_PCS_VFP)
#include <arm-linux-gnueabi/python2.7/pyconfig.h>
Here is the point, I only have x86_64-linux-gnu/python2.7/pyconfig.h in /usr/include, how can I get the arm-linux-gnueabi/python2.7/pyconfig.h? I already apt-get install python2.7-dev.
I ran into this problem when attempting to cross-compile a python wrapper module built with SWIG, but it looks to me like it will happen to anyone cross compiling python-linked C code on a Debian system.
Apparently the Debian python-dev packages are not set up with the header files to facilitate a cross compile, but it is possible to go get them manually. I'm not sure if this is a python bug or a Debian package bug, and I haven't researched if it applies to other distros.
pyconfig.h sets preprocessor defines to tell the python source code about platform-depended things like endianness and data type sizes, so the proper pyconfig.h is definitely needed to correctly compile python source. Fortunately, the pyconfig.h file should be the only file you need to grab separately, and it is available from the Debian python-dev package for your target platform.
you can download the package file for armeabi or any other architecture from https://packages.debian.org/jessie/libpython2.7-dev and extract the include directory yourself, or you can use the following commands to download the package and copy the proper files for armeabi to /usr/local/include
wget http://security.debian.org/debian-security/pool/updates/main/p/python2.7/libpython2.7-dev_2.7.9-2+deb8u2_armel.deb
dpkg -x libpython2.7-dev_2.7.9-2_armel.deb libpython2.7-dev_2.7.9-2_armel_extracted
sudo cp -r libpython2.7-dev_2.7.9-2_armel_extracted/usr/include/arm-linux-gnueabi/ /usr/local/include/
rm -r libpython2.7-dev_2.7.9-2_armel*
Note that on some cross compilers you will have to add -I /usr/local/include to the compiler options if it does not search this location by default, but to me this is better than modifying /usr/include and risking your changes being wiped out by the OS
I want to use python consistently in a variety of environments. One of those environments is cygwin. One of the components I want to use is gevent (http://www.gevent.org/intro.html). Under cygwin, I have python 2.7 running (built locally, with the one line change described here, which is required for it to build: http://www.gossamer-threads.com/lists/python/python/976956).
gevent requires libevent (http://libevent.org/).
libevent seems to build just fine under cygwin (./configure && make && make install).
However, when building gevent (pip install gevent), it fails because libevent built static libraries (such as /usr/local/lib/libevent.a) and the gevent build wants shared libraries. Thus:
gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.13-i686-2.7/gevent/core.o -L/usr/local/lib/python2.7/config -levent -lpython2.7 -o build/lib.cygwin-1.7.13-i686-2.7/gevent/core.dll
/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -levent
Meanwhile, if I try to get gevent to link statically (CFLAGS='-static' pip install gevent), -levent still fails, along with numerous warnings about how dereferencing type-punned pointer will break strict-aliasing rules and some additional ld failures:
gevent/core.c:21835: warning: dereferencing type-punned pointer will break strict-aliasing rules
....
gevent/core.c:21836: warning: dereferencing type-punned pointer will break strict-aliasing rules
gcc -shared -Wl,--enable-auto-image-base -static build/temp.cygwin-1.7.13-i686-2.7/gevent/core.o -L/usr/local/lib/python2.7/config -levent -lpython2.7 -o build/lib.cygwin-1.7.13-i686-2.7/gevent/core.dll
/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -levent
/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython2.7
... so...
I think I need to tell libevent to build .dll instead of .a, but libevent's Makefile does not actually have a .a target, and it's not clear to me how the abstractions being used would have to change to accomplish this.
So, taking a step back: how do I install gevent under python 2.7 under cygwin?
Can you try a beta release from http://code.google.com/p/gevent/downloads/list ?
It does not require any external dependencies.