Sorry if I'm duplicating a question, but I just cannot find the solution to what I'm looking for anywhere on the internet, yet I believe that this is a very simple problem.
I'm trying to extend python with some custom C++ libraries, and building my C++ libraries with CMake. I'm following the instructions on https://docs.python.org/2/extending/extending.html, but it's not compiling correctly.
When I try to build it, I get these messages:
"C:\Program Files (x86)\JetBrains\CLion 140.2310.6\bin\cmake\bin\cmake.exe" --build C:\Users\pkim2\.clion10\system\cmake\generated\76c451cd\76c451cd\Debug --target parsers -- -j 8
Linking CXX executable parsers.exe
CMakeFiles\parsers.dir/objects.a(main.cpp.obj): In function `spam_system':
C:/code/ground-trac/ground/launch/trunk/Software Support/Data Analysis Scripts/data_review_automation/parsers/main.cpp:9: undefined reference to `_imp__PyArg_ParseTuple'
C:/code/ground-trac/ground/launch/trunk/Software Support/Data Analysis Scripts/data_review_automation/parsers/main.cpp:12: undefined reference to `_imp__Py_BuildValue'
c:/mingw/bin/../lib/gcc/mingw32/4.8.1/../../../libmingw32.a(main.o):(.text.startup+0xa7): undefined reference to `WinMain#16'
collect2.exe: error: ld returned 1 exit status
CMakeFiles\parsers.dir\build.make:87: recipe for target 'parsers.exe' failed
CMakeFiles\Makefile2:59: recipe for target 'CMakeFiles/parsers.dir/all' failed
CMakeFiles\Makefile2:71: recipe for target 'CMakeFiles/parsers.dir/rule' failed
mingw32-make.exe[3]: *** [parsers.exe] Error 1
mingw32-make.exe[2]: *** [CMakeFiles/parsers.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles/parsers.dir/rule] Error 2
mingw32-make.exe: *** [parsers] Error 2
Makefile:109: recipe for target 'parsers' failed
Based on this, I suspect that this is a problem with the way I'm linking things in the CMakeLists.txt file, but I have no idea how to do it properly. This is what my CMakeLists.txt looks like right now:
cmake_minimum_required(VERSION 2.8.4)
project(parsers)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
set(SOURCE_FILES main.cpp)
include_directories(C:\\Python27\\include)
link_directories(C:\\Python27\\)
target_link_libraries(python2.7)
add_executable(parsers ${SOURCE_FILES})
How in the world do I get this thing to compile correctly? I am running Windows 7 64-bit, and using CLion as my IDE.
Your first problem is that you are using target_link_libraries wrong: you should pass it the target to which to add a link and then the library you want to link in:
target_link_libraries(parsers python2.7)
Your second problem is that you are building an executable, instead of a shared library. If you want to make your extension accessible from python it needs to be a library.
add_library(parsers SHARED ${SOURCE_FILES})
But now comes the good news: your life becomes much simpler (and more portable) if you use the built in CMake module FindPythonLibs.cmake. To build a python module you would only need to do the following:
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
find_package(PythonLibs REQUIRED)
add_library(parsers SHARED ${SOURCE_FILES})
include_directories(${PYTHON_INCLUDE_DIRS})
target_link_libraries(parsers ${PYTHON_LIBRARIES})
if you use windows, try this CMakeLists.txt
cmake_minimum_required(VERSION 3.8)
project(CSample)
set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -g -ftest-coverage -fprofile-arcs" )
link_directories(D:/Programs/mingw/mingw64/lib/)
include_directories(D:/Programs/Python/Python37/include/)
link_libraries(D:/Programs/Python/Python37/libs/python37.lib)
# Python
add_executable(CSample main.cpp)
You are using target_link_libraries() wrong. Check the docs; you probably want something like:
add_executable(parsers ${SOURCE_FILES})
target_link_libraries(parsers python2.7)
Note that the output from CMake should already tell you that something is wrong. On my machine:
CMake Error at CMakeLists.txt:8 (target_link_libraries):
Cannot specify link libraries for target "python2.7" which is not built by
this project.
Related
I want to build pyobjc-7.3, because it has fix for send2trash.
Classic building on BigSur 20.5.0 is strait forward.
cd pyobjc-7.3/pyobjc-framework-Cocoa
python3 setup.py build
though once I run same build inside nix-shell magic happens.
nix-shell -p pkgs.python39Packages.setuptools
python3 setup.py build
clang-7: error: argument unused during compilation:
'-fno-strict-overflow' [-Werror,-Wunused-command-lin\ e-argument]
ok. no big deal. let's disable warning.
CFLAGS="-Wno-unused-argument" python3 setup.py build
what? now clang is like a blind kitten.
Modules/pyobjc-api.h:19:10: fatal error: 'objc/objc.h' file not found
#include <objc/objc.h>
-isysroot option and -I has no effect.
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk
I noticed lots additions to -I flag in clang such as:
-iwithprefix /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include
it helps clang to find objc header file, though this is not the end of the story.
Modules/pyobjc-api.h:21:9: fatal error: 'Foundation/Foundation.h' file
not found
how come?!
oh there is another header files of special kind - frameworks. Wheel reinvention...
clang, take another argument
-iframeworkwithsysroot /System/Library/Frameworks
Here I get tons of type errors and I run out of ideas what could I try next:
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSString.h:138:1:
error:
function cannot return function type 'NSComparisonResult' (aka 'int (int)')
(NSComparisonResult)compare:(NSString *)string options:(NSStringCompareOptions)mask range:(NSR...
After days of trying I've found solution.
There are a few bugs are causing the problems:
First is nix provides older (10.12) sdk while setup.py thinks is 10.15.
This enables CPP sections for unsupported SDK API therefore type errors.
Following hack makes pyobjc to think that SDK is older that it is.
with pkgs;
with pkgs.lib;
with pkgs.python39Packages;
let
pyobjc-core = buildPythonPackage rec {
pname = "pyobjc-core";
version = "7.3";
name = "${pname}-${version}";
src = pkgs.python39Packages.fetchPypi {
pname = "pyobjc-core";
inherit version;
sha256 = "0x3msrzvcszlmladdpl64s48l52fwk4xlnnri8daq2mliggsx0ah";
};
preBuild=''
export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX10.12.sdk"
Second problem is with header discovery and overstrict lint from python nix
CFLAGS = "-iwithsysroot /usr/include -Wno-unused-argument";
Third problem big sur linkder is dynamic and ffi libray is not found.
Providing through nix derivation
buildInputs = [ pkgs.libffi ];
Forth problem is tests are broken
doCheck = false;
I want to be able to call my C++ code as a python package. To do this I am using pybind11 with CMakelists (following this example https://github.com/pybind/cmake_example). My problem is that I have to include GSL libraries in the compilation of the code, and these need an explicit linker -lgsl .
If I were just to compile and run the C++ without wrapping it with python, the following Cmakelists.txt file does the job
cmake_minimum_required(VERSION 3.0)
set(CMAKE_BUILD_TYPE Debug)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14")
project(myProject)
add_executable(
myexecutable
main.cpp
function1.cpp
)
find_package(GSL REQUIRED)
target_link_libraries(myexecutable GSL::gsl GSL::gslcblas)
but when using pybind11 the template I found doesn't allow the add_executable therefore target_link_libraries doesn't work.
I have trie this
project(myProject)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED YES) # See below (1)
# Set source directory
set(SOURCE_DIR "project")
# Tell CMake that headers are also in SOURCE_DIR
include_directories(${SOURCE_DIR})
set(SOURCES "${SOURCE_DIR}/functions.cpp")
# Generate Python module
add_subdirectory(lib/pybind11)
pybind11_add_module(namr ${SOURCES} "${SOURCE_DIR}/bindings.cpp")
FIND_PACKAGE(GSL REQUIRED)
target_link_libraries(GSL::gsl GSL::gslcblas)
but this produces errors in the building.
Any idea ?
Function pybind11_add_module creates a library target, which can be used for link added module with other libraries:
pybind11_add_module(namr ${SOURCES} "${SOURCE_DIR}/bindings.cpp")
target_link_libraries(namr PUBLIC GSL::gsl GSL::gslcblas)
This is explicitely stated in documentation:
This function behaves very much like CMake’s builtin add_library (in fact, it’s a wrapper function around that command). It will add a library target called <name> to be built from the listed source files. In addition, it will take care of all the Python-specific compiler and linker flags as well as the OS- and Python-version-specific file extension. The produced target <name> can be further manipulated with regular CMake commands.
I am building a python module with Cython that links against a DLL file. In order to succesfully import my module I need to have the DLL in the Windows search path. Otherwise, the typical error message is:
ImportError: DLL load failed: The specified module could not be found.
Is there a way to packaged the DLL directly into the produced pyd file to make the distribution easier?
One example of this is with the OpenCV distribution, where a (huge) pyd file is distributed and is the only file needed for the Python bindings to work.
Python's packaging & deployment is still a pain point for many of us. There is just not a silver bullet. Here are several methods:
1. OpenCV build method
The method is decribed here : https://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_bindings/py_bindings_basics/py_bindings_basics.html#bindings-basics
OpenCV generates these wrapper functions automatically from the C++
headers using some Python scripts which are located in
modules/python/src2.
Basically it parse the header files and generate the static PyObject keywords whenever it's needed. Once the header are created appropriately, it just call python setup. Honestly, it might work, but I would not advise this method.
2. Makefiles
If you already use Makefile, just create a rule to place your lib accordinlgy. Example, from my own code :
setup.py
from distutils.core import setup, Extension
setup(name='sha1_hmac', version='1.0', \
ext_modules=[Extension('sha1_hmac',
library_dirs=['C:\MinGW\lib'],
sources= ['../tools/sha1.c','sha1_hmac.c'])])
Makefile
# The hmac generation used by the webserver is done
# using the sha1.c implementation. There is a binding needed to
# glue the C code with the python script
libsha1_hmac:
ifeq ($(OS), Windows_NT)
$(PYTHON) setup.py build --compiler=mingw32
else
$(PYTHON) setup.py install --home=$(CURDIR)
endif
.PHONY: webserver
webserver: libsha1_hmac
ifeq ($(OS), Windows_NT)
mv $(shell find build -type f -name "sha1*.pyd") $(LIB)
else
mv -f $(shell find $(LIB)/python -type f -name "sha1*.so") $(LIB)
endif
$(PYTHON) hmac_server.py
3. Modern deployement tools
There are several new tools to deploy python applications, namely wheels which seem to gain traction. I don't use it, but it look like it can ease up your bundling problem :
How can I make a Python Wheel from an existing native library?
Once it wheeled, you can install it like this : pip install some-package.whl
Trying to install a Spenvis package and here's the error I'm getting when running the script:
-- Boost version: 1.41.0
-- Found the following Boost libraries:
-- python
-- Configuring done
-- Generating done
-- Build files have been written to: Desktop/build
make[2]: *** No rule to make target `/usr/local/lib64/libpython2.6.so.1.0', needed by `source/libSpenvis.so'. Stop.
make[1]: *** [source/CMakeFiles/Spenvis.dir/all] Error 2
make: *** [all] Error 2
It's looking for libpython2.6.so.1.0 in /usr/local/lib64, but the library itself is in /usr/lib64.
I'm not a super user, so I can't change/copy the library into the directory nor can I make a link to it. For reference, here's the python script I'm running followed by the CMakeLists.txt file:
"""
Main launch script
"""
import os
import sys
import shutil
from subprocess import call
"""
Define path where to find python packages
"""
loc_dir=os.getcwd()
os.chdir("python_utilities")
#Spenvis csv
#############
os.chdir("spenvis_csv")
if not os.path.exists("build"):
os.mkdir("build")
os.chdir("build")
call("cmake ../", shell=True)
call("make")
os.chdir(loc_dir)
os.chdir("python_utilities")
if not os.path.exists("lib"):
os.mkdir("lib")
for file_name in os.listdir("spenvis_csv/build/source"):
if file_name.find("libSpenvis.") !=-1:
shutil.move("spenvis_csv/build/source/%s" %(file_name),"lib/Spenvis.so")
os.chdir("..")
And the CMakeLists.txt:
cmake_minimum_required(VERSION 2.8)
# Make sure the compiler can find include files
include_directories (${PYSPENVIS_SOURCE_DIR})
# get boost
set(Boost_USE_STATIC_LIBS OFF)
#set(Boost_USE_MULTIEADED ON)
find_package(Boost COMPONENTS
python
REQUIRED)
include_directories(${Boost_INCLUDE_DIRS})
link_directories(${Boost_LIBRARY_DIRS})
# get python
include(FindPythonLibs)
set(PythonLibs_USE_STATIC_LIBS OFF)
find_package(PythonInterp)
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_DIRS})
link_directories(${PYTHON_LIBRARIES})
add_library(Spenvis SHARED pySpenvisCSV.cc SpenvisCSV.cc SpenvisCSVCollection.cc)
TARGET_LINK_LIBRARIES(Spenvis ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
Taking advice, I ran the following command in the directory with CMakeLists.txt file:
cmake -DPYTHON_LIBRARY='/usr/lib64/libpython2.6.so.1.0'
This churned out:
Boost found.
Found Boost components:
python
-- Found PythonLibs: /usr/lib64/libpython2.6.so.1.0 (found version "2.7.1")
-- Found PythonLibs: /usr/lib64/libpython2.6.so.1.0 (found suitable version "2.7.1", minimum required is "2.6")
-- Configuring done
-- Generating done
-- Build files have been written to: blahblahblah
Going back to run the install.py program now returns:
-- Boost version: 1.41.0
-- Found the following Boost libraries:
-- python
-- Found PythonLibs: /usr/local/lib64/libpython2.6.so.1.0 (found version "2.6.6")
-- Found PythonLibs: /usr/local/lib64/libpython2.6.so.1.0 (found suitable version "2.6.6", minimum required is "2.6")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/smh/Linux/Desktop/gras original/gras-03-03/python/python_utilities/spenvis_csv/build
make[2]: *** No rule to make target `/usr/local/lib64/libpython2.6.so.1.0', needed by `source/libSpenvis.so'. Stop.
So, the same error is returned, only now it is claiming it's finding python libraries. Strangely, though, the python libraries it is finding don't exist, which is to say, there is no libpython2.6.so.1.0 in /usr/local/lib64 but instead it is located in /usr/lib64.
In the CMakeCache.txt, the lines concerning the desired library are as followed:
//No help, variable specified on the command line.
PYTHON_LIBRARIES:UNINITIALIZED=/usr/lib64/libpython2.6.so.1.0
//Path to a library.
PYTHON_LIBRARY:FILEPATH=/usr/lib64/libpython2.6.so
//Dependencies for the target
Spenvis_LIB_DEPENDS:STATIC=optimized;boost_python-mt-shared;debug;boost_python-mt-shared-debug;general;/usr/lib64/libpython2.6.so;
//Details about finding PythonLibs
FIND_PACKAGE_MESSAGE_DETAILS_PythonLibs:INTERNAL=[/usr/lib64/libpython2.6.so][/usr/local/include/python2.7][v2.7.1(2.6)]
The /usr need to be insert in front of CMAKE_PREFIX_PATH, so it will be found first.
list(INSERT 0 CMAKE_PREFIX_PATH /usr)
I am trying to install this python package on my cygwin(last available version) setup, using gcc and no windows/microsoft components at all. The package has many dependencies so it can take a while for you to try to install it. I guess the error is generic and you don't require to replicate the issue to provide insights on how to fix it.
I have found this error for several(all?) .la libraries on the package:
Making all in centrality
make[4]: Entering directory '/cygdrive/d/gdrive/Thesis/tech/urv tech/graph-tool-2.2.31/src/graph/centrality'
CXX graph_betweenness.lo
CXX graph_centrality_bind.lo
CXX graph_closeness.lo
CXX graph_eigentrust.lo
CXX graph_eigenvector.lo
CXX graph_hits.lo
CXX graph_katz.lo
CXX graph_pagerank.lo
CXX graph_trust_transitivity.lo
CXXLD libgraph_tool_centrality.la
/usr/lib/gcc/i686-pc-cygwin/4.8.2/../../../../i686-pc-cygwin/bin/ld: warning: --export-dynamic is not supported for PE targets, did you mean --export-all-symbols?
.libs/graph_betweenness.o:graph_betweenness.cc:(.text+0x182b): undefined reference to `graph_tool::GraphInterface::GetNumberOfVertices()'
.libs/graph_betweenness.o:graph_betweenness.cc:(.text+0x19bd): undefined reference to `graph_tool::GraphInterface::GetNumberOfVertices()'
.libs/graph_betweenness.o:graph_betweenness.cc:(.text+0x1a68): undefined reference to `graph_tool::ValueException::ValueException(std::string const&)'
.libs/graph_betweenness.o:graph_betweenness.cc:(.text+0x1a87): undefined reference to `graph_tool::ValueException::~ValueException()'
.libs/graph_betweenness.o:graph_betweenness.cc:(.text+0x1ad1): undefined reference to `graph_tool::ValueException::ValueException(std::string const&)'
/usr/lib/gcc/i686-pc-cygwin/4.8.2/../../../../i686-pc-cygwin/bin/ld: .libs/graph_betweenness.o: bad reloc address 0x6 in section `.text$_ZN5boost16exception_detail10clone_baseD1Ev[__ZN5boost16exception_detail10clone_baseD1Ev]'
collect2: error: ld returned 1 exit status
Makefile:474: recipe for target 'libgraph_tool_centrality.la' failed
As I see .lo files are built, probably not correctly, and when it comes to generate the .la file is when I get the 'bad reloc address' error. In other words, my assumption is that because the .lo files are not correctly generated the .la built fails.
Doing some research I have found this bug reported and fixed back in 2009:
https://www.mail-archive.com/bug-binutils#gnu.org/msg07150.html
Based on it, I understand that an approach to try to fix the problem would be to replace the -export-dynamic flag by -export-all-symbols in the configure file the python library uses to install itself. There are two files that I modify: configure and [configure.ac][2]
Configure has this kind of export-dynamic mentions, related to other settings:
export_dynamic_flag_spec_CXX='${wl}--export-dynamic'
Configure.ac has this one only:
[MOD_LDFLAGS="-module -avoid-version -export-dynamic -no-undefined -Wl,-E -Wl,--as-needed"]
that I think is the key one because it refers to LD flags.
But if I do this, the project seems to break an I receive messages saying the files used in the build have the wrong version of libtool.
make[4]: Entering directory '/cygdrive/d/gdrive/Thesis/tech/urv tech/graph-tool-2.2.31/src/graph/centrality'
CXX graph_betweenness.lo
libtool: Version mismatch error. This is libtool 2.4.2, but the
libtool: definition of this LT_INIT comes from libtool 2.4.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.2
libtool: and run autoconf again.
Makefile:507: recipe for target 'graph_betweenness.lo' failed
There are mentions also in aclocal.m4 that if I try to change cause additional problems related to build files version.
In addition, in ltmain.sh it is verified for some libraries if they have been generated with the flag export-dynamic.
So, several points here:
what is the difference between the .la and the .lo libraries that the module tries to generate?
I think the problem is related to differences in static and dynamic link but really don't know how to fix it. Are there other settings involved?
In general, any hints on what to try?