We Keep Coding

DLL load failed while importing after building .whl package with cython

I have built a cython package on windows using mingw.
Everything was fine, the only warnings i got during the compilation were:
Mylib\__init__.c: In function '__Pyx_ImportType':
\__init__.c:126091:13: warning: unknown conversion type character 'z' in format [-Wformat=]
126091 | "%s.%s size changed, may indicate binary incompatibility. "
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\__init__.c:126092:24: note: format string is defined here
126092 | "Expected %zd from C header, got %zd from PyObject",
| ^
\__init__.c:126091:13: warning: unknown conversion type character 'z' in format [-Wformat=]
126091 | "%s.%s size changed, may indicate binary incompatibility. "
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\__init__.c:126092:47: note: format string is defined here
126092 | "Expected %zd from C header, got %zd from PyObject",
| ^
\__init__.c:126091:13: warning: too many arguments for format [-Wformat-extra-args]
126091 | "%s.%s size changed, may indicate binary incompatibility. "
but for the moment, since the compilation was ok (Only these warnings and no errors), i decided to test the library built in cython installing via pip the .whl package that was generated.
The rest of the output of the compilation is the following:
writing build\temp.win-amd64-3.8\Release\mylib\mylib.cp38-win_amd64.def
creating build\lib.win-amd64-3.8
D:\mingw64\mingw64\bin\gcc.exe -shared -s build\temp.win-amd64-3.8\Release\mylib\__init__.o build\temp.win-amd64-3.8\Release\mylib\mylib.cp38-win_amd64.def -L./ -LC:\Users\Utente\AppData\Local\Programs\Python\Python38\libs -LC:\Users\Utente\AppData\Local\Programs\Python\Python38\PCbuild\amd64 -llmylib -lpython38 -lvcruntime140 -o build\lib.win-amd64-3.8\mylib.cp38-win_amd64.pyd -static-libgcc -static-libstdc++ -Wl,-Bstatic,--whole-archive -lwinpthread -Wl,--no-whole-archive "-L ."
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wheel
copying build\lib.win-amd64-3.8\mylib.cp38-win_amd64.pyd -> build\bdist.win-amd64\wheel\.
running install_egg_info
Copying Mylib.egg-info to build\bdist.win-amd64\wheel\.\Mylib-1.0.0-py3.8.egg-info
running install_scripts
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build\bdist.win-amd64\wheel\Mylib-1.0.0.dist-info\WHEEL
creating 'dist\Mylib-1.0.0-cp38-cp38-win_amd64.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'mylib.cp38-win_amd64.pyd'
adding 'mylib-1.0.0.dist-info/LICENSE'
adding 'mylib-1.0.0.dist-info/METADATA'
adding 'mylib-1.0.0.dist-info/WHEEL'
adding 'mylib-1.0.0.dist-info/top_level.txt'
adding 'mylib-1.0.0.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
c:\users\utente\desktop\mylib-main\.eggs\wheel-0.37.1-py3.8.egg\wheel\bdist_wheel.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
if get_flag('Py_DEBUG',
Now, i moved to dist/ and i run:
py -3.8 -m pip install Mylib-1.0.0-cp38-cp38-win_amd64.whl
Everything was fine.
I tapped:
py -3.8 -m pip show Mylib
And the output was:
Name: Mylib
Version: 1.0.0
Summary: My library
Home-page: https://github.com/Mylib
Author: Me
Author-email: Me#gmail.com
License: UNKNOWN
Location: c:\users\utente\appdata\local\programs\python\python38\lib\site-packages
Requires:
Required-by:
Now, i moved to c:\users\utente\appdata\local\programs\python\python38\lib\site-packages
and i saw:
a .PYD file: mylib.cp38-win_amd64
and a directory: mylib.cp38-win_amd64
Inside mylib.cp38-win_amd64 there are only these files:
INSTALLER.txt
METADATA.txt
LICENSE.txt
RECORD.txt
WHEEL.txt
When i run on python3.8:
import mylib
this error occurs:
"ImportError: DLL load failed while importing mylib: The specified module could not be found."
I tried to run from the same directory of the pyd file. Same error.
I also tried to import the path with:
import mysys
sys.path.append('directory of the pyd')
Again same error.
For what i have read there could be a problem with other dll dependencies.
So, i used dumpbin on the pyd and this was the output:
File Type: DLL
Image has the following dependencies:
python38.dll
libmylib.dll
KERNEL32.dll
msvcrt.dll
Summary
1000 .CRT
1000 .bss
6000 .data
1000 .edata
4000 .idata
2000 .pdata
5000 .rdata
1000 .reloc
1000 .rsrc
51000 .text
1000 .tls
2000 .xdata
If the problem is a problem of other dll dependencies i just tought that i needed to import the dlls paths. So ,as suggested by fix pyd with other dependencies i used these commands before importing mylib:
import os
os.add_dll_directory('directory of all the dlls previously showed')
But again, same error, so the problem is not a problem of other dll dependencies.
The only thing i can imagine this problem come from is the warnings previously showed. And in particular the last line of the compilation:
c:\users\utente\desktop\mylib-main\.eggs\wheel-0.37.1-py3.8.egg\wheel\bdist_wheel.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
if get_flag('Py_DEBUG',
That could cause the fact that, maybe (?), the library should be imported in debug mode (?), and so the pyd file (for what i have read) should have the signature mylib_d.pyd(?). But i tried to rename it in this way and it seems it does not work either.
NOTE:
On unix i was able to cythonize everything and import mylib in python.
Edit 1:
It seems that if i call even only a single function of the dll library mylib that is compiled with the code in cython inside a def function defined in the pyx module then that error appears
I found the solution:
Instead of compiling a .dll shared library i compiled a .lib library.
Then, when compiling in cython i linked the .lib library.
Usign dumpbin on the generated .pyd file it seems that mylib.pyd didn't depends to any libmylib.dll library and i can import it without any problem and calling functions that call in turns functions from mylib.lib

CMake not linking Python

Sorry if I'm duplicating a question, but I just cannot find the solution to what I'm looking for anywhere on the internet, yet I believe that this is a very simple problem.
I'm trying to extend python with some custom C++ libraries, and building my C++ libraries with CMake. I'm following the instructions on https://docs.python.org/2/extending/extending.html, but it's not compiling correctly.
When I try to build it, I get these messages:
"C:\Program Files (x86)\JetBrains\CLion 140.2310.6\bin\cmake\bin\cmake.exe" --build C:\Users\pkim2\.clion10\system\cmake\generated\76c451cd\76c451cd\Debug --target parsers -- -j 8
Linking CXX executable parsers.exe
CMakeFiles\parsers.dir/objects.a(main.cpp.obj): In function `spam_system':
C:/code/ground-trac/ground/launch/trunk/Software Support/Data Analysis Scripts/data_review_automation/parsers/main.cpp:9: undefined reference to `_imp__PyArg_ParseTuple'
C:/code/ground-trac/ground/launch/trunk/Software Support/Data Analysis Scripts/data_review_automation/parsers/main.cpp:12: undefined reference to `_imp__Py_BuildValue'
c:/mingw/bin/../lib/gcc/mingw32/4.8.1/../../../libmingw32.a(main.o):(.text.startup+0xa7): undefined reference to `WinMain#16'
collect2.exe: error: ld returned 1 exit status
CMakeFiles\parsers.dir\build.make:87: recipe for target 'parsers.exe' failed
CMakeFiles\Makefile2:59: recipe for target 'CMakeFiles/parsers.dir/all' failed
CMakeFiles\Makefile2:71: recipe for target 'CMakeFiles/parsers.dir/rule' failed
mingw32-make.exe[3]: *** [parsers.exe] Error 1
mingw32-make.exe[2]: *** [CMakeFiles/parsers.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles/parsers.dir/rule] Error 2
mingw32-make.exe: *** [parsers] Error 2
Makefile:109: recipe for target 'parsers' failed
Based on this, I suspect that this is a problem with the way I'm linking things in the CMakeLists.txt file, but I have no idea how to do it properly. This is what my CMakeLists.txt looks like right now:
cmake_minimum_required(VERSION 2.8.4)
project(parsers)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
set(SOURCE_FILES main.cpp)
include_directories(C:\\Python27\\include)
link_directories(C:\\Python27\\)
target_link_libraries(python2.7)
add_executable(parsers ${SOURCE_FILES})
How in the world do I get this thing to compile correctly? I am running Windows 7 64-bit, and using CLion as my IDE.

Your first problem is that you are using target_link_libraries wrong: you should pass it the target to which to add a link and then the library you want to link in:
target_link_libraries(parsers python2.7)
Your second problem is that you are building an executable, instead of a shared library. If you want to make your extension accessible from python it needs to be a library.
add_library(parsers SHARED ${SOURCE_FILES})
But now comes the good news: your life becomes much simpler (and more portable) if you use the built in CMake module FindPythonLibs.cmake. To build a python module you would only need to do the following:
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
find_package(PythonLibs REQUIRED)
add_library(parsers SHARED ${SOURCE_FILES})
include_directories(${PYTHON_INCLUDE_DIRS})
target_link_libraries(parsers ${PYTHON_LIBRARIES})

if you use windows, try this CMakeLists.txt
cmake_minimum_required(VERSION 3.8)
project(CSample)
set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -g -ftest-coverage -fprofile-arcs" )
link_directories(D:/Programs/mingw/mingw64/lib/)
include_directories(D:/Programs/Python/Python37/include/)
link_libraries(D:/Programs/Python/Python37/libs/python37.lib)
# Python
add_executable(CSample main.cpp)

You are using target_link_libraries() wrong. Check the docs; you probably want something like:
add_executable(parsers ${SOURCE_FILES})
target_link_libraries(parsers python2.7)
Note that the output from CMake should already tell you that something is wrong. On my machine:
CMake Error at CMakeLists.txt:8 (target_link_libraries):
Cannot specify link libraries for target "python2.7" which is not built by
this project.

Cython build resulting in undefined symbol

I've got a c++ program I'm trying to wrap/convert to Cython. It uses a particular library that for some reason, will not result in a working module for importing. There is a working c++ program, by the way. Here's the setup.py:
ext_modules = [
Extension(
name="libnmfpy",
sources=["interface/nmf_lib.pyx"],
include_dirs = ["../src/", numpy.get_include()],
libraries=["nmf","mpi_cxx","mpi","m"],
library_dirs=["../build/Linux/bin.release","/usr/local/lib/","/usr/lib"],
language="c++",)
]
setup(
name = 'libnmfpy',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
)
I should mention it is libnmf that seems to be causing problems. The first build of libnmf would cause this script to generate this error:
/usr/bin/ld: ../build/Linux/bin.release/libnmf.a(nmf.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
../build/Linux/bin.release/libnmf.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
When I rebuild libnmf with -fPIC, setup generates a libnmfpy.so, but when I import that in another script, I would get the aforementioned undefined symbol:
Traceback (most recent call last):
File "test.py", line 1, in <module>
import libnmfpy
ImportError: $path/cython/libnmfpy.so: undefined symbol: _ZN4elem6lapack3SVDEiiPdiS1_
If it would help, here's something my search suggested:
nm libnmfpy.so | grep _ZN4elem6lapack3SVDEiiPdiS1_
U _ZN4elem6lapack3SVDEiiPdiS1_
nm ../build/Linux/bin.release/libnmf.a | grep _ZN4elem6lapack3SVDEiiPdiS1_
U _ZN4elem6lapack3SVDEiiPdiS1_
Which is what I guess to the cause of the error. I look in what I think is the offending library that libnmf is built on:
nm $another_path/lib/libelemental.a | grep _ZN4elem6lapack3SVDEiiPdiS1_
0000000000005290 T _ZN4elem6lapack3SVDEiiPdiS1_
I am not too familiar yet with libraries and linkers, so any help would be appreciated. Thanks
edit: a little digging made me realize something. Is there a difference between Mac OS X and Linux that I should watch out for? The people I work for that wrote this originally reported no build errors like this

You should use nm -C, to unmangle your symbols. It also looks like you are mixing static and shared libraries which is generally not a good idea. Also, gcc's linker is a one pass linker which means the order of library flags matters. You want to list the libraries in reverse dependency order. In other words, if a depends on b, then b must appear before a in the linker flags.

Well I can't try to recreate your setup and then work out and test a solution on my setup since I don't have your source, but it seems to me that when you recompiled libnmf with fpic, it was recompiled with dynamic linking, while before it used to be statically linked.
If my guess is correct, then you can try either:
compiling libnmf again with -fPIC , AND -static.
changing your setup.py - add "elemental" to the libraries list - this will make the linker fetch that lib as well.
You should note that approach #1 is usually considered less desirable, but as I said it could be that it was originally compiled that way anyways. #2, however, could take more work because if there are other libs that are required, you will have to find and add them as well.

undefined symbol: PyOS_InputHook, from shared library

I wrote a C++ "python plugin" for an a non-python C++ application.
At some point, this plugin, which is a .so, initializes the python interpreter and opens a python console.
For convenience the "readline" module is then imported and we get this error:
ImportError: /usr/lib/python2.7/lib-dynload/readline.so: undefined symbol: PyOS_InputHook
The link command (generated by cmake) goes:
/usr/bin/c++ -fPIC -Wall -Wextra -O3 -DNDEBUG -Xlinker -export-dynamic -Wl,-fwhole-program /usr/lib/libpython2.7.a -shared -Wl,-soname,libMyplugin.so -o libMyplugin.so [sources] [qt libs] -lGLU -lGL -lX11 -lXext -lc -lc -lpython2.7 -Wl,-rpath,/src:/usr/local/Trolltech/Qt-4.8.4/lib:
nm libMyplugin.so gives the following python-related symbols:
U Py_Finalize
U Py_Initialize
00000000002114a8 B PyOS_InputHook
U PyRun_InteractiveLoopFlags
U PyRun_SimpleStringFlags
We observe that PyOS_InputHook is defined in the BSS section of the plugin. Yet, python's readline.so fails to find it.
The question is why, and how to fix it.

The issue is with how the main application loads the plugin: it uses dlopen() without the flag RTLD_GLOBAL.
This implies that the symbols present in the plugin that are not currently needed (like PyOS_InputHook in this instance) are not resolved and will not be resolved for other shared libraries that will be loaded afterwards (like readline.so in this instance).
To fix this, the flag RTLD_GLOBAL should be used when loading the plugin.
If there is no control over the main application (as in this instance) and on how it uses dlopen(), it is still possible to "reload" the plugin from within the plugin itself using dlopen() with flags RTLD_NOLOAD | RTLD_GLOBAL, so as to resolve all previously unresolved symbols in the currently loaded library.
Doing this solves the issue.

How to compile .c code from Cython with gcc

Now that I've successfully installed Cython on Windows 7, I try to compile some Cython code using Cython, but gcc makes my life hard.
cdef void say_hello(name):
print "Hello %s" % name
Using gcc to compile the code throws dozens of undefined reference to -erros, and I'm pretty sure the libpython.a is available (as the installation tutorial said, undefined reference to -errors are thrown if this file is missing).
$ cython ctest.pyx
$ gcc ctest.c -I"C:\Python27\include"
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1038): undefined reference to `_imp__PyString_FromStringAndSize'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1075): undefined reference to `_imp___Py_TrueStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1086): undefined reference to `_imp___Py_ZeroStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1099): undefined reference to `_imp___Py_NoneStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x10b8): undefined reference to `_imp__PyObject_IsTrue'
c:/program files/mingw/bin/../lib/gcc/mingw32/4.5.2/../../../libmingw32.a(main.o):main.c:(.text+0xd2): undefined reference to `WinMain#16'
collect2: ld returned 1 exit status
The weird thing is, using pyximport* or a setup-script works pretty fine, but it's both not very handy when still working on a module.
How to compile those .c files generated with Cython using gcc ?
or any other compiler, important is that it will work !
*pyximport: Is it normal that only python-native functions and classes are contained in the imported module and not cdef-functions and classes ?
like:
# filename: cython_test.pyx
cdef c_foo():
print "c_foo !"
def foo():
print "foo !"
c_foo()
import pyximport as p; p.install()
import cython_test
cython_test.foo()
# foo !\nc_foo !
cython_test.c_foo()
# AttributeError, module object has no attribute c_foo
UPDATE
Calling $ gcc ctest.c "C:\Python27\libs\libpython27.a" kills the undefined reference to -erros, but this one:
c:/program files/mingw/bin/../lib/gcc/mingw32/4.5.2/../../../libmingw32.a(main.o):main.c:(.text+0xd2): undefined reference to `WinMain#16'

Try:
gcc -c -IC:\Python27\include -o ctest.o ctest.c
gcc -shared -LC:\Python27\libs -o ctest.pyd ctest.o -lpython27
-shared creates a shared library. -lpython27 links with the import library C:\Python27\libs\libpython27.a.

That is a linker (ld) error and not a compiler error. You should provide the path to the library (-l and -L) and not only to the headers (-I).