We Keep Coding

pybind with third-party C++ on Windows (ImportError: DLL load failed)

I'm trying to use a third-party c++ lib from python using pybind11.
I have managed to do it using pure C++ code, but, when I add the third-party code I get the following error:
ImportError: DLL load failed while importing recfusionbind
Here's my code:
#include <RecFusion.h>
#include <pybind11/pybind11.h>
using namespace RecFusion;
bool isValidLicense() {
return RecFusionSDK::setLicenseFile("License.dat");
}
namespace py = pybind11;
PYBIND11_MODULE(recfusionbind, m) {
m.def("isValidLicense", &isValidLicense, R"pbdoc(
Check if there is a valid license file.
)pbdoc");
#ifdef VERSION_INFO
m.attr("__version__") = VERSION_INFO;
#else
m.attr("__version__") = "dev";
#endif
}
If i change, for testing purposes, the return of the function to simply true it works just fine. But calling the third-party library gives the above mentioned error. I have put RecFusion.dll on the current directory (where the python script is), with same results.
Any tips on what I'm missing will be welcomed. Lib is 64 bits, same as my code.

This seems to me like a linking problem. Look at your dll's import table (with tools like PE explorer or IDA). Now validate that the function RecFusionSDK::setLicenseFile is indeed imported from a dll named RecFusion.dll and not some other dll name.
Also validate that the mangled function name in RecFusion.dll's export table and in your dll match. If not, this could be an ABI problem.

PyImport_Import cannot find module

I'm trying to call a python function inside my c++
When i import my python module it gives this error:
ModuleNotFoundError: No module named 'test'
Here is how i'm importing the module:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int main()
{
Py_Initialize();
PyObject *pName = PyUnicode_FromString("test");
PyObject *pModule = PyImport_Import(pName);
if (pModule == nullptr)
{
PyErr_Print();
std::exit(1);
}
Py_Finalize();
return 0;
}
I feel like this must have to do with my project structure. It currently looks like this:
project
|-- main.cpp
|-- test.py
I'm not sure if it's worth mentioning, but the executable is in the same directory, and it's being run from that directory aswell.
How can i fix this?

Use Py_SetPath before Py_Initialize() to set the sys.path.
Here is a longer list of what can be done before you initialize the interpreter: pre-init-safe
Isolate the embedded Python interpreter and set the path properly to avoid problems with partially using the modules of the installed Python version.

This was solved by setting the "PYTHONPATH" environment variable:
setenv("PYTHONPATH", ".", 1);
Thanks #Botje

As dwib mentioned I was able to overcome by the error by setting the PYTHONPATH variable in my zshrc (may be bashrc for you) file.
export PYTHONPATH=$PATH:/home/jk/jk/c
where /home/jk/jk/c is the folder where my file was residing.
Also sourced it
. ~/.zshrc

Python ctypes dll load error 487

I am using Python 2.7 and trying to load a dll with ctypes:
lib = ctypes.cdll.LoadLibrary("mylib.dll")
It sometimes throws the following error, some other times it runs ok. Also with Python 3 always throws this error:
libcrypto = ctypes.cdll.LoadLibrary("data\openvpn\libeay32") File
"C:\Python27\lib\ctypes__init__.py", line 440, in LoadLibrary
return self._dlltype(name) File "C:\Python27\lib\ctypes__init__.py", line 362, in init
self._handle = _dlopen(self._name, mode) WindowsError: [Error 487] Attempt to access invalid address

We were having a similar issue with a FIPS-certified version of OpenSSL. Unfortunately, that version of OpenSSL is required to be loaded at a specific memory address. If you try to load OpenSSL too late when starting an application, the memory address it requires has already been taken by another dependency. The best we could figure out to "fix" this is to ensure OpenSSL (libeay32.dll) is loaded into the memory address space as soon as possible.
In your case, add the following lines AT THE TOP of your Python script (NOTE: AT THE TOP means before any other line of executable code, including import statements):
import ctypes
from ctypes import cdll
from ctypes.util import find_library
cdll.LoadLibrary(find_library('libeay32'))
Alternatively, you could ensure that libeay32.dll is the first dependency listed within mylib.dll.
BEGIN EDIT
While the above code helped increase the probability of success in my case, it still loaded the OpenSSL DLL too late in some instances. To fix this, I had to create a new Python wrapper executable in C++, with the following code:
#include "Python.h"
#include "openssl\crypto.h"
auto volatile GetVersionOpenSSL{ &SSLeay };
int wmain(size_t const argc, wchar_t const *const argv[]) {
Py_Initialize();
int const result{ Py_Main(argc, const_cast<wchar_t**>(argv)) };
Py_Finalize();
return result;
}
Note that I had to take the following additional actions as well:
Add libeay32.lib as the first library in the linker command line
Disable the linker optimization to remove unused references
Call the resulting executable instead of python.exe with the same arguments, and that error should go away.

Python C Extension using numpy and gdal giving undefined symbol at runtime

I'm writing a C++ extension for python to speed up a raster image viewer created in-house.
I've got working code, but noticed that the speed hadn't increased that much and after profiling a bit deeper realised that it was due to the gdal.ReadAsArray calls, which were python callbacks from the C extension.
To get around the overhead of Python C-API when calling python objects I decided I would use the C++ libraries for gdal rather than using the Python callback to the existing gdalDataset. (space isn't a problem).
However after implementing the code for this I ran into an error at runtime(the extension compiled fine)
which was
import getRasterImage_new
ImportError: /local1/data/scratch/lib/python2.7/site-packages
/getRasterImage_new.so: undefined symbol: _ZN11GDALDataset14GetRasterYSizeEv
The code below replicates the error(some edits may be needed to run on your machines(ignore the uninitialised variables, it's just whats needed to replicate the error).
Python:
#!/usr/bin/env python
import numpy
from osgeo import gdal
import PythonCTest
print("test starting")
PythonCTest.testFunction(1)
print("test complete")
C++:
#include "Python.h"
#include "numpy/arrayobject.h"
#include "gdal_priv.h"
#include <iostream>
extern "C"{
static PyObject* PythonCTest_testFunction(PyObject* args);
static PyMethodDef PythonCTest_newMethods[] = {
{"testFunction", (PyCFunction)PythonCTest_testFunction, METH_VARARGS,
"test function"},
{NULL,NULL,0,NULL}};
PyMODINIT_FUNC initPythonCTest(void){
(void)Py_InitModule("PythonCTest",PythonCTest_newMethods);
import_array();
}
}
GDALDataset* y;
static PyObject* PythonCTest_testFunction(PyObject* args){
std::cout << "in testFunction\n";
y->GetRasterYSize();
std::cout << "doing stuff" << "\n";
return Py_None;
}
Any suggestions would be very welcome.
EDIT
You can also remove the from osgeo import gdal and the error stull occurs(only just noticed that).
EDIT 2
I forgot to say that I'm compiling my extension using distutils
current setup.py is
#!/usr/bin/env python
from distutils.core import setup, Extension
import os
os.environ["CC"] = "g++"
setup(name='PythonCTest', version='1.0', \
ext_modules=[Extension('PythonCTest', ['PythonCTest.cpp'],
extra_compile_args=['--std=c++14','-l/usr/include/gdal', '-I/usr/include/gdal'])])

A Python extension module is a dynamically loadable (shared) library. When linking a shared library, you need to specify its library dependencies, such as -lgdal and, for that matter, -lpython2.7. Failing to do so results in a library with unresolved symbols, and if those symbols are not provided by the time it is loaded, the loading will fail, as reported by Python.
To resolve the error, you need to add libraries=['gdal'] to Extension constructor. Specifying -lgdal in extra_compile_args won't work because compile arguments, as the name implies, are used for compilation and not for linking.
Note that an unresolved symbol would not go undetected when linking an executable, where the build would fail with a linker error. To get the same diagnostics when linking shared libraries, include -Wl,-zdefs in link arguments.

Loading DLL from embedded python code in c

The crux of my problem is this:
I am developing code on Windows XP in C with MS Visual Studio 10.0, and I need to embed Python to do some plotting, file management, and some other things. I had problems with sys.path finding my Pure-Python modules, but I have fixed the problem by modifying PYTHONPATH.
Now, my problem is getting python to find dynamic libraries that are pulled in by some modules. In particular, my problem is to compress a folder into a bzip2 achive of the same name.
From a normal python command prompt, this works just fine:
import tarfile
tar=tarfile.open('Code.tar.bz2','w:bz2')
tar.add('Code',arcname='Code')
tar.close()
But when I call this code from my c-code, it gives me this error:
Traceback (most recent call last):
File "<string>", line 4, in <module>
File "D:\My_Documents\Code\ScrollModel\trunk\PythonCode.py", line 20, in Colle
ctFiles
tar=tarfile.open(os.path.join(runPath,'CODE.tar.bz2'),'w:bz2')
File "c:\Python26\lib\tarfile.py", line 1671, in open
return func(name, filemode, fileobj, **kwargs)
File "c:\Python26\lib\tarfile.py", line 1737, in bz2open
raise CompressionError("bz2 module is not available")
tarfile.CompressionError: bz2 module is not available
I have a suspicion the problem is similar to what is described at section 5.6 of Embedded Python, but it is a bit hard to tell. For what its worth, if I do
Py_Initialize();
PyRun_SimpleString("import ssl\n");
Py_Finalize();
it doesn't work either and I get an ImportError.
Anyone had any problems like this? Am I missing something critical?

Try this, it works on my machine.
Create a simple Windows console application in Visual Studio 2010 (remove precompiled headers option in the wizard). Replace the generated code with this one :
#include <Python.h>
int main(int argc, char *argv[]) {
Py_Initialize();
PyRun_SimpleString("import ssl \n"
"for f in dir(ssl):\n"
" print f \n" );
Py_Finalize();
return 0;
}
With PYTHONHOME set to something like c:\Python...
add C:\Python\Include to the include path
add C:\Python\Libs to the library path
add python26.lib to the linker input (adjust with your Python version)
Build. Run from anywhere and you should see a listing of the content of the ssl module.
I also tried with Mingw. Same file, build with this command line :
gcc -Wall -o test.exe embeed.c -I%PYTHONHOME%\Include -L%PYTHONHOME%\libs -lpython26

Hey, I have asked a similar question, my operation system is Linux.
When i compile c file, option $(python-config --cflags --ldflags) should be added, as
gcc test.c $(python-config --cflags --ldflags) -o test
I think in Windows you may also check python-config option, Hope this helps!

I had a similar problem with Boost C++ DLL. Any external DLL need to be in the DLL search path.
In my experience, PYTHONPATH affects Python module (the import statement in Python will end up a LoadLibrary call), and build options have nothing to do with it.
When you load a DLL, Windows doesn't care what the process is. In other words, Python follows the same DLL loading rules as Notepad. You can confirm that you are facing a Windows path problem by copying any missing DLL in the same directory as your python extension, or to a directory in your path.
To find what DLL are required by any other executable or DLL, simply open the DLL or EXE file with DependencyWalker. There is also a "Profile" menu which will allow you to run your application and watch it search and load DLLs.