We Keep Coding

Python can't locate .so shared library with ctypes.CDLL - Windows

I am trying to run a C function in Python. I followed examples online, and compiled the C source file into a .so shared library, and tried to pass it into the ctypes CDLL() initializer function.
import ctypes
cFile = ctypes.CDLL("libchess.so")
At this point python crashes with the message:
Could not find module 'C:\Users\user\PycharmProjects\project\libchess.so' (or one of its dependencies). Try using the full path with constructor syntax.
libchess.so is in the same directory as this Python file, so I don't see why there would be an issue finding it.
I read some stuff about how shared libraries might be hidden from later versions of python, but the suggested solutions I tried did not work. Most solutions were also referring to fixes involving linux system environment variables, but I'm on Windows.
Things I've tried that have not worked:
changing "libchess.so" to "./libchess.so" or the full path
using cdll.LoadLibrary() instead of CDLL() (apparently both do the same thing)
adding the parent directory to system PATH variable
putting os.add_dll_directory(os.getcwd()) in the code before trying to load the file
Any more suggestions are appreciated.

Solved:
Detailed explanation here: https://stackoverflow.com/a/64472088/16044321
The issue is specific to how Python performs a DLL/SO search on Windows. While the ctypes docs do not specify this, the CDLL() function requires the optional argument winmode=0 to work correctly on Windows when loading a .dll or .so. This issue is also specific to Python versions greater than 3.8.
Thus, simply changing the 2nd line to cFile = ctypes.CDLL("libchess.so", winmode=0) works as expected.

Why does python throw an undefined symbol error when importing a shared object from an alternate path?

I created a python extension using Boost::Python. To make it easier to use the extension on different target machines, I have included the libboost_python36.so.1.75.0 library in the same directory as the generated extension (pyshmringbuffer.so).
I checked out pyshmringbuffer.so and libboost_python36.so.1.75.0 onto a machine other than it was compiled in the directory : /path/to/pyshmringbuffer
After setting LD_LIBRARY_PATH to: /path/to/pyshmringbuffer and changing to this directory, I am able to run python3.6 and import the shared object just fine.
The problem comes when I try to run python from an alternate directory. From any other directory, I append the python path as follows:
import sys
sys.path.append("/path/to/pyshmringbuffer")
Then, when I try to import pyshmringbuffer, I get the following undefined symbol:
ImportError: /path/to/pyshmringbuffer/pyshmringbuffer.so: undefined symbol: _ZNK5boost6python7objects21py_function_impl_base9max_arityEv
I was under the impression that all symbols are self contained within the shared object. Why does it matter where I import the shared library from?

The symbol in your error message is an internal one, generated by one of the build tools. Having one undefined suggests that one of your components was built with an incompatible tool version, or that a *.so file (shared object) is out of date in some other fashion.
The simplest way to fix this is usually to rebuild your product components from scratch, in the proper order.

I was able to resolve my issue by prepending /path/to/pyshmringbuffer to my python path using:
sys.path.insert(0,"/path/to/pyshmringbuffer")
I can't say for sure, but as #PRUNE pointed out, there is something in my python path that python is seeing before it sees the intended library.
Coincidentally, I DO have a build of libboost_python36.so.1.75.0 located elsewhere on the target machine. The path for this doesn't appear on my PYTHONPATH or LD_LIBRARY_PATH, so I wouldn't EXPECT it to have been interfering, but I can't be positive.

What does "Symbol not found / Expected in: flat namespace" actually mean?

When I import a module I built, I get this boost-python related error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: dlopen(./myMod.so, 2): Symbol not found: __ZN5boost6python7objects15function_objectERKNS1_11py_functionERKSt4pairIPKNS0_6detail7keywordES9_E
Referenced from: ./myMod.so
Expected in: flat namespace
in ./myMod.so
What does this actually mean? Why was this error raised?

Description
The problem was caused by mixing objects that compiled with libc++ and object that compiled with libstdc++.
In our case, the library myMod.so (compiled with libstdc++) need boost-python that compiled with libstdc++ (boost-python-libstdc++ from now). When boost-python is boost-python-libstdc++, it will work fine. Otherwise - on computer that its boost-python has compiled with libc++ (or another c++ library), it will have a problem loading and running it.
In our case, it happens because that libc++ developers intentionally changed the name of all of their symbols to prevent you (and save you) from mixing code from their library and code from a different one: myMod.so need a function that take an argument from the type. In libc++, this type's name is std::__1::pair. Therefore, this symbol was not found.
To understand why mixing two version of the same API is bad, consider this situation: There are two libraries: Foo and Bar. They both have a function that takes a std::string and uses it for something but they use a different c++ library. When a std::string that has been created by Foo will be passed to Bar, Bar will think that this is an instance of its c++ library's std::string and then bad things can happen (they are a completely different objects).
Note: In some cases, there would be no problem with two or more different versions of the same API in a completely different parts of a program. There will be a problem if they will pass this API's objects between them. However, checking that can be very hard, especially if they pass the API object only as a member of another object. Also, a library's initialization function can do things that should not happen twice. Another version may do these things again.
How to solve that?
You can always recompile your libraries and make them match each other.
You can link boost-python to your library as a static library. Then, it will work on almost every computer (even one that doesn't has boost-python installed). See more about that here.
Summary
myMod.so need another version of boost-python, one that compiled with a specific c++ library. Therefore, It would not work with any another version.

In my case I was receiving:
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/xmlsec.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_xmlSecDSigNs'
BACKGROUND:
M1 MacBook Pro with Montery
I was working with a python virtualenv (using pyenv) to use an earlier version of python3.8 (3.8.2), while my system had 3.8.10 installed natively.
While I was in the activated 3.8.2 virtualenv I noticed the path in dlopen() was pointing to the package in the native python install NOT the virtualenv install.
SOLUTION:
In my case, I did not need the native 3.8 version at all so I simply removed it and this solved the problem.

I encounter the same problem.
Expected in: flat namespace
Add the linker flag fixes the problem
-lboost_python37
change the dynamic library name to the one installed on the os.
By the way, my os is macOS High Sierra and I use brew to install boost_python3.

Symbol not found means the definition of the declared function or variable was not found. When a header file of a shared object is compiled with your program, linker adds symbols of declared functions and objects to your compiled program. When your program is loaded by the OS's loader, the symbols are resolved so that their definition will be loaded. It is only at this time where if the implementation is missing, loader complains it couldn't find the definition due to may be failing to resolve the actual path to the library or the library itself wasn't compiled with the implementation/source file where the definition of the function or object resides. There is a good article on this on the linux journal http://www.linuxjournal.com/article/6463.

In my case I was just failing to import all the required sources (c++ files) when compiling with Cython.
From the string after "Symbol not found" you can understand which library you are missing.

One of the solutions I found was to uninstall and reinstall it using the no-binary flag, which forces pip to compile the module from source instead of installing from precompiled wheel.
pip install --no-binary :all: <name-of-module>
Found this solution here

Here's what I've learned (osx):
If this is supposed to work (i.e. it works on another computer), you may be experiencing clang/gcc issues. To debug this, use otool -l on the .so file which is raising the error, or a suspect library (in my example it's a boost-python dylib file) and examine the contents. Anything in the /System/ folder is built with clang, and should be installed somewhere else with the gcc compiler. Never delete anything in the /System folder.

.so files are dynamic libraries (so = shared object). On Windows they are called .dll (dynamic-link library). They contain compiled code which contains functions available for usage to any executable which links them.
What is important to notice here is that those .so are not Python files. They were probably compiled from C or C++ code and contain public functions which can be used from Python code (see documentation on Extending Python with C or C++).
On your case, well, you have a corrupt .so. Try reinstalling the affected libraries, or Python, or both.

Problem
I had this same issue when running puma as part of Rails app
LoadError:
dlopen(/Users/alucard/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/puma-5.6.4/lib/puma/puma_http11.bundle, 0x0009): symbol not found in flat namespace '_ERR_load_crypto_strings'
/Users/alucard/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/puma-5.6.4/lib/puma/puma_http11.bundle
Solution
It was solved just by installing puma gem again gem install puma

How to make compiled __init__ module extensions work properly?

I'm using the 64-bit version of python 3.3 and cython 0.18 on Windows 7.
I've encountered a problem that is thoroughly described here
One issue that was not discussed in the linked thread was, that the python's distutils called the link command (link.exe) with an option to export (this is only needed on Windows) a symbol named PyInit_[module name] but, if the module name was __init__ the cython-created C file only specified a symbol with name PyInit_[package name] and the linker gave back this error: LINK : error LNK2001: unresolved external symbol.
Fortunately, I managed to fix this issue in the build_ext module.
The problem that remains is that a compiled __init__.pyd module of mypackage containing an import statement like
import math
Throws an
SystemError: Parent module '' not loaded, cannot perform relative import
When mypackage is imported (no matter from where it is imported). Without import statements it works as expected though.
If anyone has a fix, workaround or information that could solve this problem or could explain why it's not (yet) solvable please, share your knowledge.

*.pyd file fails to load, but DependancyWalker comes up clean, and ProcMon shows it loaded

I am trying to load a *.pyd with Python, but I receive the well known "Import Error: DLL load failed: the specified procedure can not be found." error.
I have already done the following:
1.) Investigated the *.pyd with Dependency Walker. GPSVC.DLL and IESHIMS.DLL came up as missing, but delay loaded, IEFRAME.DLL aslo came up as missing an export, but was also delay-loaded. It's my understanding that these are not used, and are delay load anyway, so they should not be the problem.
2.) Did an "import foo" on foo.pyd in the python command window, with ProcMon watching. ProcMon shows event "LoadImage" on "foo.pyd" with result SUCCESS.
This seems to imply that the *.pyd file loaded correctly.
So what am I missing. My windows diagnostics are telling me all is well, but python is telling me the thing cannot be loaded (usually due to a missing dll or symbol).
Ideas?
Thanks!

Is the .pyd file for the same version of Python you're using? Loading a .pyd file for the wrong Python version can produce that error message.
Dependency Walker can show you which pythonNN.dll it links to.

If you have a file foo.pyd, for import foo to succeed, there must be an externally accessible function named initfoo. Dependency Walker will show this (typically the ONLY function) if it exists. initfoo needs to be called by Python to initialise the foo module.
Note: I would expect a more explicit error message if this were the problem:
>>> import fubar
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define init function (initfubar)
>>>
You say that you are "trying to load a *.pyd file". Is that just a strange way of describing import foo or is it something else?
Did you create the pyd? If not, who did? Have you asked them? Is this pyd available on the web so that others could try to load/import it?

Ok here is the answer:
The windows diagnostics (depends, procmon, etc) were showing the DLL (or pyd) loading fine.
Python was showing that it was not loading fine.
I found that the windows tools were referring to a different Python26.dll hiding in my C:\Window\SysWOW64 folder.
This second Python26.dll (found in SysWOW64) has a symbol that is missing in the primary python26.dll (installed by the windows python installer, found in C:\Python26).
This symbol "_PyByteArray_empty_string", was apparently needed by my *.pyd file.
So when loading via windows diagnostics, the SysWOW64 dll was found, and the *.pyd loaded properly. When loading from python, the dll in C:\Python26\ was found, the symbol was missing, and load failed.
So that is WHY the problem manifested. The question now is: Why are there two versions of Python26.dll floating around, one with _PyByteArray_empty_string, and one without?
I'm using Python 2.6.6. Perhaps this symbol is removed in 2.6.6 but present in some older 2.6.x release?