We Keep Coding

Python can't locate .so shared library with ctypes.CDLL - Windows

I am trying to run a C function in Python. I followed examples online, and compiled the C source file into a .so shared library, and tried to pass it into the ctypes CDLL() initializer function.
import ctypes
cFile = ctypes.CDLL("libchess.so")
At this point python crashes with the message:
Could not find module 'C:\Users\user\PycharmProjects\project\libchess.so' (or one of its dependencies). Try using the full path with constructor syntax.
libchess.so is in the same directory as this Python file, so I don't see why there would be an issue finding it.
I read some stuff about how shared libraries might be hidden from later versions of python, but the suggested solutions I tried did not work. Most solutions were also referring to fixes involving linux system environment variables, but I'm on Windows.
Things I've tried that have not worked:
changing "libchess.so" to "./libchess.so" or the full path
using cdll.LoadLibrary() instead of CDLL() (apparently both do the same thing)
adding the parent directory to system PATH variable
putting os.add_dll_directory(os.getcwd()) in the code before trying to load the file
Any more suggestions are appreciated.

Solved:
Detailed explanation here: https://stackoverflow.com/a/64472088/16044321
The issue is specific to how Python performs a DLL/SO search on Windows. While the ctypes docs do not specify this, the CDLL() function requires the optional argument winmode=0 to work correctly on Windows when loading a .dll or .so. This issue is also specific to Python versions greater than 3.8.
Thus, simply changing the 2nd line to cFile = ctypes.CDLL("libchess.so", winmode=0) works as expected.

ImportError: DLL load failed while importing ON WINDOWS

I fixed a super-annoying case of "ImportError: DLL load failed while importing" in a way that generally applies to Windows, so let me share it with the group. Here is the question:
I installed FINUFFT via pip install finufft. When I import finufft, I get this error:
ImportError: DLL load failed while importing _finufft: The specified module could not be found.
How do I fix it?

Read to the end before doing anything.
The error means that a DLL cannot find another DLL that it was linked with. But which other DLL?
Download Dependencies.
Locate your problematic DLL. In this specific case: Locate the folder ...\Lib\site-packages\finufft\ of the FINUFFT installation that you want to fix. ...\ is the path of your standard python installation or of your python virtual environment.
Start DependenciesGui.exe and use it to open the problematic DLL, e.g. ...\finufft\_finufft.cp38-win_amd64.pyd. (A .pyd is a regular DLL with some specific entry points for python.)
On the left, you will see a complete list of the problematic DLL's direct dependencies, whose dependencies you can in turn unfold by mouse click. Apart from typical Windows-DLLs, like kernel32.dll and MSVCRT.dll, and apart from the FFTW-DLLs, which should already be in the FINUFFT-folder, there will also be some - possibly missing - Linux-DLLs. For me, it was libgcc_s_seh-1.dll, libgomp-1.dll and libstdc++-6.dll. By checking their direct dependencies, I also discovered libwinpthread-1.dll as missing.
[See EDIT below!!!] I found those DLLs in Anaconda (...\Anaconda3\Library\mingw-w64\bin\), but you can probably also get them from cygwin (...\cygwin64\bin\), git (...\Git\mingw64\bin\) or anything else that downloads mingw64 and its packages on Windows.
To solve the problem, copy the respective DLLs into ...\Lib\site-packages\finufft\ and give them the exact filenames that the FINUFFT-DLL is expecting according to Dependencies. This works because Windows and because of the Windows DLL search order.
Now, import finufft should work in the specific python environment whose FINUFFT installation you fixed. Clearly, this method can be applied anytime DLL dependencies are missing.
EDIT - correction of my answer by #CristiFati: If possible, DLLs and similar things should always be built with the same toolchain. So if you don't compile them yourself, get them from as few different places as possible, i.e. don't mix regular python, Anaconda, cygwin, etc. - if possible. Of course, Windows DLLs will have a different origin from Linux DLLs.

Trying to import cjson in a Maya 2018 Python environment

I'd like to use cjson in Maya 2018 but I'm running into some trouble. The original tool seems to be 32-bit compiled (https://pypi.org/project/python-cjson/) so I looked around and found a version that was compiled in 64-bit from https://www.lfd.uci.edu/~gohlke/pythonlibs/.
Using "C:\Program Files\Autodesk\Maya2018\Python\Scripts\pip.exe" in admin mode I managed to pip install the 64-bit wheel file and that's given me cjson.pyd in my Maya 2018 site-packages directory.
However, when I try to import this I get the error:
// Error: root : [('<maya console>', 2, '<module>', None)] //
// Error: root : DLL load failed: The specified module could not be found. //
// Error: //
Has anyone had any experience importing this module into Maya before?
I have a feeling this could be to do DLL dependencies but I'm not sure where to begin solving it.

So, traditionally you'd check your dll for dependencies using something like Dependency Walker. .pyd files are just renamed .dll files, so should work fine with this. However, Dependency Walker is mostly full of noise - unless you spot something really obvious (and you can ignore all the API-MS-WIN*, EXT-MS-* dlls straight away) then you might find it easier to enable Loader Snaps (see this old but still relevant MSDN article) which should hopefully be able to tell you which loadlibrary call is failing. Alternatively, you can use Process Monitor (procmon) form Sysinternals to see what files Maya attempts to load when you import the module in to a script. I can't remember if Maya spawns a separate process to run it's python scripts in, or if they're executed in the context of the main maya process. This is important as you'll want to filter events in procmon to just maya / the python executable as otherwise you'll be swamped by them.
Edit: Having looked through the source code there doesn't appear to be much in the way of dependencies beyond the python headers themselves and some standard library includes. I suspect it's more likely that this is compiled with a different version of the toolchain than Maya's Python. In the Python interpreter, you should see a line like this when you start it:
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)] on win32
It would be good to see what toolchain was used to compile it (the square brackets bit), and then build the extension from source again with that toolchain. Alternatively, you can try running the setup.py for this module from the project's GitHub page, which if your machine is set up to be able to compile python extensions correctly will build it from source. More details about building c extensions can be found on the Python docs page.

What does "Symbol not found / Expected in: flat namespace" actually mean?

When I import a module I built, I get this boost-python related error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: dlopen(./myMod.so, 2): Symbol not found: __ZN5boost6python7objects15function_objectERKNS1_11py_functionERKSt4pairIPKNS0_6detail7keywordES9_E
Referenced from: ./myMod.so
Expected in: flat namespace
in ./myMod.so
What does this actually mean? Why was this error raised?

Description
The problem was caused by mixing objects that compiled with libc++ and object that compiled with libstdc++.
In our case, the library myMod.so (compiled with libstdc++) need boost-python that compiled with libstdc++ (boost-python-libstdc++ from now). When boost-python is boost-python-libstdc++, it will work fine. Otherwise - on computer that its boost-python has compiled with libc++ (or another c++ library), it will have a problem loading and running it.
In our case, it happens because that libc++ developers intentionally changed the name of all of their symbols to prevent you (and save you) from mixing code from their library and code from a different one: myMod.so need a function that take an argument from the type. In libc++, this type's name is std::__1::pair. Therefore, this symbol was not found.
To understand why mixing two version of the same API is bad, consider this situation: There are two libraries: Foo and Bar. They both have a function that takes a std::string and uses it for something but they use a different c++ library. When a std::string that has been created by Foo will be passed to Bar, Bar will think that this is an instance of its c++ library's std::string and then bad things can happen (they are a completely different objects).
Note: In some cases, there would be no problem with two or more different versions of the same API in a completely different parts of a program. There will be a problem if they will pass this API's objects between them. However, checking that can be very hard, especially if they pass the API object only as a member of another object. Also, a library's initialization function can do things that should not happen twice. Another version may do these things again.
How to solve that?
You can always recompile your libraries and make them match each other.
You can link boost-python to your library as a static library. Then, it will work on almost every computer (even one that doesn't has boost-python installed). See more about that here.
Summary
myMod.so need another version of boost-python, one that compiled with a specific c++ library. Therefore, It would not work with any another version.

In my case I was receiving:
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/xmlsec.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_xmlSecDSigNs'
BACKGROUND:
M1 MacBook Pro with Montery
I was working with a python virtualenv (using pyenv) to use an earlier version of python3.8 (3.8.2), while my system had 3.8.10 installed natively.
While I was in the activated 3.8.2 virtualenv I noticed the path in dlopen() was pointing to the package in the native python install NOT the virtualenv install.
SOLUTION:
In my case, I did not need the native 3.8 version at all so I simply removed it and this solved the problem.

I encounter the same problem.
Expected in: flat namespace
Add the linker flag fixes the problem
-lboost_python37
change the dynamic library name to the one installed on the os.
By the way, my os is macOS High Sierra and I use brew to install boost_python3.

Symbol not found means the definition of the declared function or variable was not found. When a header file of a shared object is compiled with your program, linker adds symbols of declared functions and objects to your compiled program. When your program is loaded by the OS's loader, the symbols are resolved so that their definition will be loaded. It is only at this time where if the implementation is missing, loader complains it couldn't find the definition due to may be failing to resolve the actual path to the library or the library itself wasn't compiled with the implementation/source file where the definition of the function or object resides. There is a good article on this on the linux journal http://www.linuxjournal.com/article/6463.

In my case I was just failing to import all the required sources (c++ files) when compiling with Cython.
From the string after "Symbol not found" you can understand which library you are missing.

One of the solutions I found was to uninstall and reinstall it using the no-binary flag, which forces pip to compile the module from source instead of installing from precompiled wheel.
pip install --no-binary :all: <name-of-module>
Found this solution here

Here's what I've learned (osx):
If this is supposed to work (i.e. it works on another computer), you may be experiencing clang/gcc issues. To debug this, use otool -l on the .so file which is raising the error, or a suspect library (in my example it's a boost-python dylib file) and examine the contents. Anything in the /System/ folder is built with clang, and should be installed somewhere else with the gcc compiler. Never delete anything in the /System folder.

.so files are dynamic libraries (so = shared object). On Windows they are called .dll (dynamic-link library). They contain compiled code which contains functions available for usage to any executable which links them.
What is important to notice here is that those .so are not Python files. They were probably compiled from C or C++ code and contain public functions which can be used from Python code (see documentation on Extending Python with C or C++).
On your case, well, you have a corrupt .so. Try reinstalling the affected libraries, or Python, or both.

Problem
I had this same issue when running puma as part of Rails app
LoadError:
dlopen(/Users/alucard/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/puma-5.6.4/lib/puma/puma_http11.bundle, 0x0009): symbol not found in flat namespace '_ERR_load_crypto_strings'
/Users/alucard/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/puma-5.6.4/lib/puma/puma_http11.bundle
Solution
It was solved just by installing puma gem again gem install puma

Importing a dll in python on Ubuntu

I am using python 2.6.5 on an Ubuntu intalled server.
I need to integrate an API for our applicaion, in that case, i needed to use a DLL given to me by the API provider. Their example of code about api integration is written in Visual Basic... I made a search on google and found some examples of using ctypes , and i try using cdll and pydll, which caused the following error...
OSError: /home//some.dll: invalid ELF header
One possibility is using IronPython, but i do not have much information about ironpython so i am not sure if it will handle my needs completely..
Is there any available module that let me use that dll on python (or aynthing that i am missing from the exixting ones). It is hard to upgrade my python version?

DLLs may be windows creatures, but if a DLL is 'pure .NET' and doesn't utilize executables specific to windows etc., then it can work often in Linux, through Mono. (mono ipy.exe).
Ironpython's System and similiar windows modules are customized to be os agnostic (to a untested degree).
I have successfully run NHibernate, FluentNHibernate, log4net, and a few other commonly used DLLS in Ubuntu.
import clr
import sys
sys.path.append(os.path.abspath('./DLL')) #where your dlls are
clr.AddReference('System')
clr.AddReference('FluentNHibernate')
from FluentNHibernate.Cfg.Db import PostgreSQLConfiguration
The key seems to be to import DLLs in this fashion. If a dll imports another (fluentnhibernate imports nhibernate), you don't need to import Nhibernate for example.

DLLs are Windows creatures. The only way you'll be able to use a DLL is by using a Windows build of Python. You'll be able to run Windows Python on Ubuntu by having Windows installed inside a virtual machine. You also might be able to run it using Wine.
An alternative, of course, is to ask your API provider if they have a Linux version of the API.

First, check if your DLL is a .NET Assembly file. An "Assembly DLL file" has nothing to do with the assembler. It's simply a way the .NET framework stores its bytecode inside a DLL file!
Do file library.dll in Linux. If it says something like this:
PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows
then you're lucky: it's an assembly file. You can run it on Linux.
Install Mono. Install Python.NET. Forget IronPython: it's dead.
Now, in Python.NET, you can do this:
import clr
clr.AddReference('./library.dll')
# the library has just registered a namespace we can use
from LibraryName import *
but how do you know what to import?
Auto-complete.
Or use monop tool to inspect the DLL like this:
$ monop -r library.dll
Assembly Information:
LibraryName
Version=9.9.3.0
Culture=neutral
PublicKeyToken=null
LibraryName.ClassName
...
$ monop -r library.dll LibraryName.ClassName
public class ClassName {
public ClassName (string inputString);
...
}
and it will tell you everything about that library