We Keep Coding

ImportError: DLL load failed while importing ON WINDOWS

I fixed a super-annoying case of "ImportError: DLL load failed while importing" in a way that generally applies to Windows, so let me share it with the group. Here is the question:
I installed FINUFFT via pip install finufft. When I import finufft, I get this error:
ImportError: DLL load failed while importing _finufft: The specified module could not be found.
How do I fix it?

Read to the end before doing anything.
The error means that a DLL cannot find another DLL that it was linked with. But which other DLL?
Download Dependencies.
Locate your problematic DLL. In this specific case: Locate the folder ...\Lib\site-packages\finufft\ of the FINUFFT installation that you want to fix. ...\ is the path of your standard python installation or of your python virtual environment.
Start DependenciesGui.exe and use it to open the problematic DLL, e.g. ...\finufft\_finufft.cp38-win_amd64.pyd. (A .pyd is a regular DLL with some specific entry points for python.)
On the left, you will see a complete list of the problematic DLL's direct dependencies, whose dependencies you can in turn unfold by mouse click. Apart from typical Windows-DLLs, like kernel32.dll and MSVCRT.dll, and apart from the FFTW-DLLs, which should already be in the FINUFFT-folder, there will also be some - possibly missing - Linux-DLLs. For me, it was libgcc_s_seh-1.dll, libgomp-1.dll and libstdc++-6.dll. By checking their direct dependencies, I also discovered libwinpthread-1.dll as missing.
[See EDIT below!!!] I found those DLLs in Anaconda (...\Anaconda3\Library\mingw-w64\bin\), but you can probably also get them from cygwin (...\cygwin64\bin\), git (...\Git\mingw64\bin\) or anything else that downloads mingw64 and its packages on Windows.
To solve the problem, copy the respective DLLs into ...\Lib\site-packages\finufft\ and give them the exact filenames that the FINUFFT-DLL is expecting according to Dependencies. This works because Windows and because of the Windows DLL search order.
Now, import finufft should work in the specific python environment whose FINUFFT installation you fixed. Clearly, this method can be applied anytime DLL dependencies are missing.
EDIT - correction of my answer by #CristiFati: If possible, DLLs and similar things should always be built with the same toolchain. So if you don't compile them yourself, get them from as few different places as possible, i.e. don't mix regular python, Anaconda, cygwin, etc. - if possible. Of course, Windows DLLs will have a different origin from Linux DLLs.

Rust and Python on W10: LNK1181

I'm trying to write a Rust module which can be called from Python. I'm following this page:
https://developers.redhat.com/blog/2017/11/16/speed-python-using-rust#edit_src_lib_rs
On cargo build --release I get this error:
= note: LINK : fatal error LNK1181: cannot open input file
'python39.lib'
This has come up before, and this seems the most recent and relevant answer.
I seem to have tried all the possible solutions in that answer, including locating and running vcvars64.bat, as detailed here. No joy.
I have MS Visual Studio (2019) installed, with the C++ and W10 SDK components.
Significantly, this error sometimes does not occur when I don't include the --release switch. Without it, the program (sometimes) compiles and runs OK. Given that one reason for wanting to make a Rust module is performance, however, I'd like to solve this problem. Also I have now found that this successul building of a "debug" build is a) intermittent and b) partial: when it fails, some of the desired output files are created but not others.
And...
According to the page at the first link, it seems that after build, I should then be looking for a file ending .so. In a W10 OS I'm looking for a .dll file.
If it builds completely successfully, under target\debug I have myrustlib.d, myrustlib.dll, myrustlib.dll.exp, myrustlib.dll.lib, myrustlib.pdb, pyext_myrustlib.pdb, pyext_myrustlib.d and pyext_myrustlib.exe. None of these, renamed and/or given an .dll extension as applicable, can be successfully imported as a Python module, at least as documented on that page.
I've also found this more recent page for doing the same thing. I get the same 1181 error.
python39.lib
I found out where this is located in my system, under ... Python\Python39\libs. I modified my PATH env var to specifically include this path (and rebooted). Still the same error.

Workaround:
This page, using PyO3 instead of rust-cpython (actually a fork of the latter), appears to do things successfully, on my machine + OS at least.
NB at the time of writing the second example appears to contain a typo: I believe it should be
fn hashset(_: Python, m: &PyModule) -> PyResult<()> {
rather than
fn hash_set(_: Python, m: &PyModule) -> PyResult<()> {
if you want it to compile OK (author has been notified). I have managed to create a module (hashset.cp39-win_amd64.pyd in a W10 OS) which successfully enables you to import this Rust construct into Python.

Using embedded python, loading a *.pyd outside of DLLs directory fails

I have a C++ application (X-Plane) for which there is a plugin which permits the use of python scripts (XPPython3 plugin). The plugin is written in C, using python CAPI, and works great, allowing me to write python scripts which get executed within the C++ application.
On Windows 10, I want to extend my python features by importing imgui. I have a python cython-built pyd file (_imgui.cp39-win_amd64.pyd).
If I place the pyd file in C\Program Files\Python39\DLLs, it works as expected: C++ application calls CAPI to python, which loads script which imports and executes imgui code.
If I place the pyd file anywhere else, embedded python either reports "module not found" -- if the pyd isn't on sys.path(), or if it is on sys.path():
ImportError: DLL load failed while importing _imgui: The parameter is incorrect.'
Changes using: os.add_dll_directory(r'D:\somewhere\else')
Does not effect whether the module is found or not, nor does it change the 'parameter incorrect' error. (see https://bugs.python.org/issue36085 for details on this change. -- my guess is add_dll_directory changes lookup for DLLs, but not for pyd?) sys.path appears to be used for locating pyd.
Yes, the pyd is compiled with python3.9: I've compiled it both with mingw and with visual studio toolchains, in case that might be a difference.
For fun, I moved python-standard _zoneinfo.pyd from Python39\DLLs and it fails in the same way in embedded python: "The parameter is incorrect". So, that would appear to rule out my specific pyd file.
The key question is/are:
Other than placing a pyd file under PythonXX\DLLs, is there a way to load a PYD in an embedded python implementation? (I want to avoid having to tell users to move my pyd file into the Python39\DLLs directory... because they'll forget.)
Note that using IDLE or python.exe, I can load pyds without error -- anywhere on sys.path -- so they don't have to be under Python39\DLLs. It's only when trying to load from embedded python that the "Parameter is incorrect" appears. And of course, this works flawlessly on Mac.
(Bonus question: what parameter? It appears to be python passing through a windows error.)

There seems to be a simple answer, though I suspect it's better characterized as a python bug.
There is nothing magical about Python39\DLLs directory.
The problem is using absolute vs relative paths in sys.path.
Python can find modules using absolute or relative paths. So if zippy.py is in folder foobar,
sys.path.append('foobar')
import zippy
# Success
Python and find, BUT NOT LOAD pyd files using relative paths. For example, move _zoneinfo.pyd from PythonXX\LDDs to foobar
sys.path.append('foobar')
import _zoneinfo
# ImportError: DLL load failed while importing _zoneinfo: The parameter is incorrect.'
Instead, use absolute path, and it will find and load PYD:
sys.path.append(r'c:\MyTest\foobar')
import _zoneinfo
# Success

So, there is actually a way to do this—that is, ship your application with the desired libraries. The solution is to use an embedded distribution and ship this with your application. You can find the correct distribution on the official Python download page corresponding to your desired version (here's the link to the lastest 3.9 release which seems to be what you're using: https://www.python.org/downloads/release/python-392/). Look for the Windows Embeddable Package.
You can then simply drop in your .pyd file alongside the standard library files (note that if your third-party library is dependent on any other libraries, you will have to include them, as well). Shipping your application with an embeddable distribution should not only solve your current issue, but will also mean that your application will work regardless of which version of Python a user has installed (or without having Python installed at all).

What does "Symbol not found / Expected in: flat namespace" actually mean?

When I import a module I built, I get this boost-python related error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: dlopen(./myMod.so, 2): Symbol not found: __ZN5boost6python7objects15function_objectERKNS1_11py_functionERKSt4pairIPKNS0_6detail7keywordES9_E
Referenced from: ./myMod.so
Expected in: flat namespace
in ./myMod.so
What does this actually mean? Why was this error raised?

Description
The problem was caused by mixing objects that compiled with libc++ and object that compiled with libstdc++.
In our case, the library myMod.so (compiled with libstdc++) need boost-python that compiled with libstdc++ (boost-python-libstdc++ from now). When boost-python is boost-python-libstdc++, it will work fine. Otherwise - on computer that its boost-python has compiled with libc++ (or another c++ library), it will have a problem loading and running it.
In our case, it happens because that libc++ developers intentionally changed the name of all of their symbols to prevent you (and save you) from mixing code from their library and code from a different one: myMod.so need a function that take an argument from the type. In libc++, this type's name is std::__1::pair. Therefore, this symbol was not found.
To understand why mixing two version of the same API is bad, consider this situation: There are two libraries: Foo and Bar. They both have a function that takes a std::string and uses it for something but they use a different c++ library. When a std::string that has been created by Foo will be passed to Bar, Bar will think that this is an instance of its c++ library's std::string and then bad things can happen (they are a completely different objects).
Note: In some cases, there would be no problem with two or more different versions of the same API in a completely different parts of a program. There will be a problem if they will pass this API's objects between them. However, checking that can be very hard, especially if they pass the API object only as a member of another object. Also, a library's initialization function can do things that should not happen twice. Another version may do these things again.
How to solve that?
You can always recompile your libraries and make them match each other.
You can link boost-python to your library as a static library. Then, it will work on almost every computer (even one that doesn't has boost-python installed). See more about that here.
Summary
myMod.so need another version of boost-python, one that compiled with a specific c++ library. Therefore, It would not work with any another version.

In my case I was receiving:
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/xmlsec.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_xmlSecDSigNs'
BACKGROUND:
M1 MacBook Pro with Montery
I was working with a python virtualenv (using pyenv) to use an earlier version of python3.8 (3.8.2), while my system had 3.8.10 installed natively.
While I was in the activated 3.8.2 virtualenv I noticed the path in dlopen() was pointing to the package in the native python install NOT the virtualenv install.
SOLUTION:
In my case, I did not need the native 3.8 version at all so I simply removed it and this solved the problem.

I encounter the same problem.
Expected in: flat namespace
Add the linker flag fixes the problem
-lboost_python37
change the dynamic library name to the one installed on the os.
By the way, my os is macOS High Sierra and I use brew to install boost_python3.

Symbol not found means the definition of the declared function or variable was not found. When a header file of a shared object is compiled with your program, linker adds symbols of declared functions and objects to your compiled program. When your program is loaded by the OS's loader, the symbols are resolved so that their definition will be loaded. It is only at this time where if the implementation is missing, loader complains it couldn't find the definition due to may be failing to resolve the actual path to the library or the library itself wasn't compiled with the implementation/source file where the definition of the function or object resides. There is a good article on this on the linux journal http://www.linuxjournal.com/article/6463.

In my case I was just failing to import all the required sources (c++ files) when compiling with Cython.
From the string after "Symbol not found" you can understand which library you are missing.

One of the solutions I found was to uninstall and reinstall it using the no-binary flag, which forces pip to compile the module from source instead of installing from precompiled wheel.
pip install --no-binary :all: <name-of-module>
Found this solution here

Here's what I've learned (osx):
If this is supposed to work (i.e. it works on another computer), you may be experiencing clang/gcc issues. To debug this, use otool -l on the .so file which is raising the error, or a suspect library (in my example it's a boost-python dylib file) and examine the contents. Anything in the /System/ folder is built with clang, and should be installed somewhere else with the gcc compiler. Never delete anything in the /System folder.

.so files are dynamic libraries (so = shared object). On Windows they are called .dll (dynamic-link library). They contain compiled code which contains functions available for usage to any executable which links them.
What is important to notice here is that those .so are not Python files. They were probably compiled from C or C++ code and contain public functions which can be used from Python code (see documentation on Extending Python with C or C++).
On your case, well, you have a corrupt .so. Try reinstalling the affected libraries, or Python, or both.

Problem
I had this same issue when running puma as part of Rails app
LoadError:
dlopen(/Users/alucard/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/puma-5.6.4/lib/puma/puma_http11.bundle, 0x0009): symbol not found in flat namespace '_ERR_load_crypto_strings'
/Users/alucard/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/puma-5.6.4/lib/puma/puma_http11.bundle
Solution
It was solved just by installing puma gem again gem install puma

py2exe fails to generate an executable

I am using python 2.6 on XP. I have just installed py2exe, and I can successfully create a simple hello.exe from a hello.py. However, when I try using py2exe on my real program, py2exe produces a few information messages but fails to generate anything in the dist folder.
My setup.py looks like this:
from distutils.core import setup
import py2exe
setup(console=['ServerManager.py'])
and the py2exe output looks like this:
python setup.py py2exe
running py2exe
creating C:\DevSource\Scripts\ServerManager\build
creating C:\DevSource\Scripts\ServerManager\build\bdist.win32
...
...
creating C:\DevSource\Scripts\ServerManager\dist
*** searching for required modules ***
*** parsing results ***
creating python loader for extension 'wx._misc_' (C:\Python26\lib\site-packages\wx-2.8-msw-unicode\wx\_misc_.pyd -> wx._misc_.pyd)
creating python loader for extension 'lxml.etree' (C:\Python26\lib\site-packages\lxml\etree.pyd -> lxml.etree.pyd)
...
...
creating python loader for extension 'bz2' (C:\Python26\DLLs\bz2.pyd -> bz2.pyd)
*** finding dlls needed ***
py2exe seems to have found all my imports (though I was a bit surprised to see win32 mentioned, as I am not explicitly importing it). Also, my program starts up quite happily with this command:
python ServerManager.py
Clearly I am doing something fundamentally wrong, but in the absence of any error messages from py2exe I have no idea what.

I put this in all my setup.py scripts:
distutils.core.setup(
options = {
"py2exe": {
"dll_excludes": ["MSVCP90.dll"]
}
},
...
)
This keeps py2exe quiet, but you still need to make sure that dll is on the user's machine.

I've discovered that py2exe works just fine if I comment out the part of my program that uses wxPython. Also, when I use py2exe on the 'simple' sample that comes with its download (i.e. in Python26\Lib\site-packages\py2exe\samples\simple), I get this error message:
*** finding dlls needed ***
error: MSVCP90.dll: No such file or directory
So something about wxPython makes py2exe think I need a Visual Studio 2008 DLL. I don't have VS2008, and yet my program works perfectly well as a directory of Python modules. I found a copy of MSVCP90.DLL on the web, installed it in Python26/DLLs, and py2exe now works fine.
I still don't understand where this dependency has come from, since I can run my code perfectly okay without py2exe. It's also annoying that py2exe didn't give me an error message like it did with the test_wx.py sample.
Further update: When I tried to run the output from py2exe on another PC, I discovered that it needed to have MSVCR90.DLL installed; so if your target PC hasn't got Visual C++ 2008 already installed, I recommend you download and install the Microsoft Visual C++ 2008 Redistributable Package.

wxPython has nothing to do with it. Before Python 2.6, Python used Visual Studio 2003 as their Windows compiler. Beginning with 2.6, they switched to Visual Studio 2008, which requires a manifest file in some situations. This has been well documented. See the following links:
http://wiki.wxpython.org/py2exe
http://py2exe.org/index.cgi/Tutorial#Step52
Also, if you're creating a wxPython application with py2exe, then you want to set the windows parameter, NOT the console one. Maybe my tutorial will help you:
http://www.blog.pythonlibrary.org/2010/07/31/a-py2exe-tutorial-build-a-binary-series/

It looks like this is only a dependency for Python 2.6. I wasn't getting this error under 2.5, but after the upgrade I am.
This email thread has some background for why the problem exists and how to fix it:
http://www.nabble.com/py2exe,-Py26,-wxPython-and-dll-td20556399.html
I didn't want to have to install the vcredist. My application currently requires no installation and can be run by non-administrators, which is behavior I don't want to lose. So I followed the suggestions in the links and got the necessary Microsoft.VC90.CRT.manifest and msvcr90.dll by installing Python "for this user only". I also needed msvcp90.dll that I found in the WinSxS folder of an "all users" Python 2.6 install. Since I already had two of the three, I included msvcm90.dll to prevent future errors though I didn't get any immediate errors when I left it out. I put the manifest and the three DLLs in the libs folder used by my frozen application.
The trick I had to perform was including an additional copy of the manifest and msvcr90.dll in the root of my application folder next to by py2exe generated executable. This copy of the DLL is used to bootstrap the application, but then it appears to only look in the libs folder.
Hopefully that discovery helps someone else out.
Also, I had the same problem with having py2exe log a real error message. Then I realized that stderr wasn't getting redirected into my log file. Add "> build.log 2>&1" on the command line where you invoke py2exe.

import sys
sys.path.append('C:\\WINDOWS\\WinSxS\\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2')
On each Windows, you can find the file MSVCP90.dll in some subdirectory in C:\\WINDOWS\\WinSxS\\
In my case, the directory was: x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2.
Go to C:\\WINDOWS\\WinSxS\\ and use windows file search to find MSVCP90.dll.

Just for your info, for me it worked to copy the files
Microsoft.VC90.CRT.manifest
msvcr90.dll
into the directory with the .exe on the user's machine (who has no python or VC redistributable installed).
Thanks for all the hints here!

The output says you're using WX. Try running py2exe with your script specified as a GUI app instead of console. If I'm not mistaken, that tends to cause problems with py2exe.

Try this: http://www.py2exe.org/index.cgi/Tutorial#Step52
It worked for me

There is some info on the wxPython wiki.
Deploy a Python app
py2exe with wxPython and Python 2.6

On my win8.1, I do not find the path
c:/Program Files/Microsoft Visual Studio 9.0/VC/redist/x86/Microsoft.VC90.CRT
On the contrary , the dll is found in
C:/WINDOWS/WinSxS/x86_Microsoft.VC90.CRT_XXXXXXX
The XXX may vary according to your PC
You may search in the path , then add the path in you setup.py
import sys
sys.path.append('C:/WINDOWS/WinSxS/x86_Microsoft.VC90.CRT_XXXXXXX')

import sys
sys.path.append('c:/Program Files/Microsoft Visual Studio 9.0/VC/redist/x86/Microsoft.VC90.CRT')