We Keep Coding

`pyusb` fails to find `libusb` when using `pyinstaller`

Problem
My application works well when in python-only and on my machine. I am attempting to deploy to a machine which has lesser privileges.
The basic script that I am attempting to run at the moment:
import logging
import os
os.environ['PYUSB_DEBUG'] = 'debug'
os.environ['PYUSB_LOG_FILENAME'] = 'C:\\path\to\log.txt'
logging.basicConfig(level=logging.DEBUG)
import usb.core
import usb.util
devices = [d for d in usb.core.find(find_all=True,
idVendor=0x0683,
idProduct=0x2008
)]
[print(d) for d in devices]
device = devices[0]
device.set_configuration()
The imports are a bit out of whack in this example file because the os.environ variable must be set before usb import or the logging won't be set up properly to get my log.txt from the usb module.
From Python
When I execute the above script on my machine, this is the resulting log.txt:
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,767 DEBUG:usb.backend.libusb1:_LibUSB.enumerate_devices()
...
...(more logs past this point, but you can see that the core found the backend)
From pyinstaller executable
I am bundling using pyinstaller. I normally like to use --onefile, but I always verify functionality without --onefile. I am only doing the most basic build: pyinstaller --noconfirm path/to/script.py
The console error:
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_libraryOSError: USB library could not be found
Traceback (most recent call last):
File "backend_test.py", line 15, in <module>
File "usb\core.py", line 1309, in find
usb.core.NoBackendError: No backend available
[9344] Failed to execute script backend_test
Note that the USB library failed to find a backend.
The fail log.txt when attempted to run the executable:
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,838 ERROR:usb.backend.libusb1:Error loading libusb 1.0 backend
Traceback (most recent call last):
File "usb\backend\libusb1.py", line 961, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,840 ERROR:usb.backend.openusb:Error loading OpenUSB backend
Traceback (most recent call last):
File "usb\backend\openusb.py", line 745, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,842 ERROR:usb.backend.libusb0:Error loading libusb 0.1 backend
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
At the moment, I can tell that the failure is occurring in a pyinstaller hook. I don't understand precisely how this works, so the clue may be found somewhere in there.
Background & Research
Versioning
libusb==1.0.24b3
pyinstaller==4.3
pyinstaller-hooks-contrib==2022.4
pyusb==1.2.1
... (there is more, but I suspect that these are the critical-to-function packages)
libusb
Not sure why this installed the beta version. Interesting, but doesn't appear to be the smoking gun.
pyinstaller
Note that pyinstaller==4.3 is a bit out of date. This is a result of my overambitious IT department's antivirus. It doesn't appear to want to allow more recent versions of pyinstaller to build executables.
pyinstaller-hooks-contrib
This is a repository which contains several pyinstaller hooks. I know that these are "critical to function" for lots of modules, but I don't really know how this works. It is possible, even likely given that this is where something is failing, that my problem lies within this library OR with this library's interaction with the version of pyinstaller that I'm using. Version 2022.4 is the version that is installed when I python -m pip install pyinstaller.
pyusb
Current version.
Other Attempts Thus Far
I have been up and down the web and have spent a particularly long time with the SO question Pyusb on windows - no backend available. Many of the below attempts were based on that SO question, but there have been other sources as well.
There are more attempts than just these, but I have been at it long enough that I don't remember them all!
Bundling libusb-1.0.dll into Executable
It is possible to bundle dll files directly with the pyinstaller-created script using the datas directive. This was my first attempt and it worked... on my machine. As soon as I deployed, the script failed with a PermissionError. I suspect that "untrusted" dll's aren't allowed to be utilized on the target machine. My login on my machine has elevated privileges, explaining why it may have worked in my environment.
Installing libusb-win32-devel-filter.exe.
Simply didn't work.
Including C:\\Windows\\System32 on PATH
I verified that the PATH variable contained C:\\Windows\\System32, which I also verified contains the proper dll file.
Specify backend
When I modify the script to specify the backend:
...
import usb.backend.libusb1 as libusb1
be = libusb1.get_backend(find_library=lambda x: "C:\\WINDOWS\\system32\\libusb-1.0.dll")
devices = [d for d in usb.core.find(find_all=True,
idVendor=0x0683,
idProduct=0x2008,
backend=be)]
...
There is no change in behavior. I do see in the log when running in python (not pyinstaller) that the backend is immediately found:
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.__init__(<WinDLL 'C:\Windows\System32\libusb-1.0.dll', handle 7ff840bd0000 at 0x190eadbe820>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.enumerate_devices()
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58EB0>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58F40>)
...
Install libusb
No change.
Modify pyinstaller and pyinstaller-hooks-contrib versions
Tried pyinstaller==4.10 and python-hooks-contrib==2022.3, which appear to be "compatible" releases.
I would like to try to build the pyinstaller script by excluding the pyusb from pyinstaller-hooks-contrib, but I'm not sure how to do this.
Other Clues....
While looking in pyinstaller-hooks-contrib, I'm seeing the way to access DLL files is through ctypes.WinDLL.
import ctypes
print('ctypes.WinDLL', ctypes.WinDLL('libusb-1.0.dll'))
Result when running from python: <WinDLL 'libusb-1.0.dll', handle 7ff831cc0000 at 0x21076e9adf0>
Result when running from pyinstaller: ctypes.WinDLL <PyInstallerWinDLL 'libusb-1.0.dll', handle 7ff83d400000 at 0x201a9f39fa0>

I'm still not sure why this works the way that it does, but my initial instincts were correct when I had tried to include libusb-1.0.dll in the pyinstaller package. Unfortunately, there is apparently an issue with the windows implementation of the libusb-1.0.dll (I'm not making that assertion, I'm quoting another SO answer).
The answer, for me, was to add libusb0.dll to my pyinstaller build directory instead of the more recent libusb-1.0.dll:
a = Analysis(['examples\\backend_test.py'],
pathex=[],
binaries=[],
datas=[('C:\\Windows\\System32\\libusb0.dll', '.'),],
hiddenimports=[],
...])
I can't believe that it was this simple in my case, but there it is...

Installing Gumbo and its Python wrapper

I am trying to use Gumbo's python wrapper to parse HTML.
My operating system is Ubuntu 14.04.3 LTS. I am using Python2.7
I have cloned the latest version of Gumbo from github.
I followed the installation steps provided on Github.
The installation of the libraries, both the C library and the python wrapper, appeared to be successful ( No error message, both printed successful messages at the end )
C library final message:
Libraries have been installed in: /usr/local/lib
Python wrapper message:
Installed
/usr/local/lib/python2.7/dist-packages/gumbo-0.10.1-py2.7.egg
Processing dependencies for gumbo==0.10.1 Finished processing
dependencies for gumbo==0.10.1
The first problem I encountered was when I tried to open pydoc for gumbo, to better understand the library.
pydoc gumbo produced the following error:
problem in gumbo - <type 'exceptions.OSError'>: /usr/local/lib/python2.7/dist-packages/gumbo-0.10.1-py2.7.egg/gumbo/libgumbo.so: cannot open shared object file: No such file or directory
Searching for the message yielded a single result.
It was not of much use to me.
Looking at the dist-packages directory, I noticed that libgumbo.so was not in /usr/local/lib/python2.7/dist-packages/gumbo-0.10.1-py2.7.egg/gumbo/ . All other files ( soup-adapter.py, gumboc.py, etc ) where there however.
The installation of the C library placed libgumbo.so ( and some other libraries, like libgumbo.a libgumbo.la, etc ) in /usr/local/lib. So, as a work around, I created a simlink from .../dist-packages/gumbo-0.10.1-py2.7.egg/gumbo/ to /usr/local/lib.
This got pydoc gumbo to work.
I tried to import gumbo and soup-adapter in the interpreter after. I received the following error:
import soup_adapter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "soup_adapter.py", line 26, in <module>
import gumboc
File "gumboc.py", line 44, in <module>
os.path.dirname(__file__), _name_of_lib))
File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libgumbo.so: cannot open shared object file: No such file or directory
I am not sure how to proceed or how exactly to get gumbo to work.

py2exe won't properly compile my application

I've never used py2exe before. I installed the package, created a setup.py, ran it and it seemed to work without any obvious errors. I went into the dist folder it created and tried to run the exe but I'm getting this error:
Traceback (most recent call last):
File "rl2.py", line 3, in <module>
File "libtcodpy.pyc", line 41, in <module>
File "ctypes\__init__.pyc", line 428, in __getitem__
File "ctypes\__init__.pyc", line 423, in __getattr__
File "ctypes\__init__.pyc", line 353, in __init__
WindowsError: [Error 126] The specified module could not be found
How can I solve this?

What exact commands did you use to get this output?
I recommend using PyInstaller.
You can build an EXE with PyInstaller using this command:
"python pyinstaller.py -y -w -F [PATH\TO\SCRIPT.py] -n [PROJECT_NAME]"
-y: answer yes to every question
-w: windowed (does not show commandline when running)
-F: Compile into one EXE instead of a directory with .dll files
-n: assign project name
Hope I helped

You probably need to manually specify ctypes in the packages section of your setup.py.
See here

I had same issue today. HEre is the solution will work 100%
The meaning of error is the imported dll file in your code is misssng in created dist folder. so you have to copy and paste dll file in dist folder, and it will work.
I used logimove.dll file and same issue. so i put logimove.dll file in dist file and it works. :)

py2exe + pywin32 MemoryLoadLibrary import fail when bundle_files=1

I have created a simple program which uses pywin32. I want to deploy it as an executable, so I py2exe'd it. I also didn't want a huge amount of files, so I set bundle_files to 1 (meaning bundle everything together). However, when I attempt running it, I get:
Traceback (most recent call last):
File "pshelper.py", line 4, in <module>
File "zipextimporter.pyc", line 82, in load_module
File "win32.pyc", line 8, in <module>
File "zipextimporter.pyc", line 98, in load_module
ImportError: MemoryLoadLibrary failed loading win32ui.pyd
In my setup script, I tried doing packages=["win32ui"] and includes=["win32ui"] as options, but that didn't help. How can I get py2exe to include win32ui.pyd?
I don't have this problem if I don't ask it to bundle the files, so I can do that, for now, but I'd like to know how to get it to work properly.

Are you sure that the problem is that win32ui.pyd is not included? The stack trace isn't exactly the same as noted in the wiki, but check this out: http://www.py2exe.org/index.cgi/Py2exeAndWin32ui.

The work-around that has worked best so far is to simply re-implement the pywin32 functions using ctypes. That doesn't require another .pyd or .dll file so the issue is obviated.

Do care to try out PyInstaller? I've used it both on Windows 7 and Ubuntu 10.04 and it worked like magic, even when I compiled to .pyd on Windows.
I've been able to bundle all sorts of applications that I've developed with it.

I have the same problem trying to bundle psutil with py2exe. Here is what I found so far.
Traceback (most recent call last):
File "wx_gui.py", line 43, in <module>
File "zipextimporter.pyc", line 82, in load_module
File "psutil\__init__.pyc", line 85, in <module>
File "zipextimporter.pyc", line 82, in load_module
File "psutil\_psmswindows.pyc", line 15, in <module>
File "zipextimporter.pyc", line 98, in load_module
ImportError: MemoryLoadLibrary failed loading _psutil_mswindows.pyd
I get this traceback when bundle_files = 3. zipfile may be specified or may be None, I still get the problem.
First, I thought this was a missing dll because of this page:
http://www.py2exe.org/index.cgi/ProblemsToBeFixed
I've copied all the dlls I found in Python27 into the same directory as the executable and added that directory path to os.environ['path']. That didn't worked.
Then I tried to import my package directly from site-packages.
I've replaced the whole sys.path of my compiled executable with my normal sys.path
sys.path = [r'C:\Python27\Lib\idlelib', ...]
I think the .pyd module got imported but Visual c++ threw me this really ugly error message:
Runtime Error!
Program: {path}.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I suggest you try to replace the whole sys.path to see if it is working. If it is, you could always make your single executable write the necessary module into a temp directory and add it to your path. If not, I feel like this problem is going to be hard to solve.

How to fix this dll loading python error?

I use one c++ dll in my python code.
When I run my python app on my computer, it works fine but when I copy all to another computer this happen:
Traceback (most recent call last):
File "C:\users\Public\SoundLog\Code\Código Python\SoundLog\SoundLog.py", line 9, in <module>
from Auxiliar import *
File "C:\users\Public\SoundLog\Code\Código Python\SoundLog\Auxiliar\DataCollection.py", line 4, in <module>
import SoundLogDLL
File "C:\users\Public\SoundLog\Code\Código Python\SoundLog\Auxiliar\SoundLogDLL.py", line 4, in <module>
dll = cdll.LoadLibrary(os.environ['PUBLIC'] + "\\SoundLog\\DLLs\\ForPython\\SoundLogC++WrapperDLL.dll")
File "C:\Python26\lib\ctypes\__init__.py", line 431, in LoadLibrary
return self._dlltype(name)
File "C:\Python26\lib\ctypes\__init__.py", line 353, in __init__
self._handle = _dlopen(self._name, mode)
WindowsError: [Error 14001] The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail
How can I fix this problem so that my python app work in every computer?
Note:
I only need them to work on windows 7 like mine. And I tested on another win7 computer (both 32bits).

You're using a DLL which depends on a Microsoft Visual C++ runtime which isn't installed on the target computer. You have a few options:
Install or copy the Visual C++ runtime libraries to the target computer. Installation is done by adding merge modules to your installer (if you have one) or by running the redistributable installer (vcredist.exe). The various methods are documented here for VS2005 (other versions will be similar).
If you built the DLL, you can change the project settings to statically link the runtime libraries. See this answer to see how this is done.