`pyusb` fails to find `libusb` when using `pyinstaller` - python
Problem
My application works well when in python-only and on my machine. I am attempting to deploy to a machine which has lesser privileges.
The basic script that I am attempting to run at the moment:
import logging
import os
os.environ['PYUSB_DEBUG'] = 'debug'
os.environ['PYUSB_LOG_FILENAME'] = 'C:\\path\to\log.txt'
logging.basicConfig(level=logging.DEBUG)
import usb.core
import usb.util
devices = [d for d in usb.core.find(find_all=True,
idVendor=0x0683,
idProduct=0x2008
)]
[print(d) for d in devices]
device = devices[0]
device.set_configuration()
The imports are a bit out of whack in this example file because the os.environ variable must be set before usb import or the logging won't be set up properly to get my log.txt from the usb module.
From Python
When I execute the above script on my machine, this is the resulting log.txt:
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,767 DEBUG:usb.backend.libusb1:_LibUSB.enumerate_devices()
...
...(more logs past this point, but you can see that the core found the backend)
From pyinstaller executable
I am bundling using pyinstaller. I normally like to use --onefile, but I always verify functionality without --onefile. I am only doing the most basic build: pyinstaller --noconfirm path/to/script.py
The console error:
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_libraryOSError: USB library could not be found
Traceback (most recent call last):
File "backend_test.py", line 15, in <module>
File "usb\core.py", line 1309, in find
usb.core.NoBackendError: No backend available
[9344] Failed to execute script backend_test
Note that the USB library failed to find a backend.
The fail log.txt when attempted to run the executable:
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,838 ERROR:usb.backend.libusb1:Error loading libusb 1.0 backend
Traceback (most recent call last):
File "usb\backend\libusb1.py", line 961, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,840 ERROR:usb.backend.openusb:Error loading OpenUSB backend
Traceback (most recent call last):
File "usb\backend\openusb.py", line 745, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,842 ERROR:usb.backend.libusb0:Error loading libusb 0.1 backend
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
At the moment, I can tell that the failure is occurring in a pyinstaller hook. I don't understand precisely how this works, so the clue may be found somewhere in there.
Background & Research
Versioning
libusb==1.0.24b3
pyinstaller==4.3
pyinstaller-hooks-contrib==2022.4
pyusb==1.2.1
... (there is more, but I suspect that these are the critical-to-function packages)
libusb
Not sure why this installed the beta version. Interesting, but doesn't appear to be the smoking gun.
pyinstaller
Note that pyinstaller==4.3 is a bit out of date. This is a result of my overambitious IT department's antivirus. It doesn't appear to want to allow more recent versions of pyinstaller to build executables.
pyinstaller-hooks-contrib
This is a repository which contains several pyinstaller hooks. I know that these are "critical to function" for lots of modules, but I don't really know how this works. It is possible, even likely given that this is where something is failing, that my problem lies within this library OR with this library's interaction with the version of pyinstaller that I'm using. Version 2022.4 is the version that is installed when I python -m pip install pyinstaller.
pyusb
Current version.
Other Attempts Thus Far
I have been up and down the web and have spent a particularly long time with the SO question Pyusb on windows - no backend available. Many of the below attempts were based on that SO question, but there have been other sources as well.
There are more attempts than just these, but I have been at it long enough that I don't remember them all!
Bundling libusb-1.0.dll into Executable
It is possible to bundle dll files directly with the pyinstaller-created script using the datas directive. This was my first attempt and it worked... on my machine. As soon as I deployed, the script failed with a PermissionError. I suspect that "untrusted" dll's aren't allowed to be utilized on the target machine. My login on my machine has elevated privileges, explaining why it may have worked in my environment.
Installing libusb-win32-devel-filter.exe.
Simply didn't work.
Including C:\\Windows\\System32 on PATH
I verified that the PATH variable contained C:\\Windows\\System32, which I also verified contains the proper dll file.
Specify backend
When I modify the script to specify the backend:
...
import usb.backend.libusb1 as libusb1
be = libusb1.get_backend(find_library=lambda x: "C:\\WINDOWS\\system32\\libusb-1.0.dll")
devices = [d for d in usb.core.find(find_all=True,
idVendor=0x0683,
idProduct=0x2008,
backend=be)]
...
There is no change in behavior. I do see in the log when running in python (not pyinstaller) that the backend is immediately found:
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.__init__(<WinDLL 'C:\Windows\System32\libusb-1.0.dll', handle 7ff840bd0000 at 0x190eadbe820>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.enumerate_devices()
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58EB0>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58F40>)
...
Install libusb
No change.
Modify pyinstaller and pyinstaller-hooks-contrib versions
Tried pyinstaller==4.10 and python-hooks-contrib==2022.3, which appear to be "compatible" releases.
I would like to try to build the pyinstaller script by excluding the pyusb from pyinstaller-hooks-contrib, but I'm not sure how to do this.
Other Clues....
While looking in pyinstaller-hooks-contrib, I'm seeing the way to access DLL files is through ctypes.WinDLL.
import ctypes
print('ctypes.WinDLL', ctypes.WinDLL('libusb-1.0.dll'))
Result when running from python: <WinDLL 'libusb-1.0.dll', handle 7ff831cc0000 at 0x21076e9adf0>
Result when running from pyinstaller: ctypes.WinDLL <PyInstallerWinDLL 'libusb-1.0.dll', handle 7ff83d400000 at 0x201a9f39fa0>
I'm still not sure why this works the way that it does, but my initial instincts were correct when I had tried to include libusb-1.0.dll in the pyinstaller package. Unfortunately, there is apparently an issue with the windows implementation of the libusb-1.0.dll (I'm not making that assertion, I'm quoting another SO answer).
The answer, for me, was to add libusb0.dll to my pyinstaller build directory instead of the more recent libusb-1.0.dll:
a = Analysis(['examples\\backend_test.py'],
pathex=[],
binaries=[],
datas=[('C:\\Windows\\System32\\libusb0.dll', '.'),],
hiddenimports=[],
...])
I can't believe that it was this simple in my case, but there it is...
Related
Pyinstaller fails to find kernel32 dll
Currently trying to build a exe using PyInstaller. The installation goes fine, although when trying to run the exe an exception occurs Exception ignored in: <function Library.__del__ at 0x000002126637F160> Traceback (most recent call last): File "C:\Users\blah\AppData\Local\Programs\Python\Python39\lib\site-packages\pylink\library.py", line 272, in __del__ File "C:\Users\blah\AppData\Local\Programs\Python\Python39\lib\site-packages\pylink\library.py", line 390, in unload File "C:\Users\blah\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 444, in __getattr__ File "PyInstaller\loader\pyimod04_ctypes.py", line 79, in __init__ pyimod04_ctypes.install.<locals>.PyInstallerImportError: Failed to load dynlib/dll 'kernel32'. Most likely this dynlib/dll was not found when the application was frozen. I've followed the lead on some other posts and ensured the Visual C++ Redistributables are installed and also included --hidden-import ctypes in the build to no avail. Building on Windows 10 - Python 3.9 - PyInstaller 5.0.1 Any help would be great!
Taken from the comments on the original answer. Quote stevod: I ended up finding the problem, our company anti virus wouldn't let the bootloader access the required files, so we had to rebuild the bootloader manually and then it worked! The same worked also for me.
Security issues downloading application for MacOS - "library load disallowed by system policy"
I recently created an application using Pyinstaller, and have bundled it correctly. It theoretically should be able to work on other macbooks now. However, I have tried zipping the application, storing it on Google Drive, and then downloading and running it on my own and other macbooks. I have hit a security issue: Traceback (most recent call last): File "_pyinstaller_hooks_contrib/hooks/rthooks/pyi_rth_certifi.py", line 13, in <module> File "PyInstaller/loader/pyimod03_importers.py", line 531, in exec_module File "ssl.py", line 98, in <module> ImportError: dlopen(/Users/a/Downloads/main/_ssl.cpython-37m-darwin.so, 2): no suitable image found. Did find: /Users/a/Downloads/main/_ssl.cpython-37m-darwin.so: code signature in (/Users/a/Downloads/main/_ssl.cpython-37m-darwin.so) not valid for use in process using Library Validation: library load disallowed by system policy [50024] Failed to execute script pyi_rth_certifi Any ideas on how to overcome this would be really appreciated! Thank you.
I have resolved the issue and thought I'd share my answer! Before distributing the app, on MacOS you have to codesign it. This is the site that helped me: https://github.com/pyinstaller/pyinstaller/wiki/Recipe-OSX-Code-Signing. Hope this eventually helps someone.
Python code works on interpreter but not when I use pyinstaller
I am creating a bot and I am using libs like pandas, numpy and sklearn and I have also installed all these libraries in a virtual env. The bot works fine on Pycharm but when I use pyinstaller and create a standalone, it's standalone shows this error: Traceback (most recent call last): File "PyInstaller\loader\pyiboot01_bootstrap.py", line 167, in __init__ File "ctypes\__init__.py", line 373, in __init__ FileNotFoundError: Could not find module 'C:\Users\vabha\AppData\Local\Temp\_MEI38962\sklearn\.libs\vcomp140.dll' (or one of its dependencies). Try using the full path with constructor syntax. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "chandlerbing.py", line 3, in <module> from sklearn.pipeline import Pipeline . . .
Console debbug say FileNotFoundError... Let me translate it for you. vcomp140.dll is missing and need to be used you had it on temporary location and now is not any more. Try also look on this answer or you can try also Auto Py To Exe maybe will work.
You need to have all the libraries installed in the same environment as pyinstaller. You probably installed pyinstaller in a global environment so you should install the libraries in the globa environment.
Is it possible to use the python3 bindings for VirtualBox?
I am trying to use the python 3 bindings to VirtualBox but there appears to be broken dependencies. It seems odd to me that this hasn't been fixed over the ~4 years that people have been having this issue. Perhaps I'm missing something obvious. It's been known to happen. I have installed the virtualbox host modules, sdk, and extensions through my OS's pacakage manager. Then, through pip: pip install pyvbox The imports work: from virtualbox import VirtualBox, Session, Manager, WebServiceManager But then any attempt to instantiate anything results in an exception complaining about a missing vboxapi. box = VirtualBox() Traceback: Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/lib/python3.6/site-packages/virtualbox/library_ext/vbox.py", line 22, in __init__ manager = virtualbox.Manager() File "/usr/lib/python3.6/site-packages/virtualbox/__init__.py", line 130, in __init__ with import_vboxapi() as vboxapi: File "/usr/lib/python3.6/contextlib.py", line 82, in __enter__ return next(self.gen) File "/usr/lib/python3.6/site-packages/virtualbox/__init__.py", line 45, in import_vboxapi import vboxapi File "/home/$USER/.eclipse/org.eclipse.platform_4.6.3_155965261_linux_gtk_x86_64/plugins/org.python.pydev_5.7.0.201704111357/pysrc/_pydev_bundle/pydev_import_hook.py", line 20, in do_import module = self._system_import(name, *args, **kwargs) ModuleNotFoundError: No module named 'vboxapi' There is a vboxapi on PyPi, but it won't install as there is no code associated with it, nor any useful information on the PyPi page: https://pypi.python.org/pypi/vboxapi Here are a couple links to the valiant efforts of braver souls than I. It is not immediately clear to me which is the correct solution or if either are still relevant, given that they are from 3 and 4 years ago, respectively. https://github.com/GreatFruitOmsk/vboxapi-py3 https://github.com/jbuergel/vboxapi-py3 Also from 3 years ago, word of a vboxapi.diff and intergration into vboxapi: https://www.virtualbox.org/pipermail/vbox-dev/2014-April/012231.html
I'm the current maintainer of the pyvbox package. The VirtualBox SDK already supports Python 3, I use Python 3.5 to develop the library. I recommend uninstalling and reinstalling the latest version of the SDK (which at the time of writing this is 5.1.22). You can find the SDK on the VirtualBox downloads page. Unzip the archive and run the vboxapisetup.py file using your system Python with the following command: python vboxapisetup.py install You don't need to install this in any virtualenv, as pyvbox will search your system libraries in addition to virtualenv installations for better ease of use. If you have problems using the pyvbox package after running these steps, please open an issue and include as much information as possible including the steps you took, OS, where your system Python is located, which version of VirtualBox & SDK you're using, and I'll help you as best I can.
Yes you can, it is possible, very tricky to setup but it work fine for me now (Ubuntu 18.04 / python3.6 / virtualbox 6.0) .
The error:
ModuleNotFoundError: No module named 'vboxapi'
mean that python3 does not find vboxapi module, now there is two methods to "force-install" the vboxapi package to python3:
First Method [easy]: Assuming pyvbox is already installed and work fine with python2.7, in that case you can simply copy the package from python2.7 dist-packages to python3 dist-package with:
sudo cp -r /usr/lib/python2.7/dist-packages/vboxapi /usr/lib/python3/dist-packages
Second method [more tricky]: Go to VirtualBox, then download the last Software Developer Kit (SDK), actually the 6.0.4
Unzip the archive and run the vboxapisetup.py file using Python3 with the following command:
sudo python3 vboxapisetup.py install
You will get this issue:
Traceback (most recent call last):
File "vboxapisetup.py", line 90, in <module>
main(sys.argv)
File "vboxapisetup.py", line 63, in main
raise Exception("No VBOX_INSTALL_PATH defined, exiting")
Exception: No VBOX_INSTALL_PATH defined, exiting
You may directly edit the current file vboxapisetup.py and replace line 57, from vboxDest = os.environ.get("VBOX_MSI_INSTALL_PATH", None) to vboxDest = "/usr/lib/virtualbox"
Then run agin:
sudo python3 vboxapisetup.py install
And now you will get something like that:
running install
running build
running build_py
copying vboxapi/__init__.py -> build/lib/vboxapi
running install_lib
creating /usr/local/lib/python3.6/dist-packages/vboxapi
copying build/lib/vboxapi/__init__.py -> /usr/local/lib/python3.6/dist-packages/vboxapi
copying build/lib/vboxapi/VirtualBox_constants.py -> /usr/local/lib/python3.6/dist-packages/vboxapi
byte-compiling /usr/local/lib/python3.6/dist-packages/vboxapi/__init__.py to __init__.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/vboxapi/VirtualBox_constants.py to VirtualBox_constants.cpython-36.pyc
running install_egg_info
Removing /usr/local/lib/python3.6/dist-packages/vboxapi-1.0.egg-info
Writing /usr/local/lib/python3.6/dist-packages/vboxapi-1.0.egg-info
which mean that we are ok with vboxapi package installation !
Now, let's try again to load virtualbox() inside python3:
from virtualbox import VirtualBox, Session, Manager, WebServiceManager
box = VirtualBox()
this probably will raise this new issue:
Traceback (most recent call last):
File "virtualbox_python3_test.py", line XX, in <module>
vbox = virtualbox.VirtualBox()
File "/usr/local/lib/python3.6/dist-packages/virtualbox/library_ext/vbox.py", line 22, in __init__
manager = virtualbox.Manager()
File "/usr/local/lib/python3.6/dist-packages/virtualbox/__init__.py", line 143, in __init__
self.manager = vboxapi.VirtualBoxManager(mtype, mparams)
File "/usr/local/lib/python3.6/dist-packages/vboxapi/__init__.py", line 989, in __init__
self.platform = PlatformXPCOM(dPlatformParams)
File "/usr/local/lib/python3.6/dist-packages/vboxapi/__init__.py", line 750, in __init__
import xpcom.vboxxpcom
File "/usr/lib/virtualbox/sdk/bindings/xpcom/python/xpcom/vboxxpcom.py", line 78, in <module>
raise Exception('Cannot find VBoxPython module (tried: %s)' % (', '.join(_asVBoxPythons),))
Exception: Cannot find VBoxPython module (tried: VBoxPython3_6m, VBoxPython3m, VBoxPython)
If you dig you will find a lot of questions (question 1,question 2,question 3, question 4 etc...) relative to this issue on the web ...
But according to my dig & research, if you are lucky (and have a Virtualbox built with python3 native support) you can try:
cd /usr/lib/virtualbox/
sudo cp VBoxPython3_5m.so VBoxPython3_6m.so
But if you got the following error:
cp: cannot stat 'VBoxPython3_5m.so': No such file or directory
It mean that you don't have native python3 support in Virtualbox...
This could be solved like this:
Go here and download the python3-virtualbox-5.2.16 binary package (we don't care about the VirtualBox version...)
Now open python3-virtualbox-5.2.16-lp150.4.11.1.x86_64.rpm archive, browse it to /./usr/lib/virtualbox/, then extract the file VBoxPython3_6m.so, then drop this file in your current working directory, after that from this directory you have to do:
sudo cp VBoxPython3_6m.so /usr/lib/virtualbox/
And now, you can use python3 binding for virtualbox !
Solve a ValueError
I was looking a way to prove my USB-SERIAL port and I found this code:
http://www.digitalmihailo.com/usb-programming-with-python-on-linux-pyusb-version/, but I have a problem because when I run this program, then show me up this message:
Traceback (most recent call last):
File "namefile.py", line 122 in <module>
main()
File "namefile.py", line 64, in main
raise ValueError('Device not found')
ValueError: Device not found
I already install the PyUSB library, but the result does not change. Someone can help me with this little problem.
Thank you much in advance.
If you are running your example in Linux you need to load the ACMtty module so you system makes a Serial USB device available for your user space. As root and only if you have cdc-acm module compiled for your current kernel: modprobe cdc-acm Once this module is loaded and your device connected you should find a device named following this pattern: /dev/ttyACM* You may have already installed the right user space tools and libraries but you also need to install the device.