We Keep Coding

`pyusb` fails to find `libusb` when using `pyinstaller`

Problem
My application works well when in python-only and on my machine. I am attempting to deploy to a machine which has lesser privileges.
The basic script that I am attempting to run at the moment:
import logging
import os
os.environ['PYUSB_DEBUG'] = 'debug'
os.environ['PYUSB_LOG_FILENAME'] = 'C:\\path\to\log.txt'
logging.basicConfig(level=logging.DEBUG)
import usb.core
import usb.util
devices = [d for d in usb.core.find(find_all=True,
idVendor=0x0683,
idProduct=0x2008
)]
[print(d) for d in devices]
device = devices[0]
device.set_configuration()
The imports are a bit out of whack in this example file because the os.environ variable must be set before usb import or the logging won't be set up properly to get my log.txt from the usb module.
From Python
When I execute the above script on my machine, this is the resulting log.txt:
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,767 DEBUG:usb.backend.libusb1:_LibUSB.enumerate_devices()
...
...(more logs past this point, but you can see that the core found the backend)
From pyinstaller executable
I am bundling using pyinstaller. I normally like to use --onefile, but I always verify functionality without --onefile. I am only doing the most basic build: pyinstaller --noconfirm path/to/script.py
The console error:
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_libraryOSError: USB library could not be found
Traceback (most recent call last):
File "backend_test.py", line 15, in <module>
File "usb\core.py", line 1309, in find
usb.core.NoBackendError: No backend available
[9344] Failed to execute script backend_test
Note that the USB library failed to find a backend.
The fail log.txt when attempted to run the executable:
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,838 ERROR:usb.backend.libusb1:Error loading libusb 1.0 backend
Traceback (most recent call last):
File "usb\backend\libusb1.py", line 961, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,840 ERROR:usb.backend.openusb:Error loading OpenUSB backend
Traceback (most recent call last):
File "usb\backend\openusb.py", line 745, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,842 ERROR:usb.backend.libusb0:Error loading libusb 0.1 backend
Traceback (most recent call last):
File "usb\backend\libusb0.py", line 738, in get_backend
File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
At the moment, I can tell that the failure is occurring in a pyinstaller hook. I don't understand precisely how this works, so the clue may be found somewhere in there.
Background & Research
Versioning
libusb==1.0.24b3
pyinstaller==4.3
pyinstaller-hooks-contrib==2022.4
pyusb==1.2.1
... (there is more, but I suspect that these are the critical-to-function packages)
libusb
Not sure why this installed the beta version. Interesting, but doesn't appear to be the smoking gun.
pyinstaller
Note that pyinstaller==4.3 is a bit out of date. This is a result of my overambitious IT department's antivirus. It doesn't appear to want to allow more recent versions of pyinstaller to build executables.
pyinstaller-hooks-contrib
This is a repository which contains several pyinstaller hooks. I know that these are "critical to function" for lots of modules, but I don't really know how this works. It is possible, even likely given that this is where something is failing, that my problem lies within this library OR with this library's interaction with the version of pyinstaller that I'm using. Version 2022.4 is the version that is installed when I python -m pip install pyinstaller.
pyusb
Current version.
Other Attempts Thus Far
I have been up and down the web and have spent a particularly long time with the SO question Pyusb on windows - no backend available. Many of the below attempts were based on that SO question, but there have been other sources as well.
There are more attempts than just these, but I have been at it long enough that I don't remember them all!
Bundling libusb-1.0.dll into Executable
It is possible to bundle dll files directly with the pyinstaller-created script using the datas directive. This was my first attempt and it worked... on my machine. As soon as I deployed, the script failed with a PermissionError. I suspect that "untrusted" dll's aren't allowed to be utilized on the target machine. My login on my machine has elevated privileges, explaining why it may have worked in my environment.
Installing libusb-win32-devel-filter.exe.
Simply didn't work.
Including C:\\Windows\\System32 on PATH
I verified that the PATH variable contained C:\\Windows\\System32, which I also verified contains the proper dll file.
Specify backend
When I modify the script to specify the backend:
...
import usb.backend.libusb1 as libusb1
be = libusb1.get_backend(find_library=lambda x: "C:\\WINDOWS\\system32\\libusb-1.0.dll")
devices = [d for d in usb.core.find(find_all=True,
idVendor=0x0683,
idProduct=0x2008,
backend=be)]
...
There is no change in behavior. I do see in the log when running in python (not pyinstaller) that the backend is immediately found:
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.__init__(<WinDLL 'C:\Windows\System32\libusb-1.0.dll', handle 7ff840bd0000 at 0x190eadbe820>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.enumerate_devices()
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58EB0>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58F40>)
...
Install libusb
No change.
Modify pyinstaller and pyinstaller-hooks-contrib versions
Tried pyinstaller==4.10 and python-hooks-contrib==2022.3, which appear to be "compatible" releases.
I would like to try to build the pyinstaller script by excluding the pyusb from pyinstaller-hooks-contrib, but I'm not sure how to do this.
Other Clues....
While looking in pyinstaller-hooks-contrib, I'm seeing the way to access DLL files is through ctypes.WinDLL.
import ctypes
print('ctypes.WinDLL', ctypes.WinDLL('libusb-1.0.dll'))
Result when running from python: <WinDLL 'libusb-1.0.dll', handle 7ff831cc0000 at 0x21076e9adf0>
Result when running from pyinstaller: ctypes.WinDLL <PyInstallerWinDLL 'libusb-1.0.dll', handle 7ff83d400000 at 0x201a9f39fa0>

I'm still not sure why this works the way that it does, but my initial instincts were correct when I had tried to include libusb-1.0.dll in the pyinstaller package. Unfortunately, there is apparently an issue with the windows implementation of the libusb-1.0.dll (I'm not making that assertion, I'm quoting another SO answer).
The answer, for me, was to add libusb0.dll to my pyinstaller build directory instead of the more recent libusb-1.0.dll:
a = Analysis(['examples\\backend_test.py'],
pathex=[],
binaries=[],
datas=[('C:\\Windows\\System32\\libusb0.dll', '.'),],
hiddenimports=[],
...])
I can't believe that it was this simple in my case, but there it is...

Writing a network Scanner with python using scapy

I am writing a simple network scanner with python using scapy following is my code :
import scapy.all as scapy
def scan(ip):
scapy.arping(ip)
scan("192.168.1.1/24")
Error I am getting :
Traceback (most recent call last):
File "ipScanner.py", line 10, in <module>
scan("192.168.1.1/24")
File "ipScanner.py", line 8, in scan
scapy.arping(ip)
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/layers/l2.py", line 648, in arping
filter="arp and arp[7] = 2", timeout=timeout, iface_hint=net, **kargs) # noqa: E501
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/sendrecv.py", line 553, in srp
filter=filter, nofilter=nofilter, type=type)
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/arch/bpf/supersocket.py", line 242, in __init__
super(L2bpfListenSocket, self).__init__(*args, **kwargs)
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/arch/bpf/supersocket.py", line 62, in __init__
(self.ins, self.dev_bpf) = get_dev_bpf()
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/arch/bpf/core.py", line 114, in get_dev_bpf
raise Scapy_Exception("No /dev/bpf handle is available !")
scapy.error.Scapy_Exception: No /dev/bpf handle is available !
Exception ignored in: <function _L2bpfSocket.__del__ at 0x105984c20>
Traceback (most recent call last):
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/arch/bpf/supersocket.py", line 139, in __del__
self.close()
File "/Users/omairkhan/opt/anaconda3/lib/python3.7/site-packages/scapy/arch/bpf/supersocket.py", line 211, in close
if not self.closed and self.ins is not None:
AttributeError: 'L2bpfSocket' object has no attribute 'ins'
Can anyone please help understand it.
NOTE: I am running it on mac OS.

I wrote this exact program when I first started programming with matching syntax, and it ran correctly on my systems when run as administrator. I develop on Linux and Windows rather than Mac, but I will offer what I can.
Are you running this script through your IDE or calling it from the shell?
I recommend only running it from the shell. This simply gives you more control over the files like specifying which version of python the script is, and if you need administrative privileges for a script, you can elevate the script permissions in the shell.
Also, in my OS, I was taught to always use, and have experienced the mistakes of forgetting this, always add:
#!/usr/bin/env python
as the first line of every script. At least in Linux, it tells the PC how to treat the file (it tells it to treat the file as a python file--yes I acknowledge that its already running it as python). I would check to see if that is valid for MacOS file system.
Most of what I have recommended so far comes down to no /dev/bpf handle is available, only ever being an issue for me when I'm not running script as an administrator (although Linux states permission denied). And I shouldn't leave out that using Anaconda on Windows in the past (before I understood the structure of my file systems) prevented me from using common modules like pygame and scapy. I could only guess in that case Anaconda prevented the PC from knowing where to find every piece of that module by making the computer think it had its own one of that module under Anaconda directory when it was in a different PATH.

Python shell working differently on different location in the same machine inside same virtual environment

I have a cloud instance of a Linux machine (openSuSE) with multiple users.
I have created a virtual environment and installed all my required libraries (including Klein).
I have two users "a" and "b".
While logged in as "a" and inside virtualenv, when I open python shell at home directory and type
import klein
it imports normally.
Now when I change directory to
/home/b/
and run the same (open python shell, import klein) while being in the same virtualenv, it gives me an error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/a/.local/lib/python3.6/site-packages/klein/__init__.py", line 3, in <module>
from klein._plating import Plating
File "/home/a/.local/lib/python3.6/site-packages/klein/_plating.py", line 16, in <module>
from .app import _call
File "/home/a/.local/lib/python3.6/site-packages/klein/app.py", line 19, in <module>
from twisted.internet import endpoints, reactor
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 58, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory
File "/home/a/.local/lib/python3.6/site-packages/twisted/protocols/tls.py", line 63, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 158, in <module>
verifyHostname, VerificationError = _selectVerifyImplementation()
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 141, in _selectVerifyImplementation
from service_identity import VerificationError
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/__init__.py", line 7, in <module>
from . import cryptography, pyopenssl
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/cryptography.py", line 16, in <module>
from .exceptions import SubjectAltNameWarning
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/exceptions.py", line 21, in <module>
#attr.s
AttributeError: module 'attr' has no attribute 's'
Command "which python" gives same address at both location which is my virtualenv python address and that should be expected.
But what causes this weird python shell behavior.
Thank you

I solved it and a very shameful reason caused the error.
One of the modules Twisted uses is "attr" module. I had named one of my files attr.py and that is what was causing all the error.
I myself am not deleting this question if moderation has no problem, maybe somebody like me might be stuck at the same situation. It may help them.
Never name your python files same as that of any standard module unless overriding.
Also if your issue persists, then Jean's answer will definitely resolve it.

There can be multiple different Python packages that provide the same Python module. For example, there are at least two packages that provide the attr module:
https://pypi.org/project/attr/
https://pypi.org/project/attrs/
It's possible you've installed the wrong package based on the requirements. You can check what you have installed with pip freeze.

Error Import Impyla library on Windows

I'm having trouble with using impyla library on windows
I installed impyla library
pip install impyla
Error occured when I tried to import impyla libary in python code
from impala.dbapi import connect # error occured
from impala.util import as_pandas
conn = connect(host='10.xx.xx.xx', database='xx_xx', port=21050)`
Traceback (most recent call last): ...
File "D:/test/test.py", line 14, in
from impala.dbapi import connect
File "C:\Anaconda3\lib\site-packages\impala\dbapi.py", line 28, in
import impala.hiveserver2 as hs2
File "C:\Anaconda3\lib\site-packages\impala\hiveserver2.py", line 32, in
from impala._thrift_api import (
File "C:\Anaconda3\lib\site-packages\impala_thrift_api.py", line 73, in
include_dirs=[thrift_dir])
File "C:\Anaconda3\lib\site-packages\thriftpy\parser__init__.py", line 30, in load
include_dir=include_dir)
File "C:\Anaconda3\lib\site-packages\thriftpy\parser\parser.py", line 496, in parse
url_scheme))
thriftpy.parser.exc.ThriftParserError: ThriftPy does not support generating module with path in protocol 'c'
when I tried to print include_dir, which was
D:/test\thrift
I just cannot import libray at all
help me

I had the same problem with thriftpy, the problem on windows is an absolute path is like C:\foo\bar.thrift
But, the way the thrift library parses the file, it detects the C: as if it were a protocol like http: or https:
Its pretty easy to workaround you just have to strip the first two characters from the path with a slice like path[2:]
Just slice when you call thriftpy.load or in the library file
File "C:\Anaconda3\lib\site-packages\thriftpy\parser__init__.py", line 30
path = "C:\foo\bar.thrift"
thrift.load(path[2:], module_name, include_dirs=include_dirs,
include_dir=include_dir)
OR
You can go a bit deeper and make the same change that I already submitted as a patch on the github page... perhaps it will be incorporated in the next version of thrift.
File "C:\Anaconda3\lib\site-packages\thriftpy\parser\parser.py", line 488
- if url_scheme == '':
+ if len(url_scheme) <= 1:
My justification of why this change is valid is in the pull request. If its incorporated then you wont have to worry about making the same change again when you update the library. If not then just strip the two characters again.
Update:
Thriftpy Version 1: the parser fix is now on line 547: elif len(url_scheme) <= 1:
Thriftpy Version 2: the fix has already been merged.

I was encountering the same error with impyla on an Anaconda Python 3.6 distribution on Windows. Instead of installing using pip, I was able to get it working using:
conda install -c anaconda impyla
https://anaconda.org/anaconda/impyla

Python can't load libc when importing xfsm module

I'm trying to write Python code to manipulate Xerox FST files, and I've installed the Python bindings for libxfsm and the XFSM library available through http://fsmbook.com. I'm running 64-bit Ubuntu. The installation goes fine, but when I try to import the xfsm module, I get this error:
>>> import xfsm
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/xfsm/__init__.py", line
30, in <module>
import xfsm.errors # So error handler gets installed
File "/usr/local/lib/python2.7/dist-packages/xfsm/errors.py", line
8, in <module>
from xfsm.utils import *
File "/usr/local/lib/python2.7/dist-packages/xfsm/utils.py", line
39, in <module>
libc = cfsm.load_library("c")
File "/usr/local/lib/python2.7/dist-packages/xfsm/cfsm_api.py", line
366, in l oad_library
raise ImportError("%s not found." % libname)
ImportError: c not found.
Inspecting the source code for the Python interface, it looks like it is trying and failing to find libc, and failing, but I am at a loss on how to fix that.
Has anybody else gotten the Python-XFST interface working, or have any idea what's going wrong and how to fix it?

I had same problem on 64-bit Ubuntu 14.04. The problem is it can't find libc. In my case libc on my OS is name libc.so.6(I don't know why, I am new on Ubuntu). But the Python-XFST doesn't recognize it (Detail your can refer to source file)
My solution is simple, just modify xfsm/utils.py file in
**Line 39** : libc = cfsm.load_library("c")
to
**Line 39** : libc = cfsm.load_library("libc.so.6")
Remember to add the path to libc.so.6 in env.