Full Disclaimer: I have been using buildroot for the last 6 weeks. This is my first introduction to embedded Linux, thus I am still very green but have been able to solve 99% of my problems myself. for the most part the process has been straightforward.
So to occupy myself while I am stuck and home and cant work I have been working on an embedded hardware project. I've selected my hardware, built a prototype, learned buildroot basics, brought up the basic system, optimized the kernel config, built a custom device tree for my hardware and I am happy up until this point.
In parallel I have been programming the target application in python on my desktop, its dependent on a few libraries (hardware abstraction, communication, display etc) but is relatively straight forward.
I've got to the point where I have a list of requisite packages to build into my buildroot system. The buildroot tools are great here using the scanpypi tool:
~/buildroot$ utils/scanpypi diskcache -o package
Simply adding all the dependencies into /package/config.in has allowed them to be selected in menuconfig and added to the recipe.
The problem comes at build time where the building of the python module fails for the module above python-diskcache.
It has dependencies on a few things but one is slqite3, this has been added as:
the core python module "sqlite module"
external package "python-pysqlite3"
libraries > database > sqlite
However, it fails at build:
>>> python-diskcache 4.1.0 Building
Traceback (most recent call last):
File "setup.py", line 5, in <module>
import diskcache
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/__init__.py", line 9, in <module>
from .core import Cache, Disk, EmptyDirWarning, JSONDisk, UnknownFileWarning, Timeout
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/core.py", line 14, in <module>
import sqlite3
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
make[1]: *** [package/pkg-generic.mk:269: /home/buildroot/output/build/python-diskcache-4.1.0/.stamp_built] Error 1
make: *** [Makefile:84: _all] Error 2
In looking for a solution it seems like _sqlite3 is the C module external to python for communicating with an sqlite database. It should be installed with python (using 3.8) and should be enabled in buildroot with the enabling of the core sqlite module.
discussion 1
discussion 2
There are several fixes for dealing with this issue on a host (e.g. apt get install libsqlite3-dev and reinstall/reconfigure python). Obviously this is not possible in the image, and both the sqlite and python3 installs are latest builds and installed to the image at build time.
I'm really struggling to understand the problem or how I might patch it. I have a few theories based on the discussion, but I am unsure.
1) python is getting installed to the image before sqlite, so the appropriate module is not getting cp or symlink'ed to the python install.
2) there is some other unknown dependency not being met at build time and its failing silently
any ideas or assistance would be hugely appreciated.
Thanks
The problem is that Python on your build machine finds the cross-compiled _sqlite3 module, which it can't load because it's for the wrong architecture.
This usually doesn't occur, because the setup script normally doesn't go and load the package it is trying to build/install.
One workaround could be to install the host version of all the dependencies of diskcache, and set PYTHONPATH=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/:$(PYTHON3_PATH) in DISKCACHE_ENV. However, this is liable to lead to all kinds of other breakage.
A better solution, therefore, is to patch the setup.py script of diskcache so it doesn't try to import diskcache itself. It probably only does that to get a version number or something similar; that can be solved by moving the version number into a separate file and load that one instead.
As suggested by Arnout, I got my diskcache building by creating a following patch
diff --git a/setup.py b/setup.py
index b49d9c8..22cd3c8 100644
--- a/setup.py
+++ b/setup.py
## -3,8 +3,6 ## from io import open
from setuptools import setup
from setuptools.command.test import test as TestCommand
-import diskcache
-
class Tox(TestCommand):
def finalize_options(self):
## -23,8 +21,8 ## with open('README.rst', encoding='utf-8') as reader:
readme = reader.read()
setup(
- name=diskcache.__title__,
- version=diskcache.__version__,
+ name='diskcache',
+ version='5.4.0',
description='Disk Cache -- Disk and file backed persistent cache.',
long_description=readme,
author='Grant Jenks',
--
2.25.1
Related
I manually compiled python-openzwave to work with C++ library.
I would like to use it as Kodi addon (OpenELEC running on Pi 3), so can not use standard installation.
I've compiled everything, downloaded missing six and louie libs, and now try to run hello_world.py.
My current dirs structure is the following:
- root
- bin
- .lib
- config
Alarm.o
...
libopenzwave.a
libopenzwave.so
libopenzwave.so.1.4
...
- libopenzwave
driver.pxd
group.pxd
...
- louie
__init__.py
dispatcher.py
...
- openzwave
__init__.py
command.py
...
six.py
hello_world.py
But when I run hello_world.py, I get the following error -
Traceback (most recent call last):
File "hello_world.py", line 40, in <module>
from openzwave.controller import ZWaveController
File "/storage/.kodi/addons/service.multimedia.open-zwave/openzwave/controller.py", line 34, in <module>
from libopenzwave import PyStatDriver, PyControllerState
ImportError: No module named libopenzwave
If I move libopenzwave.a and libopenzwave.so to root folder, then I get the following error:
Traceback (most recent call last):
File "hello_world.py", line 40, in <module>
from openzwave.controller import ZWaveController
File "/storage/.kodi/addons/service.multimedia.open-zwave/openzwave/controller.py", line 34, in <module>
from libopenzwave import PyStatDriver, PyControllerState
ImportError: dynamic module does not define init function (initlibopenzwave)
What is wrong with my setup?
In general the steps required consist of calls to make build which handles building the .cpp files for openzwave and downloading all dependencies (including Cython); and make install which runs the setup-api, setup-lib.py (this setup script also creates the C++ Python extention for openzwave), setup-web.py and setup-manager.py.
Since you cannot run make install as you specified and are instead using the archive they provide, the only other options for creating the python extention, after building the openzwave library with make build, is generating the .so files for it without installing to standard locations.
Building the .so for the cython extention in the same folder as the Cython scripts is done by running:
python setup.py build_ext --inplace
This should create a shared library in src-lib named libopenzwave.so (it is different from the libopenzwave.so contained in the bin/ directory) which contains all the functionality specified in the extention module. You could try adding that to the libopenzwave folder.
If you pass special compiler flags during make build for building the openzwave library you should specify the same ones when executing the setup-lib.py script. This can be done by specifying the CFLAGS before executing it (as specified here) or else you might have issues like error adding symbols: File in wrong format.
Here's the description of the python-openzwave's build from the question's perspective. Almost all the steps correspond to the root Makefile's targets.
Prerequisites. There are several independent targets with little to no organization. Most use Debian-specific commands.
Cython is not needed if building from an archive (details below)
openzwave C++ library (openzwave openzwave/.lib/ target).
Build logic: openzwave/Makefile, invoked without parameters (but with inherited environment).
Inputs: openzwave/ subtree (includes libhidapi and libtinyxml, statically linked).
Outputs: openzwave/.lib/libopenzwave.{a,so}
Accepts PREFIX as envvar (/usr/local by default)
The only effect that affects us is: $(PREFIX)/etc/openzwave/ is assigned to a macro which adds a search location for config files (Options.cpp): config/ -> /etc/openzwave/ -> <custom location>.
libopenzwave Python C extension module (install-lib target - yes, the stock Makefile cannot just build it; the target doesn't even have the dependency on the library).
Build logic: setup-lib.py
Inputs: src-lib/, openzwave/.lib/libopenzwave.a
Outputs: build/<...>/libopenzwave.so - yes, the same name as openzwave's output, so avoid confusing them
By default, openzwave is linked statically with the module so you don't need to include the former into a deployment
The module does, however, need the config folder from the library. It is included by the build script when making a package.
Contrary to what Jim says, Cython is not needed to build from an archive, the archive already includes the generated .cpp.
Now, the catch is: the module itself uses pkg_resources to locate its data. So you cannot just drop the .so and config into the currect directory and call it a day. You need to make pkg_resources.get_distribution('libopenzwave') succeed.
pkg_resources claims to support "normal filesystem packages, .egg files, and unpacked .egg files."
In particular, I was able to pull this off: make an .egg (setup-lib.py bdist_egg), unpack it into the current directory and rename EGG-INFO into libopenzwave.egg-info (like it is in site-packages). A UserWarning is issued if I don't specifically add the .so's location into PYTHON_PATH/sys.path before importing the module.
openzwave,pyozwman and pyozwweb Python packages (install)
these are pure Python packages. The first one uses the C extension module, others use the first one.
Build logic: setup-api.py,setup-manager.py,setup-web.py
Input: src-*/
Output: (pure Python)
They only use pkg_resources.declare_namespace() so you're gonna be fine with just the proper files/dirs on sys.path without any .egg-info's
I'm trying to use py2exe to build an executable on 64-bit Windows 7 with Anaconda (Python 3.4) for a project of mine that depends on a lot of libraries. Some of the more complex ones include vispy (pyopengl), PyQt4, numba, and scipy. I've been stepping through various errors to try to get a working executable, but have hit a road block with no clear way forward. Currently, the py2exe command completes, but I get the following error when running the exe:
...
from numba import jit
File "C:\Anaconda3\envs\sift_py2exe\lib\site-packages\numba\__init__.py", line
13, in <module>
from .pycc.decorators import export, exportmany
File "C:\Anaconda3\envs\sift_py2exe\lib\site-packages\numba\pycc\__init__.py",
line 12, in <module>
from .cc import CC
File "C:\Anaconda3\envs\sift_py2exe\lib\site-packages\numba\pycc\cc.py", line
4, in <module>
from distutils.command import build_ext
File "C:\Anaconda3\envs\sift_py2exe\lib\distutils\command\build_ext.py", line
17, in <module>
from site import USER_BASE
ImportError: No module named 'site'
I was able to do a small workaround by adding the C:\Anaconda3\envs\sift_py2exe\Lib directory to sys.path in my main script, but I doubt that's going to help me much later. Not to mention I had more scipy DLL issues after that.
Here are the relevant parts of my setup.py:
try:
import py2exe
from llvmlite.binding.ffi import _lib_dir, _lib_name
kwargs["data_files"] = [('.', [os.path.join(_lib_dir, _lib_name), os.path.join(_lib_dir, "MSVCP120.dll"), os.path.join(_lib_dir, "MSVCR120.dll")])]
kwargs["console"] = [{
'script': 'cspov/__main__.py',
'dest_base': "SIFT",
}]
kwargs["options"] = {'py2exe': {"includes": ["vispy.app.backends._pyqt4", "PyQt4.QtNetwork"]}}
except ImportError:
print("'py2exe' and/or 'llvmlite' not available")
I've tried adding the "Lib" directory in setup.py and then including "site", but it doesn't find the module. Any ideas? Thanks.
Side note: I'm using the Microsoft DLLs from llvmlite as a quick workaround because I couldn't get it to work in any of the normal ways.
This isn't the answer I was hoping for, but I was able to get a working executable when I switched to pyinstaller. All of the other SO questions I saw relevant to my problem had similar "solutions".
Good day,
I'm a student and I would just like to ask for a minute of your time.
I'm working on a barcode reader connected via USB port to a board name Arduino Yun. This board runs a version of embedded linux derived from OpenWrt using a microprocessor named Atheros AR9331
I would like to ask you, what's necessary to make the Python Evdev binding (python-evdev.readthedocs.org/en/latest/), to be able to run in this type of MIPS microarchitecture? At the momento, it's only for Ubuntu and ArchLinux.
I'm kind of guessing that cross compilation would be needed, or the indication of the usage of a specific C compiler inside this linux.
The current python version supported for OpenWrt is 2.7.3
I already know , if you compile C code in your PC, the resulting executable will only run in this type of architecture. If you use that compiled program inside the microprocessor, it wont work.
I've used this binding without trouble within ubuntu in my PC. I followed the instructions, python setup.py install, with a previous installation of setuptools, and it worked just fine.
But regarding OpenWrt, this was not the case.
The python script I'm using requires this library within the first line of code in order to reach the data from the device (it works like a keyboard /dev/input/event0):
#!/usr/bin/env python
from evdev import InputDevice, ecodes, list_devices
from select import select
I've seen suggestions of copying the entire library inside the arduino, and run the script inside the same folder. But it doesn't work, since the evdev module has files created with the architecture of the PC and not the MIPS.
So, what are the messages displayed for the error?
If you run python setup.py install in Openwrt to try to install the evdev binding, this appears on screen:
File "setup.py", line 10, in <module>
from setuptools.command.develop import develop
ImportError: No module named setuptools.command.develop
It's obvious from here that you need the module aforementioned. So, I tried to install it with this script (pypi.python.org/pypi/setuptools):
python ez_setup.py
And the output shows this:
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-11.3.1.zip
Traceback (most recent call last):
File "ez_setup.py", line 332, in <module>
sys.exit(main())
File "ez_setup.py", line 327, in main
downloader_factory=options.downloader_factory,
File "ez_setup.py", line 287, in download_setuptools
downloader(url, saveto)
File "ez_setup.py", line 209, in download_file_curl
_clean_check(cmd, target)
File "ez_setup.py", line 169, in _clean_check
subprocess.check_call(cmd)
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['curl','https://pypi.python.org/packages/source/s/setuptools/setuptools-11.3.1.zip', '--silent', '--output', '/mnt/sda1/evdev-0.4.6/setuptools-11.3.1.zip']' returned non-zero exit status 60
I pressume this output is due to the fact that pypi doesn't exist for the python 2.7.3 in OpenWrt , only for newer versions and other architectures. Evedv binding is requiring the setuptools module in order to make things easier and standard, but if the binding is not supported for the target architecture, what's needed to be able to use it anyways?
Thanks for your time,
Good day everyone,
The solution was provided by Georgi Valkov. He is the creator of the python-evdev binding. I contacted him directly, and he was so kind that he cross compiled a version for the OpenWrt / Yun .
You can install the package using the openwrt package manager - opkg. The installation process is along the lines of:
$ opkg update
$ opkg install /path/to/python-evdev_0.4.7-1_ar71xx.ipk
To verify that the install was successful:
$ opkg files python-evdev
/usr/lib/python2.7/site-packages/evdev-0.4.7-py2.7.egg-info
/usr/lib/python2.7/site-packages/evdev/genecodes.py
/usr/lib/python2.7/site-packages/evdev/ff.py
/usr/lib/python2.7/site-packages/evdev/_input.so
/usr/lib/python2.7/site-packages/evdev/device.py
/usr/lib/python2.7/site-packages/evdev/events.py
/usr/lib/python2.7/site-packages/evdev/__init__.py
/usr/lib/python2.7/site-packages/evdev/ecodes.py
/usr/lib/python2.7/site-packages/evdev/_ecodes.so
/usr/lib/python2.7/site-packages/evdev/util.py
/usr/lib/python2.7/site-packages/evdev/uinput.py
/usr/lib/python2.7/site-packages/evdev/_uinput.so
This works just fine. Thanks.
PS. If someone needs the file, please contact me. Georgi sent me this address, but I didn't download the file from there because he sent it to me over email.
https://github.com/gvalkov/openwrt-packages-yun/blob/master/lang/python-evdev/Makefile
In the output, you can see that curl returned the status code 60. According to man curl
60 Peer certificate cannot be authenticated with known CA certifi‐
cates.
According to the setuptools page, you can instead use python ez_setup.py --insecure but obviously do that at your own risk. Alternatively you could do what the advanced instructions say and manually download the setuptools tarball, verify its md5 hash yourself, and install it using its setup.py .
I installed the p4 library on my local archlinux machine and everything works smoothly.
I was now trying to do the same on a couple of virtual machines (with same OS also 64 bits) but can't get it running...
The installation goes well and doesn't complain, but when I try to import it I get:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "P4.py", line 312, in <module>
import P4API
ImportError: /usr/lib/python2.7/site-packages/P4API.so: undefined symbol: TLSv1_method
If I look in the symbols with
nm -A /usr/lib/python2.7/site-packages/P4API.so
I can see that in fact all the SSL-related symbols are undefined (while being defined on the working one), but why on earth?
I have openssl installed and Python is installed in exactly the same way, I can't find any other library that should be useful.
Any idea?
EDIT:
apparently the only difference is that in the wrong installation of P4 P4API.so is missing librt from the "ldd" output (and librt is actually there).
Not sure how it would relate to the TLS problem, and trying to reinstall glibc and then
p4python but no luck still..
Rebuilding the P4Python against the "rt" and "ssl" libraries fixes this.
Take the P4 API off the perforce site (e.g. for Linux take this: http://filehost.perforce.com/perforce/r13.2/bin.linux26x86_64/p4api.tgz)
Take the source code of P4Python off http://public.perforce.com/guest/robert_cowham/perforce/API/python/main/p4python.zip
tar xf the P4 API archive
Unzip the p4pthon archive
Edit setup.cfg to point to the P4API directory
Edit setup.py and go to line 120 and add "ssl" and "rt" so it looks like this
else: # Assume Linux
libraries = ["client", "rpc", "supp", "ssl", "rt"] # P4API libs
extra_compile_args = ["-DOS_LINUX", "-D%s" % p4_api_ver]
Do sudo python setup.py install
I'm trying to run a python script using python 2.6.4. The hosting company has 2.4 installed so I compiled my own 2.6.4 on a similar server and then moved the files over into ~/opt/python. that part seems to be working fine.
anyhow, when I run the script below, I am getting ImportError: No module named _sqlite3 and I'm not sure what to do to fix this.
Most online threads mention that sqlite / sqlite3 is included in python 2.6 - so I'm not sure why this isn't working.
-jailshell-3.2$ ./pyDropboxValues.py
Traceback (most recent call last):
File "./pyDropboxValues.py", line 21, in
import sqlite3
File "/home/myAccount/opt/lib/python2.6/sqlite3/__init__.py", line 24, in
from dbapi2 import *
File "/home/myAccount/opt/lib/python2.6/sqlite3/dbapi2.py", line 27, in
from _sqlite3 import *
ImportError: No module named _sqlite3
I think I have everything set up right as far as the directory structure.
-jailshell-3.2$ find `pwd` -type d
/home/myAccount/opt
/home/myAccount/opt/bin
/home/myAccount/opt/include
/home/myAccount/opt/include/python2.6
/home/myAccount/opt/lib
/home/myAccount/opt/lib/python2.6
/home/myAccount/opt/lib/python2.6/distutils
/home/myAccount/opt/lib/python2.6/distutils/command
/home/myAccount/opt/lib/python2.6/distutils/tests
/home/myAccount/opt/lib/python2.6/compiler
/home/myAccount/opt/lib/python2.6/test
/home/myAccount/opt/lib/python2.6/test/decimaltestdata
/home/myAccount/opt/lib/python2.6/config
/home/myAccount/opt/lib/python2.6/json
/home/myAccount/opt/lib/python2.6/json/tests
/home/myAccount/opt/lib/python2.6/email
/home/myAccount/opt/lib/python2.6/email/test
/home/myAccount/opt/lib/python2.6/email/test/data
/home/myAccount/opt/lib/python2.6/email/mime
/home/myAccount/opt/lib/python2.6/lib2to3
/home/myAccount/opt/lib/python2.6/lib2to3/pgen2
/home/myAccount/opt/lib/python2.6/lib2to3/fixes
/home/myAccount/opt/lib/python2.6/lib2to3/tests
/home/myAccount/opt/lib/python2.6/xml
/home/myAccount/opt/lib/python2.6/xml/parsers
/home/myAccount/opt/lib/python2.6/xml/sax
/home/myAccount/opt/lib/python2.6/xml/etree
/home/myAccount/opt/lib/python2.6/xml/dom
/home/myAccount/opt/lib/python2.6/site-packages
/home/myAccount/opt/lib/python2.6/logging
/home/myAccount/opt/lib/python2.6/lib-dynload
/home/myAccount/opt/lib/python2.6/sqlite3
/home/myAccount/opt/lib/python2.6/sqlite3/test
/home/myAccount/opt/lib/python2.6/encodings
/home/myAccount/opt/lib/python2.6/wsgiref
/home/myAccount/opt/lib/python2.6/multiprocessing
/home/myAccount/opt/lib/python2.6/multiprocessing/dummy
/home/myAccount/opt/lib/python2.6/curses
/home/myAccount/opt/lib/python2.6/bsddb
/home/myAccount/opt/lib/python2.6/bsddb/test
/home/myAccount/opt/lib/python2.6/idlelib
/home/myAccount/opt/lib/python2.6/idlelib/Icons
/home/myAccount/opt/lib/python2.6/tmp
/home/myAccount/opt/lib/python2.6/lib-old
/home/myAccount/opt/lib/python2.6/lib-tk
/home/myAccount/opt/lib/python2.6/hotshot
/home/myAccount/opt/lib/python2.6/plat-linux2
/home/myAccount/opt/lib/python2.6/ctypes
/home/myAccount/opt/lib/python2.6/ctypes/test
/home/myAccount/opt/lib/python2.6/ctypes/macholib
/home/myAccount/opt/share
/home/myAccount/opt/share/man
/home/myAccount/opt/share/man/man1
And finally the contents of the sqlite3 directory:
-jailshell-3.2$ find `pwd`
/home/myAccount/opt/lib/python2.6/sqlite3
/home/myAccount/opt/lib/python2.6/sqlite3/__init__.pyo
/home/myAccount/opt/lib/python2.6/sqlite3/dump.pyc
/home/myAccount/opt/lib/python2.6/sqlite3/__init__.pyc
/home/myAccount/opt/lib/python2.6/sqlite3/dbapi2.pyo
/home/myAccount/opt/lib/python2.6/sqlite3/dbapi2.pyc
/home/myAccount/opt/lib/python2.6/sqlite3/dbapi2.py
/home/myAccount/opt/lib/python2.6/sqlite3/dump.pyo
/home/myAccount/opt/lib/python2.6/sqlite3/__init__.py
/home/myAccount/opt/lib/python2.6/sqlite3/dump.py
I feel like I need to add something into the sqlite3 directory - maybe sqlite3.so? But I don't know where to get that.
What am I doing wrong here? Please remember that I'm using a shared host so that means installing / compiling on another server and then copying the files over. Thanks! :)
Update
Just wanted to confirm that the answer from #samplebias did work out very well. I needed to have the dev package installed on the machine I was compiling from to get it to add in sqlite3.so and related files. Also found the link in the answer very helpful. Thanks #samplebias !
Python's build system uses a setup.py file to compile all of the native extensions, including sqlite3. It searches common operating system paths for the sqlite3 include and library dirs. If the sqlite3 development package is not installed Python will skip compiling the _sqlite3.so extension, but the pure Python portion of the sqlite3 package will still be installed.
You would need to have the operating system's sqlite3 development package installed when you compile Python and at runtime: sqlite3-devel on Centos, both libsqlite3-0 and libsqlite3-dev on Ubuntu.
Here's an example of the _sqlite3.so extension linkage on my Ubuntu system:
% ldd /usr/lib/python2.6/lib-dynload/_sqlite3.so | grep sqlite3
libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x00007f29ef3be000)
% dpkg -S /usr/lib/libsqlite3.so.0
libsqlite3-0: /usr/lib/libsqlite3.so.0
In general, the first thing to do is to ask your host. I seems a bit odd that SQLite is not installed (or installed properly). So they'll likely fix it quite fast if you ask them.
For python 2.4, you need sqlite and bindings, pysqlite 2 or aspw
None of the files listed in the sqlite folder is the _sqlite3.pyd Python shared library. Are you sure you compiled it when compiling Python? What does the build log say? I think there's a configure flag that needs to be passed.
Alternatively, just install pysqlite