We Keep Coding

Buildroot Python Dependencies (_sqlite3)

Full Disclaimer: I have been using buildroot for the last 6 weeks. This is my first introduction to embedded Linux, thus I am still very green but have been able to solve 99% of my problems myself. for the most part the process has been straightforward.
So to occupy myself while I am stuck and home and cant work I have been working on an embedded hardware project. I've selected my hardware, built a prototype, learned buildroot basics, brought up the basic system, optimized the kernel config, built a custom device tree for my hardware and I am happy up until this point.
In parallel I have been programming the target application in python on my desktop, its dependent on a few libraries (hardware abstraction, communication, display etc) but is relatively straight forward.
I've got to the point where I have a list of requisite packages to build into my buildroot system. The buildroot tools are great here using the scanpypi tool:
~/buildroot$ utils/scanpypi diskcache -o package
Simply adding all the dependencies into /package/config.in has allowed them to be selected in menuconfig and added to the recipe.
The problem comes at build time where the building of the python module fails for the module above python-diskcache.
It has dependencies on a few things but one is slqite3, this has been added as:
the core python module "sqlite module"
external package "python-pysqlite3"
libraries > database > sqlite
However, it fails at build:
>>> python-diskcache 4.1.0 Building
Traceback (most recent call last):
File "setup.py", line 5, in <module>
import diskcache
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/__init__.py", line 9, in <module>
from .core import Cache, Disk, EmptyDirWarning, JSONDisk, UnknownFileWarning, Timeout
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/core.py", line 14, in <module>
import sqlite3
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
make[1]: *** [package/pkg-generic.mk:269: /home/buildroot/output/build/python-diskcache-4.1.0/.stamp_built] Error 1
make: *** [Makefile:84: _all] Error 2
In looking for a solution it seems like _sqlite3 is the C module external to python for communicating with an sqlite database. It should be installed with python (using 3.8) and should be enabled in buildroot with the enabling of the core sqlite module.
discussion 1
discussion 2
There are several fixes for dealing with this issue on a host (e.g. apt get install libsqlite3-dev and reinstall/reconfigure python). Obviously this is not possible in the image, and both the sqlite and python3 installs are latest builds and installed to the image at build time.
I'm really struggling to understand the problem or how I might patch it. I have a few theories based on the discussion, but I am unsure.
1) python is getting installed to the image before sqlite, so the appropriate module is not getting cp or symlink'ed to the python install.
2) there is some other unknown dependency not being met at build time and its failing silently
any ideas or assistance would be hugely appreciated.
Thanks

The problem is that Python on your build machine finds the cross-compiled _sqlite3 module, which it can't load because it's for the wrong architecture.
This usually doesn't occur, because the setup script normally doesn't go and load the package it is trying to build/install.
One workaround could be to install the host version of all the dependencies of diskcache, and set PYTHONPATH=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/:$(PYTHON3_PATH) in DISKCACHE_ENV. However, this is liable to lead to all kinds of other breakage.
A better solution, therefore, is to patch the setup.py script of diskcache so it doesn't try to import diskcache itself. It probably only does that to get a version number or something similar; that can be solved by moving the version number into a separate file and load that one instead.

As suggested by Arnout, I got my diskcache building by creating a following patch
diff --git a/setup.py b/setup.py
index b49d9c8..22cd3c8 100644
--- a/setup.py
+++ b/setup.py
## -3,8 +3,6 ## from io import open
from setuptools import setup
from setuptools.command.test import test as TestCommand
-import diskcache
-
class Tox(TestCommand):
def finalize_options(self):
## -23,8 +21,8 ## with open('README.rst', encoding='utf-8') as reader:
readme = reader.read()
setup(
- name=diskcache.__title__,
- version=diskcache.__version__,
+ name='diskcache',
+ version='5.4.0',
description='Disk Cache -- Disk and file backed persistent cache.',
long_description=readme,
author='Grant Jenks',
--
2.25.1

ImportError: No module named _analog_swig

I am having issues getting python to import the _analog_swig gnuradio module in order to run gnuradio code on a Windows 8.1 64bit machine.
Some background: I am running Python 2.7.10 (installed in C:\Python27) and have installed the latest gnuradio binary (v3.7.11.1/v1.3 64-Bit Any CPU) from here: http://www.gcndevelopment.com/gnuradio/downloads.htm. I have installed gnuradio to C:\Program Files\GNURadio-3.7 .
I can run gnuradio companion and run flowgraphs from GRC successfully (which calls "C:\Program Files\GNURadio-3.7\bin\run_gr.bat" gnuradio-companion.py).
I have added & verified the following system variables are set:
Path: C:\Program Files\GNURadio-3.7\bin
PYTHONPATH: C:\Program Files\GNURadio-3.7\lib\site-packages
GRC_BLOCKS_PATH: C:\Program Files\GNURadio-3.7\share\gnuradio\grc\blocks
Now to the problem: If I run e.g. CMD and type:
python C:\test\top_block.py
I am returned the following ImportError:
File "C:\test\top_block.py", line 22, in <module>
from gnuradio import analog
File "C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog\__init__.py", line 33, in <module>
from analog_swig import *
File "C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog\analog_swig.py", line 17, in <module>
_analog_swig = swig_import_helper()
File "C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog\analog_swig.py", line 16, in swig_import_helper
return importlib.import_module('_analog_swig')
File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named _analog_swig
The folder content of C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog is as follows:
Comparing this to the folder content on a linux machine, which has a working install of gnuradio that works with python as I want it:
The difference seems to be that the folder in windows contains only a _analog_swig.pyc file, whereas the folder in linux contains a _analog_swig_.so file.
Any idea why the _analog_swig module can apparently not be imported in windows?
My plan is to be able to run gnuradio code directly from my python interpreter and being able to create compiled gnuradio executables so any help on how this could be fixed is much appreciated.

I've been struggling with this for the past few days, but I finally figured it out. I was trying to run GnuRadio Companion generated code in IDLE and also in PyCharm. I kept failing miserably with this same error. I finally figured it out:
-As Flexo says, the PYD file (_analog_swig.pyd) is actually a Windows DLL. The error makes it sound like Python is not finding that file, but that is not at all what was happening. The PYD file, being a DLL, has dependencies itself. Python is able to find _analog_swig.pyd just fine, but it could not find the DEPENDENCIES of that library.
-To verify if that's what wrong in your installation, download and use DependencyWalker (Google it) to check if your system can find the dependencies to _analog_swig.pyd.
-The fix for me was to add the GnuRadio-3.7/bin folder to my PATH environment variable. Inside that folder are a number of DLLs that the _analog_swig.pyd library needs to load. If you don't have the folder in your PATH, the module will fail to load in Python and throw the error you see above.
-I see that you verified that this folder is in your PATH, so this is apparently not the same problem, although your symptoms are exactly the same as mine. i.e. the GRC code would run just fine when you start with "run_gr.bat", but not when you run from a normal CMD window.
Hopefully that helps someone else that wants to use GNURadio Python code on Windows.

Friend,
As you mentioned, the GNU Companion calls \bin\run_gr.bat gnuradio-companion.py. That batch script does quite a bit of work on windows environment variables (try opening it in a text editor if you're curious).
In a sense, the run_gr.bat script puts together a temporary, custom python workspace for gnuradio so it can import anything it needs. It receives python scripts to run in this environment as command line arguments; hence, you can use it to run any GNU radio python code you want in your windows command prompt. Generally, you would call
<gnuradio_install_path>\bin\run_gr.bat <gnu_radio_code>.py
To test your import, you can try
# test.py
from gnuradio import analog
try calling the following from the command prompt, in the test.py directory:
<gnuradio_install_path>\bin\run_gr.bat test.py

Using third-party modules in pypy

The initial problem was solved, but there are more ...
__
I found this neat LTS tool online. Since it takes quite some time for training I'd like to speed things up. While searching for an approach I came across PyPy.
Pypy is now setup and seems to be working, but not with the Sequitur g2p code. When importing a module which comes from a .so file I get a No module named _sequitur_ .
I also tried installing/compiling the code with PyPy instead of Python which crashes without any useful(?) error.
I run PyPy v2.6.1 on a Ubuntu Linux 14.04 with Python 2.7.10.
Calling g2p with the default interpreter works fine.
Does the problematic .so file need to be compiled via a call from PyPy ?
Edit #1:
When trying to install g2p with PyPy instead of CPython it crashes without after this command:
error: Command "cc -O2 -fPIC -Wimplicit -DMULTIGRAM_SIZE=2 -I/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include -I/opt/pypy-2.6.1-linux64/include -c sequitur_wrap.cpp -o build/temp.linux-x86_64-2.7/sequitur_wrap.o -fpermissive" failed with exit status 1
When calling this particular command from the console, without being a part of the setup.py, there are two errors:
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:192:55: error: ‘PyArray_DescrFromType’ was not declared in this scope
PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \
&
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:194:69: error: ‘PyArray_FromAny’ was not declared in this scope
NPY_ARRAY_ENSUREARRAY, NULL)
This either didn't appear or caused an error when installing with CPhyton.
diff also shows that ndarrayobject.h under the mentioned path doesn't differ from the one in /usr/local/lib/python2.7/site-packages/numpy/andsoon .
Since I have no clue how c++ works, I'm lost there...
Edit #2:
Well, somehow it didn't work with the first include (pointing to pypy's numpy). Probably not the clean way to do it, but it helped to get rid of that problem. So I substituted the include_dirs entry in the setup.py and pointed that one to cPython's numpy include dir.
That done, the setup.py runs without any error. When I now try to import sequitur (one of the necessary files) I noticed that one .so was missing, copied it from cPython's numpy to Pypy's numpy. So far so good but now I still get this error:
$ pypy
Python 2.7.10 (f3ad1e1e1d62, Aug 28 2015, 10:45:29)
[PyPy 2.6.1 with GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import sequitur
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sequitur.py", line 32, in <module>
import sequitur_, SequenceModel, Minimization, misc
File "sequitur_.py", line 28, in <module>
_sequitur_ = swig_import_helper()
File "sequitur_.py", line 20, in swig_import_helper
import _sequitur_
ImportError: numpy.core.multiarray failed to import
Any suggestions?
Thanks

Matlab Engine Python - OSx Anaconda Segfault or DYLD_LIBRARY_PATH error with iPython

After installing the matlab python package in terminal using:
cd "matlabroot\extern\engines\python"
python setup.py install
And trying to run it, I get a segfault:
:~$ python
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import matlab.engine
Segmentation fault: 11
However, I can get around this by setting DYLD_LIBRARY_PATH after which matlab.engine works:
:~$ export DYLD_LIBRARY_PATH=/System/Library/Frameworks/Python.framework/Versions/Current/lib:$DYLD_LIBRARY_PATH
:~$ python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import matlab.engine
>>> eng = matlab.engine.start_matlab()
>>> exit()
However, when I try to launch iPython afterwards I get this error:
Traceback (most recent call last):
File "//anaconda/bin/ipython", line 4, in <module>
from IPython import start_ipython
File "//anaconda/lib/python2.7/site-packages/IPython/__init__.py", line 45, in <module>
from .config.loader import Config
File "//anaconda/lib/python2.7/site-packages/IPython/config/__init__.py", line 6, in <module>
from .application import *
File "//anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 19, in <module>
from IPython.config.configurable import SingletonConfigurable
File "//anaconda/lib/python2.7/site-packages/IPython/config/configurable.py", line 12, in <module>
from .loader import Config, LazyConfigValue
File "//anaconda/lib/python2.7/site-packages/IPython/config/loader.py", line 16, in <module>
from IPython.utils.path import filefind, get_ipython_dir
File "//anaconda/lib/python2.7/site-packages/IPython/utils/path.py", line 14, in <module>
import tempfile
File "//anaconda/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "//anaconda/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(//anaconda/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
Referenced from: //anaconda/lib/python2.7/lib-dynload/_io.so
Expected in: dynamic lookup
As you can see the python versions are different. I think this is a conflict between my system Python and Anaconda but I'm not sure how to fix it, any help much appreciated.
Thanks.

You need to install libraries using the conda command line install utility, not python. conda help search and conda help install should get you going.

This is not a direct solution, but I was stuck with the same problem and used this workaround. And I should say that I did finally make matlab.engine run in my Python environment, but that it just plain sucked. It couldn't run scripts with embedded scripts in other folders, and I also experienced that it couldn't find some build-in functions. The following is implemented for a Unix machine, but it would not take a lot of modification to make it work in Windows, I believe.
Here's what I did:
Wrote a main Matlab script that could do everything I needed from Matlab.
Ran that script through a subprocess in Python.
In my case I needed do a series of matrix operations for matrix X and return matrix S. The matrix operations required the use of a particular Matlab function. My first idea was to open a Matlab-session with matlab.engine and then manage the matrix operations in Python only calling the Matlab function when needed. Instead (as the bullets state) I wrote a Matlab function ComputeSimilarityMat that takes X, does all necessary operations, and returns S. Then I basically just ran that Matlab function from Python using a subprocess.
This is what the Python script, that manages the Matlab script through a subprocess, looks like:
import subprocess
import numpy as np
def run_matlab(X):
"""Produce trait-similarity matrix S from a trait-space matrix X
Parameters
----------
X : numpy.ndarray
Returns
-------
S : numpy.ndarray
"""
# Dump input in .csv, so Matlab can load it
np.savetxt('X.csv', X, delimiter=",")
# Code executed in Matlab. Important to end with quit;
matlab_code = "X=csvread('X.csv');" \
"S=ComputeSimilarityMat(X);" \
"csvwrite('S.csv',S);" \
"quit;"
# -nosplash suppresses the splash window on startup
matlab_call = ["matlab", "-nodesktop", "-nosplash", "-r", matlab_code]
subprocess.call(matlab_call)
# Load the .csv file that Matlab dumps after finishing
S = np.genfromtxt('X.csv', delimiter=",")
return S
I have to say, I'm sure there's a nicer way to pass an object to Matlab then to have to save and load like this, but it worked for me. Still hoping for an improvement of matlab.engine though, but until then I'm going with this approach.
Note: To run Matlab from command-line/subprocess you need to add the Matlab bin folder to your PATH variable like this:
export PATH="/Applications/MATLAB_R2015a.app/bin:$PATH"
I just put this in my .profile file.

In my case adding things to LD_LIBRARY_PATH (Ubuntu's version of DYLD_LIBRARY_PATH) only made things worse. Instead I had to make sure it did not refer to the Python installation, and had to add a symbolic link from /usr/lib instead. See https://stackoverflow.com/a/45161288/2524427

Undefined Symbol in C++ When Loading a Python Shared Library

I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.
Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.
However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.
Here is the full python error:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in
import add_newdocs
File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in
from lib import add_newdoc
File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in
from type_check import *
File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in
import numpy.core.numeric as _nx
File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in
import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in
from openravepy import *
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in
openravepy_currentversion = loadlatest()
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
return _loadversion('_openravepy_')
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
mainpackage = __import__("openravepy", globals(), locals(), [targetname])
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in
from openravepy_int import *
ImportError: numpy.core.multiarray failed to import

I experienced the same problem with my application and solved it without linking python to the executable.
The setup is as follows:
Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter
The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.
dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)
This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.
It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.

The solution was linking the python2.6 library with my executable as well.
Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.
For clarification, my program model is something like:
[My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]

Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.

there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:
http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html
search for "python" in the cmake file here:
https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt
the relevant info is:
if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
find_package(PythonLibs)
if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
if( PYTHON_EXECUTABLE )
# get the site-packages directory
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
OUTPUT_VARIABLE _python_sitepackage
RESULT_VARIABLE _python_failed)
if( ${_python_failed} EQUAL 0 )
string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
else()
message(STATUS "failed to get python site-package directory")
endif()
endif()
include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
add_library(orpythonbinding SHARED orpythonbinding.cpp)
target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
if( WIN32 )
set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
endif()
endif()
endif()