We Keep Coding

Buildroot Python Dependencies (_sqlite3)

Full Disclaimer: I have been using buildroot for the last 6 weeks. This is my first introduction to embedded Linux, thus I am still very green but have been able to solve 99% of my problems myself. for the most part the process has been straightforward.
So to occupy myself while I am stuck and home and cant work I have been working on an embedded hardware project. I've selected my hardware, built a prototype, learned buildroot basics, brought up the basic system, optimized the kernel config, built a custom device tree for my hardware and I am happy up until this point.
In parallel I have been programming the target application in python on my desktop, its dependent on a few libraries (hardware abstraction, communication, display etc) but is relatively straight forward.
I've got to the point where I have a list of requisite packages to build into my buildroot system. The buildroot tools are great here using the scanpypi tool:
~/buildroot$ utils/scanpypi diskcache -o package
Simply adding all the dependencies into /package/config.in has allowed them to be selected in menuconfig and added to the recipe.
The problem comes at build time where the building of the python module fails for the module above python-diskcache.
It has dependencies on a few things but one is slqite3, this has been added as:
the core python module "sqlite module"
external package "python-pysqlite3"
libraries > database > sqlite
However, it fails at build:
>>> python-diskcache 4.1.0 Building
Traceback (most recent call last):
File "setup.py", line 5, in <module>
import diskcache
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/__init__.py", line 9, in <module>
from .core import Cache, Disk, EmptyDirWarning, JSONDisk, UnknownFileWarning, Timeout
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/core.py", line 14, in <module>
import sqlite3
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
make[1]: *** [package/pkg-generic.mk:269: /home/buildroot/output/build/python-diskcache-4.1.0/.stamp_built] Error 1
make: *** [Makefile:84: _all] Error 2
In looking for a solution it seems like _sqlite3 is the C module external to python for communicating with an sqlite database. It should be installed with python (using 3.8) and should be enabled in buildroot with the enabling of the core sqlite module.
discussion 1
discussion 2
There are several fixes for dealing with this issue on a host (e.g. apt get install libsqlite3-dev and reinstall/reconfigure python). Obviously this is not possible in the image, and both the sqlite and python3 installs are latest builds and installed to the image at build time.
I'm really struggling to understand the problem or how I might patch it. I have a few theories based on the discussion, but I am unsure.
1) python is getting installed to the image before sqlite, so the appropriate module is not getting cp or symlink'ed to the python install.
2) there is some other unknown dependency not being met at build time and its failing silently
any ideas or assistance would be hugely appreciated.
Thanks

The problem is that Python on your build machine finds the cross-compiled _sqlite3 module, which it can't load because it's for the wrong architecture.
This usually doesn't occur, because the setup script normally doesn't go and load the package it is trying to build/install.
One workaround could be to install the host version of all the dependencies of diskcache, and set PYTHONPATH=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/:$(PYTHON3_PATH) in DISKCACHE_ENV. However, this is liable to lead to all kinds of other breakage.
A better solution, therefore, is to patch the setup.py script of diskcache so it doesn't try to import diskcache itself. It probably only does that to get a version number or something similar; that can be solved by moving the version number into a separate file and load that one instead.

As suggested by Arnout, I got my diskcache building by creating a following patch
diff --git a/setup.py b/setup.py
index b49d9c8..22cd3c8 100644
--- a/setup.py
+++ b/setup.py
## -3,8 +3,6 ## from io import open
from setuptools import setup
from setuptools.command.test import test as TestCommand
-import diskcache
-
class Tox(TestCommand):
def finalize_options(self):
## -23,8 +21,8 ## with open('README.rst', encoding='utf-8') as reader:
readme = reader.read()
setup(
- name=diskcache.__title__,
- version=diskcache.__version__,
+ name='diskcache',
+ version='5.4.0',
description='Disk Cache -- Disk and file backed persistent cache.',
long_description=readme,
author='Grant Jenks',
--
2.25.1

Error: No module named 'fcntl'

I get the following error:
Traceback (most recent call last):
File "C:/Users/aaaa/Desktop/ttttttt.py", line 5, in <module>
import reload
File "C:\Users\aaa\AppData\Local\Programs\Python\Python36\lib\site-
packages\reload.py", line 3, in <module>
import sys, time, re, os, signal, fcntl
ModuleNotFoundError: No module named 'fcntl'
So I did a pip install, which also gets an error.
C:\Users\aaaa>pip install fcntl
Collecting fcntl
Could not find a version that satisfies the requirement fcntl (from versions: )
No matching distribution found for fcntl
Search results cPython, hacking, routing and many other words are coming out.
It's a tough answer for beginners, so I want to get a more detailed solution.
How should I solve it?
#py3
import time
from selenium import webdriver
import codecs
import sys
import reload
import re
import fcntl
import os
import signal

The fcntl module is not available on Windows. The functionality it exposes does not exist on that platform.
If you're trying to lock a file, there are some other Python modules available which provide that functionality. One I've seen referenced in other answers is portalocker.

I got the same error when trying to run my flask app using gunicorn.
gunicorn --bind 127.0.0.1:5000 predict:app
The issue is that 'fcntl' is not available on windows. The alternative that can be used, as suggested by Alexey Grigorov in Ml bookcamp, is the 'waitress' package.
pip install waitress
Then write in the command prompt the following command.
waitress-serve --listen=127.0.0.1:5000 predict:app
For those still looking for the answer.

I got some info from this website https://pypi.org/project/micropython-fcntl/#files and installed as follows which solved the problem:
pip install micropython-fcntl

What you can do is install importlib with the usual:
pip install importlib
From there use the following:
from importlib import reload
Note that you will need to load your imports as 'modules':
from petshop import parrot as parrot

Scons fails to import _args_from_interpreter_flags

Im trying to compile Godot engine following the instructions here
When I run scons bin/godot as the tutorial says, I get the following error:
scons: Reading SConscript files ...
ImportError: cannot import name _args_from_interpreter_flags:
File "/home/grayfox/github/godot2/godot/SConstruct", line 9:
import multiprocessing
File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 65:
from multiprocessing.util import SUBDEBUG, SUBWARNING
File "/usr/lib64/python2.7/multiprocessing/util.py", line 40:
from subprocess import _args_from_interpreter_flags
The SConstruct file starts this way:
EnsureSConsVersion(0,14);
import string
import os
import os.path
import glob
import sys
import methods
import multiprocessing
...
If I try to run python SConstruct I get an error complaining about missing functions defined by scons (i.e. the script fails after doing all the imports).
Commenting import multiprocessing fixes the issue but I don't want to modify that file, as I would have to revert the change if I ever make a pull request. The project is quite active so I believe this has something to do with my local configuration.
Any ideas why the script is failing to import _args_from_interpreter_flags only if I execute it via scons?
[UPDATE]
I did a fresh Gentoo install and the problem persists. I did some tests and I found this:
In a python terminal>
>>> import SCons.Script
>>> from subprocess import _args_from_interpreter_flags
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name _args_from_interpreter_flags
>>> import subprocess
>>> subprocess.__file__
'/usr/lib64/python2.7/site-packages/SCons/compat/_scons_subprocess.pyc'
But the output is different if I do this:
>>> import subprocess
>>> subprocess.__file__
'/usr/lib64/python2.7/subprocess.pyc'
So I update my question: Is this a bug? Can anybody reproduce it in other distros? If it's a bug, should I report it to Gentoo or to SCons?
[ANOTHER UPDATE]
Adding temp.extend([os.path.join(x, 'lib64') for x in prefs]) did't work, same error.
Adding print sys.path at the beginning of the compact module gives:
['/usr/lib64/python-exec/python2.7/scons-local-2.3.0',
'/usr/lib64/python-exec/python2.7/scons-local',
'/usr/lib64/python2.7/site-packages/lib32/scons-2.3.0',
'/usr/lib32/scons-2.3.0',
'/usr/local/lib32/scons-2.3.0',
'/usr/lib64/python2.7/site-packages/lib/python2.7/site-packages/scons-2.3.0',
'/usr/lib/python2.7/site-packages/scons-2.3.0',
'/usr/local/lib/python2.7/site-packages/scons-2.3.0',
'/usr/lib64/scons-2.3.0',
'/usr/lib64/python2.7/site-packages/lib32/scons',
'/usr/lib32/scons',
'/usr/local/lib32/scons',
'/usr/lib64/python2.7/site-packages/lib/python2.7/site-packages/scons',
'/usr/lib/python2.7/site-packages/scons',
'/usr/local/lib/python2.7/site-packages/scons',
'/usr/lib64/scons',
'/usr/lib64/python2.7/site-packages/RBTools-0.6-py2.7.egg',
'/usr/lib64/python27.zip',
'/usr/lib64/python2.7', #It's here, so what's the problem?
'/usr/lib64/python2.7/plat-linux2',
'/usr/lib64/python2.7/lib-tk',
'/usr/lib64/python2.7/lib-old',
'/usr/lib64/python2.7/lib-dynload',
'/usr/lib64/python2.7/site-packages',
'/usr/lib64/python2.7/site-packages/gtk-2.0',
'/usr/lib64/python2.7/site-packages/wx-2.8-gtk2-unicode']

It looks as if this isn't really a problem connected to SCons directly. You might have an alien "subprocess" module/package installed in your system. Also check out Cannot import name _args_from_interpreter_flags which seems to be related.
Based on your updated question: I tried to compile Godot on my machine (Python 2.7.3, SCons 2.3.1, Ubuntu 12.04 LTS) and it's running fine, so the problem is not related to the provided SConstruct (and its supporting build description files in subfolders). The "_scons_subprocess" module gets used only when the import of the original "subprocess.py" fails. So I suspect that the SCons start script sets up a wrong sys.path, which may happen under 64bit (see issue http://scons.tigris.org/issues/show_bug.cgi?id=2657 ).
After you added "temp.extend([os.path.join(x, 'lib64') for x in prefs])", your "print sys.path" statement shows paths like "/usr/lib64/python-exec" in its output. A Google search turned up the page http://forums.gentoo.org/viewtopic-t-985402-start-0.html for me. It describes an issue with Gentoo, where programs are installed as links to pip. Please follow the given advice and see if this fixes your problem.

It's a bug in Gentoo's scons-2.3.0 and scons-2.3.1 ebuilds (see bug report). It has been fixed in versions 2.3.1-r1 and higher.

Run python script from python using different python

I have a software that has python 2.5.5. I want to send a command that would start a script in python 2.7.5 and then proceed with the script.
I tried using
#!python2.7.5
and http://redsymbol.net/articles/env-and-python-scripts-version/
But I cant get it to work...
In my python 2.5.5 I can execute script as
execfile("c:/script/test.py")
The problem is that the 2.7.5 has a module comtypes + few other. I dont know how to install it for my 2.5.5 so I'm trying to start a separate script and run it under python27. Now another reason why I want it its because I want to take the load off program. I have 2 heavy tasks to perform. The second task is the one that need comptypes so sending it to external shell/app would do perfect trick. Is there a way to do it ?
I wish I could just type run("C:/Python27/python.exe % C:/script/test,py")
Thanks, bye.
Little update. I try to run
import os
os.system("\"C:\Python27\python.exe\" D:\test\runTest.py")
But I'm getting a quick pop up and close window saying that
Import Error : no module named site...
This works if I run from external shell but not from here :(
So I've tried another approach this time to add modules to python... in any case I run this :
import os
import sys
sys.path.append("C:/python27")
sys.path.append("C:/Python27/libs")
sys.path.append("C:/Python27/Lib")
sys.path.append("C:/Python27/Lib/logging")
sys.path.append("C:/Python27/Lib/site-packages")
sys.path.append("C:/Python27/Lib/ctypes")
sys.path.append("C:/Python27/DLLs")
import PyQt4
print PyQt4
import comtypes
import logging
but it crashes with C error...
Runtime Error :
Program: c:\Pr...
R6034
An application has made attempt to load the C runtime library incorectly.
blablabla....
How can I import it ? Maybe if I can import it I can run it directly from my app rather than starting separate python...
Traceback (most recent call last):
File "<string>", line 18, in <module>
File "C:\Python27\Lib\site-packages\comtypes\__init__.py", line 22, in <module>
from ctypes import *
File "C:\Python27\Lib\ctypes\__init__.py", line 10, in <module>
from _ctypes import Union, Structure, Array
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.
Another update to isseu
so I run now
import os
os.system("start cmd {D:\test\runTest.py}")
now this works and he open CMD with c:\Python27 as directory but he dont run the file... any hitns how to fix it?

Use "raw" strings so that you don't need to escape as much; I think the backslashes are what was breaking your code since backslash is considered an escape character except in raw strings.
Also, use the subprocess module. It makes it easy to avoid manually making a safe command string (the module takes care of that for you). All you need to do is pass it a list of arguments.
Your code would then look something like this:
import subprocess
proc = subprocess.Popen([r"C:\Python27\python.exe", r"D:\test\runTest.py"])
# then either do this
proc.wait() # wait until the process finishes
# or this
while True:
# NOTE: do something else here
# poll the process until it is done
if proc.poll() is not None:
break # break out of loop
See subprocess docs for Python 2 here. Be sure to check if a feature was added after Python 2.5 (the 2.5 docs aren't available online anymore AFAIK).
UPDATE:
I just noticed that you tried to use the Python 2.7 libraries and modules in your 2.5 code. This probably won't work due to new features added after 2.5. But it got me thinking how you might be able to make 2.7 work.
It may be that your Python2.7 install can't find its libraries; this is probably why you get the error Import Error : no module named site. You can do something like the above and modify the PYTHONPATH environment variable before starting the subprocess, like this:
import os
import subprocess
paths = [r"C:\python27", r"C:\python27\libs", r"C:\python27\Lib\site-packages", r"C:\python27\DLLs"]
paths += os.environ.get('PYTHONPATH', '').split(os.pathsep)
env27 = dict(os.environ)
env27['PYTHONPATH'] = os.pathsep.join(paths)
proc = subprocess.Popen([r"C:\Python27\python.exe", r"D:\test\runTest.py"], env=env27)

python pyjamas install on debian

i have installed pyjamas on debian
http://pyjs.org/getting_started.html
however my program does not find the module, what could be the problem, i have installed pyjamas correctly using apt-get
krisdigitx-virtual-machine ~ # python jamas.py
Traceback (most recent call last):
File "jamas.py", line 3, in <module>
from pyjamas import Window
ImportError: No module named pyjamas
krisdigitx-virtual-machine ~ #
#!/usr/bin/env python
from pyjs import Window
from pyjs.ui import RootPanel, Button
from pyjs.ui import HTML
def greet(sender):
Window.alert("Hello Krishna!")
b = Button("click me", greet)
Rootpanel().add(b)
After some research:
i had to do pyjsbuild jamas.py to get the output directory, however it gives me a new error
jamas TypeError: jamas.RootPanel().add is not a function

Since you get an
ImportError: No module named pyjamas
you may need to add the module's path to $PYTHONPATH with
export PYTHONPATH=$PYTHONPATH:/usr/share/pyjamas/library

Please go to http://pyjs.org/GettingHelp.html
The first link "Getting Started" is a detailled walkthrough for an installation to start from scratch. Basically, what it says there is: Get the up-to-date source code from the git repository.
All steps to get Pyjs and Pyjs Desktop running are described in the Wiki article in the necessary detail, but still concise enough.