We Keep Coding

nrfutil - "ImportError: No module named main" on Nixos

I'm using the tool nrfutil which is implemented in Python. To be able to use it under NixOS I was using a default.nix file, that installed nrfutil into a venv. This worked for some time very well. (The last build on the build server using Nix within an alpine container could build the software I'm working on 11 days ago successfully.) When I do exactly the same things (i.e. restarting the CI server build without changes), the build fails now complaining about pip being incorrect:
$ nix-shell
New python executable in /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
Not overwriting existing python script /home/matthias/source/tbconnect/bootloader/.venv/bin/python (you must use /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7)
Installing pip, wheel...
done.
Traceback (most recent call last):
File "/home/matthias/source/tbconnect/bootloader/.venv/bin/pip", line 6, in <module>
from pip._internal.main import main
ImportError: No module named main
To me it seems that the module main should exist:
$ ls -l .venv/lib/python2.7/site-packages/pip/_internal/main.py
-rw-r--r-- 1 matthias matthias 1359 10月 15 12:27 .venv/lib/python2.7/site-packages/pip/_internal/main.py
I'm not very much into the Python environment, so I don't know any further. Has somebody any pointer for me where to continue debugging? How is Python resolving modules? Why doesn't it find the module, that seems to be present to me?
This is my default.nix that I use to install pip:
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
python27Packages.pip
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv --no-setuptools .venv
export PATH=$PWD/.venv/bin:$PATH
#pip install nrfutil
pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}
I replaced pip install nrfutil with pip help to make sure the problem is not the package I try to install itself.
I'm still using python 2.7 as the nrfutil still is not fit for Python 3.
Anyway replacing python27 with python37 did not change the error I get when trying to start pip.)
NixOS version used locally is 19.09. Nix in the CI docker container is nixos/nix:latest which is the nix package manager on Alpine Linux.
Update:
Actually it works when I replace the call to pip install nrfutil with python2.7 -m pip install nrfutil. This actually confuses me even more. python2.7 is exactly the binary that is in the shebang of pip:
[nix-shell:~/source/tbconnect/bootloader]$ type python2.7
python2.7 is /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
[nix-shell:~/source/tbconnect/bootloader]$ type pip
pip is /home/matthias/source/tbconnect/bootloader/.venv/bin/pip
[nix-shell:~/source/tbconnect/bootloader]$ head --lines 2 .venv/bin/pip
#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-
Update 2:
I found out that another way to fix the problem is to edit .venv/bin/pip. This script tried the following import:
from pip._internal.main import main
Which I think is the new module path starting with pip 19.3. But I still have pip 19.2. When I change this line to:
from pip._internal import main
Running pip by typing pip is working.
The thing is I have no idea why the pip script is trying to load the new module path while NixOS still has the old version of pip.
I also opened an issue for NixOS on GitHub: https://github.com/NixOS/nixpkgs/issues/71178

I got your shell derivation to work by dropping the Python27Packages.pip,
(nix-shell) 2d [azul:/tmp/lixo12333] $
>>> pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------------- -------
behave 1.2.6
Click 7.0
crcmod 1.7
ecdsa 0.13.3
enum34 1.1.6
future 0.18.2
intelhex 2.2.1
ipaddress 1.0.23
libusb1 1.7.1
linecache2 1.0.0
nrfutil 5.2.0
parse 1.12.1
parse-type 0.5.2
pc-ble-driver-py 0.11.4
piccata 1.0.1
pip 19.3.1
protobuf 3.10.0
pyserial 3.4
pyspinel 1.0.0a3
PyYAML 4.2b4
setuptools 41.6.0
six 1.12.0
tqdm 4.37.0
traceback2 1.4.0
virtualenv 16.4.3
wheel 0.33.6
wrapt 1.11.2
(nix-shell) 2d [azul:/tmp/lixo12333] $
and my default.nix
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv .venv
export PATH=$PWD/.venv/bin:$PATH
pip install nrfutil
#pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}

The mysqlclient nightmare

I am trying to install mysqlclient in a centos system. These are some of the issues I am facing:
mysqlclient needs to be compiled in the installation system
to compile it, according to the official sources, we need python-devel and mysql-devel
what if I am installing mysqlclient in a virtualenv with python36? Do I need python36-devel instead of python-devel? Why is this not mentioned in the documentation?
will python36-devel match the python36 of my virtualenv? What if I am using my self-compiled python37 instead of the centos python36, will that match python36-devel? It's not like I have infinite options regarding the centos packages I can install ...
And some problems unrelated to mysqlclient, but related to centos:
centos does not have python36 by default. It seems something called epel-release must be installed.
python36-devel installs 36 (!!) dependencies:
--
Installed:
python36-devel.x86_64 0:3.6.8-1.el7
Dependency Installed:
dwz.x86_64 0:0.11-3.el7 groff-base.x86_64 0:1.22.2-8.el7 perl.x86_64 4:5.16.3-294.el7_6 perl-Carp.noarch 0:1.26-244.el7 perl-Encode.x86_64 0:2.51-7.el7 perl-Exporter.noarch 0:5.68-3.el7
perl-File-Path.noarch 0:2.09-2.el7 perl-File-Temp.noarch 0:0.23.01-3.el7 perl-Filter.x86_64 0:1.49-3.el7 perl-Getopt-Long.noarch 0:2.40-3.el7 perl-HTTP-Tiny.noarch 0:0.033-3.el7 perl-PathTools.x86_64 0:3.40-5.el7
perl-Pod-Escapes.noarch 1:1.04-294.el7_6 perl-Pod-Perldoc.noarch 0:3.20-4.el7 perl-Pod-Simple.noarch 1:3.28-4.el7 perl-Pod-Usage.noarch 0:1.63-3.el7 perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 perl-Socket.x86_64 0:2.010-4.el7
perl-Storable.x86_64 0:2.45-3.el7 perl-Text-ParseWords.noarch 0:3.29-4.el7 perl-Time-HiRes.x86_64 4:1.9725-3.el7 perl-Time-Local.noarch 0:1.2300-2.el7 perl-constant.noarch 0:1.27-2.el7 perl-libs.x86_64 4:5.16.3-294.el7_6
perl-macros.x86_64 4:5.16.3-294.el7_6 perl-parent.noarch 1:0.225-244.el7 perl-podlators.noarch 0:2.5.1-3.el7 perl-srpm-macros.noarch 0:1-8.el7 perl-threads.x86_64 0:1.87-4.el7 perl-threads-shared.x86_64 0:1.43-6.el7
python-rpm-macros.noarch 0:3-24.el7 python-srpm-macros.noarch 0:3-24.el7 python3-rpm-macros.noarch 0:3-24.el7 redhat-rpm-config.noarch 0:9.1.0-87.el7.centos zip.x86_64 0:3.0-11.el7
Utterly broken, if you ask me.

How to install dependencies from requirements.txt in a Yocto recipe for a local Python project

What I should have:
I want my Yocto Project to build a package for my Python project with all dependencies inside. The project has to run out of box on the resulting read-only sdcard image.
It simply should install all requirements in the required version to the package.
What I tried without luck:
Calling pip in do_install():
"pip/pip3 is not found", even it's in RDEPENDS.
Anyway, I really prefer this way.
With inherit pypi:
When trying with inherit pypi, it tries to get also my local sources (my pyton project) from pypi. And I have always to copy the requirements to the recipe. This is not my preferred way.
Calling pip in pkg_postinst():
It tries to install the modules on first start and fails, because the system has no internet connection and it's a read-only system. It must run out of the box without installation on first boot time. Does its stuff to late.
Where I'll get around:
There should be no need to change anything in the recipes when something changes in requirements.txt.
Background information
I'm working with Yocto Rocko in a Linux environment.
In the Hostsystem, there is no pip installed. I want to run this one installed from RDEPENDS in the target system.
Building the Package (only this recipe) with:
bitbake myproject
Building the whole sdcard image:
bitbake myProject-image-base
The recipe:
myproject.bb (relevant lines):
RDEPENDS_${PN} = "python3 python3-pip"
APP_SOURCES_DIR := "${#os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../../../../app-sources')}"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = " \
file://${APP_SOURCES_DIR}/myProject \
...
"
inherit allarch # tried also with pypi and setuptools3 for the pypi way.
do_install() { # Line 116
install -d -m 0755 ${D}/myProject
cp -R --no-dereference --preserve=mode,links -v ${APP_SOURCES_DIR}/myProject/* ${D}/myProject/
pip3 install -r ${APP_SOURCES_DIR}/myProject/requirements.txt
# Tried also python ${APP_SOURCES_DIR}/myProject/setup.py install
}
# Tried also this, but it's no option because the data MUST be included in the Package:
# pkg_postinst_${PN}() {
# #!/bin/sh -e
# pip3 install -r /myProject/requirements.txt
# }
FILES_${PN} = "/myProject/*"
Resulting Errors:
Expected to install the listed modules from requirements.txt into the myProject package, so that the python app will run directly on the resulting readonly sdcard image.
With pip, I get:
| /*/tmp/work/*/myProject/0.1.0-r0/temp/run.do_install: 116: pip3: not found
| WARNING: exit code 127 from a shell command.
| ERROR: Function failed: do_install ...
When using pypi:
404 Not Found
ERROR: myProject-0.1.0-r0 do_fetch: Fetcher failure for URL: 'https://files.pythonhosted.org/packages/source/m/myproject/myproject-0.1.0.tar.gz'. Unable to fetch URL from any source.
=> But it should not fetch myProject, since it is already local and nowhere remote.
Any ideas? What would be the best way to reach to a ready to use sdcard image without the need to change recipes when requirements.txt changes?

You should use RDEPENDS_${PN} to take care of your dependencies for your app in the recipe.
For example, assuming your python app needs aws-iot-device-sdk-python module, you should add it to RDEPENDS in the recipe. In your case, it would be like this:
RDEPENDS_${PN} = "python3 \
python3-pip \
python3-aws-iot-device-sdk-python \
"
Here's the link showing the Python modules supported by OpenEmbedded Layer.
https://layers.openembedded.org/layerindex/branch/master/layer/meta-python/
If the modules you need are not there, you will likely need to create recipes for the modules.

My newest findings:
Yocto/bitbake seems to suppress interpreting the requirements, because this breaks automatic dependency resolving what could lead to conflicts.
Reason: The required modules from setup.py would not be stored as independent packages, but as part of my package. So, bitbake does not know about this modules what could conflict with other packages that probably requires same modules in different versions.
What was in my recipe:
MY_INSTALL_ARGS = "--root=${D} \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
do_install() {
PYTHONPATH=${PYTHON_SITEPACKAGES_DIR} \
${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${MY_INSTALL_ARGS}
}
If I execute this outside of bitbake as python3 setup.py install ${MY_INSTALL_ARGS}, all will be installed correctly, but in the recipe, no requirements are installed.
There is a parameter --no-deps, but I didn't find where it is set.
I think there could be one possibility to exploit the requirements out of setup.py:
Find out where to disable --no-deps in the openembedded/poky layer for easy_install.
Creating a separate PYTHON_SITEPACKAGES_DIR
Install this separate PYTHON_SITEPACKAGES_DIR in eg the home directory as private python modules dir.
This way, no python module would trigger a conflict.
Since I do not have the time to experiment with this, I'll define now one recipe per requirement.

You try installing pip?
Debian
apt-get install python-pip
apt-get install python3-pip
Centos
yum install python-pip

Python3 can't see opencv-python, numpy, PyQt5

I installed opencv-python numpy PyQt5 using brew. Unfortunately it installed only for python in version 2 but I wanted it to ver 3. So normally when I am using python2 I can import those libs, but in python3 there is just error about not module found.
When I am typing for example brew info numpy, I am getting something like this:
numpy: stable 1.15.2 (bottled), HEAD Package for scientific computing with Python https://www.numpy.org/ /usr/local/Cellar/numpy/1.15.2 (967 files, 25.5MB) Poured from bottle on 2018-10-15 at 12:13:26 From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/numpy.rb
==> Dependencies Build: gcc ✔ Recommended: python ✔, python#2 ✔
==> Options
--without-python Build without python support
--without-python#2 Build without python2 support
--HEAD Install HEAD version
==> Analytics install: 33,262 (30d), 96,001 (90d), 314,869 (365d) install_on_request: 5,934 (30d), 19,037 (90d), 56,029 (365d) build_error: 0 (30d)
So like you can see there is just python2 in "Recommended". Is there any possibility to repair this mistake and link somehow those libs to python3?
I am using macOS High Sierra.

Why don't you use pip3 for installing packages for python3? Brew is for installing applications, not libraries.

Problem solved. Recently, Python.org sites stopped supporting TLS version 1.0 and 1.1. This helped:
curl https://bootstrap.pypa.io/get-pip.py | python3

Updating the SQLite3 build on my python install

noob programmer here, I'm trying to get the SQLite3 on my Python installation up-to-date (I currently have version 3.6.11, whereas I need at least version 3.6.19, as that is the first version that supports foreign keys). Here's my problem, though: I have no idea how to do this. I know next to nothing about the command line, and I don't really know what files to replace (if at all) in my python install. And before anyone asks, I'm already using the latest Pysql version – it's what's not up to date. Can anyone give me some pointers, or maybe a guide on how to update this?
I'm on Mac OSX 10.5.8, working with python 2.6.

I suggest using the 'pip' command on the command line.
pip search sqlite
pip install pysqlite

I just came fresh from installing this both in Mavericks and Mountain Lion.
This SO article mentions using the build_static method, which they say retrieves that latest version of the sqlite amalgamation. For some reason it didn't work (ie it didn't seem to download it or use it)
What I ended up doing was
Downloaded pysqlite as a tarball
Downloaded latest sqlite source amalgamation
Unzipped pysqlite into its folder
Unzipped sqlite and copied that to the pysqlite folder
Opened setup.cfg and commented out all of the directives
In Terminal, went to the pysqlite folder, and then:
$ python setup.py build_static install
This compiled pysqlite using the libraries from the latest sqlite sources. And then in Python, I used it as:
import pysqlite2.dbapi2 as sqlite3

I recently installed python from source and used the following commands to install both SQLite from source and Python 2.7.13 from source.
for SQLite3 you can use the following commands
$SQLITE_INSTALL_LOCATION
$ curl -O http://www.sqlite.org/sqlite-autoconf-3070603.tar.gz
$ tar xvfz sqlite-autoconf-3070603.tar.gz
$ cd sqlite-autoconf-3070603
$ ./configure --prefix=$SQLITE_INSTALL_LOCATION --disable-static CFLAGS="-g"
$ make && make install
Then when I compiled my python I edited the setup.py in the root of the Python source
$ curl -O https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz
$ tar xvfz Python-2.7.13.tgz
Python-2.7.13/setup.py -- add the path to your SQLite install here:
```
...
# We hunt for #define SQLITE_VERSION "n.n.n"
# We need to find >= sqlite version 3.0.8
sqlite_incdir = sqlite_libdir = None
sqlite_inc_paths = [ '/usr/include',
'/usr/include/sqlite',
'/usr/include/sqlite3',
'/usr/local/include',
'/usr/local/include/sqlite',
'/usr/local/include/sqlite3',
$SQLITE_INSTALL_LOCATION/include,
]
...
Once you've changed the setup.py in your python source finish up compiling and installing python assuming the install location is $PYTHON_INSTALL_LOCATION
$ cd Python-2.7.13
$ LD_RUN_PATH=$SQLITE_INSTALL_LOCATION/lib ./configure --prefix=$PYTHON_INSTALL_LOCATION --enable-shared --enable-unicode=ucs4
$ LD_RUN_PATH=$SQLITE_INSTALL_LOCATION/lib make
$ LD_RUN_PATH=$SQLITE_INSTALL_LOCATION/lib make install
Once you do that you should have a Python version with SQLite3 support installed at $PYTHON_INSTALL_LOCATION/bin/python
Hope this helps!

Disclaimer: I'm not a Mac User, but by common knowledge i give you
this info.
You could follow the next instructions:
Use Homebrew
As this page mention: If you need to upgrade sqlite, you could use Homebrew.
Homebrew implies an aditional "software manager". So you should know how to use it before.
Install it from the source
As this page metion:
Download the sqlite-autoconf package
Compile it and install it:
:
$ tar xvfz sqlite-autoconf-3071502.tar.gz
$ cd sqlite-autoconf-3071502
$ ./configure --prefix=/usr/local
$ make
$ make install
Either Homebrew or Source, verfy it
>>> import sqlite3
>>> sqlite3.version_info
(2, 4, 1)
>>> sqlite3.sqlite_version_info
(3, 6, 11)
>>> from pysqlite2 import dbapi2 as sqlite3
>>> sqlite3.version_info
(2, 5, 5)
>>> sqlite3.sqlite_version_info
(3, 6, 18)
Maybe you need to uninstall previous version of pysqlite. In any way, you should read this answer to understand better the sqlite/python relationship.

https://pip.pypa.io/en/latest/installing.html
python get-pip.py
python [complete path]
python c:\folder\get-pip.py