We Keep Coding

nrfutil - "ImportError: No module named main" on Nixos

I'm using the tool nrfutil which is implemented in Python. To be able to use it under NixOS I was using a default.nix file, that installed nrfutil into a venv. This worked for some time very well. (The last build on the build server using Nix within an alpine container could build the software I'm working on 11 days ago successfully.) When I do exactly the same things (i.e. restarting the CI server build without changes), the build fails now complaining about pip being incorrect:
$ nix-shell
New python executable in /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
Not overwriting existing python script /home/matthias/source/tbconnect/bootloader/.venv/bin/python (you must use /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7)
Installing pip, wheel...
done.
Traceback (most recent call last):
File "/home/matthias/source/tbconnect/bootloader/.venv/bin/pip", line 6, in <module>
from pip._internal.main import main
ImportError: No module named main
To me it seems that the module main should exist:
$ ls -l .venv/lib/python2.7/site-packages/pip/_internal/main.py
-rw-r--r-- 1 matthias matthias 1359 10月 15 12:27 .venv/lib/python2.7/site-packages/pip/_internal/main.py
I'm not very much into the Python environment, so I don't know any further. Has somebody any pointer for me where to continue debugging? How is Python resolving modules? Why doesn't it find the module, that seems to be present to me?
This is my default.nix that I use to install pip:
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
python27Packages.pip
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv --no-setuptools .venv
export PATH=$PWD/.venv/bin:$PATH
#pip install nrfutil
pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}
I replaced pip install nrfutil with pip help to make sure the problem is not the package I try to install itself.
I'm still using python 2.7 as the nrfutil still is not fit for Python 3.
Anyway replacing python27 with python37 did not change the error I get when trying to start pip.)
NixOS version used locally is 19.09. Nix in the CI docker container is nixos/nix:latest which is the nix package manager on Alpine Linux.
Update:
Actually it works when I replace the call to pip install nrfutil with python2.7 -m pip install nrfutil. This actually confuses me even more. python2.7 is exactly the binary that is in the shebang of pip:
[nix-shell:~/source/tbconnect/bootloader]$ type python2.7
python2.7 is /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
[nix-shell:~/source/tbconnect/bootloader]$ type pip
pip is /home/matthias/source/tbconnect/bootloader/.venv/bin/pip
[nix-shell:~/source/tbconnect/bootloader]$ head --lines 2 .venv/bin/pip
#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-
Update 2:
I found out that another way to fix the problem is to edit .venv/bin/pip. This script tried the following import:
from pip._internal.main import main
Which I think is the new module path starting with pip 19.3. But I still have pip 19.2. When I change this line to:
from pip._internal import main
Running pip by typing pip is working.
The thing is I have no idea why the pip script is trying to load the new module path while NixOS still has the old version of pip.
I also opened an issue for NixOS on GitHub: https://github.com/NixOS/nixpkgs/issues/71178

I got your shell derivation to work by dropping the Python27Packages.pip,
(nix-shell) 2d [azul:/tmp/lixo12333] $
>>> pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------------- -------
behave 1.2.6
Click 7.0
crcmod 1.7
ecdsa 0.13.3
enum34 1.1.6
future 0.18.2
intelhex 2.2.1
ipaddress 1.0.23
libusb1 1.7.1
linecache2 1.0.0
nrfutil 5.2.0
parse 1.12.1
parse-type 0.5.2
pc-ble-driver-py 0.11.4
piccata 1.0.1
pip 19.3.1
protobuf 3.10.0
pyserial 3.4
pyspinel 1.0.0a3
PyYAML 4.2b4
setuptools 41.6.0
six 1.12.0
tqdm 4.37.0
traceback2 1.4.0
virtualenv 16.4.3
wheel 0.33.6
wrapt 1.11.2
(nix-shell) 2d [azul:/tmp/lixo12333] $
and my default.nix
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv .venv
export PATH=$PWD/.venv/bin:$PATH
pip install nrfutil
#pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}

GDAL libraries, who does what

I'm struggling installing GDAL on ubuntu 16.04 to work with GeoDjango (Django 2.1, python3), so I need to understand what I'm actually installing.
What is the rôle of each library/package/module ?
apt
gdal-bin (A 'C' library containing the actual functions ?)
python-gdal (The same in python, or just some kind of bridge ?)
python3-gdal (see above, but for python3. Does it need python-gdal ?)
pip
gdal
pygdal
What is the link between pip modules and apt packages here ?

Every piece of info is available, if one is willing to search for it.
DEBs (installed system-wide):
gdal-bin ([Ubtu]: Package: gdal-bin) - a collection of gdal related binaries (tools and utilities)
python3-gdal ([Ubtu]: Package: python3-gdal) - Python 3 bindings, extensions (.sos) and some wrapper scripts, which enable gdal usage from Python
python-gdal - the same thing, but for Python 2 (totally unrelated to previous item)
WHLs (installed as Python modules to the interpreter used to launch pip):
GDAL ([PyPI]: GDAL) - the sources (.tar.gz) for #2. (and / or #3.). During pip install phase, they are built and installed for current Python
pygdal ([PyPI]: pygdal) - same thing (but for VEnv?) as previous item. It seems to be a lighter version (it doesn't contain the scripts)
But, all of the above depend on libgdal ([Ubtu]: Package: libgdal1i), which is the gdal library.

Install ROS Kinetic on MacOS under a separate Anaconda environment

I'm installing ROS Kinetic on MacOS Sierra 10.12.6 under a seperate environment under Python Anaconda distribution.
Here is my python version(my seperate python environment in Anaconda is called "ros-kinetic"):
(ros-kinetic) jizhe#jizhedeMacBook-Pro  ~/ros_catkin_ws  python --version
Python 3.6.4 :: Anaconda, Inc.
I followed the guide: http://wiki.ros.org/kinetic/Installation/OSX/Homebrew/Source.
When I execute the "rosdep install" command, I got the following error:
(ros-kinetic) jizhe#jizhedeMacBook-Pro ~/ros_catkin_ws  rosdep install --from-paths src --ignore-src --rosdistro kinetic -y
ERROR: the following packages/stacks could not have their rosdep keys resolved
to system dependencies:
actionlib: No definition of [python-wxtools] for OS [osx]
catkin: No definition of [google-mock] for OS [osx]
roslisp: No definition of [google-mock] for OS [osx]
Obviously, there is something wrong with the package "python-wxtools" and "google-mock".
I've already install the wxpython via "brew install wxpython", and here is the wxpython info:
(ros-kinetic) jizhe#jizhedeMacBook-Pro  ~/ros_catkin_ws  brew info wxpython
wxpython: stable 3.0.2.0 (bottled)
Python bindings for wxWidgets
https://www.wxwidgets.org/
/usr/local/Cellar/wxpython/3.0.2.0_1 (1,108 files, 37.9MB) *
Poured from bottle on 2018-02-24 at 03:56:21
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/wxpython.rb
==> Dependencies
Required: wxmac ✔
==> Caveats
Python modules have been installed and Homebrew's site-packages is not
in your Python sys.path, so you will not be able to import the modules
this formula installed. If you plan to develop with these modules,
please run:
mkdir -p /Users/jizhe/Library/Python/2.7/lib/python/site-packages
echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")' >> /Users/jizhe/Library/Python/2.7/lib/python/site-packages/homebrew.pth
And I've already run the command:
$ mkdir -p ~/Library/Python/2.7/lib/python/site-packages
$ echo "$(brew --prefix)/lib/python2.7/site-packages" >> ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth
So, what's wrong with the package "google-mock" and "python-wxtools", how can I solve this problem?

Try installing with --skip-keys python-wxtools --skip-keys google-mock. It worked for me.

There is a ticket for this issue. Basically the workaround is to install googletest manually. Not sure what's wrong with python-wxtools.
edit: If you're using ros-install-osx, you can add the following lines to your rosdeps.yaml file:
google-mock:
osx:
homebrew:
packages: []

I found this question from a Google search, just for your information in 2021 installing ROS in conda also on macOS should be much straightforward, see the following links:
https://medium.com/robostack/cross-platform-conda-packages-for-ros-fa1974fd1de3
https://github.com/RoboStack/ros-noetic

Python 3.6.4 install OpenCV 3.4.0 and Opencv contrib on Windows 10

I am using OpenCv 3.4.0, Python 3.6.4, Cmake 3.10.2, OpenNI-Windows-x64-2.2, SensorKinect093-Bin-Win32-v5.1.2.1, Visual Studio 2015 and Windows 10. I want to build the cv2.pyd for Python 3.6.4.
I follow the step in many others way such as github and OpenCV install opencv_contrib on Windows
After configure in the Cmake, It shows the output about python 3 like this:
Python (for build): C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/python.exe
And the Cmake havn't the BUILD_opencv_python3, PYTHON3_INCLUDE_DIR and PYTHON3_LIBRARY, so I append the Bool value of BUILD_opencv_python3 and the locations of PYTHON3_INCLUDE_DIR, PYTHON3_LIBRARY manually. The locations are as followings:
PYTHON3_EXECUTABLE C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/python.exe
PYTHON3_INCLUDE_DIR C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/include
PYTHON3_INCLUDE_DIR2
PYTHON3_LIBRARY C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/libs/python36.lib
PYTHON3_LIBRARY_DEBUG
PYTHON3_NUMPY_INCLUDE_DIRS C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/lib/site-packages/numpy/core/include
PYTHON3_PACKAGES_PATH C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/Lib/site-packages
After that, Cmake can't find the Libraries of Python3, albeit it have the correct location of PYTHON3_LIBRARY. The output about Python3 in Cmake like this:
Python 3:
Interpreter: C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/python.exe (ver 3.6.4)
Libraries: NO
numpy: C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/lib/site-packages/numpy/core/include (ver 1.14.0)
packages path: C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/Lib/site-packages
Python (for build): C:/Users/zhong/AppData/Local/Programs/Python/Python36-32/python.exe
How can I configure the python 3.6.4 with Cmake correctlly? I want to use deep camera so I have to compile the Opencv with OPENNI2.
I am looking for help how can I solve it? Thank you very much.

I give up the python 3, and install the python 2.7.14, OpenCV 3.2.0 instead.
Following the step in OpenCV install opencv_contrib on Windows is easy to install...
It take me too much time to install, I suppose that OpenCV 3.4.0 do not support for Python 3.6.4.

I just found this site which contains instructions:
https://www.scivision.co/install-opencv-python-windows/
I was able to use the "pip install opencv-contrib-python" command to have it auto downloaded and installed. I am using Python 3.7 (but it should also work for the version you listed) and have an anaconda environment activated. If you use anaconda, just make sure you install pip for the environment you are using and activate it before using the pip install command.

What's the difference between homebrew python and caskroom python?

The page Installing Python on Mac OS X suggests that the OS X version is OK for learning but not great for writing real programs; solution - install from Homebrew.
I don't think the caskroom existed when they wrote this page though. Basically, I just want to install the most optimal version for doing Python programming.
When I do a search I get this output:
$ brew search python
boost-python gst-python python python3 wxpython zpython
Caskroom/cask/mod_python homebrew/python/vpython Caskroom/cask/python
homebrew/python/python-dbus homebrew/versions/gst-python010 Caskroom/cask/python3
This is what homebrew reports:
macosx-10-9:~ vagrant$ brew cask info python
python: 2.7.9
Python
https://www.python.org/
Not installed
https://github.com/caskroom/homebrew-cask/blob/master/Casks/python.rb
==> Contents
python-2.7.9-macosx10.6.pkg (pkg)
macosx-10-9:~ vagrant$ brew info python
python: stable 2.7.10 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org
/usr/local/Cellar/python/2.7.10_2 (4906 files, 77M) *
Poured from bottle
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb
So, what's the effective difference between these two packages?

Caskroom python installs the Python Mac OS X packages from https://www.python.org/downloads/mac-osx/ as they are provided there.
brew install python will install from source and under /usr/local/Cellar/python/... and properly symlink /usr/local/bin/python.
The latter is the "proper homebrew approach" (TM) and will allow updates with brew upgrade.