We Keep Coding

Build Error on apple silicon M1 with docker

I was trying to dockerize a flask application with a third-party cli (plastimatch) on my M1.
I used ubuntu:18.04 as base image. The build on more recent version would fail with the error message 'no installation candidate was found'. The first odd thing I noticed was that the exact same build would succeed on a linux server.
I used a local venv to finalize the application and as I started to dockerize everything I got the following error:
#16 22.37 note: This error originates from a subprocess, and is likely not a problem with pip.
#16 22.37 ERROR: Failed building wheel for pylibjpeg-libjpeg
#16 22.37 Failed to build pylibjpeg-openjpeg pylibjpeg-libjpeg
#16 22.37 ERROR: Could not build wheels for pylibjpeg-openjpeg, pylibjpeg-libjpeg, which is required to install pyproject.toml-based projects
These python packages are wrappers for different C++ libaries, that handle images. The local build fails and the build on our linux server runs perfectly fine.
Has anyone noticed similar problems when dockerizing there applications locally in development? And are there any solutions to it?
Here is the reference of the used Dockerfile and requirements.txt (currently missing specific versions):
FROM ubuntu:18.04 as base
RUN apt-get update -y && apt-get install -y && apt-get upgrade -y
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get install -y python3.8 python3-pip
RUN rm /usr/bin/python3 && ln -s /usr/bin/python3.8 /usr/bin/python3
RUN apt-get install -y \
plastimatch \
zlib1g \
cmake
WORKDIR /app
COPY requirements.txt requirements.txt
RUN python3 -m pip install -U --force-reinstall pip
RUN pip3 install --upgrade pip setuptools wheel
RUN pip3 install -r requirements.txt
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
FROM base as upload-dev
RUN echo "Building dev version"
COPY requirements_dev.txt requirements_dev.txt
RUN pip3 install -r requirements_dev.txt
COPY . .
python-dotenv
cython
pynrrd
flask-cors
Flask
Werkzeug
httplib2
numpy
pydicom
highdicom
dicomweb-client
Update: 01. July 2022
I could track down the error.
The problem was the missing wheel of some third party libraries. If no wheel could be located, the source code will be fetched and installed by a compiler. This crashed on my machine during the installation of libraries that use C++ at their core.
An easy approach to fix this problem would be to directly use the linux AMD64 image.
FROM --platform=linux/amd64 $YOUR_BASE_IMAGE
This would be a bit slower but for most development environments sufficient.
A detailed explanation: https://pythonspeed.com/articles/docker-build-problems-mac/

For me, the fix was to install Rosetta 2, which is included in the Docker documentation: https://docs.docker.com/desktop/mac/apple-silicon/#system-requirements
softwareupdate --install-rosetta

Problem building docker with numpy and pandas over arm64

I'm trying to build a docker image with docker-compose in my ARM64 rasperry pi but it seems to be imposible.
This is my dockerfile:
FROM python:3.6-slim
RUN apt-get update && apt-get -y install python3-dev
RUN apt-get -y install python3-numpy
RUN apt-get -y install python3-pandas
ENTRYPOINT ["python3", "app.py"]
It seems to be OK, but when app.py is run, it gives an error: "Module numpy not found", and the same for pandas module.
If I try to install numpy and pandas using pip:
RUN pip install numpy pandas
It gives me an error or, more usually, the raspberry just gets frozen and I have to unplug it to recover.
I have tried with different versions of python for the source image and also using several ubuntu images and installing python.
Any idea of how can I install numpy and pandas in docker for my raspberry pi (ARM64)?
Thanks

The problems seems to be with the python version. I'm using a python3.6 docker image but, both python3-numpy and python3-pandas packages require python3.5, so when those packages are installed a new version of python is also installed. This is why when I'm trying to import those modules the python interpreter can't found them, because they are installed for another python version.
Finaly I solved it using a generic docker debian image and installing python3.5 myself instead of using a python docker image.
FROM debian:stretch-slim
RUN apt-get update && apt-get -y dist-upgrade
RUN apt-get -y install build-essential libssl-dev libffi-dev python3.5 libblas3 libc6 liblapack3 gcc python3-dev python3-pip cython3
RUN apt-get -y install python3-numpy python3-sklearn
RUN apt-get -y install python3-pandas
COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt

(Disclaimer: The Raspberry Pi 3 B+ is probably too slow to install big dependecies like numpy)
This Dockerfile worked for me on the Raspberry Pi 3 B+ with Software-Version: Linux raspberrypi 5.10.63-v7+ (Consider updating it)
FROM python:3.9-buster
WORKDIR /
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
I am not sure, but I think it helped also to clean docker i.e. remove all images and containers with the following commands:
Warning: This commands deletes all images and containers!
$ docker container prune
$ docker image prune -a
Or reset Docker completely (deletes also volumes and networks):
$ docker system prune --volumes

I recommend to create requirements.txt file.
Inside you can declare packets to install.
The `Dockerfile':
FROM python
COPY app.py /workdir/
COPY requirements.txt /workdir/
WORKDIR /workdir
RUN pip install --trusted-host pypi.python.org -r requirements.txt
CMD python app.py
edit
I create Dockerfile which import pandas lib and then checking if it work:
cat Dockerfile
FROM python
COPY app.py /workdir/
WORKDIR /workdir
RUN python -m pip install pandas
CMD python app.py

apache-airflow fails install

I'm trying to install apache-airflow the recommended way with pip install apache-airflow. During the install of pendulum (a dependency), I get an error:
error: can't copy 'pendulum/parsing': doesn't exist or not a regular file
I think it's related to Python distutils error: "[directory]... doesn't exist or not a regular file", but that doesn't give an answer as to how one resolves this when using pip. Pulling the tar for pendulum and installing using python setup.py install works, but then when subsequently I do pip install apache-airflow again, it sees that pendulum is already installed, UNINSTALLS, and then tries to install again using pip, resulting in the same error. I'm using a docker container and installing python-setuptools with apt-get before I do any of this. Here's my dockerfile, fwiw...
FROM phusion/baseimage:0.10.1
MAINTAINER a curious dev
RUN apt-get update && apt-get install -y python-setuptools python-pip python-dev libffi-dev libssl-dev zip wget
ENV SLUGIFY_USES_TEXT_UNIDECODE=yes
RUN wget https://files.pythonhosted.org/packages/5b/57/71fc910edcd937b72aa0ef51c8f5734fbd8c011fa1480fce881433847ec8/pendulum-2.0.4.tar.gz
RUN tar -xzvf pendulum-2.0.4.tar.gz
RUN cd pendulum-2.0.4/ && python setup.py install
RUN pip install apache-airflow
CMD airflow initdb && airflow webserver -p 8080
Does anyone see anything I'm doing wrong? I haven't found anyone else with this error so I think there's something really obvious I'm missing. Thanks for reading.

Upgrade pip first.
FROM phusion/baseimage:0.10.1
RUN apt-get update && apt-get install -y python-setuptools python-pip python-dev libffi-dev libssl-dev zip wget
ENV SLUGIFY_USES_TEXT_UNIDECODE=yes
RUN pip install -U pip
RUN pip install apache-airflow
CMD airflow initdb && airflow webserver -p 8080
seems to work fine for me.

Docker build fails if pip is installed and updated to 10.0.1 in a single RUN section

How would you explain docker build failure with Dockerfile1, and it's success with Dockerfile2 (see below).
1)
// Dockerfile1
FROM ubuntu:16.04
RUN apt-get -y update && \
apt-get -y install python-pip python-dev build-essential && \
pip install --upgrade pip && \
pip install --upgrade virtualenv
docker build . fails with the following err
Collecting pip
Downloading
https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Found existing installation: pip 8.1.1
Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside
environment /usr
Successfully installed pip-10.0.1
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
from pip import main
ImportError: cannot import name main
The command '/bin/sh -c apt-get -y update && apt-get -y install
python-pip python-dev build-essential && pip install --upgrade pip && pip install --upgrade virtualenv && virtualenv /venv' returned a non-zero code: 1
However, it succeeds if we split it into two RUN.
2)
// Dockerfile2
FROM ubuntu:16.04
RUN apt-get -y update && \
apt-get -y install python-pip python-dev build-essential && \
pip install --upgrade pip
RUN pip install --upgrade virtualenv
The installation failure for pip is related to this reported issue. So my questions:
Why does docker build fail in the first case? If we just run those command in bash, there wont be any error.
Why does docker build succeed in the second case? How is it related to layering concept in docker?
Why specifying pip version in Dockerfile1 (i.e. pip install --upgrade pip=0.9.3) solves the problem too?
Update (May 6, 2018):
I've figured out the issue. What happens here is as below:
apt-get -y install python-pip installs an old version of pip whose shim script import pip's main directly.
pip install --upgrade pip installs pip 10.0.1 and moves main into an internal directory _internal. It adds its shim script to PATH.
Calling pip fails as it still calls the old shim script as it's path is cached. Running hash -d pip in between fixes the issue.
So apparently, splitting install and update into two RUN sections has similar effect as hash -d pip. Workarounds (also suggested by Andriy Maletsky) are 1) pin pip update to 9.0.3, or 2) install (latest) pip from source in the first place, or 3) use hash -r in between, or 4) use another RUN command for later use of pip.

The problem is that pip executable (/usr/bin/pip) breaks while updating pip from version 9 to version 10.
Possible solutions:
1. Do not update and use pip v9
2. Do not use apt-get to install pip. Download it manually.
Why does docker build fail in the first case? If we just run those command in bash, there wont be any error.
No, there will be an error. I ran those commands inside docker run --rm -it ubuntu:16.04 bash and got it.
Why does docker build succeed in the second case? How is it related to layering concept in docker?
I believe you made a mistake somewhere in second RUN and it's silencing an error (in that place which you didn't provide). For example, this will work (because ; used instead of && and execution doesn't break after bad command):
RUN pip install --upgrade virtualenv && \
virtualenv /venv; source /venv/bin/activate
Why specifying pip version in Dockerfile1 (i.e. pip install --upgrade pip=0.9.3) solves the problem too?
Because this pip bug appeared in version 10.
P.S. You should not update or manually change files you added to your system via apt-get (you are doing this via pip install --upgrade pip).

install opencv-python using Python3 in Ubuntu 15.04

I wish to install opencv-python via the command in Ubuntu 15.04 machine
pip3 install opencv-python
But as soon as I run this command I get the following error :
Downloading/unpacking opencv-python
Could not find any downloads that satisfy the requirement opencv-python
Cleaning up...
No distributions at all found for opencv-python
Storing debug log for failure in /home/Nadeem/.pip/pip.log
Any help would be much appreciated.
Thanks!!

You can install opencv from source.
Follow this link to do so.
Or you may need to upgrade your pip3 using the following command
pip3 install --upgrade pip
EDIT
For completeness(and in case the link is broken) I've listed here steps to compile and install OpenCV from source on Ubuntu(Tested on Ubuntu 14.04 LTS with python 3).
Step 1 Update the packages
sudo apt-get update
sudo apt-get upgrade
Step 2 Install dependencies
sudo apt-get install build-essential cmake git pkg-config # Developer tools required to compile opencv
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev # Libraries required to read various image format from disk
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev # Libraries required to read various video formats
sudo apt-get install libgtk2.0-dev # Required by opencv for GUI features
sudo apt-get install libatlas-base-dev gfortran # Packages used by opencv to optimize various functions.
pip3 install --upgrade pip
Step 3 setup virtual environment(using conda)
conda create -n opencv-exmaple-env python=3.6
source activate opencv-exmaple-env # Activate the envirnoment
Step 4 Install packages required to compile opencv
sudo apt-get install python3.6-dev # If the python version is not 3.6 then make changes to this command accordingly.
pip install numpy # This should be done after the environment in Step 3 is activated
Step 5: Build and install OpenCV 3.0 with Python 3.4+ bindings
5.1 Clone the opencv source
cd ~
mkdir opencv-source
cd opencv-source
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.3.0 # Branch you want to compile from
5.2 Clone Opencv Contrib rep
Contains exptra functionalities such as standard keypoint detectors and local invariant descriptors (such as SIFT, SURF, etc.)
cd ~
mkdir opencv-contrib
cd opencv-contrib
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib
git checkout 3.3.0 # The version you want to compile
5.3 Compile,build and install
cd ~/opencv-source/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv-contrib/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig
5.4 Link installed opencv object file to python site packages
ln -s /usr/local/lib/python3.6/site-packages/cv2.so /path-to-python-sitepackages-of-the-environment/cv2.so
6 Verify installation
import cv2
If the above code runs without error then opencv is installed successfully.

At first upgrade pip using sudo.
arsho:~/workspace $ sudo pip3 install --upgrade pip
Successfully installed pip
Now install opencv-python again using sudo command.
arsho:~/workspace $ sudo pip3 install opencv-python
Successfully installed numpy-1.13.1 opencv-python-3.3.0.10
Finally check the opencv-python version and location information using pip.
arsho:~/workspace $ pip3 show opencv-python
---
Name: opencv-python
Version: 3.3.0.10
Location: /usr/local/lib/python3.4/dist-packages
Requires: numpy
I have tested this using Ubuntu 14.04.5 LTS in https://c9.io/.