We Keep Coding

Alpine Docker image with pandas installed on Pipenv

We need an alpine based docker image that can have pandas package within pipenv
This works.
FROM python:3-alpine
RUN apk add g++ && \
pip install numpy
But, our process needs the install on pipenv and below fails with error pipenv not found
FROM python:3-alpine
RUN apk add g++ && \
pipenv install numpy
Note pipenv is installed in earlier docker statements. However, even the below fails, with pipenv not found
FROM python:3-alpine
RUN apk add g++ && \
pip install --user pipenv && \
pipenv install numpy
Any suggestions?

pipenv isn't available because pip install --user pipenv installs it in /root/.local/bin, which isn't listed in the search path ($PATH). The easiest way to fix it would be to install pipenv without the --user flag. It will then be installed in /usr/local/bin/:
FROM python:3-alpine
RUN apk add g++ && \
pip install pipenv && \
pipenv install numpy
If you run through the build steps manually, it gives you a warning about this:
docker run --rm -ti python:3-alpine /bin/sh
apk add g++
pip install --user pipenv this shows the warning below:
WARNING: The scripts pipenv and pipenv-resolver are installed in '/root/.local/bin' which is not on PATH.

How do I install a specific python version without pyenv within nvidia docker so that it interacts well with poetry?

I am trying to build a docker image that contains cuda, cudnn and python, each with specific versions that are templatable as a base for downstream users.
(In this example I have replace all the irrelevant templating with hard-coded versions, this is just FYI as a motivation).
Please note that the following questions are not duplicates:
How to install python in a docker image? does not involve poetry
Integrating Python Poetry with Docker Does not concern itself with installing dependencies
How do I integrate pyenv, poetry, and docker? This works for me already, I am looking for a different solution
I have achieved what I want using pyenv to install the specific python version within docker inside the nvidia image.
However, this solution is not optimal since the resulting image is about 1.5GB larger than what I think should be possible. Sidenote: I know that there are other ways to reduce the image size further that I have not done in this example. This is not the question here.
I have prepared a dummy pyproject.toml and poetry.lock to demonstrate the issue that I am currently facing:
pyproject.toml
[tool.poetry]
name = "example_project"
version = "1.0.0"
description = ""
authors = ["RunOrVeith"]
[tool.poetry.dependencies]
python = ">=3.8,<3.11"
scipy = "^1.9.3"
[build-system]
requires = ["poetry-core>=1.1.0"]
build-backend = "poetry.core.masonry.api"
Working Dockerfile.pyenv
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ARG PYTHON_VERSION=3.8
ENV DEBIAN_FRONTEND=noninteractive
# Set-up necessary Env vars for PyEnv
ENV PYENV_ROOT /root/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
ENV PATH="/root/.local/bin/:$PATH"
# Install essentials for pyenv https://github.com/pyenv/pyenv/wiki
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates \
curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev mecab-ipadic-utf8 git \
&& rm -rf /var/lib/apt/lists/*
# Install pyenv
RUN set -ex \
&& curl https://pyenv.run | bash \
&& pyenv update \
&& pyenv install $PYTHON_VERSION \
&& pyenv global $PYTHON_VERSION \
&& pyenv rehash \
&& pip install --upgrade pip
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example # The template that I want to provide ends here, this is just for demoing the issue
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
The version that doesn't work Dockerfile.plain
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHON_VERSION=3.8
ENV PATH="/root/.local/bin/:$PATH"
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys A4B469963BF863CC \
&& apt update \
&& apt install -y git curl \
&& apt install -y --no-install-recommends make build-essential
# Don't be confused, distutils-3.9 also installs python 3.8 https://github.com/deadsnakes/issues/issues/150
RUN apt install -y --no-install-recommends python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-distutils python${PYTHON_VERSION}-venv \
&& update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 10 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 10 \
&& apt-get install -y --no-install-recommends python3-pip python3-setuptools \
&& update-alternatives --install /usr/local/bin/pip pip /usr/bin/pip 10 \
&& update-alternatives --install /usr/local/bin/pip3 pip3 /usr/bin/pip 10 \
&& apt-get clean
WORKDIR /virtualenvs
RUN curl -sSL https://install.python-poetry.org | python${PYTHON_VERSION} - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
You can build this using
DOCKER_BUILDKIT=1 docker build -t github:example-plain --target example -f Dockerfile.plain .
and then run using
docker run -it github:example-plain bash
Here is the issue:
All the following commands are run from within the docker image.
According to poetry, everything is installed:
root#5e1ffb1f971c:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
However using regular pip, there is nothing, and imports also fail.
If I use poetry to import something, it also does not work.
root#5e1ffb1f971c:/app# pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
root#5e1ffb1f971c:/app# pip freeze
root#5e1ffb1f971c:/app# python -c "import scipy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
root#5e1ffb1f971c:/app# poetry run python -c "import scipy"
Skipping virtualenv creation, as specified in config file.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
What is also interesting is that if I upgrade pip with poetry it tells me it can't uninstall pip, I am assuming this is due to this ubuntu patch that tries to prevent me from breaking the system (even though I just install pip).
Afterwards, the poetry pip executable also points somewhere else.
root#5e1ffb1f971c:/app# poetry run pip install --upgrade pip
Skipping virtualenv creation, as specified in config file.
Collecting pip
Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-22.3.1
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
So how do I set this up so that I get a fresh python install of whichever version I configure, and it works with poetry? It is also required that the python and python3 aliases point to whatever poetry is using.
Reference with working version:
If I do the same commands with the working version using pyenv, it looks like this:
root#c0a9af7f05b4:/app# pip freeze
numpy==1.23.4
scipy==1.9.3
root#c0a9af7f05b4:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#c0a9af7f05b4:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)
root#c0a9af7f05b4:/app# pip --version
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)

No module named PyInstaller' after what appears to be a successful install

I am building a docker image. Within it I am trying to install a number of python packages within one RUN. All packages within that command are installed correctly, but PyInstaller is not for some reason, although the build logs make me think that it should have been: Successfully installed PyInstaller
The minimal Dockerfile to reproduce the issue:
FROM debian:buster
RUN apt-get update && \
apt-get install -y \
python3 \
python3-pip \
unixodbc-dev
RUN python3 -m pip install --no-cache-dir pyodbc==4.0.30 && \
python3 -m pip install --no-cache-dir Cython==0.29.19 && \
python3 -m pip install --no-cache-dir PyInstaller==3.5 && \
python3 -m pip install --no-cache-dir selenium==3.141.0 && \
python3 -m pip install --no-cache-dir bs4==0.0.1
RUN python3 -m PyInstaller
The last run command fails with /usr/bin/python3: No module named PyInstaller, all other packages can be imported as expected.
The issue is also reproducible with this Dockerfile:
FROM debian:buster
RUN apt-get update && \
apt-get install -y \
python3 \
python3-pip
RUN python3 -m pip install --no-cache-dir PyInstaller==3.5
RUN python3 -m PyInstaller
What is the reason for this issue and what is the fix?
EDIT:
When I run the layer before the last RUN, I can see that no PyInstaller is installed, but I can run python3 -m pip install --no-cache-dir PyInstaller==3.5 and then it works without changing anything else.

Although I do not fully undestand the reason behind it, it seems like the --no-cache-dir option was causing the issue. The dockerfile below builds without an issue:
FROM debian:buster
RUN apt-get update && \
apt-get install -y \
python3 \
python3-pip
RUN python3 -m pip install PyInstaller==3.5
RUN python3 -m PyInstaller --help
Edit: This seems to be an issue outside of PyInstaller, but with the specific version of pip, see https://github.com/pyinstaller/pyinstaller/issues/6963 for details.

I'm not familiar with PyInstaller but in their requirements page they wrote:
If the pip setup fails to build a bootloader, or if you do not use pip
to install, you must compile a bootloader manually. The process is
described under Building the Bootloader.
Have you try that in your Dockerfile?
(And you're totally right, it should fail... )

ERROR: Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly

I was trying to install OpenCV4 in a docker on jetson nano. It has jetpack 4.4 s os. The docker was successfully created and Tensorflow is running but while installing OpenCV using pip it is showing CMake error.
root#5abf405fb92d:~# pip3 install opencv-python
Collecting opencv-python
Downloading opencv-python-4.4.0.42.tar.gz (88.9 MB)
|████████████████████████████████| 88.9 MB 2.5 kB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.18.4)
Building wheels for collected packages: opencv-python
Building wheel for opencv-python (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpqpzwrofy
cwd: /tmp/pip-install-93nxibky/opencv-python
Complete output (9 lines):
File "/tmp/pip-build-env-o_hualnr/overlay/lib/python3.6/site-packages/skbuild/setuptools_wrap.py", line 560, in setup
cmkr = cmaker.CMaker(cmake_executable)
File "/tmp/pip-build-env-o_hualnr/overlay/lib/python3.6/site-packages/skbuild/cmaker.py", line 95, in __init__
self.cmake_version = get_cmake_version(self.cmake_executable)
File "/tmp/pip-build-env-o_hualnr/overlay/lib/python3.6/site-packages/skbuild/cmaker.py", line 82, in get_cmake_version
"Problem with the CMake installation, aborting build. CMake executable is %s" % cmake_executable)
Traceback (most recent call last):
Problem with the CMake installation, aborting build. CMake executable is cmake
----------------------------------------
ERROR: Failed building wheel for opencv-python
Failed to build opencv-python
ERROR: Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly

I had the same problem and i did this,
pip install --upgrade pip setuptools wheel
then install opencv again,
pip install opencv-python
this worked for me

If after
pip install --upgrade pip setuptools wheel
you still have the same error,
You can try to specify the older version of OpenCV to install.
Ex.
pip3 install opencv-python==3.4.13.47

Yes .. Finally found a work around.
Follow this https://github.com/mdegans/nano_build_opencv and build from source and finally gets installed.
PS: It may take a bit long for building, for me it took 10 Hrs :P.
Happy Image-Processing..

I had a similar problem and what solved it for me was not to use python:3-alpine but python:3.8-slim. E.g.:
FROM python:3.8-slim
WORKDIR /usr/src/app
RUN apt update
RUN apt -y install build-essential libwrap0-dev libssl-dev libc-ares-dev uuid-dev xsltproc
RUN apt-get update -qq \
&& apt-get install --no-install-recommends --yes \
build-essential \
gcc \
python3-dev \
mosquitto \
mosquitto-clients
RUN pip3 install --upgrade pip setuptools wheel
RUN python3 -m pip install --no-cache-dir \
numpy scipy matplotlib scikit-build opencv-contrib-python-headless \
influxdb paho-mqtt configparser Pillow \
qrcode
worked finally for me.

please check your python specifications:
- opencv -> python[
version='
>=2.7,<2.8.0a0
>=3.5,<3.6.0a0
>=3.6,<3.7.0a0
>=3.7,<3.8.0a0']

I handled it by reinstalling python3 from scratch in my MacBook:
brew reinstall python#3.9
I also reinstalled numpy and matplotlib packages experimentally.
pip3 install numpy
pip3 install matplotlib
pip3 install opencv-contrib-python
The versions:
macOS Mojave 10.14.5
Python 3.9.7
OpenCV 4.5.3
OpenCV's version is 4.5.3 by this way:
import cv2
print(cv2.__version__)
But by "pip list", it shows "opencv-contrib-python 3.4.9.31".

if you are trying to install opencv in your raspberrypi 3B, Use following steps:
sudo raspi-config
advanced -- expand filesystem
reboot your pi
Open your raspi terminal and do following stuff:
use command: sudo apt-get update
use command: sudo apt-get upgrade
check your python version and upgarde it to latest one
install pip and upgrade pip
use command: mkdir project
use command: cd project
create virtual environment
activate virtual environment
install dependencies ,can get dependencies from https://singleboardbytes.com/647/install-opencv-raspberry-pi-4.htm
if error in installing libdhf5-dev
use command: sudo apt-get install python3-h5py and reinstall libdhf5-dev
use command: pip install scikit-build
use command: pip install cython
before installing opencv ,make sure you are in virtual environment or
activate environment
use command: pip install --no-cache-dir opencv-contrib-python==4.5.3.56
Remember to use mentioned version...
Thank You...

I came across similiar situation
I had error
Failed to build opencv-python ERROR: Could not build wheels for
opencv-python which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 19.2.3, however version 22.0.4 is
available. You should consider upgrading via the 'python -m pip
install --upgrade pip' command.
I ran in command prompt with admin privileges following
python -m pip install --upgrade pip
Now "pip install opencv-python" works

install opencv-python using Python3 in Ubuntu 15.04

I wish to install opencv-python via the command in Ubuntu 15.04 machine
pip3 install opencv-python
But as soon as I run this command I get the following error :
Downloading/unpacking opencv-python
Could not find any downloads that satisfy the requirement opencv-python
Cleaning up...
No distributions at all found for opencv-python
Storing debug log for failure in /home/Nadeem/.pip/pip.log
Any help would be much appreciated.
Thanks!!

You can install opencv from source.
Follow this link to do so.
Or you may need to upgrade your pip3 using the following command
pip3 install --upgrade pip
EDIT
For completeness(and in case the link is broken) I've listed here steps to compile and install OpenCV from source on Ubuntu(Tested on Ubuntu 14.04 LTS with python 3).
Step 1 Update the packages
sudo apt-get update
sudo apt-get upgrade
Step 2 Install dependencies
sudo apt-get install build-essential cmake git pkg-config # Developer tools required to compile opencv
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev # Libraries required to read various image format from disk
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev # Libraries required to read various video formats
sudo apt-get install libgtk2.0-dev # Required by opencv for GUI features
sudo apt-get install libatlas-base-dev gfortran # Packages used by opencv to optimize various functions.
pip3 install --upgrade pip
Step 3 setup virtual environment(using conda)
conda create -n opencv-exmaple-env python=3.6
source activate opencv-exmaple-env # Activate the envirnoment
Step 4 Install packages required to compile opencv
sudo apt-get install python3.6-dev # If the python version is not 3.6 then make changes to this command accordingly.
pip install numpy # This should be done after the environment in Step 3 is activated
Step 5: Build and install OpenCV 3.0 with Python 3.4+ bindings
5.1 Clone the opencv source
cd ~
mkdir opencv-source
cd opencv-source
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.3.0 # Branch you want to compile from
5.2 Clone Opencv Contrib rep
Contains exptra functionalities such as standard keypoint detectors and local invariant descriptors (such as SIFT, SURF, etc.)
cd ~
mkdir opencv-contrib
cd opencv-contrib
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib
git checkout 3.3.0 # The version you want to compile
5.3 Compile,build and install
cd ~/opencv-source/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv-contrib/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig
5.4 Link installed opencv object file to python site packages
ln -s /usr/local/lib/python3.6/site-packages/cv2.so /path-to-python-sitepackages-of-the-environment/cv2.so
6 Verify installation
import cv2
If the above code runs without error then opencv is installed successfully.

At first upgrade pip using sudo.
arsho:~/workspace $ sudo pip3 install --upgrade pip
Successfully installed pip
Now install opencv-python again using sudo command.
arsho:~/workspace $ sudo pip3 install opencv-python
Successfully installed numpy-1.13.1 opencv-python-3.3.0.10
Finally check the opencv-python version and location information using pip.
arsho:~/workspace $ pip3 show opencv-python
---
Name: opencv-python
Version: 3.3.0.10
Location: /usr/local/lib/python3.4/dist-packages
Requires: numpy
I have tested this using Ubuntu 14.04.5 LTS in https://c9.io/.