We Keep Coding

How do I install a specific python version without pyenv within nvidia docker so that it interacts well with poetry?

I am trying to build a docker image that contains cuda, cudnn and python, each with specific versions that are templatable as a base for downstream users.
(In this example I have replace all the irrelevant templating with hard-coded versions, this is just FYI as a motivation).
Please note that the following questions are not duplicates:
How to install python in a docker image? does not involve poetry
Integrating Python Poetry with Docker Does not concern itself with installing dependencies
How do I integrate pyenv, poetry, and docker? This works for me already, I am looking for a different solution
I have achieved what I want using pyenv to install the specific python version within docker inside the nvidia image.
However, this solution is not optimal since the resulting image is about 1.5GB larger than what I think should be possible. Sidenote: I know that there are other ways to reduce the image size further that I have not done in this example. This is not the question here.
I have prepared a dummy pyproject.toml and poetry.lock to demonstrate the issue that I am currently facing:
pyproject.toml
[tool.poetry]
name = "example_project"
version = "1.0.0"
description = ""
authors = ["RunOrVeith"]
[tool.poetry.dependencies]
python = ">=3.8,<3.11"
scipy = "^1.9.3"
[build-system]
requires = ["poetry-core>=1.1.0"]
build-backend = "poetry.core.masonry.api"
Working Dockerfile.pyenv
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ARG PYTHON_VERSION=3.8
ENV DEBIAN_FRONTEND=noninteractive
# Set-up necessary Env vars for PyEnv
ENV PYENV_ROOT /root/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
ENV PATH="/root/.local/bin/:$PATH"
# Install essentials for pyenv https://github.com/pyenv/pyenv/wiki
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates \
curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev mecab-ipadic-utf8 git \
&& rm -rf /var/lib/apt/lists/*
# Install pyenv
RUN set -ex \
&& curl https://pyenv.run | bash \
&& pyenv update \
&& pyenv install $PYTHON_VERSION \
&& pyenv global $PYTHON_VERSION \
&& pyenv rehash \
&& pip install --upgrade pip
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example # The template that I want to provide ends here, this is just for demoing the issue
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
The version that doesn't work Dockerfile.plain
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHON_VERSION=3.8
ENV PATH="/root/.local/bin/:$PATH"
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys A4B469963BF863CC \
&& apt update \
&& apt install -y git curl \
&& apt install -y --no-install-recommends make build-essential
# Don't be confused, distutils-3.9 also installs python 3.8 https://github.com/deadsnakes/issues/issues/150
RUN apt install -y --no-install-recommends python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-distutils python${PYTHON_VERSION}-venv \
&& update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 10 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 10 \
&& apt-get install -y --no-install-recommends python3-pip python3-setuptools \
&& update-alternatives --install /usr/local/bin/pip pip /usr/bin/pip 10 \
&& update-alternatives --install /usr/local/bin/pip3 pip3 /usr/bin/pip 10 \
&& apt-get clean
WORKDIR /virtualenvs
RUN curl -sSL https://install.python-poetry.org | python${PYTHON_VERSION} - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
You can build this using
DOCKER_BUILDKIT=1 docker build -t github:example-plain --target example -f Dockerfile.plain .
and then run using
docker run -it github:example-plain bash
Here is the issue:
All the following commands are run from within the docker image.
According to poetry, everything is installed:
root#5e1ffb1f971c:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
However using regular pip, there is nothing, and imports also fail.
If I use poetry to import something, it also does not work.
root#5e1ffb1f971c:/app# pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
root#5e1ffb1f971c:/app# pip freeze
root#5e1ffb1f971c:/app# python -c "import scipy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
root#5e1ffb1f971c:/app# poetry run python -c "import scipy"
Skipping virtualenv creation, as specified in config file.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
What is also interesting is that if I upgrade pip with poetry it tells me it can't uninstall pip, I am assuming this is due to this ubuntu patch that tries to prevent me from breaking the system (even though I just install pip).
Afterwards, the poetry pip executable also points somewhere else.
root#5e1ffb1f971c:/app# poetry run pip install --upgrade pip
Skipping virtualenv creation, as specified in config file.
Collecting pip
Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-22.3.1
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
So how do I set this up so that I get a fresh python install of whichever version I configure, and it works with poetry? It is also required that the python and python3 aliases point to whatever poetry is using.
Reference with working version:
If I do the same commands with the working version using pyenv, it looks like this:
root#c0a9af7f05b4:/app# pip freeze
numpy==1.23.4
scipy==1.9.3
root#c0a9af7f05b4:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#c0a9af7f05b4:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)
root#c0a9af7f05b4:/app# pip --version
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)

Docker Windows image with numpy

I've built a Docker image on Windows using an official Docker Python image.
I've added to the image some other Python libraries/packages such as pip,
coverage, mypy, pylint, and numpy. The Dockerfile seems to be correct.
When I spin a container of the image, I can access the installed packages/libraries except for numpy.
root#b4d044180979:/usr/python# pip --version
pip 18.0 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
root#b4d044180979:/usr/python# which coverage
/usr/local/bin/coverage
root#b4d044180979:/usr/python# coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
root#b4d044180979:/usr/python# which numpy
root#b4d044180979:/usr/python# numpy --version
bash: numpy: command not found
root#b4d044180979:/usr/python#
Any idea why this is happening?
Why the numpy library is not recognized even-though the image building report indicates that it was successfully installed?
...
Collecting numpy
Downloading https://files.pythonhosted.org/packages/27/92/c01d3a6c58ceab0e6ec36ad3af41bc076014cc916afcb979ab4c9558f347/numpy-1.15.0-cp37-cp37m-manylinux1_x86_64.whl (13.8
MB)
Installing collected packages: numpy
Successfully installed numpy-1.15.0
...
Dockerfile
FROM python:3
RUN apt-get update && \
apt-get -y install vim
RUN pip install --upgrade pip && \
pip --version && \
pip install autopep8 && \
pip install coverage && \
pip install mypy && \
pip install pylint && \
pip install numpy && \
pip list
CMD bash

Cannot load CLoader with pyyaml

I'm working on a python project using pyyaml. I need to run it in a Docker container based on bitnami/minideb:jessie. Python version is 2.7.9.
The original code is using CLoader and I cannot change it currently.
Any reason CLoader fails to load but Loader is fine ?
>>> import yaml
>>> yaml.__version__
'3.12'
>>> from yaml import Loader
>>> from yaml import CLoader
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name CLoader
>>>
I cannot figure out what I'm missing here. Any idea ?
Running it from the Docker image python:2.7.9 does not raise any error then:
$ docker run -ti python:2.7.9 bash
#/ python
>>> from yaml import CLoader
>>> from yaml import Loader
>>>

By default, the setup.py script checks whether LibYAML is installed
and if so, builds and installs LibYAML bindings.
This is the minimum to get CLoader compiled and installed.
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y \
python3 python3-dev python3-pip gcc libyaml-dev
RUN pip3 install pyyaml
# verify
RUN python3 -c "import yaml; yaml.CLoader"

I ran into the same problem. You need to install the libyaml-dev package, then install libyaml and pyyaml from source. Here's the complete Dockerfile for minideb:jessie:
FROM bitnami/minideb:jessie
RUN apt-get update
RUN apt-get install -y \
automake \
autoconf \
build-essential \
git-core \
libtool \
libyaml-dev \
make \
python \
python-dev \
python-pip
RUN pip install --upgrade pip
RUN pip install Cython==0.29.10
RUN mkdir /libyaml
WORKDIR /libyaml
RUN git clone https://github.com/yaml/libyaml.git . && \
git checkout dist-0.2.2 && \
autoreconf -f -i && \
./configure && \
make && \
make install
RUN mkdir /pyyaml
WORKDIR /pyyaml
RUN git clone https://github.com/yaml/pyyaml.git . && \
git checkout 5.1.1 && \
python setup.py install
RUN python -c "import yaml; from yaml import CLoader; print 'Loaded CLoader!'"

A couple of additions to others' solutions:
If you want the install command to hard-fail if the libyaml C extension won't build (instead of silently falling back to a pure-Python only install), you can pass the --with-libyaml global option, eg: python setup.py --with-libyaml install.
If you're doing this with something that might ever need to be upgraded (eg implicitly via another package's requirement for a higher pyyaml version), it's better to use pip instead of directly calling setup.py, as that (currently) uses a pure distutils installation, which pip will fail to uninstall later. You'll see an error like "ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall."
Doing the required extension build with pip looks something like pip install --global-option='--with-libyaml' pyyaml.

I'm just copying the developer's answer from the issue linked above, but this happens because pyyaml only installs the libyaml bindings (CLoader & co.) if it finds the libyaml-dev package (that's the debian package, anyway) at install time. If it doesn't find it, it prints a warning and skips the libyaml bindings.
So, install libyaml-dev before installing pyyaml.

I tried all the step mentions, and the following steps fixed my issue.
Install
apt-get install -y gcc libyaml-dev
pip install --ignore-installed --global-option='--with-libyaml' pyyaml
Test
python -c "import yaml; yaml.CLoader"

install opencv-python using Python3 in Ubuntu 15.04

I wish to install opencv-python via the command in Ubuntu 15.04 machine
pip3 install opencv-python
But as soon as I run this command I get the following error :
Downloading/unpacking opencv-python
Could not find any downloads that satisfy the requirement opencv-python
Cleaning up...
No distributions at all found for opencv-python
Storing debug log for failure in /home/Nadeem/.pip/pip.log
Any help would be much appreciated.
Thanks!!

You can install opencv from source.
Follow this link to do so.
Or you may need to upgrade your pip3 using the following command
pip3 install --upgrade pip
EDIT
For completeness(and in case the link is broken) I've listed here steps to compile and install OpenCV from source on Ubuntu(Tested on Ubuntu 14.04 LTS with python 3).
Step 1 Update the packages
sudo apt-get update
sudo apt-get upgrade
Step 2 Install dependencies
sudo apt-get install build-essential cmake git pkg-config # Developer tools required to compile opencv
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev # Libraries required to read various image format from disk
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev # Libraries required to read various video formats
sudo apt-get install libgtk2.0-dev # Required by opencv for GUI features
sudo apt-get install libatlas-base-dev gfortran # Packages used by opencv to optimize various functions.
pip3 install --upgrade pip
Step 3 setup virtual environment(using conda)
conda create -n opencv-exmaple-env python=3.6
source activate opencv-exmaple-env # Activate the envirnoment
Step 4 Install packages required to compile opencv
sudo apt-get install python3.6-dev # If the python version is not 3.6 then make changes to this command accordingly.
pip install numpy # This should be done after the environment in Step 3 is activated
Step 5: Build and install OpenCV 3.0 with Python 3.4+ bindings
5.1 Clone the opencv source
cd ~
mkdir opencv-source
cd opencv-source
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.3.0 # Branch you want to compile from
5.2 Clone Opencv Contrib rep
Contains exptra functionalities such as standard keypoint detectors and local invariant descriptors (such as SIFT, SURF, etc.)
cd ~
mkdir opencv-contrib
cd opencv-contrib
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib
git checkout 3.3.0 # The version you want to compile
5.3 Compile,build and install
cd ~/opencv-source/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv-contrib/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig
5.4 Link installed opencv object file to python site packages
ln -s /usr/local/lib/python3.6/site-packages/cv2.so /path-to-python-sitepackages-of-the-environment/cv2.so
6 Verify installation
import cv2
If the above code runs without error then opencv is installed successfully.

At first upgrade pip using sudo.
arsho:~/workspace $ sudo pip3 install --upgrade pip
Successfully installed pip
Now install opencv-python again using sudo command.
arsho:~/workspace $ sudo pip3 install opencv-python
Successfully installed numpy-1.13.1 opencv-python-3.3.0.10
Finally check the opencv-python version and location information using pip.
arsho:~/workspace $ pip3 show opencv-python
---
Name: opencv-python
Version: 3.3.0.10
Location: /usr/local/lib/python3.4/dist-packages
Requires: numpy
I have tested this using Ubuntu 14.04.5 LTS in https://c9.io/.

How to compile mod_wsgi 4.5.3 with Python3.5.2 under centos7, Apache/2.4.6

I tried ./configure for mod_wsgi 4.5.3 like below:
./configure --with-python=/opt/Python352/bin
Where /opt/Python352 folder is installed with python3.5.2.
However in CentOS 7 the "configure" always builds the mod_wsgi with binary "python"(corresponding to python2.75) not with binary "python3" or "python3.5".
Requirement is to get the .so file, the latest for mod_wsgi 4.5.3, compiled using python3.5.2 and load this in Apache/2.4.6 under CentOS 7.
Thanks.

Here's an exact dump of what I use for CentOS 7.2. You can probably do without a lot of the pre-reqs, but I figured I'd include them since many are handy to have:
echo 'Python 3.5.2 is not installed, installing Python 3 pre-requisites...'
yum -y groupinstall development
echo 'Installing extra packages for Python...'
yum -y install zlib-devel openssl-devel sqlite-devel bzip2-devel python-devel openssl-devel libffi-devel openssl-perl libjpeg-turbo-devel zlib-devel giflib ncurses-devel gdbm-devel xz-devel tkinter readline-devel tk tk-devel
echo 'Installing Python 3.5.2...'
wget -q 'https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz'
tar -xzf 'Python-3.5.2.tgz'
cd ./Python-3.5.2
CXX=g++ ./configure --enable-shared
make
echo 'Moving to alternate location to keep system Python version intact...'
make altinstall
cd ..
rm Python-3.5.2.tgz
rm -rf ./Python-3.5.2
ln -fs /usr/local/bin/python3.5 /usr/bin/python3.5
echo "/usr/local/lib/python3.5" > /etc/ld.so.conf.d/python35.conf
echo "/usr/local/lib" >> /etc/ld.so.conf.d/python35.conf
ldconfig
echo 'Now, install mod_wsgi...'
wget -q "https://github.com/GrahamDumpleton/mod_wsgi/archive/4.4.21.tar.gz"
tar -xzf '4.4.21.tar.gz'
cd ./mod_wsgi-4.4.21
./configure --with-python=/usr/local/bin/python3.5
make
make install
I couldn't see an obvious problem with your question given the information, but figured a complete working example would help.