I am currently building an image from alpine:3.7.
There are two packages that I am having problems with:
pendulum (specifically python-dateutils package)
service_identity (specifically attrs package)
The error that I receive it is:
Could not find a version that satisfies the requirement setuptools (from versions: ) No matching distribution found for setuptools
Note: all packages are pre-cached on a directory using pip download.
The dockerfile looks as follows:
RUN apk add --no-cache --virtual .build-deps <dev packages>
&& apk add --no-cache --update python3
&& pip3 install --upgrade pip setuptools
RUN pip3 install -f ./python-packages --no-index -r requirements.txt ./python-packages/pkgs
....
dev-packages such as libffi-dev, libressl-dev, etc.
I'm not sure about the full list of dev-packages to build in the question, but it should be the following: g++ (GNU C++ standard library and compiler), python3-dev (python3 development files), libffi-dev (libffi development files) and openssl-dev (Toolkit for SSL v2/v3 and TLS v1 development files).
The Dockerfile is:
FROM alpine:3.7
RUN apk add --no-cache --virtual .build-deps g++ python3-dev libffi-dev openssl-dev && \
apk add --no-cache --update python3 && \
pip3 install --upgrade pip setuptools
RUN pip3 install pendulum service_identity
Apparently when upgrading pip with:
pip3 install --upgrade pip setuptools
I removed pip upgrading and installation worked. Now, I have been researching the correct way to upgrade pip on alpine and found a Dockerfile in a github repo that does this check:
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
Which makes sure that pip3 is being referred when calling just pip command by doing a symbolic link on python and system binaries' directories.
Related
I am trying to build a docker image that contains cuda, cudnn and python, each with specific versions that are templatable as a base for downstream users.
(In this example I have replace all the irrelevant templating with hard-coded versions, this is just FYI as a motivation).
Please note that the following questions are not duplicates:
How to install python in a docker image? does not involve poetry
Integrating Python Poetry with Docker Does not concern itself with installing dependencies
How do I integrate pyenv, poetry, and docker? This works for me already, I am looking for a different solution
I have achieved what I want using pyenv to install the specific python version within docker inside the nvidia image.
However, this solution is not optimal since the resulting image is about 1.5GB larger than what I think should be possible. Sidenote: I know that there are other ways to reduce the image size further that I have not done in this example. This is not the question here.
I have prepared a dummy pyproject.toml and poetry.lock to demonstrate the issue that I am currently facing:
pyproject.toml
[tool.poetry]
name = "example_project"
version = "1.0.0"
description = ""
authors = ["RunOrVeith"]
[tool.poetry.dependencies]
python = ">=3.8,<3.11"
scipy = "^1.9.3"
[build-system]
requires = ["poetry-core>=1.1.0"]
build-backend = "poetry.core.masonry.api"
Working Dockerfile.pyenv
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ARG PYTHON_VERSION=3.8
ENV DEBIAN_FRONTEND=noninteractive
# Set-up necessary Env vars for PyEnv
ENV PYENV_ROOT /root/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
ENV PATH="/root/.local/bin/:$PATH"
# Install essentials for pyenv https://github.com/pyenv/pyenv/wiki
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates \
curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev mecab-ipadic-utf8 git \
&& rm -rf /var/lib/apt/lists/*
# Install pyenv
RUN set -ex \
&& curl https://pyenv.run | bash \
&& pyenv update \
&& pyenv install $PYTHON_VERSION \
&& pyenv global $PYTHON_VERSION \
&& pyenv rehash \
&& pip install --upgrade pip
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example # The template that I want to provide ends here, this is just for demoing the issue
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
The version that doesn't work Dockerfile.plain
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHON_VERSION=3.8
ENV PATH="/root/.local/bin/:$PATH"
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys A4B469963BF863CC \
&& apt update \
&& apt install -y git curl \
&& apt install -y --no-install-recommends make build-essential
# Don't be confused, distutils-3.9 also installs python 3.8 https://github.com/deadsnakes/issues/issues/150
RUN apt install -y --no-install-recommends python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-distutils python${PYTHON_VERSION}-venv \
&& update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 10 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 10 \
&& apt-get install -y --no-install-recommends python3-pip python3-setuptools \
&& update-alternatives --install /usr/local/bin/pip pip /usr/bin/pip 10 \
&& update-alternatives --install /usr/local/bin/pip3 pip3 /usr/bin/pip 10 \
&& apt-get clean
WORKDIR /virtualenvs
RUN curl -sSL https://install.python-poetry.org | python${PYTHON_VERSION} - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
You can build this using
DOCKER_BUILDKIT=1 docker build -t github:example-plain --target example -f Dockerfile.plain .
and then run using
docker run -it github:example-plain bash
Here is the issue:
All the following commands are run from within the docker image.
According to poetry, everything is installed:
root#5e1ffb1f971c:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
However using regular pip, there is nothing, and imports also fail.
If I use poetry to import something, it also does not work.
root#5e1ffb1f971c:/app# pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
root#5e1ffb1f971c:/app# pip freeze
root#5e1ffb1f971c:/app# python -c "import scipy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
root#5e1ffb1f971c:/app# poetry run python -c "import scipy"
Skipping virtualenv creation, as specified in config file.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
What is also interesting is that if I upgrade pip with poetry it tells me it can't uninstall pip, I am assuming this is due to this ubuntu patch that tries to prevent me from breaking the system (even though I just install pip).
Afterwards, the poetry pip executable also points somewhere else.
root#5e1ffb1f971c:/app# poetry run pip install --upgrade pip
Skipping virtualenv creation, as specified in config file.
Collecting pip
Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-22.3.1
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
So how do I set this up so that I get a fresh python install of whichever version I configure, and it works with poetry? It is also required that the python and python3 aliases point to whatever poetry is using.
Reference with working version:
If I do the same commands with the working version using pyenv, it looks like this:
root#c0a9af7f05b4:/app# pip freeze
numpy==1.23.4
scipy==1.9.3
root#c0a9af7f05b4:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#c0a9af7f05b4:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)
root#c0a9af7f05b4:/app# pip --version
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)
In the below docker file:
FROM alpine:latest
ENV HOME /home/samcli
ENV PATH $HOME/.local/bin:$PATH
RUN mkdir /root/bin /aws; \
apk add --no-cache groff less bash python jq curl py-pip tzdata
RUN ln -fs /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN apk add --no-cache --virtual .build-deps gcc python2-dev python3-dev linux-headers musl-dev && \
pip install --upgrade pip; \
adduser samcli -Du 5566; \
chown -R samcli $HOME;
USER samcli
WORKDIR $HOME
RUN pip install --user --upgrade awscli aws-sam-cli;
USER root
RUN apk del .build-deps; \
rm -rf /var/cache/apk/*
Layer(RUN pip install --user --upgrade awscli aws-sam-cli;) is installing with python 2.7, despite image has python3.7 installed.
I see below deprecation error when installing python package:
Step 9/11 : RUN pip install --user --upgrade awscli aws-sam-cli;
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/
How to install aws package with python3.7?
because below command is using python2
pip install --user --upgrade awscli aws-sam-cli;
I would recommend using python offical image based on alpine so you will do not need to maintain and install the python version. Below base image is base on alpine 3.9 and python version is 3.7
FROM python:3.7-alpine3.9
ENV HOME /home/samcli
ENV PATH $HOME/.local/bin:$PATH
RUN ln -fs /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN apk add --no-cache --virtual .build-deps python2-dev python3-dev gcc linux-headers musl-dev && \
adduser samcli -Du 5566; \
chown -R samcli $HOME;
RUN apk add --no-cache groff less bash jq curl py-pip tzdata
USER samcli
WORKDIR $HOME
RUN pip install --user --upgrade awscli aws-sam-cli;
USER root
RUN apk del .build-deps; \
rm -rf /var/cache/apk/*
This Dockerfile works for me.
FROM alpine:latest
RUN apk update && apk upgrade
RUN apk --no-cache add python3 py3-pip gcc musl-dev python3-dev
RUN pip install aws-sam-cli awscli
...
I installed Python 3.6 on Ubuntu 16.04 on Docker using the ppa:jonathonf/python-3.6 repository. Now I'd like to install xapian so I can use it with Python. I have not found any ready-made packages, so I am trying to build it from sources. I set PYTHON3 and PYTHON3_LIB parameters to point to Python 3.6. During the build process I get the following error:
ImportError: libxapian.so.30: cannot open shared object file: No such file or directory
I tried xapian versions 1.3.7 and 1.4.5 without luck.
How can I install xapian?
Here's a Dockerfile to reproduce my error:
FROM ubuntu:16.04
RUN apt-get update \
&& apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:jonathonf/python-3.6
RUN apt-get update \
&& apt-get install -y python3-pip docker.io python3.6 python3.6-dev software-properties-common \
python-software-properties build-essential wget unzip cmake python3-sphinx \
&& cd /usr/local/bin \
&& ln -s /usr/bin/python3.6 python
RUN python -m pip install --upgrade pip
# install xapian 1.4.5
RUN apt-get update && apt-get install -y curl uuid-dev zlib1g-dev
WORKDIR /root
RUN curl --silent --show-error --fail --next -O https://oligarchy.co.uk/xapian/1.4.5/xapian-core-1.4.5.tar.xz
RUN curl --silent --show-error --fail --next -O https://oligarchy.co.uk/xapian/1.4.5/xapian-bindings-1.4.5.tar.xz
RUN tar xvf xapian-core-1.4.5.tar.xz
RUN tar xvf xapian-bindings-1.4.5.tar.xz
WORKDIR /root/xapian-core-1.4.5
RUN ./configure && make && make install
WORKDIR /root/xapian-bindings-1.4.5
RUN ./configure PYTHON3=/usr/bin/python3.6 PYTHON3_LIB=/usr/lib/python3.6 --with-python3 && make && make install
RUN python -c "import xapian"
The problem is that the Xapian library (libxapian.so.30) is being installed into /usr/local/lib by default, but Ubuntu doesn't know that it's been put there yet. You can tell it by adding:
RUN ldconfig
after installing the core (so before you change WORKDIR to build the bindings).
There's some helpful information about ldconfig and library search paths on Ubuntu in the answers to this Unix Stackexchange question.
I am currently building a docker image with the latest version of Django but during the build I get this:
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-9LueOB/Django/
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c INSTALL_PKGS="python-pip oracle-instantclient-tnsnames.ora afs_tools_standalone nss_wrapper openssh-clients cx_Oracle libmemcached-devel python-ldap mod_wsgi httpd MySQL-python wassh-ssm-cern wassh" && INSTALL_PKGS_BUILD="gcc python-devel zlib-devel" && yum install -y centos-release-scl && yum update -y && yum install -y --setopt=tsflags=nodocs --enablerepo=centosplus $INSTALL_PKGS && yum install -y $INSTALL_PKGS_BUILD && rpm -e --nodeps centos-logos && pip install --upgrade && pip install -r requirements.txt && rm requirements.txt && yum remove -y $INSTALL_PKGS_BUILD && yum clean all -y && rm -rf /var/cache/yum && mkdir /var/django/ && mkdir -p /var/django/home/ && mkdir -p /usr/local/bin/ && mkdir -p /run/httpd/ && chown -R root:root /run/httpd && chmod a+rw /run/httpd' returned a non-zero code: 1
On my requirements.txt I am using this:
Django==2.0
djangorestframework==3.2.1
pyapi-gitlab==7.8.4
pylibmc==1.5.0
django-logtail
python-gitlab
Any idea of why this is happening?
I believe it's because Django 2.0 doesn't support Python 2.x anymore, try to replace "python" by "python3" in your command.
EDIT: or run "pip3" instead of "pip" when you install Django>=2.0
as mentioned by Piinthesky here you'll find the relevant Django documentation
I'm trying to install awscli using pip (as per Amazon's recommendations) in a custom Docker image that comes FROM library/node:6.11.2. Here's a repro:
FROM library/node:6.11.2
RUN apt-get update && \
apt-get install -y \
python \
python-pip \
python-setuptools \
groff \
less \
&& pip --no-cache-dir install --upgrade awscli \
&& apt-get clean
CMD ["/bin/bash"]
However, with the above I'm met with:
no such option: --no-cache-dir
Presumably because I've got incorrect versions of Python and/or Pip?
I'm installing Python, Pip, and awscli in a similar way with FROM maven:3.5.0-jdk-8 and there it works just fine. I'm unsure what the relevant differences between the two images are.
Removing said option from my Dockerfile doesn't do me much good either, because then I'm met with a big pile of different errors, an excerpt here:
Installing collected packages: awscli, PyYAML, docutils, rsa, colorama, botocore, s3transfer, pyasn1, jmespath, python-dateutil, futures, six
Running setup.py install for PyYAML
checking if libyaml is compilable
### ABBREVIATED ###
ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
### ABBREVIATED ###
Bottom line: how do you properly install awscli in library/node:6.x based images?
Adding python-dev as per this other answer works, but throws an alarming number of compiler warnings (errors?), so I went with a variation of #SergeyKoralev's answer, which needed some tweaking before it worked.
Here's the changes I needed to make this work:
Change to python3 and pip3 everywhere.
Add a statement to upgrade pip itself.
Separate the awscli install in a separate RUN command.
Here's a full repro that does seem to work:
FROM library/node:6.11.2
RUN apt-get update && \
apt-get install -y \
python3 \
python3-pip \
python3-setuptools \
groff \
less \
&& pip3 install --upgrade pip \
&& apt-get clean
RUN pip3 --no-cache-dir install --upgrade awscli
CMD ["/bin/bash"]
You can probably also keep the aws install in the same RUN layer if you add a shell command before the install that refreshes things after upgrading pip. Not sure how though.
All the answers are about aws-cli version 1, If you want version 2 try the below
FROM node:lts-stretch-slim
RUN apt-get update && \
apt-get install -y \
unzip \
curl \
&& apt-get clean \
&& curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install \
&& rm -rf \
awscliv2.zip \
&& apt-get -y purge curl \
&& apt-get -y purge unzip
CMD ["/bin/bash"]
As you have correctly stated, pip installing on the docker image you are using is an older one not supporting --no-cache-dir. You can try updating that or you can also fix the second problem which is about missing python source headers. This can be fixed by installing python-dev package. Just add that to the list of packages installed in the Dockerfile:
FROM library/node:6.11.2
RUN apt-get update && \
apt-get install -y \
python \
python-dev \
python-pip \
python-setuptools \
groff \
less \
&& pip install --upgrade awscli \
&& apt-get clean
CMD ["/bin/bash"]
You can then run aws which should be on your path.
Your image is based on Debian Jessie, so you are installing Python 2.7. Try using Python 3.x:
apt-get install -y python3-pip
pip3 install awscli
Install AWS CLI in docker container using below command:
apt upgrade -y;apt update;apt install python3 python3-pip python3-setuptools -y; python3 -m pip --no-cache-dir install --upgrade awscli
To check the assumed role or AWS identity run below command:
aws sts get-caller-identity