I am using docker tutorial (https://docs.docker.com/language/python/build-images/) to build a simple python app. Using freeze command I made requirements.txt file which consists a lot of packages.
When I want to build the docker image, I am getting this error:
Step 4/6 : RUN pip3 install -r requirements.txt ---> Running in
f92acd21d271
ERROR: Could not find a version that satisfies the requirement
apt-clone==0.2.1 (from versions: none)
ERROR: No matching distribution found for apt-clone==0.2.1
The command '/bin/sh -c pip3 install -r requirements.txt' returned a
non-zero code: 1
This is my dockerfile contents (same as what is mentioned in the tutorial):
# syntax=docker/dockerfile:1
FROM python:3.8-slim-buster
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD [ "python3", "-m", "flask","run","--host=0.0.0.0" ]
It's not related to apt-clone==0.2.1 package. Whatever I try to install in the docker image, it fails. I tried apt update and installing pip3 in the dockerfile too but didn't work.
What did I miss?
pip3 freeze outputs the package and its version installed in the current environment, no matter the package installed by pip or with other methods.
In fact, apt-clone==0.2.1 comes from debian package repo, not from pypi.org, see next:
$ pip3 freeze | grep apt-clone
$ apt-get install -y apt-clone
$ dpkg -l apt-clone
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii apt-clone 0.4.1 all Script to create state bundles
$ dpkg -L apt-clone
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/apt-clone
/usr/share/doc/apt-clone/copyright
/usr/share/doc/apt-clone/changelog.gz
/usr/share/man
/usr/share/man/man8
/usr/share/man/man8/apt-clone.8.gz
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/apt_clone.py
/usr/lib/python3/dist-packages/apt_clone-0.2.1.egg-info
/usr/bin
/usr/bin/apt-clone
$ pip3 freeze | grep apt-clone
apt-clone==0.2.1
You could see from above, the apt_clone.py & apt_clone-0.2.1.egg-info are installed by debian package apt-clone, the 0.4.1 is just the debian package version, while 0.2.1 is the python package version.
So, for apt-clone similar, you need to install them with apt although they are seen in pip3 freeze.
There is no package named apt-clone in the public PyPI repository, so pip3 obviously cannot find it.
If you actually have a Python package named like this, where did it come from?
If you created a package with this name locally, you need to install it too in your Docker image somehow.
If you are inside an organization, maybe you have a local PyPI with different packages than the public one, and then you need to configure pip3 to use it (try pip3 -i http://your.local.pypi/simple -r requirements.txt where obviously the URL needs to be something else, but we can't guess what).
If pip3 freeze says you have this package, it must have come from somewhere, but we don't know where. You have to figure that out, or supply more details to help us help you.
There is a Debian package named apt-clone but that obviously isn't something pip can install. Did you actually mean apt-get install -y apt-clone==0.2.1? (I can only find version 0.4.1 in Debian Buster, though.)
... #atline's answer explains what happened (you should probably accept that) but the simple fix is to manually populate requirements.txt with your actual requirements. pip3 freeze is a nice shorthand for that if you are in a virtual environment, but it's not a proper replacement for a manually curated requirements.txt or setup.py (or its modern replacements, currently moving from setup.cfg to pyproject.toml). If flask is the only package your app needs, simply put pip3 install flask and don't create a requirements.txt.
Since the beginning of this year our python dataflow jobs result in an error on worker startup:
Processing /var/opt/google/dataflow/setuptools-51.1.1.tar.gz
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
ERROR: No matching distribution found for wheel
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rvfx0hn9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the logs for full command output."
My requirements.txt is
ndjson==0.3.1
apache-beam[gcp]==2.26.0
dataclasses-json==0.5.2
google-cloud-storage==1.35.0
google-cloud-secret-manager==2.0.0
I am using the following Dockerfile to create a Flex-Template:
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}
# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
# https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev git && rm -rf /var/lib/apt/lists/*
ADD utils utils
COPY requirements.txt .
COPY main.py .
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
RUN pip install -U apache-beam[gcp]==2.24.0
RUN pip install -U -r ./requirements.txt
Has anyone a solution for the issue? I did try to set a specific setuptools version (51.1.1) as suggested in the following post leading to the same error message. Has anyone faced the same issue before
The issue was due a conflict in the dataclasses-json (The exact reason I couldn't find out). After removing it from the requirements.txt the image can successfully be buildt:
ndjson==0.3.1
google-cloud-storage==1.31.2
google-cloud-secret-manager==2.0.0
pycryptodome==3.9.8
I am trying to get Cython, cartopy and shapely running in a docker container so I can leverage a python library traffic. I am currently getting an error with Cython:
Collecting Cython==0.26 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/87/6c/53a9e636c9dbe7acd5c002422c1a7a48a367f3b4c0cf6490908f43398ca6/Cython-0.26-cp27-cp27mu-manylinux1_x86_64.whl (7.0MB)
Collecting geos (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/11/9b/a190f02fb92f465a7640b9ee7da732d91610415a1102f6e9bb08125a3fef/geos-0.2.2.tar.gz (365kB)
Collecting cartopy (from -r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/e5/92/fe8838fa8158931906dfc4f16c5c1436b3dd2daf83592645b179581403ad/Cartopy-0.17.0.tar.gz (8.9MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-Se89QB/cartopy/setup.py", line 42, in <module>
raise ImportError('Cython 0.15.1+ is required to install cartopy.')
ImportError: Cython 0.15.1+ is required to install cartopy.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-Se89QB/cartopy/
The command '/bin/sh -c pip install --no-cache-dir -r requirements.txt' returned a non-zero code: 1
Below is my setup:
Dockerfile:
FROM ubuntu:latest
WORKDIR /usr/src/app
#apt-get install -y build-essential -y python python-dev python-pip python-virtualenv libmysqlclient-dev curl&& \
RUN \
apt-get update && \
apt-get install -y build-essential -y python python-dev python-pip python-virtualenv libmysqlclient-dev curl&& \
rm -rf /var/lib/apt/lists/*
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
# Install cron
RUN apt-get update
RUN apt-get install cron
# Add crontab file in the cron directory
ADD crontab /etc/cron.d/simple-cron
# Add shell script and grant execution rights
ADD script.sh /script.sh
RUN chmod +x /script.sh
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/simple-cron
# Create the log file to be able to run tail
RUN touch /var/log/cron.log
# Run the command on container startup
CMD cron && tail -f /var/log/cron.log
requirements.txt
Cython==0.26
geos
cartopy
shapely
traffic
Try installing Cython first with pip, before using your requirements.txt.
cartopy has a lot of dependencies, and some -- especially Proj -- may not be resolvable using PIP or apt-get. numpy and cython may be resolved by installing them separately and just prior to installing cartopy (like u/dopplershift suggests) -- but Proj will never resolve, grr.
My solution was to use conda install, which solves the dependencies for you. Unfortunately Docker and Conda don't play well together, but you can kind of work around it using miniconda. Try this:
FROM ubuntu:latest
FROM python:3.8.5
RUN mkdir /app
ADD . /app
WORKDIR /app
# cartopy cannot be installed using PIP because the proj never gets resolved.
# The proj dependency never gets resolved because there are two Python packages
# called proj, and PIP always loads the wrong one. The conda install command,
# however, using the conda-forge channel, does know how to resolve the dependency
# issues, including packages like numpy.
#
# Here we install miniconda, just so we can use the conda install command
# for cartopy.
FROM continuumio/miniconda3
RUN conda install -c conda-forge cartopy
Since 2022-09-09 this is much easier, because Cartopy v0.21.0 does not depend on PROJ.
Solution
Dockerfile:
FROM python:3.11-slim-bullseye
RUN apt update && apt install -y git gcc build-essential python3-dev libgeos-dev
RUN python3 -m pip install --upgrade pip setuptools wheel
ADD requirements.txt .
RUN python3 -m pip install --no-cache-dir --compile -r requirements.txt
# add files and set cmd/entrpypoint down here
requirements.txt:
Cartopy==0.21.0
Test
docker build -t cartopy -f Dockerfile .
docker run -it cartopy pip freeze
Results in:
Cartopy==0.21.0
certifi==2022.12.7
contourpy==1.0.6
cycler==0.11.0
fonttools==4.38.0
kiwisolver==1.4.4
matplotlib==3.6.2
numpy==1.23.5
packaging==22.0
Pillow==9.3.0
pyparsing==3.0.9
pyproj==3.4.0
pyshp==2.3.1
python-dateutil==2.8.2
Shapely==1.8.5.post1
six==1.16.0
What I am trying to do?
Install all dependencies mentioned in requirements.txt using downloaded wheel files i.e. offline installation of packages in Docker
What I have done?
By following this thread I managed to download all my wheels into a wheelhouse folder using mkdir wheelhouse && pip download -r requirements.txt -d wheelhouse and I created a compressed tarball wheelhouse.tar.gz containing all my downloaded .whl files along with a requirements.txt
When I try to install the wheels locally (outside Docker) using pip install -r wheelhouse/requirements.txt --no-index --find-links wheelhouse, it works!
But when I run the same in Docker, it doesn't with the following error:
Processing ./wheelhouse/beautifulsoup4-4.8.2-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement blis==0.4.1 (from -r ./wheelhouse/requirements.txt (line 2)) (from versions: none)
ERROR: No matching distribution found for blis==0.4.1 (from -r ./wheelhouse/requirements.txt (line 2))
While actually, the wheel for blis 0.4.1 is present in my wheelhouse directory.
Can anyone please help me identify why it doesn't run on Docker and runs on local?
Dockerfile
FROM python:3
COPY . /app
WORKDIR /app
RUN tar -zxf ./wheelhouse.tar.gz
RUN pip install -r ./wheelhouse/requirements.txt --no-index --find-links ./wheelhouse
Screenshot of wheelhouse directory:
I am having a really hard time trying to install a stable data science package configuration in docker. This should be easier with such mainstream, relevant tools.
The following is the Dockerfile that used to work, with a bit of a hack, removing pandas from the package core and installing it separately, specifying pandas<0.21.0, because, allegedly, higher versions conflict with numpy.
FROM alpine:3.6
ENV PACKAGES="\
dumb-init \
musl \
libc6-compat \
linux-headers \
build-base \
bash \
git \
ca-certificates \
freetype \
libgfortran \
libgcc \
libstdc++ \
openblas \
tcl \
tk \
libssl1.0 \
"
ENV PYTHON_PACKAGES="\
numpy \
matplotlib \
scipy \
scikit-learn \
nltk \
"
RUN apk add --no-cache --virtual build-dependencies python3 \
&& apk add --virtual build-runtime \
build-base python3-dev openblas-dev freetype-dev pkgconfig gfortran \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& python3 -m ensurepip \
&& rm -r /usr/lib/python*/ensurepip \
&& pip3 install --upgrade pip setuptools \
&& ln -sf /usr/bin/python3 /usr/bin/python \
&& ln -sf pip3 /usr/bin/pip \
&& rm -r /root/.cache \
&& pip install --no-cache-dir $PYTHON_PACKAGES \
&& pip3 install 'pandas<0.21.0' \ #<---------- PANDAS
&& apk del build-runtime \
&& apk add --no-cache --virtual build-dependencies $PACKAGES \
&& rm -rf /var/cache/apk/*
# set working directory
WORKDIR /usr/src/app
# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt # other than data science packages go here
RUN pip install -r requirements.txt
# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
COPY . /usr/src/app
# run server
CMD ["/usr/src/app/entrypoint.sh"]
The configuration above used to work. What happens now is that build does go through, but pandas fails at import with the following error:
ImportError: Missing required dependencies ['numpy']
Since numpy 1.16.1 was installed, I don't know which numpy pandas is trying to find anymore...
Does anyone know how to obtain a stable solution for this?
NOTE: A solution consisting of a pull from a turnkey docker image for data science with at least the packages mentioned above, into Dockerfile above, would be also very welcomed.
EDIT 1:
If I move install of data packages into requirements.txt, as suggested in the comments, like so:
requirements.txt
(...)
numpy==1.16.1 # or numpy==1.16.0
scikit-learn==0.20.2
scipy==1.2.1
nltk==3.4
pandas==0.24.1 # or pandas== 0.23.4
matplotlib==3.0.2
(...)
and Dockerfile:
# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip install -r requirements.txt
It breaks again at pandas, complaining about numpy.
Collecting numpy==1.16.1 (from -r requirements.txt (line 61))
Downloading https://files.pythonhosted.org/packages/2b/26/07472b0de91851b6656cbc86e2f0d5d3a3128e7580f23295ef58b6862d6c/numpy-1.16.1.zip (5.1MB)
Collecting scikit-learn==0.20.2 (from -r requirements.txt (line 62))
Downloading https://files.pythonhosted.org/packages/49/0e/8312ac2d7f38537361b943c8cde4b16dadcc9389760bb855323b67bac091/scikit-learn-0.20.2.tar.gz (10.3MB)
Collecting scipy==1.2.1 (from -r requirements.txt (line 63))
Downloading https://files.pythonhosted.org/packages/a9/b4/5598a706697d1e2929eaf7fe68898ef4bea76e4950b9efbe1ef396b8813a/scipy-1.2.1.tar.gz (23.1MB)
Collecting nltk==3.4 (from -r requirements.txt (line 64))
Downloading https://files.pythonhosted.org/packages/6f/ed/9c755d357d33bc1931e157f537721efb5b88d2c583fe593cc09603076cc3/nltk-3.4.zip (1.4MB)
Collecting pandas==0.24.1 (from -r requirements.txt (line 65))
Downloading https://files.pythonhosted.org/packages/81/fd/b1f17f7dc914047cd1df9d6813b944ee446973baafe8106e4458bfb68884/pandas-0.24.1.tar.gz (11.8MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 359, in get_provider
module = sys.modules[moduleOrReq]
KeyError: 'numpy'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-_e5z6o6_/pandas/setup.py", line 732, in <module>
ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
File "/tmp/pip-install-_e5z6o6_/pandas/setup.py", line 475, in maybe_cythonize
numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1144, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 361, in get_provider
__import__(moduleOrReq)
ModuleNotFoundError: No module named 'numpy'
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-_e5z6o6_/pandas/
EDIT 2:
This seems like an open pandas issue. For more details please refer to:
pandas-dev github
"Unfortunately, this means that a requirements.txt file is insufficient for setting up a new environment with pandas installed (like in a docker container)".
**ImportError**:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
1. Check that you are using the Python you expect (you're using /usr/local/bin/python),
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy versions you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
EDIT 3
requirements.txt ---> https://pastebin.com/0icnx0iu
EDIT 4
As of 01/12/20, the accepted solution started not to work anymore.
Now, build breaks not at pandas, but at scipy but after numpy, while building scipy's wheel. This is the log:
----------------------------------------
ERROR: Failed building wheel for scipy
Running setup.py clean for scipy
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s6nahssd/scipy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-s6nahssd/scipy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
cwd: /tmp/pip-install-s6nahssd/scipy
Complete output (9 lines):
`setup.py clean` is not supported, use one of the following instead:
- `git clean -xdf` (cleans all files)
- `git clean -Xdf` (cleans all versioned files, doesn't touch
files that aren't checked into the git repo)
Add `--force` to your command to use it anyway if you must (unsupported).
----------------------------------------
ERROR: Failed cleaning build dir for scipy
Successfully built numpy
Failed to build scipy
ERROR: Could not build wheels for scipy which use PEP 517 and cannot be installed directly
From the error it seems that building process is using python3.6, while I use FROM alpine:3.7.
Full log here -> https://pastebin.com/Tw4ubxSA
And this is the current Dockerfile:
https://pastebin.com/3SftEufx
If you're not bound to Alpine 3.6, using Alpine 3.7 (or later) should work.
On Alpine 3.6, installing matplotlib failed for me with the following:
Collecting matplotlib
Downloading https://files.pythonhosted.org/packages/26/04/8b381d5b166508cc258632b225adbafec49bbe69aa9a4fa1f1b461428313/matplotlib-3.0.3.tar.gz (36.6MB)
Complete output from command python setup.py egg_info:
Download error on https://pypi.org/simple/numpy/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
Couldn't find index page for 'numpy' (maybe misspelled?)
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
No local packages or working download links found for numpy>=1.10.0
However, on Alpine 3.7, it worked. This may be due to a numpy versioning issue (see here), but I'm not able to tell for sure. Past that problem, packages were built and installed successfully - taking a good while, about 30 minutes (since Alpine's musl-libc is not compatible to Python's Wheels format, all packages installed with pip have to be built from source).
Note that one important change is needed: you should only remove the build-runtime virtual package (apk del build-runtime) after pip install. Also, if applicable, you could replace numpy 1.16.1 with 1.16.2, which is the shipped version (otherwise 1.16.2 will be uninstalled and 1.16.1 built from source, further increasing the build time) - I haven't tried this, though.
For reference, here's my slightly modified Dockerfile and docker build output.
Note:
Usually Alpine is chosen as the base for minimizing the image size (Alpine is also otherwise very slick, but has compatibility issues with mainland Linux apps due to glibc/musl). Having to build Python packages from source kind of beats that purpose, since you get a very bloated image - 900MB before any cleanup, which also takes ages to build. The image could be greatly compacted by removing all intermediate compilation artifacts, build dependencies etc., but still.
If you can't get the Python package versions you need to work on Alpine, without having to build them from source, I would suggest trying other small and more compatible base images such as debian-slim, or even ubuntu.
Edit:
Following "Edit 3" with added requirements, here are updated Dockerfile and Docker build output.
The following packages were added for satisfying build dependencies:
postgresql-dev libffi-dev libressl-dev libxml2 libxml2-dev libxslt libxslt-dev libjpeg-turbo-dev zlib-dev
For packages that failed to build due to specific headers, I used Alpine's package contents search to locate the missing package.
Specifically for cffi, the ffi.h header was missing, which needs the libffi-dev package: https://pkgs.alpinelinux.org/contents?file=ffi.h&path=&name=&branch=v3.7.
Alternatively, when a package build failure is not very clear, the installation instructions of the specific package could be referred to, for example, Pillow.
The new image size, before any compaction, is 1.04GB. For cutting it down a bit, you could remove the Python and pip caches:
RUN apk del build-runtime && \
find -type d -name __pycache__ -prune -exec rm -rf {} \; && \
rm -rf ~/.cache/pip
This will bring image size down to 661MB, when using docker build --squash.
Try adding this to your requirements.txt file:
numpy==1.16.0
pandas==0.23.4
I've been facing the same error since yesterday and this change solved it for me.
An older Q&A at Why does it take ages to install Pandas on Alpine Linux relates.
If your aim to get a stable solution without knowing the nuts and bolts, for python 3 you can just build off the following (copy & paste of my answer from https://stackoverflow.com/a/50443531/1021819)
FROM python:3.7-alpine
RUN echo "#testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk add --update --no-cache py3-numpy py3-pandas#testing
If your goal is to understand how to achieve a stable build, the discussion there and related images might help too...
FROM python:3.8-alpine
RUN apk --update add gcc build-base freetype-dev libpng-dev openblas-dev
RUN pip install --no-cache-dir matplotlib pandas
This may not be completely relevant, since this the first answer that pops up when searching for numpy/pandas installation failed in Alpine, I am adding this answer.
The following fix worked for me(But it takes longer to install pandas/numpy)
apk update
apk --no-cache add curl gcc g++
ln -s /usr/include/locale.h /usr/include/xlocale.h
Now try installing pandas/numpy