I'm reading this dockerfile for letsencrypt on Alpine:
https://github.com/CognitiveScale/lets-alpine/blob/master/Dockerfile
As I know, if I instlalled just pip with apk, or even apk-get on ubuntu, shouldn't the package manager also download any other needed libraries for pip to work? Why does this list of libs must be typed in the dockerfile?
RUN apk add --update \
python python-dev py-pip \
gcc musl-dev linux-headers \
augeas-dev openssl-dev libffi-dev ca-certificates dialog \
&& rm -rf /var/cache/apk/*
I'm asking this because, what if I want to create images based on alpine, how am I going to know all the needed libs?
These Alpine packages are not needed for pip itself, presumably they are needed to build the Python modules that you will install with pip later.
You need to read module descriptions to determine their dependencies. Alternatively, you can follow the "trial and error" route and add the required Alpine packages when some Python modules fail to build.
Related
I'm using nvidia/cudagl:11.4.2-base-ubuntu20.04, which does not include python. This is ok for me, since I need a very specific python version (3.9.9) anyhow. One way should be to compile python myself (e.g. according to https://askubuntu.com/a/890959). However, I thought maybe I could use a multi-stage build and just copy the required python executables from python:3.9.9-slim, so that I don't have to deal with the compilation and its remnants.
Is that a feasible idea? And what exactly would need to be copied over?
It's not really feasible to copy installed binaries from one image to another. If you know the application is only a static binary, or you know it has very controlled library dependencies, it could work, but Python installs things in many different directories and includes its own library tree. You'll be much better off using the distribution package manager to install the specific version of Python you need.
Since your base image is based on Ubuntu, there's an add-on Ubuntu repository that contains many different Python versions; also see Python 3.7 on Ubuntu 20.04. Translating that answer to Dockerfile format, you'd get something like:
FROM nvidia/cudagl:11.4.2-base-ubuntu20.04
...
# USER root (if you're not already root)
RUN apt-add-repository ppa:deadsnakes/ppa \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
python3.9
As of this writing that includes Python 3.9.10 rather than 3.9.9, but this shouldn't be a significant difference for you.
As David Maze pointed out, it is apparently not a good idea to copy over a Python install. Unfortunately, the popular deadsnakes repository that he suggested, does not supply patch versions of Python, i.e. only the latest patch of Python 3.9 (at the time of writing 3.9.10) and not previous versions like 3.9.9.
I found that pyenv is a pretty convenient way to install a specific Python version (originally posted here):
FROM ubuntu:16.04
ENV PYTHON_VERSION 2.7.10
#Set of all dependencies needed for pyenv to work on Ubuntu systems
RUN apt-get update \
&& apt-get install -y --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev mecab-ipadic-utf8 git
# Set-up necessary Env vars for PyEnv
ENV PYENV_ROOT /root/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
# Install pyenv
RUN set -ex \
&& curl https://pyenv.run | bash \
&& pyenv update \
&& pyenv install $PYTHON_VERSION \
&& pyenv global $PYTHON_VERSION \
&& pyenv rehash
# Optional : Checks Pyenv version on container start-up
ENTRYPOINT [ "pyenv","version" ]
I'm trygin to install a Django project on Docker, for testing purposes, but while i got the installation of the Pygame library, it launches an error(Many actually) saying that no Pygame version is available to install, from 2.0.1 to 1.9.2, and in some way, all of them show an error similar to /bin/sh: sdl2-config: not found. I tried searching, and it seems i need an SDL2 package for it to work. Alpine has a SDL2 package, tried to install it, but didn't work, searching a little more found that maybe i need a more specific one called python:pygame-sdl2, but the only thing i found respecting Alpine was this Google result:
It mentions that the package is absent on Alpine, but i can't find that same line if i make click on the result link.
Does anyone knows if it's possible install a Pygame project on an Alpine Image, or which would be the next recommendable image to mount a Django project?
This is my Dockerfile, just in case:
FROM python:3.8-alpine as base
ENV PYTHONUNBUFFERED 1
RUN apk add --no-cache --virtual .build-deps \
ca-certificates gcc postgresql-dev linux-headers musl-dev libffi-dev jpeg-dev zlib-dev geos sdl2\
&& rm -rf /var/cache/apk/*
RUN pip3 install --upgrade pip
COPY requirements.txt .
RUN pip install -r requirements.txt
FROM base as debug
RUN pip3 install debugpy
As #DavidMaze mentioned, the package sdl2-dev is the one required, there are 2 versions of packages, the normal ones and the -devs, and seems that python libraries require the latest ones, reason as why didn't work when i installed the sdl2. After this i only had to deal with a few other packages missing(The worst one was portmidi, since it was on the Edge branch), but after all that, the library has already been compiled.
I have a Dockerfile like the following, app code is omitted:
FROM python:3
# Binary dependencies
RUN apt update && apt install -y gfortran libopenblas-dev liblapack-dev
# Wanted Python packages
RUN python3 -m pip install mysqlclient numpy scipy pandas matplotlib
It works fine but produces an image of 1.75 GB in size (while code is about 50 MB). How can I reduce such huge volume??
I also tried to use Alpine Linux, like this:
FROM python:3-alpine
# Binary dependencies for numpy & scipy; though second one doesn't work anyway
RUN apk add --no-cache --virtual build-dependencies \
gfortran gcc g++ libstdc++ \
musl-dev lapack-dev freetype-dev python3-dev
# For mysqlclient
RUN apk --no-cache add mariadb-dev
# Wanted Python packages
RUN python3 -m pip install mysqlclient numpy scipy pandas matplotlib
But Alpine leads to many different strange errors. Error from the upper code:
File "scipy/odr/setup.py", line 28, in configuration
blas_info['define_macros'].extend(numpy_nodepr_api['define_macros'])
KeyError: 'define_macros'
So, how one can get minimal possible (or at least just smaller) image of Python 3 with mentioned packages?
There are several things you can do to make your Docker image smaller.
Use the python:3-slim Docker image as a base. The -slim images do not include packages needed for compiling software.
Pin the Python version, let's say to 3.8. Some packages do not have wheel files for python 3.9 yet, so you might have to compile them. It is good practice, in general, to use a more specific tag because the python:3-slim tag will point to different versions of python at different points in time.
You can also omit the installation of gfortran, libopenblas-dev, and liblapack-dev. Those packages are necessary for building numpy/scipy, but if you install the wheel files, which are pre-compiled, you do not need to compile any code.
Use --no-cache-dir in pip install to disable the cache. If you do not include this, then pip's cache counts toward the Docker image size.
There are no linux wheels for mysqlclient, so you will have to compile it. You can install build dependencies, install the package, then remove build dependencies in a single RUN instruction. Keep in mind that libmariadb3 is a runtime dependency of this package.
Here is a Dockerfile that implements the suggestions above. It makes a Docker image 354 MB large.
FROM python:3.8-slim
# Install mysqlclient (must be compiled).
RUN apt-get update -qq \
&& apt-get install --no-install-recommends --yes \
build-essential \
default-libmysqlclient-dev \
# Necessary for mysqlclient runtime. Do not remove.
libmariadb3 \
&& rm -rf /var/lib/apt/lists/* \
&& python3 -m pip install --no-cache-dir mysqlclient \
&& apt-get autoremove --purge --yes \
build-essential \
default-libmysqlclient-dev
# Install packages that do not require compilation.
RUN python3 -m pip install --no-cache-dir \
numpy scipy pandas matplotlib
Using alpine linux was a good idea, but alpine uses muslc instead of glibc, so it is not compatible with most pip wheels. The result is that you would have to compile numpy/scipy.
I'm running my python (django) app inside a container (using django cookiecutter project template). Everything was working well, but suddenly it failed to build with the following error:
Step 5/13 : RUN pip install -r /requirements/local.txt ---> Running in 4118795a1036 /usr/local/bin/python: can't open file /usr/local/bin/pip': [Errno 13] Permission denied ERROR: Service 'django' failed to build: The command '/bin/sh -c pip install -r /requirements/local.txt' returned a non-zero code: 2
I followed the official docker documentation about permissions when I first installed docker (https://docs.docker.com/install/linux/linux-postinstall/)
Image used: python:3.6-alpine
Docker version 19.03.1, build 74b1e89
docker-compose version 1.24.0, build 0aa59064
Ubuntu 18.04.3 LTS
I have already tried to prune the docker system (docker system prune) and tried to build the project with sudo (sudo docker-compose build) with no success.
Here is how my Dockerfile looks like:
FROM python:3.6-alpine
ENV PYTHONUNBUFFERED 1
RUN apk update \
# psycopg2 dependencies
&& apk add --virtual build-deps gcc python3-dev musl-dev \
&& apk add mariadb-dev \
&& apk add mysql-client \
# Pillow dependencies
&& apk add jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev \
# CFFI dependencies
&& apk add libffi-dev py-cffi \
# Translations dependencies
&& apk add gettext \
# https://docs.djangoproject.com/en/dev/ref/django-admin/#dbshell
&& apk add postgresql-client
# Requirements are installed here to ensure they will be cached.
COPY ./requirements /requirements
RUN pip install -r /requirements/local.txt
(...)
Has anyone ever faced this problem or knows any way to fix it? I've been struggling with it for hours and I've (literally) run out the ideas. Any help would be really appreciated.
Trying removing python3-dev from the packages you install, as well as py-cffi. The Python in the alpine packages is different than the Python in the Docker image, you don't want to install Alpine Python stuff if you can help it.
I would suggest avoiding Alpine. The motivation for Alpine is smaller images, but because binary wheels don't work with Alpine you need to compile packages... which means you need to install a compiler, which makes your images bigger. If you switch to Debian base image python:3.6-slim-buster (and I would argue you should for other reasons too: https://pythonspeed.com/articles/base-image-python-docker-images/) you will simplify your life.
I am trying to set up the compiled version of CPython, on Ubuntu 12.04, by following the python developer guide.
Even after installing the dependent packages lzma and sqlite3, build fails indicating that the dependent modules were not found.
Exact Error:
*Python build finished, but the necessary bits to build these modules were not found:
_lzma _sqlite3 _tkinter
To find the necessary bits, look in setup.py in detect_modules() for the module's name.*
I could not locate the package tkinter. Appreciate any help.
I was able to build Python 3.3 without modifying setup.py after installing the following packages on my Ubuntu Precise box.
build-essential
zlib1g-dev
libbz2-dev
libncurses5-dev
libreadline6-dev
libsqlite3-dev
libssl-dev
libgdbm-dev
liblzma-dev
tk8.5-dev
In general, see Python Developer's Guide for dependencies. There it says:
"If you want to build all optional modules, install the following packages and their dependencies":
sudo apt-get install build-essential gdb lcov pkg-config \
libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
lzma lzma-dev tk-dev uuid-dev zlib1g-dev
The lack of finding lzma and sqlite3 may be because your paths (LD_LIBRARY_PATH in particular) were incorrect. How did you install those two packages; did you use the package manager? If you installed manually, where did you install them? Also, did you install the development versions, if you used the package manager to install lzma and sqlite3? When installing from source, you'll need the development versions, so Python's source can find the necessary include files.
Further, you may have to edit setup.py to indicate where these packages can be found.
As for tkinter: this relies on tcl/tk, so check that you have the development versions of these packages installed if you're installing python/tkinter from source.
This works for me (Python 3.4, Ubuntu 13.04) meaning "make" completes cleanly:
sudo apt-get install build-essential python-dev libncurses*-dev \
liblzma-dev libgdbm-dev libsqlite3-dev \
libbz2-dev tk-dev
Install:
cd python3.4
make clean && ./configure && make && sudo make install
I used the instructions here:
python-on-debian-wheezy
But I also had to install tk-dev which wasn't listed there.
Struggled a bit with this on Ubuntu 20.04 in 2021 (in case anyone lands here looking for a newer set of instructions). Found this article that was very useful:
https://linoxide.com/ubuntu-how-to/install-python-3-9-on-ubuntu-20-04-lts/
On Ubuntu you can install the dependencies with apt so it's just a matter of knowing which. The build commands I used were the following:
# Update repo, very important on fresh server install
apt update
# Install dependencies
apt install gcc build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev
# Configure with optimizations
./configure --enable-optimizations
make -j 4 # 4 cores
make test # Shows you anything you missed
# https://docs.python.org/3/using/unix.html#building-python
make altinstall
I chose not to install sqlite or tkinter because I didn't need them, but the process is the same. Just include those dependencies found in #simp76's answer.
I just ran through this process on a fresh install of Ubuntu 20.04 on a DO droplet and it worked flawlessly.