I have started learning Docker and I have developed a Python package (not published anywhere, it is just used internally) that installs and works fine locally (here I will call it mypackage). However, when trying to install it in a Docker container, Python in the container fails to recognise it even though during the build of the image no error was raised. The Dockerfile looks like this:
# install Ubuntu 20.04
FROM ubuntu:20.04
# update Ubuntu packages
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update
RUN apt upgrade -y
RUN apt install -y apt-utils \
build-essential \
curl \
mysql-server \
libmysqlclient-dev \
libffi-dev \
libssl-dev \
libxml2-dev \
libxslt1-dev \
unzip \
zlib1g-dev
# install Python 3.9
RUN apt-get install -y software-properties-common gcc && \
add-apt-repository -y ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3.9-dev python3.9-distutils python3-pip python3-apt python3.9-venv
# make symlink (overriding default Python3.8 installed with Ubuntu)
RUN rm /usr/bin/python3
RUN ln -s /usr/bin/python3.9 /usr/bin/python3
# copy package files and source code
RUN mkdir mypackage
COPY pyproject.toml setup.cfg setup.py requirements.txt ./mypackage/
COPY src mypackage/src/
# add path
ENV PACKAGE_PATH=/mypackage/
ENV PATH="$PACKAGE_PATH/:$PATH"
# install mypackage
RUN pip3 install -e ./mypackage
CMD ["python3.9", "main.py"]
So the above runs successfully, but if I run sudo docker run -it test_image bin/bash and run pip3 list, the package will not be there and a ModuleNotFoundError when running code depending on mypackage. Interestingly if I create a virtual environment by replacing this:
ENV PACKAGE_PATH=/mypackage/
ENV PATH="$PACKAGE_PATH/:$PATH"
by this:
ENV VIRTUAL_ENV=/opt/venv
RUN python3.9 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
it works. Ideally, I want to know why I need to create a virtual environment and how can I run local packages in a container without creating virtual environments.
Related
I have a couple of Python modules that I use inside my Docker container and they require a higher version of Python that what's being used. I install Python and install the modules using:
RUN apt-get update || : && apt-get install python3 -y
RUN apt-get install -y python3-pip
COPY requirements.txt /project
RUN pip3 install -r requirements.txt
Expecting I would be using the latest version of Python in my Docker container but when I go into it's shell and run python3 --version is comes as 3.4.2 which is incredibly old for my program. How do I make the default Python to be the latest I installed above without messing over the System-level python?
The image runtime I'm using for the Docker container is: node:9-slim
I don't think you can find a prebuilt python3.9 package on a debian 8 distribution as your environment is pretty old.
The only solution is you build the python3.9 out from source code in your base container. A full workable Dockerfile as next:
FROM node:9-slim
RUN apt update; \
apt install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev; \
wget https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz; \
tar -zxvf Python-3.9.7.tgz; \
cd Python-3.9.7; \
./configure --prefix=/usr/local/python3; \
make && make install; \
ln -sf /usr/local/python3/bin/python3.9 /usr/bin/python3; \
ln -sf /usr/local/python3/bin/pip3.9 /usr/bin/pip3
Verify it:
$ docker build -t myimage:1 .
$ docker run --rm -it myimage:1 python3 --version
Python 3.9.7
$ docker run --rm -it myimage:1 pip3 --version
pip 21.2.3 from /usr/local/python3/lib/python3.9/site-packages/pip (python 3.9)
Python Program does create folder and put some files over there. But when i try to run the program inside docker via CMD
It creates the folder and put files over there and upon completion, the folder somehow gets removed or doesnt show inside the docker image.
I have tried the following things:
Check Folder Exist after creating - It shows folder created over there.
Check inside the docker image using bash - It doesnt show the folder and contents.
The dockerfile is
FROM ubuntu:18.04
# Upgrade installed packages
RUN apt update
RUN apt upgrade -y
ENV TZ=Europe/London
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get install -y libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
WORKDIR /code
RUN apt-get -y install python3-pip
RUN apt-get -y install python3-venv
RUN apt -y install python3-setuptools libffi-dev python3-dev
RUN apt install -y curl
RUN apt install -y unzip
RUN apt-get install -y build-essential swig
WORKDIR /code
RUN python3 -m venv .env
RUN . .env/bin/activate && pip install --upgrade pip && curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | LC_ALL=C.UTF-8 xargs -n 1 -L 1 pip install
COPY requirements.txt requirements.txt
RUN . .env/bin/activate && pip install pyenchant && pip install -r requirements.txt
RUN apt install -y libgl1-mesa-glx
RUN apt-get install -y libglib2.0-0
RUN apt-get install -y libenchant1c2a
RUN mkdir embeddings
COPY . .
RUN curl -L http://nlp.stanford.edu/data/glove.6B.zip --output glove.zip
RUN unzip -o glove.zip -d embeddings/
RUN . .env/bin/activate && python nltk_install.py
CMD . .env/bin/activate && python main.py
Changes to filesystem are not stored in docker image. They exist in container created from an image but if you use 'docker run' command a new container is created.
I'm using python3.7-slim-buster docker image for my django project. Now I want to use Geo features of django. But it seems I have to install GDAL. So, I do RUN apt-get install gdal and it raises exception "E: Unable to locate package gdal-bin".
Here is my docker file:
FROM python:3.7-slim-buster
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# DB vars
ENV DB_USER_NAME ${DB_USER_NAME}
ENV DB_NAME ${DB_NAME}
ENV DB_HOST ${DB_HOST}
ENV DB_PORT ${DB_PORT}
ENV DB_PASSWORD ${DB_PASSWORD}
ENV DJANGO_SECRET_KEY ${DJANGO_SECRET_KEY}
RUN apt-get install -y gdal-bin python-gdal python3-gdal
RUN ["adduser", "${USER_NAME}", "--disabled-password", "--ingroup", "www-data", "--quiet"]
USER ${USER_NAME}
ADD ${PROJECT_NAME}/ /home/${USER_NAME}/${PROJECT_NAME}
WORKDIR /home/${USER_NAME}/${PROJECT_NAME}
ENV PATH="/home/${USER_NAME}/.local/bin:\${PATH}:/usr/local/python3/bin"
RUN pip install --user -r requirements.txt
CMD python manage.py runserver 0.0.0.0:9000
#CMD gunicorn ${PROJECT_NAME}.wsgi:application --bind 0.0.0.0:8000
EXPOSE 8000
you need to do the following:
RUN apt-get update
RUN apt-get install -y software-properties-common && apt-get update
RUN apt-get install -y python3.7-dev
RUN add-apt-repository ppa:ubuntugis/ppa && apt-get update
RUN apt-get install -y gdal-bin libgdal-dev
ARG CPLUS_INCLUDE_PATH=/usr/include/gdal
ARG C_INCLUDE_PATH=/usr/include/gdal
RUN pip install GDAL
If you can use other base image, here is one with gdal installed:
FROM osgeo/gdal:ubuntu-small-3.2.0
That's because your image doesn't have repository which contain gdal-bin package. So you have to add repository (you can see the guideline here) and install it:
RUN add-apt-repository ppa:ubuntugis/ppa && apt-get update && apt-get install -y gdal-bin python-gdal python3-gdal
I want to install Pyrender with headless rendering on Ubuntu 14.04. Specifically, I'd like have it installed within a Dockerfile. How can I do it so that OSMesa (and everything else) installs correctly?
Here are the lines from my Dockerfile that got things working (Ubuntu 14.04, python 3.6). It mostly involves following the installation guide, with some extra stuff to make sure deps get installed properly (llvm-6.0 is the main thing that's tricky).
If you're not trying to run in Docker, you can basically just run this stuff (in order) from the command line.
# Install pyrender
RUN pip3 install pyrender
# Copy and rename an apt lib file so that apt-add-repository
# works (cleaner way would be to symlink it but Dockerfiles don't seem
# to like symlinks). Might be due to some screwy python3.6/3.4 conflicts
# on my Docker image
RUN cp /usr/lib/python3/dist-packages/apt_pkg.cpython-34m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so
# Add new apt repositories and then apt-add some OSMesa deps
RUN add-apt-repository "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-6.0 main"
RUN add-apt-repository -y ppa:ubuntu-toolchain-r/test
RUN apt-get update && apt-get install --yes llvm-6.0 freeglut3 freeglut3-dev pkg-config
# Download OSMesa, then build and install it
RUN curl -o mesa-18.3.3.tar.gz ftp://ftp.freedesktop.org/pub/mesa/mesa-18.3.3.tar.gz
RUN tar xfv mesa-18.3.3.tar.gz
WORKDIR ./mesa-18.3.3
RUN ./configure --prefix=/usr/local \
--enable-opengl --disable-gles1 --disable-gles2 \
--disable-va --disable-xvmc --disable-vdpau \
--enable-shared-glapi \
--disable-texture-float \
--enable-gallium-llvm --enable-llvm-shared-libs \
--with-gallium-drivers=swrast,swr \
--disable-dri --with-dri-drivers= \
--disable-egl --with-egl-platforms= --disable-gbm \
--disable-glx \
--disable-osmesa --enable-gallium-osmesa \
ac_cv_path_LLVM_CONFIG=llvm-config-6.0
RUN make -j8
RUN make install
# Add some new environment variables so the OSMesa libs can be found
ENV MESA_HOME /usr/local
ENV LIBRARY_PATH $LIBRARY_PATH:$MESA_HOME/lib
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:$MESA_HOME/lib
ENV C_INCLUDE_PATH $C_INCLUDE_PATH:$MESA_HOME/include/
ENV CPLUS_INCLUDE_PATH $CPLUS_INCLUDE_PATH:$MESA_HOME/include/
# Get rid of the crappy old version of pyopengl, install a sweet new one
RUN pip3 uninstall -y pyopengl
RUN pip3 install git+https://github.com/mmatl/pyopengl.git
I installed Python 3.6 on Ubuntu 16.04 on Docker using the ppa:jonathonf/python-3.6 repository. Now I'd like to install xapian so I can use it with Python. I have not found any ready-made packages, so I am trying to build it from sources. I set PYTHON3 and PYTHON3_LIB parameters to point to Python 3.6. During the build process I get the following error:
ImportError: libxapian.so.30: cannot open shared object file: No such file or directory
I tried xapian versions 1.3.7 and 1.4.5 without luck.
How can I install xapian?
Here's a Dockerfile to reproduce my error:
FROM ubuntu:16.04
RUN apt-get update \
&& apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:jonathonf/python-3.6
RUN apt-get update \
&& apt-get install -y python3-pip docker.io python3.6 python3.6-dev software-properties-common \
python-software-properties build-essential wget unzip cmake python3-sphinx \
&& cd /usr/local/bin \
&& ln -s /usr/bin/python3.6 python
RUN python -m pip install --upgrade pip
# install xapian 1.4.5
RUN apt-get update && apt-get install -y curl uuid-dev zlib1g-dev
WORKDIR /root
RUN curl --silent --show-error --fail --next -O https://oligarchy.co.uk/xapian/1.4.5/xapian-core-1.4.5.tar.xz
RUN curl --silent --show-error --fail --next -O https://oligarchy.co.uk/xapian/1.4.5/xapian-bindings-1.4.5.tar.xz
RUN tar xvf xapian-core-1.4.5.tar.xz
RUN tar xvf xapian-bindings-1.4.5.tar.xz
WORKDIR /root/xapian-core-1.4.5
RUN ./configure && make && make install
WORKDIR /root/xapian-bindings-1.4.5
RUN ./configure PYTHON3=/usr/bin/python3.6 PYTHON3_LIB=/usr/lib/python3.6 --with-python3 && make && make install
RUN python -c "import xapian"
The problem is that the Xapian library (libxapian.so.30) is being installed into /usr/local/lib by default, but Ubuntu doesn't know that it's been put there yet. You can tell it by adding:
RUN ldconfig
after installing the core (so before you change WORKDIR to build the bindings).
There's some helpful information about ldconfig and library search paths on Ubuntu in the answers to this Unix Stackexchange question.