Python Program does create folder and put some files over there. But when i try to run the program inside docker via CMD
It creates the folder and put files over there and upon completion, the folder somehow gets removed or doesnt show inside the docker image.
I have tried the following things:
Check Folder Exist after creating - It shows folder created over there.
Check inside the docker image using bash - It doesnt show the folder and contents.
The dockerfile is
FROM ubuntu:18.04
# Upgrade installed packages
RUN apt update
RUN apt upgrade -y
ENV TZ=Europe/London
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get install -y libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
WORKDIR /code
RUN apt-get -y install python3-pip
RUN apt-get -y install python3-venv
RUN apt -y install python3-setuptools libffi-dev python3-dev
RUN apt install -y curl
RUN apt install -y unzip
RUN apt-get install -y build-essential swig
WORKDIR /code
RUN python3 -m venv .env
RUN . .env/bin/activate && pip install --upgrade pip && curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | LC_ALL=C.UTF-8 xargs -n 1 -L 1 pip install
COPY requirements.txt requirements.txt
RUN . .env/bin/activate && pip install pyenchant && pip install -r requirements.txt
RUN apt install -y libgl1-mesa-glx
RUN apt-get install -y libglib2.0-0
RUN apt-get install -y libenchant1c2a
RUN mkdir embeddings
COPY . .
RUN curl -L http://nlp.stanford.edu/data/glove.6B.zip --output glove.zip
RUN unzip -o glove.zip -d embeddings/
RUN . .env/bin/activate && python nltk_install.py
CMD . .env/bin/activate && python main.py
Changes to filesystem are not stored in docker image. They exist in container created from an image but if you use 'docker run' command a new container is created.
Related
I have started learning Docker and I have developed a Python package (not published anywhere, it is just used internally) that installs and works fine locally (here I will call it mypackage). However, when trying to install it in a Docker container, Python in the container fails to recognise it even though during the build of the image no error was raised. The Dockerfile looks like this:
# install Ubuntu 20.04
FROM ubuntu:20.04
# update Ubuntu packages
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update
RUN apt upgrade -y
RUN apt install -y apt-utils \
build-essential \
curl \
mysql-server \
libmysqlclient-dev \
libffi-dev \
libssl-dev \
libxml2-dev \
libxslt1-dev \
unzip \
zlib1g-dev
# install Python 3.9
RUN apt-get install -y software-properties-common gcc && \
add-apt-repository -y ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3.9-dev python3.9-distutils python3-pip python3-apt python3.9-venv
# make symlink (overriding default Python3.8 installed with Ubuntu)
RUN rm /usr/bin/python3
RUN ln -s /usr/bin/python3.9 /usr/bin/python3
# copy package files and source code
RUN mkdir mypackage
COPY pyproject.toml setup.cfg setup.py requirements.txt ./mypackage/
COPY src mypackage/src/
# add path
ENV PACKAGE_PATH=/mypackage/
ENV PATH="$PACKAGE_PATH/:$PATH"
# install mypackage
RUN pip3 install -e ./mypackage
CMD ["python3.9", "main.py"]
So the above runs successfully, but if I run sudo docker run -it test_image bin/bash and run pip3 list, the package will not be there and a ModuleNotFoundError when running code depending on mypackage. Interestingly if I create a virtual environment by replacing this:
ENV PACKAGE_PATH=/mypackage/
ENV PATH="$PACKAGE_PATH/:$PATH"
by this:
ENV VIRTUAL_ENV=/opt/venv
RUN python3.9 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
it works. Ideally, I want to know why I need to create a virtual environment and how can I run local packages in a container without creating virtual environments.
I have issue with my dockerfile:
I tried to add latest python version (3.9 or 3.8) but when
My dockerfile:
FROM jenkins/jnlp-slave:latest
USER root
ENV AWS_ACCESS_KEY_ID ****************
ENV AWS_SECRET_ACCESS_KEY ***************
ENV AWS_DEFAULT_REGION ***********
ENV AWS_DEFAULT_OUTPUT ************
RUN apt-get update
RUN apt-get install vim nano expect sshpass curl apt-transport-https -y
**RUN apt install python3.9 -y**
RUN apt-get update \
&& apt-get install -y python3-pip python3-dev \
&& cd /usr/local/bin \
&& ln -s /usr/bin/python3 python \
&& pip3 install --upgrade pip
RUN apt-get install -y libcurl4-gnutls-dev librtmp-dev
COPY pythonRequirements.txt pythonRequirements.txt
RUN mkdir ~/.aws/ && touch ~/.aws/config
RUN apt-get update
RUN apt-get install -y tcpdump
RUN apt-get install jq -y
RUN apt-get install -y pkg-config
RUN apt install libcairo2-dev -y
RUN pip3 install -r pythonRequirements.txt --no-cache-dir
RUN apt-get install -y libsm6 libxext6 libxrender-dev python3-tk
RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
RUN echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | tee -a /etc/apt/sources.list.d/kubernetes.list
RUN apt-get update
RUN apt-get install -y kubectl
RUN mkdir ~/.kube
RUN apt-get install xvfb -y
RUN pip3 install psycopg2-binary
RUN aws s3 cp s3://*************************
RUN aws s3 cp s3://*************************
RUN aws s3 cp s3://*************************
RUN aws s3 cp s3://*************************
When I tried it and then I checked the python version I got python 3.5!!!
My docker file must include FROM jenkins/jnlp-slave:latest
Following is my docker file
FROM ubuntu:18.04
# Upgrade installed packages
RUN apt update
RUN apt upgrade -y
ENV TZ=Europe/London
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get install -y libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
WORKDIR /code
RUN apt-get -y install python3-pip
RUN apt-get -y install python3-venv
RUN apt -y install python3-setuptools libffi-dev python3-dev
RUN apt install -y curl
RUN apt install -y unzip
RUN apt-get install -y build-essential swig
WORKDIR /code
RUN python3 -m venv .env
RUN . .env/bin/activate && pip install --upgrade pip && curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | LC_ALL=C.UTF-8 xargs -n 1 -L 1 pip install
COPY requirements.txt requirements.txt
RUN . .env/bin/activate && pip install pyenchant && pip install -r requirements.txt
RUN apt install -y libgl1-mesa-glx
RUN apt-get install -y libglib2.0-0
RUN apt-get install -y libenchant1c2a
RUN mkdir embeddings
COPY . .
RUN curl -L http://nlp.stanford.edu/data/glove.6B.zip --output glove.zip
RUN unzip -o glove.zip -d embeddings/
RUN . .env/bin/activate && python nltk_install.py
CMD . .env/bin/activate && python main.py
It works when built using docker build .
Docker-compose file
version: "3.9" # optional since v1.27.0
services:
pythonscripts:
build: .
volumes:
- "/mnt/d/code/data1/:/code/data/"
But when i try with docker-compose using docker-compose up. it gives following error:
Successfully built 06caf1786bfe
Successfully tagged openbotsdocumentsautomlpythonscripts_pythonscripts:latest
WARNING: Image for service pythonscripts was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating openbotsdocumentsautomlpythonscripts_pythonscripts_1 ... done
Attaching to openbotsdocumentsautomlpythonscripts_pythonscripts_1
pythonscripts_1 | /bin/sh: 1: .: Can't open .env/bin/activate
openbotsdocumentsautomlpythonscripts_pythonscripts_1 exited with code 2
My docker build is failing after there is an update to this package (https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2) I want a version of this package before the update happened (24-Jun-2020 15:09 ).
after adding the URL to source.list, apt-get update -y is failing.
Docker File :
FROM python:3.6.8
WORKDIR /app
ADD . /app
RUN apt-get update -y
RUN apt-get install python-pip -y
RUN apt-get -y update && apt-get install -y --no-install-recommends apt-utils apt-transport-https
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update -y
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql17
RUN ACCEPT_EULA=Y apt-get -y install mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
RUN /bin/bash -c "source ~/.bashrc"
RUN apt-get install unixodbc-dev
RUN pip install -r requirements.txt
EXPOSE 4040
CMD ["newrelic-admin","run-program","gunicorn", "-c", "gunicorn_config.py", "main:app"]
That error usually indicates a problem with the package repository on the server side (you can test from other client machines/networks to be sure). If you can report it to someone at Microsoft, it usually gets fixed pretty quickly and you can try again after that.
I'm new to docker and I created a docker image and this is how my docker file looks like.
FROM python:3.8.3
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-client \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get install -y gcc libtool-ltdl-devel xmlsec1-1.2.20 xmlsec1-devel-1.2.20 xmlsec1 openssl-
1.2.20 xmlsec1-openssl-devel-1.2.20 \
&& apt-get -y install curl gnupg \
&& curl -sL https://deb.nodesource.com/setup_14.x | bash - \
&& apt-get -y install nodejs
WORKDIR /app/
COPY . /app
RUN pip install -r production_requirements.txt \
&& front_end/noa-frontend/npm install
This image is used in docker-compose.yml's app service. So when I run the docker-compose build, I'm getting the below error saying it couldn't find the package. Those are few dependencies which I want to install in order to install a python package.
In the beginning, I've run the apt-get update to update the package lists.
Can anyone please help me with this issue.
Updated Dockerfile
FROM python:3.8.3
RUN apt-get update
RUN apt-get install -y postgresql-client\
&& apt-get install -y gcc libtool-ltdl-devel xmlsec1-1.2.20 xmlsec1-
devel-1.2.20 xmlsec1 openssl-1.2.20 xmlsec1-openssl-devel-1.2.20 \
&& apt-get -y install curl gnupg \
&& curl -sL https://deb.nodesource.com/setup_14.x | bash - \
&& apt-get -y install nodejs
WORKDIR /app/
COPY . /app
RUN pip install -r production_requirements.txt \
&& front_end/noa-frontend/npm install
You are trying to use apt-get install after doing rm -rf /var/lib/apt/lists/*. That is guaranteed not to end well. Try removing the rm command initially to see if that helps. If you really need to keep the size of the image down then put the rm command as the very last command in the run statement.
If you really want to reduce your image size then try switching to using python:3.8-slim or python:3.8-alpine. Alpine is a different OS to the default of Ubuntu, but its package manager can be told not to cache files locally. eg.
FROM python:3.8-alpine
RUN apk add --no-cache postgresql-client
RUN apk add --no-cache gcc libtool-ltdl-devel xmlsec1-1.2.20 xmlsec1-devel-1.2.20 xmlsec1 \
openssl-1.2.20 xmlsec1-openssl-devel-1.2.20
RUN apk add --no-cache curl gnupg
RUN apk add --no-cache nodejs
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash -
WORKDIR /app/
COPY . /app
RUN pip install -r production_requirements.txt \
&& front_end/noa-frontend/npm install
Certain bits of software might be available under different package names, so you'll have to check that out.
The instruction rm -rf /var/lib/apt/lists/* is more or less negating apt-get update. APT is no longer able to access the list of available packages after that. Move the rm to the end (and perhaps consider using the safer apt-get clean all).