I am relatively new to Docker and, as an experiment, I am trying to create just a generic Django development container with the following Dockerfile:
FROM python
ENV PYTHONUNBUFFERED 1
RUN apt-get update && apt-get dist-upgrade -y
RUN mkdir /code
WORKDIR /code
RUN python3 -m venv djangoProject
RUN /bin/bash -c "source /code/djangoProject/bin/activate && python3 -m pip install --upgrade pip && pip install django"
EXPOSE 8000
The image seems to build okay, but when I go to run the container:
docker container run -v /home/me/dev/djangoRESTreact/code:/code -it --rm djangodev /bin/bash
My local mount, /home/me/dev/djangoRESTreact/code, is not populated with the djangoProject venv directory I was expecting from this Dockerfile and mount. The docker container also has an empty directory at /code. If I run python3 -m venv djangoProject directly inside the container, the venv directory is created and I can see it both on the host and within the container.
Any idea why my venv is not being created in the image and subsequent container?
I'm pulling my hair out.
Thanks in advance!
You don't need venvs in a Docker container at all, so don't bother with one.
FROM python
ENV PYTHONUNBUFFERED 1
RUN apt-get update && apt-get dist-upgrade -y
RUN mkdir /code
WORKDIR /code
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install django
EXPOSE 8000
To answer your question, though, you're misunderstanding how -v mounts work; they mount a thing from your host onto a directory in the container. The /code/... created in your dockerfile is essentially overridden by the volume mount, which is why you don't see the venv at all.
When you mount a volume into a container, the volume covers up anything that was already in the container at that location. This is the exact same way that every other mount on Linux works. Also, volumes are only mounted when building containers, not when running them. Thus, the venv that you put in that location while building isn't visible without running. If you want your venv to be visible, then you need to put it in the volume, not just in the container at the same place.
Mounting the volume with -v causes /home/me/dev/djangoRESTreact/code on the host to be mounted at /code in the container. This mounts over anything that was placed there during the build (your venv).
If you run the container without the -v flag, you'll probably find the venv directory exists.
You should probably avoid creating a venv within the container, as it's an isolated environment.
Instead just copy your requirements.txt into the container, and install them directly in the container. Something like:
COPY ./requirements.txt /requirements.txt
RUN pip install -U pip && pip install -r /requirements.txt
Related
I am running docker compose up which consists of multiple containers on of which is python 3.* and all the containers have volumes attached to them.
also I have already created requirements.txt file
I have entered python container and install x packages then I did
pip freeze > requirements.txt
I then I stoped the containers and restart the containers again, but python container didn't start and the log says modules x is not found, so what I did is that O deleted the container and created a new one and it worked,
my questions is, Is there any way to not deleting the container (I think its over kill)
but some-who still able to manage installing packages in the container?
Dockerfile
FROM python:3.6
RUN apt-get update
RUN apt-get install -y gettext
RUN mkdir -p /var/www/server
COPY src/requirements.txt /var/www/server/
WORKDIR /var/www/server
RUN pip install -r ./requirements.txt
EXPOSE 8100
ENTRYPOINT sleep 3 && python manage.py migrate && python manage.py runserver 0.0.0.0:8100
You should move your project source files into the container during build and within in it run the pip install -r requirements.txt.
Below is an example to give you an idea:
--- Other build commands follow
WORKDIR /usr/src/service
COPY ./service . # Here, I am moving everything of the service folder/module into WORKDIR within docker
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt
-- Other build commands follow.
Finally, you will use docker-compose build service to build the define service in the docker-compose.yml pointing to the Dockerfile in the build context.
...
build:
context: .
dockerfile: service/Dockerfile
...
Broadly, set up your Dockerfile such that you need to do the least-changing and most time-costly work first
FROM FOO
RUN get os-level and build dependencies
COPY only exactly files needed to identify dependencies
RUN install dependencies that takes a long time
RUN install more frequently-changing dependencies
COPY rest of your wanted content
ENTRYPOINT define me
as #coldly says in their Answer, write your dependencies into a requirements file and install them during the container build!
I've python code that includes tqdm code.
the bash build docker image and container however I can't see any output from the container(in CLI).
#!/bin/sh
docker build . -t traffic
docker run -d --name traffic_con traffic
docker wait traffic_con
docker cp -a traffic_con:/usr/TrafficMannager/out/data/. ./out/data/
docker rm /traffic_con
docker rmi /traffic
I've tried to run the container on interactive mode (-it) however it's throwing an error
[EDIT:]
Docker file:
FROM cityflowproject/cityflow
# Create a folder we'll work in
WORKDIR /usr/TrafficMannager
# Upgrade installed packages
RUN apt-get update && apt-get upgrade -y && apt-get clean
# Install vim to open & edit code\text files
RUN apt-get install -y vim
# Install all python code depentences
RUN pip install gym && \
pip install numpy && \
pip install IPython && \
pip install torch && \
python -m pip install python-dotenv &&\
pip install tqdm
COPY . .
CMD chmod u+x script/container_instructions.sh; ./script/container_instructions.sh
container_instructions:
#!/bin/sh
pip install lib/extern/CityFlow/.
python main.py
You run the Docker container in the background, then immediately docker wait for it. If you run the container in the foreground instead, you'll see its output on stdout, and the docker run command will complete when the container exits.
docker run --name traffic_con traffic # without -d
Given the wrapper script you show, you may find this setup much easier to run in a Python virtual environment. Ignore all the Docker parts and run:
python3 -m venv venv
./venv/bin/pip install gym numpy IPython torch python-dotenv tqdm lib/extern/CityFlow
./venv/bin/python3 main.py
The script will directly write to ./out/data on the host system, without the long-winded privileged script to copy data out.
If you really do need a container here, you can also mount the output directory into the container to avoid the manual copy step.
#!/bin/sh
docker build . -t traffic
docker run --rm -v "$PWD/out/data:/usr/TrafficMannager/out/data" traffic
docker rmi traffic
I am building a Docker based Flask API that needs to connect to a remote Oracle Database. I can get it to work on my machine outside of Docker but when I go to containerize it I get the error. I have tried every article I can find on stackoverflow and I still get the error:
load by OS failure: libclntsh.so: cannot open shared object file: No such file or directory
I have tried 3 different ways:
FROM python:3.9-buster
ENV DPI_DEBUG_LEVEL=64
# Installing Oracle instant client
# INSTALL TOOLS
RUN apt-get update \
&& apt-get -y install unzip \
&& apt-get -y install libaio1 libaio-dev \
&& mkdir -p /opt/data/api
ADD ./oracle-instantclient/ /opt/data
ADD ./install-instantclient.sh /opt/data
ADD ./requirements.txt /opt/data
WORKDIR /opt/data
ENV ORACLE_HOME=/opt/oracle/instantclient
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
ENV OCI_HOME=/opt/oracle/instantclient
ENV OCI_LIB_DIR=/opt/oracle/instantclient
ENV OCI_INCLUDE_DIR=/opt/oracle/instantclient/sdk/include
RUN ./install-instantclient.sh
# Python set up
# set working directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# add and install requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# add app
COPY . .
# add entrypoint.sh
COPY ./entrypoint.sh .
RUN chmod +x /usr/src/app/entrypoint.sh
This was the first way I tried for instant client:
# Install system dependencies and clean up rpms afterwards
RUN apt-get update \
&& apt-get -y install alien unzip libaio1 \
&& apt-get clean
# ZIP Install
ENV DPI_DEBUG_LEVEL=64
ENV INSTANT_CLIENT_FILE=instantclient-basic-linux.x64-19.13.0.0.0dbru.zip
RUN mkdir -p /opt/oracle
ADD ./resources/${INSTANT_CLIENT_FILE} /opt/oracle
RUN apt-get -y install unzip
RUN unzip /opt/oracle/${INSTANT_CLIENT_FILE} -d /opt/oracle
RUN ln -s /opt/oracle/instantclient_19_13/libclntsh.so.19.13 /usr/lib/libclntsh.so
RUN rm -rf /opt/oracle/${INSTANT_CLIENT_FILE}
# This needs to be set to the path that was created when the unzip occurred
# I figured out what the directory name after /opt/oracle was going to be by
# unzipping the file on my computer
ENV ORACLE_HOME=/opt/oracle/instantclient_19_13
ENV LD_LIBRARY_PATH=${ORACLE_HOME}
ENV ORACLE_BASE=${ORACLE_HOME}
ENV PATH="${ORACLE_HOME}:${PATH}"
RUN sh -c "echo ${ORACLE_HOME} > /etc/ld.so.conf.d/oracle-instantclient.conf"
RUN ldconfig
Then tried RPM install:
# RPM Install
ENV INSTANT_CLIENT_FILE_NAME=oracle-instantclient-basic-21.4.0.0.0-1.el8.x86_64
RUN mkdir /resources
COPY ./resources/${INSTANT_CLIENT_FILE_NAME}.rpm /resources
RUN alien -ct --scripts /resources/${INSTANT_CLIENT_FILE_NAME}.rpm
#RUN alien --scripts --to-deb /resources/${INSTANT_CLIENT_FILE_NAME}.tgz
RUN apt-get -y install ./resources/${INSTANT_CLIENT_FILE_NAME}.deb
RUN rm -rf ./resources/${INSTANT_CLIENT_FILE_NAME}.rpm
RUN rm -rf ./resources/${INSTANT_CLIENT_FILE_NAME}.deb
Each time I get the error but I have either specified LD_LIBRARY_PATH directly or I have run:
RUN sh -c "echo ${ORACLE_HOME} > /etc/ld.so.conf.d/oracle-instantclient.conf"
RUN ldconfig
And if I run ldconfig -p in the container then I see my entries. Or if I look at the environment variables in the container everything is set and I can see them. But I still get the error about not being able to find the files. Any other suggestions would be greatly appreciated.
So I have been working on this for two days. And there was a part of my setup that I was over looking that turned out to be causing the issue. I'm on a Mac Mini M1 and the reason that nothing I tried worked. I was missing an important part in my Dockerfile. I needed to add --platform=linux/amd64. I didn't know this because I just switched to the Mac Mini 2 days ago and this wasn't something I needed to do before. Hopefully if someone runs into the same issue they will find this and it will help them.
Too long for a comment, and you have a few scenarios, so here are some thoughts.
Use Oracle's container which already has cx_Oracle? Look for the *-oracledb container on https://github.com/oracle/docker-images/pkgs/container/oraclelinux7-python
Never set ORACLE_HOME with Instant Client.
What are those OCI_HOME, OCI_LIB_DIR and OCI_INCLUDE_DIR variables for? They are not used by cx_Oracle install or runtime.
With RPMs on Ubuntu I do:
alien -i --scripts oracle-instantclient19.13-basic-19.13.0.0.0-1.x86_64.rpm
alien -i --scripts oracle-instantclient19.13-sqlplus-19.13.0.0.0-1.x86_64.rpm
apt-get install libaio1
Then I don't need to create symlinks, or run ldconfig. I.e. it should 'just work'.
Perhaps check my blog post series Docker for Oracle Database Applications in Node.js and Python which has some Dockerfile examples for Python cx_Oracle?
I have a Flask API that connects to an Azure SQL database, deployed on Azure App Service in a Docker Image.
It works fine but I am trying to keep consistency between my development, staging and production environments using Alembic/Flask-Migrate to apply database upgrades.
I saw on Miguel Grinberg's Docker Deployment Tutorial, that this can be achieved by adding the flask db upgrade command to a boot.sh script, like so:
#!/bin/sh
flask db upgrade
exec gunicorn -w 4 -b :5000 --access-logfile - --error-logfile - app:app
My problem is that, when running the boot.sh script, I receive the error:
Usage: flask db [OPTIONS] COMMAND [ARGS]...
Try 'flask db --help' for help.
'.ror: No such command 'upgrade
Which indicates the script cannot find the Flask-Migrate library. This actually happens if I try other site-packages, such as just trying to run flask commands.
The weird thing is:
gunicorn works just fine
The API works just fine
I can run flask db upgrade with no problem if I fire up the container and open a terminal session with docker exec -i -t api /bin/sh
Obviously, there's a problem with my Dockerfile. I would massively appreciate any help here as I'm relatively new to Docker and Linux so I'm sure I'm missing something obvious:
EDIT: It also works just fine if I add the following line to my Dockerfile, just before the entrypoint CMD:
RUN flask db upgrade
Dockerfile
FROM python:3.8-alpine
# Dependencies for pyodbc on Linux
RUN apk update
RUN apk add curl sudo build-base unixodbc-dev unixodbc freetds-dev
RUN apk add gcc musl-dev libffi-dev openssl-dev
RUN apk add --no-cache tzdata
RUN rm -rf /var/cache/apk/*
RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.5.2.2-1_amd64.apk
RUN sudo sudo apk add --allow-untrusted msodbcsql17_17.5.2.2-1_amd64.apk
RUN mkdir /code
WORKDIR /code
COPY requirements.txt requirements.txt
RUN python -m pip install --default-timeout=100 -r requirements.txt
RUN python -m pip install gunicorn
ADD . /code/
COPY boot.sh /usr/local/bin/
RUN chmod u+x /usr/local/bin/boot.sh
EXPOSE 5000
ENTRYPOINT ["sh", "boot.sh"]
I ended up making some major changes to my Dockerfile and boot.sh script. I'll share these as best I can below:
Problem 1: Entrypoint script cannot access directories
My main issue was that I had an inconsistent folder structure in my directory. There were 2 boot.sh scripts and the one being run on entrypoint either had the wrong permissions or was in the wrong place to find my site packages.
I simplified the copying of files from my local machine to the Docker image like so:
RUN mkdir /code
WORKDIR /code
COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install --default-timeout=100 -r requirements.txt
RUN venv/bin/pip install gunicorn
COPY app app
COPY migrations migrations
COPY api.py config.py boot.sh ./
RUN chmod u+x boot.sh
EXPOSE 5000
ENTRYPOINT ["./boot.sh"]
The changes involved:
Setting up a virtualenv and installing all site packages in there
Making sure the config.py, boot.sh, and api.py files were in the root directory of the application folder (./)
Changing the entrypoint command from ["bin/sh", "boot.sh"] to just ["./boot.sh"]
Moving migrations files into the relevant folder for the upgrade script
I was then able to activate the virtual environment in the entrypoint file, and run the flask upgrade commands (NB: I had a problem with line endings being CRLF instead of LF in boot.sh, so make sure to change it if on Windows):
#!/bin/bash
source venv/bin/activate
flask db upgrade
exec gunicorn -w 4 -b :5000 --access-logfile - --error-logfile - api:app
Problem 2: Alpine Linux Too Slow
My other issue was that my image was taking forever to build (upwards of 45 mins) on Alpine Linux. Turns out this is a pretty well-established issue when using some of the libraries in my API (Pandas, Numpy).
I switched to a Debian build so that I could makes changes more quickly to my Docker image.
Including the installation of pyodbc to connect to Azure SQL Server, the first half of my Dockerfile now looks like:
FROM python:3.8-slim-buster
RUN apt-get update
RUN apt-get install -y apt-utils curl sudo gcc g++ gnupg2
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get install -y libffi-dev libgssapi-krb5-2 unixodbc-dev unixodbc freetds-dev
RUN sudo apt-get update
RUN sudo ACCEPT_EULA=Y apt-get install msodbcsql17
RUN apt-get clean -y
Where the curl commands and below come from the official MS docs on installing pyodbc on Debian
Full dockerfile:
FROM python:3.8-slim-buster
RUN apt-get update
RUN apt-get install -y apt-utils curl sudo gcc g++ gnupg2
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get install -y libffi-dev libgssapi-krb5-2 unixodbc-dev unixodbc freetds-dev
RUN sudo apt-get update
RUN sudo ACCEPT_EULA=Y apt-get install msodbcsql17
RUN apt-get clean -y
RUN mkdir /code
WORKDIR /code
COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install --default-timeout=100 -r requirements.txt
RUN venv/bin/pip install gunicorn
COPY app app
COPY migrations migrations
COPY api.py config.py boot.sh ./
RUN chmod u+x boot.sh
EXPOSE 5000
ENTRYPOINT ["./boot.sh"]
I think this is the key information.
Which indicates the script cannot find the Flask-Migrate library. This actually happens if I try other site-packages, such as just trying to run flask commands.
To me this may indicate that the problem is not specific to Flask-Migrate but to all packages - as you write. This may mean on of following two.
First, it can mean that the packages are not correctly installed. However, this is unlikely as you write that it works when you manually start the container.
Second, something is wrong with how you execute your boot.sh script. For example, try changing
ENTRYPOINT ["sh", "boot.sh"]
to
ENTRYPOINT ["/bin/sh", "boot.sh"]
HTH!
I am working on setting up a dockerised selenium grid. I can send my python tests [run with pytest] from a pytest container [see below] by attaching to it.
But I have setup another LAMP container that is going to control pytest.
So I want to make the pytest container standalone,running idle and waiting for commands from the LAMP container.
I have this Dockerfile:
# Starting from base image
FROM ubuntu
#-----------------------------------------------------
# Set the Github personal token
ENV GH_TOKEN blablabla
# Install Python & pip
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y python python-pip python-dev && pip install --upgrade pip
# Install nano for #debugging
RUN apt-get install -y nano
# Install xvfb
RUN apt-get install -y xvfb
# Install GIT
RUN apt-get update -y && apt-get install git -y
# [in the / folder]
RUN git clone https://$GH_TOKEN:x-oauth-basic#github.com/user/project.git /project
# Install dependencies via pip
WORKDIR /project
RUN pip install -r dependencies.txt
#-----------------------------------------------------
#
CMD ["/bin/bash"]
I start the pytest container manually [for development] with this:
docker run -dit -v /project --name pytest repo/user:py
The thing is that I finished development and I want to have the pytest container launched from docker-compose and connect it to other containers [with link and volume].
I just cannot make it to stay up .
I used this :
pytest:
image: repo/user:py
volumes:
- "/project"
command: "/bin/bash tail -f /dev/null"
but didnt work.
So, inside the Dockerfile, should I use a specific CMD or ENTRYPOINT ?
Should I use some command from the docker-compose file?
I just enabled it on one of my projects recently. I use a multistage build. At present I put tests in the same folder as the source test_*.py. From my experience with this, it doesn't feel natural, I prefer tests to be in its own folder that is excluded by default.
FROM python:3.7.6 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip3 install --compile -r requirements.txt && rm -rf /root/.cache
COPY src /app
# TODO precompile
# Build stage test - run tests
FROM build AS test
RUN pip3 install pytest pytest-cov && rm -rf /root/.cache
RUN pytest --doctest-modules \
--junitxml=xunit-reports/xunit-result-all.xml \
--cov \
--cov-report=xml:coverage-reports/coverage.xml \
--cov-report=html:coverage-reports/
# Build stage 3 - Complete the build setting the executable
FROM build AS final
CMD [ "python", "./service.py" ]
In order to exclude the test files from coverage. .coveragerc must be present.
[run]
omit = test_*
The test target runs the required tests and generates coverage and execution reports. These are NOT suitable for Azure DevOps and SonarQube. To make it suitable
sed -i~ 's#/app#$(Build.SourcesDirectory)/app#' $(Pipeline.Workspace)/b/coverage-reports/coverage.xml
To run tests
#!/usr/bin/env bash
set -e
DOCKER_BUILDKIT=1 docker build . --target test --progress plain
I am not exactly sure how your tests execute, but I think I have a similar use case. You can see how I do this in my Envoy project in cmd.sh, and a sample test.
Here is how I run my tests. I'm using pytest as well, but thats not important:
1. use docker-compose to bring up the stack, without the tests
2. wait for the stack to be ready for requests. for me this means poll for a 200 response
3. run the test container separately but make sure it uses the same network as the compose stack.
This can be done in several ways. You can put this all in a Bash script and control it all from your host.
In my case I do this all from a Python container. Its a little to wrap your head around, but the idea is there is a Python test container which the host starts. Then the container itself uses compose to bring up the stack back on the host (dockerception). And then in the test container we run the pytest test. When its done, it composes down the stack and pushes up the return code.
At first you must get list of your images with "docker images".
Then watch the list and ensure your image is exists.
So run your docker image with "docker run".
Do not forget this note: you must CMD in DockerFile to run pytest with your container.
my Dockerfile
FROM python:3.6-slim
COPY . /python-test-calculator
WORKDIR /python-test-calculator
RUN pip freeze > requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
RUN mkdir reports
CMD cd reports
CMD [python", "-m", "pytest", "--junitxml=reports/result.xml"]
CMD tail -f /dev/null