We Keep Coding

Install local package through a Dockerfile

I have started learning Docker and I have developed a Python package (not published anywhere, it is just used internally) that installs and works fine locally (here I will call it mypackage). However, when trying to install it in a Docker container, Python in the container fails to recognise it even though during the build of the image no error was raised. The Dockerfile looks like this:
# install Ubuntu 20.04
FROM ubuntu:20.04
# update Ubuntu packages
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update
RUN apt upgrade -y
RUN apt install -y apt-utils \
build-essential \
curl \
mysql-server \
libmysqlclient-dev \
libffi-dev \
libssl-dev \
libxml2-dev \
libxslt1-dev \
unzip \
zlib1g-dev
# install Python 3.9
RUN apt-get install -y software-properties-common gcc && \
add-apt-repository -y ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3.9-dev python3.9-distutils python3-pip python3-apt python3.9-venv
# make symlink (overriding default Python3.8 installed with Ubuntu)
RUN rm /usr/bin/python3
RUN ln -s /usr/bin/python3.9 /usr/bin/python3
# copy package files and source code
RUN mkdir mypackage
COPY pyproject.toml setup.cfg setup.py requirements.txt ./mypackage/
COPY src mypackage/src/
# add path
ENV PACKAGE_PATH=/mypackage/
ENV PATH="$PACKAGE_PATH/:$PATH"
# install mypackage
RUN pip3 install -e ./mypackage
CMD ["python3.9", "main.py"]
So the above runs successfully, but if I run sudo docker run -it test_image bin/bash and run pip3 list, the package will not be there and a ModuleNotFoundError when running code depending on mypackage. Interestingly if I create a virtual environment by replacing this:
ENV PACKAGE_PATH=/mypackage/
ENV PATH="$PACKAGE_PATH/:$PATH"
by this:
ENV VIRTUAL_ENV=/opt/venv
RUN python3.9 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
it works. Ideally, I want to know why I need to create a virtual environment and how can I run local packages in a container without creating virtual environments.

How to install db-core or any adapter in Windows 10 with Python (pip)

For the last couple of days I've struggled to install Dbt in my Windows 10 box. It seems the best way is to emulate Linux, with WSL.
So, in order to help others to save their time and a few neurons, I decided to post a quick recipe in this thread. I summarized the whole process in 7 steps, together with a nice and complete tutorial
Enable WSL
https://learn.microsoft.com/en-us/windows/wsl/install
Install Linux Ubuntu
https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-10#1-overview
Install Python
As python3 comes with Ubuntu by default, you won't need to do anything in this step. Otherwise, you can always got to:
https://packaging.python.org/en/latest/tutorials/installing-packages/#requirements-for-installing-packages
Install Pip
https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment
Install VirtualEnv
https://docs.python.org/3/library/venv.html
I hope it helps. If not you can always post a message in this thread!
Best wishes,
I

Another way you can run dbt-core on Windows is with Docker. I'm currently on Windows 10 and use a Docker image for my dbt project without needing WSL. Below is my Dockerfile and requirements.txt file with dbt-core and dbt-snowflake but feel free to swap the packages you need.
In my repo, my dbt project is in a folder at the root level named dbt.
requirements.txt
dbt-core==1.1.0
dbt-snowflake==1.1.0
Dockerfile
FROM public.ecr.aws/docker/library/python:3.8-slim-buster
COPY . /dbt
# Update and install system packages
RUN apt-get update -y && \
apt-get install --no-install-recommends -y -q \
git libpq-dev python-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Install dbt
RUN pip install -U pip
RUN pip install -r dbt/requirements.txt
# TEMP FIX due to dependency updates. See https://github.com/dbt-labs/dbt-core/issues/4745
RUN pip install --force-reinstall MarkupSafe==2.0.1
# Install dbt dependencies
WORKDIR /dbt
RUN dbt deps
# Specify profiles directory
ENV DBT_PROFILES_DIR=.dbt
# Expose port for dbt docs
EXPOSE 8080
And then you can build and run it (I personally put both of these commands in a dbt_run.sh file and run with bash dbt_run.sh):
docker build -t dbt_image .
docker run \
-p 8080:8080 \
--env-file .env \
-it \
--mount type=bind,source="$(pwd)",target=/dbt \
dbt_image bash
If you make changes to your dbt project while the container is running they will be reflected in the container which makes it great for developing locally. Hope this helps!

Default pip package PATH in python:3.8-slim-buster docker image

Im installing gunicorn pip package in my docker python:3.8-slim-buster image and when I use CMD gunicorn im told /bin/sh: 1: gunicorn: not found.
So im considering changing the path but i have a few questions to do so :
should i use (in my Dockerfile):
pip --target=path_already_in_PATH install gunicorn
ENV PYTHONPATH "${PYTHONPATH}:good_path"
ENV PATH="/default_pip_path:${PATH}"
I dont know which option is better and what to put in good_path, path_already_in_PATH and default_pip_path
This is my Dockerfile :
FROM python:3.8-slim-buster
RUN apt-get update \
# dependencies for building Python packages
&& apt-get install -y build-essential netcat
# cleaning up unused files
# && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
# && rm -rf /var/lib/apt/lists/*
RUN addgroup --system kr1p \
&& adduser --system --ingroup kr1p kr1p
WORKDIR /app
COPY app .
RUN chown -R kr1p:kr1p /app
USER kr1p
RUN pip install -r requirements.txt
ENV PYTHONUNBUFFERED 1
CMD gunicorn
I ve also tried python -m gunicorn but it's the same and also CMD ["gunicorn"]
And the docker-compose.yml
---
version: '3.7'
services:
app:
container_name: app
build:
context: .
dockerfile: ./app/Dockerfile
volumes:
- app:/app
ports:
- 5000:5000
volumes:
app:
name: app
I noticed pip says "Defaulting to user installation because normal site-packages is not writeable" at the begining of the installation probably because i've created a new user
It's another issue but pip also tells me at the end : #10 385.5 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
What is the proper way to set a virtualenv to avoid issues?

Ah, so the problem shows up in the docker build output:
Step 8/10 : RUN pip install gunicorn
---> Running in 5ec725d1c957
Defaulting to user installation because normal site-packages is not writeable
Collecting gunicorn
Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.5/79.5 KB 2.3 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=3.0 in /usr/local/lib/python3.8/site-packages (from gunicorn) (57.5.0)
Installing collected packages: gunicorn
WARNING: The script gunicorn is installed in '/home/kr1p/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed gunicorn-20.1.0
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removing intermediate container 5ec725d1c957
---> c42800562d88
Step 9/10 : ENV PYTHONUNBUFFERED 1
---> Running in 8d9342ec2288```
Namely: " WARNING: The script gunicorn is installed in '/home/kr1p/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location."
This is because it's running as your non-root kr1p user, so it's actually ending up in $HOME/.local/bin/gunicorn instead.
I would either:
add that dir to the PATH statically in the dockerfile, like:
ENV PATH=/home/kr1p/.local/bin:$PATH
or, install dependencies as root, prior to switching down to the unpriv user for copying source files and other setup.
USER root
COPY requirements.txt /reqs.txt
RUN pip install --root-user-action=ignore -r /reqs.txt
USER kr1p
COPY --chown kr1p app/ ./
The root-user-action is just to suppress a message about how you should be using virtualenvs, which doesn't necessarily apply when walling things off inside a container instead. This requires a newer pip than that which comes with debian-buster though, so I ended up removing it (and you're just stuck with that warning if you use the install while root approach).
As a full working example for the PATH modifying approach, see:
FROM python:3.8-slim-buster
RUN apt-get update \
# dependencies for building Python packages
&& apt-get install -y build-essential netcat
# cleaning up unused files
# && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
# && rm -rf /var/lib/apt/lists/*
# sets kr1p home dir to /app
RUN adduser --home /app --system --group kr1p
ENV PATH=/app/.local/bin:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY app/requirements.txt .
USER kr1p
RUN pip install -r /app/requirements.txt
COPY --chown=kr1p:kr1p app .
# otherwise a shell runs gunicorn, and signals don't get passed down properly
CMD ["gunicorn", "--help"]
(There were a few other things wrong like a missing = in your ENV statement, etc.)

Dockerfile: /bin/sh: 1: apt-get: not found

When building a Docker file, I get the error
"/bin/sh: 1: apt-get: not found"
docker file:
FROM python:3.8
FROM ubuntu:20.04
ENV PATH="/env/bin/activate"
RUN apt-get update -y && apt-get upgrade -y
WORKDIR /var/www/html/
COPY . .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD ["python", "manage.py"]

You are setting the PATH to /env/bin/activate and that is then the only place where apt-get is searched for. There is no need to activate a virtual env inside the container, just get rid of that line. pip can install the packages in requirements.txt to the "system" Python without issues.
You cannot layer 2 images like you are attempting to do, with multiple FROM statements. Just use FROM python:3.8 and drop the ubuntu. Multiple FROM statements are used in multi-stage builds where you have intermediate images which produce artifacts that are copied to the final image.
So just do:
FROM python:3.8
RUN apt-get update -y && apt-get upgrade -y
WORKDIR /var/www/html/
COPY . .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD ["python", "manage.py"]
.. although why you would put Python code in /var/www/html beats me. Probably you don't.

Run python mysql client on slim python 3.6 docker image

I have a working service running on a python:3.6-jessie image.
I am trying to reduce the size of it to speed up serverless cold starts.
I have tried the images python:3.6-alpine, python:3.6-slim-buster and python:3.6-slim-jessie.
With all of them I end up having to install many additional packages and I end up with the follwing error that I cannot fix with more packages:
ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
My current Dockerfile is
FROM python:3.6-jessie as build
ENV PYTHONUNBUFFERED 0
ENV FLASK_APP "api/app.py"
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
WORKDIR /opt/venv
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
FROM python:3.6-slim-jessie
COPY --from=build /opt/venv /opt/venv
WORKDIR /opt/venv
RUN apt-get update
RUN apt-get --assume-yes install gcc
RUN apt-get --assume-yes install python-mysqldb
ENV PATH="/opt/venv/bin:$PATH"
RUN rm -rf configs tests draw_results env .idea .git .pytest_cache
EXPOSE 8000
CMD ["/opt/venv/run.sh"]
The relevant lines from requirements.txt:
mysqlclient==1.4.2.post1
PyMySQL==0.9.3
Flask-SQLAlchemy==2.3.2
SQLAlchemy==1.3.0
The run.sh is just my gunicorn start command.
Is there any package I can use to fix this last issue, is there some other mysql library I should be using or some other way for me to fix this. Or should I just stick to full python:3.6 images when I want a mysql client?

I'm using python:3.7-slim and using the following command
RUN apt-get -y install default-libmysqlclient-dev

Try to add this line to the dockerfile:
RUN apt-get install -y libmysqlclient-dev

For python slim-buster (debian os) use can run this command on Dockerfile.
RUN apt-get update && apt-get install -y default-mysql-client
This worked for me.
I have used python:3.10.6-slim-buster