I changed my local dev. to Docker. I use the Django framework. For frontend I use the gulp build command to “create” my files. Now I tried a lot, looked into the Cookiecutter and Saleor project but still having issues to install npm in a way that I can call the gulp build command in my Docker container.
I already tried to add:
RUN curl -sL https://deb.nodesource.com/setup_10.x | bash -
RUN apt-get update && apt-get install -y \
nodejs \
COPY ./package.json /app/
RUN npm install
While npm is installed, I still can’t run the command gulp build in my Container. It just says gulp is an unknown command. So it seems npm doesn’t install the defined packages in my package.json file. Anyone here who already solved that and can give me some tips?
Dockerfile
# Pull base image
FROM python:3.7
# Define environment variable
ENV PYTHONUNBUFFERED 1
RUN apt-get update && apt-get install -y \
# Language dependencies
gettext \
# In addition, when you clean up the apt cache by removing /var/lib/apt/lists
# it reduces the image size, since the apt cache is not stored in a layer.
&& rm -rf /var/lib/apt/lists/*
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
# Install Python dependencies
RUN pip install pipenv
RUN pipenv install --system --deploy --dev
docker-compose.py
version: '3'
services:
web:
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
env_file: .env
volumes:
- .:/app
ports:
- "8000:8000"
depends_on:
- db
entrypoint: ./compose/local/django/entrypoint.sh
container_name: myproject
db:
image: postgres
ports:
- "5432:5432"
environment:
# Password will be required if connecting from a different host
- POSTGRES_PASSWORD=password
Update 1:
# Pull base image
FROM combos/python_node:3_10
# Define environment variable
ENV PYTHONUNBUFFERED 1
RUN apt-get update && apt-get install -y \
# Language dependencies
gettext \
# In addition, when you clean up the apt cache by removing /var/lib/apt/lists
# it reduces the image size, since the apt cache is not stored in a layer.
&& rm -rf /var/lib/apt/lists/*
# COPY webpack.config.js app.json package.json package-lock.json /app/
# WORKDIR /app
# RUN npm install
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
RUN npm install
# Install Python dependencies
RUN pip install pipenv
RUN pipenv install --system --deploy --dev
Are you sure npm install is being called in the correct directory? You copy package.json to /app but run npm install from an unknown folder. You can try:
COPY ./package.json /app/
RUN cd /app/ && npm install
But I think you'd want to install gulp globally anyway, so you can skip package.json and just use:
RUN npm install -g gulp-cli
This way whatever calls gulp should have it in PATH and not just that specific directory.
Also, if you want to get a Docker image with both Python 3.7 and Node.js 10 already installed, you can use combos/python_node:3.7_10. It's rebuilt daily to contain the latest versions of both images.
Related
Im installing gunicorn pip package in my docker python:3.8-slim-buster image and when I use CMD gunicorn im told /bin/sh: 1: gunicorn: not found.
So im considering changing the path but i have a few questions to do so :
should i use (in my Dockerfile):
pip --target=path_already_in_PATH install gunicorn
ENV PYTHONPATH "${PYTHONPATH}:good_path"
ENV PATH="/default_pip_path:${PATH}"
I dont know which option is better and what to put in good_path, path_already_in_PATH and default_pip_path
This is my Dockerfile :
FROM python:3.8-slim-buster
RUN apt-get update \
# dependencies for building Python packages
&& apt-get install -y build-essential netcat
# cleaning up unused files
# && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
# && rm -rf /var/lib/apt/lists/*
RUN addgroup --system kr1p \
&& adduser --system --ingroup kr1p kr1p
WORKDIR /app
COPY app .
RUN chown -R kr1p:kr1p /app
USER kr1p
RUN pip install -r requirements.txt
ENV PYTHONUNBUFFERED 1
CMD gunicorn
I ve also tried python -m gunicorn but it's the same and also CMD ["gunicorn"]
And the docker-compose.yml
---
version: '3.7'
services:
app:
container_name: app
build:
context: .
dockerfile: ./app/Dockerfile
volumes:
- app:/app
ports:
- 5000:5000
volumes:
app:
name: app
I noticed pip says "Defaulting to user installation because normal site-packages is not writeable" at the begining of the installation probably because i've created a new user
It's another issue but pip also tells me at the end : #10 385.5 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
What is the proper way to set a virtualenv to avoid issues?
Ah, so the problem shows up in the docker build output:
Step 8/10 : RUN pip install gunicorn
---> Running in 5ec725d1c957
Defaulting to user installation because normal site-packages is not writeable
Collecting gunicorn
Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.5/79.5 KB 2.3 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=3.0 in /usr/local/lib/python3.8/site-packages (from gunicorn) (57.5.0)
Installing collected packages: gunicorn
WARNING: The script gunicorn is installed in '/home/kr1p/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed gunicorn-20.1.0
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removing intermediate container 5ec725d1c957
---> c42800562d88
Step 9/10 : ENV PYTHONUNBUFFERED 1
---> Running in 8d9342ec2288```
Namely: " WARNING: The script gunicorn is installed in '/home/kr1p/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location."
This is because it's running as your non-root kr1p user, so it's actually ending up in $HOME/.local/bin/gunicorn instead.
I would either:
add that dir to the PATH statically in the dockerfile, like:
ENV PATH=/home/kr1p/.local/bin:$PATH
or, install dependencies as root, prior to switching down to the unpriv user for copying source files and other setup.
USER root
COPY requirements.txt /reqs.txt
RUN pip install --root-user-action=ignore -r /reqs.txt
USER kr1p
COPY --chown kr1p app/ ./
The root-user-action is just to suppress a message about how you should be using virtualenvs, which doesn't necessarily apply when walling things off inside a container instead. This requires a newer pip than that which comes with debian-buster though, so I ended up removing it (and you're just stuck with that warning if you use the install while root approach).
As a full working example for the PATH modifying approach, see:
FROM python:3.8-slim-buster
RUN apt-get update \
# dependencies for building Python packages
&& apt-get install -y build-essential netcat
# cleaning up unused files
# && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
# && rm -rf /var/lib/apt/lists/*
# sets kr1p home dir to /app
RUN adduser --home /app --system --group kr1p
ENV PATH=/app/.local/bin:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY app/requirements.txt .
USER kr1p
RUN pip install -r /app/requirements.txt
COPY --chown=kr1p:kr1p app .
# otherwise a shell runs gunicorn, and signals don't get passed down properly
CMD ["gunicorn", "--help"]
(There were a few other things wrong like a missing = in your ENV statement, etc.)
I am trying to build this docker image with docker compose:
FROM python:3.7-slim
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
RUN apt-get update && apt-get install -y \
build-essential \
make \
gcc \
python3-dev \
mongodb
# Create working directory and copy all files
COPY . /app
WORKDIR /app
# Pip install requirements
RUN pip install --user -r requirements.txt
# Port to expose
EXPOSE 8000
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "main.py", "runserver"]
but i get this error:
Package 'mongodb' has no installation candidate
When I run the same exact docker image with python:3.4-slim it works. Why?
That's because python:3.4-slim uses Debian stretch (9) for its base and the mongodb package is available in its repos. But for python:3.7-slim, the base is bullseye (11) and mongodb is no longer in its repos.
I'd recommend not to install mongodb in the image that you're building above but rather use a separate mongodb container.
I have a Flask API that connects to an Azure SQL database, deployed on Azure App Service in a Docker Image.
It works fine but I am trying to keep consistency between my development, staging and production environments using Alembic/Flask-Migrate to apply database upgrades.
I saw on Miguel Grinberg's Docker Deployment Tutorial, that this can be achieved by adding the flask db upgrade command to a boot.sh script, like so:
#!/bin/sh
flask db upgrade
exec gunicorn -w 4 -b :5000 --access-logfile - --error-logfile - app:app
My problem is that, when running the boot.sh script, I receive the error:
Usage: flask db [OPTIONS] COMMAND [ARGS]...
Try 'flask db --help' for help.
'.ror: No such command 'upgrade
Which indicates the script cannot find the Flask-Migrate library. This actually happens if I try other site-packages, such as just trying to run flask commands.
The weird thing is:
gunicorn works just fine
The API works just fine
I can run flask db upgrade with no problem if I fire up the container and open a terminal session with docker exec -i -t api /bin/sh
Obviously, there's a problem with my Dockerfile. I would massively appreciate any help here as I'm relatively new to Docker and Linux so I'm sure I'm missing something obvious:
EDIT: It also works just fine if I add the following line to my Dockerfile, just before the entrypoint CMD:
RUN flask db upgrade
Dockerfile
FROM python:3.8-alpine
# Dependencies for pyodbc on Linux
RUN apk update
RUN apk add curl sudo build-base unixodbc-dev unixodbc freetds-dev
RUN apk add gcc musl-dev libffi-dev openssl-dev
RUN apk add --no-cache tzdata
RUN rm -rf /var/cache/apk/*
RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.5.2.2-1_amd64.apk
RUN sudo sudo apk add --allow-untrusted msodbcsql17_17.5.2.2-1_amd64.apk
RUN mkdir /code
WORKDIR /code
COPY requirements.txt requirements.txt
RUN python -m pip install --default-timeout=100 -r requirements.txt
RUN python -m pip install gunicorn
ADD . /code/
COPY boot.sh /usr/local/bin/
RUN chmod u+x /usr/local/bin/boot.sh
EXPOSE 5000
ENTRYPOINT ["sh", "boot.sh"]
I ended up making some major changes to my Dockerfile and boot.sh script. I'll share these as best I can below:
Problem 1: Entrypoint script cannot access directories
My main issue was that I had an inconsistent folder structure in my directory. There were 2 boot.sh scripts and the one being run on entrypoint either had the wrong permissions or was in the wrong place to find my site packages.
I simplified the copying of files from my local machine to the Docker image like so:
RUN mkdir /code
WORKDIR /code
COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install --default-timeout=100 -r requirements.txt
RUN venv/bin/pip install gunicorn
COPY app app
COPY migrations migrations
COPY api.py config.py boot.sh ./
RUN chmod u+x boot.sh
EXPOSE 5000
ENTRYPOINT ["./boot.sh"]
The changes involved:
Setting up a virtualenv and installing all site packages in there
Making sure the config.py, boot.sh, and api.py files were in the root directory of the application folder (./)
Changing the entrypoint command from ["bin/sh", "boot.sh"] to just ["./boot.sh"]
Moving migrations files into the relevant folder for the upgrade script
I was then able to activate the virtual environment in the entrypoint file, and run the flask upgrade commands (NB: I had a problem with line endings being CRLF instead of LF in boot.sh, so make sure to change it if on Windows):
#!/bin/bash
source venv/bin/activate
flask db upgrade
exec gunicorn -w 4 -b :5000 --access-logfile - --error-logfile - api:app
Problem 2: Alpine Linux Too Slow
My other issue was that my image was taking forever to build (upwards of 45 mins) on Alpine Linux. Turns out this is a pretty well-established issue when using some of the libraries in my API (Pandas, Numpy).
I switched to a Debian build so that I could makes changes more quickly to my Docker image.
Including the installation of pyodbc to connect to Azure SQL Server, the first half of my Dockerfile now looks like:
FROM python:3.8-slim-buster
RUN apt-get update
RUN apt-get install -y apt-utils curl sudo gcc g++ gnupg2
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get install -y libffi-dev libgssapi-krb5-2 unixodbc-dev unixodbc freetds-dev
RUN sudo apt-get update
RUN sudo ACCEPT_EULA=Y apt-get install msodbcsql17
RUN apt-get clean -y
Where the curl commands and below come from the official MS docs on installing pyodbc on Debian
Full dockerfile:
FROM python:3.8-slim-buster
RUN apt-get update
RUN apt-get install -y apt-utils curl sudo gcc g++ gnupg2
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get install -y libffi-dev libgssapi-krb5-2 unixodbc-dev unixodbc freetds-dev
RUN sudo apt-get update
RUN sudo ACCEPT_EULA=Y apt-get install msodbcsql17
RUN apt-get clean -y
RUN mkdir /code
WORKDIR /code
COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install --default-timeout=100 -r requirements.txt
RUN venv/bin/pip install gunicorn
COPY app app
COPY migrations migrations
COPY api.py config.py boot.sh ./
RUN chmod u+x boot.sh
EXPOSE 5000
ENTRYPOINT ["./boot.sh"]
I think this is the key information.
Which indicates the script cannot find the Flask-Migrate library. This actually happens if I try other site-packages, such as just trying to run flask commands.
To me this may indicate that the problem is not specific to Flask-Migrate but to all packages - as you write. This may mean on of following two.
First, it can mean that the packages are not correctly installed. However, this is unlikely as you write that it works when you manually start the container.
Second, something is wrong with how you execute your boot.sh script. For example, try changing
ENTRYPOINT ["sh", "boot.sh"]
to
ENTRYPOINT ["/bin/sh", "boot.sh"]
HTH!
I've been trying to make the image of flask+postgresql smaller with docker multistage build, but its crashing because its not finding the flask app.py. Any idea why that is happening?
Error:
web_1 | Usage: flask run [OPTIONS]
web_1 |
web_1 | Error: The file/path provided (app.py) does not appear to exist. Please verify the path is correct. If app is not on PYTHONPATH, ensure the extension is .py
first-web-page_web_1 exited with code 2
Dockerfile that's not working:
FROM python:3.7-alpine as compile-image
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY . .
RUN apk update && apk add postgresql-dev gcc musl-dev && pip install -r requirements.txt
FROM python:3.7-alpine AS build-image
COPY --from=compile-image /opt/venv /opt/venv
# Make sure we use the virtualenv:
ENV PATH="/opt/venv/bin:$PATH"
ENV FLASK_APP app.py
ENV FLASK_RUN_HOST 0.0.0.0
CMD ["flask", "run"]
EXPOSE 5000
Dockerfile that's working that is not multistage:
FROM python:3.7-alpine as build
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN apk update && apk add postgresql-dev gcc musl-dev && pip install -r requirements.txt
FROM build
COPY --from=build /code .
ENV FLASK_APP app.py
ENV FLASK_RUN_HOST 0.0.0.0
CMD ["flask", "run"]
EXPOSE 5000
COPY . /code/
Size of the working image is about 130Mb after its uploded to the registry is it even possible to make it smaller then that?
There's a lot going on here.
First, you would typically use two containers for this sort of project: one for the database, and one for the webapp. This comes with a few advantages:
You can use the official postgresql image, rather than rolling your own
You can upgrade one component without shutting down the other (a nice feature of you are making frequent changes to your web app but would like the database to just stay up and running)
You can scale your webapp separately from the database
You can rely on docker to restart a failed service
With respect to your question:
When you COPY . ., you're not copying your application into /opt/venv. It's going into the root directory, /. In your second stage, you do this:
COPY --from=compile-image /opt/venv /opt/venv
You never copy your app into the final image.
Secondly, when you this:
RUN apk update && apk add postgresql-dev gcc musl-dev && pip install -r requirements.txt
You're not install your Python dependencies into your virtual environment, because you've never activated it.
I think you'd be better off using two separate images to start two separate containers (and then using something like docker-compose to manage your application stack).
I've managed to make the image smaller (from 140MiB ~ 24MiB) but not with multistage builds, just by removing the dependence that were needed for postgresql to be installed. If anyone has an idea why its not working with multistage I would appreciate it.
FROM python:3.7-alpine
RUN mkdir /code
ENV FLASK_APP app.py
ENV FLASK_RUN_HOST 0.0.0.0
ARG BUILD_DEPS="gcc musl-dev postgresql-dev"
ARG RUNTIME_DEPS="libpq"
WORKDIR /code
COPY requirements.txt /code/
RUN apk add --no-cache --virtual .build_deps ${BUILD_DEPS} && \
apk add --no-cache ${RUNTIME_DEPS} && \
pip install --no-cache-dir -r requirements.txt && \
apk del .build_deps
CMD ["flask", "run"]
EXPOSE 5000
COPY . /code/
I have some tar files which I want to move to docker image and extract them there and then run python web app script. I have sh script that extracts those files.
If I run the script using RUN command then they get extracted but they are not present in the final container.
I also used entrypoint but it executes and then closes container not executing "main python script".
Is there a way how to execute this install script and then continue running the main script and not closing?
Dockerfile
# Use an official Python runtime as a parent image
FROM python:2.7-slim
RUN apt-get update \
&& apt-get install -y curl \
&& curl -sL https://deb.nodesource.com/setup_4.x | bash \
&& apt-get install -y nodejs \
&& apt-get install -y git \
&& npm install -g bower \
&& npm install -g gulp#^3.9.1
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
RUN pip install -r requirements.txt \
&& npm install \
&& bower install --allow-root \
&& gulp default
# Define environment variable
ENV PATH "$PATH:/app/tree-tagger/cmd/"
ENV PATH "$PATH:/app/tree-tagger/bin/"
ENV TREETAGGER "/app/tree-tagger/cmd/"
ENV TREETAGGER_HOME "/app/tree-tagger/cmd/"
CMD python app.py
ENTRYPOINT sh tree-tagger/install-tagger.sh
Here is docker-compose script on top of that
web:
build: .
ports:
- "8080:8080"
volumes:
- .:/app
links:
- db
db:
image: mongo:3.0.2
You can just use RUN for that. RUN runs on container side so anything new that gets created by it - will be included in container as well.
Your issue most likely comes from the fact that your docker-compose file is mounting a volume that you previously wrote to.
Basically, what you're trying to do is:
1. During container creation, copy current dir to /app and do something with it
2. After container is created - run it while mounting current dir to /app again.
So you end up with anything newly created in /app/ to be overwritten. If you skip the volume part in docker-compose, it will all work as expected. But if you prefer to mount current app code to /app anyway (so you don't have to rebuild image during development every time your code change), there is a way. Just change your docker-compose to:
web:
build: .
ports:
- "8080:8080"
volumes:
- .:/app
- /app/tree-tagger/
links:
- db
db:
image: mongo:3.0.2
Now you can just change your dockerfile to what you had before (with RUN instead of ENTRYPOINT) and /app/tree-tagger/ won't be replaced this time. So:
...
RUN sh tree-tagger/install-tagger.sh
CMD python app.py