We Keep Coding

Running pytest command outside of docker container failes because container stopped

I use docker selenium grid and pytest to execute tests. What i now do is:
Spin up selenium grid via a makerfile
Spin up the docker container (with a volume pointing to my local pc for the tests). The container also runs the pytest command.
This all works good, except that i would rather split the second action and be able to run the test on an already running container. Preferred setup:
Spin up selenium grid + docker container with pyton+pytest
A command to run the tests (with the container as interpretor)
When i tried to do this, i faced the issue that the python+pytest container stops running when the commands are all done. There is no long living process.
Dockerfile
FROM python:3.9.0-alpine
RUN apk add tk
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN ls ..
CMD pytest --junitxml ../r/latest.xml
My docker-compose file looks like:
docker-compose.yml
version: "3.0"
services:
pytest:
container_name: pytest
build:
context: .
dockerfile: Dockerfile
volumes:
- ./t:/t
- ./r:/r
working_dir: /t/
networks:
default:
name: my_local_network #same as selenium grid
It does not 'feel' good to have this pytest command in the container settings itself.

Container shutting down
That's because the CMD pytest --junitxml ../r/latest.xml line will execute once and when complete it will exit the container.
To run a cmd on an existing container
You can run commands on an existing docker container using this command:
docker exec <container_name> python -m pytest
Where <container_name> would be pytest in your case, since that is what the container is called in your docker-compose.yml file.
See here for more info: https://docs.docker.com/engine/reference/commandline/exec/
Using Make
If you want to extend this to a makefile command:
docker:
docker-compose up -d
ci-tests: docker
docker exec <container_name> python -m pytest
To both spin up AND run tests you can use:
make ci-tests
You could run selenium-grid in docker too if you wanted to make this solution completely portable: https://www.conductor.com/nightlight/running-selenium-grid-using-docker-compose/

airflow initdb in Docker throws ImportError: cannot import name 'import_string'

I am trying to dockerize airflow, my Dockerfile looks like this
FROM python:3.5.2
RUN mkdir -p /src/airflow
RUN mkdir -p /src/airflow/logs
RUN mkdir -p /src/airflow/plugins
WORKDIR /src
COPY . .
RUN pip install psycopg2
RUN pip install -r requirements.txt
COPY airflow.cfg /src/airflow
ENV AIRFLOW_HOME /src/airflow
ENV PYTHONPATH "${PYTHONPATH}:/src"
RUN airflow initdb
EXPOSE 8080
ENTRYPOINT ./airflow-start.sh
while my docker-compose.yml looks like this
version: "3"
services:
airflow:
container_name: airflow
network_mode: host
build:
context: .
dockerfile: Dockerfile
ports:
- 8080:8080
The output of $ docker-compose build comes up like normal, every step executes and then
Step 12/14 : RUN airflow initdb
---> Running in 8b7ebe406978
[2020-04-21 10:34:21,419] {__init__.py:45} INFO - Using executor LocalExecutor
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 17, in <module>
from airflow.bin.cli import CLIFactory
File "/usr/local/lib/python3.5/site-packages/airflow/bin/cli.py", line 59, in <module>
from airflow.www.app import cached_app
File "/usr/local/lib/python3.5/site-packages/airflow/www/app.py", line 20, in <module>
from flask_cache import Cache
File "/usr/local/lib/python3.5/site-packages/flask_cache/__init__.py", line 24, in <module>
from werkzeug import import_string
ImportError: cannot import name 'import_string'
ERROR: Service 'airflow' failed to build: The command '/bin/sh -c airflow initdb' returned a non-zero code: 1
postgres is running on host system.
I have tried multiple ways but this keeps on happening.
I even tried puckel/docker-airflow image and the same error occurred.
Can someone tell me what am I doing wrong?
Project Structure:
root
-airflow_dags
-Dockerfile
-docker-compose.yml
-airflow-start.sh
-airflow.cfg
In case it's relevant: airflow-start.sh
In airflow.cfg:
dags_folder = /src/airflow_dags/
sql_alchemy_conn = postgresql://airflow:airflow#localhost:5432/airflow

If possible get your code running without touching docker ... run it directly on your host ... of course this means your host ( your laptop or wherever you are executing your commands, could be a remote VPS debian box ) must have the same OS as your Dockerfile, I see in this case FROM python:3.5.2 is actually using debian 8
Short of doing above launch a toy container which does nothing yet executes and lets you login to it to manually run your commands to aid troubleshooting ... so use following as this toy container's Dockerfile
FROM python:3.5.2
CMD ["/bin/bash"]
so now issue this
docker build --tag saadi_now . # creates image saadi_now
now launch that image
docker run -d saadi_now sleep infinity # launches container
docker ps # lets say its container_id is b91f8cba6ed1
now login to that running container
docker exec -ti b91f8cba6ed1 bash
cool so you are now inside the docker container so run the commands which were originally in the real Dockfile ... this sometime makes it easier to troubleshoot
one by one add to this toy Dockerfile your actual commands from the real Dockerfile and redo above until you discover the underlying issues

Most likely this is related to either a bug in airflow with the werkzeug package, or your requirements might be clobbering something.
I recommend checking the versions of airflow, flask, and werkzueg that are used in the environment. It may be that you need to pin the version of flask or werkzueg as discussed here.

docker entrypoint behaviour with django

I'm trying to make my first django container with uwsgi. It works as follows:
FROM python:3.5
RUN apt-get update && \
apt-get install -y && \
pip3 install uwsgi
COPY ./projects.thux.it/requirements.txt /opt/app/requirements.txt
RUN pip3 install -r /opt/app/requirements.txt
COPY ./projects.thux.it /opt/app
COPY ./uwsgi.ini /opt/app
COPY ./entrypoint /usr/local/bin/entrypoint
ENV PYTHONPATH=/opt/app:/opt/app/apps
WORKDIR /opt/app
ENTRYPOINT ["entrypoint"]
EXPOSE 8000
#CMD ["--ini", "/opt/app/uwsgi.ini"]
entrypoint here is a script that detects whether to call uwsgi (in case there are no args) or python manage in all other cases.
I'd like to use this container both as an executable (dj migrate, dj shell, ... - dj here is python manage.py the handler for django interaction) and as a long-term container (uwsgi --ini uwsgi.ini). I use docker-compose as follows:
web:
image: thux-projects:3.5
build: .
ports:
- "8001:8000"
volumes:
- ./projects.thux.it/web/settings:/opt/app/web/settings
- ./manage.py:/opt/app/manage.py
- ./uwsgi.ini:/opt/app/uwsgi.ini
- ./logs:/var/log/django
And I manage in fact to serve the project correctly but to interact with django to "check" I need to issue:
docker-compose exec web entrypoint check
while reading the docs I would have imagined I just needed the arguments (without entrypoint)
Command line arguments to docker run will be appended after
all elements in an exec form ENTRYPOINT, and will override all
elements specified using CMD. This allows arguments to be passed to
the entry point, i.e., docker run -d will pass the -d argument
to the entry point.
The working situation with "repeated" entrypoint:
$ docker-compose exec web entrypoint check
System check identified no issues (0 silenced).
The failing one if I avoid 'entrypoint':
$ docker-compose exec web check
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"check\": executable file not found in $PATH": unknown

docker exec never uses a container's entrypoint; it just directly runs the command you give it.
When you docker run a container, the entrypoint and command you give to start it are combined to produce a single command line, and that command becomes the main container process. On the other hand, when you docker exec a command in a running container, it's interpreted literally; there aren't two parts of the command line to assemble, and the container's entrypoint isn't considered at all.
For the use case you describe, you don't need an entrypoint script to process the command in an unusual way. You can create a symlink to the manage.py script to give a shorter alias to run it, but make the default command be the uwsgi runner.
RUN chmod +x manage.py
RUN ln -s /opt/app/manage.py /usr/local/bin/dj
CMD ["uwsgi", "--ini", "/opt/app/uwsgi.ini"]
# Runs uwsgi:
docker run -p 8000:8000 myimage
# Manually trigger database migrations:
docker run --rm myimage dj migrate

docker-compose: Why is my python application being invoked here?

I've been scratching my head for a while with this. I have the following Dockerfile for my python application:
# Use an official Python runtime as a parent image
FROM frankwolf/rpi-python3
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
RUN chmod 777 docker-entrypoint.sh
# Install any needed packages specified in requirements.txt
RUN pip3 install --trusted-host pypi.python.org -r requirements.txt
# Run __main__.py when the container launches
CMD ["sudo", "python3", "__main__.py", "-debug"] # Not sure if I need sudo here
docker-compose file:
version: "3"
services:
mongoDB:
restart: unless-stopped
volumes:
- "/data/db:/data/db"
ports:
- "27017:27017"
- "28017:28017"
image: "andresvidal/rpi3-mongodb3:latest"
mosquitto:
restart: unless-stopped
ports:
- "1883:1883"
image: "mjenz/rpi-mosquitto"
FG:
privileged: true
network_mode: "host"
depends_on:
- "mosquitto"
- "mongoDB"
volumes:
- "/home/pi:/home/pi"
#image: "arkfreestyle/fg:v1.8"
image: "test:latest"
entrypoint: /app/docker-entrypoint.sh
restart: unless-stopped
And this what docker-entrypoint.sh looks like:
#!/bin/sh
if [ ! -f /home/pi/.initialized ]; then
echo "Initializing..."
echo "Creating .initialized"
# Create .initialized hidden file
touch /home/pi/.initialized
else
echo "Initialized already!"
sudo python3 __main__.py -debug
fi
Here's what I am trying to do:
(This stuff already works)
1) I need a docker image which runs my python application when I run it in a container. (this works)
2) I need a docker-compose file which runs 2 services + my python application, BUT before running my python application I need to do some initialization work, for this I created a shell script which is docker-entrypoint.sh. I want to do this initialization work ONLY ONCE when I deploy my application on a machine for the first time. So I'm creating a .initialized hidden file which I'm using as a check in my shell script.
I read that using entrypoint in a docker-compose file overwrites any old entrypoint/cmd given to the Dockerfile. So that's why in the else portion of my shell script I'm manually running my code using "sudo python3 main.py -debug", this else portion works fine.
(This is the main question)
In the if portion, I do not run my application in the shell script. I've tested the shell script itself separately, both if and else statements work as I expect, but when I run "sudo docker-compose up", the first time when my shell script hits the if portion it echoes the two statements, creates the hidden file and THEN RUNS MY APPLICATION. The console output appears in purple/pink/mauve for the application, while the other two services print their logs out in yellow and cyan. I'm not sure if the colors matter, but in the normal condition my application logs are always green, in fact the first two echoes "Initializing" and "Creating .initialized" are also green! so I thought I'd mention this detail. After those two echoes, my application mysteriously begins and logs console output in purple...
Why/how is my application being invoked in the if statement of the shell script?
(This is only happens if I run through docker-compose, not if I just run the shell script with sh docker-entrypoint.sh)

Problem 1
Using ENTRYPOINT and CMD at the same time has some strange effects.
Problem 2
This happens to your container:
It is started the first time. The .initialized file does not exist.
The if case is executed. The file is created.
The script and therefore the container ends.
The restart: unless-stopped option restarts the container.
The .initialized file exists now, the else case is run.
python3 __main__.py -debug is executed.
BTW the USER command in the Dockerfile or the user option in Docker Compose are better options than sudo.

Docker container/image running but there is no port number

I am trying to get a django project that I have built to run on docker and create an image and container for my project so that I can push it to my dockerhub profile.
Now I have everything set up and I've created the initial image of my project. However, when I run it I am not getting any port number attached to the container. I need this to test and see if the container is actually working.
Here is what I have:
Successfully built a047506ef54b
Successfully tagged test_1:latest
(MySplit) omars-mbp:mysplit omarjandali$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
test_1 latest a047506ef54b 14 seconds ago 810MB
(MySplit) omars-mbp:mysplit omarjandali$ docker run --name testing_first -d -p 8000:80 test_1
01cc8173abfae1b11fc165be3d900ee0efd380dadd686c6b1cf4ea5363d269fb
(MySplit) omars-mbp:mysplit omarjandali$ docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
01cc8173abfa test_1 "python manage.py ru…" 13 seconds ago Exited (1) 11 seconds ago testing_first
(MySplit) omars-mbp:mysplit omarjandali$ Successfully built a047506ef54b
You can see there is no port number so I don't know how to access the container through my local machine on my web browser.
dockerfile:
FROM python:3
WORKDIR tab/
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "manage.py", "runserver", "0.0.0.0"]

This line from the question helps reveal the problem;
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
01cc8173abfa test_1 "python manage.py ru…" 13 seconds ago Exited (1) 11 seconds ago testing_first
Exited (1) (from the STATUS column) means that the main process has already exited with a status code of 1 - usually meaning an error. This would have freed up the ports, as the docker container stops running when the main process finishes for any reason.
You need to view the logs in order to diagnose why.
docker logs 01cc will show the logs of the docker container that has the ID starting with 01cc. You should find that reading these will help you on your way. Knowing this command will help you immensely in debugging weirdness in docker, whether the container is running or stopped.
An alternative 'quick' way is to drop the -d in your run command. This will make your container run inline rather than as a daemon.

Created Dockerise django seed project
django-admin.py startproject djangoapp
Need a requirements.txt file outlining the Python dependencies
cd djangoapp/
RUN follwoing command to create the files required for dockerization
cat <<EOF > requirements.txt
Django
psycopg2
EOF
Dockerfile
cat <<EOF > Dockerfile
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /app
WORKDIR /app
ADD requirements.txt /app/
RUN pip install -r requirements.txt
ADD . /app/
EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
EOF
docker-compose.yml
cat <<EOF > docker-compose.yml
version: "3.2"
services:
web:
image: djangoapp
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
EOF
Run the application with
docker-compose up -d

When you created the container you published the ports. Your container would be accessible via port 8000 if it successfully built. However, as Shadow pointed out, your container exited with an error. That is why you must add the -a flag to your docker container ls command. docker container ls only shows running containers without the -a flag.
I recommend forgoing the detached flag -d to see what is causing the error. Then creating a new container after you have successfully launched the one you are working on. Or simply run the following commands once you fix the issue. docker stop testing_first then docker container rm testing_first finally run the same command you ran before. docker run --name testing_first -d -p 8000:80 test_1
I ran into similar problems with the first docker instances I attempted to run as well.