docker container exited immediately after python script execution:
docker run -t -i -v /root/test.py:/test.py zookeeper python test.py
(test.py starts zookeeper service )
The command is successful but exits immediately with out starting container. I could NOT start the container with "docker start container id".
Manually running "python test.py" is successful inside container but not during "docker run ...."
Just starting the server is not enough. When the CMD exits, so does the container. Thus, if you start a service that's a daemon, you need to keep your process alive. This can be achieved by, for example, tailing the service log file. supervisord is another way to run processes and keep the CMD alive.
For example, you might do
CMD /test.py && tail -F /var/log/zookeeper.log
Running from the commandline you could do something similar
docker run -t -i -v /root/test.py:/test.py zookeeper bash -c "python test.py && tail -F /var/log/zookeeper.log"
Related
I have the docker container that executes the python file - I want it to restart on the failure of the script - usually memory errors.
Docker is running, python file works but after script failure container just exit.
The container just --restart always policy does not work - what am I doing wrong?
docker command:
sudo docker run --init --gpus all \
--ipc host --privileged --net host \
-p 8888:8888 -p49053:49053
--restart always \
-v /mnt/disks/sde:/home/sliceruser/data \
-v /mnt/disks/sdb:/home/sliceruser/dataOld \
slicerpicai:latest
end of docker file
ENTRYPOINT [ "/bin/bash", "start.sh","-l", "-c" ]
start.sh
cd /home/sliceruser/data/piCaiCode
git pull
python3.8 /home/sliceruser/data/piCaiCode/Three_chan_baseline_hyperParam.py
He restart policy works, you might just not see it. I suggest to check the number of retries so far on the container:
docker inspect -f "{{ .RestartCount }}" my-container
Also docker tries indefinitely (with --retry always) but it does wait always longer if the start keeps failing.
If you say your script has memory issues, it would be good to address those before looking at issues with Docker. if the reason why the container stops lies outside docker, that obviously stops the container from restarting as well. So I recommend checking the container logs and thinking of what you do in order to manually restart the container after a failure.
For more details check also the official reference of docker run
If you want to reproduce what I wrote above do the following:
Open 1 terminal and run:
docker stats
Open a second terminal and run:
docker run -d --name testcontainer --restart always alpine:latest sh -c "sleep 5 && exit 2"
This will start a container that "crashes" every 5s.
In the same terminal run:
# check the status and see how it waits longer and longer to restart
docker container ls --filter name="testcontainer"
# check the number of restarts so far
docker inspect -f "{{ .RestartCount }}" testcontainer
Friendly footnote: I think you are lucky it doesn't restart because this is such an unsecure container. ;)
when I run "docker exec -it docker-name bash" on centOS7 service ,it will go into docker container and can run " python xx.py config.yaml " to execute some works .
but if I use Jenkins shell run "docker exec -it docker-name bash" ,it will have no response ,I write "python xx.py config.yaml " behind ,Jenkins show [ python: can't open file 'xxx.py': [Errno 2] No such file or directory ] ,I think this error is not into the docker container ,so can't find the python file that in the docker container .How can I enter the docker container with Jenkins shell .
When you run docker exec -it docker-name bash, you get an interactive shell inside the container that gets connected to your console and the next command you type to the console is executed in that shell.
But Jenkins has no console. It is executing a script, with the standard input connected to a null device (which always returns end of file on read). So in effect it is executing the equivalent of
docker exec -it docker-name bash </dev/null (the /dev/null is the null device and < connects it to standard input of the command). And if you do that on your console, nothing happens and you'll get your original prompt again.
But you don't have to, and shouldn't be, running bash in this case at all. You give docker exec the command you want to run in the container and it runs it there. So you just do
docker exec -i docker-name python xx.py config.yaml
and that runs the python command, prints any output and when the command ends, disconnects from the container again.
I've omitted the -t because that instructs docker to use the terminal (console), but Jenkins does not have any console, just the -i, instructing it to connect the stdin, stdout and stderr, is good enough.
Now there is also a way to send the commands on the standard input of the bash similar to what the console would do, but I strongly recommend reading the documentation of bash before attempting that.
I have a shell script called kill.sh that helps me restart a python script I've written. I normally use pkill -f main.py to kill my forever-running python script. However, when I wrote it into a shell script it does not work.
My script
pkill -f main.py
ps aux | grep main.py # Still shows the process running.
While just executing pkill -f main.py in bash command line works as expected. Why is this?
This is not a satisfactory answer, as I cannot find out the root cause of why pkill -f does not work in a script. I ended up using a systemd Service file to manage my python process. Here's an example fyi.
[Unit]
Description=Service Name
[Service]
Environment=PYTHONUNBUFFERED=1
ExecStart=/path/to/python /path/to/python/script.py
Restart=on-failure
RestartSec=5s
WorkingDirectory=/python/project/dir/
Name the file main.service and place it in /lib/systemd/system/
Running the service systemctl start main.service
Stop the service systemctl stop main.service
Restart the service systemctl restart main.service
Show status and output systemctl status main.service -l
Now I don't have to worry about multiple processes running. If the program dies it'll even restart.
I have a Python-based ROS2 node running inside a Docker container and I am trying to handle the graceful shutdown of the node by capturing the SIGTERM/SIGINT signals and/or by catching the KeyboardInterrupt exception.
The problem is when I run the node in a container using docker-compose. I cannot seem to catch the "moment" when the container is being stopped/killed. I've explicitly added the STOPSIGNAL in the Dockerfile and the stop_signal in the docker-compose file.
Here is a sample of the node code:
import signal
import sys
import rclpy
def stop_node(*args):
print("Stopping node..")
rclpy.shutdown()
return True
def main():
rclpy.init(args=sys.argv)
print("Creating node..")
node = rclpy.create_node("mynode")
print("Running node..")
while rclpy.ok():
rclpy.spin_once(node)
if __name__ == '__main__':
try:
signal.signal(signal.SIGINT, stop_node)
signal.signal(signal.SIGTERM, stop_node)
main()
except:
stop_node()
Here is a sample Dockerfile to re-create the image:
FROM osrf/ros2:nightly
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
RUN apt-get update && \
apt-get install -y vim
WORKDIR /nodes
COPY mynode.py .
ADD run-node.sh /run-node.sh
RUN chmod +x /run-node.sh
STOPSIGNAL SIGTERM
Here is the sample docker-compose.yml:
version: '3'
services:
mynode:
container_name: mynode-container
image: mynode
entrypoint: /bin/bash -c "/run-node.sh"
privileged: true
stdin_open: false
tty: true
stop_signal: SIGTERM
Here is the run-node.sh script:
source /opt/ros/$ROS_DISTRO/setup.bash
python3 /nodes/mynode.py
When I manually run the node inside the container (using python3 mynode.py or by /run-node.sh) or when I do docker run -it mynode /bin/bash -c "/run-node.sh", I get the "Stopping node.." message. But when I do docker-compose up, I never see that message when I stop the container, by Ctrl+C or by docker-compose down.
$ docker-compose up
Creating network "ros-node_default" with the default driver
Creating mynode-container ... done
Attaching to mynode-container
mynode-container | Creating node..
mynode-container | Running node..
^CGracefully stopping... (press Ctrl+C again to force)
Stopping mynode-container ... done
$
I've tried:
moving the calls to signal.signal
using atexit instead of signal
using docker stop and docker kill --signal
I've also checked this Python inside docker container, gracefully stop question but there's no clear solution there, and I'm not sure if using ROS/rclpy makes my setup different (also, my host machine is Ubuntu 18.04, while that user was on Windows).
Is it possible to catch the stopping of the container in my stop_node method?
When your docker-compose.yml file says:
entrypoint: /bin/bash -c "/run-node.sh"
Since that's a bare string, Docker wraps it in a /bin/sh -c wrapper. So your container's main process is something like
/bin/sh -c '/bin/bash -c "/run-node.sh"'
In turn, the bash script stays running. It launches a Python script, and stays running as its parent until that script exits. (The two levels of sh -c wrappers may or may not stay running.)
The important part here is that this wrapper shell, not your script, is the main container process that receives signals, and (it turns out) won't receive SIGTERM unless it's explicitly coded to.
The most important restructuring to do here is to have your wrapper script exec the Python script. That causes it to replace the wrapper, so it becomes the main process and receives signals. If nothing else changing the last line to
exec python3 /nodes/mynode.py
will likely help.
I would go a little further here and make sure as much of this code is built into your Docker image, and try to minimize the number of explicit shell wrappers. "Do some initialization, then exec something" is an extremely common Docker pattern, and you can write this script and make it your image's entrypoint:
#!/bin/sh
# Do the setup
# ("." is the same as "source", but standard)
. "/opt/ros/$ROS_DISTRO/setup.bash"
# Run the main CMD
exec "$#"
Similarly, your main script should start with a "shebang" line like
#!/usr/bin/env python3
import ...
Your Dockerfile already contains the setup to be able to run the wrapper directly, you may need a similar RUN chmod line for the main script. But then you can add
ENTRYPOINT ["/run-node.sh"]
CMD ["/nodes/my-node.py"]
Since both scripts are executable and have the "shebang" lines you can run them directly. Using the JSON syntax keeps Docker from adding an additional shell wrapper. Since your entrypoint script will now run whatever the command is, it's easy to change that separately. For example, if you want an interactive shell that's done the environment variable setup to try to debug your container startup, you can override just the command part
docker run --rm -it mynode sh
I'm opening a docker container and running an inline bash script. The bash script runs python code but I'm not always sure what that code will be.
Because it's arbitrary code, I'd like to enable a kill switch to close this container within 30 seconds. Is there a way to do that within this code that I'm running:
docker run my/image sh -c '$(curl -ss -o python_file.py https://www.example.com); \
python python_file.py'
Basically before running the python file, I'd like to start a timer. And if that timer hits 30 seconds, I run docker kill this specific container.
I've tried the following but it's not working.
timeout 30 docker run my/image sh -c '$(curl -ss -o python_file.py https://www.example.com); \
python python_file.py'