Hello i have created a script in python to run with PM2 a process monitoring tool available in NPM, the code is taken from the accepted answer of this question and is following
import signal
import sys
import time
def signal_handler(sig, frame):
print('You pressed Ctrl+C!')
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print('Press Ctrl+C to exit')
while True:
time.sleep(0.2)
NOTE: signal.pause() is not available for windows so i used infinite loop as an alternative (code blocking)
Now coming to the problem when I run the script manually from the CMD e.g., py test.py and then after pressing CTRL + C it captures the SIGINT perfectly, but when the same script is executed with PM2 e.g., pm2 start "py test.py" --name SigTestApp then stopping the PM2 process by typing pm2 stop SigTestApp simply kills the app and SIGINT is not being detected by the script
e.g., no message is shown "You pressed Ctrl+C!"
Related
I made a simple python webserver based on this example, which runs inside Docker
FROM python:3-alpine
WORKDIR /app
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh
COPY src src
CMD ["python", "/app/src/api.py"]
ENTRYPOINT ["/app/entrypoint.sh"]
Entrypoint:
#!/bin/sh
echo starting entrypoint
set -x
exec "$#"
Stopping the container took very long, altough the exec statement with the JSON array syntax should pass it to the python process. I assumed a problem with SIGTERM no being passed to the container. I added the following to my api.pyscript to detect SIGTERM
def terminate(signal,frame):
print("TERMINATING")
if __name__ == "__main__":
signal.signal(signal.SIGTERM, terminate)
webServer = HTTPServer((hostName, serverPort), MyServer)
print("Server started http://%s:%s" % (hostName, serverPort))
webServer.serve_forever()
Executed without Docker python3 api/src/api.py, I tried
kill -15 $(ps -guaxf | grep python | grep -v grep | awk '{print $2}')
to send SIGTERM (15 is the number code of it). The script prints TERMINATING, so my event handler works. Now I run the Docker container using docker-compose and press CTRL + C. Docker says gracefully stopping... (press Ctrl+C again to force) but doesn't print my terminating message from the event handler.
I also tried to run docker-compose in detached mode, then run docker-compose kill -s SIGTERM api and view the logs. Still no message from the event handler.
Since the script runs as pid 1 as desired and setting init: true in docker-compose.yml doesn't seem to change anything, I took a deeper drive in this topic. This leads me figuring out multiple mistakes I did:
Logging
The approach of printing a message when SIGTERM is catched was designed as simple test case to see if this basically works before I care about stopping the server. But I noticed that no message appears for two reasons:
Output buffering
When running a long term process in python like the HTTP server (or any while True loop for example), there is no output displayed when starting the container attached with docker-compose up (no -d flag). To receive live logs, we need to start python with the -u flag or set the env variable PYTHONUNBUFFERED=TRUE.
No log piping after stop
But the main problem was not the output buffering (this is just a notice since I wonder why there was no log output from the container). When canceling the container, docker-compose stops piping logs to the console. This means that from a logical perspective it can't display anything that happens AFTER CTRL + C is pressed.
To fetch those logs, we need to wait until docker-compose has stopped the container and run docker-compose logs. It will print all, including those generated after CTRL + C is pressed. Using docker-compose logs I found out that SIGTERM is passed to the container and my event handler works.
Stopping the webserver
With those knowledge I tried to stop the webserver instance. First this doesn't work because it's not enough to just call webServer.server_close(). Its required to exit explicitely after any cleanup work is done like this:
def terminate(signal,frame):
print("Start Terminating: %s" % datetime.now())
webServer.server_close()
sys.exit(0)
When sys.exit() is not called, the process keeps running which results in ~10s waiting time before Docker kills it.
Full working example
Here a demo script that implement everything I've learned:
from http.server import BaseHTTPRequestHandler, HTTPServer
import signal
from datetime import datetime
import sys, os
hostName = "0.0.0.0"
serverPort = 80
class MyServer(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type", "text/html")
self.end_headers()
self.wfile.write(bytes("Hello from Python Webserver", "utf-8"))
webServer = None
def terminate(signal,frame):
print("Start Terminating: %s" % datetime.now())
webServer.server_close()
sys.exit(0)
if __name__ == "__main__":
signal.signal(signal.SIGTERM, terminate)
webServer = HTTPServer((hostName, serverPort), MyServer)
print("Server started http://%s:%s with pid %i" % ("0.0.0.0", 80, os.getpid()))
webServer.serve_forever()
Running in a container, it could be stopped very fast without waiting for Docker to kill the process:
$ docker-compose up --build -d
$ time docker-compose down
Stopping python-test_app_1 ... done
Removing python-test_app_1 ... done
Removing network python-test_default
real 0m1,063s
user 0m0,424s
sys 0m0,077s
Docker runs your application, per default, in foreground, so, as PID 1, this said, the process with the PID 1 as a special meaning and specific protections in Linux.
This is highlighted in docker run documentation:
Note
A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. As a result, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
Source: https://docs.docker.com/engine/reference/run/#foreground
In order to fix this, you can run the container, in a single container mode, with the flag --init of docker run:
You can use the --init flag to indicate that an init process should be used as the PID 1 in the container. Specifying an init process ensures the usual responsibilities of an init system, such as reaping zombie processes, are performed inside the created container.
Source: https://docs.docker.com/engine/reference/run/#specify-an-init-process
The same configuration is possible in docker-compose, simply by specifying init: true on the container.
An example would be:
version: "3.8"
services:
web:
image: alpine:latest
init: true
Source: https://docs.docker.com/compose/compose-file/#init
This question already has an answer here:
Capturing SIGINT using KeyboardInterrupt exception works in terminal, not in script
(1 answer)
Closed 5 years ago.
I'm trying to:
launch a background process (a python script)
run some bash commands
Then send control-C to shut down the background process once the foreground tasks are finished
Minimal example of the what I've tried - Python test.py:
import sys
try:
print("Running")
while True:
pass
except KeyboardInterrupt:
print("Escape!")
Bash test.sh:
#!/bin/bash
python3 ./test.py &
pid=$!
# ... do something here ...
sleep 2
# Send an interrupt to the background process
# and wait for it to finish cleanly
echo "Shutdown"
kill -SIGINT $pid
wait
result=$?
echo $result
exit $result
But the bash script seems to be hanging on the wait and the SIGINT signal is not being sent to the python process.
I'm using Mac OS X, and am looking for a solution that works for bash on linux + mac.
Edit: Bash was sending interrupts but Python was not capturing them when being run as a background job. Fixed by adding the following to the Python script:
import signal
signal.signal(signal.SIGINT, signal.default_int_handler)
The point is SIGINT is used to terminate foreground process. You should directly use kill $pid to terminate background process.
BTW, kill $pid is equal to kill -15 $pid or kill -SIGTERM $pid.
Update
You can use signal module to deal with this situation.
import signal
import sys
def handle(signum, frame):
sys.exit(0)
signal.signal(signal.SIGINT, handle)
print("Running")
while True:
pass
I'm trying to catch SIGINT (or keyboard interrupt) in Python 2.7 program. This is how my Python test script test looks:
#!/usr/bin/python
import time
try:
time.sleep(100)
except KeyboardInterrupt:
pass
except:
print "error"
Next I have a shell script test.sh:
./test & pid=$!
sleep 1
kill -s 2 $pid
When I run the script with bash, or sh, or something bash test.sh, the Python process test stays running and is not killable with SIGINT. Whereas when I copy test.sh command and paste it into (bash) terminal, the Python process test shuts down.
I cannot get what's going on, which I'd like to understand. So, where is difference, and why?
This is not about how to catch SIGINT in Python! According to docs – this is the way, which should work:
Python installs a small number of signal handlers by default: SIGPIPE ... and SIGINT is translated into a KeyboardInterrupt exception
It is indeed catching KeyboardInterrupt when SIGINT is sent by kill if the program is started directly from shell, but when the program is started from bash script run on background, it seems that KeyboardInterrupt is never raised.
There is one case in which the default sigint handler is not installed at startup, and that is when the signal mask contains SIG_IGN for SIGINT at program startup. The code responsible for this can be found here.
The signal mask for ignored signals is inherited from the parent process, while handled signals are reset to SIG_DFL. So in case SIGINT was ignored the condition if (Handlers[SIGINT].func == DefaultHandler) in the source won't trigger and the default handler is not installed, python doesn't override the settings made by the parent process in this case.
So let's try to show the used signal handler in different situations:
# invocation from interactive shell
$ python -c "import signal; print(signal.getsignal(signal.SIGINT))"
<built-in function default_int_handler>
# background job in interactive shell
$ python -c "import signal; print(signal.getsignal(signal.SIGINT))" &
<built-in function default_int_handler>
# invocation in non interactive shell
$ sh -c 'python -c "import signal; print(signal.getsignal(signal.SIGINT))"'
<built-in function default_int_handler>
# background job in non-interactive shell
$ sh -c 'python -c "import signal; print(signal.getsignal(signal.SIGINT))" &'
1
So in the last example, SIGINT is set to 1 (SIG_IGN). This is the same as when you start a background job in a shell script, as those are non interactive by default (unless you use the -i option in the shebang).
So this is caused by the shell ignoring the signal when launching a background job in a non interactive shell session, not by python directly. At least bash and dash behave this way, I've not tried other shells.
There are two options to deal with this situation:
manually install the default signal handler:
import signal
signal.signal(signal.SIGINT, signal.default_int_handler)
add the -i option to the shebang of the shell script, e.g:
#!/bin/sh -i
edit: this behaviour is documented in the bash manual:
SIGNALS
...
When job control is not in effect, asynchronous commands ignore SIGINT and SIGQUIT in addition to these inherited handlers.
which applies to non-interactive shells as they have job control disabled by default, and is actually specified in POSIX: Shell Command Language
On a Raspberry Pi 2 running Jessie I have two displays, a (default) HDMI display, and an LCD touch screen (which requires a couple of SDL-related variables to be set using os.environ).
I have two pygame programs, lcd.py and hdmi.py which, when run from separate SSH terminals, coexist nicely, the lcd.py dsiplays a few buttons, the hdmi.py displays a slideshow on the attached HDMI display.
If I run them separately in two SSH terminals (as the 'pi' user, using sudo python PROGRAM), the lcd.py displays to the LCD and the hdmi.py displays a slideshow to the HDMI screen.
However, I cannot figure out how to have lcd.py invoke the hdmi.py program as a completely independent process (so it has its own env variables, and can drive the HDMI display independently of the parent process driving the LCD display).
The lcd.py program has a button that when touched invokes the routine startSlideShow()
However, the various things I tried in lcd.py startSlideShow() to launch hdmi.py all fail:
def startSlideShow():
# when running in SSH the correct command is
# sudo python /home/pi/Desktop/code/hdmi.py
# or sudo /usr/bin/python /home/pi/Desktop/code/hdmi.py
# tried various ways of invoking hdmi.py via
# os.fork(), os.spawnl(), subprocess.Popen()
WHAT GOES HERE?
No ongoing inter-process communication is needed. Other than when the lcd.py program needs to "launch" the hdmi.py program, they do not need to communicate, and it does not really matter to me whether or not terminating lcd terminates the hdmi.py program.
Things I tried in startSlideShow() that do not work:
cmd = "sudo /usr/bin/python /home/pi/Desktop/code/hdmi.py"
pid = os.spawnl(os.P_NOWAIT, cmd)
# lcd.py keeps running, but creates a zombie process [python]<defunct> instead of running hdmi.py
And
cmd = "sudo /usr/bin/python /home/pi/Desktop/code/hdmi.py"
pid = os.spawnl(os.P_DETACH, cmd)
# lcd.py exits with error AttributeError: 'module' object has no attribute 'P_DETACH'
And
cmd = "sudo /usr/bin/python /home/pi/Desktop/code/hdmi.py"
pid = os.spawnl(os.P_WAIT, cmd)
# no error message, returns a pid of 127, but does nothing, and no such process exists when I run `ps aux` in another SSH terminal
And
cmd = ["/usr/bin/python", "/home/pi/Desktop/code/hdmi.py" ]
pid = subprocess.Popen(cmd, stdout=subprocess.PIPE).pid # call subprocess
# runs the hdmi.py program, but output goes to LCD not HDMI
# (the 2 programs alternately take over the same screen)
And
cmd = ["/usr/bin/python", "/home/pi/Desktop/code/hdmi.py" ]
pid = subprocess.Popen(cmd).pid
# as above, both programs run on same display, which flashes between the two programs
And
pid = os.spawnlp(os.P_NOWAIT, "/usr/bin/python", "/home/pi/Desktop/code/hdmi.py")
# invokes python interpreter, get >>> on SSH terminal
And
pid = os.spawnlp(os.P_NOWAIT, "/usr/bin/python /home/pi/Desktop/code/hdmi.py")
# creates zombie process [python] <defunct>
In order to give the subprocess a different value for environment variables than the parent process you can use the env argument to the POpen constructor and supply the values you want. This should allow you, for example, to supply a different DISPLAY value.
I am starting my script locally via:
sudo python run.py remote
This script happens to also open a subprocess (if that matters)
webcam = subprocess.Popen('avconv -f video4linux2 -s 320x240 -r 20 -i /dev/video0 -an -metadata title="OfficeBot" -f flv rtmp://6f7528a4.fme.bambuser.com/b-fme/xxx', shell = True)
I want to know how to terminate this script when I SSH in.
I understand I can do:
sudo pkill -f "python run.py remote"
or use:
ps -f -C python
to find the process ID and kill it that way.
However none of these gracefully kill the process, I want to able to trigger the equilivent of CTRL/CMD C to register an exit command (I do lots of things on shutdown that aren't triggered when the process is simply killed).
Thank you!
You should use "signals" for it:
http://docs.python.org/2/library/signal.html
Example:
import signal, os
def handler(signum, frame):
print 'Signal handler called with signal', signum
signal.signal(signal.SIGINT, handler)
#do your stuff
then in terminal:
kill -INT $PID
or ctrl+c if your script is active in current shell
http://en.wikipedia.org/wiki/Unix_signal
also this might be useful:
How do you create a daemon in Python?
You can use signals for communicating with your process. If you want to emulate CTRL-C the signal is SIGINT (which you can raise by kill -INT and process id. You can also modify the behavior for SIGTERM which would make your program shut down cleanly under a broader range of circumstances.