We Keep Coding

Twisted application ignoring a certain UNIX signal - is it possible?

Let's say we have the following situation:
kill <pid> sends SIGTERM
kill -<SIGNAL> <pid> sends <SIGNAL>
Sometimes, during development, I need to kill my application and restart it, at the moment - using the first type of command. But, if I have a production console opened, I have a chance to kill our production (let's say, I've forgotten about it - THIS HAPPENED RIGHT NOW).
The solution that came into my mind is based on ignoring SIGTERM in production mode, but killing the app gracefully in development mode. This way, if, for some reason, I want to kill our prod, I'll need to specify a SIGNAL to do it, and it'll be impossible to be done accidentally.
The app is built on Twisted.
Twisted has a number useful of methods to use signals with it - for example:
reactor.addSystemEventTrigger('before', 'shutdown', shutdown_callback)
But is it possible to make it ignore a certain signal? I need only one, and I don't want to go this [reactor.run(installSignalHandlers=False)] way (some people say that it doesn't even work) - it'll require me to rewrite the whole signal handling by myself, and that's not what I'm looking for.
Thanks!

Twisted installs some signal handlers by default but the only one it really tightly integrates with is SIGCHLD on POSIX so that it can do child process management correctly.
You can just use the Python signal module to change the signal-handling behavior of any signal you want (even SIGCHLD, just be aware this will probably break reactor.spawnProcess).
Twisted itself doesn't provide any APIs for customizing signal handling behavior.

This is what I've done in the end (using twistd module with startReactor() override):
def signal_handler(signum, frame):
if signum == signal.SIGTERM:
if is_prod():
log.critical("Received SIGTERM on PRODUCTION call system, ignoring!")
else:
log.critical("Received SIGTERM on DEV call system, shutting down!")
reactor.stop()
elif any([
signum == signal.SIGQUIT,
signum == signal.SIGINT,
signum == signal.SIGILL,
signum == signal.SIGFPE,
signum == signal.SIGABRT,
signum == signal.SIGBUS,
signum == signal.SIGPIPE,
signum == signal.SIGSYS,
signum == signal.SIGSEGV,
signum == signal.SIGHUP
]):
log.critical(f"*Received {signal.Signals(signum).name}, shutting down.*")
reactor.stop()
def register_signals():
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGILL, signal_handler)
signal.signal(signal.SIGFPE, signal_handler)
signal.signal(signal.SIGABRT, signal_handler)
signal.signal(signal.SIGBUS, signal_handler)
signal.signal(signal.SIGPIPE, signal_handler)
signal.signal(signal.SIGSYS, signal_handler)
signal.signal(signal.SIGSEGV, signal_handler)
signal.signal(signal.SIGHUP, signal_handler)
# ...
class ApplicationRunner(twistd._SomeApplicationRunner):
def startReactor(self, reactor, oldstdout, oldstderr):
self._exitSignal = None
from twisted.internet import reactor
try:
reactor.run(installSignalHandlers=False)
except BaseException:
close = False
if self.config["nodaemon"]:
file = oldstdout
else:
file = open("TWISTD-CRASH.log", "a")
close = True
try:
traceback.print_exc(file=file)
file.flush()
finally:
if close:
file.close()
def createOrGetApplication(self):
return application
def run(self):
self.preApplication()
self.application = self.createOrGetApplication()
self.postApplication()
register_signals()
twistd._SomeApplicationRunner = ApplicationRunner
twistd.run()
Basically, this code gets access to the execution of the inner reactor, adds the required parameter, and takes all signal handling on itself. Not the best solution, but that's all we have now.
Known bug:
OS kills processes with SIGTERM during the restart, so if the OS triggers the process shutdown, it will send SIGTERM there, and then OS will hang. The solution is to check the following before denying the SIGTERM request:
Existing SSH connections (if there are no connections, no user could make such a mistake, so the process shutdown should proceed).
Bash history for shutdown, reboot, poweroff, and other stuff like that (Poweruser wants to shut down the server, so we should proceed with the process shut down)
Any other system-specific conditions.

SocketServer never exits

I'm starting a simple TCP server using SocketServer:
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
pass
...
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
try:
...
finally:
server.shutdown()
However, after the program ends it doesn't terminate and seems to be stuck. It doesn't respond to keyboard events (CTRL-C) and the only way to exit is to call os._exit(0) or to just close the shell window.
I've searched a bit about it but I still don't see what I'm missing: The thread is marked as daemon, and the server is shut-down at the end.
I'm running Python 2.7.9 under Windows 8.1

Ctrl-c raises KeyboardInterrupt exception and the issue is SocketServer internally catches the error but never bothers to come out. You can add an extra signal handler for signal.SIGINT & then call server.start().
Like this,
import signal
import sys
def signal_handler(signal, frame):
logger.log("DEBUG","You pressed Ctrl+C!")
server.stop()
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
server_thread.start()
signal.pause()

How to really test signal handling in Python?

My code is simple:
def start():
signal(SIGINT, lambda signal, frame: raise SystemExit())
startTCPServer()
So I register my application with signal handling of SIGINT, then I start a start a TCP listener.
here are my questions:
How can I using python code to send a SIGINT signal?
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
If I run start() in my test, it will block and how can I send a signal to it?

First of, testing the signal itself is a functional or integration test, not a unit test. See What's the difference between unit, functional, acceptance, and integration tests?
You can run your Python script as a subprocess with subprocess.Popen(), then use the Popen.send_signal() method to send signals to that process, then test that the process has exited with Popen.poll().

How can I using python code to send a SIGINT signal?
You can use os.kill, which slightly misleadingly, can used to send any signal to any process by its ID. The process ID of the application/test can be found by os.getpid(), so you would have...
pid = os.getpid()
# ... other code discussed later in the answer ...
os.kill(pid, SIGINT)
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
The usual way in a test you can check that some code raises SystemExit, is with unittest.TestCase::assertRaises...
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
# ... other code discussed later in the answer ...
with self.assertRaises(SystemExit):
start.start()
If I run start() in my test, it will block and how can I send a signal to it?
This is the trick: you can start another thread which then sends a signal back to the main thread which is blocking.
Putting it all together, assuming your production start function is in start.py:
from signal import (
SIGINT,
signal,
)
import socketserver
def startTCPServer():
# Taken from https://docs.python.org/3.4/library/socketserver.html#socketserver-tcpserver-example
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
self.request.sendall(self.data.upper())
HOST, PORT = "localhost", 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
def start():
def raiseSystemExit(_, __):
raise SystemExit
signal(SIGINT, raiseSystemExit)
startTCPServer()
Then your test code could be like the following, say in test.py
import os
from signal import (
SIGINT,
)
import threading
import time
import unittest
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
pid = os.getpid()
def trigger_signal():
# You could do something more robust, e.g. wait until port is listening
time.sleep(1)
os.kill(pid, SIGINT)
thread = threading.Thread(target=trigger_signal)
thread.daemon = True
thread.start()
with self.assertRaises(SystemExit):
start.start()
if __name__ == '__main__':
unittest.main()
and run using
python test.py
The above is the same technique as in the answer at https://stackoverflow.com/a/49500820/1319998

Signal Handling in Windows

In Windows I am trying to create a python process that waits for SIGINT signal.And when it receives SIGINT I want it to just print a message and wait for another occurrence of SIGINT.So I used signal handler.
Here is my signal_receiver.py code.
import signal, os, time
def handler(signum, frame):
print 'Yes , Received', signum
signal.signal(signal.SIGINT, handler)
print 'My process Id' , os.getpid()
while True:
print 'Waiting for signal'
time.sleep(10)
When this process running ,I just send SIGINT to this procees from some other python process using,
os.kill(pid,SIGINT).
But when the signal_receiver.py receives SIGINT it just quits the execution .But expected behavior is to print the message inside the handler function and continue execution.
Can some one please help me to solve this issue.Is it a limitation in windows ,because the same works fine in linux.
Thanks in advance.

When you press CTRL+C, the process receives a SIGINT and you are catching it correctly, because otherwise it would throw a KeyboardInterrupt error.
On Windows, when time.sleep(10) is interrupted, although you catch SIGINT, it still throws an InterruptedError. Just add a try/except statement inside time.sleep to catch this exception, for example:
import signal
import os
import time
def handler(signum, frame):
if signum == signal.SIGINT:
print('Signal received')
if __name__ == '__main__':
print('My PID: ', os.getpid())
signal.signal(signal.SIGINT, handler)
while True:
print('Waiting for signal')
try:
time.sleep(5)
except InterruptedError:
pass
Note: tested on Python3.x, it should also work on 2.x.

How do I make a Twisted application handle SIGTERM?

I have the following sample code. The code for handling SIGINT works perfectly but it does not seem to be doing the same when I send a SIGTERM. What could be wrong?
def signal_handler(signal, frame):
print 'Terminating...'
reactor.removeAll()
reactor.stop()
def run():
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
site = server.Site(stuff)
reactor.listenTCP(8080, site)
reactor.run()
Why isn't SIGTERM calling signal_handler?
Updated with the answer
I added reactor.addSystemEventTrigger('before', 'shutdown', shutdown) before running the reactor and then called reactor.sigTerm() from the shutdown method.

There can only be one handler for a particular signal. reactor.run() installs its own handler for SIGTERM that replaces yours.
Fortunately, the reactor's SIGTERM handler essentially does the same thing as yours (but more correctly).