I have the following sample code. The code for handling SIGINT works perfectly but it does not seem to be doing the same when I send a SIGTERM. What could be wrong?
def signal_handler(signal, frame):
print 'Terminating...'
reactor.removeAll()
reactor.stop()
def run():
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
site = server.Site(stuff)
reactor.listenTCP(8080, site)
reactor.run()
Why isn't SIGTERM calling signal_handler?
Updated with the answer
I added reactor.addSystemEventTrigger('before', 'shutdown', shutdown) before running the reactor and then called reactor.sigTerm() from the shutdown method.
There can only be one handler for a particular signal. reactor.run() installs its own handler for SIGTERM that replaces yours.
Fortunately, the reactor's SIGTERM handler essentially does the same thing as yours (but more correctly).
Related
Let's say we have the following situation:
kill <pid> sends SIGTERM
kill -<SIGNAL> <pid> sends <SIGNAL>
Sometimes, during development, I need to kill my application and restart it, at the moment - using the first type of command. But, if I have a production console opened, I have a chance to kill our production (let's say, I've forgotten about it - THIS HAPPENED RIGHT NOW).
The solution that came into my mind is based on ignoring SIGTERM in production mode, but killing the app gracefully in development mode. This way, if, for some reason, I want to kill our prod, I'll need to specify a SIGNAL to do it, and it'll be impossible to be done accidentally.
The app is built on Twisted.
Twisted has a number useful of methods to use signals with it - for example:
reactor.addSystemEventTrigger('before', 'shutdown', shutdown_callback)
But is it possible to make it ignore a certain signal? I need only one, and I don't want to go this [reactor.run(installSignalHandlers=False)] way (some people say that it doesn't even work) - it'll require me to rewrite the whole signal handling by myself, and that's not what I'm looking for.
Thanks!
Twisted installs some signal handlers by default but the only one it really tightly integrates with is SIGCHLD on POSIX so that it can do child process management correctly.
You can just use the Python signal module to change the signal-handling behavior of any signal you want (even SIGCHLD, just be aware this will probably break reactor.spawnProcess).
Twisted itself doesn't provide any APIs for customizing signal handling behavior.
This is what I've done in the end (using twistd module with startReactor() override):
def signal_handler(signum, frame):
if signum == signal.SIGTERM:
if is_prod():
log.critical("Received SIGTERM on PRODUCTION call system, ignoring!")
else:
log.critical("Received SIGTERM on DEV call system, shutting down!")
reactor.stop()
elif any([
signum == signal.SIGQUIT,
signum == signal.SIGINT,
signum == signal.SIGILL,
signum == signal.SIGFPE,
signum == signal.SIGABRT,
signum == signal.SIGBUS,
signum == signal.SIGPIPE,
signum == signal.SIGSYS,
signum == signal.SIGSEGV,
signum == signal.SIGHUP
]):
log.critical(f"*Received {signal.Signals(signum).name}, shutting down.*")
reactor.stop()
def register_signals():
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGILL, signal_handler)
signal.signal(signal.SIGFPE, signal_handler)
signal.signal(signal.SIGABRT, signal_handler)
signal.signal(signal.SIGBUS, signal_handler)
signal.signal(signal.SIGPIPE, signal_handler)
signal.signal(signal.SIGSYS, signal_handler)
signal.signal(signal.SIGSEGV, signal_handler)
signal.signal(signal.SIGHUP, signal_handler)
# ...
class ApplicationRunner(twistd._SomeApplicationRunner):
def startReactor(self, reactor, oldstdout, oldstderr):
self._exitSignal = None
from twisted.internet import reactor
try:
reactor.run(installSignalHandlers=False)
except BaseException:
close = False
if self.config["nodaemon"]:
file = oldstdout
else:
file = open("TWISTD-CRASH.log", "a")
close = True
try:
traceback.print_exc(file=file)
file.flush()
finally:
if close:
file.close()
def createOrGetApplication(self):
return application
def run(self):
self.preApplication()
self.application = self.createOrGetApplication()
self.postApplication()
register_signals()
twistd._SomeApplicationRunner = ApplicationRunner
twistd.run()
Basically, this code gets access to the execution of the inner reactor, adds the required parameter, and takes all signal handling on itself. Not the best solution, but that's all we have now.
Known bug:
OS kills processes with SIGTERM during the restart, so if the OS triggers the process shutdown, it will send SIGTERM there, and then OS will hang. The solution is to check the following before denying the SIGTERM request:
Existing SSH connections (if there are no connections, no user could make such a mistake, so the process shutdown should proceed).
Bash history for shutdown, reboot, poweroff, and other stuff like that (Poweruser wants to shut down the server, so we should proceed with the process shut down)
Any other system-specific conditions.
How do you kill a websocket server programmatically? I'll be deploying this server to production along side other things. And I like to build a single python script that sends a kill signal to everything. I cannot figure out how to kill this thing without a user keyboard interrupt or a kill -9.
sys.exit() didn't work.
psutil and terminate() didn't work either
import os
import psutil
current_system_pid = os.getpid()
ThisSystem = psutil.Process(current_system_pid)
ThisSystem.terminate()
I'm out of ideas. For now I'm killing it on the command line with kill -9.
When I kill it varous ways, it tend to see this message below, but the scrip is still running
2020-12-12 12:24:54-0500 [autobahn.twisted.websocket.WebSocketServerFactory] (TCP Port 8080 Closed)
2020-12-12 12:24:54-0500 [-] Stopping factory <autobahn.twisted.websocket.WebSocketServerFactory object at 0x110680f28>
autobahn install:
pip install autobahn[twisted]
Code:
from autobahn.twisted.websocket import WebSocketServerProtocol, WebSocketServerFactory
import sys
from twisted.python import log
from twisted.internet import reactor
class MyServerProtocol(WebSocketServerProtocol):
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
def onMessage(self, payload, isBinary):
print("Text message received: {0}".format(payload.decode('utf8')))
# echo back message verbatim
# self.sendMessage(payload, isBinary)
def onClose(self, wasClean, code, reason):
print("WebSocket connection closed: {0}".format(reason))
def StopWebsocketServer():
PrintAndLog_FuncNameHeader("Begin")
reactor.stop()
PrintAndLog_FuncNameHeader("End")
if __name__ == '__main__':
# TODO remove the logging that came in the example
log.startLogging(sys.stdout)
factory = WebSocketServerFactory("ws://127.0.0.1:8080")
factory.protocol = MyServerProtocol
# note to self: if using putChild, the child must be bytes...
reactor.listenTCP(Port_ws, factory)
reactor.run()
Solution using #Jean-Paul Calderone's answer:
import os
import signal
os.kill(os.getpid(), signal.SIGKILL)
I have an external python script that sends a kill signal to each of my python scripts. The kill signal is simply the existence of a file that every script knows to look for. Once that kill signal appears, each script knows it has x seconds before it will be killed. This way they have a few seconds to gracefully finish something.
twisted.internet.reactor.stop() is how you cause the reactor to shut down. This is usually what results in a Twisted-based program exiting (though of course it doesn't necessarily have to, if the program does more things after the reactor shuts down - but this is uncommon).
However, it sounds like you don't want to know what Python code to run inside the process to end it. You want to know what some other process can do to your Twisted-based process to make it exit. You gave two solutions - KeyboardInterrupt and SIGKILL. You didn't mention why either of these two solutions is inappropriate. They seem fine to me.
If you're uncomfortable with SIGKILL (which you shouldn't be, after all, your program might meet an untimely demise for many reasons and you should be prepared to deal with this) then what you might have overlooked about KeyboardInterrupt is that it is merely the exception that is raised inside a Python program by the default SIGINT handler.
If you send SIGINT to a Twisted-based process then, under normal usage, this will stop the reactor and allow an orderly shutdown.
I'm starting a simple TCP server using SocketServer:
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
pass
...
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
try:
...
finally:
server.shutdown()
However, after the program ends it doesn't terminate and seems to be stuck. It doesn't respond to keyboard events (CTRL-C) and the only way to exit is to call os._exit(0) or to just close the shell window.
I've searched a bit about it but I still don't see what I'm missing: The thread is marked as daemon, and the server is shut-down at the end.
I'm running Python 2.7.9 under Windows 8.1
Ctrl-c raises KeyboardInterrupt exception and the issue is SocketServer internally catches the error but never bothers to come out. You can add an extra signal handler for signal.SIGINT & then call server.start().
Like this,
import signal
import sys
def signal_handler(signal, frame):
logger.log("DEBUG","You pressed Ctrl+C!")
server.stop()
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
server_thread.start()
signal.pause()
My code is simple:
def start():
signal(SIGINT, lambda signal, frame: raise SystemExit())
startTCPServer()
So I register my application with signal handling of SIGINT, then I start a start a TCP listener.
here are my questions:
How can I using python code to send a SIGINT signal?
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
If I run start() in my test, it will block and how can I send a signal to it?
First of, testing the signal itself is a functional or integration test, not a unit test. See What's the difference between unit, functional, acceptance, and integration tests?
You can run your Python script as a subprocess with subprocess.Popen(), then use the Popen.send_signal() method to send signals to that process, then test that the process has exited with Popen.poll().
How can I using python code to send a SIGINT signal?
You can use os.kill, which slightly misleadingly, can used to send any signal to any process by its ID. The process ID of the application/test can be found by os.getpid(), so you would have...
pid = os.getpid()
# ... other code discussed later in the answer ...
os.kill(pid, SIGINT)
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
The usual way in a test you can check that some code raises SystemExit, is with unittest.TestCase::assertRaises...
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
# ... other code discussed later in the answer ...
with self.assertRaises(SystemExit):
start.start()
If I run start() in my test, it will block and how can I send a signal to it?
This is the trick: you can start another thread which then sends a signal back to the main thread which is blocking.
Putting it all together, assuming your production start function is in start.py:
from signal import (
SIGINT,
signal,
)
import socketserver
def startTCPServer():
# Taken from https://docs.python.org/3.4/library/socketserver.html#socketserver-tcpserver-example
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
self.request.sendall(self.data.upper())
HOST, PORT = "localhost", 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
def start():
def raiseSystemExit(_, __):
raise SystemExit
signal(SIGINT, raiseSystemExit)
startTCPServer()
Then your test code could be like the following, say in test.py
import os
from signal import (
SIGINT,
)
import threading
import time
import unittest
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
pid = os.getpid()
def trigger_signal():
# You could do something more robust, e.g. wait until port is listening
time.sleep(1)
os.kill(pid, SIGINT)
thread = threading.Thread(target=trigger_signal)
thread.daemon = True
thread.start()
with self.assertRaises(SystemExit):
start.start()
if __name__ == '__main__':
unittest.main()
and run using
python test.py
The above is the same technique as in the answer at https://stackoverflow.com/a/49500820/1319998
I'm trying to make a small application using SimpleXMLRPCServer and I'm wondering how to properly exit it when receiving SIGTERM.
The reason is because I will make a start/stop script for the application later and I want it to perform various things before stopping.
My current code is:
import SimpleXMLRPCServer
import signal
import sys
if __name__ == "__main__":
print "setting up xmlrpc server"
server = SimpleXMLRPCServer.SimpleXMLRPCServer(("localhost", 8000))
def signal_handler(signum, frame):
print "received signal"
server.server_close()
# perform clean up, etc. here...
print "exiting, gracefully"
sys.exit(0)
# signals
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGHUP, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
print "serving forever"
server.serve_forever()
It is working, but I'm not quite sure that I'm doing it the "correct" way. Any thoughts or ideas?
Also, are there any other signals that I should be listening for?