We Keep Coding

How to really test signal handling in Python?

My code is simple:
def start():
signal(SIGINT, lambda signal, frame: raise SystemExit())
startTCPServer()
So I register my application with signal handling of SIGINT, then I start a start a TCP listener.
here are my questions:
How can I using python code to send a SIGINT signal?
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
If I run start() in my test, it will block and how can I send a signal to it?

First of, testing the signal itself is a functional or integration test, not a unit test. See What's the difference between unit, functional, acceptance, and integration tests?
You can run your Python script as a subprocess with subprocess.Popen(), then use the Popen.send_signal() method to send signals to that process, then test that the process has exited with Popen.poll().

How can I using python code to send a SIGINT signal?
You can use os.kill, which slightly misleadingly, can used to send any signal to any process by its ID. The process ID of the application/test can be found by os.getpid(), so you would have...
pid = os.getpid()
# ... other code discussed later in the answer ...
os.kill(pid, SIGINT)
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
The usual way in a test you can check that some code raises SystemExit, is with unittest.TestCase::assertRaises...
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
# ... other code discussed later in the answer ...
with self.assertRaises(SystemExit):
start.start()
If I run start() in my test, it will block and how can I send a signal to it?
This is the trick: you can start another thread which then sends a signal back to the main thread which is blocking.
Putting it all together, assuming your production start function is in start.py:
from signal import (
SIGINT,
signal,
)
import socketserver
def startTCPServer():
# Taken from https://docs.python.org/3.4/library/socketserver.html#socketserver-tcpserver-example
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
self.request.sendall(self.data.upper())
HOST, PORT = "localhost", 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
def start():
def raiseSystemExit(_, __):
raise SystemExit
signal(SIGINT, raiseSystemExit)
startTCPServer()
Then your test code could be like the following, say in test.py
import os
from signal import (
SIGINT,
)
import threading
import time
import unittest
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
pid = os.getpid()
def trigger_signal():
# You could do something more robust, e.g. wait until port is listening
time.sleep(1)
os.kill(pid, SIGINT)
thread = threading.Thread(target=trigger_signal)
thread.daemon = True
thread.start()
with self.assertRaises(SystemExit):
start.start()
if __name__ == '__main__':
unittest.main()
and run using
python test.py
The above is the same technique as in the answer at https://stackoverflow.com/a/49500820/1319998

Twisted reactor is stopped, but program doesn't end?

So I'm writing a small script to use with Deluge. Deluge uses Twisted, and I really don't have a firm grasp on how it works. Normally I'd just look up more info on it, but getting started with Twisted would take a long time and is beyond the scope of this little project. So I figured I would just ask here.
Now, I have this code. I'll try to explain the specifig parts I need help with
import base64
import processargs
from deluge.ui.client import client
from twisted.internet import reactor
from deluge.log import setupLogger
setupLogger()
options = processargs.readConfig(os.path.expanduser("~/.deluge-automator"))
d = client.connect(
host=options['host'],
port=int(options['port']),
username=options['username'],
password=options['password']
)
def start():
#other code
t = client.core.add_torrent_file(tfile,
base64.encodestring(data), None)
t.addCallback(on_torrent_added_success, tfile)
t.addErrback(on_torrent_added_fail)
def handle_stop_signal(SIGNAL, stack):
client.disconnect()
reactor.stop()
def on_torrent_added_success(result, tfile):
#other code
start()
def on_torrent_added_fail(result):
print "Add torrent failed!"
print "result: ", result
def on_connect_success(result):
#other code
start()
d.addCallback(on_connect_success)
def on_connect_fail(result):
print "Connection failed!"
print "result: ", result
d.addErrback(on_connect_fail)
signal.signal(signal.SIGTERM, handle_stop_signal)
signal.signal(signal.SIGINT, handle_stop_signal)
reactor.run()
When a torrent is successfully added, it should go back to start(), and it does, but I think it loses the reactor or something. Because now whenever it recieves a SIGTERM or SIGINT, the reactor closes, but doesn't quit the program:
± % python2 main.py
Connection was successful!
result: 10
^C^CConnection failed!
result: [Failure instance: Traceback: <class 'twisted.internet.error.ReactorNotRunning'>: Can't stop reactor that isn't running.
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:551:_runCallbacks
/usr/lib/python2.7/site-packages/deluge/ui/client.py:412:__on_login
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:368:callback
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:464:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:551:_runCallbacks
main.py:70:on_connect_success
main.py:32:start
main.py:49:handle_stop_signal
/usr/lib/python2.7/site-packages/twisted/internet/base.py:577:stop
]
So the reactor gets stopped, but it doesn't quit the program. I have to keyboard interrupt twice. Once to stop the reactor, and a second time to throw the error. Is there a certain way to set up a loop like this?

reactor handles sigint, sigterm itself (there might be a parameter of reactor.run() that disables that). Install reactor.addSystemEventTrigger('before', 'shutdown', client.disconnect) instead.
See twisted: catch keyboardinterrupt and shutdown properly.

How to gracefully exit application started with twistd?

I have a jabber client that is reading from its stdin and posting PubSub messages. If I get EOF on stdin, I want to terminate the client.
I first tried sys.exit(), but this causes an exception and the client does not exit. I then did some searching and found out that I should call reactor.stop(), but I am unable to make this work. The following code in my client:
from twisted.internet import reactor
reactor.stop()
Results in exceptions.AttributeError: 'module' object has no attribute 'stop'
What do I need to do to cause twistd to shut my application down and exit?
EDIT 2
The original problem was caused by some symlinks messing up the module import. After fixing that problem, I get a new exception:
twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.
After the exception, twistd shuts down. I think this may be caused by the call to MyClient.loop in MyClient.connectionInitialized. Perhaps I need to defer the call until later?
EDIT
Here's the .tac file for my client
import sys
from twisted.application import service
from twisted.words.protocols.jabber.jid import JID
from myApp.clients import MyClient
clientJID = JID('client#example.com')
serverJID = JID('pubsub.example.com')
password = 'secret'
application = service.Application('XMPP client')
xmppClient = client.XMPPClient(clientJID, password)
xmppClient.logTraffic = True
xmppClient.setServiceParent(application)
handler = MyClient(clientJID, serverJID, sys.stdin)
handler.setHandlerParent(xmppClient)
Which I'm invoking with
twistd -noy sentry/myclient.tac < input.txt
Here's the code for MyClient:
import os
import sys
import time
from datetime import datetime
from wokkel.pubsub import PubSubClient
class MyClient(PubSubClient):
def __init__(self, entity, server, file, sender=None):
self.entity = entity
self.server = server
self.sender = sender
self.file = file
def loop(self):
while True:
line = self.file.readline()
if line:
print line
else:
from twisted.internet import reactor
reactor.stop()
def connectionInitialized(self):
self.loop()

from twisted.internet import reactor
reactor.stop()
that should work. The fact that it doesn't means something else is wrong on your application. I can't figure out what's wrong from the information you provided.
Can you provide more (all) of the code?
EDIT:
Ok, now the problem is that you don't stop your own while True loop, so it will keep looping and eventually stop the reactor again.
Try this:
from twisted.internet import reactor
reactor.stop()
return
Now, I suspect your loop isn't very good thing for a event-driven framework. While you're just printing lines, it is fine, but depending on what you want to really do (I suspect you'll do more than just print lines) you'll have to refactor that loop to work with events.

Use reactor.callFromThread(reactor.stop) instead of reactor.stop. This should solve the issue.

I used to do this way (in sigint handler of a non-twistd called application):
reactor.removeAll()
reactor.iterate()
reactor.stop()
I'm not 100% sure it is the right way, but twisted is happy
the same application started in a tac is handled directly by twistd signal handler, I've found this question because I have some rpc client requests that I would to wait for and handle result before exiting and looks like twistd is just killing the reactor without letting the call finish

Python : How to close a UDP socket while is waiting for data in recv?

let's consider this code in python:
import socket
import threading
import sys
import select
class UDPServer:
def __init__(self):
self.s=None
self.t=None
def start(self,port=8888):
if not self.s:
self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.bind(("",port))
self.t=threading.Thread(target=self.run)
self.t.start()
def stop(self):
if self.s:
self.s.close()
self.t.join()
self.t=None
def run(self):
while True:
try:
#receive data
data,addr=self.s.recvfrom(1024)
self.onPacket(addr,data)
except:
break
self.s=None
def onPacket(self,addr,data):
print addr,data
us=UDPServer()
while True:
sys.stdout.write("UDP server> ")
cmd=sys.stdin.readline()
if cmd=="start\n":
print "starting server..."
us.start(8888)
print "done"
elif cmd=="stop\n":
print "stopping server..."
us.stop()
print "done"
elif cmd=="quit\n":
print "Quitting ..."
us.stop()
break;
print "bye bye"
It runs an interactive shell with which I can start and stop an UDP server.
The server is implemented through a class which launches a thread in which there's a infinite loop of recv/onPacket callback inside a try/except block which should detect the error and the exits from the loop.
What I expect is that when I type "stop" on the shell the socket is closed and an exception is raised by the recvfrom function because of the invalidation of the file descriptor.
Instead, it seems that recvfrom still to block the thread waiting for data even after the close call.
Why this strange behavior ?
I've always used this patter to implements an UDP server in C++ and JAVA and it always worked.
I've tried also with a "select" passing a list with the socket to the xread argument, in order to get an event of file descriptor disruption from select instead that from recvfrom, but select seems to be "insensible" to the close too.
I need to have a unique code which maintain the same behavior on Linux and Windows with python 2.5 - 2.6.
Thanks.

The usual solution is to have a pipe tell the worker thread when to die.
Create a pipe using os.pipe. This gives you a socket with both the reading and writing ends in the same program. It returns raw file descriptors, which you can use as-is (os.read and os.write) or turn into Python file objects using os.fdopen.
The worker thread waits on both the network socket and the read end of the pipe using select.select. When the pipe becomes readable, the worker thread cleans up and exits. Don't read the data, ignore it: its arrival is the message.
When the master thread wants to kill the worker, it writes a byte (any value) to the write end of the pipe. The master thread then joins the worker thread, then closes the pipe (remember to close both ends).
P.S. Closing an in-use socket is a bad idea in a multi-threaded program. The Linux close(2) manpage says:
It is probably unwise to close file descriptors while they may be in use by system calls in other threads in the same process. Since a file descriptor may be re-used, there are some obscure race conditions that may cause unintended side effects.
So it's lucky your first approach did not work!

This is not java. Good hints:
Don't use threads. Use asynchronous IO.
Use a higher level networking framework
Here's an example using twisted:
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor, stdio
from twisted.protocols.basic import LineReceiver
class UDPLogger(DatagramProtocol):
def datagramReceived(self, data, (host, port)):
print "received %r from %s:%d" % (data, host, port)
class ConsoleCommands(LineReceiver):
delimiter = '\n'
prompt_string = 'myserver> '
def connectionMade(self):
self.sendLine('My Server Admin Console!')
self.transport.write(self.prompt_string)
def lineReceived(self, line):
line = line.strip()
if line:
if line == 'quit':
reactor.stop()
elif line == 'start':
reactor.listenUDP(8888, UDPLogger())
self.sendLine('listening on udp 8888')
else:
self.sendLine('Unknown command: %r' % (line,))
self.transport.write(self.prompt_string)
stdio.StandardIO(ConsoleCommands())
reactor.run()
Example session:
My Server Admin Console!
myserver> foo
Unknown command: 'foo'
myserver> start
listening on udp 8888
myserver> quit

Running SimpleXMLRPCServer in separate thread and shutting down

I have a class that I wish to test via SimpleXMLRPCServer in python. The way I have my unit test set up is that I create a new thread, and start SimpleXMLRPCServer in that. Then I run all the test, and finally shut down.
This is my ServerThread:
class ServerThread(Thread):
running = True
def run(self):
self.server = #Creates and starts SimpleXMLRPCServer
while (self.running):
self.server.handle_request()
def stop(self):
self.running = False
self.server.server_close()
The problem is, that calling ServerThread.stop(), followed by Thread.stop() and Thread.join() will not cause the thread to stop properly if it's already waiting for a request in handle_request. And since there doesn't seem to be any interrupt or timeout mechanisms here that I can use, I am at a loss for how I can cleanly shut down the server thread.

I had the same problem and after hours of research i solved it by switching from using my own handle_request() loop to serve_forever() to start the server.
serve_forever() starts an internal loop like yours. This loop can be stopped by calling shutdown(). After stopping the loop it is possible to stop the server with server_close().
I don't know why this works and the handle_request() loop don't, but it does ;P
Here is my code:
from threading import Thread
from xmlrpc.server import SimpleXMLRPCServer
from pyWebService.server.service.WebServiceRequestHandler import WebServiceRquestHandler
class WebServiceServer(Thread):
def __init__(self, ip, port):
super(WebServiceServer, self).__init__()
self.running = True
self.server = SimpleXMLRPCServer((ip, port),requestHandler=WebServiceRquestHandler)
self.server.register_introspection_functions()
def register_function(self, function):
self.server.register_function(function)
def run(self):
self.server.serve_forever()
def stop_server(self):
self.server.shutdown()
self.server.server_close()
print("starting server")
webService = WebServiceServer("localhost", 8010)
webService.start()
print("stopping server")
webService.stop_server()
webService.join()
print("server stopped")

Two suggestions.
Suggestion One is to use a separate process instead of a separate thread.
Create a stand-alone XMLRPC server program.
Start it with subprocess.Popen().
Kill it when the test is done. In standard OS's (not Windows) the kill works nicely. In Windows, however, there's no trivial kill function, but there are recipes for this.
The other suggestion is to have a function in your XMLRPC server which causes server self-destruction. You define a function that calls sys.exit() or os.abort() or raises a similar exception that will stop the process.

This is my way. send SIGTERM to self. (Works for me)
Server code
import os
import signal
import xmlrpc.server
server = xmlrpc.server.SimpleXMLRPCServer(("0.0.0.0", 8000))
server.register_function(lambda: os.kill(os.getpid(), signal.SIGTERM), 'quit')
server.serve_forever()
Client code
import xmlrpc.client
c = xmlrpc.client.ServerProxy("http://localhost:8000")
try:
c.quit()
except ConnectionRefusedError:
pass