I have been using Pyro 3 for a little while now, with great success, but occasionally I have noticed, that when a signal such as SIGHUP or SIGINT arrives while Pyro is doing some remote communications, the process hangs, hence the question, is Pyro signal safe?
Thanks in advance.
Seems the issue here is by default Python sets up a handlers for SIGINT and SIGTERM which raise exceptions. If you therfore receive a signal while doing some Pyro comms, the exception is raised, and off it goes to look for an appropriate except clause, not finishing what it was doing, if you then try and use Pyro again, for example in the except/finally clause, you can get issues. In my case it was sending some messages from finally to a log via a queue which was proxied to another process using Pyro.
Related
I am writing a simple multithreaded chat server in Python3. Sometimes i want to interrupt another thread that is waiting in socket.recv() to allow the theread to perform cleanup tasks.
Is calling socket.shutdown(socket.SHUT_RDWR) from another thread safe?
Calling socket.shutdown from another thread is reasonably safe, and, as #kinokikuf mentioned, is an appropriate way to terminate a blocking recv() in another thread, as long as you can guarantee that the socket is not yet closed when the call to shutdown() happens.
If you cannot guarantee that the socket isn't yet closed, you may get an exception when calling shutdown or you might even terminate another unrelated connection if a race-condition happens and the socket handle is reused by the OS!
The Linux man page for the shutdown() system call doesn't explicitly state that blocking recv() calls will be terminated, only that "further receptions and transmissions will be disallowed". But in practice this seems to include blocking recv() calls on the same socket on both Unixoid systems and Windows.
Another way would be to use socket.settimeout() to circumvent the indefinite blocking problem and regularly check (e.g. with global variables) inside the thread if the connection should be closed.
Yet another reasonable way to interrupt the recv() would be to send a non-lethal signal to the thread, which will interrupt the recv() system call, but there seems to be no wrapper for the low-level pthread_kill function in Python and since Python 3.5 system calls will be automatically restarted if interrupted, which makes this option unusable.
Consider the following twisted code, using deferLater:
import random
from twisted.internet.task import deferLater
from twisted.internet import reactor
def random_exception(msg='general'):
if random.random() < 0.5:
raise Exception("Random exception with 50%% likelihood occurred in %s!" % msg)
def dolater():
random_exception('dolater')
print "it's later!"
def whoops(failure):
failure.trap(Exception)
print failure
defer = deferLater(reactor, 10, dolater)
defer.addErrback(whoops)
reactor.run()
An exception is raised during the 10 second sleep (namely a KeyboardInterrupt), however, it seems that the whoops method is never called. My assumption is that since I add the errBack after the deferred kicks off, it's never properly registered. Advice appreciated.
EDIT:
Alright, no one likes my use of the signal (not the exception) KeyboardInterrupt to show an error condition outside of the defer. I thought pretty hard about an actual exception that might occur out of the defer callback, but couldn't think of a particularly good one, most everything would be some kind of signal (or developer error), so signal handling is fine for now- but that wasn't really the heart of the question.
As I understand it, twisted's callback/errback system handles errors within the callback structure - e.g. if dolater raises an Exception of some kind. To show this, I have added an exception that could occur during dolater, to show that if the exception occurs in dolater, the errback handles the exception just fine.
My concern was if something went wrong while the reactor was just reacting normally, and the only thing I could get to go wrong was a keyboard interrupt, then I wanted whoops to fire. It appears that if I put other async events into the reactor and raise exceptions from there, then the dolater code wouldn't be affected, and I would have to add errbacks to those other async events. There is no master error handling for an entire twisted program.
So signals it is, until I can find some way to cause the reactor to fail without a signal.
If by KeyboardInterrupt you mean a signal (ctrl-c, SIGINT, etc), then what you need to do is setup a signal handler with your whoops function as the callback.
By following two previous answers from #jean-paul-calderone twisted: catch keyboardinterrupt and shutdown properly and twisted - interrupt callback via KeyboardInterrupt, I tried the following, and I think it matches your need:
def dolater():
print "it's later!"
def whoops(signal, stackframe):
print "I'm here because of signal number " + str(signal)
reactor.stop()
defer = task.deferLater(reactor, 10, dolater)
signal.signal(signal.SIGINT, whoops)
reactor.run()
That will call whoops on a SIGINT. I put a reactor.stop() in the whoops because otherwise the reactor would just keep on running, take that out if you really want it to keep running in the face of a ctrl-c.
Note: I'm not explicitly showing how to fire a err-back in the signal system because (at least to my understanding) that doesn't really map to how defer'ed should be used. I imagine if you found a way to get the defer'ed into the signal handler you could fire its errback but I think thats out of the expected use-case for twisted and may have crazy consequences.
The problem is with the actual exception you're trying to catch, specifically KeyboardInterrupt is not a subclass of Exception, thus can not be catched with it. If you'd just change the line:
failure.trap(Exception)
into:
failure.trap(KeyboardInterrupt)
it surely would catch it. More on Python's exception hierarchy can be found in the official Python docs: https://docs.python.org/2/library/exceptions.html
Twisted is a library for doing many things concurrently. The things are kept as isolated as possible (given that this is still Python, there's still global state, etc).
If you have a TCP server with two clients connect to it and one of them sends you some bad data that triggers a bug in your parser that leads to an exception being raised, that exception isn't going to cause the other client to receive any error. Nor would you want it to, I hope (at least not automatically).
Similarly, if you have a client connected to your server and you start a delayed call with deferLater and the client triggers that bug, you wouldn't want the error to be delivered to the errback on the Deferred returned by deferLater.
The idea here is that separate event sources are generally treated separately (until you write some code that glues them together somehow).
For the ten seconds that are passing between when you call deferLater and when Twisted begins to run the function you passed to deferLater, any errors that happen - including you hitting C-c on your keyboard to make Python raise a KeyboardInterrupt - aren't associated with that delayed call and they won't be delivered to the errback you attach to its Deferred.
Only exceptions raised by your dolater function will cause the errback chain of that Deferred to begin execution.
I have a long-running Python process that I want to be able to terminate in the event it gets hung-up and stops reporting progress. But I want to signal it in a way that allows it to safely cleanup, in case it hasn't completely hung-up and there's still something running that can respond to signals gracefully. What's the best order of signals to send before outright killing it?
I'm currently doing something like:
def safe_kill(pid):
for sig in [SIGTERM, SIGABRT, SIGINT, SIGKILL]:
os.kill(pid, sig)
time.sleep(1)
if not pid_exists(pid):
return
Is there a better order? I know SIGKILL bypasses the process entirely, but is there any significant difference between SIGTERM/SIGABRT/SIGINT or do they all have the same effect as far as Python is concerned?
I believe the proper way for stopping a process is SIGTERM followed by SIGKILL after a small timeout.
I don't think that SIGINT and SIGABRT are necessary if that process handles signals in a standard way. SIGINT is usually handled the same way as SIGTERM and SIGABRT is usually used by process itself on abort() (wikipedia).
Anything more complex than a small script usually implements custom SIGTERM handling to shutdown gracefully (cleaning up all the resources, etc).
For example, take a look at Upstart. It is an init daemon - it starts and stops most of processes in Ubuntu and some other distributions. The default Upstart behavior for stopping a process is to send SIGTERM, wait 5 seconds and send SIGKILL (source - upstart cookbook).
You probably should do some testing to determine the best timeout for your process.
You need to register a signal handler, as you would do in C.
import signal
import sys
def clean_termination(signal):
# perform your cleanup
sys.exit(1)
# register the signal handler for the signals specified in the question
signal.signal(signal.SIGTERM, clean_termination)
signal.signal(signal.SIGABRT, clean_termination)
Note that Python maps the SIGINT signal to a KeyboardInterrupt exception, that you can catch with a regular except statement.
I have a Python service that opens a Pika BlockingConnection in a secondary thread, and listens to incoming messages.
When the connection to RabbitMQ is broken (if RabbitMQ dies, for example), an exception is raised, as it should. I do not catch the exception and let it go unhandled.
In Windows, when the exception is thrown, the process is terminated, which is what I want. In OS X, on the other hand, only the thread dies and the process keeps running.
How can I tell OS X to behave like Windows, or Windows to behave like OS X? I don't like this inconsistent behavior.
(Obviously I can catch the exception and terminate the process myself, but I'm asking about the default behavior)
The behavior difference you describe makes me think that the thread is a daemon thread on OS X. Do you create it yourself, or it is created inside some third-party code? Can it be created as a daemonic or non-daemonic thread depending on OS?
Try checking what isDaemon() method returns (at least if threading interface is used).
What are the rules surrounding Python threads and how Unix signals are handled?
Is KeyboardInterrupt, which is triggered by SIGINT but handled internally by the Python runtime, handled differently?
First, when setting up signal handlers using the signal module, you must create them in the main thread. You will receive an exception if you try to create them in a separate thread.
Signal handlers registered via the signal.signal() function will always be called in the main thread. On architectures which support sending signals to threads, at the C level I believe the Python runtime ignores all signals on threads and has a signal handler on the main thread, which it uses to dispatch to your Python-code signal handler.
The documentation for the thread module states that the KeyboardInterrupt exception (which is ordinarily triggered by SIGINT) can be delivered to an arbitrary thread unless you have the signal module available to you, which all Unix systems should have. In that case, it's delivered to the main thread. If you're on a system without signal, you'll have to catch KeyboardInterrupt in your thread and call thread.interrupt_main() to re-raise it in the main thread.
More information can be found in the Python docs for the thread and signal modules.
From the signal documentation:
Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), pause(), setitimer() or getitimer(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.