I have a thread waiting on input, but in the event that no input is provided, I need to exit the program. How can i exit the program? in this example the exit should be triggered by keyboard ctrl+c however I would also like to do this without interaction ie via a timeout or other event.
import threading
import signal
import sys
import time
shutdown = False
def shutdownHook(sigNum, currentStackFrame):
global shutdown
print('shutdown')
shutdown = True
def readInput():
print('readInput')
print(sys.stdin.readline())
print('done reading input')
if __name__ == '__main__':
signal.signal(signal.SIGINT, shutdownHook)
signal.signal(signal.SIGTERM, shutdownHook)
inputThread = threading.Thread(name='input', target=readInput)
inputThread.start()
print('started input')
while not shutdown:
time.sleep(1)
print('waiting ' + str(shutdown))
print('current thread' + str(threading.current_thread()))
print('end of program ' + str(shutdown))
sys.exit(0)
You may use signal.alarm() to send a SIGALRM to your program after a certain amount of time (define here in second):
if __name__ == '__main__':
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, shutdownHook)
signal.alarm(5)
Here is the complete working example from the documentation:
Here is a minimal example program. It uses the alarm() function to
limit the time spent waiting to open a file; this is useful if the
file is for a serial device that may not be turned on, which would
normally cause the os.open() to hang indefinitely. The solution is to
set a 5-second alarm before opening the file; if the operation takes
too long, the alarm signal will be sent, and the handler raises an
exception.
import signal, os
def handler(signum, frame):
print('Signal handler called with signal', signum)
raise OSError("Couldn't open device!")
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)
# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)
signal.alarm(0) # Disable the alarm
As for why your program is not quitting is because quoted the doc
Python signal handlers are always executed in the main Python thread,
even if the signal was received in another thread. This means that
signals can’t be used as a means of inter-thread communication. You
can use the synchronization primitives from the threading module
instead. Besides, only the main thread is allowed to set a new signal handler.
That means your thread cannot receive no signals the way you design the program. In fact if you try to set a signal in your thread you will receive a ValueError:
ValueError: signal only works in main thread
That's why your program keeps turning after receiving a SIGTERM. Because the thread did not received the signal.
See here: Kill python thread using os for alternative solution.
Make the thread as Deamon thread, this way it will also shutdown when main thread is exited.
inputThread = threading.Thread(name='input', target=readInput)
inputThread.setDaemon(True) # add this line
inputThread.start()
Also you can add a time lapse for no activity within specified period.
time_limit_for_shutdown_in_secs = 10
secs = 0
while not shutdown:
if secs > time_limit_for_shutdown_in_secs: break
time.sleep(1)
print('waiting ' + str(shutdown))
secs += 1
print('current thread' + str(threading.current_thread()))
print('end of program ' + str(shutdown))
sys.exit(0)
Related
In my Python application, I have a function that consumes message from Amazon SQS FIFO queue.
def consume_msgs():
sqs = boto3.client('sqs',
region_name='us-east-1',
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
print('STARTING WORKER listening on {}'.format(QUEUE_URL))
while 1:
response = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=1,
WaitTimeSeconds=10,
)
messages = response.get('Messages', [])
for message in messages:
try:
print('{} > {}'.format(threading.currentThread().getName(), message.get('Body')))
body = json.loads(message.get('Body'))
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=message.get('ReceiptHandle'))
except Exception as e:
print('Exception in worker > ', e)
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=message.get('ReceiptHandle'))
time.sleep(10)
In order to scale up, I am using multi threading to process messages.
if __name__ == '__main__:
for i in range(3):
t = threading.Thread(target=consume_msgs, name='worker-%s' % i)
t.setDaemon(True)
t.start()
while True:
print('Waiting')
time.sleep(5)
The application runs as service. If I need to deploy new release, it has to be restarted. Is there a way have the threads exist gracefully when main process is being terminated? In stead of killing the threads abruptly, they finish with current message first and stop receiving the next messages.
Since your threads keep looping, you cannot just join them, but you need to signal them it's time to break out of the loop too in order to be able to do that. This docs hint might be useful:
Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
With that, I've put the following example together, which can hopefully help a bit:
from threading import Thread, Event
from time import sleep
def fce(ident, wrap_up_event):
cnt = 0
while True:
print(f"{ident}: {cnt}", wrap_up_event.is_set())
sleep(3)
cnt += 1
if wrap_up_event.is_set():
break
print(f"{ident}: Wrapped up")
if __name__ == '__main__':
wanna_exit = Event()
for i in range(3):
t = Thread(target=fce, args=(i, wanna_exit))
t.start()
sleep(5)
wanna_exit.set()
A single event instance is passed to fce which would just keep running endlessly, but when done with each iteration, before going back to the top check, if the event has been set to True. And before exiting from the script, we set this event to True from the controlling thread. Since the threads are no longer marked as daemon threads, we do not have to explicitly join them.
Depending on how exactly you want to shutdown your script, you will need to handle the incoming signal (SIGTERM perhaps) or KeyboardInterrupt exception for SIGINT. And perform your clean-up before exiting, the mechanics of which remain the same. Apart from not letting python just stop execution right away, you need to let your threads know they should not re-enter the loop and wait for them to be joined.
The SIGINT is a bit simpler, because it's exposed as a python exception and you could do for instance this for the "main" bit:
if __name__ == '__main__':
wanna_exit = Event()
for i in range(3):
t = Thread(target=fce, args=(i, wanna_exit))
t.start()
try:
while True:
sleep(5)
print('Waiting')
except KeyboardInterrupt:
pass
wanna_exit.set()
You can of course send SIGINT to a process with kill and not only from the controlling terminal.
My code is simple:
def start():
signal(SIGINT, lambda signal, frame: raise SystemExit())
startTCPServer()
So I register my application with signal handling of SIGINT, then I start a start a TCP listener.
here are my questions:
How can I using python code to send a SIGINT signal?
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
If I run start() in my test, it will block and how can I send a signal to it?
First of, testing the signal itself is a functional or integration test, not a unit test. See What's the difference between unit, functional, acceptance, and integration tests?
You can run your Python script as a subprocess with subprocess.Popen(), then use the Popen.send_signal() method to send signals to that process, then test that the process has exited with Popen.poll().
How can I using python code to send a SIGINT signal?
You can use os.kill, which slightly misleadingly, can used to send any signal to any process by its ID. The process ID of the application/test can be found by os.getpid(), so you would have...
pid = os.getpid()
# ... other code discussed later in the answer ...
os.kill(pid, SIGINT)
How can I test whether if the application receives a signal of SIGINT, it will raise a SystemExit exception?
The usual way in a test you can check that some code raises SystemExit, is with unittest.TestCase::assertRaises...
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
# ... other code discussed later in the answer ...
with self.assertRaises(SystemExit):
start.start()
If I run start() in my test, it will block and how can I send a signal to it?
This is the trick: you can start another thread which then sends a signal back to the main thread which is blocking.
Putting it all together, assuming your production start function is in start.py:
from signal import (
SIGINT,
signal,
)
import socketserver
def startTCPServer():
# Taken from https://docs.python.org/3.4/library/socketserver.html#socketserver-tcpserver-example
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
self.request.sendall(self.data.upper())
HOST, PORT = "localhost", 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
def start():
def raiseSystemExit(_, __):
raise SystemExit
signal(SIGINT, raiseSystemExit)
startTCPServer()
Then your test code could be like the following, say in test.py
import os
from signal import (
SIGINT,
)
import threading
import time
import unittest
import start
class TestStart(unittest.TestCase):
def test_signal_handling(self):
pid = os.getpid()
def trigger_signal():
# You could do something more robust, e.g. wait until port is listening
time.sleep(1)
os.kill(pid, SIGINT)
thread = threading.Thread(target=trigger_signal)
thread.daemon = True
thread.start()
with self.assertRaises(SystemExit):
start.start()
if __name__ == '__main__':
unittest.main()
and run using
python test.py
The above is the same technique as in the answer at https://stackoverflow.com/a/49500820/1319998
In Windows I am trying to create a python process that waits for SIGINT signal.And when it receives SIGINT I want it to just print a message and wait for another occurrence of SIGINT.So I used signal handler.
Here is my signal_receiver.py code.
import signal, os, time
def handler(signum, frame):
print 'Yes , Received', signum
signal.signal(signal.SIGINT, handler)
print 'My process Id' , os.getpid()
while True:
print 'Waiting for signal'
time.sleep(10)
When this process running ,I just send SIGINT to this procees from some other python process using,
os.kill(pid,SIGINT).
But when the signal_receiver.py receives SIGINT it just quits the execution .But expected behavior is to print the message inside the handler function and continue execution.
Can some one please help me to solve this issue.Is it a limitation in windows ,because the same works fine in linux.
Thanks in advance.
When you press CTRL+C, the process receives a SIGINT and you are catching it correctly, because otherwise it would throw a KeyboardInterrupt error.
On Windows, when time.sleep(10) is interrupted, although you catch SIGINT, it still throws an InterruptedError. Just add a try/except statement inside time.sleep to catch this exception, for example:
import signal
import os
import time
def handler(signum, frame):
if signum == signal.SIGINT:
print('Signal received')
if __name__ == '__main__':
print('My PID: ', os.getpid())
signal.signal(signal.SIGINT, handler)
while True:
print('Waiting for signal')
try:
time.sleep(5)
except InterruptedError:
pass
Note: tested on Python3.x, it should also work on 2.x.
I wrote a program that uses threads to keep a connection alive while the main program loops until it either has an exception or is manually closed. My program runs in 1 hour intervals and the timeout for the connection is 20 minutes, thus I spawn a thread for every connection element that exist inside of my architecture. Thus, if we have two servers to connect to it connects to both these serves and stays connected and loops through each server retrieving data.
the program I wrote works correctly, however I can't seem to find a way to handle when the program it's self throws an exception. This is to say I can't find an appropriate way to dispose of the threads when the main program excepts. When the program excepts it will just hang open because of the thread not excepting as well and it won't close correctly and will have to be closed manually.
Any suggestions on how to handle cleaning up threads on program exit?
This is my thread:
def keep_vc_alive(vcenter,credentials, api):
vm_url = str(vcenter._proxy.binding.url).split('/')[2]
while True:
try:
logging.info('staying connected %s' % str(vm_url))
vcenter.keep_session_alive()
except:
logging.info('unable to call current time of vcenter %s attempting to reconnect.' % str(vm_url))
try:
vcenter = None
connected,api_version,uuid,vcenter = vcenter_open(60, api, * credentials)
except:
logging.critical('unable to call current time of vcenter %s killing application, please have administrator restart the module.' % str(vm_url))
break
time.sleep(60*10)
Then my exception clean up code is as follows, obviously I know.stop() doesn't work, but I honestly have no idea how to do what it is im trying to do.
except Abort: # Exit without clearing the semaphore
logging.exception('ApplicationError')
try:
config_values_vc = metering_config('VSphere',['vcenter-ip','username','password','api-version'])
for k in xrange(0, len(config_values_vc['username'])): # Loop through each vcenter server
vc_thread[config_values_vc['vcenter-ip'][k]].stop()
except:
pass
#disconnect vcenter
try:
for vcenter in list_of_vc_connections:
list_of_vc_connections[vcenter].disconnect()
except:
pass
try: # Close the db is it is open (db is defined)
db.close()
except:
pass
sys.exit(1)
except SystemExit:
raise
except:
logging.exception('ApplicationError')
semaphore('ComputeLoader', False)
logging.critical('Unexpected error: %s' % sys.exc_info()[0])
raise
Instead of sleeping, wait on a threading.Event():
def keep_vc_alive(vcenter,credentials, api, event): # event is a threading.Event()
vm_url = str(vcenter._proxy.binding.url).split('/')[2]
while not event.is_set(): # If the event got set, we exit the thread
try:
logging.info('staying connected %s' % str(vm_url))
vcenter.keep_session_alive()
except:
logging.info('unable to call current time of vcenter %s attempting to reconnect.' % str(vm_url))
try:
vcenter = None
connected,api_version,uuid,vcenter = vcenter_open(60, api, * credentials)
except:
logging.critical('unable to call current time of vcenter %s killing application, please have administrator restart the module.' % str(vm_url))
break
event.wait(timeout=60*10) # Wait until the timeout expires, or the event is set.
Then, in your main thread, set the event in the exception handling code:
except Abort: # Exit without clearing the semaphore
logging.exception('ApplicationError')
event.set() # keep_alive thread will wake up, see that the event is set, and exit
The generally accepted way to stop threads in python is to use the threading.Event object.
The algorithm followed usually is something like the following:
import threading
...
threads = []
#in the main program
stop_event = threading.Event()
#create thread and store thread and stop_event together
thread = threading.Thread(target=keep_vc_alive, args=(stop_event))
threads.append((thread, stop_event))
#execute thread
thread.start()
...
#in thread (i.e. keep_vc_alive)
# check is_set in stop_event
while not stop_event.is_set():
#receive data from server, etc
...
...
#in exception handler
except Abort:
#set the stop_events
for thread, stop_event in threads:
stop_event.set()
#wait for threads to stop
while 1:
#check for any alive threads
all_finished = True
for thread in threads:
if thread.is_alive():
all_finished = False
#keep cpu down
time.sleep(1)
I want to use Redis's pubsub feature to implement comet, but pubsub doesn't have timeout, so if I use ps.listen(), it will block, even if client closes browser.
Greenlet has a timeout feature when spawn process, but I don't know how to combine them.
Flask's pseudo code:
#app.route('/')
def comet():
rc = redis.Redis()
ps = rc.pubsub()
ps.subscribe('foo')
for item in ps.listen():
if item['type'] == 'message':
return item['data']
# ps.listen() will block, so how to make it timeout after 30 s?
Because you're not threading (and I'm assuming this is intentional and in some cases wise) you must use a type of interrupt. Signals are a type of interrupt on Unix systems to allow you to return to a callback during a call that could block.
This example of a file open which will never return is in line with what you want to do. It's taken from http://docs.python.org/library/signal.html#module-signal
But a warning. Because Python uses a Global Interpreter Lock to perform OS signal handling it is subject to some stability problems. These problems should be rare normally though.
import signal, os
def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)
# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)
signal.alarm(0) # Disable the alarm
p = redis.pubsub(
ignore_subscribe_messages=True
)
p.subscribe(
DB_PREFIX + CHANEL_KEY
)
message = None
timeout = 20
stop_time = time.time() + timeout
# little hack for setting get_message polling timeout
# because redis-py have bug and first call this method processed
# without timeout
while time.time() < stop_time:
message = p.get_message(timeout=stop_time - time.time())
if message:
break
if message:
data = json.loads(message["data"])
else:
raise HTTPRequestTimeout