We Keep Coding

Python: Unable to kill process with keyboard interrupt?

I have a decorator written as such:
import threading
from time import sleep
from functools import wraps
import sys
import os
def repeat_periodically(f):
""" Repeat wrapped function every second """
#wraps(f)
def wrap(self, *args, **kwargs):
def wrap_helper(*args, **kwargs):
try:
threading.Timer(1.0, wrap_helper).start()
f(self)
except KeyboardInterrupt:
try:
sys.exit(1)
except:
os._exit(1)
wrap_helper()
return wrap
I'm not sure if it continues to open a new thread every single time it calls itself, but regardless, I'm unable to kill the process when I hit CTRL + C. I've also added the same try-except block in the function that I've decorated:
#repeat_periodically
def get_stats(self):
try:
# log some state information
except KeyboardInterrupt:
try:
sys.exit(1)
except:
os._exit(1)
My program just continues to run and all I see in the terminal is
^C <the stuff that I am logging>
<the stuff that I am logging>
<the stuff that I am logging>
In other words, it just keeps logging, even though I'm trying to kill it with CTRL + C.
Update:
I should mention that the above process is spun up from another thread:
tasks = [
{'target': f, 'args': (arg1)},
{'target': g},
]
for task in tasks:
t = threading.Thread(**task)
t.start()
Specifically it is the second task that spins up the Timer. However, if I set t.daemon = True, the process just runs once and exits. The first task uses watchdog. I've essentially used the example code from the watchdog documentation:
def watch_for_event_file(Event):
path = sys.argv[1] if len(sys.argv) > 1 else '.'
event_handler = LoggingCreateHandler(Event)
observer = Observer()
observer.schedule(event_handler, path)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
(Sorry for all the updates)

From the Thread documentation:
The entire Python program exits when no alive non-daemon threads are left.
So making your Timer threads as daemon threads should solve your problem. So replace:
threading.Timer(1.0, wrap_helper).start()
with:
t = threading.Timer(1.0, wrap_helper)
t.daemon = True
t.start()

File download with concurrent.futures or multiprocessing - How to stop on Ctrl-C?

I've read a lot of questions on SO and elsewhere on this topic but can't get it working. Perhaps it's because I'm using Windows, I don't know.
What I'm trying to do is download a bunch of files (whose URLs are read from a CSV file) in parallel. I've tried using multiprocessing and concurrent.futures for this with no success.
The main problem is that I can't stop the program on Ctrl-C - it just keeps running. This is especially bad in the case of processes instead of threads (I used multiprocessing for that) because I have to kill each process manually every time.
Here is my current code:
import concurrent.futures
import signal
import sys
import urllib.request
class Download(object):
def __init__(self, url, filename):
self.url = url
self.filename = filename
def perform_download(download):
print('Downloading {} to {}'.format(download.url, download.filename))
return urllib.request.urlretrieve(download.url, filename=download.filename)
def main(argv):
args = parse_args(argv)
queue = []
with open(args.results_file, 'r', encoding='utf8') as results_file:
# Irrelevant CSV parsing...
queue.append(Download(url, filename))
def handle_interrupt():
print('CAUGHT SIGINT!!!!!!!!!!!!!!!!!!!11111111')
sys.exit(1)
signal.signal(signal.SIGINT, handle_interrupt)
with concurrent.futures.ThreadPoolExecutor(max_workers=args.num_jobs) as executor:
futures = {executor.submit(perform_download, d): d for d in queue}
try:
concurrent.futures.wait(futures)
except KeyboardInterrupt:
print('Interrupted')
sys.exit(1)
I'm trying to catch Ctrl-C in two different ways here but none of them works. The latter one (except KeyboardInterrupt) actually gets run but the process won't exit after calling sys.exit.
Before this I used the multiprocessing module like this:
try:
pool = multiprocessing.Pool(processes=args.num_jobs)
pool.map_async(perform_download, queue).get(1000000)
except Exception as e:
pool.close()
pool.terminate()
sys.exit(0)
So what is the proper way to add ability to terminate all worker threads or processes once you hit Ctrl-C in the terminal?
System information:
Python version: 3.6.1 32-bit
OS: Windows 10

You are catching the SIGINT signal in a signal handler and re-routing it as a SystemExit exception. This prevents the KeyboardInterrupt exception to ever reach your main loop.
Moreover, if the SystemExit is not raised in the main thread, it will just kill the child thread where it is raised.
Jesse Noller, the author of the multiprocessing library, explains how to deal with CTRL+C in a old blog post.
import signal
from multiprocessing import Pool
def initializer():
"""Ignore CTRL+C in the worker process."""
signal.signal(SIGINT, SIG_IGN)
pool = Pool(initializer=initializer)
try:
pool.map(perform_download, dowloads)
except KeyboardInterrupt:
pool.terminate()
pool.join()

I don't believe the accepted answer works under Windows, certainly not under current versions of Python (I am running 3.8.5). In fact, it won't run at all since SIGINT and SIG_IGN will be undefined (what is needed is signal.SIGINT and signal.SIG_IGN).
This is a know problem under Windows. A solution I have come up with is essentially the reverse of the accepted solution: The main process must ignore keyboard interrupts and we initialize the process pool to initially set a global flag ctrl_c_entered to False and to set this flag to True if Ctrl-C is entered. Then any multiprocessing worker function (or method) is decorated with a special decorator, handle_ctrl_c, that firsts tests the ctrl_c_entered flag and only if False does it run the worker function after re-enabling keyboard interrupts and establishing a try/catch handler for keyboard interrups. If the ctrl_c_entered flag was True or if a keyboard interrupt occurs during the execution of the worker function, the value returned is an instance of KeyboardInterrupt, which the main process can check to determine whether a Ctrl-C was entered.
Thus all submitted tasks will be allowed to start but will immediately terminate with a return value of a KeyBoardInterrupt exception and the actual worker function will never be called by the decorator function once a Ctrl-C has been entered.
import signal
from multiprocessing import Pool
from functools import wraps
import time
def handle_ctrl_c(func):
"""
Decorator function.
"""
#wraps(func)
def wrapper(*args, **kwargs):
global ctrl_c_entered
if not ctrl_c_entered:
# re-enable keyboard interrups:
signal.signal(signal.SIGINT, default_sigint_handler)
try:
return func(*args, **kwargs)
except KeyboardInterrupt:
ctrl_c_entered = True
return KeyboardInterrupt()
finally:
signal.signal(signal.SIGINT, pool_ctrl_c_handler)
else:
return KeyboardInterrupt()
return wrapper
def pool_ctrl_c_handler(*args, **kwargs):
global ctrl_c_entered
ctrl_c_entered = True
def init_pool():
# set global variable for each process in the pool:
global ctrl_c_entered
global default_sigint_handler
ctrl_c_entered = False
default_sigint_handler = signal.signal(signal.SIGINT, pool_ctrl_c_handler)
#handle_ctrl_c
def perform_download(download):
print('begin')
time.sleep(2)
print('end')
return True
if __name__ == '__main__':
signal.signal(signal.SIGINT, signal.SIG_IGN)
pool = Pool(initializer=init_pool)
results = pool.map(perform_download, range(20))
if any(map(lambda x: isinstance(x, KeyboardInterrupt), results)):
print('Ctrl-C was entered.')
print(results)

How can I end an infinite loop with socket operations inside after finishing current iteration?

I have an infinite loop in which there are operations that are mandatory to be completely executed before exiting the loop. Namely, I am using the socket library for connecting to an external device and I need to wait the read instructions to be finished before interrupting the loop.
I have tried using a signal handler (like in this question) for raising a flag when a Keyboard interrupt is detected.
Current code:
import videosensor
import signal
def signal_handler(signal, frame):
"""Raises a flag when a keyboard interrupt is raised."""
global interrupted
interrupted = True
if __name__ == '__main__':
camera = videosensor.VideoSensor(filename)
interrupted = False
signal.signal(signal.SIGINT, signal_handler)
while not interrupted:
location = camera.get_register()
#...
#More irrelevant stuff is executed.
#...
time.sleep(0.01)
#This code has to be executed after exiting while loop
camera_shutdown(camera)
In the previous code, videosensor.VideoSensor is a class containing socket operations for getting data from an external device. The get_register() method used in the main routine is the following:
def get_register(self):
"""Read the content of the specified register.
"""
#Do some stuff
value = socket.recv(2048)
return value
The problem:
I wanted the while loop to be continually executed until the user pressed a key or used the Keyboard Interrupt, but after the current iteration was finished. Instead, using the previous solution does not work as desired, as it interrupts the ongoing instruction, and if it is reading the socket, an error is raised:
/home/.../client.pyc
in read_register(self, regkey)
164 reg = self._REGISTERS[regkey]
165 self.send('r,{}\n'.format(reg))
--> 166 value = socket.recv(2048)
167 #Convert the string input into a valid value e.g. list or int
168 formatted_result = ast.literal_eval(value)
error: [Errno 4] Interrupted system
EDIT: It seems, from an answer below, that there is no way of using the Keyboard Interrupt and avoid the socket read function to be aborted. Despite there are solutions for catching the error, they don't avoid the read cancellation.
I am interested, though, in finding a way of getting a user input e.g. specific key press, that raises the flag, which will be checked at the end of the loop, without interrupting the main routine execution until this check.
EDIT2: The used OS is the Linux distribution Ubuntu 14.04

After quick SO search I found this solution for your issue
Basically, there's nothing you can do: when you send a SIGINT to your process, the socket will return a SIGINT as well. The best you can do, then, is to actively ignore the issue, by catching the socket EINTR error and going on with your loop:
import errno
try:
# do something
value = conn.recv(2048)
except socket.error as (code, msg):
if code != errno.EINTR:
raise
An alternative solution to avoid issues with C-c breaking reads, is to use parallel execution, to read your socket in a routine, and handle user input on the other:
import asyncio
async def camera_task(has_ended, filename):
camera = videosensor.VideoSensor(filename)
try:
while not has_ended.is_set():
location = camera.get_register()
#...
#More irrelevant stuff is executed.
#...
await asyncio.sleep(0.01)
finally:
#This code has to be executed after exiting while loop
camera_shutdown(camera)
async def input_task(shall_end):
while True:
i = input("Press 'q' to stop the script…")
if i == 'q':
shall_end.set()
def main():
filename = …
#
end_event = asyncio.Event()
asyncio.Task(camera_task(end_event, filename))
asyncio.Task(input_task(end_event))
asyncio.get_event_loop().run_forever()
or with threading
import threading, time
def camera_task(has_ended, filename):
camera = videosensor.VideoSensor(filename)
try:
while not has_ended.is_set():
location = camera.get_register()
#...
#More irrelevant stuff is executed.
#...
time.sleep(0.01)
finally:
#This code has to be executed after exiting while loop
camera_shutdown(camera)
def input_task(shall_end):
while True:
i = input("Press 'q' to stop the script…")
if i == 'q':
shall_end.set()
def main():
filename = …
#
end_event = threading.Event()
threads = [
threading.Thread(target=camera_task, args=(end_event, filename)),
threading.Thread(target=input_task, args=(end_event,))
]
# start threads
for thread in threads:
thread.start()
# wait for them to end
for thread in threads:
thread.join()
or with multiprocessing:
import multiprocessing, time
def camera_task(has_ended, filename):
camera = videosensor.VideoSensor(filename)
try:
while not has_ended.is_set():
location = camera.get_register()
#...
#More irrelevant stuff is executed.
#...
time.sleep(0.01)
finally:
#This code has to be executed after exiting while loop
camera_shutdown(camera)
def input_task(shall_end):
while True:
i = input("Press 'q' to stop the script…")
if i == 'q':
shall_end.set()
def main():
filename = …
#
end_event = multiprocessing.Event()
processes = [
multiprocessing.Process(target=camera_task, args=(end_event, filename)),
multiprocessing.Process(target=input_task, args=(end_event,))
]
# start processes
for process in processes:
process.start()
# wait for them to end
for process in processes:
process.join()
disclaimer: those codes are untested, and there might be some typos or little errors, but I believe the overall logic should be 👌

You created your custom signal handler but did not overide the default keyboard interrupt behaviour. Add signal.signal(signal.SIGINT, signal_handler) to your code to accomplish this:
import videosensor
import signal
# Custom signal handler
def signal_handler(signal, frame):
"""Raises a flag when a keyboard interrupt is raised."""
global interrupted
interrupted = True
# Necessary to override default keyboard interrupt
signal.signal(signal.SIGINT, signal_handler)
if __name__ == '__main__':
# Main programme

If I understand correctly, you do not want socket.recv() to be interrupted, but you do want to use signals to let the user indicate that the I/O loop should be terminated once the current I/O operation has completed.
With the assumption that you are using Python 2 on a Unix system, you can solve your problem by calling signal.siginterrupt(signal.SIGINT, False) before entering the loop. This will cause system calls to be restarted when a signal occurs rather than interrupting it and raising an exception.
In your case this means that the socket.recv() operation will be restarted after your signal handler is called and therefore get_register() will not return until a message is received on the socket. If that is what you want your code will be:
interrupted = False
old_handler = signal.signal(signal.SIGINT, signal_handler) # install signal handler
signal.siginterrupt(signal.SIGINT, False) # do not interrupt system calls
while not interrupted:
location = camera.get_register()
if location == '':
# remote connection closed
break
#...
#More irrelevant stuff is executed.
#...
time.sleep(0.01)
That's one way to do it, but it does require that your code is running on a Unix platform.
Another way, which might work on other platforms, is to handle the exception, ignore further SIGINT signals (in case the user hits interrupt again), and then perform a final socket.recv() before returning from the get_register() function:
import errno
def get_register(s):
"""Read the content of the specified register.
"""
#Do some stuff
try:
old_handler = None
return s.recv(2048)
except socket.error as exc:
if exc.errno == errno.EINTR:
old_handler = signal.signal(signal.SIGINT, signal.SIG_IGN) # ignore this signal
return s.recv(2048) # system call was interrupted, restart it
else:
raise
finally:
if old_handler is not None:
signal.signal(signal.SIGINT, old_handler) # restore handler
Signal handling can get tricky and there might be race conditions in the above that I am not aware of. Try to use siginterrupt() if possible.

Multiple thread with Autobahn, ApplicationRunner and ApplicationSession

python-running-autobahnpython-asyncio-websocket-server-in-a-separate-subproce
can-an-asyncio-event-loop-run-in-the-background-without-suspending-the-python-in
Was trying to solve my issue with this two links above but i have not.
I have the following error : RuntimeError: There is no current event loop in thread 'Thread-1'.
Here the code sample (python 3):
from autobahn.asyncio.wamp import ApplicationSession
from autobahn.asyncio.wamp import ApplicationRunner
from asyncio import coroutine
import time
import threading
class PoloniexWebsocket(ApplicationSession):
def onConnect(self):
self.join(self.config.realm)
#coroutine
def onJoin(self, details):
def on_ticker(*args):
print(args)
try:
yield from self.subscribe(on_ticker, 'ticker')
except Exception as e:
print("Could not subscribe to topic:", e)
def poloniex_worker():
runner = ApplicationRunner("wss://api.poloniex.com:443", "realm1")
runner.run(PoloniexWebsocket)
def other_worker():
while True:
print('Thank you')
time.sleep(2)
if __name__ == "__main__":
polo_worker = threading.Thread(None, poloniex_worker, None, (), {})
thank_worker = threading.Thread(None, other_worker, None, (), {})
polo_worker.start()
thank_worker.start()
polo_worker.join()
thank_worker.join()
So, my final goal is to have 2 threads launched at the start. Only one need to use ApplicationSession and ApplicationRunner. Thank you.

A separate thread must have it's own event loop. So if poloniex_worker needs to listen to a websocket, it needs its own event loop:
def poloniex_worker():
asyncio.set_event_loop(asyncio.new_event_loop())
runner = ApplicationRunner("wss://api.poloniex.com:443", "realm1")
runner.run(PoloniexWebsocket)
But if you're on a Unix machine, you will face another error if you try to do this. Autobahn asyncio uses Unix signals, but those Unix signals only work in the main thread. You can simply turn off Unix signals if you don't plan on using them. To do that, you have to go to the file where ApplicationRunner is defined. That is wamp.py in python3.5 > site-packages > autobahn > asyncio on my machine. You can comment out the signal handling section of the code like so:
# try:
# loop.add_signal_handler(signal.SIGTERM, loop.stop)
# except NotImplementedError:
# # signals are not available on Windows
# pass
All this is a lot of work. If you don't absolutely need to run your ApplicationSession in a separate thread from the main thread, it's better to just run the ApplicationSession in the main thread.

Catch Keyboard Interrupt in program that is waiting on an Event

The following program hangs the terminal such that it ignores Ctrl+C. This is rather annoying since I have to restart the terminal every time one of the threads hang.
Is there any way to catch the KeyboardInterrupt while waiting on an event?
import threading
def main():
finished_event = threading.Event()
startThread(finished_event)
finished_event.wait()#I want to stop the program here
print('done!')
def startThread(evt):
"""Start a thread that will trigger evt when it is done"""
#evt.set()
if __name__ == '__main__':
main()

If you want to avoid polling, you can use the pause() function of the signal module instead of finished_event.wait(). signal.pause() is a blocking function and gets unblocked when a signal is received by the process. In this case, when ^C is pressed, SIGINT signal unblocks the function. Note that the function does not work on Windows according to the documentation. I've tried it on Linux and it worked for me.
I came across this solution in this SO thread.

Update: On the current Python 3 finished_event.wait() works on my Ubuntu machine (starting with Python 3.2). You don't need to specify the timeout parameter, to interrupt it using Ctrl+C. You need to pass the timeout parameter on CPython 2.
Here's a complete code example:
#!/usr/bin/env python3
import threading
def f(event):
while True:
pass
# never reached, otherwise event.set() would be here
event = threading.Event()
threading.Thread(target=f, args=[event], daemon=True).start()
try:
print('Press Ctrl+C to exit')
event.wait()
except KeyboardInterrupt:
print('got Ctrl+C')
There could be bugs related to Ctrl+C. Test whether it works in your environment.
Old polling answer:
You could try to allow the interpreter to run the main thread:
while not finished_event.wait(.1): # timeout in seconds
pass
If you just want to wait until the child thread is done:
while thread.is_alive():
thread.join(.1)

You could also patch the Event.wait() function in the following manner:
def InterruptableEvent():
e = threading.Event()
def patched_wait():
while not e.is_set():
e._wait(3)
e._wait = e.wait
e.wait = patched_wait
return e
>>> event = InterruptableEvent()
>>> try:
... event.wait()
... except KeyboardInterrupt:
... print "Received KeyboardInterrupt"
...
^CReceived KeyboardInterrupt
This works because wait() with a timeout argument will raise a KeyboardInterrupt.

Based on #Pete's answer, but with subclassing and using the actual Event.wait method, just with smaller timeouts to allow handling of KeyboardInterrupts and such in between:
class InterruptableEvent(threading.Event):
def wait(self, timeout=None):
wait = super().wait # get once, use often
if timeout is None:
while not wait(0.01): pass
else:
wait(timeout)