We Keep Coding

Finish loop before exiting with a KeyboardInterrupt

I have some program logic that works as follows:
for i in range(10**6):
foo(i)
print("foo executed with parameter", i)
bar(i)
print("bar executed with parameter", i)
The problem arises when I want to interrupt the loop with Ctrl+C to raise a KeyboardInterrupt.
I make sure that the function bar(i) always runs after the function foo(i); i.e. the loop cannot be interrupted between the two function calls (or, if it is, I want to complete the loop before exiting). If the interrupt is received while one of the functions is being executed, I want both to finish before exiting.
How is this achievable (with a try...except statement or otherwise) in Python 3?

Thank you to Omer Ben Haim for providing an answer in the comments.
Indeed, the SIGINT signal can be captured using the signal module. Here is some proof-of-concept code that demonstrates this:
import signal
import time
stop = False
def handler(sig, frame):
global stop
stop = True
signal.signal(signal.SIGINT, handler)
############
i = 0
while i<10**6 and not stop:
print("Part 1:", i)
time.sleep(0.5)
print("Part 2:", i)
time.sleep(0.5)
i += 1

This could be done rather easily using a flag.
When a KeyboardInterrupt is raised(E.g. Ctrl+c is pressed), you capture it and set the quit_loop flag to True.
When we finish what we need to do, we check if we need to quit the loop and if we do then we break.
If you want to fully exit the program you can replace break with exit().
quit_loop = False
for i in range(10):
try:
foo()
except KeyboardInterrupt:
quit_loop = True
bar()
if quit_loop:
break

How can I elegantly organize this multithreaded python code?

I am working on a Python service that subscribes real-time streaming data from one messaging broker and publishes to another broker, in some situations I also need to get snapshot data from other data source on network disconnection or system recovery. While the streaming data comes from one thread, and some service events happen in another thread, I decided to create a data processing thread to just pop the queue one by one. I got it to work but later I tried to keep the snapshot fetching logic in a separate thread and that's where things get messy.
I know this is a long question with a lot of specific nuances but I tried to make the example here as clear as I can.
So here is what the 1st attempt looks like, and it works well:
import queue
import threading
def process_data(data_queue, data_store):
# data_store is my internal cache data structure.
# so for simplicity and demonstration purpose, I assume the following:
# if its type is dict, it's snapshot data
# if its type is tuple, it's a key/value pair and that's an incremental update data
# if it is -1, we terminate the queue processing
# if it is -2, we need to retrieve a snapshot
while True:
x = data_queue.get()
if isinstance(x, dict):
data_store.on_snapshot(x)
elif isinstance(x, tuple):
k, v = x
data_store.on_update(k, v)
elif isinstance(x, int):
if x == -1:
data_queue.task_done()
break
elif x == -2:
get_snapshot() # this is potentially a long blocking call
else:
print('unknown int', x)
else:
print('unknown data', x)
data_queue.task_done()
if __name__ == '__main__':
data_store = DataStore()
data_queue = queue.Queue()
# start other threads that write data to the queue
start_data_writer1(data_queue)
start_data_writer2(data_queue)
start_thread_for_some_event(data_queue) # may put -2 in the queue for snapshot
process_thread = threading.Thread(
target=process_data,
args=(data_queue, data_store))
process_thread.start()
data_queue.put(-2) # signal a snapshot fetching
do_something_else()
try:
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print('terminating...')
finally:
# to break out of the infinite loop in process_data()
data_queue.put(-1)
process_thread.join()
data_queue.join()
This way works however I don't particularly like the fact that I am calling get_snapshot() function in the processing thread. I thought the idea of the processing thread is to be busy at popping data off the queue all the time, unless there's nothing to pop. In the above implementation, during the get_snapshot call, it is possible that queue could be built up due to other writer threads.
So I tried something else and I also wanted to be able to exit the program gracefully. That's where things get really ugly. I created a new thread for occasionally fetching the snapshot and used condition object for thread communication. This is what I did on top of the existing code:
snapshot_lock = threading.Lock()
cond = threading.Condition(snapshot_lock)
need_for_snapshot = False # used to trigger snapshots
keep_snapshot_thread = True # flag if the snapshot thread is done
# then I need to add this new function to run snapshot fetching
def fetch_snapshot(data_queue):
global need_for_snapshot
global keep_snapshot_thread
def _need_snapshot():
global need_for_snapshot
return need_for_snapshot
while True:
with cond:
cond.wait_for(_need_snapshot)
if not keep_snapshot_thread:
break
data_queue.put(get_snapshot()) # the long blocking function
need_for_snapshot = False
# in process_data() function, everything stays the same except for the `if` statement for handling `x == 2`
def process_data(data_queue, data_store):
global need_for_snapshot
while True:
x = data_queue.get()
# omitting some old code
elif isinstance(x, int):
if x == -1:
data_queue.task_done()
break
elif x == -2:
**with cond:
need_for_snapshot = True
cond.notify()**
# more code omitted
if __name__ == '__main__':
# same code as before except for the finally part
try:
# start other threads...omitting some code
# when a snapshot is needed in these threads
# do the following
# with cond:
# need_for_snapshot = True
# cond.notify()
# start snapshot worker thread
snapshot_thread = threading.Thread(
target=fetch_snapshot, args=(data_queue,))
process_thread = threading.Thread(
target=process_data,
args=(data_queue, data_store))
snapshot_thread.start()
process_thread.start()
data_queue.put(-2) # signal fetching a snapshot
# omitting more code here...
finally:
keep_snapshot_thread = False
# we don't technically need to trigger another snapshot now
# but the code below is to unblock the cond.wait_for() part
# since keep_snapshot_thread flag is just flipped, we can use
# it to break out of the infinite loop in fetch_snapshot thread.
# This is the part that I feel hacky...
with cond:
need_for_snapshot = True
cond.notify()
snapshot_t.join()
data_queue.put(-1) # signal the termination of process_thread
process_t.join()
data_queue.join()
I think I got this to work, especially that the program can exit gracefully when I hit ctrl-c but it is so ugly and tricky that I had to play with it quick a bit to get it to work correctly.
Is there some way I can write it more elegantly? Is there some sort of pattern that we generally use to solve this type of problem? Thank you so much for your help.

The standard technique for handling multiple producers and multiple consumers is to use an Event is_done and a joinable Queue work.
The worker queues do nothing but:
while not event.is_set():
try:
job = work.get(timeout=5)
except Empty:
continue
handle the job
work.task_done()
Your main worker does the following:
start the jobs that produce work
wait for them to be done
work.join() # wait for the queue to be empty
event.set() # tell the workers they can exit
perform any cleanup necessary
Note that the goal is to decouple the workers and the producers as much as possible. Trying to create complicated logic tying them together is almost certain to produce race cases.
An alternative would be to create a sentinel object like "END" indicating that everything is done. Once all the producers are done, the main thread would push a number of sentinel objects equal to the number of workers onto the work queue, and the call work.join(). Each worker thread would call work.get() inside a loop and exit when it saw the sentinel. Remember to call work.task_done() on the sentinel, too!
Again. You want simple logic and to use the tools provided by multithreading.

What is the best way to user-input data while in a loop?

I have a Python program running a (nested) loop which will run for a fairly long time, and I want the user to be able to pause and/or abort it with just pressing p and c, respectively.
I am running this in an IPython console, so I don't really have access to msvctr.getch and I kinda want to keep it platform independent.
Obviously, input() blocks, which is exactly what I do not want. So I tried threading, which works when used as intended, but when hitting CTRLC the thread does not stop. This is likely because any legitimate method to stop the thread (atexit, global variable or lambda stop_thread) isn't executed because the thread blocks.
import threading
import queue
q = queue.SimpleQueue()
stop_thread = False
def handle_input(q, stopped):
s = ''
while not stopped():
s = input()
q.put(s)
thread = threading.Thread(target=handle_input,
args=[q, lambda: stop_thread])
thread.start()
for i in range(very_long_time):
#Do something time consuming
if not q.empty():
s = q.get_nowait()
if 'p' in s:
print('Paused...', end='\r')
s = s.replace('p', '')
while True:
if not q.empty():
s += q.get_nowait()
if 'p' in s or 'c' in s:
s = s.replace('p', '')
break
time.sleep(0.5)
if 'c' in s:
print('\rAborted training loop...' + ' '*50, end='\r')
s = s.replace('c', '')
stop_thread = True
# Another method of stopping the thread
# thread.__getattribute__('_tstate_lock').release()
# thread._stop()
# thread.join()
break
This works in principle, but breaks when interrupting.
The thread does not seem to stop, which poses a problem when running this again in the same console, because it does not even ask for user input then.
Additionally, this prints my 'c' or 'p' and a newline, which I can't get rid of, because IPython doesn't allow all ANSI escapes.
Is there a fix to my method, or even better, a cleaner alternative?

You can try using the keyboard module, which (among other things) lets you bind event hooks to keyboard presses.
In this case, I would create a set of global variables/flags (say, paused and abort), initially set to False, and then make some hotkeys for p and c respectively to toggle them:
paused = False
abort = False
def toggle_paused():
global paused
paused = not paused
def trigger_abort():
abort = True
keyboard.add_hotkey('p', toggle_paused())
keyboard.add_hotkey('c', trigger_abort())
And then change your loop to check for paused and abort on every iteration (assuming, that is, that each iteration is fairly quick). What you're already doing would more-or-less work - just remove the queues and threading stuff you've already set up (IIRC keyboard's events run on their own threads anyway), de-indent the if conditions, and change the conditions to if paused: and if abort: respectively.
You can also lace the rest of your code with things that look for pause or abort flags, so that your program can gracefully pause or exit at a convenient time for it. You can also extend the toggle_paused() and trigger_abort() to do whatever you need them to (e.g. have trigger_abort() print "Trying to abort program (kill me if I'm not done in 5 seconds)" or something.
Although, as #Tomerikoo suggested in a comment, creating the threat with the daemon=True option is the best answer, if it's possible with the way your program is designed. If this is all your program does then using daemon threads wouldn't work, because your program would just quit immediately, but if this is a background operation then you can use a daemon thread to put it in the background where it won't obstruct the rest of the user's experience.

Python, anticipating an endless loop

I have code that can potentially have an endless loop. I would like to be able to stop this in code, is there somethng that can do that? Some thing like:
for i to range(0,5):
timer = on
run(x)
#while run is running
if timer.time > 30:
exit run(x)
print("out of time ")
print(timer.time)
So the output could be: 3, 2, out of time 30, 2, out of time 30
I'm afraid this might require threading, which I have attempted in C and Java, for school, but never python.

A few options:
If you can modify run's "endless" loop, make it check for the time. This is probably the easiest.
If on Linux/UNIX, you can use signal.alarm with a SIGALRM handler to get interrupted after a fixed period of time.
You can spawn a separate thread and kill the other thread when time elapses. This is rather nasty, and should only be used if you're on Windows (or care about portability) and have no way to modify run.
You could spawn run into a separate process and kill the process when time expires (e.g. with multiprocessing).
Here's an example of interrupting a calculation using signal.alarm:
import signal
class TimeoutException(Exception):
pass
def alarm_handler(*args):
raise TimeoutException()
def tryrun(func, timeout=30):
oldhandler = signal.signal(signal.SIGALRM, alarm_handler)
try:
signal.alarm(timeout)
func()
except TimeoutException:
print "Timeout"
else:
print "Success"
finally:
signal.alarm(0) # disarm alarm
signal.signal(signal.SIGALRM, oldhandler)
import time
tryrun(lambda: time.sleep(10), 5) # prints Timeout
tryrun(lambda: time.sleep(2), 5) # prints Success

making a programme run indefinitely in python

Is there any way to make a function (the ones I'm thinking of are in the style of the simple ones I've made which generate the fibonnacci sequence from 0 to a point, and all the primes between two points) run indefinitely. E.g. until I press a certain key or until a time has passed, rather than until a number reaches a certain point?
Also, if it is based on time then is there any way I could just extend the time and start it going from that point again, rather than having to start again from 0? I am aware there is a time module, i just don't know much about it.

The simplest way is just to write a program with an infinite loop, and then hit control-C to stop it. Without more description it's hard to know if this works for you.
If you do it time-based, you don't need a generator. You can just have it pause for user input, something like a "Continue? [y/n]", read from stdin, and depending on what you get either exit the loop or not.

If you really want your function to run and still wants user (or system) input, you have two solutions:
multi-thread
multi-process
It will depend on how fine the interaction. If you just want to interrupt the function and don't care about the exit, then multi-process is fine.
In both cases, you can rely on some shared resources (file or shared memory for multi-thread, variable with associated mutex for multi-thread) and check for the state of that resource regularly in your function. If it is set up to tell you to quit, just do it.
Example on multi-thread:
from threading import Thread, Lock
from time import sleep
class MyFct(Thread):
def __init__(self):
Thread.__init__(self)
self.mutex = Lock()
self._quit = False
def stopped(self):
self.mutex.acquire()
val = self._quit
self.mutex.release()
return val
def stop(self):
self.mutex.acquire()
self._quit = True
self.mutex.release()
def run(self):
i = 1
j = 1
print i
print j
while True:
if self.stopped():
return
i,j = j,i+j
print j
def main_fct():
t = MyFct()
t.start()
sleep(1)
t.stop()
t.join()
print "Exited"
if __name__ == "__main__":
main_fct()

You could use a generator for this:
def finished():
"Define your exit condition here"
return ...
def count(i=0):
while not finished():
yield i
i += 1
for i in count():
print i
If you want to change the exit condition you could pass a value back into the generator function and use that value to determine when to exit.

As in almost all languages:
while True:
# check what you want and eventually break
print nextValue()
The second part of your question is more interesting:
Also, if it is based on time then is there anyway I could just extend the time and start it going from that point again rather than having to start again from 0
you can use a yield instead of return in the function nextValue()

If you use a child thread to run the function while the main thread waits for character input it should work. Just remember to have something that stops the child thread (in the example below the global runthread)
For example:
import threading, time
runthread = 1
def myfun():
while runthread:
print "A"
time.sleep(.1)
t = threading.Thread(target=myfun)
t.start()
raw_input("")
runthread = 0
t.join()
does just that

If you want to exit based on time, you can use the signal module's alarm(time) function, and the catch the SIGALRM - here's an example http://docs.python.org/library/signal.html#example
You can let the user interrupt the program in a sane manner by catching KeyboardInterrupt. Simply catch the KeyboardInterrupt exception from outside you main loop, and do whatever cleanup you want.
If you want to continue later where you left off, you will have to add some sort persistence. I would pickle a data structure to disk, that you could read back in to continue the operations.
I haven't tried anything like this, but you could look into using something like memoizing, and caching to the disk.

You could do something like this to generate fibonnacci numbers for 1 second then stop.
fibonnacci = [1,1]
stoptime = time.time() + 1 # set stop time to 1 second in the future
while time.time() < stoptime:
fibonnacci.append(fibonnacci[-1]+fibonnacci[-2])
print "Generated %s numbers, the last one was %s." % (len(fibonnacci),fibonnacci[-1])
I'm not sure how efficient it is to call time.time() in every loop - depending on the what you are doing inside the loop, it might end up taking a lot of the performance away.