We Keep Coding

How to make a secondary process continually update a variable through ProcessPoolExecutor in python

Well met!
I'm trying to use pyautogui to run some simple checks, I'm attempting to make the main process detect a visual input, then start a sub process that continually updates a shared variable with the Y position of a different image as it moves through the screen until it disappears.
Unfortunately I'm barely a programmer so I keep getting stuck on the execution, so I wanted to ask for help. This is the code I wrote,
import pyautogui
import time
import importlib
foobar = importlib.import_module("cv2")
foocat = importlib.import_module("concurrent")
import numpy
import concurrent.futures
with concurrent.futures.ProcessPoolExecutor() as executor:
CheckingInput = executor.submit(CheckPositionInput)
CheckingImage = executor.submit(CheckPositionImage)
print(XMark, YMark)
print(time.time() - startingtime)
def CheckPositionInput():
Checked = False
global XImage, YImage, XMark, YMark, AreaOnScreen
while not Checked:
print('Searching')
if pyautogui.locateCenterOnScreen('Area.png', confidence=0.8) != None:
Checked = True
AreaOnScreen = True
XMark, YMark = pyautogui.locateCenterOnScreen('Area.png', confidence=0.8)
def CheckPositionImage():
global XImage, YImage, XMark, YMark, AreaOnScreen
print('start')
while not AreaOnScreen:
print('Waiting')
while AreaOnScreen:
if pyautogui.locateCenterOnScreen('Image.png', confidence=0.6) != None:
XMark, YMark = pyautogui.locateCenterOnScreen('Image.png', confidence=0.6)
print(YMark)
print('Checking')
The problems I've run into go from the while loop in CheckPositionImage closing and dying after a single loop, to the while loop in CheckPositionImage getting stuck and stopping the check position process, and that no matter what I try I can't manage to update the crucial Ymark variable properly outside the process.

It's important to understand that global variables are not read/write sharable across multiple processes. A child process can possibly inherit such a variable value (depends on what platform you are running) and read its value, but once a process assigns a new value to that variable, this change is not reflected back to any other process. This is because every process runs in its own address space and can only modify its copy of a global variable. You would need to use instead shared memory variables created by the main process and passed to its child processes. But let's ignore this fundamental problem for now and assume that global variables were sharable.
If I follow your code correctly, this is what you appear to be doing:
The main process submits two tasks to a multiprocessing pool to be processed by worker functions CheckPositionInput and CheckPositionImage and then waits for both tasks to complete to print out global variables XMark and YMark, presumably set by the CheckPositionImage function.
CheckPositionImage is effectively doing nothing until CheckPositionInput sets global variable AreaOnScreen to True, which only occurs after the call pyautogui.locateCenterOnScreen('Area.png', confidence=0.8) returns a value that is not None. When this occurs, checked is set to True and your loop terminates effectively terminating the task.
When varibale AreaOnScreen is set to True (in step 2. above), function CheckPositionImage finally enters into a loop calling pyautogui.locateCenterOnScreen('Image.png', confidence=0.6). When this function returns a value that is not None a couple of print statements are issued and the loop is re-iterated.
To the extent that my analysis is correct, I have a few comments:
This CheckPositionImage task never ends since variable AreaOnSCreen is neither ever reset to False nor is a return nor break statement issued in the loop. I assume this is an oversight and once we are returned a non-None value from our call to pyautogui.locateCenterOnScreen, we should return. My assumption is based on the fact that without this termination occurring, the main process's block beginning with concurrent.futures.ProcessPoolExecutor() as executor: will never complete (there is am implicit wait for all submitted tasks to complete) and you will therefore never fall through to the subsequent print statements.
You never initialize variable startingtime.
Function CheckPositionInput sets global variables XMark and YMark, whose values are never referenced by either the main process or function pyautogui.locateCenterOnScreen('Image.png', confidence=0.6). What is the point in calling this function a second time with identical arguments to set these variables that are never read?
You have processes running, but the actual processing is essentially sequential: The main process does nothing until both child processes it has created end and one child process does nothing useful until the other child process sets a flag when its terminating. I see, therefore, no reason for using multiprocessing at all. Your code could be simply (note that I have renamed variables and functions according to Python's PEP8 coding conventions:
import pyautogui
import time
# What is the purpose of these next 3 commented-out statements?
#import importlib
#foobar = importlib.import_module("cv2")
#foocat = importlib.import_module("concurrent")
def search_and_check():
print('Searching...')
while True:
if pyautogui.locateCenterOnScreen('Area.png', confidence=0.8) != None:
# What is the purpose of this second call, which I have commented out?
# Note that the values set, i.e. xMark and yMark, are never referenced.
#xMark, yMark = pyautogui.locateCenterOnScreen('Area.png', confidence=0.8)
break
print('Checking...')
while True:
result = pyautogui.locateCenterOnScreen('Image.png', confidence=0.6)
if result != None:
return result
starting_time = time.time()
xMark, yMark = search_and_check()
print(xMark, yMark)
print(time.time() - starting_time)
Could/should the two different calls to pyautogui.locateCenterOnScreen be done in parallel?

How can I elegantly organize this multithreaded python code?

I am working on a Python service that subscribes real-time streaming data from one messaging broker and publishes to another broker, in some situations I also need to get snapshot data from other data source on network disconnection or system recovery. While the streaming data comes from one thread, and some service events happen in another thread, I decided to create a data processing thread to just pop the queue one by one. I got it to work but later I tried to keep the snapshot fetching logic in a separate thread and that's where things get messy.
I know this is a long question with a lot of specific nuances but I tried to make the example here as clear as I can.
So here is what the 1st attempt looks like, and it works well:
import queue
import threading
def process_data(data_queue, data_store):
# data_store is my internal cache data structure.
# so for simplicity and demonstration purpose, I assume the following:
# if its type is dict, it's snapshot data
# if its type is tuple, it's a key/value pair and that's an incremental update data
# if it is -1, we terminate the queue processing
# if it is -2, we need to retrieve a snapshot
while True:
x = data_queue.get()
if isinstance(x, dict):
data_store.on_snapshot(x)
elif isinstance(x, tuple):
k, v = x
data_store.on_update(k, v)
elif isinstance(x, int):
if x == -1:
data_queue.task_done()
break
elif x == -2:
get_snapshot() # this is potentially a long blocking call
else:
print('unknown int', x)
else:
print('unknown data', x)
data_queue.task_done()
if __name__ == '__main__':
data_store = DataStore()
data_queue = queue.Queue()
# start other threads that write data to the queue
start_data_writer1(data_queue)
start_data_writer2(data_queue)
start_thread_for_some_event(data_queue) # may put -2 in the queue for snapshot
process_thread = threading.Thread(
target=process_data,
args=(data_queue, data_store))
process_thread.start()
data_queue.put(-2) # signal a snapshot fetching
do_something_else()
try:
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print('terminating...')
finally:
# to break out of the infinite loop in process_data()
data_queue.put(-1)
process_thread.join()
data_queue.join()
This way works however I don't particularly like the fact that I am calling get_snapshot() function in the processing thread. I thought the idea of the processing thread is to be busy at popping data off the queue all the time, unless there's nothing to pop. In the above implementation, during the get_snapshot call, it is possible that queue could be built up due to other writer threads.
So I tried something else and I also wanted to be able to exit the program gracefully. That's where things get really ugly. I created a new thread for occasionally fetching the snapshot and used condition object for thread communication. This is what I did on top of the existing code:
snapshot_lock = threading.Lock()
cond = threading.Condition(snapshot_lock)
need_for_snapshot = False # used to trigger snapshots
keep_snapshot_thread = True # flag if the snapshot thread is done
# then I need to add this new function to run snapshot fetching
def fetch_snapshot(data_queue):
global need_for_snapshot
global keep_snapshot_thread
def _need_snapshot():
global need_for_snapshot
return need_for_snapshot
while True:
with cond:
cond.wait_for(_need_snapshot)
if not keep_snapshot_thread:
break
data_queue.put(get_snapshot()) # the long blocking function
need_for_snapshot = False
# in process_data() function, everything stays the same except for the `if` statement for handling `x == 2`
def process_data(data_queue, data_store):
global need_for_snapshot
while True:
x = data_queue.get()
# omitting some old code
elif isinstance(x, int):
if x == -1:
data_queue.task_done()
break
elif x == -2:
**with cond:
need_for_snapshot = True
cond.notify()**
# more code omitted
if __name__ == '__main__':
# same code as before except for the finally part
try:
# start other threads...omitting some code
# when a snapshot is needed in these threads
# do the following
# with cond:
# need_for_snapshot = True
# cond.notify()
# start snapshot worker thread
snapshot_thread = threading.Thread(
target=fetch_snapshot, args=(data_queue,))
process_thread = threading.Thread(
target=process_data,
args=(data_queue, data_store))
snapshot_thread.start()
process_thread.start()
data_queue.put(-2) # signal fetching a snapshot
# omitting more code here...
finally:
keep_snapshot_thread = False
# we don't technically need to trigger another snapshot now
# but the code below is to unblock the cond.wait_for() part
# since keep_snapshot_thread flag is just flipped, we can use
# it to break out of the infinite loop in fetch_snapshot thread.
# This is the part that I feel hacky...
with cond:
need_for_snapshot = True
cond.notify()
snapshot_t.join()
data_queue.put(-1) # signal the termination of process_thread
process_t.join()
data_queue.join()
I think I got this to work, especially that the program can exit gracefully when I hit ctrl-c but it is so ugly and tricky that I had to play with it quick a bit to get it to work correctly.
Is there some way I can write it more elegantly? Is there some sort of pattern that we generally use to solve this type of problem? Thank you so much for your help.

The standard technique for handling multiple producers and multiple consumers is to use an Event is_done and a joinable Queue work.
The worker queues do nothing but:
while not event.is_set():
try:
job = work.get(timeout=5)
except Empty:
continue
handle the job
work.task_done()
Your main worker does the following:
start the jobs that produce work
wait for them to be done
work.join() # wait for the queue to be empty
event.set() # tell the workers they can exit
perform any cleanup necessary
Note that the goal is to decouple the workers and the producers as much as possible. Trying to create complicated logic tying them together is almost certain to produce race cases.
An alternative would be to create a sentinel object like "END" indicating that everything is done. Once all the producers are done, the main thread would push a number of sentinel objects equal to the number of workers onto the work queue, and the call work.join(). Each worker thread would call work.get() inside a loop and exit when it saw the sentinel. Remember to call work.task_done() on the sentinel, too!
Again. You want simple logic and to use the tools provided by multithreading.

What is the best way to user-input data while in a loop?

I have a Python program running a (nested) loop which will run for a fairly long time, and I want the user to be able to pause and/or abort it with just pressing p and c, respectively.
I am running this in an IPython console, so I don't really have access to msvctr.getch and I kinda want to keep it platform independent.
Obviously, input() blocks, which is exactly what I do not want. So I tried threading, which works when used as intended, but when hitting CTRLC the thread does not stop. This is likely because any legitimate method to stop the thread (atexit, global variable or lambda stop_thread) isn't executed because the thread blocks.
import threading
import queue
q = queue.SimpleQueue()
stop_thread = False
def handle_input(q, stopped):
s = ''
while not stopped():
s = input()
q.put(s)
thread = threading.Thread(target=handle_input,
args=[q, lambda: stop_thread])
thread.start()
for i in range(very_long_time):
#Do something time consuming
if not q.empty():
s = q.get_nowait()
if 'p' in s:
print('Paused...', end='\r')
s = s.replace('p', '')
while True:
if not q.empty():
s += q.get_nowait()
if 'p' in s or 'c' in s:
s = s.replace('p', '')
break
time.sleep(0.5)
if 'c' in s:
print('\rAborted training loop...' + ' '*50, end='\r')
s = s.replace('c', '')
stop_thread = True
# Another method of stopping the thread
# thread.__getattribute__('_tstate_lock').release()
# thread._stop()
# thread.join()
break
This works in principle, but breaks when interrupting.
The thread does not seem to stop, which poses a problem when running this again in the same console, because it does not even ask for user input then.
Additionally, this prints my 'c' or 'p' and a newline, which I can't get rid of, because IPython doesn't allow all ANSI escapes.
Is there a fix to my method, or even better, a cleaner alternative?

You can try using the keyboard module, which (among other things) lets you bind event hooks to keyboard presses.
In this case, I would create a set of global variables/flags (say, paused and abort), initially set to False, and then make some hotkeys for p and c respectively to toggle them:
paused = False
abort = False
def toggle_paused():
global paused
paused = not paused
def trigger_abort():
abort = True
keyboard.add_hotkey('p', toggle_paused())
keyboard.add_hotkey('c', trigger_abort())
And then change your loop to check for paused and abort on every iteration (assuming, that is, that each iteration is fairly quick). What you're already doing would more-or-less work - just remove the queues and threading stuff you've already set up (IIRC keyboard's events run on their own threads anyway), de-indent the if conditions, and change the conditions to if paused: and if abort: respectively.
You can also lace the rest of your code with things that look for pause or abort flags, so that your program can gracefully pause or exit at a convenient time for it. You can also extend the toggle_paused() and trigger_abort() to do whatever you need them to (e.g. have trigger_abort() print "Trying to abort program (kill me if I'm not done in 5 seconds)" or something.
Although, as #Tomerikoo suggested in a comment, creating the threat with the daemon=True option is the best answer, if it's possible with the way your program is designed. If this is all your program does then using daemon threads wouldn't work, because your program would just quit immediately, but if this is a background operation then you can use a daemon thread to put it in the background where it won't obstruct the rest of the user's experience.

How to start a thread again with an Event object in Python?

I want to make a thread and control it with an event object. Detailedly speaking, I want the thread to be executed whenever the event object is set and to wait itselt, repeatedly.
The below shows a sketchy logic I thought of.
import threading
import time
e = threading.Event()
def start_operation():
e.wait()
while e.is_set():
print('STARTING TASK')
e.clear()
t1 = threading.Thread(target=start_operation)
t1.start()
e.set() # first set
e.set() # second set
I expected t1 to run once the first set has been commanded and to stop itself(due to e.clear inside it), and then to run again after the second set has been commanded. So, accordign to what I expected, it should print out 'STARTING TASK' two times. But it shows it only once, which I don't understand why. How am I supposed to change the code to make it run the while loop again, whenever the event object is set?

The first problem is that once you exit a while loop, you've exited it. Changing the predicate back won't change anything. Forget about events for a second and just look at this code:
i = 0
while i == 0:
i = 1
It obviously doesn't matter if you set i = 0 again later, right? You've already left the while loop, and the whole function. And your code is doing exactly the same thing.
You can fix problem that by just adding another while loop around the whole thing:
def start_operation():
while True:
e.wait()
while e.is_set():
print('STARTING TASK')
e.clear()
However, that still isn't going to work—except maybe occasionally, by accident.
Event.set doesn't block; it just sets the event immediately, even if it's already set. So, the most likely flow of control here is:
background thread hits e.wait() and blocks.
main thread hits e.set() and sets event.
main thread hits e.set() and sets event again, with no effect.
background thread wakes up, does the loop once, calls e.clear() at the end.
background thread waits forever on e.wait().
(The fact that there's no way to avoid missed signals with events is effectively the reason conditions were invented, and that anything newer than Win32 and Python doesn't bother with events… But a condition isn't sufficient here either.)
If you want the main thread to block until the event is clear, and only then set it again, you can't do that. You need something extra, like a second event, which the main thread can wait on and the background thread can set.
But if you want to keep track of multiple set calls, without missing any, you need to use a different sync mechanism. A queue.Queue may be overkill here, but it's dead simple to do in Python, so let's just use that. Of course you don't actually have any values to put on the queue, but that's OK; you can just stick a dummy value there:
import queue
import threading
q = queue.Queue()
def start_operation():
while True:
_ = q.get()
print('STARTING TASK')
t1 = threading.Thread(target=start_operation)
t1.start()
q.put(None)
q.put(None)
And if you later want to add a way to shut down the background thread, just change it to stick values on:
import queue
import threading
q = queue.Queue()
def start_operation():
while True:
if q.get():
return
print('STARTING TASK')
t1 = threading.Thread(target=start_operation)
t1.start()
q.put(False)
q.put(False)
q.put(True)

making a programme run indefinitely in python

Is there any way to make a function (the ones I'm thinking of are in the style of the simple ones I've made which generate the fibonnacci sequence from 0 to a point, and all the primes between two points) run indefinitely. E.g. until I press a certain key or until a time has passed, rather than until a number reaches a certain point?
Also, if it is based on time then is there any way I could just extend the time and start it going from that point again, rather than having to start again from 0? I am aware there is a time module, i just don't know much about it.

The simplest way is just to write a program with an infinite loop, and then hit control-C to stop it. Without more description it's hard to know if this works for you.
If you do it time-based, you don't need a generator. You can just have it pause for user input, something like a "Continue? [y/n]", read from stdin, and depending on what you get either exit the loop or not.

If you really want your function to run and still wants user (or system) input, you have two solutions:
multi-thread
multi-process
It will depend on how fine the interaction. If you just want to interrupt the function and don't care about the exit, then multi-process is fine.
In both cases, you can rely on some shared resources (file or shared memory for multi-thread, variable with associated mutex for multi-thread) and check for the state of that resource regularly in your function. If it is set up to tell you to quit, just do it.
Example on multi-thread:
from threading import Thread, Lock
from time import sleep
class MyFct(Thread):
def __init__(self):
Thread.__init__(self)
self.mutex = Lock()
self._quit = False
def stopped(self):
self.mutex.acquire()
val = self._quit
self.mutex.release()
return val
def stop(self):
self.mutex.acquire()
self._quit = True
self.mutex.release()
def run(self):
i = 1
j = 1
print i
print j
while True:
if self.stopped():
return
i,j = j,i+j
print j
def main_fct():
t = MyFct()
t.start()
sleep(1)
t.stop()
t.join()
print "Exited"
if __name__ == "__main__":
main_fct()

You could use a generator for this:
def finished():
"Define your exit condition here"
return ...
def count(i=0):
while not finished():
yield i
i += 1
for i in count():
print i
If you want to change the exit condition you could pass a value back into the generator function and use that value to determine when to exit.

As in almost all languages:
while True:
# check what you want and eventually break
print nextValue()
The second part of your question is more interesting:
Also, if it is based on time then is there anyway I could just extend the time and start it going from that point again rather than having to start again from 0
you can use a yield instead of return in the function nextValue()

If you use a child thread to run the function while the main thread waits for character input it should work. Just remember to have something that stops the child thread (in the example below the global runthread)
For example:
import threading, time
runthread = 1
def myfun():
while runthread:
print "A"
time.sleep(.1)
t = threading.Thread(target=myfun)
t.start()
raw_input("")
runthread = 0
t.join()
does just that

If you want to exit based on time, you can use the signal module's alarm(time) function, and the catch the SIGALRM - here's an example http://docs.python.org/library/signal.html#example
You can let the user interrupt the program in a sane manner by catching KeyboardInterrupt. Simply catch the KeyboardInterrupt exception from outside you main loop, and do whatever cleanup you want.
If you want to continue later where you left off, you will have to add some sort persistence. I would pickle a data structure to disk, that you could read back in to continue the operations.
I haven't tried anything like this, but you could look into using something like memoizing, and caching to the disk.

You could do something like this to generate fibonnacci numbers for 1 second then stop.
fibonnacci = [1,1]
stoptime = time.time() + 1 # set stop time to 1 second in the future
while time.time() < stoptime:
fibonnacci.append(fibonnacci[-1]+fibonnacci[-2])
print "Generated %s numbers, the last one was %s." % (len(fibonnacci),fibonnacci[-1])
I'm not sure how efficient it is to call time.time() in every loop - depending on the what you are doing inside the loop, it might end up taking a lot of the performance away.