We Keep Coding

How to allow a class's variables to be modified concurrently by multiple threads

I have a class (MyClass) which contains a queue (self.msg_queue) of actions that need to be run and I have multiple sources of input that can add tasks to the queue.
Right now I have three functions that I want to run concurrently:
MyClass.get_input_from_user()
Creates a window in tkinter that has the user fill out information and when the user presses submit it pushes that message onto the queue.
MyClass.get_input_from_server()
Checks the server for a message, reads the message, and then puts it onto the queue. This method uses functions from MyClass's parent class.
MyClass.execute_next_item_on_the_queue()
Pops a message off of the queue and then acts upon it. It is dependent on what the message is, but each message corresponds to some method in MyClass or its parent which gets run according to a big decision tree.
Process description:
After the class has joined the network, I have it spawn three threads (one for each of the above functions). Each threaded function adds items from the queue with the syntax "self.msg_queue.put(message)" and removes items from the queue with "self.msg_queue.get_nowait()".
Problem description:
The issue I am having is that it seems that each thread is modifying its own queue object (they are not sharing the queue, msg_queue, of the class of which they, the functions, are all members).
I am not familiar enough with Multiprocessing to know what the important error messages are; however, it is stating that it cannot pickle a weakref object (it gives no indication of which object is the weakref object), and that within the queue.put() call the line "self._sem.acquire(block, timeout) yields a '[WinError 5] Access is denied'" error. Would it be safe to assume that this failure in the queue's reference not copying over properly?
[I am using Python 3.7.2 and the Multiprocessing package's Process and Queue]
[I have seen multiple Q/As about having threads shuttle information between classes--create a master harness that generates a queue and then pass that queue as an argument to each thread. If the functions didn't have to use other functions from MyClass I could see adapting this strategy by having those functions take in a queue and use a local variable rather than class variables.]
[I am fairly confident that this error is not the result of passing my queue to the tkinter object as my unit tests on how my GUI modifies its caller's queue work fine]
Below is a minimal reproducible example for the queue's error:
from multiprocessing import Queue
from multiprocessing import Process
import queue
import time
class MyTest:
def __init__(self):
self.my_q = Queue()
self.counter = 0
def input_function_A(self):
while True:
self.my_q.put(self.counter)
self.counter = self.counter + 1
time.sleep(0.2)
def input_function_B(self):
while True:
self.counter = 0
self.my_q.put(self.counter)
time.sleep(1)
def output_function(self):
while True:
try:
var = self.my_q.get_nowait()
except queue.Empty:
var = -1
except:
break
print(var)
time.sleep(1)
def run(self):
process_A = Process(target=self.input_function_A)
process_B = Process(target=self.input_function_B)
process_C = Process(target=self.output_function)
process_A.start()
process_B.start()
process_C.start()
# without this it generates the WinError:
# with this it still behaves as if the two input functions do not modify the queue
process_C.join()
if __name__ == '__main__':
test = MyTest()
test.run()

Indeed - these are not "threads" - these are "processes" - while if you were using multithreading, and not multiprocessing, the self.my_q instance would be the same object, placed at the same memory space on the computer,
multiprocessing does a fork of the process, and any data in the original process (the one in execution in the "run" call) will be duplicated when it is used - so, each subprocess will see its own "Queue" instance, unrelated to the others.
The correct way to have various process share a multiprocessing.Queue object is to pass it as a parameter to the target methods. The simpler way to reorganize your code so that it works is thus:
from multiprocessing import Queue
from multiprocessing import Process
import queue
import time
class MyTest:
def __init__(self):
self.my_q = Queue()
self.counter = 0
def input_function_A(self, queue):
while True:
queue.put(self.counter)
self.counter = self.counter + 1
time.sleep(0.2)
def input_function_B(self, queue):
while True:
self.counter = 0
queue.put(self.counter)
time.sleep(1)
def output_function(self, queue):
while True:
try:
var = queue.get_nowait()
except queue.Empty:
var = -1
except:
break
print(var)
time.sleep(1)
def run(self):
process_A = Process(target=self.input_function_A, args=(queue,))
process_B = Process(target=self.input_function_B, args=(queue,))
process_C = Process(target=self.output_function, args=(queue,))
process_A.start()
process_B.start()
process_C.start()
# without this it generates the WinError:
# with this it still behaves as if the two input functions do not modify the queue
process_C.join()
if __name__ == '__main__':
test = MyTest()
test.run()
As you can see, since your class is not actually sharing any data through the instance's attributes, this "class" design does not make much sense for your application - but for grouping the different workers in the same code block.
It would be possible to have a magic-multiprocess-class that would have some internal method to actually start the worker-methods and share the Queue instance - so if you have a lot of those in a project, there would be a lot less boilerplate.
Something along:
from multiprocessing import Queue
from multiprocessing import Process
import time
class MPWorkerBase:
def __init__(self, *args, **kw):
self.queue = None
self.is_parent_process = False
self.is_child_process = False
self.processes = []
# ensure this can be used as a colaborative mixin
super().__init__(*args, **kw)
def run(self):
if self.is_parent_process or self.is_child_process:
# workers already initialized
return
self.queue = Queue()
processes = []
cls = self.__class__
for name in dir(cls):
method = getattr(cls, name)
if callable(method) and getattr(method, "_MP_worker", False):
process = Process(target=self._start_worker, args=(self.queue, name))
self.processes.append(process)
process.start()
# Setting these attributes here ensure the child processes have the initial values for them.
self.is_parent_process = True
self.processes = processes
def _start_worker(self, queue, method_name):
# this method is called in a new spawned process - attribute
# changes here no longer reflect attributes on the
# object in the initial process
# overwrite queue in this process with the queue object sent over the wire:
self.queue = queue
self.is_child_process = True
# call the worker method
getattr(self, method_name)()
def __del__(self):
for process in self.processes:
process.join()
def worker(func):
"""decorator to mark a method as a worker that should
run in its own subprocess
"""
func._MP_worker = True
return func
class MyTest(MPWorkerBase):
def __init__(self):
super().__init__()
self.counter = 0
#worker
def input_function_A(self):
while True:
self.queue.put(self.counter)
self.counter = self.counter + 1
time.sleep(0.2)
#worker
def input_function_B(self):
while True:
self.counter = 0
self.queue.put(self.counter)
time.sleep(1)
#worker
def output_function(self):
while True:
try:
var = self.queue.get_nowait()
except queue.Empty:
var = -1
except:
break
print(var)
time.sleep(1)
if __name__ == '__main__':
test = MyTest()
test.run()

Update long process' progress via callback instead of polling

in my Python script I am triggering a long process (drive()) that is encapsulated into a class method:
car.py
import time
class Car(object):
def __init__(self, sleep_time_in_seconds, miles_to_drive):
self.sleep_time_in_seconds = sleep_time_in_seconds
self.miles_to_drive = miles_to_drive
def drive(self):
for mile in range(self.miles_to_drive):
print('driving mile #{}'.format(mile))
time.sleep(self.sleep_time_in_seconds)
app.py
from car import Car
sleep_time = 2
total_miles = 5
car = Car(sleep_time_in_seconds=sleep_time, miles_to_drive=total_miles)
car.drive()
def print_driven_distance_in_percent(driven_miles):
print("Driven distance: {}%".format(100 * driven_miles / total_miles))
In the main script app.py I'd like to know the progress of the drive() process. One way of solving this would be to create a loop that polls the current progress from the Car class. If the Car class would inherit from Thread - polling seems to be an expected pattern as far as I have googled...
I'm just curious whether it's possible to somehow notify the main script from within the Car class about the current progress.
I thought about maybe creating a wrapper class that I can pass as argument to the Car class, and the car instance then can call the wrapper class' print_progress function.
Or is there a more pythonic way to notify the caller script on demand?
Thanks
EDIT:
Based on Artiom Kozyrev's answer - this is what I wanted to achieve:
import time
from threading import Thread
from queue import Queue
def ask_queue(q):
"""
The function to control status of our status display thread
q - Queue - need to show status of task
"""
while True:
x = q.get() # take element from Queue
if x == "STOP":
break
print("Process completed in {} percents".format(x))
print("100% finished")
class MyClass:
"""My example class"""
def __init__(self, name, status_queue):
self.name = name
self.status_queue = status_queue
def my_run(self):
"""
The function we would like to monitor
"""
# th = Thread(target=MyClass.ask_queue, args=(self.status_queue,), ) # monitoring thread
# th.start() # start monitoring thread
for i in range(100): # start doing our main function we would like to monitor
print("{} {}".format(self.name, i))
if i % 5 == 0: # every 5 steps show status of progress
self.status_queue.put(i) # send status to Queue
time.sleep(0.1)
self.status_queue.put("STOP") # stop Queue
# th.join()
if __name__ == "__main__":
q = Queue()
th = Thread(target=ask_queue, args=(q,), ) # monitoring thread
th.start() # start monitoring thread
# tests
x = MyClass("Maria", q)
x.my_run()
th.join()
Thanks to all!!

Thanks for interesting question, typically you do not need to use status as a separate thread for the case, you can just print status in the method you would like to monitor, but for training purpose you solve the issue the follwoing way, please follow comments and feel free to ask:
import time
from threading import Thread
from queue import Queue
class MyClass:
"""My example class"""
def __init__(self, name, status_queue):
self.name = name
self.status_queue = status_queue
#staticmethod
def ask_queue(q):
"""
The function to control status of our status display thread
q - Queue - need to show status of task
"""
while True:
x = q.get() # take element from Queue
if x == "STOP":
break
print("Process completed in {} percents".format(x))
print("100% finished")
def my_run(self):
"""
The function we would like to monitor
"""
th = Thread(target=MyClass.ask_queue, args=(self.status_queue,), ) # monitoring thread
th.start() # start monitoring thread
for i in range(100): # start doing our main function we would like to monitor
print("{} {}".format(self.name, i))
if i % 5 == 0: # every 5 steps show status of progress
self.status_queue.put(i) # send status to Queue
time.sleep(0.1)
self.status_queue.put("STOP") # stop Queue
th.join()
if __name__ == "__main__":
# tests
x = MyClass("Maria", Queue())
x.my_run()
print("*" * 200)
x.my_run()

Python pyttsx, how to use external loop

In my program I need class(which can be some thread) to check some list like "say_list" and when other classes add some text to it, pyttsx say that text.
I search in pyttsx docs and I find some external loops feature but I can not find example which work correctly.
I want something like this:
import pyttsx
import threading
class VoiceAssistant(threading.Thread):
def __init__(self):
super(VoiceAssistant, self).__init__()
self.engine = pyttsx.init()
self.say_list = []
def add_say(self, msg):
self.say_list.append(msg)
def run(self):
while True:
if len(self.say_list) > 0:
self.engine.say(self.say_list[0])
self.say_list.remove(self.say_list[0])
if __name__ == '__main__':
va = VoiceAssistant()
va.start()
Thanks.

I can get the proper results by using python's built in Queue class:
import pyttsx
from Queue import Queue
from threading import Thread
q = Queue()
def say_loop():
engine = pyttsx.init()
while True:
engine.say(q.get())
engine.runAndWait()
q.task_done()
def another_method():
t = Thread(target=say_loop)
t.daemon = True
t.start()
for i in range(0, 3):
q.put('Sally sells seashells by the seashore.')
print "end of another method..."
def third_method():
q.put('Something Something Something')
if __name__=="__main__":
another_method()
third_method()
q.join() # ends the loop when queue is empty
Above is a simple example I whipped up. It uses the 'queue/consumer' model to allow separate functions/classes to access the same queue, then a worker that will execute anytime the queue has items. Should be pretty easy to adapt to your needs.
Further reading about Queues: https://docs.python.org/2/library/queue.html
There appears to be an interface for this in the docs you linked to, but it seemed like you were already on the separate thread track so this seemed closer to what you wanted.
And here's the modified version of your code:
import pyttsx
from Queue import Queue
import threading
class VoiceAssistant(threading.Thread):
def __init__(self):
super(VoiceAssistant, self).__init__()
self.engine = pyttsx.init()
self.q = Queue()
self.daemon = True
def add_say(self, msg):
self.q.put(msg)
def run(self):
while True:
self.engine.say(self.q.get())
self.engine.runAndWait()
self.q.task_done()
if __name__ == '__main__':
va = VoiceAssistant()
va.start()
for i in range(0, 3):
va.add_say('Sally sells seashells by the seashore.')
print "now we want to exit..."
va.q.join() # ends the loop when queue is empty

pyttsx3 loop pronunciation
import pyttsx3
voice_text = 'test'
def loopSound():
def onEnd(name, completed):
print('finishing', name, completed)
engine.say(voice_text, 'voice_a')
engine = pyttsx3.init()
engine.connect('finished-utterance', onEnd)
engine.say(voice_text, 'voice_e')
engine.startLoop()
loopSound()

When I'm testing about multiprocessing and threading with python, and I meet a odd situation

I am using process pools(including 3 processes). In every process, I have set (created) some threads by using the thread classes to speed handle something.
At first, everything was OK. But when I wanted to change some variable in a thread, I met an odd situation.
For testing or to know what happens, I set a global variable COUNT to test. Honestly, I don't know this is safe or not. I just want to see, by using multiprocessing and threading can I change COUNT or not?
#!/usr/bin/env python
# encoding: utf-8
import os
import threading
from Queue import Queue
from multiprocessing import Process, Pool
# global variable
max_threads = 11
Stock_queue = Queue()
COUNT = 0
class WorkManager:
def __init__(self, work_queue_size=1, thread_pool_size=1):
self.work_queue = Queue()
self.thread_pool = [] # initiate, no have a thread
self.work_queue_size = work_queue_size
self.thread_pool_size = thread_pool_size
self.__init_work_queue()
self.__init_thread_pool()
def __init_work_queue(self):
for i in xrange(self.work_queue_size):
self.work_queue.put((func_test, Stock_queue.get()))
def __init_thread_pool(self):
for i in xrange(self.thread_pool_size):
self.thread_pool.append(WorkThread(self.work_queue))
def finish_all_threads(self):
for i in xrange(self.thread_pool_size):
if self.thread_pool[i].is_alive():
self.thread_pool[i].join()
class WorkThread(threading.Thread):
def __init__(self, work_queue):
threading.Thread.__init__(self)
self.work_queue = work_queue
self.start()
def run(self):
while self.work_queue.qsize() > 0:
try:
func, args = self.work_queue.get(block=False)
func(args)
except Queue.Empty:
print 'queue is empty....'
def handle(process_name):
print process_name, 'is running...'
work_manager = WorkManager(Stock_queue.qsize()/3, max_threads)
work_manager.finish_all_threads()
def func_test(num):
# use a global variable to test what happens
global COUNT
COUNT += num
def prepare():
# prepare test queue, store 50 numbers in Stock_queue
for i in xrange(50):
Stock_queue.put(i)
def main():
prepare()
pools = Pool()
# set 3 process
for i in xrange(3):
pools.apply_async(handle, args=('process_'+str(i),))
pools.close()
pools.join()
global COUNT
print 'COUNT: ', COUNT
if __name__ == '__main__':
os.system('printf "\033c"')
main()
Now, finally the result of COUNT is just 0.I am unable to understand whats happening here?

You print the COUNT var in the father process. Variables doesn't sync across processes because they doesn't share memory, that means that the variable stay 0 at the father process and is increased in the subprocesses
In the case of threading, threads share memory, that means that they share the variable count, so they should have COUNT as more than 0 but again they are at the subprocesses, and when they change the variable, it doesn't update it in other processes.

multi threads modify a global list in python

i want to add an item into a global list every 2 seconds in one thread,
and save the list into database before empty it every 3 seconds in another thread.
i create two local varibles to monitor the total added items and total saveditems, they should be equal every 6 senconds,but it is not.
here is my code:
import datetime
import psutil,os,time
from threading import *
class AddToList(Thread):
totalAdded=0
def run(self):
lock=RLock()
lock.acquire()
while True:
entryList.append("AddToList at "+str(datetime.datetime.now()))
self.totalAdded=self.totalAdded+len(entryList)
print("totalAdded:"+str(self.totalAdded))
time.sleep(2)
lock.release()
class SaveList(Thread):
totalSaved=0
'''save entry to server'''
def __init__(self):
Thread.__init__(self)
def run(self):
lock=RLock()
lock.acquire()
while True:
#save list to database,then empty the list
self.totalSaved=self.totalSaved+len(entryList)
del entryList[:]
print("totalSaved:"+str(self.totalSaved))
time.sleep(3)
lock.release()
if __name__=="__main__":
global entryList
entryList=[]
addClass= AddToList()
addClass.start()
saveClass=SaveList()
saveClass.start()
result:
totalAdded:2
totalSaved:2
totalAdded:3
totalSaved:3totalAdded:4
totalAdded:6
totalSaved:5
totalAdded:7
totalSaved:6
totalAdded:8
totalAdded:10
totalSaved:8
totalAdded:11
totalSaved:9
totalAdded:12
totalAdded:14
totalSaved:11
totalAdded:15
totalSaved:12
...........
...........
totalAdded:51
totalSaved:39totalAdded:52
totalAdded:54
totalSaved:41
totalAdded:55
totalSaved:42
totalAdded:56
totalAdded:58
totalSaved:44
totalAdded:59
totalSaved:45totalAdded:60
......
......
i anm new to python and searched a lot about threading ,Lock and RLock ,but with no luck.
where am wrong?

To make Lock and RLock work you must use the same object in every thread. The lock objects must have the same "visibility" of the object that you want to "protect".
Here is a new version of you code which should work. It also avoid using things like global variables etc.
import datetime
import time
import threading
class AddToList(threading.Thread):
def __init__(self, lock, entryList):
threading.Thread.__init__(self)
self.totalAdded = 0
self.entryList = entryList
self.lock = lock
def run(self):
while True:
self.lock.acquire()
entryList.append("AddToList at {}".format(datetime.datetime.now()))
self.totalAdded += 1
self.lock.release()
print("totalAdded: {}".format(self.totalAdded))
time.sleep(2)
class SaveList(threading.Thread):
def __init__(self, lock, entryList):
threading.Thread.__init__(self)
self.totalSaved = 0
self.entryList = entryList
self.lock = lock
def run(self):
while True:
self.lock.acquire()
self.totalSaved += len(self.entryList)
del self.entryList[:]
self.lock.release()
print("totalSaved: {}".format(self.totalSaved))
time.sleep(3)
if __name__=="__main__":
lock=threading.Lock()
entryList=[]
addClass = AddToList(lock, entryList)
addClass.start()
saveClass = SaveList(lock, entryList)
saveClass.start()
Some things to note:
Use Lock instead of RLock when you don't have any particular needs. RLock is much slower.
As already pointed out by someone it is better avoid using global variables when not needed. Also Class variables should be used only when it makes sense.
When you use a lock you should try to limit as much as possible the code between acquire and release. In you previous code you never release the lock.