We Keep Coding

Pythonic way of observing a variable in multithreading?

Imagine having the two threads observeT and upT. observeT observes the value of an instance attribute (instance.a) and should 'alert' (print a note in this example) if its value is 7. Then there's the thread upT, which increases the value of the instance attribute by 1 at a time (instance.a += 1).
However, due to the randomly chosen thread to continue with Python's Lock we can't make sure that the observer thread (observeT) catches the moment when the value of instance.a was increased to 7.
How do I make sure that the observer is called every time after upT releases to lock? Note that it is important to keep the threads upT and observeT split.
Please see the following code for more details:
from threading import Lock, Thread
class MyClass():
a: int
def __new__(cls):
instance = super().__new__(cls)
instance.a = 0
return instance
instance = MyClass()
lock = Lock()
def up():
for i in range(100000):
with lock:
instance.a += 1
def observe():
while True:
with lock:
a = instance.a
if a == 7:
print("This is 7!")
if instance.a == 100000:
break
observeT = Thread(target=observe)
upT = Thread(target=up)
observeT.start()
upT.start()
upT.join()
observeT.join()
Thank you for your help!

Is this what you're looking for?
from threading import Thread, Lock, Condition
class MyClass:
def __init__(self, a_lock):
self.cond = Condition(a_lock)
self.canproceed = False
self.a = 0
def __setattr__(self, key, value):
super().__setattr__(key, value)
if key == 'a':
if value == 7 or value == 100000:
self.cond.notify()
if value == 7:
while not self.canproceed:
self.cond.wait()
lock = Lock()
instance = MyClass(lock)
def up():
for i in range(100000):
with lock:
instance.a += 1
def observe():
with instance.cond:
while instance.a != 7:
instance.cond.wait()
print("This is 7!")
instance.canproceed = True
instance.cond.notify()
while instance.a != 100000:
instance.cond.wait()
observeT = Thread(target=observe)
upT = Thread(target=up)
observeT.start()
upT.start()
upT.join()
observeT.join()
Output:
This is 7!

When I'm testing about multiprocessing and threading with python, and I meet a odd situation

I am using process pools(including 3 processes). In every process, I have set (created) some threads by using the thread classes to speed handle something.
At first, everything was OK. But when I wanted to change some variable in a thread, I met an odd situation.
For testing or to know what happens, I set a global variable COUNT to test. Honestly, I don't know this is safe or not. I just want to see, by using multiprocessing and threading can I change COUNT or not?
#!/usr/bin/env python
# encoding: utf-8
import os
import threading
from Queue import Queue
from multiprocessing import Process, Pool
# global variable
max_threads = 11
Stock_queue = Queue()
COUNT = 0
class WorkManager:
def __init__(self, work_queue_size=1, thread_pool_size=1):
self.work_queue = Queue()
self.thread_pool = [] # initiate, no have a thread
self.work_queue_size = work_queue_size
self.thread_pool_size = thread_pool_size
self.__init_work_queue()
self.__init_thread_pool()
def __init_work_queue(self):
for i in xrange(self.work_queue_size):
self.work_queue.put((func_test, Stock_queue.get()))
def __init_thread_pool(self):
for i in xrange(self.thread_pool_size):
self.thread_pool.append(WorkThread(self.work_queue))
def finish_all_threads(self):
for i in xrange(self.thread_pool_size):
if self.thread_pool[i].is_alive():
self.thread_pool[i].join()
class WorkThread(threading.Thread):
def __init__(self, work_queue):
threading.Thread.__init__(self)
self.work_queue = work_queue
self.start()
def run(self):
while self.work_queue.qsize() > 0:
try:
func, args = self.work_queue.get(block=False)
func(args)
except Queue.Empty:
print 'queue is empty....'
def handle(process_name):
print process_name, 'is running...'
work_manager = WorkManager(Stock_queue.qsize()/3, max_threads)
work_manager.finish_all_threads()
def func_test(num):
# use a global variable to test what happens
global COUNT
COUNT += num
def prepare():
# prepare test queue, store 50 numbers in Stock_queue
for i in xrange(50):
Stock_queue.put(i)
def main():
prepare()
pools = Pool()
# set 3 process
for i in xrange(3):
pools.apply_async(handle, args=('process_'+str(i),))
pools.close()
pools.join()
global COUNT
print 'COUNT: ', COUNT
if __name__ == '__main__':
os.system('printf "\033c"')
main()
Now, finally the result of COUNT is just 0.I am unable to understand whats happening here?

You print the COUNT var in the father process. Variables doesn't sync across processes because they doesn't share memory, that means that the variable stay 0 at the father process and is increased in the subprocesses
In the case of threading, threads share memory, that means that they share the variable count, so they should have COUNT as more than 0 but again they are at the subprocesses, and when they change the variable, it doesn't update it in other processes.

Why doesn’t this thread terminate?

I am trying to use the python threading module. As I am sysadmin, I struggle a little bit when developing; and this concept is kind of new for me. I launch two threads and I want to stop them, when the main thread sets a flag to False:
class My_Thread( threading.Thread):
def __init__(self, thread_id, thread_name, count):
threading.Thread.__init__(self)
self.thread_id = thread_id
self.thread_name = thread_name
self.count = count
def run(self):
do_job(self.thread_name, self.thread_id, self.count)
def do_job(t_name, t_id, count):
while not get_kill():
print "It is "+str(time.time())+" and I am "+str(t_name)
print get_kill()
time.sleep(count)
kill = False
def get_kill():
return kill
def set_kill(state):
kill = state
if __name__ == '__main__':
a = My_Thread(1, "Thread-1", 2)
b = My_Thread(2, "Thread-2", 1)
a.start()
b.start()
while(True):
try:
pass
except KeyboardInterrupt,ki:
set_kill(True)
sys.exit(0)
But the value is never read as changed in both threads and they don't exit. Why is this value not properly read from threads?

The problem
In set_kill(), you are creating a new local variable kill setting it to state, and returning from the function. You are not actually updating the value of kill in the global scope.
To do that, you would need to have:
def set_kill(state):
global kill
kill = state
A better way
Using globals like that is generally considered bad practice, you probably want to convert your kill variable and functions into an object, to encapsulate that data and behaviour together:
class Kill(object):
kill = False
def get(self):
return self.kill
def set(self, value):
self.kill = value
Which you would use like this:
class MyThread(Thread):
def __init__(self, thread_id, thread_name, count, kill):
self.kill = kill
...
def do_job(self, ...):
while not self.kill.get():
...
if __name__ == '__main__':
kill = Kill()
a = My_Thread(1, "Thread-1", 2, kill)
b = My_Thread(2, "Thread-2", 1, kill)
...
kill.set(True)

Threading lock in python not working as desired

I am trying to protect data inside my thread from the main thread. I have the following code:
lock = threading.Lock()
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
with lock:
print 'acquired'
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
print 'released\n'
ags_list = ['x']
createstuff(ags_list)
rn =random.randint(5,50)
print 'random no:', rn
ags_list[0] = rn
It produces the Output:
acquired
random no: 10
Value: [10], : Thread-1
released
Why does changing the list in main thread cause the list inside another thread to mutate even though it is locked? What can I do to prevent it? Thanks.

because the lock only works out if you're using everywhere you're mutating the list, it's not a magic spell that works everywhere if you call it at only one place.
To protect the list you need to add the lock context on both threads:
lock = threading.Lock()
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
with lock:
print 'thread: acquired'
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
print 'thread released'
ags_list = ['x']
createstuff(ags_list)
with lock:
print 'thread: acquired'
rn =random.randint(5,50)
print 'random no:', rn
ags_list[0] = rn
print 'thread: released'
You could create a thread safe list such as:
class ThreadSafeList(list):
def __init__(self, *args):
super(ThreadSafeList, self).__init__(*args)
self.lock = threading.Lock()
def __setitem__(self, idx, value):
with self.lock:
print 'list acquired'
super(ThreadSafeList, self)[idx] = value
print 'list released'
and then use it:
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
args_list = ThreadSafeList(['x'])
createstuff(args_list)
rn =random.randint(5,50)
print 'random no:', rn
args_list[0] = rn
of course that's only an example that needs to be completed and improved. Here I preferred to focus on the point.
Though you do not need a lock in the thread, because accessing a value from a list is (afaict) an atomic read only action, so the mutation of the list can only happen before or after the value is being accessed within the list, not as it is accessing the value. So in the end you should not have any race issue in your example.
If you were modifying the list's value, or doing a non atomic access to data, then the lock could be useful.
N.B.: in case you thought it could work any other way: the mutex mechanism (implemented through Lock) does not protect data, it protects two threads of execution from executing at the same time. If you assert a lock in Thread A, before asserting the same lock in Thread B, Thread B will wait for Thread A to deassert the lock until doing its job.

In python list is passed by reference, so any changes on that list are reflected anywhere else that list is being used.
Here is a link to docs that might help clear a few more things up.
lock = threading.Lock()
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
with lock:
print 'acquired'
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
print 'released\n'
ags_list = ['x']
# you would need to create a copy of different copy of that list
new_ags_list = ags_list[:] #here
createstuff(ags_list)
rn =random.randint(5,50)
print 'random no:', rn
ags_list[0] = rn

Python threading and handing over values

Im trying to update threads which continuously run with new values every now and then.
class Test:
def __init__(self, num):
#testing reasons
self.num = num
def printloop(self, num):
self.num = num
#running is set to True sometime in the beginning
while running:
print(self.num)
time.sleep(3)
if not running:
print("finished")
def setnum(self, num):
self.num = num
I create threads like this:
t1 = threading.Thread(target=test.printloop,args=("1"))
This works and prints the proper arg.
But how can I update single threads with new values - if needed? Not all of the threads might need to be updated. The setnum method in my class there is obviously not working since it would update the value for all of the threads.
Do I need to limit the thread lifetime and join and wait for them to finish. Then recreating them with new values?
Or should I define a variable for each thread - how do I do that dynamically?
Or is there a better way im not seeing?
Thanks!
Edit:
I suppose i'll end up with something like:
test1 = Test(1)
..
test5 = Test(5)
t1 = threading.Thread(target=test1.printloop,args=("1"))
t5 = threading.Thread(target=test5.printloop,args=("5"))
and then use a method on each to set the Values?

For single integer values you can make Test a subclass of thread (and have run call printloop). Then other threads can call setnum safely. Due to the GIL and the fact that you are setting a single value this is safe, if you were doing a more complex update you would have to wrap setnum and the inner loop in printloop in a lock to prevent race conditions.
EDIT: A simple example
from threading import Thread
from time import sleep
class Output(Thread):
def __init__(self, num):
super(Output, self).__init__()
self.num = num
self.running = False
def run(self):
self.running = True
while self.running:
print self.num
sleep(1)
def stop(self):
self.running = False
def set_num(self, num):
self.num = num
output = Output(0)
output.start()
sleep(3)
output.set_num(1)
sleep(3)
output.stop()
output.join()