I have a problem trying to link threads memory. I want that the counter shares the memory between threads that all of them only count to a certain number(100 in this case) and finally it is returned to the main thread. The problem is that even with lock all of the threads have a single count
import threading
from threading import Thread, Lock
import time
import multiprocessing
import random
def create_workers(n_threads, counter):
# counter = 0
workers = []
for n in range(n_threads):
worker = DataCampThread('Thread - ' + str(n), counter)
workers.append(worker)
for worker in workers:
worker.start()
for worker in workers:
worker.join()
return counter
def thread_delay(thread_name, num, delay):
num += 1
time.sleep(delay)
print(thread_name, '-------->', num)
return num
class DataCampThread(Thread):
def __init__(self, name, cou):
Thread.__init__(self)
self.name = name
self.counter = cou
delay = random.randint(1, 2)
self.delay = delay
self.lock = Lock()
def run(self):
print('Starting Thread:', self.name)
while self.counter < 100:
self.lock.acquire()
self.counter = thread_delay(self.name, self.counter, self.delay)
self.lock.release()
print('Execution of Thread:', self.name, 'is complete!')
if __name__ == '__main__':
# create the agent
n_threads = 3#multiprocessing.cpu_count()
counter = 0
create_workers(n_threads, counter)
print(counter)
print("Thread execution is complete!")
As I mentioned in the comments, I'm not really sure what you're trying to do — but here's an uninformed guess to (hopefully) expedite things.
Based on your response to the initial version of my answer about wanting to avoid a global variable, the counter is now a class attribute that will automatically be shared by all instances of the class. Each thread has its own name and randomly selected amount of time it delays between updates to the shared class attribute named counter.
Note: The test code redefines the print() function to prevent it from being used by more that one thread at a time.
import threading
from threading import Thread, Lock
import time
import random
MAXVAL = 10
class DataCampThread(Thread):
counter = 0 # Class attribute.
counter_lock = Lock() # Control concurrent access to shared class attribute.
def __init__(self, name):
super().__init__() # Initialize base class.
self.name = name
self.delay = random.randint(1, 2)
def run(self):
print('Starting Thread:', self.name)
while True:
with self.counter_lock:
if self.counter >= MAXVAL:
break # Exit while loop (also releases lock).
# self.counter += 1 # DON'T USE - would create an instance-level attribute.
type(self).counter += 1 # Update class attribute.
print(self.name, '-------->', self.counter)
time.sleep(self.delay)
print('Execution of Thread:', self.name, 'is complete!')
def main(n_threads, maxval):
''' Create and start worker threads, then wait for them all to finish. '''
workers = [DataCampThread(name=f'Thread #{i}') for i in range(n_threads)]
for worker in workers:
worker.start()
# Wait for all treads to finish.
for worker in workers:
worker.join()
if __name__ == '__main__':
import builtins
def print(*args, **kwargs):
''' Redefine print to prevent concurrent printing. '''
with print.lock:
builtins.print(*args, **kwargs)
print.lock = Lock() # Function attribute.
n_threads = 3
main(n_threads, MAXVAL)
print()
print('Thread execution is complete!')
print('final counter value:', DataCampThread.counter)
Sample output:
Starting Thread: Thread #0
Starting Thread: Thread #1
Thread #0 --------> 1
Starting Thread: Thread #2
Thread #1 --------> 2
Thread #2 --------> 3
Thread #1 --------> 4
Thread #0 --------> 5
Thread #2 --------> 6
Thread #2 --------> 7
Thread #1 --------> 8
Thread #0 --------> 9
Thread #2 --------> 10
Execution of Thread: Thread #1 is complete!
Execution of Thread: Thread #0 is complete!
Execution of Thread: Thread #2 is complete!
Thread execution is complete!
final counter value: 10
Related
I am facing some issues while implementing multi-threading in python. The issue is very specific to my use case. Having gone through numerous posts on the same, I deployed the most widely suggested/used method for doing so.
I start by defining my thread class as follows.
class myThread(Thread):
def __init__(self, graphobj, q):
Thread.__init__(self)
self.graphobj = graphobj
self.q = q
def run(self):
improcess(self.graphobj, self.q)
Post which I define my function that does all the processing required.
def improcess(graphobj, q):
while not exitFlag:
queueLock.acquire()
if not q.empty():
photo_id = q.get()
queueLock.release()
# Complete processing
else:
queueLock.release()
Now comes the part where I am stuck. I am able to run the below mentioned code exactly as it is without any issues. However if I try and wrap the same in a function as such it breaks down.
def train_control(graphobj, photo_ids):
workQueue = Queue(len(photo_ids))
for i in range(1,5):
thread = myThread(graphobj=graphobj, q=workQueue)
thread.start()
threads.append(thread)
queueLock.acquire()
for photo_id in photo_ids:
workQueue.put(photo_id)
queueLock.release()
while not workQueue.empty():
pass
exitFlag = 1
for t in threads:
t.join()
By breaking down I mean that the threads complete their work but they don't stop waiting i.e. the exitFlag is never set to 1. I am unsure as to how to make this work.
Unfortunately the design of our systems is such that this piece of codes needs to be wrapped in a function which can be invoked by another module, so pulling it out is not really an option.
Looking forward to hearing from experts on this. Thanks in advance.
Edit : Forgot to mention this in the first draft. I globally initialize exitFlag and set its value to 0.
Below is the minimum, verifiable code snippet that I created to capture this problem:
import threading
import Queue
globvar01 = 5
globvar02 = 7
exitFlag = 0
globlist = []
threads = []
queueLock = threading.Lock()
workQueue = Queue.Queue(16)
class myThread(threading.Thread):
def __init__(self, threadID, q):
threading.Thread.__init__(self)
self.threadID = threadID
self.q = q
def run(self):
print "Starting thread " + str(self.threadID)
myfunc(self.threadID, self.q)
print "Exiting thread " + str(self.threadID)
def myfunc(threadID, q):
while not exitFlag:
queueLock.acquire()
if not workQueue.empty():
thoughtnum = q.get()
queueLock.release()
print "Processing thread " + str(threadID)
if (thoughtnum < globvar01):
globlist.append([1,2,3])
elif (thoughtnum < globvar02):
globlist.append([2,3,4])
else:
queueLock.release()
def controlfunc():
for i in range(1,5):
thread = myThread(i, workQueue)
thread.start()
threads.append(thread)
queueLock.acquire()
for i in range(1,11):
workQueue.put(i)
queueLock.release()
# Wait for queue to empty
while not workQueue.empty():
pass
exitFlag = 1
# Wait for all threads to complete
for t in threads:
t.join()
print "Starting main thread"
controlfunc()
print "Exiting Main Thread"
From your MCVE, the only thing missing is:
while not workQueue.empty():
pass
global exitFlag # Need this or `exitFlag` is a local variable only.
exitFlag = 1
You could eliminate the queueLock and the exitFlag, however, by using a sentinel value in the Queue to shut down the worker threads, and it eliminates the spin-waiting. Worker threads will sleep on a q.get() and the main thread won't have to spin-wait for an empty queue:
#!python2
from __future__ import print_function
import threading
import Queue
debug = 1
console = threading.Lock()
def tprint(*args,**kwargs):
if debug:
name = threading.current_thread().getName()
with console:
print('{}: '.format(name),end='')
print(*args,**kwargs)
globvar01 = 5
globvar02 = 7
globlist = []
threads = []
workQueue = Queue.Queue(16)
class myThread(threading.Thread):
def __init__(self, threadID, q):
threading.Thread.__init__(self)
self.threadID = threadID
self.q = q
def run(self):
tprint("Starting thread " + str(self.threadID))
myfunc(self.threadID, self.q)
tprint("Exiting thread " + str(self.threadID))
def myfunc(threadID, q):
while True:
thoughtnum = q.get()
tprint("Processing thread " + str(threadID))
if thoughtnum is None:
break
elif thoughtnum < globvar01:
globlist.append([1,2,3])
elif thoughtnum < globvar02:
globlist.append([2,3,4])
def controlfunc():
for i in range(1,5):
thread = myThread(i, workQueue)
thread.start()
threads.append(thread)
for i in range(1,11):
workQueue.put(i)
# Wait for all threads to complete
for t in threads:
workQueue.put(None)
for t in threads:
t.join()
tprint("Starting main thread")
controlfunc()
tprint("Exiting Main Thread")
Output:
MainThread: Starting main thread
Thread-1: Starting thread 1
Thread-2: Starting thread 2
Thread-3: Starting thread 3
Thread-4: Starting thread 4
Thread-1: Processing thread 1
Thread-2: Processing thread 2
Thread-3: Processing thread 3
Thread-4: Processing thread 4
Thread-1: Processing thread 1
Thread-2: Processing thread 2
Thread-3: Processing thread 3
Thread-4: Processing thread 4
Thread-1: Processing thread 1
Thread-2: Processing thread 2
Thread-3: Processing thread 3
Thread-4: Processing thread 4
Thread-1: Processing thread 1
Thread-2: Processing thread 2
Thread-3: Exiting thread 3
Thread-4: Exiting thread 4
Thread-1: Exiting thread 1
Thread-2: Exiting thread 2
MainThread: Exiting Main Thread
You need to make sure exitFlag is set to 0 (False) before spawning any threads otherwise in impprocess() they won't do anything and the queue will remain non-empty.
This problem could happen if you have exitFlag as a global and it's not cleared from a previous run.
I'm trying to get a hold of multi-threading in Python.
I wrote this snippet of code:
import requests
import threading
from time import time,sleep
start_time = time()
class myThread(threading.Thread):
def __init__(self,threadID,name):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
def run(self):
print(self.name + " Starting")
start = time()
url = 'https://stackoverflow.com'
for i in range(20):
res = requests.post(url,timeout=5)
end = time()
print("Final Thread Time: " + str(end-start) + 'For Thread {}'.format(self.threadID))
threads = []
threadID = 1
for i in range(1,5):
thread = myThread(threadID,'thread{}'.format(threadID))
thread.start()
threads.append(thread)
threadID += 1
for t in threads:
t.join()
final_time_end = time()
print("Final Time: " + str(start_time - final_time_end))
So basically it's sending 20 post requests to a url in each thread.
The result is:
Final Thread Time: 6.0695300102 For Thread 3
Final Thread Time: 6.8553800583 For Thread 1
Final Thread Time: 6.9735219479 For Thread 5
Final Thread Time: 6.5822350979 For Thread 4
Final Thread Time: 11.330765152 For Thread 2
I have a dual core CPU,what I don't understand is why almost each time I run this script,4 threads finish at nearly same time(6s),but 1 thread takes about 11s(btw sometimes they all finish at the same time).
This happens when I run this code with 4 or 3 threads as well.the last one usually takes more time to finish.
What's happening in the last thread that takes more time?
Am very much new to python. So what i might be asking may not be correct. What am looking to do is. Create a thread from mainss and start the thread. When the thread is started i want it to access the variable of the mainss class from where thread is created and Modify the variable values. And i want the execution of mainss to sleep until thread modifies one of its variable value. How can i achieve this? Here is the code i tried below. Comment in the code of mythread.py class is where I need to modify the value of count variable of mainss class
main.py
#!/usr/bin/python
import time
from myThread import myThread
class mainss():
def __init__(self):
print "s"
def callThread(self):
global count
count = 1
# Create new threads
thread1 = myThread(1, "Thread-1", 1, count)
thread1.start()
# time.sleep(10) until count value is changed by thread to 3
print "Changed Count value%s " % count
print "Exiting"
m = mainss()
m.callThread()
myThread.py
#!/usr/bin/python
import threading
import time
exitFlag = 0
class myThread (threading.Thread):
def __init__(self, threadID, name, counter, count):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.counter = counter
self.count = count
def run(self):
print_time(self.name, 1, 5, self.count)
def print_time(threadName, delay, counter, count):
from main import mainss
while counter:
if exitFlag:
threadName.exit()
time.sleep(delay)
count = count + 1
print "count %s" % (count)
# here i want to modify count of mainss class
counter -= 1
Thanks in advance
Using Multiprocessing, a Manager dictionary is used to communicate between or to processes https://pymotw.com/3/multiprocessing/communication.html#managing-shared-state Note that the Manager dictionary can be changed while the process is running. Multiprocessing also has a wait feature https://pymotw.com/3/multiprocessing/communication.html#signaling-between-processes
I would use a threading.Event and a Queue
Something like this, (Please note that i didn't test this myself and obviously you gonna have to make some changes.)
main.py
import Queue
import threading
from myThread import myThread
class mainss:
def __init__(self):
self.queue = Queue.Queue()
self.event = threading.Event()
def callThread(self):
self.queue.put(1) # Put a value in the queue
t = myThread(self.queue, self.event)
t.start()
self.event.wait() # Wait for the value to update
count = self.queue.get()
print "Changed Count value %s" % count
if __name__ == '__main__':
m = mainss()
m.callThread()
myThread.py
import threading
class myThread(threading.Thread):
def __init__(self, queue, event):
super(myThread, self).__init__()
self.queue = queue
self.event = event
def run(self):
while True:
count = self.queue.get() # Get the value (1)
count += 1
print "count %s" % (count)
self.queue.put(count) # Put updated value
self.event.set() # Notify main thread
break
I have this example code to explain my problem:
import threading
import time
class thread1(threading.Thread):
def __init__(self, lock):
threading.Thread.__init__(self)
self.daemon = True
self.start()
self.lock = lock
def run(self):
while True:
self.lock.acquire(True)
print ('write done by t1')
self.lock.release()
class thread2(threading.Thread):
def __init__(self, lock):
threading.Thread.__init__(self)
self.daemon = True
self.start()
self.lock = lock
def run(self):
while True:
self.lock.acquire(True)
print ('write done by t2')
self.lock.release()
if __name__ == '__main__':
lock = threading.Lock()
t1 = thread1(lock)
t2 = thread2(lock)
lock.acquire(True)
counter = 0
while True:
print("main...")
counter = counter + 1
if(counter==5 or counter==10):
lock.release() # Here I want to unlock both threads to run just one time and then wait until I release again
time.sleep(1)
t1.join()
t2.join()
What I'm having some issues is the following:
I want to have two threads (thread1 and thread2) that are launched at the beginning of the program, but they should wait until the main() counter reaches 5 or 10.
When the main() counter reaches 5 or 10, it should signal/trigger/unlock the threads, and both threads should run just once and then wait until a new unlock.
I was expecting the code to have the following output (Each line is 1 second running):
main...
main...
main...
main...
main...
write done by t1
write done by t2
main...
main...
main...
main...
main...
write done by t1
write done by t2
Instead I have a different behaviour, such as starting with:
write done by t1
write done by t1
write done by t1
write done by t1
(etc)
And after 5 seconds the
write done by t2
A lot of times...
Can someone help me explaining what is wrong and how can I improve this?
In __init__() of thread1 and thread2, start() is invoked before self.lock is assigned.
t1 and t2 are created before the main thread acquires the lock. That makes these two threads start printing before the main thread locks them. It is the reason your code print the first several lines of "write done by x".
After the counter reaches 5, the main thread releases the lock, but it never locks it again. That makes t1 and t2 keep running.
It never quits unless you kill it...
I suggest you to use Condition Object instead of Lock.
Here is an example based on your code.
import threading
import time
class Thread1(threading.Thread):
def __init__(self, condition_obj):
super().__init__()
self.daemon = True
self.condition_obj = condition_obj
self.start()
def run(self):
with self.condition_obj:
while True:
self.condition_obj.wait()
print('write done by t1')
class Thread2(threading.Thread):
def __init__(self, condition_obj):
super().__init__()
self.daemon = True
self.condition_obj = condition_obj
self.start()
def run(self):
with self.condition_obj:
while True:
self.condition_obj.wait()
print('write done by t2')
if __name__ == '__main__':
condition = threading.Condition()
t1 = Thread1(condition)
t2 = Thread2(condition)
counter = 0
while True:
print("main...")
counter += 1
if counter == 5 or counter == 10:
with condition:
condition.notify_all()
time.sleep(1)
t1.join()
t2.join()
I'm learning multithread in python. I write some code to practice it
import threading
import time
Total = 0
class myThead(threading.Thread):
def __init__(self, num):
threading.Thread.__init__(self)
self.num = num
self.lock = threading.Lock()
def run(self):
global Total
self.lock.acquire()
print "%s acquired" % threading.currentThread().getName()
for i in range(self.num):
Total += 1
print Total
print "%s released" % threading.currentThread().getName()
self.lock.release()
t1 = myThead(100)
t2 = myThead(100)
t1.start()
t2.start()
if i pass 100 to thread t1 and t2, they go correctly.
Thread-1 acquired
100
Thread-1 released
Thread-2 acquired
200
Thread-2 released
But when i try with bigger numbler. For example, i pass 10000. It prints out unexpected output.
Thread-1 acquired
Thread-2 acquired
14854
Thread-1 released
15009
Thread-2 released
I try many times but no thing changes. So i think Lock object in python have timeout. If Lock acquire for long time, it will allow other thread can go. Can anyone explain me about it. Thank you!
No, locks do not have a timeout. What is happening is that they are not actually sharing the same lock, as a new one is created every time you instantiate the object in the init method. If all instances of that class will always share the same lock, then you could throw it in as a class property. However, explicit is better than implicit. I would personally put the lock as an argument in the init method. Something like this.
import threading
import time
Total = 0
class myThead(threading.Thread):
def __init__(self, num, lock):
threading.Thread.__init__(self)
self.num = num
self.lock = lock
def run(self):
global Total
self.lock.acquire()
print "%s acquired" % threading.currentThread().getName()
for i in range(self.num):
Total += 1
print Total
print "%s released" % threading.currentThread().getName()
self.lock.release()
threadLock = threading.Lock()
t1 = myThead(100, threadLock)
t2 = myThead(100, threadLock)
t1.start()
t2.start()
That way both instances of the class share the same lock.
Each thread gets its own lock, so acquiring t1's lock doesn't stop t2 from acquiring its own lock.
Perhaps you could make lock a class attribute, so all instances of myThread share one.
class myThead(threading.Thread):
lock = threading.Lock()
def __init__(self, num):
threading.Thread.__init__(self)
self.num = num
Result:
Thread-1 acquired
10000
Thread-1 released
Thread-2 acquired
20000
Thread-2 released