I am trying to protect data inside my thread from the main thread. I have the following code:
lock = threading.Lock()
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
with lock:
print 'acquired'
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
print 'released\n'
ags_list = ['x']
createstuff(ags_list)
rn =random.randint(5,50)
print 'random no:', rn
ags_list[0] = rn
It produces the Output:
acquired
random no: 10
Value: [10], : Thread-1
released
Why does changing the list in main thread cause the list inside another thread to mutate even though it is locked? What can I do to prevent it? Thanks.
because the lock only works out if you're using everywhere you're mutating the list, it's not a magic spell that works everywhere if you call it at only one place.
To protect the list you need to add the lock context on both threads:
lock = threading.Lock()
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
with lock:
print 'thread: acquired'
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
print 'thread released'
ags_list = ['x']
createstuff(ags_list)
with lock:
print 'thread: acquired'
rn =random.randint(5,50)
print 'random no:', rn
ags_list[0] = rn
print 'thread: released'
You could create a thread safe list such as:
class ThreadSafeList(list):
def __init__(self, *args):
super(ThreadSafeList, self).__init__(*args)
self.lock = threading.Lock()
def __setitem__(self, idx, value):
with self.lock:
print 'list acquired'
super(ThreadSafeList, self)[idx] = value
print 'list released'
and then use it:
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
args_list = ThreadSafeList(['x'])
createstuff(args_list)
rn =random.randint(5,50)
print 'random no:', rn
args_list[0] = rn
of course that's only an example that needs to be completed and improved. Here I preferred to focus on the point.
Though you do not need a lock in the thread, because accessing a value from a list is (afaict) an atomic read only action, so the mutation of the list can only happen before or after the value is being accessed within the list, not as it is accessing the value. So in the end you should not have any race issue in your example.
If you were modifying the list's value, or doing a non atomic access to data, then the lock could be useful.
N.B.: in case you thought it could work any other way: the mutex mechanism (implemented through Lock) does not protect data, it protects two threads of execution from executing at the same time. If you assert a lock in Thread A, before asserting the same lock in Thread B, Thread B will wait for Thread A to deassert the lock until doing its job.
In python list is passed by reference, so any changes on that list are reflected anywhere else that list is being used.
Here is a link to docs that might help clear a few more things up.
lock = threading.Lock()
def createstuff(data):
t= threading.Thread(target=func, args=(data,))
t.start()
def func(val):
with lock:
print 'acquired'
time.sleep(2)
print ('Value: %s, : %s'%(val, threading.currentThread().getName()))
print 'released\n'
ags_list = ['x']
# you would need to create a copy of different copy of that list
new_ags_list = ags_list[:] #here
createstuff(ags_list)
rn =random.randint(5,50)
print 'random no:', rn
ags_list[0] = rn
Related
I am trying to do an exercise about the use of multi-threading in python. This is the task "Write a program that increments a counter shared by two or more threads up untile a certain threshold. Consider various numbers of threads you can use and various initial values and thresholds. Every thread increases the value of the counter by one, if this is lower than the threashold, every 2 seconds."
My attempt at solving the problem is the following:
from threading import Thread
import threading
import time
lock = threading.Lock()
class para:
def __init__(self, value):
self.para = value
class myT(Thread):
def __init__(self,nome,para, end, lock):
Thread.__init__(self)
self.nome = nome
self.end = end
self.para = para
self.lock = lock
def run(self):
while self.para.para < self.end:
self.lock.acquire()
self.para.para += 1
self.lock.release()
time.sleep(2)
print(self.nome, self.para.para)
para = para(1)
threads = []
for i in range(2):
t = myT('Thread' + str(i), para, 15, lock)
threads.append(t)
for i in range(len(threads)):
threads[i].start()
threads[i].join()
print('End code')
I have found an issue:
for i in range(len(threads)):
threads[i].start()
threads[i].join()
The for cycle makes just one thread start while the others are not started (in fact, the output is just the Thread with name 'Thread0' incresing the variable. While if i type manually:
threads[0].start()
threads[1].start()
threads[0].join()
threads[1].join()
I get the correct output, meanining that both threads are working at the same time
Writing the join outside the for and implementing a for just for the join seems to solve the issue, but i do not completely understand why:
for i in range(len(threads)):
threads[i].start()
for i in range(len(threads)):
threads[i].join()
I wanted to ask here for an explanation of the correct way to solve the task using multi-threading in python
Here's an edit of your code and some observations.
Threads share the same memory space therefore, there's no need to pass the reference to the Lock object - that can be in global space.
The Lock object supports enter and exit and can therefore be used in the style of a work manager.
In the first loop we build a list of all threads and also start them. Once they're all started we use another loop to join them.
So now it looks like this:
from threading import Thread, Lock
class para:
def __init__(self, value):
self.para = value
class myT(Thread):
def __init__(self, nome, para, end):
super().__init__()
self.nome = nome
self.end = end
self.para = para
def run(self):
while self.para.para < self.end:
with LOCK:
self.para.para += 1
print(self.nome, self.para.para)
para = para(1)
LOCK = Lock()
threads = []
NTHREADS = 2
for i in range(NTHREADS):
t = myT(f'Thread-{i}', para, 15)
threads.append(t)
t.start()
for t in threads:
t.join()
print('End code')
I have the below sample pretty basic code for working with conditions in Python:
import threading
import random
import time
class Producer(threading.Thread):
"""
Produces random integers to a list
"""
def __init__(self, integers, condition):
"""
Constructor.
#param integers list of integers
#param condition condition synchronization object
"""
threading.Thread.__init__(self)
self.integers = integers
self.condition = condition
def run(self):
"""
Thread run method. Append random integers to the integers list
at random time.
"""
while True:
integer = random.randint(0, 256)
self.condition.acquire()
print 'condition acquired by %s' % self.name
self.integers.append(integer)
print '%d appended to list by %s' % (integer, self.name)
print 'condition notified by %s' % self.name
self.condition.notify()
print 'condition released by %s' % self.name
self.condition.release()
time.sleep(1)
class Consumer(threading.Thread):
"""
Consumes random integers from a list
"""
def __init__(self, integers, condition):
"""
Constructor.
#param integers list of integers
#param condition condition synchronization object
"""
threading.Thread.__init__(self)
self.integers = integers
self.condition = condition
def run(self):
"""
Thread run method. Consumes integers from list
"""
while True:
self.condition.acquire()
print 'condition acquired by %s' % self.name
while True:
if self.integers:
integer = self.integers.pop()
print '%d popped from list by %s' % (integer, self.name)
break
print 'condition wait by %s' % self.name
self.condition.wait()
print 'condition released by %s' % self.name
self.condition.release()
def main():
integers = []
condition = threading.Condition()
t1 = Producer(integers, condition)
t2 = Consumer(integers, condition)
t1.start()
t2.start()
t1.join()
t2.join()
if __name__ == '__main__':
main()
As per my understanding when the consumer calls the wait() method it would release the condition and go into sleep.
When the producer notifies the threads after it calls notify() it seems that neither of the consumers are reacquiring the condition before they try to pop from the integer list.
Is this not a race condition ?
The consumers do not need to reacquire the condition after being woken up from wait() because they do not release it until after resuming from wait().
What they release is a lock that is always associated with the condition, either explicitly or implicitly.
From the docs:
A condition variable is always associated with some kind of lock; this can be passed in or one will be created by default. [...] The lock is part of the condition object: you don’t have to track it separately.
The lock is acquired and released implictly by acquiring / releasing the condition, as well as when calling wait() resp. waking up from it.
The acquire() and release() methods also call the corresponding methods of the associated lock.
[..] The wait() method releases the lock, and then blocks until another thread awakens it by calling notify() or notify_all(). Once awakened, wait() re-acquires the lock and returns.
So there is always a maximum of one thread that can hold the lock, and thus modify the shared resource safely at any given point in time.
I have a very simple threading example using Python 3.4.2. In this example I am creating a five threads that just returns the character string "Result" and appends it to an array titled thread. In another for loop iterated five times the threads are joined to the term x. I am trying to print the result x, which should yield a list that looks like ['Resut','Result','Result','Result','Result'] but instead the print command only yields the title of the thread and the fact that it is closed. Im obviously misunderstanding how to use threads in python. If someone could provide an example of how to adequately complete this test case I would be very grateful.
import threading
def Thread_Test():
return ("Result")
number = 5
threads = []
for i in range(number):
Result = threading.Thread(target=Thread_Test)
threads.append(Result)
Result.start()
for x in threads:
x.join()
print (x)
There is a difference between creating a thread and trying to get values out of a thread. Generally speaking, you should never try to use return in a thread to provide a value back to its caller. That is not how threads work. When you create a thread object, you have to figure out a different way of get any values calculated in the thread to some other part of your program. The following is a simple example showing how values might be returned using a list.
#! /usr/bin/env python3
import threading
def main():
# Define a few variables including storage for threads and values.
threads_to_create = 5
threads = []
results = []
# Create, start, and store all of the thread objects.
for number in range(threads_to_create):
thread = threading.Thread(target=lambda: results.append(number))
thread.start()
threads.append(thread)
# Ensure all threads are done and show the results.
for thread in threads:
thread.join()
print(results)
if __name__ == '__main__':
main()
If you absolutely insist that you must have the ability to return values from the target of a thread, it is possible to override some methods in threading.Thread using a child class to get the desired behavior. The following shows more advanced usage and demonstrates how multiple methods require a change in case someone desires to inherit from and override the run method of the new class. This code is provided for completeness and probably should not be used.
#! /usr/bin/env python3
import sys as _sys
import threading
def main():
# Define a few variables including storage for threads.
threads_to_create = 5
threads = []
# Create, start, and store all of the thread objects.
for number in range(threads_to_create):
thread = ThreadWithReturn(target=lambda: number)
thread.start()
threads.append(thread)
# Ensure all threads are done and show the results.
print([thread.returned for thread in threads])
class ThreadWithReturn(threading.Thread):
def __init__(self, group=None, target=None, name=None,
args=(), kwargs=None, *, daemon=None):
super().__init__(group, target, name, args, kwargs, daemon=daemon)
self.__value = None
def run(self):
try:
if self._target:
return self._target(*self._args, **self._kwargs)
finally:
del self._target, self._args, self._kwargs
def _bootstrap_inner(self):
try:
self._set_ident()
self._set_tstate_lock()
self._started.set()
with threading._active_limbo_lock:
threading._active[self._ident] = self
del threading._limbo[self]
if threading._trace_hook:
_sys.settrace(threading._trace_hook)
if threading._profile_hook:
threading. _sys.setprofile(threading._profile_hook)
try:
self.__value = True, self.run()
except SystemExit:
pass
except:
exc_type, exc_value, exc_tb = self._exc_info()
self.__value = False, exc_value
if _sys and _sys.stderr is not None:
print("Exception in thread %s:\n%s" %
(self.name, threading._format_exc()), file=_sys.stderr)
elif self._stderr is not None:
try:
print((
"Exception in thread " + self.name +
" (most likely raised during interpreter shutdown):"), file=self._stderr)
print((
"Traceback (most recent call last):"), file=self._stderr)
while exc_tb:
print((
' File "%s", line %s, in %s' %
(exc_tb.tb_frame.f_code.co_filename,
exc_tb.tb_lineno,
exc_tb.tb_frame.f_code.co_name)), file=self._stderr)
exc_tb = exc_tb.tb_next
print(("%s: %s" % (exc_type, exc_value)), file=self._stderr)
finally:
del exc_type, exc_value, exc_tb
finally:
pass
finally:
with threading._active_limbo_lock:
try:
del threading._active[threading.get_ident()]
except:
pass
#property
def returned(self):
if self.__value is None:
self.join()
if self.__value is not None:
valid, value = self.__value
if valid:
return value
raise value
if __name__ == '__main__':
main()
please find the below simple example for queue and threads,
import threading
import Queue
import timeit
q = Queue.Queue()
number = 5
t1 = timeit.default_timer()
# Step1: For example, we are running multiple functions normally
result = []
def fun(x):
result.append(x)
return x
for i in range(number):
fun(i)
print result ," # normal result"
print (timeit.default_timer() - t1)
t2 = timeit.default_timer()
#Step2: by using threads and queue
def fun_thrd(x,q):
q.put(x)
return
for i in range(number):
t1 = threading.Thread(target = fun_thrd, args=(i,q))
t1.start()
t1.join()
thrd_result = []
while True:
if not q.empty():
thrd_result.append(q.get())
else:
break
print thrd_result , "# result with threads involved"
print (timeit.default_timer() - t2)
t3 = timeit.default_timer()
#step :3 if you want thread to be run without depending on the previous thread
threads = []
def fun_thrd_independent(x,q):
q.put(x)
return
def thread_indep(number):
for i in range(number):
t = threading.Thread(target = fun_thrd_independent, args=(i,q))
t.start()
threads.append(t)
thread_indep(5)
for j in threads:
j.join()
thread_indep_result = []
while True:
if not q.empty():
thread_indep_result.append(q.get())
else:
break
print thread_indep_result # result when threads are independent on each other
print (timeit.default_timer() - t3)
output:
[0, 1, 2, 3, 4] # normal result
3.50475311279e-05
[0, 1, 2, 3, 4] # result with threads involved
0.000977039337158
[0, 1, 2, 3, 4] result when threads are independent on each other
0.000933170318604
It will hugely differ according to the scale of the data
Hope this helps, Thanks
I need to use 3 threads to print array items sequentially using Python.
Each Thread will print one array item.
I need the threads to sleep for a random number of seconds and then print the item.
This function will be executed N times. This N value is given by the user.
The items must be printed on a specific order, which means I have to somehow block the other threads to execute while the previous one is not done.
I've been trying a lot of different solutions but I can't figure out how to make it work.
I've tried to use Semaphores, Lock and Events but without success on the synchronization. In all cases it would print the sequence randomly, according to the time.sleep and not with the sequence itself. How can I block the thread from executing the function and check if the previous thread was finished in order to allow the sequence to work?
Which tool should I use to make it work? Any help is appreciated.
class myThread(threading.Thread):
def __init__(self, group=None, target=None, name=None,
args=(), kwargs=None, verbose=None):
super(myThread,self).__init__()
self.target = target
self.name = name
return
def run(self):
while True:
if not q.empty():
semaphore.acquire()
try:
time_sleep = random.randrange(0,10)
print "thread " + self.name + ". Dormir por " + str(time_sleep) + " segundos"
time.sleep(time_sleep)
print cores[int(self.name)]
if int(self.name) == len(cores) - 1:
item = q.get()
print 'Executou sequencia ' + str(item + 1) + ' vezes. Ainda irá executar ' + str(q.qsize()) + ' vezes'
e.set()
finally:
semaphore.release()
if int(self.name) != len(cores) - 1:
e.wait()
return
if __name__ == '__main__':
for i in range(2):
q.put(i)
for i in range(3):
t = myThread(name=i)
t.start()
There are many, many approaches to this. A simple one is to use a shared Queue of numbers.
Each thread can sleep for however long it wants to, take a number from the queue when it wakes up, and print it. They will come out in the order they were pushed to the queue.
If your numbers are sequential, or can generated dynamically, you can also do it in constant-memory using a shared counter, as described in this answer.
If you didn't care about order you could just use a lock to synchronize access. In this case, though, how about a list of events. Each thread gets its own event slot and hands it to the next event in the list when done. This scheme could be fancied up by returning a context manager so that you don't need to release explicitly.
import threading
class EventSlots(object):
def __init__(self, num_slots):
self.num_slots = num_slots
self.events = [threading.Event() for _ in range(num_slots)]
self.events[0].set()
def wait_slot(self, slot_num):
self.events[slot_num].wait()
self.events[slot_num].clear()
def release_slot(self, slot_num):
self.events[(slot_num + 1) % self.num_slots].set()
def worker(event_slots, slot_num):
for i in range(5):
event_slots.wait_slot(slot_num)
print('slot', slot_num, 'iteration', i)
event_slots.release_slot(slot_num)
NUM = 3
slots = EventSlots(NUM)
threads = []
for slot in range(NUM):
t = threading.Thread(target=worker, args=(slots, slot))
t.start()
threads.append(t)
for t in threads:
t.join()
I'm working on a project in Python using the "thread" module.
How can I make a global variable (in my case I need it to be True or False) that all the threads in my project (about 4-6) can access?
We can define the variable outside the thread classes and declare it global inside the methods of the classes.
Please see below trivial example which prints AB alternatively. Two variables flag and val are shared between two threads Thread_A and Thread_B. Thread_A prints val=20 and then sets val to 30. Thread_B prints val=30, since val is modified in Thread_A. Thread_B then sets val to 20 which is again used in Thread_A. This demonstrates that variable val is shared between two threads. Similarly variable flag is also shared between two threads.
import threading
import time
c = threading.Condition()
flag = 0 #shared between Thread_A and Thread_B
val = 20
class Thread_A(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
global flag
global val #made global here
while True:
c.acquire()
if flag == 0:
print "A: val=" + str(val)
time.sleep(0.1)
flag = 1
val = 30
c.notify_all()
else:
c.wait()
c.release()
class Thread_B(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
global flag
global val #made global here
while True:
c.acquire()
if flag == 1:
print "B: val=" + str(val)
time.sleep(0.5)
flag = 0
val = 20
c.notify_all()
else:
c.wait()
c.release()
a = Thread_A("myThread_name_A")
b = Thread_B("myThread_name_B")
b.start()
a.start()
a.join()
b.join()
Output looks like
A: val=20
B: val=30
A: val=20
B: val=30
A: val=20
B: val=30
A: val=20
B: val=30
Each thread prints the value which was modified in another thread.
With no clue as to what you are really trying to do, either go with nio's approach and use locks, or consider condition variables:
From the docs
# Consume one item
cv.acquire()
while not an_item_is_available():
cv.wait()
get_an_available_item()
cv.release()
# Produce one item
cv.acquire()
make_an_item_available()
cv.notify()
cv.release()
You can use this to let one thread tell another a condition has been met, without having to think about the locks explicitly. This example uses cv to signify that an item is available.
How about using a threading.Event object per this description?
For example in the script below, worker1 and worker2 share an Event, and when worker2 changes its value this is seen by worker1:
import time
from threading import Thread, Event
shared_bool = Event()
def worker1(shared_bool):
while True:
if shared_bool.is_set():
print("value is True, quitting")
return
else:
print("value is False")
time.sleep(1)
def worker2(shared_bool):
time.sleep(2.5)
shared_bool.set()
t1 = Thread(target=worker1, args=(shared_bool, ))
t2 = Thread(target=worker2, args=(shared_bool, ))
t1.start()
t2.start()
t1.join()
t2.join()
Prints out:
value is False
value is False
value is False
value is True, quitting