from threading import Thread
import threading
import time
def procesa1():
lock = threading.Lock()
lock.acquire()
for i in range(3):
print(threading.get_ident())
lock.release()
if __name__ == "__main__":
hilos = []
for i in range(5):
hilos.append(Thread(target=procesa1))
for thread in hilos:
thread.start()
The 3 print in each thread should come out in a row, shouldn't they?
A thread WINS the lock, prints its 3 outputs in a row (not interspersed with those of the other threads) and then the 3 outputs of the next thread.
This is an example of the multiple random outputs:
13108
13108
13108
12780
12780
12780
7756 // what?
6844
6844
6844
7756 // what?
7756 // what?
11936 //good
11936 //good
11936 //good, they are consecutive
The lock is supposed to be for a critical "atomic" section accessed by one thread AT A TIME.
What am I losing?
The lock doesn't help to synchronize threads because each thread is creating its own lock. You need to create a single lock and pass it as an argument to each thread, so they can share it.
This works as expected (I simplified the syntax using with):
def procesa1(lock):
with lock:
for i in range(3):
print(threading.get_ident())
if __name__ == "__main__":
lock = threading.Lock()
hilos = []
for i in range(5):
hilos.append(Thread(target=procesa1, args=(lock,)))
for thread in hilos:
thread.start()
Output:
34360446720
34360446720
34360446720
34360360224
34360360224
34360360224
[....]
Related
I am working on python 3 and my class is as below.
class MyClass():
def values(self):
***values***
i =0
def check_values(self):
for i in ValueList[i:i+1]:
self.server_connect()
new_value = self.update.values(i)
def run(self):
self.check_values()
if __name__ == "__main__"
format1 = "%(asctime)s: %(message)s"
logging.basicConfig(format=format1, level=logging.INFO,
datefmt="%H:%M:%S")
for i in range(4):
thread = threading.Thread(target=MyClass().run())
threads.append(thread)
i += 1
print("the %s thread is running", thread)
thread.start()
There are no threads getting created but code works.
I am not able to catch what I am doing wrong here.
EDIT
First, I would like to thank you for response and time given for the answer.
I have to update code and inherit other class as per new update from team as below.
class MyClass(MainServer):
Now, the server has it's own run function as below.
class MainServer(object):
***constructor***
***other functions ***
def run(self):
self.add_arguments()
self.parse_arguments()
self.check_values()
Now, without run(), my code is not properly running.
while including run() as below.
*** main ***
update_perform = MyClass()
for i range(4):
thread = threading.Thread(target=Myclass().run()) <-- code starts from here
threads.append(thread)
i += 1
print("the %s thread is running", thread)
thread.start() <-- not reaching till here
As per my knowledge I will require thread.start() to start threading. So I have tried below option
class MyClass(MainServer):
***code as above***
def check_values(self):
self.server_authenticate()
update_value = self.update.values()
def run(self):
self.server_connect()
i = 0
threads = list()
for i in ValueList[i:i+1]:
print("Updating the value = ", i)
thread = threading.Thread(target=check_values(), args=[i])
thread.start()
i += 1
print("Currently running thread", thread)
threads.append(thread)
for thread in threads:
thread.join()
Here thread is executing from start and in print I can see as below
for threading :-
Currently running threads = <Thread(Thread-8, stopped 14852)>
But for the value I can see only one is in process as below
for value :-
Updating the value = 10 <- first value
So, now threads may be getting created but the values are not getting executed in parallel.
Which I am not able to figure out.
modify the run function like this
def run(self):
self.check_values()
I am new to python and threading. I am trying to run multiple threads at a time. Here is my basic code :
import threading
import time
threads = []
print "hello"
class myThread(threading.Thread):
def __init__(self,i):
threading.Thread.__init__(self)
print "i = ",i
for j in range(0,i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = myThread(i)
thread.start()
While 1 thread is waiting for time.sleep(5) i want another thread to start. In short, all the threads should run parallel.
You might have some misunderstandings on how to subclass threading.Thread, first of all __init__() method is roughly what represents a constructor in Python, basically it'll get executed every time you create an instance, so in your case when thread = myThread(i) executes, it'll block till the end of __init__().
Then you should move your activity into run(), so that when start() is called, the thread will start to run. For example:
import threading
import time
threads = []
print "hello"
class myThread(threading.Thread):
def __init__(self, i):
threading.Thread.__init__(self)
self.i = i
def run(self):
print "i = ", self.i
for j in range(0, self.i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = myThread(i)
thread.start()
P.S. Because of the existence of GIL in CPython, you might not be able to fully take advantages of all your processors if the task is CPU-bound.
Here is an example on how you could use threading based on your code:
import threading
import time
threads = []
print "hello"
def doWork(i):
print "i = ",i
for j in range(0,i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = threading.Thread(target=doWork, args=(i,))
threads.append(thread)
thread.start()
# you need to wait for the threads to finish
for thread in threads:
thread.join()
print "Finished"
import threading
import subprocess
def obj_func(simid):
simid = simid
workingdir = './' +str (simid) # the working directory for the simulation
cmd = './run_delwaq.sh' # cmd is a bash commend to launch the external execution
subprocess.Popen(cmd, cwd=workingdir).wait()
def example_subprocess_files():
num_threads = 4
jobs = []
# Launch the threads and give them access to the objective function
for i in range(num_threads):
workertask = threading.Thread(target=obj_func(i))
jobs.append(workertask)
for j in jobs:
j.start()
for j in jobs:
j.join()
print('All the work finished!')
if __name__ == '__main__':
example_subprocess_files()
This one not works for my case that the task is not print but CPU-Intensive task. The thread are excluded in serial.
I use Queue to provide tasks that threads can work on. After all work is done from Queue, I see the threads are still alive while I expected them being released. Here is my code. You can see the active threads number is increasing after a batch of task(in the same queue) increases from the console. How could I release the threads after a batch of work get done?
import threading
import time
from Queue import Queue
class ThreadWorker(threading.Thread):
def __init__(self, task_queue):
threading.Thread.__init__(self)
self.task_queue = task_queue
def run(self):
while True:
work = self.task_queue.get()
#do some work
# do_work(work)
time.sleep(0.1)
self.task_queue.task_done()
def get_batch_work_done(works):
task_queue = Queue()
for _ in range(5):
t = ThreadWorker(task_queue)
t.setDaemon(True)
t.start()
for work in range(works):
task_queue.put(work)
task_queue.join()
print 'get batch work done'
print 'active threads count is {}'.format(threading.activeCount())
if __name__ == '__main__':
for work_number in range(3):
print 'start with {}'.format(work_number)
get_batch_work_done(work_number)
Do a non blocking read in a loop and use the exception handling to terminate
def run(self):
try:
while True:
work = self.task_queue.get(True, 0.1)
#do some work
# do_work(work)
except Queue.Empty:
print "goodbye"
In the below program I have posted 5 jobs to the queue, but have created only 3 threads. When I run the program, only 3 jobs are completed. How am I supposed to complete all 5 jobs with only 3 threads? Is there a way to the make a thread that has completed its job take the next job?
import time
import Queue
import threading
class worker(threading.Thread):
def __init__(self,qu):
threading.Thread.__init__(self)
self.que=qu
def run(self):
print "Going to sleep.."
time.sleep(self.que.get())
print "Slept .."
self.que.task_done()
q = Queue.Queue()
for j in range(3):
work = worker(q);
work.setDaemon(True)
work.start()
for i in range(5):
q.put(1)
q.join()
print "done!!"
You need to have your worker threads run in a loop. You can use a sentinel value (like None or custom class) to tell the workers to shut down after you've put all your actual worked items in the queue:
import time
import Queue
import threading
class worker(threading.Thread):
def __init__(self,qu):
threading.Thread.__init__(self)
self.que=qu
def run(self):
for item in iter(self.que.get, None): # This will call self.que.get() until None is returned, at which point the loop will break.
print "Going to sleep.."
time.sleep(item)
print "Slept .."
self.que.task_done()
self.que.task_done()
q = Queue.Queue()
for j in range(3):
work = worker(q);
work.setDaemon(True)
work.start()
for i in range(5):
q.put(1)
for i in range(3): # Shut down all the workers
q.put(None)
q.join()
print "done!!"
Another option would be to use a multiprocessing.dummy.Pool, which is a thread pool that Python manages for you:
import time
from multiprocessing.dummy import Pool
def run(i):
print "Going to sleep..."
time.sleep(i)
print "Slept .."
p = Pool(3) # 3 threads in the pool
p.map(run, range(5)) # Calls run(i) for each element i in range(5)
p.close()
p.join()
print "done!!"
I'm about to put this design into use in an application, but I'm fairly new to threading and Queue stuff in python. Obviously the actual application is not for saying hello, but the design is the same - i.e. there is a process which takes some time to set-up and tear down, but I can do multiple tasks in one hit. Tasks will arrive at random times, and often in bursts.
Is this a sensible and thread safe design?
class HelloThing(object):
def __init__(self):
self.queue = self._create_worker()
def _create_worker(self):
import threading, Queue
def worker():
while True:
things = [q.get()]
while True:
try:
things.append(q.get_nowait())
except Queue.Empty:
break
self._say_hello(things)
[q.task_done() for task in xrange(len(things))]
q = Queue.Queue()
n_worker_threads = 1
for i in xrange(n_worker_threads):
t = threading.Thread(target=worker)
t.daemon = True
t.start()
return q
def _say_hello(self, greeting_list):
import time, sys
# setup stuff
time.sleep(1)
# do some things
sys.stdout.write('hello {0}!\n'.format(', '.join(greeting_list)))
# tear down stuff
time.sleep(1)
if __name__ == '__main__':
print 'enter __main__'
import time
hello = HelloThing()
hello.queue.put('world')
hello.queue.put('cruel world')
hello.queue.put('stack overflow')
time.sleep(2)
hello.queue.put('a')
hello.queue.put('b')
time.sleep(2)
for i in xrange(20):
hello.queue.put(str(i))
#hello.queue.join()
print 'finish __main__'
The thread safety is handled by Queue implementation (also you must handle in your _say_hello implementation if it is required).
Burst handler problem: A burst should be handled by a single thread only.(ex: let's say your process setup/teardown takes 10 seconds; at second 1 all threads will be busy with burst from sec 0, on second 5 a new task(or burst) but no thread available to handle them/it). So a burst should be defined by max number of tasks (or maybe "infinite") for a specific time-window. An entry in queue should be a list of tasks.
How can you group burst tasks list?
I provide a solution as code, more easy to explain ...
producer_q = Queue()
def _burst_thread():
while True:
available_tasks = [producer_q.get()]
time.sleep(BURST_TIME_WINDOW)
available_tasks.extend(producer_q.get() # I'm the single consumer, so will be at least qsize elements
for i in range(producer_q.qsize()))
consumer_q.push(available_tasks)
If you want to have a maximum of messages in a burst, you just need to slice the available_tasks in multiple lists.