The following code works for me, the problem is that each thread has to wait until throws to end or at least the perception that I have because when I put the sleep (10) the waiting time is indicated and then continuous.
What I wish is that the haul thread without having to wait for the internal code to run.
It is my code (example):
import threading
from time import sleep
class MyThread(threading.Thread):
def __init__(self, num):
threading.Thread.__init__(self)
self.num = num
def run(self):
print "I'm the thread", self.num
sleep(10)
print "I'm the thread, after 10 seg"
print "I'm the main thread"
for i in range(0, 10):
t = MyThread(i)
t.start()
t.join()
Thanks in advances.
Use 2 for loops: 1 to start the threads and one to wait for them:
# create all threads
ts = [MyThread(i) for i in range(10)]
# start all threads
for t in ts:
t.start()
# wait for all threads
for t in ts:
t.join()
Related
I am new to python and threading. I am trying to run multiple threads at a time. Here is my basic code :
import threading
import time
threads = []
print "hello"
class myThread(threading.Thread):
def __init__(self,i):
threading.Thread.__init__(self)
print "i = ",i
for j in range(0,i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = myThread(i)
thread.start()
While 1 thread is waiting for time.sleep(5) i want another thread to start. In short, all the threads should run parallel.
You might have some misunderstandings on how to subclass threading.Thread, first of all __init__() method is roughly what represents a constructor in Python, basically it'll get executed every time you create an instance, so in your case when thread = myThread(i) executes, it'll block till the end of __init__().
Then you should move your activity into run(), so that when start() is called, the thread will start to run. For example:
import threading
import time
threads = []
print "hello"
class myThread(threading.Thread):
def __init__(self, i):
threading.Thread.__init__(self)
self.i = i
def run(self):
print "i = ", self.i
for j in range(0, self.i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = myThread(i)
thread.start()
P.S. Because of the existence of GIL in CPython, you might not be able to fully take advantages of all your processors if the task is CPU-bound.
Here is an example on how you could use threading based on your code:
import threading
import time
threads = []
print "hello"
def doWork(i):
print "i = ",i
for j in range(0,i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = threading.Thread(target=doWork, args=(i,))
threads.append(thread)
thread.start()
# you need to wait for the threads to finish
for thread in threads:
thread.join()
print "Finished"
import threading
import subprocess
def obj_func(simid):
simid = simid
workingdir = './' +str (simid) # the working directory for the simulation
cmd = './run_delwaq.sh' # cmd is a bash commend to launch the external execution
subprocess.Popen(cmd, cwd=workingdir).wait()
def example_subprocess_files():
num_threads = 4
jobs = []
# Launch the threads and give them access to the objective function
for i in range(num_threads):
workertask = threading.Thread(target=obj_func(i))
jobs.append(workertask)
for j in jobs:
j.start()
for j in jobs:
j.join()
print('All the work finished!')
if __name__ == '__main__':
example_subprocess_files()
This one not works for my case that the task is not print but CPU-Intensive task. The thread are excluded in serial.
I have this example code to explain my problem:
import threading
import time
class thread1(threading.Thread):
def __init__(self, lock):
threading.Thread.__init__(self)
self.daemon = True
self.start()
self.lock = lock
def run(self):
while True:
self.lock.acquire(True)
print ('write done by t1')
self.lock.release()
class thread2(threading.Thread):
def __init__(self, lock):
threading.Thread.__init__(self)
self.daemon = True
self.start()
self.lock = lock
def run(self):
while True:
self.lock.acquire(True)
print ('write done by t2')
self.lock.release()
if __name__ == '__main__':
lock = threading.Lock()
t1 = thread1(lock)
t2 = thread2(lock)
lock.acquire(True)
counter = 0
while True:
print("main...")
counter = counter + 1
if(counter==5 or counter==10):
lock.release() # Here I want to unlock both threads to run just one time and then wait until I release again
time.sleep(1)
t1.join()
t2.join()
What I'm having some issues is the following:
I want to have two threads (thread1 and thread2) that are launched at the beginning of the program, but they should wait until the main() counter reaches 5 or 10.
When the main() counter reaches 5 or 10, it should signal/trigger/unlock the threads, and both threads should run just once and then wait until a new unlock.
I was expecting the code to have the following output (Each line is 1 second running):
main...
main...
main...
main...
main...
write done by t1
write done by t2
main...
main...
main...
main...
main...
write done by t1
write done by t2
Instead I have a different behaviour, such as starting with:
write done by t1
write done by t1
write done by t1
write done by t1
(etc)
And after 5 seconds the
write done by t2
A lot of times...
Can someone help me explaining what is wrong and how can I improve this?
In __init__() of thread1 and thread2, start() is invoked before self.lock is assigned.
t1 and t2 are created before the main thread acquires the lock. That makes these two threads start printing before the main thread locks them. It is the reason your code print the first several lines of "write done by x".
After the counter reaches 5, the main thread releases the lock, but it never locks it again. That makes t1 and t2 keep running.
It never quits unless you kill it...
I suggest you to use Condition Object instead of Lock.
Here is an example based on your code.
import threading
import time
class Thread1(threading.Thread):
def __init__(self, condition_obj):
super().__init__()
self.daemon = True
self.condition_obj = condition_obj
self.start()
def run(self):
with self.condition_obj:
while True:
self.condition_obj.wait()
print('write done by t1')
class Thread2(threading.Thread):
def __init__(self, condition_obj):
super().__init__()
self.daemon = True
self.condition_obj = condition_obj
self.start()
def run(self):
with self.condition_obj:
while True:
self.condition_obj.wait()
print('write done by t2')
if __name__ == '__main__':
condition = threading.Condition()
t1 = Thread1(condition)
t2 = Thread2(condition)
counter = 0
while True:
print("main...")
counter += 1
if counter == 5 or counter == 10:
with condition:
condition.notify_all()
time.sleep(1)
t1.join()
t2.join()
I use Queue to provide tasks that threads can work on. After all work is done from Queue, I see the threads are still alive while I expected them being released. Here is my code. You can see the active threads number is increasing after a batch of task(in the same queue) increases from the console. How could I release the threads after a batch of work get done?
import threading
import time
from Queue import Queue
class ThreadWorker(threading.Thread):
def __init__(self, task_queue):
threading.Thread.__init__(self)
self.task_queue = task_queue
def run(self):
while True:
work = self.task_queue.get()
#do some work
# do_work(work)
time.sleep(0.1)
self.task_queue.task_done()
def get_batch_work_done(works):
task_queue = Queue()
for _ in range(5):
t = ThreadWorker(task_queue)
t.setDaemon(True)
t.start()
for work in range(works):
task_queue.put(work)
task_queue.join()
print 'get batch work done'
print 'active threads count is {}'.format(threading.activeCount())
if __name__ == '__main__':
for work_number in range(3):
print 'start with {}'.format(work_number)
get_batch_work_done(work_number)
Do a non blocking read in a loop and use the exception handling to terminate
def run(self):
try:
while True:
work = self.task_queue.get(True, 0.1)
#do some work
# do_work(work)
except Queue.Empty:
print "goodbye"
In the below program I have posted 5 jobs to the queue, but have created only 3 threads. When I run the program, only 3 jobs are completed. How am I supposed to complete all 5 jobs with only 3 threads? Is there a way to the make a thread that has completed its job take the next job?
import time
import Queue
import threading
class worker(threading.Thread):
def __init__(self,qu):
threading.Thread.__init__(self)
self.que=qu
def run(self):
print "Going to sleep.."
time.sleep(self.que.get())
print "Slept .."
self.que.task_done()
q = Queue.Queue()
for j in range(3):
work = worker(q);
work.setDaemon(True)
work.start()
for i in range(5):
q.put(1)
q.join()
print "done!!"
You need to have your worker threads run in a loop. You can use a sentinel value (like None or custom class) to tell the workers to shut down after you've put all your actual worked items in the queue:
import time
import Queue
import threading
class worker(threading.Thread):
def __init__(self,qu):
threading.Thread.__init__(self)
self.que=qu
def run(self):
for item in iter(self.que.get, None): # This will call self.que.get() until None is returned, at which point the loop will break.
print "Going to sleep.."
time.sleep(item)
print "Slept .."
self.que.task_done()
self.que.task_done()
q = Queue.Queue()
for j in range(3):
work = worker(q);
work.setDaemon(True)
work.start()
for i in range(5):
q.put(1)
for i in range(3): # Shut down all the workers
q.put(None)
q.join()
print "done!!"
Another option would be to use a multiprocessing.dummy.Pool, which is a thread pool that Python manages for you:
import time
from multiprocessing.dummy import Pool
def run(i):
print "Going to sleep..."
time.sleep(i)
print "Slept .."
p = Pool(3) # 3 threads in the pool
p.map(run, range(5)) # Calls run(i) for each element i in range(5)
p.close()
p.join()
print "done!!"
I need a sleep() method which can be aborted (as described here or here).
My approach is to let a threading.Event.wait() timeout at the specified duration:
def abortable_sleep(secs, abort_event):
abort_event.wait(timeout=secs)
abort_event.clear()
After calling abortable_sleep(10, _abort) I can now (from another thread) call _event.set(_abort) to let abortable_sleep() terminate before the 10 seconds.
Example:
def sleeping_thread():
_start = time.perf_counter()
print("%f thread started" % (time.perf_counter() - _start))
abortable_sleep(5, _abort)
print("%f thread stopped" % (time.perf_counter() - _start))
if __name__ == '__main__':
_abort = threading.Event()
while True:
threading.Thread(target=sleeping_thread).start()
time.sleep(3)
_abort.set()
time.sleep(1)
Output:
0.000001 thread started
3.002668 thread stopped
0.000002 thread started
3.003014 thread stopped
0.000001 thread started
3.002928 thread stopped
0.000001 thread started
This code is working as expected but I still have some questions:
isn't there an easier way to have s.th. likea sleep() which can be aborted?
can this be done more elegant? E.g. this way I have to be careful with the Event instance which is not bound to an instance of abortable_sleep()
do I have to expect performance issues with high frequency loops like while True: abortable_sleep(0.0001)? How is the wait()-timeout implemented?
I have a wrapper class which basically slaps some sleep semantics on top of an Event. The nice thing is that you only have to pass around a Sleep object, which you can call sleep() on several times if you like (sleep() is not thread safe though) and that you can wake() from another thread.
from threading import Event
class Sleep(object):
def __init__(self, seconds, immediate=True):
self.seconds = seconds
self.event = Event()
if immediate:
self.sleep()
def sleep(self, seconds=None):
if seconds is None:
seconds = self.seconds
self.event.clear()
self.event.wait(timeout=seconds)
def wake(self):
self.event.set()
Usage example:
if __name__ == '__main__':
from threading import Thread
import time
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(created)d - %(message)s')
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("sleep")
s = Sleep(3)
logger.info("awake")
def wake_it(sleeper):
time.sleep(1)
logger.info("wakeup!")
sleeper.wake()
logger.info("sleeping again")
s = Sleep(60, immediate=False)
Thread(target=wake_it, args=[s]).start()
s.sleep()
logger.info("awake again")
The above might output something like this:
1423750549 - sleep
1423750552 - awake
1423750552 - sleeping again
1423750553 - wakeup!
1423750553 - awake again
Exactly what you did, but encapsulated in a class.
Due to race conditions, your solution is not always perfectly correct. You should use a threading.BoundedSemaphore() instead. Call aquire() immediately after creating it. When you want to sleep, call acquire() with a timeout, then call release() if the acquire() returned true. To abort the sleep early, call release() from a different thread; this will raise ValueError if there is no sleep in progress.
Using an event instead is problematic if the other thread calls set() at the wrong time (i.e. at any time other than when you are actually waiting on the event).
I'd wrap the sleep/abort function up in a new class:
class AbortableSleep():
def __init__(self):
self._condition = threading.Condition()
def __call__(self, secs):
with self._condition:
self._aborted = False
self._condition.wait(timeout=secs)
return not self._aborted
def abort(self):
with self._condition:
self._condition.notify()
self._aborted = True
I'd then also supply a Thread subclass to manage the sharing of the wakeup routine on a per-thread basis:
class ThreadWithWakeup(threading.Thread):
def __init__(self, *args, **kwargs):
self.abortable_sleep = AbortableSleep()
super(ThreadWithWakeup, self).__init__(*args, **kwargs)
def wakeup(self):
self.abortable_sleep.abort()
Any other thread with access to this thread can call wakeup() to abort the current abortable_sleep() (if one is in progress).
Using ThreadWithWakeup
You can create threads using the ThreadWithWakeup class, and use it like this:
class MyThread(ThreadWithWakeup):
def run(self):
print "Sleeper: sleeping for 10"
if self.abortable_sleep(10):
print "Sleeper: awoke naturally"
else:
print "Sleeper: rudely awoken"
t = MyThread()
t.start()
print "Main: sleeping for 5"
for i in range(5):
time.sleep(1)
print i + 1
print "Main: waking thread"
t.wakeup()
The output of which looks like:
Sleeper: sleeping for 10
Main: sleeping for 5
1
2
3
4
5
Main: waking thread
Sleeper: rudely awoken
Using AbortableSleep on its own
You can also use the AbortableSleep class on its own, which is handy if you can't use the ThreadWithWakeup class for some reason (maybe you're in the main thread, maybe something else creates the threads for you, etc.):
abortable_sleep = AbortableSleep()
def run():
print "Sleeper: sleeping for 10"
if abortable_sleep(10):
print "Sleeper: awoke naturally"
else:
print "Sleeper: rudely awoken"
threading.Thread(target=run).start()
print "Main: sleeping for 5"
for i in range(5):
time.sleep(1)
print i + 1
print "Main: aborting"
abortable_sleep.abort()