We Keep Coding

How to continue the program flow after one of several threads returns a value?

In my program, there is a section where I utilize from multiple threads for simulating a distributed environment. All threads are trying to crack a password. As you can see, all threads call the same target function func with different arguments. This function returns a result whenever a trial that cracks the password is found.
def func(self, inp):
trial = 0
while (crackPwd(inp, trial) != True):
trial += 1
return inp
threads = []
for inp in range(inpAmount):
thr = threading.Thread(target=func, args=(inp))
threads.append(thr)
thr.start()
for thr in threads:
thr.join()
However what I want to do is to stop other threads after one of the threads cracks the password. I mean, I want to continue with the program flow after a thread returns a result from func(). I tried to find a solution but none of them seems to match my problem. Now, I got results from all threads and lose so much time waiting for all threads to finish. I will appreciate your help.

Could you use an instance of the threading Event class?
By mocking the crackPwd function mentioned in your code to sleep until the sleep_time is 10 seconds (with 10% probability) I tested with:
import time
import random
import threading
def crackPwd(inp, trial):
sleep_time = random.randint(1, 10)
time.sleep(sleep_time)
return sleep_time
def func(inp):
trial = 0
while (crackPwd(inp, trial) != 10) and (not pwd_cracked.isSet()):
trial += 1
pwd_cracked.set()
return inp
threads = []
for inp in range(10):
pwd_cracked = threading.Event()
thr = threading.Thread(target=func, args=(inp, ))
threads.append(thr)
thr.start()
for thr in threads:
thr.join()
So for your original code:
def func(self, inp):
trial = 0
while (crackPwd(inp, trial) != True) and (not pwd_cracked.isSet()):
trial += 1
pwd_cracked.set()
return inp
threads = []
for inp in range(inpAmount):
pwd_cracked = threading.Event()
thr = threading.Thread(target=func, args=(inp, ))
threads.append(thr)
thr.start()
for thr in threads:
thr.join()

Python: threading

I want to start thread multiple times, but only when it is not running.
There is a simple model what I am trying:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
while True:
print(hum['h'])
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
if not H.isAlive():
H.start()
print(threading.active_count())
Also what i don't understand is that:
When I run program it prints: 0. Then after 3 seconds it prints:1 and so on after every 3 second it is increased by 1.
But I thought it would print: 0. Then after 3 second it would print: 1. And then immediately increasing fast.
Because after starting first thread it would immediately start the next one and so on. why does this happen?
How not to start a thread 'up', if it's already running?

Not sure if I got your question completely, but here are some thoughts.
When I run your code I get an increasing number of active threads, as you are creating a new thread every time, checking its status (which will always be not alive) and then starting it.
What you want to do instead is to check the status of the last run thread and if that's not alive start a new one. In order to do that you should create a new thread if the old one is done:
def up (x, r):
time.sleep(3)
r['h'] = x + 1
def main():
hum = {'h' : 0}
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
while True:
# print(hum['h'])
if not H.isAlive():
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
H.start()
print(threading.active_count())

What happens in your code:
Print the value of hum['h']
Create a thread (note you create it, you are not starting it yet)
Set the value of a property
If the thread is not started then start it
Print the count of active threads (active, NOT started)
Since you replace the H variable every time, you'll have a new thread every time that gets immediately started.
If you add a print that says "starting" in the if for the is alive, you'll see that it gets called every time.

You can use join() to wait for the thread to finish:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
while True:
print(hum['h'])
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
H.start()
H.join()
print(threading.active_count())
If you don't want to wait, you can just save the current running thread in a variable and check it in the loop:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
current_thread = None
while True:
print(hum['h'])
if current_thread is None:
current_thread = threading.Thread(target = up, args=(hum['h'],hum))
current_thread.daemon=True
current_thread.start()
elif not current_thread.isAlive():
current_thread = threading.Thread(target = up, args=(hum['h'],hum))
current_thread.daemon=True
current_thread.start()

Python: multithreading complex objects

class Job(object):
def __init__(self, name):
self.name = name
self.depends = []
self.waitcount = 0
def work(self):
#does some work
def add_dependent(self, another_job)
self.depends.append(another_job)
self.waitcount += 1
so, waitcount is based on the number of jobs you have in depends
job_board = {}
# create a dependency tree
for i in range(1000):
# create random jobs
j = Job(<new name goes here>)
# add jobs to depends if dependent
# record it in job_board
job_board[j.name] = j
# example
# jobC is in self.depends of jobA and jobB
# jobC would have a waitcount of 2
rdyQ = Queue.Queue()
def worker():
try:
job = rdyQ.get()
success = job.work()
# if this job was successful create dependent jobs
if success:
for dependent_job in job.depends:
dependent_job.waitcount -= 1
if dependent_job.waitcount == 0:
rdyQ.put(dependent_job)
and then i would create threads
for i in range(10):
t = threading.Thread( target=worker )
t.daemon=True
t.start()
for job_name, job_obj in job_board.iteritems():
if job_obj.waitcount == 0:
rdyQ.put(job_obj)
while True:
# until all jobs finished wait
Now here is an example:
# example
# jobC is in self.depends of jobA and jobB
# jobC would have a waitcount of 2
now in this scenario, if both jobA and jobB are running and they both tried to decrement waitcount of jobC, weird things were happening
so i put a lock
waitcount_lock = threading.Lock()
and changed this code to:
# if this job was successful create dependent jobs
if success:
for dependent_job in job.depends:
with waitcount_lock:
dependent_job.waitcount -= 1
if dependent_job.waitcount == 0:
rdyQ.put(dependent_job)
and strange things still happen
i.e. same job was being processed by multiple threads, as if the job was put into the queue twice
is it not a best practice to have/modify nested objects when complex objects are being pass amongst threads?

Here's a complete, executable program that appears to work fine. I expect you're mostly seeing "weird" behavior because, as I suggested in a comment, you're counting job successors instead of job predecessors. So I renamed things with "succ" and "pred" in their names to make that much clearer. daemon threads are also usually a Bad Idea, so this code arranges to shut down all the threads cleanly when the work is over. Note too the use of assertions to verify that implicit beliefs are actually true ;-)
import threading
import Queue
import random
NTHREADS = 10
NJOBS = 10000
class Job(object):
def __init__(self, name):
self.name = name
self.done = False
self.succs = []
self.npreds = 0
def work(self):
assert not self.done
self.done = True
return True
def add_dependent(self, another_job):
self.succs.append(another_job)
another_job.npreds += 1
def worker(q, lock):
while True:
job = q.get()
if job is None:
break
success = job.work()
if success:
for succ in job.succs:
with lock:
assert succ.npreds > 0
succ.npreds -= 1
if succ.npreds == 0:
q.put(succ)
q.task_done()
jobs = [Job(i) for i in range(NJOBS)]
for i, job in enumerate(jobs):
# pick some random successors
possible = xrange(i+1, NJOBS)
succs = random.sample(possible,
min(len(possible),
random.randrange(10)))
for succ in succs:
job.add_dependent(jobs[succ])
q = Queue.Queue()
for job in jobs:
if job.npreds == 0:
q.put(job)
print q.qsize(), "ready jobs initially"
lock = threading.Lock()
threads = [threading.Thread(target=worker,
args=(q, lock))
for _ in range(NTHREADS)]
for t in threads:
t.start()
q.join()
# add sentinels so threads end cleanly
for t in threads:
q.put(None)
for t in threads:
t.join()
for job in jobs:
assert job.done
assert job.npreds == 0
CLARIFYING THE LOCK
In a sense, the lock in this code protects "too much". The potential problem it's addressing is that multiple threads may try to decrement the .npreds member of the same Job object simultaneously. Without mutual exclusion, the stored value at the end of that may be anywhere from 1 smaller than its initial value, to the correct result (the initial value minus the number of threads trying to decrement it).
But there's no need to also mutate the queue under lock protection. Queues do their own thread-safe locking. So, e.g., the code could be written like so instead:
for succ in job.succs:
with lock:
npreds = succ.npreds = succ.npreds - 1
assert npreds >= 0
if npreds == 0:
q.put(succ)
It's generally best practice to hold a lock for as little time as possible. However, I find this rewrite harder to follow. Pick your poison ;-)

Producers and Consumers - Multiple Threads in Python

So I've got this code for Producers and Consumers;
import threading
import time
import random
N = 8
buffer = N * [None]
free = threading.Semaphore(N)
items = threading.Semaphore(0)
def prod():
n = 0
i = 0
while True:
time.sleep(random.random())
free.acquire()
buffer[i] = n
i = (i + 1) % N
n += 1
items.release()
def cons():
i = 0
while True:
time.sleep(random.random())
items.acquire()
print(buffer[i])
i = (i + 1) % N
free.release()
def main():
p = threading.Thread(target=prod, args=[])
c = threading.Thread(target=cons, args=[])
p.start()
c.start()
p.join()
c.join()
main()
But I want to be able to have three threads each for the producer and consumer. Can someone suggest a way I could do this using a third semaphore? Thanks.

Assuming this is not a homework about semaphores and you want a real solution, you should use the Queue object, which can handle all of this by itself. If I understood it correctly, you want three producers and three consumers that share one buffer that can have at maximum 8 items. If that's the case, the code can be simplified to something like this:
import threading
import Queue
def prod(queue):
n = 0
while True:
time.sleep(random.random())
queue.put(n)
n += 1
def cons(queue):
while True:
time.sleep(random.random())
n = queue.get()
print n
def main():
N = 8
queue = Queue.Queue(N)
threads = []
for i in range(3):
threads.append(threading.Thread(target=cons, args=[queue])))
threads.append(threading.Thread(target=prod, args=[queue])))
for thread in threads:
thread.start()
for thread in threads:
thread.join() # this will never really finish, because the threads run forever
If you are interested how is the queue implemented internally, you can see the source code here.

Rendezvous threads - thread doesn't finish processing when join() unblocks

I'm going through the "Little Book of Semaphores" right now, and I'm having a problem with the first Barrier problem. In the below code , I'm trying to have 3 threads rendezvous before continuing. This part works fine - I always get 3 "before"s pushed to the queue. However, I don't always get 3 "after"s pushed to the queue. Sometimes I do, but not always. What am I doing wrong?
import threading
import random
import Queue
import time
num_loops = 1
class myThread(threading.Thread):
def __init__(self, id, count, n, q, locks):
threading.Thread.__init__(self)
self.id = id
self.q = q
self.n = n
self.locks = locks
self.count = count
return
def run(self):
time.sleep(random.random()/100)
self.q.put("before")
with self.locks['mutex']:
self.count[0] += 1
if self.count[0] == self.n:
locks['barrier'].release()
locks['barrier'].acquire()
locks['barrier'].release()
time.sleep(random.random()/100)
self.q.put("after")
if __name__ == '__main__':
total = 10
incorrect = 0
num_threads = 3
for _ in range(total):
q = Queue.Queue()
locks = {'mutex': threading.Semaphore(1),
'barrier': threading.Semaphore(0),
}
threads = []
count = [0]
for i in range(num_threads):
t = myThread(i, count, num_threads, q, locks)
t.start()
threads.append(t)
for i in threads:
t.join()
print "join"
one_loop = ['before']*num_threads + ['after']*num_threads
total_loop = one_loop * num_loops
result = []
while not q.empty():
result.append(q.get())
print result
if result != total_loop:
incorrect += 1
print "%s out of %s is wrong" % (incorrect, total)

I found the problem. You do not join all the threads. The line:
for i in threads:
t.join()
print "join"
Should be:
for i in threads:
i.join() # changed line
print "join"
Joining t is first just waiting for the last thread created, then in the rest of the iterations a no-op.