I would like to stop/kill N thread when I receive a socket event.
I don't find any way to do that with my current app structure.
Here is the code to start threads :
for i in range(news_viewers):
t = threading.Thread(target=bot, args=(i + 1,))
t.daemon = True
t.name = "Viewer"
t.start()
When I receive the event I would like to kill/stop N thread named Viewer on my list :
for i in range(number_of_threads_to_kill):
#number_of_threads_to_kill is received by sockerIO
for t in threading.enumerate():
if 'Viewer' in t.getName():
#I NEED TO CLOSE N THREAD HERE
print('CLOSE THIS THREAD')
I can't find a way to do it, I tried many things but none of them worked.
threading.enumerate() return this :
.., <Thread(Viewer, started daemon 41868)>, <Thread(Viewer, started daemon 53872)>, <Thread(Viewer, started daemon 54748)>, <Thread(Viewer, started daemon 50028)>,...
Can someone help me to setup that ?
I finally found a way to close thread.
When I start a thread I add the id of the thread in threads_ids array :
global stop_ids, threads_ids
for i in range(news_viewers):
tid = random.randint(1, 1000)
threads_ids.append(tid)
t = threading.Thread(target=bot, args=(tid,))
t.daemon = True
t.name = "Viewer"
t.start()
When I want to close a Thread I add the N first id on stop_ids array :
global stop_ids, threads_ids
n = threads_ids[:number_of_thread_to_close]
stop_ids = stop_ids + n
On my bot function I check every N second if the id of the current thread is on stop_ids :
def bot(id):
global stop_ids
...
q = True
while q:
if id in stop_ids:
stop_ids.remove(id)
driver.quit()
q = False
break
...
Related
I want to start thread multiple times, but only when it is not running.
There is a simple model what I am trying:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
while True:
print(hum['h'])
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
if not H.isAlive():
H.start()
print(threading.active_count())
Also what i don't understand is that:
When I run program it prints: 0. Then after 3 seconds it prints:1 and so on after every 3 second it is increased by 1.
But I thought it would print: 0. Then after 3 second it would print: 1. And then immediately increasing fast.
Because after starting first thread it would immediately start the next one and so on. why does this happen?
How not to start a thread 'up', if it's already running?
Not sure if I got your question completely, but here are some thoughts.
When I run your code I get an increasing number of active threads, as you are creating a new thread every time, checking its status (which will always be not alive) and then starting it.
What you want to do instead is to check the status of the last run thread and if that's not alive start a new one. In order to do that you should create a new thread if the old one is done:
def up (x, r):
time.sleep(3)
r['h'] = x + 1
def main():
hum = {'h' : 0}
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
while True:
# print(hum['h'])
if not H.isAlive():
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
H.start()
print(threading.active_count())
What happens in your code:
Print the value of hum['h']
Create a thread (note you create it, you are not starting it yet)
Set the value of a property
If the thread is not started then start it
Print the count of active threads (active, NOT started)
Since you replace the H variable every time, you'll have a new thread every time that gets immediately started.
If you add a print that says "starting" in the if for the is alive, you'll see that it gets called every time.
You can use join() to wait for the thread to finish:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
while True:
print(hum['h'])
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
H.start()
H.join()
print(threading.active_count())
If you don't want to wait, you can just save the current running thread in a variable and check it in the loop:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
current_thread = None
while True:
print(hum['h'])
if current_thread is None:
current_thread = threading.Thread(target = up, args=(hum['h'],hum))
current_thread.daemon=True
current_thread.start()
elif not current_thread.isAlive():
current_thread = threading.Thread(target = up, args=(hum['h'],hum))
current_thread.daemon=True
current_thread.start()
dispQ = Queue.Queue()
stop_thr_event = threading.Event()
def worker (stop_event):
while not stop_event.wait(0):
try:
job = dispQ.get(timeout=1)
job.do_work() # does some work and may put more jobs in the dispQ
dispQ.task_done()
except Queue.Empty, msg:
continue
t1 = threading.Thread( target=worker, args=(stop_thr_event,) )
t1.daemon = True
t1.start()
t2 = threading.Thread( target=worker, args=(stop_thr_event,) )
t2.daemon = True
t2.start()
# put some initial jobs in the dispQ here
while True:
if dispQ.qsize() == 0:
break
else:
# do something important
stop_thr_event.set()
t1.join()
t2.join()
The Problem:
dispQ.qsize is 0 when inside worker func, job is creating more items and hence breaking out the loop (after breaking out of the loop, there are more jobs in dispQ)
I need do something like:
if dispQ.qsize() == 0 and there is no item being worked inside the worker func, then break out of the while loop
i.e. task_done() hasnt called yet after get() was called
it would be nice if it had dispQ.join() with a timeout
Here is the source for Queue.join, it looks like you can just use queue.unfinished_tasks > 0 to check if any tasks are running or in queue. You may want to acquire (then release) the lock to ensure that the queue doesn't change while you're checking its status. (I believe) You don't need to do this if you can guarantee that no items will be added to the queue after the last task finishes.
def join(self):
# ...
self.all_tasks_done.acquire()
try:
while self.unfinished_tasks:
self.all_tasks_done.wait()
finally:
self.all_tasks_done.release()
My multi-threading script raising this error :
thread.error : can't start new thread
when it reached 460 threads :
threading.active_count() = 460
I assume the old threads keeps stack up, since the script didn't kill them. This is my code:
import threading
import Queue
import time
import os
import csv
def main(worker):
#Do Work
print worker
return
def threader():
while True:
worker = q.get()
main(worker)
q.task_done()
def main_threader(workers):
global q
global city
q = Queue.Queue()
for x in range(20):
t = threading.Thread(target=threader)
t.daemon = True
print "\n\nthreading.active_count() = " + str(threading.active_count()) + "\n\n"
t.start()
for worker in workers:
q.put(worker)
q.join()
How do I kill the old threads when their job is done? (Is return not enough?)
Your threader function never exits, so your threads never die. Since you're just processing one fixed set of work and never adding items after you start working, you could set the threads up to exit when the queue is empty.
See the following altered version of your code and the comments I added:
def threader(q):
# let the thread die when all work is done
while not q.empty():
worker = q.get()
main(worker)
q.task_done()
def main_threader(workers):
# you don't want global variables
#global q
#global city
q = Queue.Queue()
# make sure you fill the queue *before* starting the worker threads
for worker in workers:
q.put(worker)
for x in range(20):
t = threading.Thread(target=threader, args=[q])
t.daemon = True
print "\n\nthreading.active_count() = " + str(threading.active_count()) + "\n\n"
t.start()
q.join()
Notice that I removed global q, and instead I pass q to the thread function. You don't want threads created by a previous call to end up sharing a q with new threads (edit although q.join() prevents this anyway, it's still better to avoid globals).
I am writing a server for some library (python).
I want that while the server is working in his loop it will open 1 thread to do something else.
I am controlling this thread with queue and until there is no return value in the queue i don't want the server to open another thread.
try:
#we have a return in the queqe
returnValue = threadQueue.get(False)
startAnotherThread = True
except Empty:
print "Waiting for return value from thread thread....."
if there is some return value in the queue then startAnotherThread will tell to some if statement to open another thread.
i don't know why it's not working mabye some one have an idea?
Solved:
Init before the server loop:
# thread queue
threadQueue = Queue()
# thread numbering
threadNumber = 0
# thread start
threadCanStart = True
Inside the server loop:
if threadCanStart:
myThread(threadNumber, threadQueue).start()
threadNumber += 1
threadCanStart = False
try:
#we have a return in the quqe
returnValue = threadQueue.get(False)
print "Queue return: ", returnValue
threadCanStart = True
except Empty:
print "Waiting for thread....."
I have some issue with too many Threads unfinished.
I think that queue command .join() just close queue and not the threads using it.
In my script I need to check 280k domains and for each domain get list of his MX records and obtain an IPv6 address of servers if it has it.
I used threads and thanks for them the script its many times faster. But there is a problem, although there is join() for the queue, number of alive threads is growing till an error occur that informs that cant create any new thread (limitation of OS?).
How can I terminate/close/stop/reset threads after each For loop when I am retrieving new domain from database?
Thread Class definition...
class MX_getAAAA_thread(threading.Thread):
def __init__(self,queue,id_domain):
threading.Thread.__init__(self)
self.queue = queue
self.id_domain = id_domain
def run(self):
while True:
self.mx = self.queue.get()
res = dns.resolver.Resolver()
res.lifetime = 1.5
res.timeout = 0.5
try:
answers = res.query(self.mx,'AAAA')
ip_mx = str(answers[0])
except:
ip_mx = "N/A"
lock.acquire()
sql = "INSERT INTO mx (id_domain,mx,ip_mx) VALUES (" + str(id_domain) + ",'" + str(self.mx) + "','" + str(ip_mx) + "')"
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
print "MX" , '>>' , ip_mx, ' :: ', str(self.mx)
lock.release()
self.queue.task_done()
Thread class in use...
(The main For-loop is not here, this is just part of his body)
try:
answers = resolver.query(domain, 'MX')
qMX = Queue.Queue()
for i in range(len(answers)):
t = MX_getAAAA_thread(qMX,id_domain)
t.setDaemon(True)
threads.append(t)
t.start()
for mx in answers:
qMX.put(mx.exchange)
qMX.join()
except NoAnswer as e:
print "MX - Error: No Answer"
except Timeout as etime:
print "MX - Error: dns.exception.Timeout"
print "end of script"
I tried to:
for thread in threads:
thread.join()
after the queue was done, but thread.join() never stops waiting, despite fact that there is no need to wait, because when queue.join() executes there is nothing to do for threads.
What I often do when my thread involves an infinite loop like this, is to change the condition to something I can control from the outside. For example like this:
def run(self):
self.keepRunning = True
while self.keepRunning:
# do stuff
That way, I can change the keepRunning property from the outside and set it to false to gracefully terminate the thread the next time it checks the loop condition.
Btw. as you seem to spawn exactly one thread for each item you put into the queue, you don’t even need to have the threads loop at all, although I would argue that you should always enforce a maximum limit of threads that can be created in this way (i.e. for i in range(min(len(answers), MAX_THREAD_COUNT)):)
Alternative
In your case, instead of terminating the threads in each for-loop iteration, you could just reuse the threads. From what I gather from your thread’s source, all that makes a thread unique to an iteration is the id_domain property you set on its creation. You could however just provide that as well with your queue, so the threads are completely independent and you can reuse them.
This could look like this:
qMX = Queue.Queue()
threads = []
for i in range(MAX_THREAD_COUNT):
t = MX_getAAAA_thread(qMX)
t.daemon = True
threads.append(t)
t.start()
for id_domain in enumerateIdDomains():
answers = resolver.query(id_domain, 'MX')
for mx in answers:
qMX.put((id_domain, mx.exchange)) # insert a tuple
qMX.join()
for thread in threads:
thread.keepRunning = False
Of course, you would need to change your thread a bit then:
class MX_getAAAA_thread(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
def run(self):
self.keepRunning = True
while self.keepRunning:
id_domain, mx = self.queue.get()
# do stuff
I do not see why you need a Queue in the first place.
After all in your design every thread just processes one task.
You should be able to pass that task to the thread on creation.
This way you do not need a Queue and you get rid of the while-loop:
class MX_getAAAA_thread(threading.Thread):
def __init__(self, id_domain, mx):
threading.Thread.__init__(self)
self.id_domain = id_domain
self.mx = mx
Then you can rid of the while-loop inside the run-method:
def run(self):
res = dns.resolver.Resolver()
res.lifetime = 1.5
res.timeout = 0.5
try:
answers = res.query(self.mx,'AAAA')
ip_mx = str(answers[0])
except:
ip_mx = "N/A"
with lock:
sql = "INSERT INTO mx (id_domain,mx,ip_mx) VALUES (" + str(id_domain) + ",'" + str(self.mx) + "','" + str(ip_mx) + "')"
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
print "MX" , '>>' , ip_mx, ' :: ', str(self.mx)
Create one thread for each task
for mx in answers:
t = MX_getAAAA_thread(qMX, id_domain, mx)
t.setDaemon(True)
threads.append(t)
t.start()
and join them
for thread in threads:
thread.join()
Joining the threads will do the trick, but the joins in your case are blocking indefinitely because your threads aren't ever exiting your run loop. You need to exit the run method so that the threads can be joined.