As titles describe, I create a separate thread to do a long task in Flask.
import schedule
import time
start_time = time.time()
def job():
print("I'm working..." + str(time.time() - start_time))
def run_schedule():
while True:
schedule.run_pending()
time.sleep(1)
When I press Ctrl + c to terminate the server, the thread still prints. How can I stop the thread when server exits?
You may want to set your thread as daemon.
A thread runs until it ends by itself or it is explicity killed.
A daemon thread runs with the same conditions and if at least one other non-daemonic thread is running: this means that if you end your main thread and no other threads are running, all daemonic thread will end as well.
if you're using threading module, you may set the thread as daemonic by changing his boolean:
import threading
your_thread.daemon = True
if you're using thread module, it should be one of the kwargs
Related
I am calling a 3rd party function that I have not written so I cannot edit it.
It has its own logic that may take some time to finish.
I wrapped that 3rd party call into a thread and I wanted to immediately cancel it once I received some event.
import threading
import time
# Assume this is the 3rd party function
def call_3rd_party_function():
for i in range(50):
print("Executing task " + str(i))
time.sleep(0.25)
def task(stop_event: threading.Event):
print("Starting background thread!")
while not stop_event.is_set():
call_3rd_party_function()
stop_event = threading.Event()
t = threading.Thread(target=task, args=(stop_event,))
t.setDaemon(True)
t.start()
time.sleep(5)
stop_event.set()
t.join()
print("Main thread end!")
In the code above, example after 5 seconds.. I want to cancel that task, but my output is not what I want as the 3rd party function still continue with its tasks to completion.
Starting background thread!
Executing task 0
Executing task 1
Executing task 2
Executing task 3
.
.
Executing task 49
Main thread end!
Is there a way to do this? I wrapped it in a thread since I want it to run on its own so as not to block the main thread from doing some other things.
Any hints?
I had some performance issues with a multi-threading code to parallelize multiple telnet probes.
Slow
My first implementation was is really slow, same a if the tasks were run sequencially:
for printer in printers:
…
thread = threading.Thread(target=collect, args=(task, printers_response), kwargs=kw)
threads.append(thread)
for thread in threads:
thread.start()
thread.join()
Blastlingly Fast
for printer in printers:
…
thread = threading.Thread(target=collect, args=(task, printers_response), kwargs=kw)
threads.append(thread)
thread.start() # <----- moved this
for thread in threads:
thread.join()
Question
I don't get why moving the start() method change the performance so much.
In your first implementation you are actually running the code sequentially because by calling join() immediately after start() the main thread is blocked until the started thread is finished.
thread.join() is blocking every thread as soon as they are created in your first implementation.
According to threading.Thread.join() documentation:
Wait until the thread terminates.
This blocks the calling thread until the thread whose join() method is called terminates -- either normally or through an unhandled exception or until the optional timeout occurs".
In your slow example you start the thread and wait till it is complete, then you iterate to the next thread.
Example
from threading import Thread
from time import sleep
def foo(a, b):
while True:
print(a + ' ' + b)
sleep(1)
ths = []
for i in range(3):
th = Thread(target=foo, args=('hi', str(i)))
ths.append(th)
for th in ths:
th.start()
th.join()
Produces
hi 0
hi 0
hi 0
hi 0
In your slow solution you are basically not using multithreading at all. Id's running a thread, waiting to finish it and then running another - there is no difference in running everything in one thread and this solution - you are running them in series.
The second one on the other hand starts all threads and then joins them. This solution limits the execution time to the longest execution time of one single thread - you are running them in parallel.
Self-taught programming student, so I apologize for all the amateur mistakes. I want to learn some deeper subjects, so I'm trying to understand threading, and exception handling.
import threading
import sys
from time import sleep
from random import randint as r
def waiter(n):
print "Starting thread " + str(n)
wait_time = r(1,10)
sleep(wait_time)
print "Exiting thread " + str(n)
if __name__=='__main__':
try:
for i in range(5):
t = threading.Thread(target=waiter, args=(i+1,))
t.daemon = True
t.start()
sleep(3)
print 'All threads complete!'
sys.exit(1)
except KeyboardInterrupt:
print ''
sys.exit(1)
This script just starts and stops threads after a random time and will kill the program if it receives a ^C. I've noticed that it doesn't print when some threads finish:
Starting thread 1
Starting thread 2
Starting thread 3
Exiting thread 3
Exiting thread 2
Starting thread 4
Exiting thread 1
Exiting thread 4
Starting thread 5
All threads complete!
In this example, it never states it exits thread 5. I find I can fix this if I comment out the t.daemon = True statement, but then exception handling waits for any threads to finish up.
Starting thread 1
Starting thread 2
^C
Exiting thread 1
Exiting thread 2
I can understand that when dealing with threads, it's best that they complete what they're handling before exiting, but I'm just curious as to why this is. I'd really appreciate any answers regarding the nature of threading and daemons to guide my understanding.
The whole point of a daemon thread is that if it's not finished by the time the main thread finishes, it gets summarily killed. Quoting the docs:
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property or the daemon constructor argument.
Note Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
Now, look at your logic. The main thread only sleeps for 3 seconds after starting thread 5. But thread 5 can sleep for anywhere from 1-10 seconds. So, about 70% of the time, it's not going to be finished by the time the main thread wakes up, prints "All threads complete!", and exits. But thread 5 is still asleep for another 5 seconds. In which case thread 5 will be killed without ever going to print "Exiting thread 5".
If this isn't the behavior you want—if you want the main thread to wait for all the threads to finish—then don't use daemon threads.
I have a process which spawns 2 types of thread classes. One thread is responsible for consuming a job_queue (100Threads of this class are usually running). And second thread is a killing thread. I am using a result_done flag which is being set from thread2 and the issue is my threads1 wait for X seconds and then check if result_done flag is set.
def run(self):
while True:
try:
val = self.job_queue.get(True,self.maxtimeout)
except:
pass
if self.result_done.isset():
return
Now, if maxtimeout is set to 500seconds and I set the result_done flag from another thread, this thread will wait for 500 seconds before exiting( if there's no data in the queue).
What I want to achieve is that all the threads die gracefully along with the current process, properly terminating the db,websocket,http connections etc as soon as result_done event is set from any of the threads of the process.
I am using python multiprocess library to create the process which spawns these threads.
Update: All threads are daemon=True threads.
To avoid waiting maxtimeout time, you could use Event.wait() method:
def run(self):
while not self.result_done.is_set():
try:
val = self.job_queue.get_nowait()
except Empty:
if self.result_done.wait(1): # wait a second to avoid busy loop
return
Event.wait(timeout) returns without waiting the full timeout if the event is set during the call.
I'm trying to stop a thread in python when the main script is killed. But since the thread is started hourly how do i stop the thread immediately?
def interval():
###the actual work being done here###
try:
threading.Timer(3600, interval).start()
except (KeyboardInterrupt, SystemExit):
print "Exiting"
cleanup_stop_interval();
sys.exit()
interval()
You might consider using sched.scheduler instead of threading.Timer here. There are some differences to be aware of:
sched.scheduler runs everything in the main process, not in
threads.
If the current process takes longer than delay seconds, the
scheduled event will start after the current call to interval
completes. threading.Timer works differently -- if the work done in
interval takes longer than an hour, more than one thread would run
interval concurrently.
I'm guessing you really do not want more than one interval to be running concurrently, so sched.scheduler may be more appropriate here than threading.Timer.
import timeit
import sched
import time
import logging
import sys
logger = logging.getLogger(__name__)
logging.basicConfig(level = logging.DEBUG,
format = '%(threadName)s: %(asctime)s: %(message)s',
datefmt = '%H:%M:%S')
schedule = sched.scheduler(timeit.default_timer, time.sleep)
delay = 5 # change to 3600 to schedule event in 1 hour
def interval():
logger.info('All work and no play makes Jack a dull boy.')
schedule.enter(delay = delay, priority = 1, action = interval, argument = ())
# Uncomment this to see how scheduled events are delayed if interval takes a
# long time.
# time.sleep(10)
schedule.enter(delay = 0, priority = 1, action = interval, argument = ())
try:
schedule.run()
except (KeyboardInterrupt, SystemExit):
print('Exiting')
sys.exit()
It is a bad idea to kill a thread just like that, cause it could
holding resources
be a parent to other threads
proper way of handling this is to have an exit_request flag that each threads checks on regular interval to see if it is time for him to exit, and wait for it to exit by using join()
You can't stop a thread from another thread. What you probably want is to make the thread a daemon thread, meaning the process will exit even if the thread is still active:
http://docs.python.org/2/library/threading.html#threading.Thread.daemon
daemon
A boolean value indicating whether this thread is a daemon thread (True) or not (False). This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False.
The entire Python program exits when no alive non-daemon threads are left.
Note you could still have problems where the process ended in the middle of this thread doing its work. If you want to protect against that, then have the thread wake up frequently to check if it should exit cleanly.