I saw the following code in a thread tutorial:
from time import sleep, perf_counter
from threading import Thread
start = perf_counter()
def foo():
sleep(5)
threads = []
for i in range(100):
t = Thread(target=foo,)
t.start()
threads.append(t)
for i in threads:
i.join()
end = perf_counter()
print(f'Took {end - start}')
When I run it it prints Took 5.014557975. Okay, that part is fine. It does not take 500 seconds as the non threaded version would.
What I don't understand is how .join works. I noticed without calling .join I got Took 0.007060926999999995 which indicates that the main thread ended before the child threads. Since '.join()' is supposed to block, when the first iteration of the loop occurs won't it be blocked and have to wait 5 seconds till the second iteration? How does it still manage to run?
I keep reading python threading is not truly multithreaded and it only appears to be (runs on a single core), but if that is the case then how exactly is the background time running if it's not parallel?
So '.join()' is supposed to block, so when the first iteration of the loop occurs wont it be blocked and it has to wait 5 seconds till the second iteration?
Remember all the threads are started at the same time and all of them take ~5s.
The second for loop waits for all the threads to finish. It will take roughly 5s for the first thread to finish, but the remaining 99 threads will finish roughly at the same time, and so will the remaining 99 iterations of the loop.
By the time you're calling join() on the second thread, it is either already finished or will be within a couple of milliseconds.
I keep reading python threading is not truly multithreaded and it only appears to be (runs on a single core), but if that is the case then how exactly is the background time running if it's not parallel?
It's a topic that has been discussed a lot, so I won't add another page-long answer.
Tl;dr: Yes, Python Multithreading doesn't help with CPU-intensive tasks, but it's just fine for tasks that spend a lot of time on waiting for something else (Network, Disk-I/O, user input, a time-based event).
sleep() belongs to the latter group of tasks, so Multithreading will speed it up, even though it doesn't utilize multiple cores simultaneously.
The OS is in control when the thread starts and the OS will context-switch (I believe that is the correct term) between threads.
time functions access a clock on your computer via the OS - that clock is always running. As long as the OS periodically gives each thread time to access a clock the thread's target can tell if it has been sleeping long enough.
The threads are not running in parallel, the OS periodically gives each one a chance to look at the clock.
Here is a little finer detail for what is happening. I subclassed Thread and overrode its run and join methods to log when they are called.
Caveat The documentation specifically states
only override __init__ and run methods
I was surprised overriding join didn't cause problems.
from time import sleep, perf_counter
from threading import Thread
import pandas as pd
c = {}
def foo(i):
c[i]['foo start'] = perf_counter() - start
sleep(5)
# print(f'{i} - start:{start} end:{perf_counter()}')
c[i]['foo end'] = perf_counter() - start
class Test(Thread):
def __init__(self,*args,**kwargs):
self.i = kwargs['args'][0]
super().__init__(*args,**kwargs)
def run(self):
# print(f'{self.i} - started:{perf_counter()}')
c[self.i]['thread start'] = perf_counter() - start
super().run()
def join(self):
# print(f'{self.i} - joined:{perf_counter()}')
c[self.i]['thread joined'] = perf_counter() - start
super().join()
threads = []
start = perf_counter()
for i in range(10):
c[i] = {}
t = Test(target=foo,args=(i,))
t.start()
threads.append(t)
for i in threads:
i.join()
df = pd.DataFrame(c)
print(df)
0 1 2 3 4 5 6 7 8 9
thread start 0.000729 0.000928 0.001085 0.001245 0.001400 0.001568 0.001730 0.001885 0.002056 0.002215
foo start 0.000732 0.000931 0.001088 0.001248 0.001402 0.001570 0.001732 0.001891 0.002058 0.002217
thread joined 0.002228 5.008274 5.008300 5.008305 5.008323 5.008327 5.008330 5.008333 5.008336 5.008339
foo end 5.008124 5.007982 5.007615 5.007829 5.007672 5.007899 5.007724 5.007758 5.008051 5.007549
Hopefully you can see that all the threads are started in sequence very close together; once thread 0 is joined nothing else happens till it stops (foo ends) then each of the other threads are joined and terminate.
Sometimes a thread terminates before it is even joined - for threads one plus foo ends before the thread is joined.
Related
I am trying to run a while loop so I can run code while the timer is still counting down.I also want to see the timer as it counts down. I have tried to find something similar on stack overflow, but haven't been able to get the result I'm looking for.
print("Input minutes and seconds")
min = int(input("Minutes: "))
sec = int(input("Seconds: "))
while min & sec > 0:
# do some code
# I want the program to count down from whatever I input
print("")
You should run your timer on a different thread. If you don't need the timer to affect the main code being run this should work:
import threading
import time
def timer_function(seconds):
'''Countdown from number of seconds given'''
for t in range(seconds, -1, -1):
time.sleep(1)
print(t)
if __name__ == "__main__":
print("Input minutes and seconds")
min = int(input("Minutes: "))
sec = int(input("Seconds: "))
x = threading.Thread(target=timer_function, args=(min * 60 + sec,))
x.start()
# Run some other code in parallel with the timer
x.join() # will wait for the timer function to finish
print('All done')
If you need the timer to stop the main thread (the code being run on the main function) then you need send some signal through a variable from the timer thread.
There might be some libraries that handle thread timeouts better if you would like to look it up :)
Getting very precise timeout timing is somewhat troublesome, as the operation of reporting on the timing can affect the timing itself if not carefully written.
Something like this is often best accomplished with two threads in parallel
Making the threads "daemon threads" allows you to end them by quitting the program (otherwise the program will wait for them to be .join()ed)
You can rely on threading.Event() to clearly communicate into a thread and provide a .wait() method, which will either wait for a given timeout or immediately end when the event .is_set()
import threading
import time
def function_to_call(): # replace me with your function
time.sleep(1000)
def timer_fn(timeout, event, delay=5): # 5 seconds between reports
time_end = time.time() + timeout
while not event.is_set():
time_remaining = int(time_end - time.time())
if time_remaining <= 0:
return
print(f"{int(time_remaining)}s remaining")
event.wait((min(delay, time_remaining))) # wait for event
timeout = int(input("minutes: ") or 0) * 60 + int(input("seconds: ") or 0)
E = threading.Event() # Event to kill t2
# making t1 a daemon lets it not prevent shutdown
t1 = threading.Thread(target=function_to_call, daemon=True)
t2 = threading.Thread(target=timer_fn, args=(timeout, E), daemon=True)
# begin both threads
t1.start()
t2.start()
# wait for t1 to exit (but not t2)
t1.join(timeout=timeout)
# t1 may join faster by ending, otherwise reach here after timeout
E.set() # set the Event to quickly end t2 (timeout or t1 returned)
t2.join() # not technically necessary, but cleans up t2
# program quits and t1 is killed if it's still running
Note that the timing display is actually separate from the thread ending, and ending the function early is done here by stopping the program. The function will continue to run if the program is not stopped!
If more advanced task control is needed, consider
modifying your callable function to repeatedly check another threading.Event throughout it to see if it's time is up (yet)
use multiprocessing, which is much more featureful than threading, though it may need to copy memory into the new process, which can be slow if you have a tremendous amount
use subprocess, creating a script just for the purpose of a timed end (here you can kill the subprocess with a simple .kill() or setting a timeout= argument when creating it! (though again you'll find some inefficiency in copying/streaming input into and out of the new process)
use os.fork() and [os.kill()](https://docs.python.org/3/library/os.html#os.kill) to split your process (advanced, but usually much more efficient than multiprocessing` due to how memory is (or rather is not) copied)
if your function can be made asynchronous, which allows multiple tasks to collaborate in the same namespace similar to threading, but in a more friendly way (though how this behaves is fundamentally different from the other techniques given, it can be very efficient if you have many tasks which don't rely on local resources, such as a webserver or database)
Though further fundamentally different, you may find further or more benefit in designing your task to be a collection of work to iterate over (maybe in a list) and consider a library like tqdm to report on its status
Try this,
Code :
import time
print("Input minutes and seconds")
min = int(input("Minutes: "))
sec = int(input("Seconds: "))
t = min*60 + sec
while t :
mins, secs = divmod(t, 60)
timer = '{:02d}:{:02d}'.format(mins, secs)
print(timer, end="\r")
time.sleep(1)
t -= 1
print('Timer ended !!')
Output :
Input minutes and seconds
Minutes: 1
Seconds: 5
01:05
01:04
01:03
01:02
01:01
01:00
00:59
00:58
.
.
.
00:01
Timer ended !!
You should ideally use threads and spawn a daemon to keep track, but that might be overkill for your case. I've made a more simple implementation which is hopefully understandable, otherwise please ask and I'll improve the comments/explanation:
import time
set_time = int(input('For how many seconds do you want to run this?'))
start_time = time.time() # lets get the current time
inner_time = time.time() # Seperate vars so we don't overwrite the main loop
count = 0 #To keep track of how many seconds we've been going
while (time.time() - start_time) < set_time: #Lets run this until we've reached our time
if(time.time() - inner_time) >= 1: #If 1 sec has passed
inner_time = time.time() #Reset time
count += 1 #Increase second by 1
print("Code has been running for "+str(count)+" seconds") #Inform the user
#Do something here...
print(str(set_time)+" seconds have now elapsed") #Done and dusted
This is the output:
For how long do you want to run this?5
Code has been running for 1 seconds
Code has been running for 2 seconds
Code has been running for 3 seconds
Code has been running for 4 seconds
5 seconds have now elapsed
I have a python code in which I can run a maximum of 10 threads at a time due to GPU and compute limitations. I have 100 folders that I want to process and I want each thread to process one folder. Here is some sample code that I have written to achieve this.
def random_wait(thread_id):
# print('Inside wait')
rand_number = random.randint(3, 9)
# print(f'Random number : {rand_number}')
print(f'Thread {thread_id} waiting for {rand_number} seconds')
time.sleep(rand_number)
print(f'Thread {thread_id} completed execution')
if __name__=='__main__':
total_runs = 6
thread_limit = 3
running_threads = list()
for i in range(total_runs):
print(f'Active threads : {threading.active_count()}')
if threading.active_count() > thread_limit:
print(f'Active thread count exceeded')
# check if an existing thread is alive and for it to finish execution
for running_thread in running_threads:
if not running_thread.is_alive():
# Remove thread
running_threads.remove(running_thread)
print(f'Removing thread: {running_thread}')
else:
thread = threading.Thread(target=random_wait, args=(i,), kwargs={})
running_threads.append(thread)
print(f'Starting thread : {i}')
thread.start()
In this code, I am checking if the number of active threads exceeds the thread limit that I have specified, and the process refrains from creating new threads unless there's space for one more thread to be executed.
I am able to refrain the process from starting new threads. However, I lose the threads that I wanted to start and the code just ends up starting and stopping the first three threads. How can I achieve starting a new thread/processing as soon as there's space for one more? Is there a better way in which I just start 10 threads, but as soon as one thread completes, I assign it to start processing another folder?
You should use a ThreadPoolExecutor from the Python standard library concurrent.futures, it automatically manages a fixed number of threads. If you need to execute the same function with different arguments in parallel (as in a parallel for-loop), you can use the .map() method:
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(10) as e:
results = e.map(work, (arg_1, arg_2, ..., arg_n))
If you need to schedule different work in parallel you should use the .submit() method:
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(10) as e:
future_1 = e.submit(work_1, arg_1)
future_2 = e.submit(work_2, arg_2)
result_1 = future_1.result()
result_2 = future_2.result()
In the second case, .submit() returns a Future object which encapsulates the asynchronous execution of the work. You should store that future and get the result when needed. Note that the context manager (with statement) ensures that the .shutdown() method is call before leaving it, so all works are done after this point.
I am trying to use python threading and am having problems getting the threads to work independently. They seem to be running in sequential order and waiting for one to finish before starting to process the next thread. I have read other posts suggesting that I need to get more work into the threads to differentiate actual CPU work vs the CPU work of starting and managing the threads, and that a sleep timer could be used to simulate this. So I tried that and then measured the task durations.
So my code is below. It first runs three tasks sequentially with a 2 second timer. This takes about 6 seconds to run as expected. The next section starts three threads and they should run roughly in parallel if my understanding of threading is correct. I have played with the timers to test the overall duration of this section of code, expecting that if one timer is larger than the other two, the code will execute in an interval closest to that larger one. but what I am seeing is that is taking the same amount of time as the three running in sequence - one after the other.
I got onto this because I am writing some code to read an asynchronous queue in the background. After launching the thread to read the queue, my code seems to stop and wait until the queue reader is stopped, which it normally doesn't as it waits for messages to come in. So what happens is that it never executes the next section of code and it seems to be waiting for the thread to complete.
Also I checked the number of threads active and it remains at the same number, and when I check for the thread ID in the code (not shown) I get the same thread number coming back for every thread.
I am new to python and am using the jupyter compiler environment. Is there a compile option or some other limitation that I am not aware of that is preventing the threading? Am I just not getting the concept? I dont believe that this is related to CPU cores / threading as it would be done through logical thread cores within the python compiled code. I also ran a similar program in a command shell environment and got the same sequential performance.
Cut and paste this code to see what it does. What am I missing?
'''
import threading
import logging
import datetime
import time
import random
class Parallel:
def work(self, interval):
time.sleep(interval)
name = self.__repr__()
print (name, " is complete after ", interval, " seconds")
# SetupLogger()
logging.getLogger().setLevel(logging.DEBUG)
logging.debug("thread program start time is %s", datetime.datetime.now())
thread1 = Parallel()
thread2 = Parallel()
thread3 = Parallel()
print ("sequential threads::")
thread1.work(2.0)
thread2.work(2.0)
thread3.work(2.0)
logging.info("parallel threads start time is %s ", datetime.datetime.now())
start = time.time()
work1 = threading.Thread(target=thread1.work(1), daemon=True)
work1.start()
print ("thread 1 is started and there are ", threading.activeCount(), " threads active")
work2 = threading.Thread(target=thread2.work(2), daemon=False)
work2.start()
print ("thread 2 is started and there are ", threading.activeCount(), " threads active")
work3 = threading.Thread(target=thread3.work(5), daemon=False)
work3.start()
print ("thread 3 is started and there are ", threading.activeCount(), " threads active")
# wait for all to complete
print ("now wait for all to finish at ", datetime.datetime.now())
work1.join()
work2.join()
work3.join()
end = time.time()
logging.info ("parallel threads end time is %s with %s elapsed", datetime.datetime.now(), str(end-start))
print ("all threads completed at:", datetime.datetime.now())
'''
In the line that initializes the thread, you are actually executing the function instead of passing its reference to the thread.
thread1.work() ----> this will actually execute the function when the program runs and encounters this statement
So when your program reaches this line,
work1 = threading.Thread(target=thread1.work(1), daemon=True)
and encounters target=thread1.work(1), it simply calls the function right there and the actual thread does nothing.
thread1.work is a reference to the function, which you need to pass to your Thread object.
So just remove the parenthesis and your code becomes
work1 = threading.Thread(target=thread1.work, daemon=True, args=(1,))
and this will behave as you expect.
This is my first time to use multi-threading..
I write a code to process every file in a directory like:
list_battle=[]
start = time.time()
for filepath in pathlib.Path(dir_battle).glob('**/*'):
battle_json = gzip.GzipFile(filepath,'rb').read().decode("utf-8")
battle_id = eval(type_cmd)
list_battle.append((battle_id, battle_json))
end = time.time()
print(end - start)
it shows the code runs 8.74 seconds.
Then, I tried to use multi-threading as follows:
# define function to process each file
def get_file_data(path, cmd, result_list):
data_json = gzip.GzipFile(path,'rb').read().decode("utf-8")
data_id = eval(cmd)
result_list.append((battle_id, battle_json))
# start to run multi-threading
pool = Pool(5)
start = time.time()
for filepath in pathlib.Path(dir_battle).glob('**/*'):
pool.apply_async( get_file_data(filepath, type_cmd, list_battle) )
end = time.time()
print(end - start)
However, the result shows it takes 12.36 seconds!
In my view, in single threading, in each turn of loop, the loop waits for the single thread to finish codes and then starts the next turn; while in multi-processing, 1st turn, the loop calls thread1 to run the codes, then 2nd turn calls thread2 to run.... during this job dispatching for other 4 threads, thread1 is running and when the 6th turn arrives, it should finishes its job and the loop could directly ask it to run the code of 6th trun...
So this should be quicker than single thread...Why the code with multi-processing runs even slower? How to address this issue? What is wrong with my thinking?
Any help is appreciated.
Multiprocessing does not reduce your processing time unless your process has a lot of dead time (waiting). The main purpose of multiprocessing is parallelism of different tasks at the burden of context switching. Whenever you switch from one task to another, interupting the previous one, your program needs to store all variables for the former task and get the ones from the new one. This takes time as well.
This means the shorter the time you spend per task, the less efficient (in regards of computing time) your multiprocessing is.
from time import sleep
from threading import Thread
from threading import get_ident
from queue import *
_urls = Queue()
class URLParser(Thread):
def run(self):
global _urls
while _urls.qsize() > 0:
url = archive_urls.get()
_urls.task_done()
# .... Omitted some code here where I do some work on the url, for brevity
print(str(get_ident()) + ' consumed:' + url + ' qsize:' + str(_urls.qsize()))
sleep(0.002)
How long should my thread sleep so that other threads can consume from the queue ? I do not want to specify a number too large, as then I may be wasting time, or too small and miss out on the benefit of all the threads consuming...
Sleeping in your worker thread is unnecessary. It's done sometimes in examples to visualize the flow of a multi-threaded program, but there's no reason you should do it in production code.
When one thread has finished a task, it may as well pick up the next task immediately. It's not going to make your program faster that another thread takes the task instead. The only thing you would achieve is that the worker threads sit idle every now and then.