I have an application that has 3 unique tasks that need to be executed once per second and need to run in parallel. I made the application to have 3 long running threads, basically a thread with a while True and it only breaks when the application is stopped.
I have a Java developer friend that suggested instead to make the task perform then stop and basically execute the same function over and over again, essentially creating a new thread each second. The while True is outside of the threads and creates them over and over. The idea being that if for some reason one dies or hangs, it doesn't cause issues. Any unforeseen errors, the next second it moves on and tries again. I like this idea but it's slightly harder to implement.
Curious what is the best approach for Python:
3 long running threads that loop inside the thread
1 while loop that creates new threads each second
One last note, I need to have some data that carries over each run, so for #2 I would have a class created and a class method would be called as a thread.
#2 would look something like this:
import concurrent.futures
import time
class Test:
def __init__(self) -> None:
super().__init__()
self.counter = 0
def run(self):
print('Test')
self.counter += 1
test1 = Test()
test2 = Test()
with concurrent.futures.ThreadPoolExecutor() as executer:
while True:
t1 = executer.submit(test1.run)
t2 = executer.submit(test2.run)
time.sleep(1.0)
Related
I saw the following code in a thread tutorial:
from time import sleep, perf_counter
from threading import Thread
start = perf_counter()
def foo():
sleep(5)
threads = []
for i in range(100):
t = Thread(target=foo,)
t.start()
threads.append(t)
for i in threads:
i.join()
end = perf_counter()
print(f'Took {end - start}')
When I run it it prints Took 5.014557975. Okay, that part is fine. It does not take 500 seconds as the non threaded version would.
What I don't understand is how .join works. I noticed without calling .join I got Took 0.007060926999999995 which indicates that the main thread ended before the child threads. Since '.join()' is supposed to block, when the first iteration of the loop occurs won't it be blocked and have to wait 5 seconds till the second iteration? How does it still manage to run?
I keep reading python threading is not truly multithreaded and it only appears to be (runs on a single core), but if that is the case then how exactly is the background time running if it's not parallel?
So '.join()' is supposed to block, so when the first iteration of the loop occurs wont it be blocked and it has to wait 5 seconds till the second iteration?
Remember all the threads are started at the same time and all of them take ~5s.
The second for loop waits for all the threads to finish. It will take roughly 5s for the first thread to finish, but the remaining 99 threads will finish roughly at the same time, and so will the remaining 99 iterations of the loop.
By the time you're calling join() on the second thread, it is either already finished or will be within a couple of milliseconds.
I keep reading python threading is not truly multithreaded and it only appears to be (runs on a single core), but if that is the case then how exactly is the background time running if it's not parallel?
It's a topic that has been discussed a lot, so I won't add another page-long answer.
Tl;dr: Yes, Python Multithreading doesn't help with CPU-intensive tasks, but it's just fine for tasks that spend a lot of time on waiting for something else (Network, Disk-I/O, user input, a time-based event).
sleep() belongs to the latter group of tasks, so Multithreading will speed it up, even though it doesn't utilize multiple cores simultaneously.
The OS is in control when the thread starts and the OS will context-switch (I believe that is the correct term) between threads.
time functions access a clock on your computer via the OS - that clock is always running. As long as the OS periodically gives each thread time to access a clock the thread's target can tell if it has been sleeping long enough.
The threads are not running in parallel, the OS periodically gives each one a chance to look at the clock.
Here is a little finer detail for what is happening. I subclassed Thread and overrode its run and join methods to log when they are called.
Caveat The documentation specifically states
only override __init__ and run methods
I was surprised overriding join didn't cause problems.
from time import sleep, perf_counter
from threading import Thread
import pandas as pd
c = {}
def foo(i):
c[i]['foo start'] = perf_counter() - start
sleep(5)
# print(f'{i} - start:{start} end:{perf_counter()}')
c[i]['foo end'] = perf_counter() - start
class Test(Thread):
def __init__(self,*args,**kwargs):
self.i = kwargs['args'][0]
super().__init__(*args,**kwargs)
def run(self):
# print(f'{self.i} - started:{perf_counter()}')
c[self.i]['thread start'] = perf_counter() - start
super().run()
def join(self):
# print(f'{self.i} - joined:{perf_counter()}')
c[self.i]['thread joined'] = perf_counter() - start
super().join()
threads = []
start = perf_counter()
for i in range(10):
c[i] = {}
t = Test(target=foo,args=(i,))
t.start()
threads.append(t)
for i in threads:
i.join()
df = pd.DataFrame(c)
print(df)
0 1 2 3 4 5 6 7 8 9
thread start 0.000729 0.000928 0.001085 0.001245 0.001400 0.001568 0.001730 0.001885 0.002056 0.002215
foo start 0.000732 0.000931 0.001088 0.001248 0.001402 0.001570 0.001732 0.001891 0.002058 0.002217
thread joined 0.002228 5.008274 5.008300 5.008305 5.008323 5.008327 5.008330 5.008333 5.008336 5.008339
foo end 5.008124 5.007982 5.007615 5.007829 5.007672 5.007899 5.007724 5.007758 5.008051 5.007549
Hopefully you can see that all the threads are started in sequence very close together; once thread 0 is joined nothing else happens till it stops (foo ends) then each of the other threads are joined and terminate.
Sometimes a thread terminates before it is even joined - for threads one plus foo ends before the thread is joined.
What is the best way to update a gui from another thread in python.
I have main function (GUI) in thread1 and from this i'm referring another thread (thread2), is it possible to update GUI while working in Thread2 without cancelling work at thread2, if it is yes how can I do that?
any suggested reading about thread handling. ?
Of course you can use Threading to run several processes simultaneously.
You have to create a class like this :
from threading import Thread
class Work(Thread):
def __init__(self):
Thread.__init__(self)
self.lock = threading.Lock()
def run(self): # This function launch the thread
(your code)
if you want run several thread at the same time :
def foo():
i = 0
list = []
while i < 10:
list.append(Work())
list[i].start() # Start call run() method of the class above.
i += 1
Be careful if you want to use the same variable in several threads. You must lock this variable so that they do not all reach this variable at the same time. Like this :
lock = threading.Lock()
lock.acquire()
try:
yourVariable += 1 # When you call lock.acquire() without arguments, block all variables until the lock is unlocked (lock.release()).
finally:
lock.release()
From the main thread, you can call join() on the queue to wait until all pending tasks have been completed.
This approach has the benefit that you are not creating and destroying threads, which is expensive. The worker threads will run continuously, but will be asleep when no tasks are in the queue, using zero CPU time.
I hope it will help you.
first of all I have to mention I'm not a programmer but a mechanical engineer so please don't crucify me if I misinterpret something or say some nonsense.
I want to write a python code witch will be "simulating" a real life problem. The real life problem is something like FIFO queue, where objects are taken from on different stations and they spend there some time and then they are returned back to queue.
What I understand what I need is to write an asynchronous Programm, because I have one Function which is putting objects to queue (let say every 15 second) and then I have some stations which take only one object from this queue and then are working on it for some time again (simple timer and a print "Hi im working on object x, will return it in: minutes".
I'm not sure if I can do it with Threading? What if I had 100 stations which work asynchronous is it possible to start 100 Threads? Because as I understand every thread should have one timer?
I would ask to give me a little push to the simplest direction to solve it, it doesn't have to be pretty but functional and easy for me.
thank you in advance for each idea!
Best regards,
MM.
Of course you can use Threading to run several processes simultaneously.
You have to create a class like this :
from threading import Thread
class Work(Thread):
def __init__(self):
Thread.__init__(self)
self.lock = threading.Lock()
def run(self): # This function launch the thread
(your code)
if you want run several thread at the same time :
def foo():
i = 0
list = []
while i < 10:
list.append(Work())
list[i].start() # Start call run() method of the class above.
i += 1
Be careful if you want to use the same variable in several threads. You must lock this variable so that they do not all reach this variable at the same time. Like this :
lock = threading.Lock()
lock.acquire()
try:
yourVariable += 1 # When you call lock.acquire() without arguments, block all variables until the lock is unlocked (lock.release()).
finally:
lock.release()
From the main thread, you can call join() on the queue to wait until all pending tasks have been completed.
This approach has the benefit that you are not creating and destroying threads, which is expensive. The worker threads will run continuously, but will be asleep when no tasks are in the queue, using zero CPU time.
I hope it will help you.
I was attempting to create a thread class that could be terminated by an exception (since I am trying to have the thread wait on an event) when I created the following:
import sys
class testThread(threading.Thread):
def __init__(self):
super(testThread,self).__init__()
self.daemon = True
def run(self):
try:
print('Running')
while 1:
pass
except:
print('Being forced to exit')
test1 = testThread()
test2 = testThread()
print(test1.daemon)
test1.run()
test2.run()
sys.exit()
However, running the program will only print out one Running message, until the other is terminated. Why is that?
The problem is that you're calling the run method.
This is just a plain old method that you implement, which does whatever you put in its body. In this case, the body is an infinite loop, so calling run just loops forever.
The way to start a thread is the start method. This method is part of the Thread class, and what it does is:
Start the thread’s activity.
It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.
So, if you call this, it will start a new thread, make that new thread run your run() method, and return immediately, so the main thread can keep doing other stuff.1 That's what you want here.
1. As pointed out by Jean-François Fabre, you're still not going to get any real parallelism here. Busy loops are never a great idea in multithreaded code, and if you're running this in CPython or PyPy, almost all of that busy looping is executing Python bytecode while holding the GIL, and only one thread can hold the GIL at a time. So, from a coarse view, things look concurrent—three threads are running, and all making progress. But if you zoom in, there's almost no overlap where two threads progress at once, usually not even enough to make up for the small scheduler overhead.
This question already has answers here:
Run certain code every n seconds [duplicate]
(7 answers)
Closed 8 years ago.
I have a threaded class whose loop needs to execute 4 times every second. I know that I can do something like
do_stuff()
time.sleep(0.25)
but the problem is that is doesn't account for the time it takes to do_stuff(). Effectively this needs to be a real-time thread. Is there a way to accomplish this? Ideally the thread would still be put to sleep when not executing code.
The simple solution
import threading
def work ():
threading.Timer(0.25, work).start ()
print "stackoverflow"
work ()
The above will make sure that work is run with an interval of four times per second, the theory behind this is that it will "queue" a call to itself that will be run 0.25 seconds into the future, without hanging around waiting for that to happen.
Because of this it can do it's work (almost) entirely uninterrupted, and we are extremely close to executing the function exactly 4 times per second.
More about threading.Timer can be read by following the below link to the python documentation:
docs.python.org - 16.2.7. Timer Objects
RECOMMENDED] The more advanced/dynamic solution
Even though the previous function works as expected you could create a helper function to aid in dealing with future timed events.
Something as the below will be sufficient for this example, hopefully the code will speak for itself - it is not as advanced as it might appear.
See this as an inspiration when you might implement your own wrapper to fit your exact needs.
import threading
def do_every (interval, worker_func, iterations = 0):
if iterations != 1:
threading.Timer (
interval,
do_every, [interval, worker_func, 0 if iterations == 0 else iterations-1]
).start ()
worker_func ()
def print_hw ():
print "hello world"
def print_so ():
print "stackoverflow"
# call print_so every second, 5 times total
do_every (1, print_so, 5)
# call print_hw two times per second, forever
do_every (0.5, print_hw)
I did a bit different approach with one Thread, looping in a while loop.
For me the advantages are:
Only one Thread, the other solutions mentioned here starting and stopping threads for every interval
More control for the Interval, you are able to stop the IntervalTimer with .stop() method
Code:
from threading import Thread, Event
# StoppableThread is from user Dolphin, from http://stackoverflow.com/questions/5849484/how-to-exit-a-multithreaded-program
class StoppableThread(Thread):
def __init__(self):
Thread.__init__(self)
self.stop_event = Event()
def stop(self):
if self.isAlive() == True:
# set event to signal thread to terminate
self.stop_event.set()
# block calling thread until thread really has terminated
self.join()
class IntervalTimer(StoppableThread):
def __init__(self, interval, worker_func):
super().__init__()
self._interval = interval
self._worker_func = worker_func
def run(self):
while not self.stop_event.is_set():
self._worker_func()
sleep(self._interval)
Usage:
def hw():
print("Hello World")
interval = IntervalTimer(1,hw)
interval.start()
sleep(10)
interval.stop()