Is there any way to have a thread terminate itself once it is finished its target function?
I have an application that requires running a wait for connection function every 2min. This function takes around 5 seconds to run.
import time
import threading
count = 0
def fivesecondwait():
time.sleep(5)
print("Finished side function")
t = threading.Thread(target = twentysecondwait)
while True:
if count == 10:
count = 0
t.start()
print("Running Main Application")
count += 1
time.sleep(1)
If I try and join the thread the loop stops running, but if I do not include a join it results in the error "Threads can only be started once". I need the function to be called every 10 seconds and to not block the execution of the loop. So is there any way to have the thread terminate itself so it can be used again?
This is for a measurement device that needs to continue measuring every 50ms while repeatedly waiting for 3-4 seconds for a connection from an MQTT host.
Your question and your code don't quite seem to match up, but this much is true; You said, "it results in the error "Threads can only be started once", and your code appears to create a single thread instance, which it then it attempts to start() more than one time.
The error message is self explanatory. Python will not allow you to call t.start() more than one time on the same thread instance, t. The reason why is, a thread isn't just an object. It's an object that represents an execution of your code (i.e., a trip through your code.)
You can't take the same trip more than one time. Even if you go to the same beach that you visited last year, even if you follow the same route, and stop to rest in all the same places, it's still a different trip. Threads obey the same model.
If you want to run twentysecondwait() multiple times "in the background," there's nothing wrong with that, but you'll have to create and start a new thread instance for each execution.
Is there a practical limit to how many thread instances can be created?
There probably is no practical limit to how many can be sequentially created and destroyed,* but each running thread occupies significant memory and uses other resources as well. So, yes, There is a limit** to how many threads can be "alive" at the same time.
Your example thread function takes 20 seconds to execute. If the program starts a new one every ten seconds, then the number of live threads will increase by one every 20 seconds—180 per hour. The program certainly will not be able to sustain that for 90 days.
* A better performing alternative to contually creating and destroying threads is to use a thread pool such as Python's concurrent.futures.ThreadPoolExecutor.
** The limit is going to depend on what version of Python you are running, on what OS you are running, on how much memory your computer has, etc. The limit probably is not well defined in any case.
A thread will automatically terminate itself when it finished running.
But:
a) you can't start the same thread twice and
b) you can't re-start a thread that terminated.
Simple solution: always start a new thread.
while True:
if count == 10:
count = 0
t = threading.Thread(target = fivesecondwait)
t.start()
...
[...] so it can be used again?
No. You could however keep a single thread running and use synchronization objects to tell the thread when to do something and when not.
This will require more knowledge but be less resource intense.
Related
I currently have a python program that makes API calls indefinitely. Each time it needs to make a new API call it starts a new thread like so:
Thread(target=self.make_api_call, args=(foo, bar)).start()
This works fine, however, after a few days of leaving the program running it always fails to create new threads and fails.
I get the following error multiple times (I assume it is for every thread that the program attempts to create and start):
internal buffer error : Memory allocation failed : growing buffer
Why is this the case? I believe all of my threads terminate after a short period of time (I don't have any reason to believe that any of the threads do not terminate). Is there a limit to the number of threads that can be created in a python program.
I have also looked into using a threadpool; will a thread pool solve my above problem? If so, how would the following logic be translated into a threadpool?
threads = []
for call in range(1, call + 1):
t = Thread(target=self.make_api_call, args=(foo, bar, call))
threads.append(t)
for thread in threads:
thread.start()
for thread in threads:
thread.join()
i am learning about threads and got one thing confusing.
from threading import Thread
from time import sleep
global a
a=0
def th1():
lasta=0
while(a<200):
if(a!=lasta):
lasta=a
print(a)
thrd=Thread(target=th1)
print(a)
thrd.start()
for i in range (1,200):
a+=1
sleep(0)
this prints numbers from 0 to 199, but
from threading import Thread
from time import sleep
global a
a=0
def th1():
lasta=0
while(a<200):
if(a!=lasta):
lasta=a
print(a)
thrd=Thread(target=th1)
print(a)
thrd.start()
for i in range (1,200):
a+=1
this code only prints 0 and 199.
I think what's happening is that in second code there's not the (lats say) stop statement that would make program execute different portion of the code while first one stops the loop and gives another thread the possibility to execute. then it checks if 0 seconds went and continues from for loop. i dont know if im right, please if you could help me explain what is really going on i would be glad.
also how can i master this kind of things? for example to run two threads continuously and let them do stuff according to one global variable. because as i clearly see even though i'm using different thread, they aren't really doing stuff together, they are still waiting for each other
Thanks!
Your specific question is quite correct: a 0-second time-out is quite different from no statement at all. As you guess, this suspends the running thread and allows another thread to receive control of the CPU.
If you use merely multi-threading, then you get the interleaving situation you describe here: you have two logical execution threads, but only one logical processor. For parallel processing, you need multi-processing. There are many tutorials and examples available on line; simply search for "Python multiprocessing tutorial".
Your request "how can i master this kind of things?" is far too general to be a Stack Overflow question. You master this knowledge as you master any knowledge: you find materials that match your learning styles, work through those materials, and practice.
How multiple threads will run depends on the environment. Threads do not 'do stuff together'. They run somewhat independently.
On a single processor machine, or in a program which is running multiple threads on a single processor, they never run at the same time. Each thread gets a time slice. It runs for a fixed amount of time (unless it yields early, as with sleep). When its time slice is finished, the next thread runs for a fixed amount of time, or until it yields.
In your first example, the main thread yields each time it increments, so the th1 thread will run once per increment, and will see each number.
In the second example, the main thread will run a full timeslice before the th1 thread is given execution time. This is sufficient for the loop in the main thread to cycle many times. When the th1 thread runs again, it has 'missed' many values of a.
I am using Threading module in python. How to know how many max threads I can have on my system?
I am using Threading module in python. How to know how many max
threads I can have on my system?
There doesn't seem to be a hard-coded or configurable MAX value that I've ever found, but there is definitely a limit. Run the following program:
import threading
import time
def mythread():
time.sleep(1000)
def main():
threads = 0 #thread counter
y = 1000000 #a MILLION of 'em!
for i in range(y):
try:
x = threading.Thread(target=mythread, daemon=True)
threads += 1 #thread counter
x.start() #start each thread
except RuntimeError: #too many throws a RuntimeError
break
print("{} threads created.\n".format(threads))
if __name__ == "__main__":
main()
I suppose I should mention that this is using Python 3.
The first function, mythread(), is the function which will be executed as a thread. All it does is sleep for 1000 seconds then terminate.
The main() function is a for-loop which tries to start one million threads. The daemon property is set to True simply so that we don't have to clean up all the threads manually.
If a thread cannot be created Python throws a RuntimeError. We catch that to break out of the for-loop and display the number of threads which were successfully created.
Because daemon is set True, all threads terminate when the program ends.
If you run it a few times in a row you're likely to see that a different number of threads will be created each time. On the machine from which I'm posting this reply, I had a minimum 18,835 during one run, and a maximum of 18,863 during another run. And the more you fiddle with the code, as in, the more code you add to this in order to experiment or find more information, you'll find the fewer threads can/will be created.
So, how to apply this to real world.
Well, a server may need the ability to start a triple-digit number of threads, but in most other cases you should re-evaluate your game plan if you think you're going to be generating a large number of threads.
One thing you need to consider if you're using Python: if you're using a standard distribution of Python, your system will only execute one Python thread at a time, including the main thread of your program, so adding more threads to your program or more cores to your system doesn't really get you anything when using the threading module in Python. You can research all of the pedantic details and ultracrepidarian opinions regarding the GIL / Global Interpreter Lock for more info on that.
What that means is that cpu-bound (computationally-intensive) code doesn't benefit greatly from factoring it into threads.
I/O-bound (waiting for file read/write, network read, or user I/O) code, however, benefits greatly from multithreading! So, start a thread for each network connection to your Python-based server.
Threads can also be great for triggering/throwing/raising signals at set periods, or simply to block out the processing sections of your code more logically.
I have a scenario I'm trying to wrap my head arround.
So I have two different servers I need to connect to, each having a different response time (one is fast, one is slow). There's a for loop which runs for a fixed amount of time for both servers, creating threads and starting them. Basically I'm making multiple asynchronous requests to these servers, storing their reply time in a list.
Here's the code:
while time.time() - start_time < run_time:
thread = Thread(target=thread_method, args=())
thread.start()
thread_list.append(thread)
# make sure all threads finish before continuing
for thread in thread_list:
thread.join()
and each thread has something like this:
start_time = time.time()
call_the_server()
end_time = time.time()
latency_list.append(start_time-end_time))
Executing the same script for each server, with the same run_time, give me let's say X items, while the second one X*3.
My thought is that creating the thread, starting it and then moving to the next one in the loop might result in the same number of items in both cases, the only difference being that one script run might finish a little later, as each thread might end later (due to the higher response time of the server).
I could really use an explanation for why i'm getting these results.
Answered by Alexander Ravikovich in a comment.
https://42bits.wordpress.com/2010/10/24/python-global-interpreter-lock-gil-explained-pycon-tech-talk/
or
https://www.youtube.com/watch?v=Obt-vMVdM8s
here's some example code
while True: #main-loop
if command_received:
thread = Thread(target = doItNOW)
thread.start()
......
def doItNOW():
some_blocking_operations()
my problem is I need "some_blocking_operations" to start IMMEDIATELY (as soon as command_received is True).
but since they're blocking i can't execute them on my main loop
and i can't change "some_blocking_operations" to be non-blocking either
for "IMMEDIATELY" i mean as soon as possible, not more than 10ms delay.
(i once got a whole second of delay).
if it's not possible, a constant delay would also be acceptable. (but it MUST be constant. with very few milliseconds of error)
i'm currently working on a linux system (Ubuntu, but it may be another one in the future. always linux)
a python solution would be amazing.. but a different one would be better than nothing
any ideas?
thanks in advance
from threading import Thread
class worker(Thread):
def __init__(self, someParameter=True):
Thread.__init__(self)
# This is how you "send" parameters/variables
# into the thread on start-up that the thread can use.
# This is just an example in case you need it.
self.someVariable = someParameter
self.start() # Note: This makes the thread self-starting,
# you could also call .start() in your main loop.
def run():
# And this is how you use the variable/parameter
# that you passed on when you created the thread.
if self.someVariable is True:
some_blocking_operations()
while True: #main-loop
if command_received:
worker()
This is a non-blocking execution of some_blocking_operations() in a threaded manner. I'm not sure if what you're looking for is to actually wait for the thread to finish or not, or if you even care?
If all you want to do is wait for a "command" to be received and then to execute the blocking operations without waiting for it, verifying that it completes, then this should work for you.
Python mechanics of threading
Python will only run in one CPU core, what you're doing here is simply running multiple executions on overlapping clock invervals in the CPU. Meaning every other cycle in the CPU an execution will be made in the main thread, and the other your blocking call will get a chance to run a execution. They won't actually run in parallel.
There are are some "You can, but..." threads.. Like this one:
is python capable of running on multiple cores?