This question already has answers here:
How to share a variable between 2 threads
(1 answer)
What is the use of join() in threading?
(12 answers)
Closed 1 year ago.
I have a while True loop that needs to be run in the background to update a variable and a function that needs to return that same variable.
I tried this as a test:
import threading
downspeed = 0
def updating():
while True:
downspeed = downspeed+1
def main():
while True:
print(downspeed)
u = threading.Thread(name='background', target=updating)
m = threading.Thread(name='foreground', target=main)
u.start()
m.start()
But it only returns 0
Your code is not really a sufficient test. The test ought to include:
Allowing some time for the threads to run to see what they do.
Make the threads sleep, at least for a little time, to allow task switches to occur.
See this code:
import threading
import time
can_run = True
downspeed = 0
def updating():
global downspeed
while can_run:
downspeed = downspeed+1
time.sleep(0.1) # Allow taskswitches
def main():
while can_run:
print(downspeed)
time.sleep(1) # Allow taskswitches
u = threading.Thread(name='background', target=updating)
m = threading.Thread(name='foreground', target=main)
u.start()
m.start()
time.sleep(10) # Allow time for the threads to do what they want
can_run = False
print('Done')
u.join()
m.join()
There are no problems sharing variables in python because of the GIL. Although this only makes changes atomic.
Related
This question already has answers here:
Is there any way to kill a Thread?
(31 answers)
Closed 2 years ago.
I want to create threads that will add something to an array, but, if they don't do that in less than 2 seconds, I want to terminate them.
This is a prof of concept, so the code is simple. Every second I want a thread to add that item in the list, so a thread runs after 0, 1, 2, 3 and 4 seconds. The idea is to not let the thread 3 and 4 run.
import threading, time
myList = []
def foo(value):
global myList
time.sleep(value)
print("Value: {}".format(value))
myList.append(value)
threads = []
for i in range(5):
th = threading.Thread(target=foo, args=(i,))
threads.append(th)
for th in threads:
th.start()
What do I do now? I tried using some other logic like using
th.join(timeout)
But that doesn't seem to work.
As I said in a comment you can't really "kill" a thread (externally). However they can "commit suicide" by returning or raising a exception.
Below is example of doing the latter when the thread's execution time has exceeded a given amount of time. Note that this is not the same as doing a join(timeout) call, which only blocks until the thread ends or the specified amount of time has elapsed. That's why the printing of value and its appending to the list happens regardless of whether the thread finishes before the call to join() times-out or not.
I got the basic idea of using sys.settrace() from the tutorial titled Different ways to kill a Thread — although my implementation is slightly different. Also note that this approach likely introduces a significant amount of overhead.
import sys
import threading
import time
class TimelimitedThread(threading.Thread):
def __init__(self, *args, time_limit, **kwargs):
self.time_limit = time_limit
self._run_backup = self.run # Save superclass run() method.
self.run = self._run # Change it to custom version.
super().__init__(*args, **kwargs)
def _run(self):
self.start_time = time.time()
sys.settrace(self.globaltrace)
self._run_backup() # Call superclass run().
self.run = self._run_backup # Restore original.
def globaltrace(self, frame, event, arg):
return self.localtrace if event == 'call' else None
def localtrace(self, frame, event, arg):
if(event == 'line' and
time.time()-self.start_time > self.time_limit): # Over time?
raise SystemExit() # Terminate thread.
return self.localtrace
THREAD_TIME_LIMIT = 2.1 # Secs
threads = []
my_list = []
def foo(value):
global my_list
time.sleep(value)
print("Value: {}".format(value))
my_list.append(value)
for i in range(5):
th = TimelimitedThread(target=foo, args=(i,), time_limit=THREAD_TIME_LIMIT)
threads.append(th)
for th in threads:
th.start()
for th in threads:
th.join()
print('\nResults:')
print('my_list:', my_list)
Output:
Value: 0
Value: 1
Value: 2
Results:
my_list: [0, 1, 2]
Join() is used to wait for the respective thread to finish. To terminate a thread, use stop().. You can try as follows:
time.sleep(N)
th.join()
I'm quite new on python and working on a school project with this logic: Users have to answer a series of questions as fast as they can, within the given time.
For instance, the time allotted is 30 seconds, I wood loop through a dictionary of questions and get the answer. On timeout, the loop will start, even if the script is still waiting for an input.
def start_test():
for item on questions:
print(item)
answers.append(input(' : '))
I've tried using multiprocessing and multithreading, but I found out that stdin doesn't work subprocesses.
I'm looking for something like:
while duration > 0:
start_test()
def countdown():
global duration
while duration > 0:
duration -= 1
time.sleep(1)
# something lime start_test().stop()
But I can't figure out how to run the countdown function in parallel with the start_test function.
Any ideas?
So as far as I know the input is accessible via main thread only. I might be wrong.
However if that is the case, you need a non-blocking input.
Check this blog. The answer below is based on that.
Note: This is a really quick and dirty solution.
I have checked this on Linux.
If it doesn't work on Windows try this
link for further reference.
import _thread
import sys
import select
import time
def start_test():
questions = ['1','2','3']
answers = []
for item in questions:
print(item)
# Input in a non-blocking way
loop_flag = True
while loop_flag:
# Read documenation and examples on select
ready = select.select([sys.stdin], [], [], 0)[0]
if not ready:
# Check if timer has expired
if timeout:
return answers
else:
for file in ready:
line = file.readline()
if not line: # EOF, input is closed
loop_flag = False
break
elif line.rstrip():
# We have some input
answers.append(line)
# So as to get out of while
loop_flag = False
# Breaking out of for
break
return answers
def countdown():
global timeout
time.sleep(30)
timeout = True
# Global Timeout Flag
timeout = False
timer = _thread.start_new_thread(countdown, ())
answers = start_test()
print(answers)
I want to move some functions to an external file for making it clearer.
lets say i have this example code (which does indeed work):
import threading
from time import sleep
testVal = 0
def testFunc():
while True:
global testVal
sleep(1)
testVal = testVal + 1
print(testVal)
t = threading.Thread(target=testFunc, args=())
t.daemon = True
t.start()
try:
while True:
sleep(2)
print('testval = ' + str(testVal))
except KeyboardInterrupt:
pass
now i want to move testFunc() to a new python file. My guess was the following but the global variables don't seem to be the same.
testserver.py:
import threading
import testclient
from time import sleep
testVal = 0
t = threading.Thread(target=testclient.testFunc, args=())
t.daemon = True
t.start()
try:
while True:
sleep(2)
print('testval = ' + str(testVal))
except KeyboardInterrupt:
pass
and testclient.py:
from time import sleep
from testserver import testVal as val
def testFunc():
while True:
global val
sleep(1)
val = val + 1
print(val)
my output is:
1
testval = 0
2
3
testval = 0 (testval didn't change)
...
while it should:
1
testval = 1
2
3
testval = 3
...
any suggestions? Thanks!
Your immediate problem is not due to multithreading (we'll get to that) but due to how you use global variables. The thing is, when you use this:
from testserver import testVal as val
You're essentially doing this:
import testserver
val = testserver.testVal
i.e. you're creating a local reference val that points to the testserver.testVal value. This is all fine and dandy when you read it (the first time at least) but when you try to assign its value in your function with:
val = val + 1
You're actually re-assigning the local (to testclient.py) val variable, not setting the value of testserver.testVal. You have to directly reference the actual pointer (i.e. testserver.testVal += 1) if you want to change its value.
That being said, the next problem you might encounter might stem directly from multithreading - you can encounter a race-condition oddity where GIL pauses one thread right after reading the value, but before actually writing it, and the next thread reading it and overwriting the current value, then the first thread resumes and writes the same value resulting in single increase despite two calls. You need to use some sort of mutex to make sure that all non-atomic operations execute exclusively to one thread if you want to use your data this way. The easiest way to do it is with a Lock that comes with the threading module:
testserver.py:
# ...
testVal = 0
testValLock = threading.Lock()
# ...
testclient.py:
# ...
with testserver.testValLock:
testserver.testVal += 1
# ...
A third and final problem you might encounter is a circular dependency (testserver.py requires testclient.py, which requires testserver.py) and I'd advise you to re-think the way you want to approach this problem. If all you want is a common global store - create it separately from modules that might depend on it. That way you ensure proper loading and initializing order without the danger of unresolveable circular dependencies.
I am using process pools(including 3 processes). In every process, I have set (created) some threads by using the thread classes to speed handle something.
At first, everything was OK. But when I wanted to change some variable in a thread, I met an odd situation.
For testing or to know what happens, I set a global variable COUNT to test. Honestly, I don't know this is safe or not. I just want to see, by using multiprocessing and threading can I change COUNT or not?
#!/usr/bin/env python
# encoding: utf-8
import os
import threading
from Queue import Queue
from multiprocessing import Process, Pool
# global variable
max_threads = 11
Stock_queue = Queue()
COUNT = 0
class WorkManager:
def __init__(self, work_queue_size=1, thread_pool_size=1):
self.work_queue = Queue()
self.thread_pool = [] # initiate, no have a thread
self.work_queue_size = work_queue_size
self.thread_pool_size = thread_pool_size
self.__init_work_queue()
self.__init_thread_pool()
def __init_work_queue(self):
for i in xrange(self.work_queue_size):
self.work_queue.put((func_test, Stock_queue.get()))
def __init_thread_pool(self):
for i in xrange(self.thread_pool_size):
self.thread_pool.append(WorkThread(self.work_queue))
def finish_all_threads(self):
for i in xrange(self.thread_pool_size):
if self.thread_pool[i].is_alive():
self.thread_pool[i].join()
class WorkThread(threading.Thread):
def __init__(self, work_queue):
threading.Thread.__init__(self)
self.work_queue = work_queue
self.start()
def run(self):
while self.work_queue.qsize() > 0:
try:
func, args = self.work_queue.get(block=False)
func(args)
except Queue.Empty:
print 'queue is empty....'
def handle(process_name):
print process_name, 'is running...'
work_manager = WorkManager(Stock_queue.qsize()/3, max_threads)
work_manager.finish_all_threads()
def func_test(num):
# use a global variable to test what happens
global COUNT
COUNT += num
def prepare():
# prepare test queue, store 50 numbers in Stock_queue
for i in xrange(50):
Stock_queue.put(i)
def main():
prepare()
pools = Pool()
# set 3 process
for i in xrange(3):
pools.apply_async(handle, args=('process_'+str(i),))
pools.close()
pools.join()
global COUNT
print 'COUNT: ', COUNT
if __name__ == '__main__':
os.system('printf "\033c"')
main()
Now, finally the result of COUNT is just 0.I am unable to understand whats happening here?
You print the COUNT var in the father process. Variables doesn't sync across processes because they doesn't share memory, that means that the variable stay 0 at the father process and is increased in the subprocesses
In the case of threading, threads share memory, that means that they share the variable count, so they should have COUNT as more than 0 but again they are at the subprocesses, and when they change the variable, it doesn't update it in other processes.
I have an application that fires up a series of threads. Occassionally, one of these threads dies (usually due to a network problem). How can I properly detect a thread crash and restart just that thread? Here is example code:
import random
import threading
import time
class MyThread(threading.Thread):
def __init__(self, pass_value):
super(MyThread, self).__init__()
self.running = False
self.value = pass_value
def run(self):
self.running = True
while self.running:
time.sleep(0.25)
rand = random.randint(0,10)
print threading.current_thread().name, rand, self.value
if rand == 4:
raise ValueError('Returned 4!')
if __name__ == '__main__':
group1 = []
group2 = []
for g in range(4):
group1.append(MyThread(g))
group2.append(MyThread(g+20))
for m in group1:
m.start()
print "Now start second wave..."
for p in group2:
p.start()
In this example, I start 4 threads then I start 4 more threads. Each thread randomly generates an int between 0 and 10. If that int is 4, it raises an exception. Notice that I don't join the threads. I want both group1 and group2 list of threads to be running. I found that if I joined the threads it would wait until the thread terminated. My thread is supposed to be a daemon process, thus should rarely (if ever) hit the ValueError Exception this example code is showing and should be running constantly. By joining it, the next set of threads doesn't begin.
How can I detect that a specific thread died and restart just that one thread?
I have attempted the following loop right after my for p in group2 loop.
while True:
# Create a copy of our groups to iterate over,
# so that we can delete dead threads if needed
for m in group1[:]:
if not m.isAlive():
group1.remove(m)
group1.append(MyThread(1))
for m in group2[:]:
if not m.isAlive():
group2.remove(m)
group2.append(MyThread(500))
time.sleep(5.0)
I took this method from this question.
The problem with this, is that isAlive() seems to always return True, because the threads never restart.
Edit
Would it be more appropriate in this situation to use multiprocessing? I found this tutorial. Is it more appropriate to have separate processes if I am going to need to restart the process? It seems that restarting a thread is difficult.
It was mentioned in the comments that I should check is_active() against the thread. I don't see this mentioned in the documentation, but I do see the isAlive that I am currently using. As I mentioned above, though, this returns True, thus I'm never able to see that a thread as died.
I had a similar issue and stumbled across this question. I found that join takes a timeout argument, and that is_alive will return False once the thread is joined. So my audit for each thread is:
def check_thread_alive(thr):
thr.join(timeout=0.0)
return thr.is_alive()
This detects thread death for me.
You could potentially put in an a try except around where you expect it to crash (if it can be anywhere you can do it around the whole run function) and have an indicator variable which has its status.
So something like the following:
class MyThread(threading.Thread):
def __init__(self, pass_value):
super(MyThread, self).__init__()
self.running = False
self.value = pass_value
self.RUNNING = 0
self.FINISHED_OK = 1
self.STOPPED = 2
self.CRASHED = 3
self.status = self.STOPPED
def run(self):
self.running = True
self.status = self.RUNNING
while self.running:
time.sleep(0.25)
rand = random.randint(0,10)
print threading.current_thread().name, rand, self.value
try:
if rand == 4:
raise ValueError('Returned 4!')
except:
self.status = self.CRASHED
Then you can use your loop:
while True:
# Create a copy of our groups to iterate over,
# so that we can delete dead threads if needed
for m in group1[:]:
if m.status == m.CRASHED:
value = m.value
group1.remove(m)
group1.append(MyThread(value))
for m in group2[:]:
if m.status == m.CRASHED:
value = m.value
group2.remove(m)
group2.append(MyThread(value))
time.sleep(5.0)