Related
I want to run multiple threads in parallel. Each thread picks up a task from a task queue and executes that task.
from threading import Thread
from Queue import Queue
import time
class link(object):
def __init__(self, i):
self.name = str(i)
def run_jobs_in_parallel(consumer_func, jobs, results, thread_count,
async_run=False):
def consume_from_queue(jobs, results):
while not jobs.empty():
job = jobs.get()
try:
results.append(consumer_func(job))
except Exception as e:
print str(e)
results.append(False)
finally:
jobs.task_done()
#start worker threads
if jobs.qsize() < thread_count:
thread_count = jobs.qsize()
for tc in range(1,thread_count+1):
worker = Thread(
target=consume_from_queue,
name="worker_{0}".format(str(tc)),
args=(jobs,results,))
worker.start()
if not async_run:
jobs.join()
def create_link(link):
print str(link.name)
time.sleep(10)
return True
def consumer_func(link):
return create_link(link)
# create_link takes a while to execute
jobs = Queue()
results = list()
for i in range(0,10):
jobs.put(link(i))
run_jobs_in_parallel(consumer_func, jobs, results, 25, async_run=False)
Now what is happening is, let say we have 10 link objects in jobs queue, while the threads are running in parallel, multiple threads are executing same task. How can I prevent this from happening?
Note - the above sample code does not have the problem describe above, but i have exactly same code except create_link method does some complex stuff.
I think what you need is a lock object (docs,tutorial+examples). If you create an instance of such an object you can 'lock' some parts of your code, ensuring that only one thread executes this part at a time.
I guess in your case you want to lock the line job = jobs.get().
First you have to create the lock in a scope where all threads have access to it. (You don't want a lock for every thread but a single lock for all your threads. That means creating the lock within your thread just before acquiring it won't work)
import threading
lock = threading.Lock()
then you can use it on your line like:
lock.acquire()
job = jobs.get()
lock.release()
or
with lock:
job = jobs.get()
The first thread to reach acquire() will lock the lock. other threads that try to acquire() the lock will pause until the lock gets unlocked again by calling release().
What's the proper way to tell a looping thread to stop looping?
I have a fairly simple program that pings a specified host in a separate threading.Thread class. In this class it sleeps 60 seconds, the runs again until the application quits.
I'd like to implement a 'Stop' button in my wx.Frame to ask the looping thread to stop. It doesn't need to end the thread right away, it can just stop looping once it wakes up.
Here is my threading class (note: I haven't implemented looping yet, but it would likely fall under the run method in PingAssets)
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
def run(self):
config = controller.getConfig()
fmt = config['timefmt']
start_time = datetime.now().strftime(fmt)
try:
if onlinecheck.check_status(self.asset):
status = "online"
else:
status = "offline"
except socket.gaierror:
status = "an invalid asset tag."
msg =("{}: {} is {}. \n".format(start_time, self.asset, status))
wx.CallAfter(self.window.Logger, msg)
And in my wxPyhton Frame I have this function called from a Start button:
def CheckAsset(self, asset):
self.count += 1
thread = PingAssets(self.count, asset, self)
self.threads.append(thread)
thread.start()
Threaded stoppable function
Instead of subclassing threading.Thread, one can modify the function to allow
stopping by a flag.
We need an object, accessible to running function, to which we set the flag to stop running.
We can use threading.currentThread() object.
import threading
import time
def doit(arg):
t = threading.currentThread()
while getattr(t, "do_run", True):
print ("working on %s" % arg)
time.sleep(1)
print("Stopping as you wish.")
def main():
t = threading.Thread(target=doit, args=("task",))
t.start()
time.sleep(5)
t.do_run = False
if __name__ == "__main__":
main()
The trick is, that the running thread can have attached additional properties. The solution builds
on assumptions:
the thread has a property "do_run" with default value True
driving parent process can assign to started thread the property "do_run" to False.
Running the code, we get following output:
$ python stopthread.py
working on task
working on task
working on task
working on task
working on task
Stopping as you wish.
Pill to kill - using Event
Other alternative is to use threading.Event as function argument. It is by
default False, but external process can "set it" (to True) and function can
learn about it using wait(timeout) function.
We can wait with zero timeout, but we can also use it as the sleeping timer (used below).
def doit(stop_event, arg):
while not stop_event.wait(1):
print ("working on %s" % arg)
print("Stopping as you wish.")
def main():
pill2kill = threading.Event()
t = threading.Thread(target=doit, args=(pill2kill, "task"))
t.start()
time.sleep(5)
pill2kill.set()
t.join()
Edit: I tried this in Python 3.6. stop_event.wait() blocks the event (and so the while loop) until release. It does not return a boolean value. Using stop_event.is_set() works instead.
Stopping multiple threads with one pill
Advantage of pill to kill is better seen, if we have to stop multiple threads
at once, as one pill will work for all.
The doit will not change at all, only the main handles the threads a bit differently.
def main():
pill2kill = threading.Event()
tasks = ["task ONE", "task TWO", "task THREE"]
def thread_gen(pill2kill, tasks):
for task in tasks:
t = threading.Thread(target=doit, args=(pill2kill, task))
yield t
threads = list(thread_gen(pill2kill, tasks))
for thread in threads:
thread.start()
time.sleep(5)
pill2kill.set()
for thread in threads:
thread.join()
This has been asked before on Stack. See the following links:
Is there any way to kill a Thread in Python?
Stopping a thread after a certain amount of time
Basically you just need to set up the thread with a stop function that sets a sentinel value that the thread will check. In your case, you'll have the something in your loop check the sentinel value to see if it's changed and if it has, the loop can break and the thread can die.
I read the other questions on Stack but I was still a little confused on communicating across classes. Here is how I approached it:
I use a list to hold all my threads in the __init__ method of my wxFrame class: self.threads = []
As recommended in How to stop a looping thread in Python? I use a signal in my thread class which is set to True when initializing the threading class.
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
self.signal = True
def run(self):
while self.signal:
do_stuff()
sleep()
and I can stop these threads by iterating over my threads:
def OnStop(self, e):
for t in self.threads:
t.signal = False
I had a different approach. I've sub-classed a Thread class and in the constructor I've created an Event object. Then I've written custom join() method, which first sets this event and then calls a parent's version of itself.
Here is my class, I'm using for serial port communication in wxPython app:
import wx, threading, serial, Events, Queue
class PumpThread(threading.Thread):
def __init__ (self, port, queue, parent):
super(PumpThread, self).__init__()
self.port = port
self.queue = queue
self.parent = parent
self.serial = serial.Serial()
self.serial.port = self.port
self.serial.timeout = 0.5
self.serial.baudrate = 9600
self.serial.parity = 'N'
self.stopRequest = threading.Event()
def run (self):
try:
self.serial.open()
except Exception, ex:
print ("[ERROR]\tUnable to open port {}".format(self.port))
print ("[ERROR]\t{}\n\n{}".format(ex.message, ex.traceback))
self.stopRequest.set()
else:
print ("[INFO]\tListening port {}".format(self.port))
self.serial.write("FLOW?\r")
while not self.stopRequest.isSet():
msg = ''
if not self.queue.empty():
try:
command = self.queue.get()
self.serial.write(command)
except Queue.Empty:
continue
while self.serial.inWaiting():
char = self.serial.read(1)
if '\r' in char and len(msg) > 1:
char = ''
#~ print('[DATA]\t{}'.format(msg))
event = Events.PumpDataEvent(Events.SERIALRX, wx.ID_ANY, msg)
wx.PostEvent(self.parent, event)
msg = ''
break
msg += char
self.serial.close()
def join (self, timeout=None):
self.stopRequest.set()
super(PumpThread, self).join(timeout)
def SetPort (self, serial):
self.serial = serial
def Write (self, msg):
if self.serial.is_open:
self.queue.put(msg)
else:
print("[ERROR]\tPort {} is not open!".format(self.port))
def Stop(self):
if self.isAlive():
self.join()
The Queue is used for sending messages to the port and main loop takes responses back. I've used no serial.readline() method, because of different end-line char, and I have found the usage of io classes to be too much fuss.
Depends on what you run in that thread.
If that's your code, then you can implement a stop condition (see other answers).
However, if what you want is to run someone else's code, then you should fork and start a process. Like this:
import multiprocessing
proc = multiprocessing.Process(target=your_proc_function, args=())
proc.start()
now, whenever you want to stop that process, send it a SIGTERM like this:
proc.terminate()
proc.join()
And it's not slow: fractions of a second.
Enjoy :)
My solution is:
import threading, time
def a():
t = threading.currentThread()
while getattr(t, "do_run", True):
print('Do something')
time.sleep(1)
def getThreadByName(name):
threads = threading.enumerate() #Threads list
for thread in threads:
if thread.name == name:
return thread
threading.Thread(target=a, name='228').start() #Init thread
t = getThreadByName('228') #Get thread by name
time.sleep(5)
t.do_run = False #Signal to stop thread
t.join()
I find it useful to have a class, derived from threading.Thread, to encapsulate my thread functionality. You simply provide your own main loop in an overridden version of run() in this class. Calling start() arranges for the object’s run() method to be invoked in a separate thread.
Inside the main loop, periodically check whether a threading.Event has been set. Such an event is thread-safe.
Inside this class, you have your own join() method that sets the stop event object before calling the join() method of the base class. It can optionally take a time value to pass to the base class's join() method to ensure your thread is terminated in a short amount of time.
import threading
import time
class MyThread(threading.Thread):
def __init__(self, sleep_time=0.1):
self._stop_event = threading.Event()
self._sleep_time = sleep_time
"""call base class constructor"""
super().__init__()
def run(self):
"""main control loop"""
while not self._stop_event.isSet():
#do work
print("hi")
self._stop_event.wait(self._sleep_time)
def join(self, timeout=None):
"""set stop event and join within a given time period"""
self._stop_event.set()
super().join(timeout)
if __name__ == "__main__":
t = MyThread()
t.start()
time.sleep(5)
t.join(1) #wait 1s max
Having a small sleep inside the main loop before checking the threading.Event is less CPU intensive than looping continuously. You can have a default sleep time (e.g. 0.1s), but you can also pass the value in the constructor.
Sometimes you don't have control over the running target. In those cases you can use signal.pthread_kill to send a stop signal.
from signal import pthread_kill, SIGTSTP
from threading import Thread
from itertools import count
from time import sleep
def target():
for num in count():
print(num)
sleep(1)
thread = Thread(target=target)
thread.start()
sleep(5)
pthread_kill(thread.ident, SIGTSTP)
result
0
1
2
3
4
[14]+ Stopped
I'm writing a Linux driver for a USB HID device in Python. The device has two ways it sends data, both of which are needed: feature reports (synchronous) and input reports (asynchronous). Using the hidapi Cython library I have only one instance of the device to work with, but I need to set up a listener for hid.read() that will run constantly AND allow synchronous methods to be called at will by the application to send feature reports.
Currently I have the listener in one thread and the synchronous calls in another. When I run the program, my synchronous calls are not happening, though they work fine if I never start the listener; so it appears the listener thread is taking over.
Below is the troubling piece of code:
app.py
# app.py
import threading
import time
import myhiddriver
# Code that sets mydevice
class Reader:
def start(self, device):
self.requests = myhiddriver.DeviceRequest(device)
# Wait a bit before firing, for testing
time.sleep(3)
self.requests.request_swipe_card()
def start_listener(self, device):
self.listener = myhiddriver.DeviceListener(device)
reader = Reader()
thread1 = threading.Thread(target=reader.start, args=(mydevice,))
thread1.daemon = True
thread2 = threading.Thread(target=reader.start_listener, args=(mydevice,))
thread2.daemon = True
thread1.start()
thread2.start()
# Keep this puppy running
while True:
pass
myhiddriver.py
import threading
LOCK = threading.Lock()
class DeviceRequest:
def __init__(self, device):
# Lock it up first
LOCK.acquire()
self.device = device
LOCK.release()
def request_swipe_card(self):
# Lock this up too
LOCK.acquire()
self.device.set_feature_report(insert data here)
LOCK.release()
class DeviceListener:
def __init__(self, device):
# Lock me up
LOCK.acquire()
self.device = device
self.start_listener()
LOCK.release()
def start_listener(self):
while True:
# Should I be locking this up?
LOCK.acquire()
data = self.device.read(255)
LOCK.release()
if data:
print data
else:
pass
My Question: Why is my synchronous call (request_swipe_card) not following through? Or, how can I better architect this to have an endless listener AND the ability to make synchronous calls on the same object?
from the looks of the code, it is because you are locking it up
When the state is unlocked, acquire() changes the state to locked and
returns immediately. When the state is locked, acquire() blocks until
a call to release() in another thread changes it to unlocked
here is the problem:
class DeviceListener:
def __init__(self, device):
LOCK.acquire() # first you lock it up
self.device = device
self.start_listener()
LOCK.release()
def start_listener(self):
while True: # because of the loop, the lock wouldn't get release even if the LOCK below doesn't exist
LOCK.acquire() # it end up blocking here and oops, it locked up xD
data = self.device.read(255) # so it wouldn't be able to read here
LOCK.release()
if data:
print data
else:
pass
and when the request_swipe_card end up being call in the other thread, it end up blocking there too
def request_swipe_card(self):
LOCK.acquire() # right here xD
self.device.set_feature_report(insert data here)
LOCK.release()
I'm sorry if it is a stupid question. I am trying to use a number of classes of multi-threading to finish different jobs, which involves invoking these multi-threadings at different times for many times. But I am not sure which method to use. The code looks like this:
class workers1(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
do some stuff
class workers2(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
do some stuff
class workers3(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
do some stuff
WorkerList1=[workers1(i) for i in range(X)]
WorkerList2=[workers2(i) for i in range(XX)]
WorkerList2=[workers3(i) for i in range(XXX)]
while True:
for thread in WorkerList1:
thread.run (start? join? or?)
for thread in WorkerList2:
thread.run (start? join? or?)
for thread in WorkerList3:
thread.run (start? join? or?)
do sth .
I am trying to have all the threads in all the WorkerList to start functioning at the same time, or at least start around the same time. After sometime once they were all terminated, I would like to invoke all the threads again.
If there were no loop, I can just use .start; but since I can only start a thread once, start apparently does not fit here. If I use run, it seems that all the threads start sequentially, not only the threads in the same list, but also threads from different lists.
Can anyone please help?
there are a lot of misconceptions here:
you can only start a specific instance of a thread once. but in your case, the for loop is looping over different instances of a thread, each instance being assigned to the variable thread in the loop, so there is no problem at all in calling the start() method over each thread. (you can think of it as if the variable thread is an alias of the Thread() object instantiated in your list)
run() is not the same as join(): calling run() performs as if you were programming sequentially. the run() method does not start a new thread, it simply execute the statements in in the method, as for any other function call.
join() does not start executing anything: it only waits for a thread to finish. in order for join() to work properly for a thread, you have to call start() on this thread first.
additionally, you should note that you cannot restart a thread once it has finished execution: you have to recreate the thread object for it to be started again. one workaround to get this working is to call Thread.__init__() at the end of the run() method. however, i would not recommend doing this since this will disallow the use of the join() method to detect the end of execution of the thread.
If you would call thread.start() in the loops, you would actually start every thread only once, because all the entries in your list are distinct thread objects (it does not matter they belong to the same class). You should never call the run() method of a thread directly -- it is meant to be called by the start() method. Calling it directly would not call it in a separate thread.
The code below creates a class that is just a thread but the start and calls the initialization of the Thread class again so that the thread doesn't know it has been called.
from threading import Thread
class MTThread(Thread):
def __init__(self, name = "", target = None):
self.mt_name = name
self.mt_target = target
Thread.__init__(self, name = name, target = target)
def start(self):
super().start()
Thread.__init__(self, name = self.mt_name, target = self.mt_target)
def run(self):
super().run()
Thread.__init__(self, name = self.mt_name, target = self.mt_target)
def code():
#Some code
thread = MTThread(name = "SomeThread", target = code)
thread.start()
thread.start()
I had this same dilemma and came up with this solution which has worked perfectly for me. It also allows a thread-killing decorator to be used efficiently.
The key feature is the use of a thread refresher which is instantiated and .started in main. This thread-refreshing thread will run a function that instantiates and starts all other (real, task-performing) threads. Decorating the thread-refreshing function with a thread-killer allows you to kill all threads when a certain condition is met, such as main terminating.
#ThreadKiller(arg) #qu'est-ce que c'est
def RefreshThreads():
threadTask1 = threading.Thread(name = "Task1", target = Task1, args = (anyArguments))
threadTask2 = threading.Thread(name = "Task2", target = Task2, args = (anyArguments))
threadTask1.start()
threadTask2.start()
#Main
while True:
#do stuff
threadRefreshThreads = threading.Thread(name = "RefreshThreads", target = RefreshThreads, args = ())
threadRefreshThreads.start()
from threading import Thread
from time import sleep
def runA():
while a==1:
print('A\n')
sleep(0.5)
if __name__ == "__main__":
a=1
t1 = Thread(target = runA)
t1.setDaemon(True)
t1.start()
sleep(2)
a=0
print(" now def runA stops")
sleep(3)
print("and now def runA continue")
a=1
t1 = Thread(target = runA)
t1.start()
sleep(2)
I have threaded code where each thread needs to write to the same file. To prevent concurrency issues, I am using a Lock object.
My question is whether I am using the Lock correctly. If I set the lock from within each thread, is that lock global or only specific to that specific thread?
Basically, should I create a Lock first and pass its reference to each thread, or is it ok to set it from within the thread like I do here:
import time
from threading import Thread, Lock
def main():
for i in range(20):
agent = Agent(i)
agent.start()
class Agent(Thread):
def __init__(self, thread_num):
Thread.__init__(self)
self.thread_num = thread_num
def run(self):
while True:
print 'hello from thread %s' % self.thread_num
self.write_result()
def write_result(self):
lock = Lock()
lock.acquire()
try:
f = open('foo.txt', 'a')
f.write('hello from thread %s\n' % self.thread_num)
f.flush()
f.close()
finally:
lock.release()
if __name__ == '__main__':
main()
For your use case one approach could be to write a file subclass that locks:
class LockedWrite(file):
""" Wrapper class to a file object that locks writes """
def __init__(self, *args, **kwds):
super(LockedWrite, self).__init__(*args, **kwds)
self._lock = Lock()
def write(self, *args, **kwds):
self._lock.acquire()
try:
super(LockedWrite, self).write(*args, **kwds)
finally:
self._lock.release()
To use in your code just replace following functions:
def main():
f = LockedWrite('foo.txt', 'a')
for i in range(20):
agent = Agent(i, f)
agent.start()
class Agent(Thread):
def __init__(self, thread_num, fileobj):
Thread.__init__(self)
self.thread_num = thread_num
self._file = fileobj
# ...
def write_result(self):
self._file.write('hello from thread %s\n' % self.thread_num)
This approach puts file locking in the file itself which seems cleaner IMHO
Create the lock outside the method.
class Agent(Thread):
mylock = Lock()
def write_result(self):
self.mylock.acquire()
try:
...
finally:
self.mylock.release()
or if using python >= 2.5:
class Agent(Thread):
mylock = Lock()
def write_result(self):
with self.mylock:
...
To use that with python 2.5 you must import the statement from the future:
from __future__ import with_statement
The lock() method returns a lock object for every call. So every thread ( actually every call to write_result ) will have a different lock object. And there will be no locking.
The lock that's used needs to be common to all threads, or at least ensure that two locks can't lock the same resource at the same time.
You can simplify things a bit (at the cost of slightly more overhead) by designating a single thread (probably created exclusively for this purpose) as the sole thread that writes to the file, and have all other threads delegate to the file-writer by placing the string that they want to add to the file into a queue.Queue object.
Queues have all of the locking built-in, so any thread can safely call Queue.put() at any time. The file-writer would be the only thread calling Queue.get(), and can presumably spend much of its time blocking on that call (with a reasonable timeout to allow the thread to cleanly respond to a shutdown request). All of the synchronization issues will be handled by the Queue, and you'll be spared having to worry about whether you've forgotten some lock acquire/release somewhere... :)
The lock instance should be associated with the file instance.
In other words, you should create both the lock and file at the same time and pass both to each thread.
I'm pretty sure that the lock needs to be the same object for each thread. Try this:
import time
from threading import Thread, Lock
def main():
lock = Lock()
for i in range(20):
agent = Agent(i, lock)
agent.start()
class Agent(Thread, Lock):
def __init__(self, thread_num, lock):
Thread.__init__(self)
self.thread_num = thread_num
self.lock = lock
def run(self):
while True:
print 'hello from thread %s' % self.thread_num
self.write_result()
def write_result(self):
self.lock.acquire()
try:
f = open('foo.txt', 'a')
f.write('hello from thread %s\n' % self.thread_num)
f.flush()
f.close()
finally:
lock.release()
if __name__ == '__main__':
main()