I am using the multiprocessing.Pool class within an object and attempting the following:
from multiprocessing import Lock, Pool
class A:
def __init__(self):
self.lock = Lock()
self.file = open('test.txt')
def function(self, i):
self.lock.acquire()
line = self.file.readline()
self.lock.release()
return line
def anotherfunction(self):
pool = Pool()
results = pool.map(self.function, range(10000))
pool.close()
pool.join()
return results
However I am getting a runtime error stating that lock objects should only be shared between processes through inheritance. I am fairly new to Python and multiprocessing. How can I get put on the right track?
multiprocessing.Lock instances can be attributes of multiprocessing.Process instances. When a process is created in the main process with a lock attribute, the lock exists in the main process’s address space. When the process’s start method is invoked and runs a subprocess which invokes the process’s run method, the lock has to be serialized/deserialized to the subprocess address space. This works as expected:
from multiprocessing import Lock, Process
class P(Process):
def __init__(self, *args, **kwargs):
Process.__init__(self, *args, **kwargs)
self.lock = Lock()
def run(self):
print(self.lock)
if __name__ == '__main__':
p = P()
p.start()
p.join()
Prints:
<Lock(owner=None)>
Unfortuantely, this does not work when you are dealing with multiprocessing.Pool instances. In your example, self.lock is created in the main process by the __init__ method. But when Pool.map is called to invoke self.function, the lock cannot be serialized/deserialized to the already-running pool process that will be running this method.
The solution is to initialize each pool process with a global variable set to this lock (there is no point in having this lock being an attribute of the class now). The way to do this is to use the initializer and initargs parameters of the pool __init__ method. See the documentation:
from multiprocessing import Lock, Pool
def init_pool_processes(the_lock):
'''Initialize each process with a global variable lock.
'''
global lock
lock = the_lock
class Test:
def function(self, i):
lock.acquire()
with open('test.txt', 'a') as f:
print(i, file=f)
lock.release()
def anotherfunction(self):
lock = Lock()
pool = Pool(initializer=init_pool_processes, initargs=(lock,))
pool.map(self.function, range(10))
pool.close()
pool.join()
if __name__ == '__main__':
t = Test()
t.anotherfunction()
What's the proper way to tell a looping thread to stop looping?
I have a fairly simple program that pings a specified host in a separate threading.Thread class. In this class it sleeps 60 seconds, the runs again until the application quits.
I'd like to implement a 'Stop' button in my wx.Frame to ask the looping thread to stop. It doesn't need to end the thread right away, it can just stop looping once it wakes up.
Here is my threading class (note: I haven't implemented looping yet, but it would likely fall under the run method in PingAssets)
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
def run(self):
config = controller.getConfig()
fmt = config['timefmt']
start_time = datetime.now().strftime(fmt)
try:
if onlinecheck.check_status(self.asset):
status = "online"
else:
status = "offline"
except socket.gaierror:
status = "an invalid asset tag."
msg =("{}: {} is {}. \n".format(start_time, self.asset, status))
wx.CallAfter(self.window.Logger, msg)
And in my wxPyhton Frame I have this function called from a Start button:
def CheckAsset(self, asset):
self.count += 1
thread = PingAssets(self.count, asset, self)
self.threads.append(thread)
thread.start()
Threaded stoppable function
Instead of subclassing threading.Thread, one can modify the function to allow
stopping by a flag.
We need an object, accessible to running function, to which we set the flag to stop running.
We can use threading.currentThread() object.
import threading
import time
def doit(arg):
t = threading.currentThread()
while getattr(t, "do_run", True):
print ("working on %s" % arg)
time.sleep(1)
print("Stopping as you wish.")
def main():
t = threading.Thread(target=doit, args=("task",))
t.start()
time.sleep(5)
t.do_run = False
if __name__ == "__main__":
main()
The trick is, that the running thread can have attached additional properties. The solution builds
on assumptions:
the thread has a property "do_run" with default value True
driving parent process can assign to started thread the property "do_run" to False.
Running the code, we get following output:
$ python stopthread.py
working on task
working on task
working on task
working on task
working on task
Stopping as you wish.
Pill to kill - using Event
Other alternative is to use threading.Event as function argument. It is by
default False, but external process can "set it" (to True) and function can
learn about it using wait(timeout) function.
We can wait with zero timeout, but we can also use it as the sleeping timer (used below).
def doit(stop_event, arg):
while not stop_event.wait(1):
print ("working on %s" % arg)
print("Stopping as you wish.")
def main():
pill2kill = threading.Event()
t = threading.Thread(target=doit, args=(pill2kill, "task"))
t.start()
time.sleep(5)
pill2kill.set()
t.join()
Edit: I tried this in Python 3.6. stop_event.wait() blocks the event (and so the while loop) until release. It does not return a boolean value. Using stop_event.is_set() works instead.
Stopping multiple threads with one pill
Advantage of pill to kill is better seen, if we have to stop multiple threads
at once, as one pill will work for all.
The doit will not change at all, only the main handles the threads a bit differently.
def main():
pill2kill = threading.Event()
tasks = ["task ONE", "task TWO", "task THREE"]
def thread_gen(pill2kill, tasks):
for task in tasks:
t = threading.Thread(target=doit, args=(pill2kill, task))
yield t
threads = list(thread_gen(pill2kill, tasks))
for thread in threads:
thread.start()
time.sleep(5)
pill2kill.set()
for thread in threads:
thread.join()
This has been asked before on Stack. See the following links:
Is there any way to kill a Thread in Python?
Stopping a thread after a certain amount of time
Basically you just need to set up the thread with a stop function that sets a sentinel value that the thread will check. In your case, you'll have the something in your loop check the sentinel value to see if it's changed and if it has, the loop can break and the thread can die.
I read the other questions on Stack but I was still a little confused on communicating across classes. Here is how I approached it:
I use a list to hold all my threads in the __init__ method of my wxFrame class: self.threads = []
As recommended in How to stop a looping thread in Python? I use a signal in my thread class which is set to True when initializing the threading class.
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
self.signal = True
def run(self):
while self.signal:
do_stuff()
sleep()
and I can stop these threads by iterating over my threads:
def OnStop(self, e):
for t in self.threads:
t.signal = False
I had a different approach. I've sub-classed a Thread class and in the constructor I've created an Event object. Then I've written custom join() method, which first sets this event and then calls a parent's version of itself.
Here is my class, I'm using for serial port communication in wxPython app:
import wx, threading, serial, Events, Queue
class PumpThread(threading.Thread):
def __init__ (self, port, queue, parent):
super(PumpThread, self).__init__()
self.port = port
self.queue = queue
self.parent = parent
self.serial = serial.Serial()
self.serial.port = self.port
self.serial.timeout = 0.5
self.serial.baudrate = 9600
self.serial.parity = 'N'
self.stopRequest = threading.Event()
def run (self):
try:
self.serial.open()
except Exception, ex:
print ("[ERROR]\tUnable to open port {}".format(self.port))
print ("[ERROR]\t{}\n\n{}".format(ex.message, ex.traceback))
self.stopRequest.set()
else:
print ("[INFO]\tListening port {}".format(self.port))
self.serial.write("FLOW?\r")
while not self.stopRequest.isSet():
msg = ''
if not self.queue.empty():
try:
command = self.queue.get()
self.serial.write(command)
except Queue.Empty:
continue
while self.serial.inWaiting():
char = self.serial.read(1)
if '\r' in char and len(msg) > 1:
char = ''
#~ print('[DATA]\t{}'.format(msg))
event = Events.PumpDataEvent(Events.SERIALRX, wx.ID_ANY, msg)
wx.PostEvent(self.parent, event)
msg = ''
break
msg += char
self.serial.close()
def join (self, timeout=None):
self.stopRequest.set()
super(PumpThread, self).join(timeout)
def SetPort (self, serial):
self.serial = serial
def Write (self, msg):
if self.serial.is_open:
self.queue.put(msg)
else:
print("[ERROR]\tPort {} is not open!".format(self.port))
def Stop(self):
if self.isAlive():
self.join()
The Queue is used for sending messages to the port and main loop takes responses back. I've used no serial.readline() method, because of different end-line char, and I have found the usage of io classes to be too much fuss.
Depends on what you run in that thread.
If that's your code, then you can implement a stop condition (see other answers).
However, if what you want is to run someone else's code, then you should fork and start a process. Like this:
import multiprocessing
proc = multiprocessing.Process(target=your_proc_function, args=())
proc.start()
now, whenever you want to stop that process, send it a SIGTERM like this:
proc.terminate()
proc.join()
And it's not slow: fractions of a second.
Enjoy :)
My solution is:
import threading, time
def a():
t = threading.currentThread()
while getattr(t, "do_run", True):
print('Do something')
time.sleep(1)
def getThreadByName(name):
threads = threading.enumerate() #Threads list
for thread in threads:
if thread.name == name:
return thread
threading.Thread(target=a, name='228').start() #Init thread
t = getThreadByName('228') #Get thread by name
time.sleep(5)
t.do_run = False #Signal to stop thread
t.join()
I find it useful to have a class, derived from threading.Thread, to encapsulate my thread functionality. You simply provide your own main loop in an overridden version of run() in this class. Calling start() arranges for the object’s run() method to be invoked in a separate thread.
Inside the main loop, periodically check whether a threading.Event has been set. Such an event is thread-safe.
Inside this class, you have your own join() method that sets the stop event object before calling the join() method of the base class. It can optionally take a time value to pass to the base class's join() method to ensure your thread is terminated in a short amount of time.
import threading
import time
class MyThread(threading.Thread):
def __init__(self, sleep_time=0.1):
self._stop_event = threading.Event()
self._sleep_time = sleep_time
"""call base class constructor"""
super().__init__()
def run(self):
"""main control loop"""
while not self._stop_event.isSet():
#do work
print("hi")
self._stop_event.wait(self._sleep_time)
def join(self, timeout=None):
"""set stop event and join within a given time period"""
self._stop_event.set()
super().join(timeout)
if __name__ == "__main__":
t = MyThread()
t.start()
time.sleep(5)
t.join(1) #wait 1s max
Having a small sleep inside the main loop before checking the threading.Event is less CPU intensive than looping continuously. You can have a default sleep time (e.g. 0.1s), but you can also pass the value in the constructor.
Sometimes you don't have control over the running target. In those cases you can use signal.pthread_kill to send a stop signal.
from signal import pthread_kill, SIGTSTP
from threading import Thread
from itertools import count
from time import sleep
def target():
for num in count():
print(num)
sleep(1)
thread = Thread(target=target)
thread.start()
sleep(5)
pthread_kill(thread.ident, SIGTSTP)
result
0
1
2
3
4
[14]+ Stopped
I was talking with my friend,after looking at example from Beasley's book
class ActorExit(Exception):
pass
class Actor:
def __init__(self):
self._mailbox = Queue()
def send(self, msg):
self._mailbox.put(msg)
def recv(self):
msg = self._mailbox.get()
if msg is ActorExit:
raise ActorExit()
return msg
def close(self):
self.send(ActorExit)
def start(self):
self._terminated = Event()
t = Thread(target=self._bootstrap)
t.daemon = True
t.start()
def _bootstrap(self):
try:
self.run()
except ActorExit:
pass
finally:
self._terminated.set()
def join(self):
self._terminated.wait()
def run(self):
while True:
msg = self.recv()
class PrintActor(Actor):
def run(self):
while True:
msg = self.recv()
print('Got:', msg)
My friend argues that sole purpose of Event is to block the main thread until the other thread performs set operation.
Is that true?
How can we watch thread execution?
Python threads, how do Event and Queue work together?
They don't. You can use Events without queues and queues without Events, there's no dependency on each other. Your example just happens to use both.
My friend argues that sole purpose of Event is to block the main thread until the other thread performs set operation. Is that true?
Calling .wait() on an Event-object will block any calling thread until the internal flag is .set().
If you look at the source for Event, you'll find that Events just consist of a Condition variable with a lock and a boolean flag + methods to handle and communicate (to waiting threads) state changes of that flag.
class Event:
"""Class implementing event objects.
Events manage a flag that can be set to true with the set() method and reset
to false with the clear() method. The wait() method blocks until the flag is
true. The flag is initially false.
"""
def __init__(self):
self._cond = Condition(Lock())
self._flag = False
...
How can we watch thread execution?
A simple method would be to apply some sort of utility function that prints out what you're interested in, for example:
def print_info(info=""):
"""Print calling function's name and thread with optional info-text."""
calling_func = sys._getframe(1).f_code.co_name
thread_name = threading.current_thread().getName()
print(f"<{thread_name}, {calling_func}> {info}", flush=True)
Another possibility would be to use logging like in this answer.
Not sure what Beazly wanted to demonstrate with the code you showed, but it deems a little over-engineered to me for this simple task. Involving Events here on top is unnecessary when you already use a queue. You can initialize thread termination by passing a sentinel-value.
Here's a simplified version of your example with sentinel ('STOP') and some info-prints with print_info from above:
import sys
import time
import threading
from queue import Queue
class Actor(threading.Thread):
def __init__(self):
super().__init__(target=self.run)
self.queue = Queue()
def send(self, msg):
self.queue.put(msg)
print_info(f"sent: {msg}") # DEBUG
def close(self):
print_info() # DEBUG
self.send('STOP')
def run(self):
for msg in iter(self.queue.get, 'STOP'):
pass
class PrintActor(Actor):
def run(self):
for msg in iter(self.queue.get, 'STOP'):
print_info(f"got: {msg}") # DEBUG
if __name__ == '__main__':
pa = PrintActor()
pa.start()
pa.send("Hello")
time.sleep(2)
pa.send("...World!")
time.sleep(2)
pa.close()
pa.join()
Output:
<MainThread, send> sent: Hello
<Thread-1, run> got: Hello
<MainThread, send> sent: ...World!
<Thread-1, run> got: ...World!
<MainThread, close>
<MainThread, send> sent: STOP
I have a little threading / locking dilema. I am trying to achieve something like this:
import threading
import time
class A:
def __init__(self):
self.lock = threading.Lock()
def print_stuff(self):
with self.lock:
## do some hazard with stuff !
print("OK")
def other_function(self):
with self.lock:
## some more hazard stuff
pass
def worker(a):
while alive:
"""thread worker function"""
a.print_stuff()
a = A()
alive = True
threads = []
for i in range(5):
t = threading.Thread(target=worker, args=(a,))
threads.append(t)
t.start()
try:
while True:
pass
except KeyboardInterrupt:
alive = False
print('interrupted!')
There is a class A - collection of functions mainly, but with some properties also. It will be use for communication with other devices via HTTP or different protocol.
My idea is to lock every function to prevent two functions of A instance being called at the same time. To my surprise, it works without lock also, but so far I am not doing anything hazardous in the functions.
My questions:
Is the implemented lock sufficient enough to prevent any touch of the same resources by multiple threads (the workers will only call functions, they will not touch the properties of A instance)?
Is it better to make the lock global and lock the function directly in worker (instead in the A instance)? See example:
lock = threading.Lock()
def worker(a):
while alive:
"""thread worker function"""
with lock:
a.print_stuff()
I'm writing a Linux driver for a USB HID device in Python. The device has two ways it sends data, both of which are needed: feature reports (synchronous) and input reports (asynchronous). Using the hidapi Cython library I have only one instance of the device to work with, but I need to set up a listener for hid.read() that will run constantly AND allow synchronous methods to be called at will by the application to send feature reports.
Currently I have the listener in one thread and the synchronous calls in another. When I run the program, my synchronous calls are not happening, though they work fine if I never start the listener; so it appears the listener thread is taking over.
Below is the troubling piece of code:
app.py
# app.py
import threading
import time
import myhiddriver
# Code that sets mydevice
class Reader:
def start(self, device):
self.requests = myhiddriver.DeviceRequest(device)
# Wait a bit before firing, for testing
time.sleep(3)
self.requests.request_swipe_card()
def start_listener(self, device):
self.listener = myhiddriver.DeviceListener(device)
reader = Reader()
thread1 = threading.Thread(target=reader.start, args=(mydevice,))
thread1.daemon = True
thread2 = threading.Thread(target=reader.start_listener, args=(mydevice,))
thread2.daemon = True
thread1.start()
thread2.start()
# Keep this puppy running
while True:
pass
myhiddriver.py
import threading
LOCK = threading.Lock()
class DeviceRequest:
def __init__(self, device):
# Lock it up first
LOCK.acquire()
self.device = device
LOCK.release()
def request_swipe_card(self):
# Lock this up too
LOCK.acquire()
self.device.set_feature_report(insert data here)
LOCK.release()
class DeviceListener:
def __init__(self, device):
# Lock me up
LOCK.acquire()
self.device = device
self.start_listener()
LOCK.release()
def start_listener(self):
while True:
# Should I be locking this up?
LOCK.acquire()
data = self.device.read(255)
LOCK.release()
if data:
print data
else:
pass
My Question: Why is my synchronous call (request_swipe_card) not following through? Or, how can I better architect this to have an endless listener AND the ability to make synchronous calls on the same object?
from the looks of the code, it is because you are locking it up
When the state is unlocked, acquire() changes the state to locked and
returns immediately. When the state is locked, acquire() blocks until
a call to release() in another thread changes it to unlocked
here is the problem:
class DeviceListener:
def __init__(self, device):
LOCK.acquire() # first you lock it up
self.device = device
self.start_listener()
LOCK.release()
def start_listener(self):
while True: # because of the loop, the lock wouldn't get release even if the LOCK below doesn't exist
LOCK.acquire() # it end up blocking here and oops, it locked up xD
data = self.device.read(255) # so it wouldn't be able to read here
LOCK.release()
if data:
print data
else:
pass
and when the request_swipe_card end up being call in the other thread, it end up blocking there too
def request_swipe_card(self):
LOCK.acquire() # right here xD
self.device.set_feature_report(insert data here)
LOCK.release()