I am trying to create a simple callback that can be registered to an object from another thread. The initial object that calls the callback is running on its own thread in this case.
This is best illustrated through the following example:
from pprint import pprint
import sys
import weakref
import threading
import time
class DummyController(object):
def __init__(self):
self.name = "fortytwo"
def callback(self):
print("I am number : " + self.name)
class SomeThread(threading.Thread):
def __init__(self, listener):
threading.Thread.__init__(self)
self.listener = listener
def run(self):
time.sleep(1)
dummy = DummyController()
self.listener.register_callback(dummy.callback)
time.sleep(5)
del dummy
class Listener(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.runner = weakref.WeakMethod(self.default_callback)
self.counter = 20
def default_callback(self):
print("Not implemented")
def register_callback(self, function):
self.runner = weakref.WeakMethod(function)
def run(self):
while self.counter:
try:
self.runner()()
except Exception as e:
pprint(e)
self.counter -= 1
time.sleep(1)
listen = Listener()
some = SomeThread(listen)
listen.start()
some.start()
Now the above code works just fine. But I am concerned about thread-safety here. Reading through weakref docs, it isn't very clear if weakref is really thread safe or not, except for the line:
Changed in version 3.2: Added support for thread.lock, threading.Lock, and code objects.
I might be simply not reading that right. Do I need to add locking, or is everything actually fine and pretty thread safe?
Many thanks
OK, I understand. This is not a problem about thread safe, but just a problem about weak reference.
There is an executable example:
from pprint import pprint
import sys
import weakref
import threading
import time
import gc
class SomeThread(threading.Thread):
def __init__(self, listener):
threading.Thread.__init__(self)
self.listener = listener
def run(self):
class test: # simplify this example.
def callback(self, count):
print(count)
time.sleep(1)
dummy = test()
self.listener.register_callback(dummy.callback)
time.sleep(5)
del dummy
gc.collect() # add this line to do garbage collecting.
class Listener(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.runner = weakref.WeakMethod(self.default_callback)
self.counter = 20
def default_callback(self):
print("Not implemented")
def register_callback(self, function):
self.runner = weakref.WeakMethod(function)
def run(self):
while self.counter:
try:
self.runner()(self.counter)
except Exception as e:
pprint(e)
self.counter -= 1
time.sleep(1)
listen = Listener()
some = SomeThread(listen)
listen.start()
some.start()
output:
TypeError('default_callback() takes 1 positional argument but 2 were given',)
TypeError('default_callback() takes 1 positional argument but 2 were given',)
18
17
16
15
TypeError("'NoneType' object is not callable",)
TypeError("'NoneType' object is not callable",)
TypeError("'NoneType' object is not callable",)
If you explicitly call gc.collect(), callback loses its last strong reference and then it becomes None. As you will never know when will gc collect garbage, there is a potential issue.
It is no matter you use thread or not, just a normal behave of weak reference.
BTW, be careful that exiting SomeThread.run will also implicitly del dummy, you can test it by removing del dummy and moving gc.collect() into try block.
Related
I'm trying to create a class with methods that must not block the execution (I guess it is called "non blocking"), but only one of those could be executing at a given time.
Suppose this:
import time
import random
class Foo():
def do_stuff_a(self):
print("Start a")
time.sleep(0.5)
print("Finished a")
def do_stuff_b(self):
print("Start b")
time.sleep(0.5)
print("Finished b")
def do_other_stuff():
print("doing other stsuff")
foo = Foo()
for x in range(15):
time.sleep(0.1)
# These calls should be non-blocking
if random.randint(0,1) == 1:
foo.do_stuff_a()
else:
foo.do_stuff_b()
do_other_stuff()
do_stuff_a and do_stuff_b should not block the flow, so do_other_stuff() should be executing every 0.1 seconds. Additionally, do_stuff_a and b should be skipped if there is any of them still running (i.e. methods of class Foo cannot run in parallel, if there is one running, the call to any of them should be just skipped (rather than put in a queue for later).
Any pointer friends on how to get started?
This is what I have come up with; I need two decorators, one to manage the threading of the methods (works fine) the other one to make sure that only one method is executed, and skip the calls if it is running (rather than leaving it in a queue)
import time
import random
import threading
def thread(func):
def wrapper( self, *args, **kwargs):
th = threading.Thread(target=func,args=(self, *args))
th.start()
return wrapper
class Foo():
def __init__(self):
self.lock = threading.Lock()
def non_parallel(fun):
def wrapper( self, *args, **kwargs):
if self.lock.locked(): return
self.lock.acquire()
try:
fun( self , *args)
except ValueError as e:
print (e)
self.lock.release()
return wrapper
#thread
#non_parallel
def do_stuff_a(self):
print("Start a\n")
print("Finished a\n")
#thread
#non_parallel
def do_stuff_b(self):
print("Start b\n")
time.sleep(1)
print("Finished b\n")
def do_other_stuff():
print("doing other stsuff\n")
foo = Foo()
for x in range(5):
time.sleep(0.1)
foo.do_stuff_a()
foo.do_stuff_b()
do_other_stuff()
I have a class (MyClass) which contains a queue (self.msg_queue) of actions that need to be run and I have multiple sources of input that can add tasks to the queue.
Right now I have three functions that I want to run concurrently:
MyClass.get_input_from_user()
Creates a window in tkinter that has the user fill out information and when the user presses submit it pushes that message onto the queue.
MyClass.get_input_from_server()
Checks the server for a message, reads the message, and then puts it onto the queue. This method uses functions from MyClass's parent class.
MyClass.execute_next_item_on_the_queue()
Pops a message off of the queue and then acts upon it. It is dependent on what the message is, but each message corresponds to some method in MyClass or its parent which gets run according to a big decision tree.
Process description:
After the class has joined the network, I have it spawn three threads (one for each of the above functions). Each threaded function adds items from the queue with the syntax "self.msg_queue.put(message)" and removes items from the queue with "self.msg_queue.get_nowait()".
Problem description:
The issue I am having is that it seems that each thread is modifying its own queue object (they are not sharing the queue, msg_queue, of the class of which they, the functions, are all members).
I am not familiar enough with Multiprocessing to know what the important error messages are; however, it is stating that it cannot pickle a weakref object (it gives no indication of which object is the weakref object), and that within the queue.put() call the line "self._sem.acquire(block, timeout) yields a '[WinError 5] Access is denied'" error. Would it be safe to assume that this failure in the queue's reference not copying over properly?
[I am using Python 3.7.2 and the Multiprocessing package's Process and Queue]
[I have seen multiple Q/As about having threads shuttle information between classes--create a master harness that generates a queue and then pass that queue as an argument to each thread. If the functions didn't have to use other functions from MyClass I could see adapting this strategy by having those functions take in a queue and use a local variable rather than class variables.]
[I am fairly confident that this error is not the result of passing my queue to the tkinter object as my unit tests on how my GUI modifies its caller's queue work fine]
Below is a minimal reproducible example for the queue's error:
from multiprocessing import Queue
from multiprocessing import Process
import queue
import time
class MyTest:
def __init__(self):
self.my_q = Queue()
self.counter = 0
def input_function_A(self):
while True:
self.my_q.put(self.counter)
self.counter = self.counter + 1
time.sleep(0.2)
def input_function_B(self):
while True:
self.counter = 0
self.my_q.put(self.counter)
time.sleep(1)
def output_function(self):
while True:
try:
var = self.my_q.get_nowait()
except queue.Empty:
var = -1
except:
break
print(var)
time.sleep(1)
def run(self):
process_A = Process(target=self.input_function_A)
process_B = Process(target=self.input_function_B)
process_C = Process(target=self.output_function)
process_A.start()
process_B.start()
process_C.start()
# without this it generates the WinError:
# with this it still behaves as if the two input functions do not modify the queue
process_C.join()
if __name__ == '__main__':
test = MyTest()
test.run()
Indeed - these are not "threads" - these are "processes" - while if you were using multithreading, and not multiprocessing, the self.my_q instance would be the same object, placed at the same memory space on the computer,
multiprocessing does a fork of the process, and any data in the original process (the one in execution in the "run" call) will be duplicated when it is used - so, each subprocess will see its own "Queue" instance, unrelated to the others.
The correct way to have various process share a multiprocessing.Queue object is to pass it as a parameter to the target methods. The simpler way to reorganize your code so that it works is thus:
from multiprocessing import Queue
from multiprocessing import Process
import queue
import time
class MyTest:
def __init__(self):
self.my_q = Queue()
self.counter = 0
def input_function_A(self, queue):
while True:
queue.put(self.counter)
self.counter = self.counter + 1
time.sleep(0.2)
def input_function_B(self, queue):
while True:
self.counter = 0
queue.put(self.counter)
time.sleep(1)
def output_function(self, queue):
while True:
try:
var = queue.get_nowait()
except queue.Empty:
var = -1
except:
break
print(var)
time.sleep(1)
def run(self):
process_A = Process(target=self.input_function_A, args=(queue,))
process_B = Process(target=self.input_function_B, args=(queue,))
process_C = Process(target=self.output_function, args=(queue,))
process_A.start()
process_B.start()
process_C.start()
# without this it generates the WinError:
# with this it still behaves as if the two input functions do not modify the queue
process_C.join()
if __name__ == '__main__':
test = MyTest()
test.run()
As you can see, since your class is not actually sharing any data through the instance's attributes, this "class" design does not make much sense for your application - but for grouping the different workers in the same code block.
It would be possible to have a magic-multiprocess-class that would have some internal method to actually start the worker-methods and share the Queue instance - so if you have a lot of those in a project, there would be a lot less boilerplate.
Something along:
from multiprocessing import Queue
from multiprocessing import Process
import time
class MPWorkerBase:
def __init__(self, *args, **kw):
self.queue = None
self.is_parent_process = False
self.is_child_process = False
self.processes = []
# ensure this can be used as a colaborative mixin
super().__init__(*args, **kw)
def run(self):
if self.is_parent_process or self.is_child_process:
# workers already initialized
return
self.queue = Queue()
processes = []
cls = self.__class__
for name in dir(cls):
method = getattr(cls, name)
if callable(method) and getattr(method, "_MP_worker", False):
process = Process(target=self._start_worker, args=(self.queue, name))
self.processes.append(process)
process.start()
# Setting these attributes here ensure the child processes have the initial values for them.
self.is_parent_process = True
self.processes = processes
def _start_worker(self, queue, method_name):
# this method is called in a new spawned process - attribute
# changes here no longer reflect attributes on the
# object in the initial process
# overwrite queue in this process with the queue object sent over the wire:
self.queue = queue
self.is_child_process = True
# call the worker method
getattr(self, method_name)()
def __del__(self):
for process in self.processes:
process.join()
def worker(func):
"""decorator to mark a method as a worker that should
run in its own subprocess
"""
func._MP_worker = True
return func
class MyTest(MPWorkerBase):
def __init__(self):
super().__init__()
self.counter = 0
#worker
def input_function_A(self):
while True:
self.queue.put(self.counter)
self.counter = self.counter + 1
time.sleep(0.2)
#worker
def input_function_B(self):
while True:
self.counter = 0
self.queue.put(self.counter)
time.sleep(1)
#worker
def output_function(self):
while True:
try:
var = self.queue.get_nowait()
except queue.Empty:
var = -1
except:
break
print(var)
time.sleep(1)
if __name__ == '__main__':
test = MyTest()
test.run()
I'm a newbie to Python and learning about threads. I have created a sample Producer-Consumer code wherein I add a movie to a list in Producer thread and pop the front element from the same list in Consumer thread. The problem is while printing the items of the movie List along with thread name I'm getting incorrect thread name in Producer thread. This is my code
Producer.py
from threading import Thread
from threading import RLock
import time
class Producer(Thread):
def __init__(self):
Thread.__init__(self)
Thread.name = 'Producer'
self.movieList = list()
self.movieListLock = RLock()
def printMovieList(self):
self.movieListLock.acquire()
if len(self.movieList) > 0:
for movie in self.movieList:
print(Thread.name, movie)
print('\n')
self.movieListLock.release()
def pushMovieToList(self, movie):
self.movieListLock.acquire()
self.movieList.append(movie)
self.printMovieList()
self.movieListLock.release()
def run(self):
for i in range(6):
self.pushMovieToList('Avengers' + str(i + 1))
time.sleep(1)
Consumer.py
from threading import Thread
import time
class Consumer(Thread):
def __init__(self):
Thread.__init__(self)
Thread.name = 'Consumer'
self.objProducer = None
def popMovieFromList(self):
self.objProducer.movieListLock.acquire()
if len(self.objProducer.movieList) > 0:
movie = self.objProducer.movieList.pop(0)
print(Thread.name, ':', movie)
print('\n')
self.objProducer.movieListLock.release()
def run(self):
while True:
time.sleep(1)
self.popMovieFromList()
Main.py
from Producer import *
from Consumer import *
def main():
objProducer = Producer()
objConsumer = Consumer()
objConsumer.objProducer = objProducer
objProducer.start()
objConsumer.start()
objProducer.join()
objConsumer.join()
main()
I am not sure whether you solve this problem.
Hope my answer will be helpful.
You can check the document of threading.
Here it says that Thread.name may set same name for multiple thread.
name
A string used for identification purposes only. It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor.
I think Thread.name is a static variable so it shares in different thread.
If you want to set name of thread, you can set it in thread object like this:
class Producer(Thread):
def __init__(self):
Thread.__init__(self)
self.name= 'Producer'
and get it by threading.current_thread().name.
if len(self.movieList) > 0:
for movie in self.movieList:
print(threading.current_thread().name, movie)
Hope you enjoy it!
I am trying to write a class to handle signals using the signal python module. The reason for having a class is to avoid the use of globals. This is the code I came up with, but unfortunately it is not working:
import signal
import constants
class SignalHandler (object):
def __init__(self):
self.counter = 0
self.break = False
self.vmeHandlerInstalled = False
def setVmeHandler(self):
self.vmeBufferFile = open('/dev/vme_shared_memory0', 'rb')
self.vmeHandlerInstalled = True
signal.signal(signal.SIGUSR1, self.traceHandler)
signal.siginterrupt(signal.SIGUSR1, False)
#...some other stuff...
def setBreakHandler(self):
signal.signal(signal.SIGINT, self.newBreakHandler)
signal.siginterrupt(signal.SIGINT, False)
def newBreakHandler(self, signum, frame):
self.removeVMEHandler()
self.break = True
def traceHandler(self, signum, frame):
self.counter += constants.Count
def removeVMEHandler(self):
if not self.vmeHandlerInstalled: return
if self.vmeBufferFile is None: return
signal.signal(signal.SIGUSR1, signal.SIG_DFL)
self.vmeHandlerInstalled = False
On the main program I use this class in the following way:
def run():
sigHandler = SignalHandler()
sigHandler.setBreakHandler()
sigHandler.setVmeHandler()
while not sigHandler.break:
#....do some stuff
if sigHandler.counter >= constants.Count:
#...do some stuff
This solution is not working, as it appears that the handler for the signal.SIGUSR1 installed in the setVmeHandler method never gets called.
So my question is: is it possible to handle signal inside a class or shall I use globals?
To answer your question, I created the following simple code:
import signal
import time
class ABC(object):
def setup(self):
signal.signal(signal.SIGUSR1, self.catch)
signal.siginterrupt(signal.SIGUSR1, False)
def catch(self, signum, frame):
print("xxxx", self, signum, frame)
abc = ABC()
abc.setup()
time.sleep(20)
If I run it:
python ./test.py
Then in another window send a USR1 signal:
kill -USR1 4357
The process prints the expected message:
('xxxx', <__main__.ABC object at 0x7fada09c6190>, 10, <frame object at 0x7fada0aaf050>)
So I think the answer is Yes, it possible to handle signal inside a class.
As for why you code doesn't work, sorry, I have no idea.
I got a similar problem as toti08, referring to setVmeHandler(self), and found out the handler must have matching parameters i.e. (self, signum,frame).
i want to add an item into a global list every 2 seconds in one thread,
and save the list into database before empty it every 3 seconds in another thread.
i create two local varibles to monitor the total added items and total saveditems, they should be equal every 6 senconds,but it is not.
here is my code:
import datetime
import psutil,os,time
from threading import *
class AddToList(Thread):
totalAdded=0
def run(self):
lock=RLock()
lock.acquire()
while True:
entryList.append("AddToList at "+str(datetime.datetime.now()))
self.totalAdded=self.totalAdded+len(entryList)
print("totalAdded:"+str(self.totalAdded))
time.sleep(2)
lock.release()
class SaveList(Thread):
totalSaved=0
'''save entry to server'''
def __init__(self):
Thread.__init__(self)
def run(self):
lock=RLock()
lock.acquire()
while True:
#save list to database,then empty the list
self.totalSaved=self.totalSaved+len(entryList)
del entryList[:]
print("totalSaved:"+str(self.totalSaved))
time.sleep(3)
lock.release()
if __name__=="__main__":
global entryList
entryList=[]
addClass= AddToList()
addClass.start()
saveClass=SaveList()
saveClass.start()
result:
totalAdded:2
totalSaved:2
totalAdded:3
totalSaved:3totalAdded:4
totalAdded:6
totalSaved:5
totalAdded:7
totalSaved:6
totalAdded:8
totalAdded:10
totalSaved:8
totalAdded:11
totalSaved:9
totalAdded:12
totalAdded:14
totalSaved:11
totalAdded:15
totalSaved:12
...........
...........
totalAdded:51
totalSaved:39totalAdded:52
totalAdded:54
totalSaved:41
totalAdded:55
totalSaved:42
totalAdded:56
totalAdded:58
totalSaved:44
totalAdded:59
totalSaved:45totalAdded:60
......
......
i anm new to python and searched a lot about threading ,Lock and RLock ,but with no luck.
where am wrong?
To make Lock and RLock work you must use the same object in every thread. The lock objects must have the same "visibility" of the object that you want to "protect".
Here is a new version of you code which should work. It also avoid using things like global variables etc.
import datetime
import time
import threading
class AddToList(threading.Thread):
def __init__(self, lock, entryList):
threading.Thread.__init__(self)
self.totalAdded = 0
self.entryList = entryList
self.lock = lock
def run(self):
while True:
self.lock.acquire()
entryList.append("AddToList at {}".format(datetime.datetime.now()))
self.totalAdded += 1
self.lock.release()
print("totalAdded: {}".format(self.totalAdded))
time.sleep(2)
class SaveList(threading.Thread):
def __init__(self, lock, entryList):
threading.Thread.__init__(self)
self.totalSaved = 0
self.entryList = entryList
self.lock = lock
def run(self):
while True:
self.lock.acquire()
self.totalSaved += len(self.entryList)
del self.entryList[:]
self.lock.release()
print("totalSaved: {}".format(self.totalSaved))
time.sleep(3)
if __name__=="__main__":
lock=threading.Lock()
entryList=[]
addClass = AddToList(lock, entryList)
addClass.start()
saveClass = SaveList(lock, entryList)
saveClass.start()
Some things to note:
Use Lock instead of RLock when you don't have any particular needs. RLock is much slower.
As already pointed out by someone it is better avoid using global variables when not needed. Also Class variables should be used only when it makes sense.
When you use a lock you should try to limit as much as possible the code between acquire and release. In you previous code you never release the lock.