I have the following code and store in a file called run_thread.py
from mymodule import Builder
def run_thread():
threads = []
for chunk in chunks:
thread = threading.Thread(target=Builder.work, args=(a, b, c))
threads.append(thread)
try:
thread.start()
except:
pass
for thread in threads:
try:
thread.join()
except:
pass
When I try to mock/patch Builder.work in my unittest, the original Builder.work still run, which is NOT what I'm expecting.
import unittest
from ddt import ddt, data, unpack
from mock import patch
#ddt
class TestRunThread(unittest.TestCase):
#patch('run_thread.Builder.work')
def test_run_thread(self, mock_work):
run_thread()
It did not work for me because Builder.work() is a static method.
When patch a class function, property, class attribute, this is used according to document
#patch.object(Builder, 'work')
def test_run_thread(self, mock_work):
# this is tested to work with static method
mock_work.return_value = ['a', 'bbbb']
class property and class attribute(not instance member) can also use
#patch('mymodule.Builder.class_attribute_or_property')
def test_mock_cls_attribute_or_property(mock_attribute):
# this patch method tested not working for static method
Related
In a custom class I have the following code:
class CustomClass():
triggerQueue: multiprocessing.Queue
def __init__(self):
self.triggerQueue = multiprocessing.Queue()
def poolFunc(queueString):
print(queueString)
def listenerFunc(self):
pool = multiprocessing.Pool(5)
while True:
try:
queueString = self.triggerQueue.get_nowait()
pool.apply_async(func=self.poolFunc, args=(queueString,))
except queue.Empty:
break
What I intend to do is:
add a trigger to the queue (not implemented in this snippet) -> works as intended
run an endless loop within the listenerFunc that reads all triggers from the queue (if any are found) -> works as intended
pass trigger to poolFunc which is to be executed asynchronosly -> not working
It works as soon as I source my poolFun() outside of the class like
def poolFunc(queueString):
print(queueString)
class CustomClass():
[...]
But why is that so? Do I have to pass the self argument somehow? Is it impossible to perform it this way in general?
Thank you for any hint!
There are several problems going on here.
Your instance method, poolFunc, is missing a self parameter.
You are never properly terminating the Pool. You should take advantage of the fact that a multiprocessing.Pool object is a context manager.
You're calling apply_async, but you're never waiting for the results. Read the documentation: you need to call the get method on the AsyncResult object to receive the result; if you don't do this before your program exits your poolFunc function may never run.
By making the Queue object part of your class, you won't be able to pass instance methods to workers.
We can fix all of the above like this:
import multiprocessing
import queue
triggerQueue = multiprocessing.Queue()
class CustomClass:
def poolFunc(self, queueString):
print(queueString)
def listenerFunc(self):
results = []
with multiprocessing.Pool(5) as pool:
while True:
try:
queueString = triggerQueue.get_nowait()
results.append(pool.apply_async(self.poolFunc, (queueString,)))
except queue.Empty:
break
for res in results:
print(res.get())
c = CustomClass()
for i in range(10):
triggerQueue.put(f"testval{i}")
c.listenerFunc()
You can, as you mention, also replace your instance method with a static method, in which case we can keep triggerQueue as part of the class:
import multiprocessing
import queue
class CustomClass:
def __init__(self):
self.triggerQueue = multiprocessing.Queue()
#staticmethod
def poolFunc(queueString):
print(queueString)
def listenerFunc(self):
results = []
with multiprocessing.Pool(5) as pool:
while True:
try:
queueString = self.triggerQueue.get_nowait()
results.append(pool.apply_async(self.poolFunc, (queueString,)))
except queue.Empty:
break
for r in results:
print(r.get())
c = CustomClass()
for i in range(10):
c.triggerQueue.put(f"testval{i}")
c.listenerFunc()
But we still need to reap the pool_async results.
Okay, I found an answer and a workaround:
the answer is based the anser of noxdafox to this question.
Instance methods cannot be serialized that easily. What the Pickle protocol does when serialising a function is simply turning it into a string.
For a child process would be quite hard to find the right object your instance method is referring to due to separate process address spaces.
A functioning workaround is to declare the poolFunc() as static function like
#staticmethod
def poolFunc(queueString):
print(queueString)
I need to pass a Manager instance to other processes as I need instances of proxy objects created in parallel and later on be re-used again in separate processes. However, it appears that I can' pass a Manager as an argument to a function that is ought to be ran by the other process. See an example:
from multiprocessing.managers import BaseManager
from multiprocessing import Pool
from functools import partial
class MyManager(BaseManager):
pass
class MyClass():
def __init__(self, i):
self.i = i
def my_fun(i, manager):
return manager.MyClass(i)
MyManager.register('MyClass', MyClass)
manager = MyManager()
manager.start()
f = partial(my_fun, manager=manager)
with Pool(4) as p:
res = [r.i for r in p.map(f, list(range(10)))]
print(res)
The following exception will arise if I run the code above:
TypeError: Pickling an AuthenticationString object is disallowed for security reasons
Interestingly, but passing Manager inside of args argument of the Pool.Process works, but I still need map functionality.
First of all, the proxy that is automatically generated for your class does not support the access of attributes. So if you want to access the i attribute of your managed class, you will need to explicitly define your own proxy class. It will be easier to just define, for example, a method get_i to return that attribute. I would typically define the get_i method in a subclass of the original class created just for the purpose of being used as the managed class implementation. In the code below I have defined such a method (although I have not bothered to create a special subclass) and a custom proxy class to show you how you would do this.
I just see no way of passing the manager instance to another process. The solution I came up with (there may be better ones) is to create a thread that will accept requests via the connections exposed by a multiprocessing.Pipe instance. You will need to enforce single threading of these requests not only because you cannot have multiple processes sending to the same connection concurrently but also because it is the only way to ensure that the response a requestor gets back matches up with its request.
The idea is that the my_fun function sends via its connection the argument i for which it wants to create a MyClass instance. A daemon thread running in the main process, function create_MyClass for which manager is defined, receives this argument, creates the desired class instance and sends the result back. Essentially create_MyClass behaves like a factory "method". The manner in which this "method" is "called", i.e. sending a message via a Pipe-created connection to a thread running in a different process, is actually similar to what happens when you make a method call on a managed class's proxy reference.
from multiprocessing.managers import BaseManager, NamespaceProxy
from multiprocessing import Pool, Pipe, Lock
from threading import Thread
class MyManager(BaseManager):
pass
class MyClass():
def __init__(self, i):
self.i = i
def get_i(self):
return self.i
class MyClassProxy(NamespaceProxy):
_exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'get_i')
def get_i(self):
return self._callmethod('get_i')
def init_pool(the_connection, the_lock):
global connection, lock
connection = the_connection
lock = the_lock
def my_fun(i):
with lock:
connection.send(i) # send argument
my_class = connection.recv() # get result
return my_class
def create_MyClass(connection):
while True:
i = connection.recv()
my_class = manager.MyClass(i)
connection.send(my_class)
if __name__ == '__main__':
MyManager.register('MyClass', MyClass, MyClassProxy)
manager = MyManager()
manager.start()
lock = Lock()
connection1, connection2 = Pipe(duplex=True)
# Give one of the bi-directional connections to the daemon thread:
Thread(target=create_MyClass, args=(connection1,), daemon=True).start()
# Initialize each process in the pool with the other bi-directional connection
# and a lock to ensure single-threading of the requests:
with Pool(4, initializer=init_pool, initargs=(connection2, lock)) as p:
res = [r.i for r in p.map(my_fun, list(range(10)))]
print(res)
Prints:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
I'm new to threading and python. I would like to understand how to pass multiple arguments from one class to another class in python using threading.
I'm using a main thread to call a class- Process then inside the run I'm doing some business logic and calling another class- build using thread and passing multiple arguments.
The run of build class is getting executed but Inside the build class, I'm unable to access those arguments and hence not able to proceed further.
Not sure if my approach is right? Any suggestions will be appreciated.
Below is my main class :
from threading import Thread
import logging as log
from process import Process
if __name__ == '__main__':
try:
proc = Process()
proc.start()
except Exception as e:
#log some error
Inside Process:
#all the dependencies are imported
class Process(Thread):
'''
classdocs
'''
def __init__(self):
'''
Constructor
'''
Thread.__init__(self)
#other intializations
def run(self):
#some other logic
self.notification(pass_some_data)
#inside notification I'm calling another thread
def notification(self,passed_data):
#passed data is converted dict1
#tup1 is being formed from another function.
#build is a class, and if i don't pass None, i get groupname error.
th = build(None,(tup1,),(dict1,))
th.start()
#inside build
class build(Thread):
def _init_(self,tup1,dict1):
super(build,self).__init__(self)
self.tup1 = tup1
self.dict1 = dict1
def run(self):
#some business logic
#I'm unable to get the arguments being passed here.
I've tried to stub out classes constructed in the main function so that I can test against main and assert that classes are initialized with specific data. However main function still does not pick up the mocked instances. How can I pass along the mocked instance to main.
from unittest.mock import patch
from contextlib import contextmanager
#contextmanager
def use_mocked(method, cls, ret_value):
class MockedClass(cls):
pass
def func(cls):
return ret_value
def fullname(o):
return o.__module__ + "." + o.__name__
setattr(MockedClass, method, classmethod(func))
with patch(fullname(cls), MockedClass):
yield
This is the patching utility to make sure main is passed the mocked reference. I may be confused on my understanding of how its functioning.
def test_main():
magic_b = MagicMock(spec_set=Benchmark, wraps=Benchmark)
with use_mocked("__new__", DataStream, magic_b):
main.main()
magic_b.assert_called_once_with() # fails
in the main module, I have a main method defined as...
import benchmark.Benchmark
def main():
b = benchmark.Benchmark() # <- this is not the mocked instance
...
I relied on the same patch utility in unittest.mock but instead just used it in the form of a decorator around my test. patch() now is passed in the Benchmark class which main imports (it is important to patch in main and not in benchmark modules it self ie. don't patch benchmark.Benchmark). Main module remains untouched and tests now pass.
import main
#patch("main.Benchmark")
# b here is a ref to MagicMock class mocking Benchmark;
# it is substituted into the execution of main module,
# patch provides it as a param so you can assert against it.
def test_main(b):
main.main()
b.assert_called_once_with()
First of all, here are my two python files:
sred.py:
import _thread,time
class Thread:
def __init__(self,time:int,say:str):
self.time=time
self.say=say
def create():
id = _thread.get_ident()
for i in range(5):
print("HALLO", id)
return
from sred import Thread
import time,_thread
_thread.start_new_thread(Thread.create,())
The second one:
main.py
from sred import Thread
import time,_thread
_thread.start_new_thread(Thread.create,())
when executing this it doesn't print anything out, why?
UPDATE:
import _thread
class Thread:
#classmethod
def create():
id = _thread.get_ident()
for i in range(5):
print("HALLO", id)
return
main.py:
from sred import Thread
import time,_thread
_thread.start_new_thread(Thread().create,())
Is this now right, or is there still something wrong?
The create method is missing self as a parameter -- it looks like it should also be a #classmethod if you want to call it as it's written now. Note that your __init__ method is never getting called, because you never instantiate any Thread objects. You may want it to read:
_thread.start_new_thread(Thread().create, ())
i.e., instantiate a thread, then pass its create method to be executed in the new thread. I'm not sure what's happening, but I suspect that something is erroring and the stacktrace is being suppressed by something.
Also, you need to delete the space after the for statement -- it's significant, and it should be throwing you a syntax error about an unexpected indent.
EDIT:
This version runs on my machine:
import _thread
class Thread:
def create(self):
id = _thread.get_ident()
for i in range(5):
print("HALLO", id)
return
_thread.start_new_thread(Thread().create, ())