I am using multiprocessing from Python for testing purpose and there are somethings I don't understand.
A priori, processes have their own memory space so we can't share and Python class between them.
But look at my code:
import sys,
import time
from multiprocessing import Queue, Process
class MainClass():
def __init__(self, q):
self.q = q
print("Queue in Main", q)
def start_p(self):
p = Proc(self.q)
p.processing()
def run_p(self):
p = Process(target=self.start_p, args=())
p.start()
return p
class Proc():
def __init__(self, q):
self.q = q
def processing(self):
print("Queue in process", self.q)
n = ''
try:
n = self.q.get(0) # Get first item in the "queue"
except KeyError as e:
print("NOK", e)
print("GET: ", n)
print('Size: ', self.q.qsize())
if __name__ == "__main__":
# Creating a queue
q = Queue()
# Add 10 numbers in the queue
for i in range(0,10):
q.put(i)
# Add the queue to Main Class
s = MainClass(q)
print("Initial queue size", q.qsize())
# Starting 3 process
p1 = s.run_p()
p2 = s.run_p()
p3 = s.run_p()
#time.sleep(2)
print("Final queue size", q.qsize())
I have created a queue on main process, with 10 numbers.
Then, I ran 3 process so each one run a task consting of just getting (and delete) first item in a queue.
What I misunderstand is how can this program work and return a final queue 7 ?
It seems the queue is shared...but the object itself (multiprocessing) is located in different memory place...
But there is no "pointer" mecanism in python ?
The result when I run the programm below:
Behaviour is almost the same on linux except the memory adress is the same for all instances.
Please can someone explain me ?
Your basic premise is not entirely correct. While multiple processes indeed have their own memory space, that does not mean they cannot exchange data.
In fact there are multiple mechanisms for programs to share data. These are generally called "inter process communication" or IPC. For example;
shared memory
pipes
sockets
Under the covers, a multiprocessing.Queue uses a multiprocessing.Pipe.
This in turn is a Python wrapper for an operating system communication pipe. On ms-windows this is an operating system primitive called a named pipe. On many other operating systems pipes are based on file descriptors.
The point is that even when the Queue is inherited in the child processes, they are all connected to the same operating system object.
That is basically why they can communicate with each other.
(Note that this is of necessity a simplification; If you really want to know how it works, read the multiprocessing module Python code.)
Related
I have a script that creates a class and try's to launch an object of that class in a separate process;
class Task():
def __init__(self, messageQueue):
self.messageQueue = messageQueue
def run(self):
startTime = time.time()
while time.time() -startTime < 60:
try:
message = self.messageQueue.get_nowait()
print message
self.messageQueue.task_done()
except Queue.Empty:
print "No messages"
time.sleep(1)
def test(messageQueue):
task = Task(messageQueue)
task.run()
if __name__ == '__main__':
messageQueue = Queue.Queue()
p = Process(target=test, args=(messageQueue,))
p.start()
time.sleep(5)
messageQueue.put("hello")
Instead of seeing the message "hello" printed out after 5 seconds, I just get a continuous stream of "No messages". What am I doing wrong?
The problem is that you're using Queue.Queue, which only handles multiple threads within the same process, not multiple processes.
The multiprocessing module comes with its own replacement, multiprocessing.Queue, which provides the same functionality, but works with both threads and processes.
See Pipes and Queues in the multiprocessing doc for more details—but you probably don't need any more details; the multiprocessing.Queue is meant to be as close to a multi-process clone of Queue.Queue as possible.
If you want to understand the under-the-covers difference:
A Queue.Queue is a deque with condition variables wrapped around it. It relies on the fact that code running in the same interpreter can access the same objects to share the deque, and uses the condition variables to protect the deque from races as well as for signaling.
A multiprocessing.Queue is a more complicated thing that pickles objects and passes them over a pipe between the processes. Races aren't a problem, but signaling still is, so it also has the equivalent of condition variables, but obviously not the ones from threading.
I am sure many similar questions have been asked before, but after reading many of them I am still not very sure what I should do. So, I have a Python script to control some external instruments (a camera and a power meter). I have written class for both instruments by calling the C functions in the .dll files using ctypes. Right now it looks something like this:
for i in range(10):
power_reading = newport.get_reading(N=100,interval=1) # take power meter reading
img = camera.capture(N=10)
value = image_processing(img) # analyze the img (ndarray) to get some values
results.append([power_reading,value]) # add both results to a list
I want to start executing the first two lines at the same time. Both newport.get_reading and camera.capture take about 100ms-1s to run (they will run for the same time if I choose the correct arguments). I don't need them to start at EXACTLY the same time, but ideally the delay should be smaller than about 10-20% of the total run time (so less than 0.2s delay when take each take about 1s to run). From what I have read, I can use the multiprocessing module. So I try something like this based on this post:
def p_get_reading(newport,N,interval,return_dict):
reading = newport.get_reading(N,interval,return_dict)
return_dict['power_meter'] = reading
def p_capture(camera,N,return_dict):
img = camera.capture(N)
return_dict['image'] = img
for i in range(10):
manager = multiprocessing.Manager()
return_dict = manager.dict()
p = multiprocessing.Process(target=p_capture, args=(camera,10))
p.start()
p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1))
p2.start()
p.join()
p2.join()
print(return_dict)
I have a few problems/questions:
I need to get the return values from both function calls. Using my current method, return_dict is only showing the entry for capture_img but not the power meter reading, why is that? It also read that I can use Queue, what is the best method for my current purpose?
How can I know whether both functions indeed start running at the same time? I am thinking of using the time module to record both the start and end time of both functions, maybe using some wrapper function to do the time logging, but will the use of multiprocessing pose any restrictions?
I usually run my code on an IDE (spyder), and from what I have read, I need to run in command prompt to see the output (I have some print statements inside the functions for debugging purposes). Can I still run in IDE for having both functions run at the same time?
Using a Lock may help with synchronisation:
import multiprocessing
def p_get_reading(newport, N, interval, lock, return_dict):
lock.acquire()
lock.release()
reading = newport.get_reading(N, interval)
return_dict['power_meter'] = reading
def p_capture(camera, N, lock, return_dict):
lock.acquire()
lock.release()
img = camera.capture(N)
return_dict['image'] = img
if __name__ == "__main__":
for i in range(10):
manager = multiprocessing.Manager()
return_dict = manager.dict()
lock = multiprocessing.Lock()
lock.acquire()
p = multiprocessing.Process(target=p_capture, args=(camera,10,lock,return_dict))
p.start()
p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1,lock,return_dict))
p2.start()
lock.release()
p.join()
p2.join()
print(return_dict)
The two Process objects can now be created and start()ed in any order as the main routine has already acquired the lock. Once released, the two processes will fight between themselves to acquire and release the lock, and be ready almost at the same time.
Also, note the use of if __name__ == "__main__" as this helps when multiprocessing makes new processes.
I must say this seems like an abuse of a Lock
An answer to your first question is simply no if you are doing in normal way, but yes if you want it to be. No because the target function cannot communicate back to spawning thread using a return. One way to do it is to use a queue and wrapper functions as following:
from threading import Thread
from Queue import Queue
def p_get_reading(newport,N,interval,return_dict):
reading = newport.get_reading(N,interval,return_dict)
return_dict.update({'power_meter': reading})
return return_dict
def p_capture(camera,N,return_dict):
img = camera.capture(N)
return_dict.update({'image': img})
return return_dict
def wrapper1(func, arg1, arg2, queue):
queue.put(func(arg1, arg2))
def wrapper2(func, arg1, arg2, arg3, queue):
queue.put(func(arg1, arg2, arg3))
q = Queue()
Thread(target=wrapper1, args=(p_capture, camera, 10 , q)).start()
Thread(target=wrapper2, args=(p_get_reading, newport, 100, 1, q)).start()
Now q holds the updated and returned dict from p_capture() and p_get_reading().
I have code that makes unique combinations of elements. There are 6 types, and there are about 100 of each. So there are 100^6 combinations. Each combination has to be calculated, checked for relevance and then either be discarded or saved.
The relevant bit of the code looks like this:
def modconffactory():
for transmitter in totaltransmitterdict.values():
for reciever in totalrecieverdict.values():
for processor in totalprocessordict.values():
for holoarray in totalholoarraydict.values():
for databus in totaldatabusdict.values():
for multiplexer in totalmultiplexerdict.values():
newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
data_I_need = dosomethingwith(newconfiguration)
saveforlateruse_if_useful(data_I_need)
Now this takes a long time and that is fine, but now I realize this process (making the configurations and then calculations for later use) is only using 1 of my 8 processor cores at a time.
I've been reading up about multithreading and multiprocessing, but I only see examples of different processes, not how to multithread one process. In my code I call two functions: 'dosomethingwith()' and 'saveforlateruse_if_useful()'. I could make those into separate processes and have those run concurrently to the for-loops, right?
But what about the for-loops themselves? Can I speed up that one process? Because that is where the time consumption is. (<-- This is my main question)
Is there a cheat? for instance compiling to C and then the os multithreads automatically?
I only see examples of different processes, not how to multithread one process
There is multithreading in Python, but it is very ineffective because of GIL (Global Interpreter Lock). So if you want to use all of your processor cores, if you want concurrency, you have no other choice than use multiple processes, which can be done with multiprocessing module (well, you also could use another language without such problems)
Approximate example of multiprocessing usage for your case:
import multiprocessing
WORKERS_NUMBER = 8
def modconffactoryProcess(generator, step, offset, conn):
"""
Function to be invoked by every worker process.
generator: iterable object, the very top one of all you are iterating over,
in your case, totalrecieverdict.values()
We are passing a whole iterable object to every worker, they all will iterate
over it. To ensure they will not waste time by doing the same things
concurrently, we will assume this: each worker will process only each stepTH
item, starting with offsetTH one. step must be equal to the WORKERS_NUMBER,
and offset must be a unique number for each worker, varying from 0 to
WORKERS_NUMBER - 1
conn: a multiprocessing.Connection object, allowing the worker to communicate
with the main process
"""
for i, transmitter in enumerate(generator):
if i % step == offset:
for reciever in totalrecieverdict.values():
for processor in totalprocessordict.values():
for holoarray in totalholoarraydict.values():
for databus in totaldatabusdict.values():
for multiplexer in totalmultiplexerdict.values():
newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
data_I_need = dosomethingwith(newconfiguration)
saveforlateruse_if_useful(data_I_need)
conn.send('done')
def modconffactory():
"""
Function to launch all the worker processes and wait until they all complete
their tasks
"""
processes = []
generator = totaltransmitterdict.values()
for i in range(WORKERS_NUMBER):
conn, childConn = multiprocessing.Pipe()
process = multiprocessing.Process(target=modconffactoryProcess, args=(generator, WORKERS_NUMBER, i, childConn))
process.start()
processes.append((process, conn))
# Here we have created, started and saved to a list all the worker processes
working = True
finishedProcessesNumber = 0
try:
while working:
for process, conn in processes:
if conn.poll(): # Check if any messages have arrived from a worker
message = conn.recv()
if message == 'done':
finishedProcessesNumber += 1
if finishedProcessesNumber == WORKERS_NUMBER:
working = False
except KeyboardInterrupt:
print('Aborted')
You can adjust WORKERS_NUMBER to your needs.
Same with multiprocessing.Pool:
import multiprocessing
WORKERS_NUMBER = 8
def modconffactoryProcess(transmitter):
for reciever in totalrecieverdict.values():
for processor in totalprocessordict.values():
for holoarray in totalholoarraydict.values():
for databus in totaldatabusdict.values():
for multiplexer in totalmultiplexerdict.values():
newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
data_I_need = dosomethingwith(newconfiguration)
saveforlateruse_if_useful(data_I_need)
def modconffactory():
pool = multiprocessing.Pool(WORKERS_NUMBER)
pool.map(modconffactoryProcess, totaltransmitterdict.values())
You probably would like to use .map_async instead of .map
Both snippets do the same, but I would say in the first one you have more control over the program.
I suppose the second one is the easiest, though :)
But the first one should give you the idea of what is happening in the second one
multiprocessing docs: https://docs.python.org/3/library/multiprocessing.html
you can run your function in this way:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
p = Pool(5)
print(p.map(f, [1, 2, 3]))
https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers
I have a large codebase to parallelise. I can avoid rewriting the method signatures of hundreds of functions by using a single global queue. I know it's messy; please don't tell me that if I'm using globals I'm doing something wrong in this case it really is the easiest choice. The code below works but i don't understand why. I declare a global multiprocessing.Queue() but don't declare that it should be shared between processes (by passing it as a parameter to the worker). Does python automatically place this queue in shared memory? Is it safe to do this on a larger scale?
Note: You can tell that the queue is shared between the processes: the worker processes start doing work on empty queues and are idle for one second before the main queue pushes some work onto the queues.
import multiprocessing
import time
outqueue = None
class WorkerProcess(multiprocessing.Process):
def __init__(self):
multiprocessing.Process.__init__(self)
self.exit = multiprocessing.Event()
def doWork(self):
global outqueue
ob = outqueue.get()
ob = ob + "!"
print ob
time.sleep(1) #simulate more hard work
outqueue.put(ob)
def run(self):
while not self.exit.is_set():
self.doWork()
def shutdown(self):
self.exit.set()
if __name__ == '__main__':
global outqueue
outqueue = multiprocessing.Queue()
procs = []
for x in range(10):
procs.append(WorkerProcess())
procs[x].start()
time.sleep(1)
for x in range(20):
outqueue.put(str(x))
time.sleep(10)
for p in procs:
p.shutdown()
for p in procs:
p.join()
try:
while True:
x = outqueue.get(False)
print x
except:
print "done"
Assuming you're using Linux, the answer is in the way the OS creates a new process.
When a process spawns a new one in Linux, it actually forks the parent one. The result is a child process with all the properties of the parent one. Basically a clone.
In your example you are instantiating the Queue and then creating the new processes. Therefore the children processes will have a copy of the same queue and will be able to use it.
To see things broken just try to first create the processes and then creating the Queue object. You'll see the children having the global variable still set as None while the parent will have a Queue.
It is safe, yet not recommended, to share a Queue as a global variable on Linux. On Windows, due to the different process creation approach, sharing a queue through a global variable won't work.
As mentioned in the programming guidelines
Explicitly pass resources to child processes
On Unix using the fork start method, a child process can make use of a shared resource created in a parent process using a global resource. However, it is better to pass the object as an argument to the constructor for the child process.
Apart from making the code (potentially) compatible with Windows and the other start methods this also ensures that as long as the child process is still alive the object will not be garbage collected in the parent process. This might be important if some resource is freed when the object is garbage collected in the parent process.
For more info about Linux forking you can read its man page.
While attempting to store multiprocessing's process instance in multiprocessing list-variable 'poolList` I am getting a following exception:
SimpleQueue objects should only be shared between processes through inheritance
The reason why I would like to store the PROCESS instances in a variable is to be able to terminate all or just some of them later (if for example a PROCESS freezes). If storing a PROCESS in variable is not an option I would like to know how to get or to list all the PROCESSES started by mutliprocessing POOL. That would be very similar to what .current_process() method does. Except .current_process gets only a single process while I need all the processes started or all the processes currently running.
Two questions:
Is it even possible to store an instance of the Process (as a result of mp.current_process()
Currently I am only able to get a single process from inside of the function that the process is running (from inside of myFunct() using .current_process() method).
Instead I would like to to list all the processes currently running by multiprocessing. How to achieve it?
import multiprocessing as mp
poolList=mp.Manager().list()
def myFunct(arg):
print 'myFunct(): current process:', mp.current_process()
try: poolList.append(mp.current_process())
except Exception, e: print e
for i in range(110):
for n in range(500000):
pass
poolDict[arg]=i
print 'myFunct(): completed', arg, poolDict
from multiprocessing import Pool
pool = Pool(processes=2)
myArgsList=['arg1','arg2','arg3']
pool=Pool(processes=2)
pool.map_async(myFunct, myArgsList)
pool.close()
pool.join()
To list the processes started by a Pool()-instance(which is what you mean if I understand you correctly), there is the pool._pool-list. And it contains the instances of the processes.
However, it is not part of the documented interface and hence, really should not be used.
BUT...it seems a little bit unlikely that it would change just like that anyway. I mean, should they stop having an internal list of processes in the pool? And not call that _pool?
And also, it annoys me that there at least isn't a get processes-method. Or something.
And handling it breaking due to some name change should not be that difficult.
But still, use at your own risk:
from multiprocessing import pool
# Have to run in main
if __name__ == '__main__':
# Create 3 worker processes
_my_pool = pool.Pool(3)
# Loop, terminate, and remove from the process list
# Use a copy [:] of the list to remove items correctly
for _curr_process in _my_pool._pool[:]:
print("Terminating process "+ str(_curr_process.pid))
_curr_process.terminate()
_my_pool._pool.remove(_curr_process)
# If you call _repopulate, the pool will again contain 3 worker processes.
_my_pool._repopulate_pool()
for _curr_process in _my_pool._pool[:]:
print("After repopulation "+ str(_curr_process.pid))
The example creates a pool and manually terminates all processes.
It is important that you remember to delete the process you terminate from the pool yourself i you want Pool() to continue working as usual.
_my_pool._repopulate increases the number of working processes to 3 again, not needed to answer the question, but gives a little bit of behind-the-scenes insight.
Yes you can get all active process and perform action based on name of process
e.g
multiprocessing.Process(target=foo, name="refresh-reports")
and then
for p in multiprocessing.active_children():
if p.name == "foo":
p.terminate()
You're creating a managed List object, but then letting the associated Manager object expire.
Process objects are shareable because they aren't pickle-able; that is, they aren't simple.
Oddly the multiprocessing module doesn't have the equivalent of threading.enumerate() -- that is, you can't list all outstanding processes. As a workaround, I just store procs in a list. I never terminate() a process, but do sys.exit(0) in the parent. It's rough, because the workers will leave things in an inconsistent state, but it's okay for smaller programs
To kill a frozen worker, I suggest: 1) worker receives "heartbeat" jobs in a queue every now and then, 2) if parent notices worker A hasn't responded to a heartbeat in a certain amount of time, then p.terminate(). Consider restating the problem in another SO question, as it's interesting.
To be honest the map stuff is much easier than using a Manager.
Here's a Manager example I've used. A worker adds stuff to a shared list. Another worker occasionally wakes up, processes everything on the list, then goes back to sleep. The code also has verbose logs, which are essential for ease in debugging.
source
# producer adds to fixed-sized list; scanner uses them
import logging, multiprocessing, sys, time
def producer(objlist):
'''
add an item to list every sec; ensure fixed size list
'''
logger = multiprocessing.get_logger()
logger.info('start')
while True:
try:
time.sleep(1)
except KeyboardInterrupt:
return
msg = 'ding: {:04d}'.format(int(time.time()) % 10000)
logger.info('put: %s', msg)
del objlist[0]
objlist.append( msg )
def scanner(objlist):
'''
every now and then, run calculation on objlist
'''
logger = multiprocessing.get_logger()
logger.info('start')
while True:
try:
time.sleep(5)
except KeyboardInterrupt:
return
logger.info('items: %s', list(objlist))
def main():
logger = multiprocessing.log_to_stderr(
level=logging.INFO
)
logger.info('setup')
# create fixed-length list, shared between producer & consumer
manager = multiprocessing.Manager()
my_objlist = manager.list( # pylint: disable=E1101
[None] * 10
)
multiprocessing.Process(
target=producer,
args=(my_objlist,),
name='producer',
).start()
multiprocessing.Process(
target=scanner,
args=(my_objlist,),
name='scanner',
).start()
logger.info('running forever')
try:
manager.join() # wait until both workers die
except KeyboardInterrupt:
pass
logger.info('done')
if __name__=='__main__':
main()