This is for Python 3.2.2. I'm only just learning how Python and multiprocessing work and this simple example has been tripping me:
from multiprocessing import Pipe, Process
def f(r):
print(r.recv())
if __name__ == '__main__':
q, r = Pipe()
p = Process(target=f, args=(r,))
p.start()
q.send([42, None, 'hello'])
p.join()
The main thread creates a new Process, p, and sends r, a bidirectional connection object, to function f(). When process p is started, I expect r.recv() to block (which, as far as I understand, means that this process will wait indefinitely until something comes through the pipe) until the main process sends some objects through with q.send.
Then p.join() should make the main process wait until p has run its course.
But nothing whatsoever happens. If I add a print statement to f(), nothing happens there, either, as if f() never even runs and p.start() is nonfunctional.
Can you explain why this won't work and what might be fixed?
I know it's been a while, but for others with this problem, you have the ends of your pipe reversed. You're trying to use the receiving end to send, and trying to receive with the sending end. I find that adding duplex=True to the Pipe constructor makes it much easier to deal with the different ends.
Source: https://docs.python.org/2/library/multiprocessing.html#pipes-and-queues
From experience I've found that I can't print from processes that I've started. You could try reversing your program:
from multiprocessing import Pipe, Process
def f(r):
r.send([42, None, 'hello'])
if __name__ == '__main__':
q, r = Pipe()
p = Process(target=f, args=(r,))
p.start()
print(q.recv())
p.join()
Related
I'm learning multiprocessing with python and it doesn't seem to work with the print() function and IDLE's shell.
Also, making a process as Daemon doesn't seems to work either as the process doesn't get killed when the Main program ends.
here is the code I wrote, I hope some1 could explain what could be wrong:
import multiprocessing
import time
def proc1(x): # Creates a function to be used inside a process
for i in range(x):
print("proc1 is running") # It seems like the Child Processes doesn't print the "print()" function into the IDLE's shell
if __name__ == '__main__': # Important!!! we dont want to create endless subprocesses by mistake
proc = multiprocessing.Process(name='proc of Python' , target=proc1 , args=[300000])
proc.daemon = True # make the process a Daemon and get killed with the end of the Main Program - doesn't seems to work in this example, I can see the process keep running on Task Manager
proc2 = multiprocessing.Process(name='proc2 of Python' , target=proc1 , args=[300000])
proc2.start()
proc.start()
# proc.join()
print('Multi Processing is hard!!\n\n')
A basic example of multiprocessing Process class runs when executed from file, but not from IDLE. Why is that and can it be done?
from multiprocessing import Process
def f(name):
print('hello', name)
p = Process(target=f, args=('bob',))
p.start()
p.join()
Yes. The following works in that function f is run in a separate (third) process.
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
However, to see the print output, at least on Windows, one must start IDLE from a console like so.
C:\Users\Terry>python -m idlelib
hello bob
(Use idlelib.idle on 2.x.) The reason is that IDLE runs user code in a separate process. Currently the connection between the IDLE process and the user code process is via a socket. The fork done by multiprocessing does not duplicate or inherit the socket connection. When IDLE is started via an icon or Explorer (in Windows), there is nowhere for the print output to go. When started from a console with python (rather than pythonw), output goes to the console, as above.
The following code, does not print "here". What is the problem?
I tested it on both my machines (windows 7, Ubuntu 12.10), and
http://www.compileonline.com/execute_python_online.php
It does not print "here" in all cases.
from multiprocessing import Queue, Process
def runLang(que):
print "start"
myDict=dict()
for i in xrange(10000):
myDict[i]=i
que.put(myDict)
print "finish"
def run(fileToAnalyze):
que=Queue()
processList=[]
dicList=[]
langs= ["chi","eng"]
for lang in langs:
p=Process(target=runLang,args=(que,))
processList.append(p)
p.start()
for p1 in processList:
p1.join()
print "here"
for _ in xrange(len(langs)):
item=que.get()
print item
dicList.append(item)
if __name__=="__main__":
processList = []
for fileToAnalyse in ["abc.txt","def.txt"]:
p=Process(target=run,args=(fileToAnalyse,))
processList.append(p)
p.start()
for p1 in processList:
p1.join()
This is because when you put lots of items into a multiprocessing.Queue, they eventually get buffered in memory, once the underlying Pipe is full. The buffer won't get flushed until something starts reading from the other end of the Queue, which will allow the Pipe to accept more data. A Process cannot terminate until the buffer for all its Queue instances have been entirely flushed to their underlying Pipe. The implication of this is that if you try to join a process without having another process/thread calling get on its Queue, you could deadlock. This is mentioned in the docs:
Warning
As mentioned above, if a child process has put items on a queue (and
it has not used JoinableQueue.cancel_join_thread), then that process
will not terminate until all buffered items have been flushed to the
pipe.
This means that if you try joining that process you may get a deadlock
unless you are sure that all items which have been put on the queue
have been consumed. Similarly, if the child process is non-daemonic
then the parent process may hang on exit when it tries to join all its
non-daemonic children.
Note that a queue created using a manager does not have this issue.
You can fix the issue by not calling join until after you empty the Queue in the parent:
for _ in xrange(len(langs)):
item = que.get()
print(item)
dicList.append(item)
# join after emptying the queue.
for p in processList:
p.join()
print("here")
I am using multiprocessing module's Process class to spawn multiple processes, those processes execute some script and then dies.What I wanted, a timeout to be applied on each process, so that a process would die if cant execute in time timeout. I am using join(timeout) on Process objects.
Since the join() function doesn;t kill the process, it just blocks the process until it finishes
Now my question : Is there any side-effects of using join() with timeout ..like, would the processes be cleaned automatically, after the main process dies ?? or I have to kill those processes manually ??
I am a newbie to python and its multiprocessing module, please be patient.
My Code, which is creating Processes in a for loop ::
q = Queue()
jobs = [
Process(
target=get_current_value,
args=(q,),
kwargs=
{
'device': device,
'service_list': service_list,
'data_source_list': data_source_list
}
) for device in device_list
]
for j in jobs:
j.start()
for k in jobs:
k.join()
The timeout argument just tells join how long to wait for the Process to exit before giving up. If timeout expires, the Process does not exit; the join call simply unblocks. If you want to end your workers when the timeout expires, you need to do so manually. You can either use terminate, as suggested by wRAR, to uncleanly shut things down, or use some other signaling mechanism to tell the children to shutdown cleanly:
p = Process(target=worker, args=(queue,))
p.start()
p.join(50)
if p.isalive(): # join timed out without the process actually finishing
#p.terminate() # unclean shutdown
If you don't want to use terminate, the alternative approach is really dependent on what the workers are doing. If they're consuming from a queue, you can use a sentinel:
def worker(queue):
for item in iter(queue.get, None): # None will break the loop
# Do normal work
if __name__ == "__main__":
queue = multiprocessing.Queue()
p = multiprocessing.Process(target=worker, args=(queue,))
p.start()
# Do normal work here
# Time to shut down
queue.put(None)
Or you could use an Event, if you're doing some other operation in a loop:
def worker(event):
while not event.is_set():
# Do work here
if __name__ == "__main__":
event= multiprocessing.Event()
p = multiprocessing.Process(target=worker, args=(event,))
p.start()
# Do normal work here
# Time to shut down
event.set()
Using terminate could be just fine, though, unless your child processes are using resources that could be corrupted if the process is unexpectedly shut down (like writing to a file or db, or holding a lock). If you're just doing some calculations in the worker, using terminate won't hurt anything.
join() does nothing with the child process. If you really want to terminate worker process in a non-clean manner you should use terminate() (you should understand the consequences).
If you want children to be terminated when the main process exits you should set daemon attribute on them.
From within my python script, I want to start another python script which will run in the background waiting for the instruction to terminate.
Host Python script (H1) starts subprocess P1.
P1 performs some short lived work & returns a sentinel to indicate that it is now going to sleep awaiting instructions to terminate.
H1 polls for this sentinel repeatedly. When it receives the sentinel, it performs some other IO bound task and when that completes, tells P1 to die gracefully (meaning close any resources that you have acquired).
Is this feasible to do with the subprocess module ?
Yes, start the process with :
p=subprocess.Popen([list for the script to execute], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
You can then read from p.stdout and p.stderr to watch for your sentinel and write to p.stdin to send messages to the child process. If you are running on a posix system, you might consider using pexpect instead; it doesn't support MS Windows, but it handles communicating with child processes better than subprocess.
"""H1"""
from multiprocessing import Process, Pipe
import sys
def P1(conn):
print 'P1: some short lived work'
sys.stdout.flush()
conn.send('work done')
# wait for shutdown command...
conn.recv()
conn.close()
print 'P1: shutting down'
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=P1, args=(child_conn,))
p.start()
print parent_conn.recv()
print 'H1: some other IO bound task'
parent_conn.send("game over")
p.join()
Output:
P1: some short lived work
work done
H1: some other IO bound task
P1: shutting down