I'm learning multiprocessing in Python while I found this odd behaviour between daemon and non-daemon process with respect to the main process.
My Code:
import multiprocessing
import time
def worker(name,num):
print name, 'Starting'
time.sleep(num)
print name, 'Exiting'
if __name__ == '__main__':
print 'main starting'
p1=multiprocessing.Process(target=worker, args=("p1",7,))
p2=multiprocessing.Process(target=worker, args=("p2",5,))
p2.daemon=True
p1.start()
p2.start()
time.sleep(4)
print 'main exiting'
The output I'm getting is:
main starting
p1 Starting
p2 Starting
main exiting
p1 Exiting
Expected Output:
main starting
p1 Starting
p2 Starting
main exiting
p2 Exiting
p1 Exiting
After few searches, I found this answer and inserted the following line to my code.
logger = multiprocessing.log_to_stderr(logging.INFO)
And the output I got is,
main starting
[INFO/p1] child process calling self.run()
p1 Starting
[INFO/p2] child process calling self.run()
p2 Starting
main exiting
[INFO/MainProcess] process shutting down
[INFO/MainProcess] calling terminate() for daemon p2
[INFO/MainProcess] calling join() for process p2
[INFO/MainProcess] calling join() for process p1
p1 Exiting
[INFO/p1] process shutting down
[INFO/p1] process exiting with exitcode 0
As per my understanding,
When a main process exits, it terminates all of its child daemon processes. (Which can be seen here)
A main process can not exit before all of its child non-daemon processes exit.
But here, why the main process is trying to shut down before p1 exits?
p1 starts and sleeps for 7 seconds.
p2 starts and sleeps for 5 seconds.
main process sleeps for 4 seconds.
main process wakes up after 4 seconds and waits for p1 to exit.
p2 wakes up after 5 seconds and exits.
p1 wakes up after 7 seconds and exits.
main process exits.
Wouldn't be the above a normal time-line for the above program?
Can anyone please explain what's happening here and why?
EDIT
After adding the line p1.join() at the end of the code, I'm getting the following output:
main starting
[INFO/Process-1] child process calling self.run()
p1 Starting
[INFO/Process-2] child process calling self.run()
p2 Starting
main exiting
p2 Exiting
[INFO/Process-2] process shutting down
[INFO/Process-2] process exiting with exitcode 0
p1 Exiting
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
When you see
[INFO/MainProcess] calling join() for process p1
it means the main process has not exited yet -- it's in the process of shutting down, but of course will not be done shutting down until that join returns... which will happen only once the joined process is done.
So the timeline is indeed as you expect it to be -- but since that join to p1 is the very last think the main process ever does, you don't see that in the output or logging from it. (main has taken all the termination-triggered action, but as a process it's still alive until then).
To verify, use (on a Unixy system) ps from another terminal while running this (perhaps with slightly longer delays to help you check): you will never see a single Python process running out of this complex -- there will be two (main and p1) until the end.
Python Multiprocess| Daemon & Join
Question end up adding more wait towards the behavior of daemon flag and join method, hence here a quick explanation with a simple script.
The daemon process is terminated automatically before the main program exits, to avoid leaving orphaned processes running but is not terminated with the main program. So any outstanding processes left by the daemoned function won't run!
However if the join() method is called on the daemoned function then the main programm will wait the left over processes.
INPUT
import multiprocessing
import time
import logging
def daemon():
p = multiprocessing.current_process()
print('Starting:', p.name, p.pid, flush=True)
print('---' * 15)
time.sleep(2)
print('===' * 15 + ' < ' + f'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush=True)
def non_daemon():
p = multiprocessing.current_process()
print('Starting:', p.name, p.pid, flush=True)
print('---'*15)
print('===' * 15 + ' < ' + f'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush=True)
if __name__ == '__main__':
print('==='*15 + ' < ' + 'MAIN PROCESS' + ' > ' + '==='*15)
logger = multiprocessing.log_to_stderr(logging.INFO)
# DAEMON
daemon_proc = multiprocessing.Process(name='MyDaemon', target=daemon)
daemon_proc.daemon = True
# NO-DAEMON
normal_proc = multiprocessing.Process(name='non-daemon', target=non_daemon)
normal_proc.daemon = False
daemon_proc.start()
time.sleep(2)
normal_proc.start()
# daemon_proc.join()
OUTPUT NO JOIN METHOD##
============================================= < MAIN PROCESS > =============================================
Starting: MyDaemon 8448
---------------------------------------------
[INFO/MyDaemon] child process calling self.run()
[INFO/MainProcess] process shutting down
[INFO/MainProcess] calling terminate() for daemon MyDaemon
[INFO/MainProcess] calling join() for process MyDaemon
[INFO/MainProcess] calling join() for process non-daemon
Starting: non-daemon 6688
---------------------------------------------
============================================= < Exiting=('non-daemon', 6688) > =============================================
[INFO/non-daemon] child process calling self.run()
[INFO/non-daemon] process shutting down
[INFO/non-daemon] process exiting with exitcode 0
Process finished with exit code 0
The function daemon() is 'spleeping' 2 second, hence as Python reaches the bottom of the script the program shut down but not daemon().
Note:
[INFO/MainProcess] calling terminate() for daemon MyDaemon
Now if the last line of the script daemon_proc.join() is not commented.
OUTPUT + JOIN METHOD
============================================= < MAIN PROCESS > =============================================
Starting: MyDaemon 13588
---------------------------------------------
[INFO/MyDaemon] child process calling self.run()
============================================= < Exiting=('MyDaemon', 13588) > =============================================
[INFO/MyDaemon] process shutting down
[INFO/MyDaemon] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[INFO/MainProcess] calling join() for process non-daemon
Starting: non-daemon 13608
---------------------------------------------
============================================= < Exiting=('non-daemon', 13608) > =============================================
[INFO/non-daemon] child process calling self.run()
[INFO/non-daemon] process shutting down
[INFO/non-daemon] process exiting with exitcode 0
Process finished with exit code 0
Related
#coding=utf8
from multiprocessing import Process
from time import sleep
import os
def foo():
print("foo")
for i in range(11000):
sleep(1)
print("{}: {}".format(os.getpid(), i))
if __name__ == "__main__":
p = Process(target=foo)
#p.daemon = True
p.start()
print("parent: {}".format(os.getpid()))
sleep(20)
raise Exception("invalid")
An exception is raised in the main-process, but child- and main-process keep running. Why?
When the MainProcess is shutting down, non-daemonic child processes are simply joined. This happens in _exit_function() which is registered with atexit.register(_exit_function). You can look it up in multiprocessing.util.py if you are curious.
You can also insert multiprocessing.log_to_stderr(logging.DEBUG) before you start your process to see the log messages:
parent: 9670
foo
[INFO/Process-1] child process calling self.run()
9675: 0
9675: 1
...
9675: 18
Traceback (most recent call last):
File "/home/...", line 26, in <module>
raise Exception("invalid")
Exception: invalid
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[INFO/MainProcess] calling join() for process Process-1
9675: 19
I am writing a python script which has 2 child processes. The main logic occurs in one process and another process waits for some time and then kills the main process even if the logic is not done.
I read that calling os_exit(1) stops the interpreter, so the entire script is killed automatically. I've used it like shown below:
import os
from multiprocessing import Process, Lock
from multiprocessing.sharedctypes import Array
# Main process
def main_process(shared_variable):
shared_variable.value = "mainprc"
time.sleep(20)
print("Task finished normally.")
os._exit(1)
# Timer process
def timer_process(shared_variable):
threshold_time_secs = 5
time.sleep(threshold_time_secs)
print("Timeout reached")
print("Shared variable ",shared_variable.value)
print("Task is shutdown.")
os._exit(1)
if __name__ == "__main__":
lock = Lock()
shared_variable = Array('c',"initial",lock=lock)
process_main = Process(target=main_process, args=(shared_variable))
process_timer = Process(target=timer_process, args=(shared_variable))
process_main.start()
process_timer.start()
process_timer.join()
The timer process calls os._exit but the script still waits for the main process to print "Task finished normally." before exiting.
How do I make it such that if timer process exits, the entire program is shutdown (including main process)?
Thanks.
My objective here is to make the child threads run in background while the main threads should start them and then exit:
I tried the below code:
import time
import logging
import threading
logging.basicConfig(filename="threading.log",level=logging.DEBUG)
def worker(count):
for c in range(count,-1,-1):
threadname = threading.currentThread().getName()
threadcount = threading.active_count()
threadnames = threading.enumerate()
logging.debug("Child thread: {} continuing with threadcount {} {} and counter value: {}".format(threadname,threadcount,threadnames,c))
time.sleep(2)
mainthread = threading.currentThread().getName()
print ("Starting main thread:",mainthread)
t1 = threading.Thread(target=worker,args=(10,))
t1.setDaemon(True)
t1.start()
time.sleep(5)
print ("Attempting to close main thread:",mainthread)
But as soon as the main thread exits i think the child thread also exits since i have this output in the threading.log(which i created from child thread)
DEBUG:root:Child thread: Thread-1 continuing with threadcount 2 [<_MainThread(MainThread, started 1160)>, <Thread(Thread-1, started daemon 10232)>] and counter value: 10
DEBUG:root:Child thread: Thread-1 continuing with threadcount 2 [<_MainThread(MainThread, started 1160)>, <Thread(Thread-1, started daemon 10232)>] and counter value: 9
DEBUG:root:Child thread: Thread-1 continuing with threadcount 2 [<_MainThread(MainThread, started 1160)>, <Thread(Thread-1, started daemon 10232)>] and counter value: 8
I know using join() would not be the answer since the main thread would block.
I dont want the main thread to block at all.
Is there a solution to the issue?
Thanks in advance.
You couldn't do that, this is not a problem about python, but a problem about system.
No matter child thread or child process will exit if main process exit. This was controlled by system, so you can do nothing on it.
Or you can change your mind, why you have to exit your main process?
You can put main process to run as a daemon and listen for some variables or something like that (you can try with json)
If you will close main process all other threads etc... will be closed.
you ought to explain why you want this behavior because it shouldn't be necessary, by making the main thread wait with join it becomes dormant, not really consuming any more power/memory than if it wasn't there.
However the t1.setDaemon(True) is what makes the child threads stop when the main process does, comment out or remove this line and it should do what you want. but this it nor good practice.
example:
import time
import logging
import threading
logging.basicConfig(filename="threading.log",level=logging.DEBUG)
def worker(count):
for c in range(count,-1,-1):
threadname = threading.currentThread().getName()
threadcount = threading.active_count()
threadnames = threading.enumerate()
logging.debug("Child thread: {} continuing with threadcount {} {} and counter value: {}".format(threadname,threadcount,threadnames,c))
time.sleep(2)
mainthread = threading.currentThread().getName()
print ("Starting main thread:",mainthread)
t1 = threading.Thread(target=worker,args=(10,))
#t1.setDaemon(True) # do not set daemon
t1.start()
time.sleep(5)
print ("Attempting to close main thread:",mainthread)
I'm trying to understand multiprocessing module. Below is my code.
from multiprocessing import Process, current_process
#from time import time
import time
def work(delay):
p = current_process()
print p.name, p.pid, p.deamon
time.sleep(delay)
print 'Finised deamon work'
def main():
print 'Starting Main Process'
p = Process(target=work, args=(2,))
p.deamon = True
p.start()
print 'Exiting Main Process'
if __name__ == '__main__':
main()
Output:
Starting Main Process
Exiting Main Process
Process-1 7863 True
Finised deamon work
I expect main process to exit before deamon process(sleep for 2 secs). Since main process exits, deamon process should also exit. But output is confusing me.
Expected Output:
Starting Main Process
Exiting Main Process
Process-1 7863 True
Is my understanding of multiprocessing module wrong?
I have a python script: zombie.py
from multiprocessing import Process
from time import sleep
import atexit
def foo():
while True:
sleep(10)
#atexit.register
def stop_foo():
p.terminate()
p.join()
if __name__ == '__main__':
p = Process(target=foo)
p.start()
while True:
sleep(10)
When I run this with python zombie.py & and kill the parent process with kill -2, the stop() is correctly called and both processes terminate.
Now, suppose I have a bash script zombie.sh:
#!/bin/sh
python zombie.py &
echo "done"
And I run ./zombie.sh from the command line.
Now, stop() never gets called when the parent gets killed. If I run kill -2 on the parent process, nothing happens. kill -15 or kill -9 both just kill the parent process, but not the child:
[foo#bar ~]$ ./zombie.sh
done
[foo#bar ~]$ ps -ef | grep zombie | grep -v grep
foo 27220 1 0 17:57 pts/3 00:00:00 python zombie.py
foo 27221 27220 0 17:57 pts/3 00:00:00 python zombie.py
[foo#bar ~]$ kill -2 27220
[foo#bar ~]$ ps -ef | grep zombie | grep -v grep
foo 27220 1 0 17:57 pts/3 00:00:00 python zombie.py
foo 27221 27220 0 17:57 pts/3 00:00:00 python zombie.py
[foo#bar ~]$ kill 27220
[foo#bar ~]$ ps -ef | grep zombie | grep -v grep
foo 27221 1 0 17:57 pts/3 00:00:00 python zombie.py
What is going on here? How can I make sure the child process dies with the parent?
Neither the atexit nor the p.daemon = True will truely ensure that the child process will die with the father. Receiving a SIGTERM will not trigger the atexit routines.
To make sure the child gets killed upon its father's death you will have to install a signal handler in the father. This way you can react on most signals (SIGQUIT, SIGINT, SIGHUP, SIGTERM, ...) but not on SIGKILL; there simply is no way to react on that signal from within the process which receives it.
Install a signal handler for all useful signals and in that handler kill the child process.
Update: This solution doesn't work for processes killed by a signal.
Your child process is not a zombie. It is alive.
If you want the child process to be killed when its parent exits normally then set p.daemon = True before p.start(). From the docs:
When a process exits, it attempts to terminate all of its daemonic child processes.
Looking at the source code, it is clear that multiprocessing uses atexit callback to kill its daemonic children i.e., it won't work if the parent is killed by a signal. For example:
#!/usr/bin/env python
import logging
import os
import signal
import sys
from multiprocessing import Process, log_to_stderr
from threading import Timer
from time import sleep
def foo():
while True:
sleep(1)
if __name__ == '__main__':
log_to_stderr().setLevel(logging.DEBUG)
p = Process(target=foo)
p.daemon = True
p.start()
# either kill itself or exit normally in 5 seconds
if '--kill' in sys.argv:
Timer(5, os.kill, [os.getpid(), signal.SIGTERM]).start()
else: # exit normally
sleep(5)
Output
$ python kill-orphan.py
[INFO/Process-1] child process calling self.run()
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[INFO/MainProcess] calling terminate() for daemon Process-1
[INFO/MainProcess] calling join() for process Process-1
[DEBUG/MainProcess] running the remaining "atexit" finalizers
Notice "calling terminate() for daemon" line.
Output (with --kill)
$ python kill-orphan.py --kill
[INFO/Process-1] child process calling self.run()
The log shows that if the parent is killed by a signal then "atexit" callback is not called (and ps shows that the child is alive in this case). See also Multiprocess Daemon Not Terminating on Parent Exit.