I have gone through the python(3.7) documentation to understand the concept of multiprocessing. For Process object there are two methods terminate() and kill. Terminating a process is done using the SIGTERM, and killing the process uses the SIGKILL signal on Unix.
When we terminate the process and check is_alive() status of the process it gives False. but, when I use kill and check the status for the process it gives is_alive() status True. I don't know why it's giving True after killing the process if it is the same as the terminate process.
def test():
print("in test method")
time.sleep(3)
if __name__ == '__main__':
p1 = Process(target=test)
p1.start() # start process
# kill process after 1 sec
time.sleep(1)
p1.kill()
print(p1.is_alive()) # why process alive status true when kill process
time.sleep(3)
print(p1.is_alive())
Whether you use kill or terminate, these methods only start the termination of the process. You must then wait for the process to fully terminate by using join (if you try to use sleep, you are only guessing as to how long to sleep for the process to be fully terminated):
p1.kill()
p1.join() # wait for p1 to fully complete
print(p1.is_alive()) # prints False
I think the reason of this is that the time between p1.kill() and print(p1.is_alive()) functions is so short. So I think that, the interpreter is trying to print on the screen whether or not the process is still active before this kill command is finished.
For example if you put time.sleep(0.001) between p1.kill() and print(p1.is_alive()) functions, False value will be printed rather than True value.
import time
from multiprocessing import Process
def test():
print("in test method")
time.sleep(3)
if __name__ == '__main__':
p1 = Process(target=test)
p1.start() # start process
# kill process after 1 sec
time.sleep(1)
p1.kill()
time.sleep(0.001) # Wait for p1.kill() function is finished.
print(p1.is_alive()) # why process alive status true when kill process
time.sleep(3)
print(p1.is_alive())
Related
I'm doing a small project in python/tkinter and I have been looking for a way to check if a process has finished but "without waiting". I have tried with:
process = subprocess.Popen(command)
while process.poll() is None:
print('Running!')
print('Finished!')
or:
process = subprocess.Popen(command)
stdoutdata, stderrdata = process.communicate()
print('Finished!')
Both codes execute the command and print "Finished!" when the process ends, but the main program freezes (waiting) and that's what I want to avoid. I need the GUI to stay functional while the process is running and then run some code right after it finishes. Any help?
It's common that you use the Thread module for that purpose:
For example:
# import Thread
from threading import Thread
import time
# create a function that checks if the process has finished
process = True
def check():
while process:
print('Running')
time.sleep(1) # here you can wait as much as you want without freezing the program
else:
print('Finished')
# call the function with the use of Thread
Thread(target=check).start()
# or if you want to keep a reference to it
t = Thread(target=check)
# you might also want to set thread daemon to True so as the Thread ends when the program closes
t.deamon = True
t.start()
This way when you do process=False the program will end and the output will show 'Finished'
The answer might be right in front of me on the link below but I still don't understand. I'm sure after someone explains this to me, Darwin will be making a call to me.
The example is at this link here, although I've made some changes to try to experiment and help my understanding.
Here's the code:
import multiprocessing
import time
import sys
def daemon():
p = multiprocessing.current_process()
print 'Starting: ', p.name, p.pid
sys.stdout.flush()
time.sleep(2)
print 'Exiting: ', p.name, p.pid
sys.stdout.flush()
def non_daemon():
p = multiprocessing.current_process()
print 'Starting: ', p.name, p.pid
sys.stdout.flush()
time.sleep(6)
print 'Exiting: ', p.name, p.pid
sys.stdout.flush()
if __name__ == '__main__':
d = multiprocessing.Process(name='daemon', target=daemon)
d.daemon = True
n = multiprocessing.Process(name='non-daemon', target=non_daemon)
n.daemon = False
d.start()
time.sleep(1)
n.start()
# d.join()
And the output of the code is:
Starting: daemon 6173
Starting: non-daemon 6174
Exiting: non-daemon 6174
If the join() at the end is uncommented, then the output is:
Starting: daemon 6247
Starting: non-daemon 6248
Exiting: daemon 6247
Exiting: non-daemon 6248
I'm confused b/c the sleep of the daemon is 2 sec, whereas the non-daemon is 6 sec. Why doesn't it print out the "Exiting" message in the first case? The daemon should have woken up before the non-daemon and printed the message.
The explanation from the site is as such:
The output does not include the “Exiting” message from the daemon
process, since all of the non-daemon processes (including the main
program) exit before the daemon process wakes up from its 2 second
sleep.
but I changed it such that the daemon should have woken up before the non-daemon does. What am I missing here? Thanks in advance for your help.
EDIT: Forgot to mention I'm using python 2.7 but apparently this problem is also in python 3.x
This was a fun one to track down. The docs are somewhat misleading, in that they describe the non-daemon processes as if they are all equivalent; the existence of any non-daemon process means the process "family" is alive. But that's not how it's implemented. The parent process is "more equal" than others; multiprocessing registers an atexit handler that does the following:
for p in active_children():
if p.daemon:
info('calling terminate() for daemon %s', p.name)
p._popen.terminate()
for p in active_children():
info('calling join() for process %s', p.name)
p.join()
So when the main process finishes, it first terminates all daemon child processes, then joins all child processes to wait on non-daemon children and clean up resources from daemon children.
Because it performs cleanup in this order, a moment after your non-daemon Process starts, the main process begins cleanup and forcibly terminates the daemon Process.
Note that fixing this can be as simple as joining the non-daemon process manually, not just joining the daemon process (which defeats the whole point of a daemon completely); that prevents the atexit handler from being called, delaying the cleanup that would terminate the daemon child.
It's arguably a bug (one that seems to exist up through 3.5.1; I reproed myself), but whether it's a behavior bug or a docs bug is arguable.
I am using multiprocessing module's Process class to spawn multiple processes, those processes execute some script and then dies.What I wanted, a timeout to be applied on each process, so that a process would die if cant execute in time timeout. I am using join(timeout) on Process objects.
Since the join() function doesn;t kill the process, it just blocks the process until it finishes
Now my question : Is there any side-effects of using join() with timeout ..like, would the processes be cleaned automatically, after the main process dies ?? or I have to kill those processes manually ??
I am a newbie to python and its multiprocessing module, please be patient.
My Code, which is creating Processes in a for loop ::
q = Queue()
jobs = [
Process(
target=get_current_value,
args=(q,),
kwargs=
{
'device': device,
'service_list': service_list,
'data_source_list': data_source_list
}
) for device in device_list
]
for j in jobs:
j.start()
for k in jobs:
k.join()
The timeout argument just tells join how long to wait for the Process to exit before giving up. If timeout expires, the Process does not exit; the join call simply unblocks. If you want to end your workers when the timeout expires, you need to do so manually. You can either use terminate, as suggested by wRAR, to uncleanly shut things down, or use some other signaling mechanism to tell the children to shutdown cleanly:
p = Process(target=worker, args=(queue,))
p.start()
p.join(50)
if p.isalive(): # join timed out without the process actually finishing
#p.terminate() # unclean shutdown
If you don't want to use terminate, the alternative approach is really dependent on what the workers are doing. If they're consuming from a queue, you can use a sentinel:
def worker(queue):
for item in iter(queue.get, None): # None will break the loop
# Do normal work
if __name__ == "__main__":
queue = multiprocessing.Queue()
p = multiprocessing.Process(target=worker, args=(queue,))
p.start()
# Do normal work here
# Time to shut down
queue.put(None)
Or you could use an Event, if you're doing some other operation in a loop:
def worker(event):
while not event.is_set():
# Do work here
if __name__ == "__main__":
event= multiprocessing.Event()
p = multiprocessing.Process(target=worker, args=(event,))
p.start()
# Do normal work here
# Time to shut down
event.set()
Using terminate could be just fine, though, unless your child processes are using resources that could be corrupted if the process is unexpectedly shut down (like writing to a file or db, or holding a lock). If you're just doing some calculations in the worker, using terminate won't hurt anything.
join() does nothing with the child process. If you really want to terminate worker process in a non-clean manner you should use terminate() (you should understand the consequences).
If you want children to be terminated when the main process exits you should set daemon attribute on them.
I am reading The Python Standard Library by Example and get confused when I arrived page 509.
Up to this point, the example programs have implicitly waited to exit until all threads have completed their work. Programs sometimes spawn a thread as a daemon that runs without blocking the main program from exiting.
but after I run some codes, I get result that is opposite. The code is like this:
#!/usr/bin/env python
# encoding: utf-8
#
# Copyright (c) 2008 Doug Hellmann All rights reserved.
#
"""Creating and waiting for a thread.
"""
#end_pymotw_header
import threading
import time
def worker():
"""thread worker function"""
print 'Worker'
# time.sleep(10000)
return
threads = []
for i in range(5):
t = threading.Thread(target=worker)
threads.append(t)
t.start()
print "main Exit"
and sometime the result is this:
Worker
Worker
WorkerWorker
main Exit
Worker
So I want to ask when will main thread exit in python after it starts several thread?
The main thread will exit whenever it is finished executing all the code in your script that is not started in a separate thread.
Since the t.start() will start the thread and then return to the main thread, the main thread will simply continue to execute until it reaches the bottom of your script and then exit.
Since you started the other threads in a non-daemon mode, they will continue running until they are finished.
If you wanted the main thread to not exit until all the threads have finished, you should explicitly join them to the main thread. The join will cause the thread calling join to wait until the the thread being joined is finished.
for i in range(5):
threads[i].join()
print "main Exit"
As #codesparkle pointed out, the more Pythonic way to write this would be to skip the index variable entirely.
for thread in threads:
thread.join()
print "main Exit"
According to the threading docs:
The entire Python program exits when only daemon threads are left
This agrees with the quote you give, but the slight difference in wording shows the result you get. The 'main thread' exits when you would expect it to. Note that the worker threads keep running at this point - you can see this in the test output you give. So, the main thread has finished, but the whole process is still running because there are other threads still running.
The difference is that if some of those worker threads were daemonised, they would be forcibly killed when the last non-daemon thread finished. If all of the workers were daemon threads, then the entire process would finish - and you would be back at your systems shell prompt - very soon after you print 'main exit', and it would be very rare (though not impossible, owing to race conditions) for any worker to print after that.
Your main thread will exit as soos as for loop completes its execution. Your main thread is starting new asynchronous threads. Which means that it will not wait untill new thread finishes its execution. So in your case main thread will start 5 threads in parallel and exit itself.
Note that Main does not exit when you print main Exit, but after it. Consider this program:
import threading
import time
def worker():
"""thread worker function"""
print 'Worker'
time.sleep(1)
print 'Done'
return
class Exiter(object):
def __init__(self):
self.a = 5.0
print 'I am alive'
def __del__(self):
print 'I am dying'
exiter = Exiter()
threads = []
for i in range(5):
t = threading.Thread(target=worker)
threads.append(t)
t.start()
print "main Exit"
I have created an object whose sole purpose is to print "I am dying" when it is being finalised. I am not deleting it explicitly anywhere, so it will only die when the main thread finishes, that is, when Python starts to kill everything to return memory to the OS.
If you run this you will see that workers are working when the main thread is done, but the objects are still alive. I am dying always comes after all the workers have finished their job.
For example as follow:
class ThreadA(Thread):
def __init__(self, mt):
Thread.__init__(self)
self.mt = mt
def run(self):
print 'T1: sleeping...'
time.sleep(4)
print 'current thread is ', self.isAlive()
print 'main thread is ', self.mt.isAlive()
print 'T1: raising...'
if __name__ == '__main__':
mt = threading.currentThread()
ta = ThreadA(mt)
ta.start()
logging.debug('main end')
>
T1: sleeping...
(MainThread) main end
current thread is True
main thread is False
T1: raising...
you can see the main thread active state is false?
I am creating a Python program that calls an external command periodically. The external command takes a few
seconds to complete. I want to reduce the possibility of the external command terminating
badly by adding a signal handler for SIGINT. Basically, I want SIGINT to attempt to wait until the command
executes before terminating the Python program. The problem is that the external perogram seems to be
getting the SIGINT as well, causing it to end abruptly. I am invoking the command using an external thread, since
the Python documentation for signal mentions that only the main thread receives the signal, according to http://docs.python.org/2/library/signal.html.
Can someone help with this.
Here is a stub of my code. Imagine that the external program is /bin/sleep:
import sys
import time
import threading
import signal
def sleep():
import subprocess
global sleeping
cmd = ['/bin/sleep', '10000']
sleeping = True
p = subprocess.Popen(cmd)
p.wait()
sleeping = False
def sigint_handler(signum, frame):
if sleeping:
print 'busy, will terminate shortly'
while(sleeping): time.sleep(0.5)
sys.exit(0)
else:
print 'clean exit'
sys.exit(0)
sleeping = False
signal.signal(signal.SIGINT, sigint_handler)
while(1):
t1 = threading.Thread(target=sleep)
t1.start()
time.sleep(500)
The expected behavior is that pressing Ctrl+C N seconds after the program starts will result in
it waiting (10000 - N) seconds and then exiting. What is happening is the program immediately terminates.
Thanks!
The problem is the way signal handlers are modified when executing a new process. From POSIX:
A child created via fork(2) inherits a copy of its parent's signal dis‐
positions. During an execve(2), the dispositions of handled signals
are reset to the default; the dispositions of ignored signals are left
unchanged.
So what you need to do is:
Ignore the SIGINT signal
Start the external program
Set the SIGINT handler as desired
That way, the external program will ignore SIGINT.
Of course, this leaves a (very) small time window when your script won't respond to SIGINT. But that's something you'll have to live with.
For example:
sleeping = False
while(1):
t1 = threading.Thread(target=sleep)
signal.signal(signal.SIGINT, signal.SIG_IGN)
t1.start()
signal.signal(signal.SIGINT, sigint_handler)
time.sleep(500)