We Keep Coding

Why use os.setsid() in Python?

I know os.setsid() is to change the process(forked) group id to itself, but why we need it?
I can see some answer from Google is:
To keep the child process running while the parent process exit.
But according to my test below, without os.setsid() the child process won't exit as well even if the parent process exit(or being killed). So why we need to add os.setsid()? Thanks.
import os
import time
import sys
mainPid = os.getpid()
print("Main Pid: %s" % mainPid)
pid = os.fork()
if pid > 0:
time.sleep(3)
print("Main process quit")
sys.exit(0)
#os.setsid()
for x in range(1, 10):
print("spid: %s, ppid: %s pgid: %s" % (os.getpid(), os.getppid(), os.getpgid(0)))
time.sleep(1)

Calling setsid is usually one of the steps a process goes through when becoming a so called daemon process. (We are talking about Linux/Unix OS).
With setsid the association with the controlling terminal breaks. This means that the process will be NOT affected by a logout.
There are other way how to survive a logout, but the purpose of this 'daemonizing' process is to create a background process as independent from the outside world as possible.
That's why all inherited descriptors are closed; cwd is set to an appropriate directory, often the root directory; and the process leaves the session it was started from.
A double fork approach is generally recommended. At each fork the parent exits and the child continues. Actually nothing changes except the PID, but that's exactly what is needed here.
First fork before the setsid makes sure the process is not a process group leader. That is required for a succesfull setsid.
The second fork after the setsid makes sure that a new association with a controlling terminal won't be started merely by opening a terminal device.
NOTE: when a daemon process is started from systemd, the systemd can arrange everything described above so the process does not have to.

Well, double fork to daemonize is a good example. However, It's better to understand what is process group and session.
Session ID (SID)
This is just the PID of the session leader. If PID == SID, then this process is a session leader.
Sessions and process groups are just ways to treat a number of related processes as a unit. All the members of a process group always belong to the same session, but a session may have multiple process groups.
Normally, a shell will be a session leader, and every pipeline executed by that shell will be a process group. This is to make it easy to kill the children of a shell when it exits. (See exit(3) for the gory details.)
Basically, if you log into a machine, your shell starts a session. If you want to keep your process running even when you log out, you should start a new session for the child.
The difference with double forked process is that you can still attach a control terminal to that process since it's a session leader, whereas the daemon process created by double fork can not be attached to the terminal anymore.

In some cases, the child process will be able to continue running even after the parent exits, but this is not foolproof. The child will also exit when the parent exits in some situations.
As of Python 3.2, you can use subprocess.Popen() and pass start_new_session=True to accomplish fully detach the child process from the parent.
The docs state:
If start_new_session is true the setsid() system call will be made in the child process prior to the execution of the subprocess. (POSIX only)
https://docs.python.org/3/library/subprocess.html#subprocess.Popen

how to kill subprocesses when parent exits in python?

code in fork_child.py
from subprocess import Popen
child = Popen(["ping", "google.com"], stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out, err = child.communicate()
I run it from a terminal window as -
$python fork_child.py
From another terminal window if I get the PID of fork_child.py and kill it with SIGTERM, "ping" doesn't get killed. How do I make sure that ping too gets killed when fork_child receives a SIGTERM ?

Children don't automatically die when the parent process is killed. They die if:
The parent forwards the signal and waits for the children to terminate
When the child tries to communicate with the parent, for example via stdio. That only works if the parent also created the file descriptors which the child uses.
The signals module contains examples how to write a signal handler.
So you need to:
collect all children in a list
install a signal handler
in the handler, iterate over all the child processes
For each child process, invoke child.terminate() followed by child.wait()
The wait() is necessary to allow the OS to garbage collect the child process. If you forget it, you may end up with zombie processes.

A simple way to kill the whole process tree in the shell is to kill its process group i.e., instead of kill $pid, run:
$ kill -TERM -$pid
Notice: the pid is negated.
Shell creates a new process group for each command (pipeline) therefore you won't kill innocent bystanders.
If descendant processes do not create their own independent process groups; they all die.
See Best way to kill all child processes.

Starting a child process with stdout/stderr redirected to a file [duplicate]

I'm wondering if this is the correct way to execute a system process and detach from parent, though allowing the parent to exit without creating a zombie and/or killing the child process. I'm currently using the subprocess module and doing this...
os.setsid()
os.umask(0)
p = subprocess.Popen(['nc', '-l', '8888'],
cwd=self.home,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
os.setsid() changes the process group, which I believe is what lets the process continue running when it's parent exits, as it no longer belongs to the same process group.
Is this correct and also is this a reliable way of performing this?
Basically, I have a remote control utility that communicate through sockets and allows to start processes remotely, but I have to ensure that if the remote control dies, the processes it started continue running unaffected.
I was reading about double-forks and not sure if this is necessary and/or subprocess.POpen close_fds somehow takes care of that and all that's needed is to change the process group?
Thanks.
Ilya

For Python 3.8.x, the process is a bit different. Use the start_new_session parameter available since Python 3.2:
import shlex
import subprocess
cmd = "<full filepath plus arguments of child process>"
cmds = shlex.split(cmd)
p = subprocess.Popen(cmds, start_new_session=True)
This will allow the parent process to exit while the child process continues to run. Not sure about zombies.
The start_new_session parameter is supported on all POSIX systems, i.e. Linux, MacOS, etc.
Tested on Python 3.8.1 on macOS 10.15.5

popen on Unix is done using fork. That means you'll be safe with:
you run Popen in your parent process
immediately exit the parent process
When the parent process exits, the child process is inherited by the init process (launchd on OSX) and will still run in the background.
The first two lines of your python program are not needed, this perfectly works:
import subprocess
p = subprocess.Popen(['nc', '-l', '8888'],
cwd="/",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
I was reading about double-forks and not sure if this is necessary
This would be needed if your parent process keeps running and you need to protect your children from dying with the parent. This answer shows how this can be done.
How the double-fork works:
create a child via os.fork()
in this child call Popen() which launches the long running process
exit child: Popen process is inherited by init and runs in the background
Why the parent has to immediately exit? What happens if it doesn't exit immediately?
If you leave the parent running and the user stops the process e.g. via ctrl-C (SIGINT) or ctrl-\ (SIGQUIT) then it would kill both the parent process and the Popen process.
What if it exits one second after forking?
Then, during this 1s period your Popen process is vulnerable to ctrl-c etc. If you need to be 100% sure then use the double forking.

What signal is passed to child process on terminating main program?

On a Windows 7 machine:-
I have a main (Python) program that I start on a command prompt [main process].
This program spawns a child (Python) program [child process].
I close the command prompt.
Result:-
Child process ends immediately.
On the other hand if I end the main program from task manager, I observe that the child process is still running.
I was wondering why the 2 approaches do not have the same results? Is it sending some different signal in the two cases?

Comments to the question pointed me to get the answer.
I was using subprocess.Popen(args) to spawn the child process. This would spawn the child process successfully but the child process would be launched in the same command window as its parent.
Going through subprocess Popen doc, I found some additional arguments to be passed in order to launch child process in another command window.
Launching the child with the following arguments solved my problem.
subprocess.Popen(args, shell=True, creationflags=subprocess.CREATE_NEW_CONSOLE)
The last argument subprocess.CREATE_NEW_CONSOLE is only for windows.

Python spawn off a child subprocess, detach, and exit

I'm wondering if this is the correct way to execute a system process and detach from parent, though allowing the parent to exit without creating a zombie and/or killing the child process. I'm currently using the subprocess module and doing this...
os.setsid()
os.umask(0)
p = subprocess.Popen(['nc', '-l', '8888'],
cwd=self.home,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
os.setsid() changes the process group, which I believe is what lets the process continue running when it's parent exits, as it no longer belongs to the same process group.
Is this correct and also is this a reliable way of performing this?
Basically, I have a remote control utility that communicate through sockets and allows to start processes remotely, but I have to ensure that if the remote control dies, the processes it started continue running unaffected.
I was reading about double-forks and not sure if this is necessary and/or subprocess.POpen close_fds somehow takes care of that and all that's needed is to change the process group?
Thanks.
Ilya

For Python 3.8.x, the process is a bit different. Use the start_new_session parameter available since Python 3.2:
import shlex
import subprocess
cmd = "<full filepath plus arguments of child process>"
cmds = shlex.split(cmd)
p = subprocess.Popen(cmds, start_new_session=True)
This will allow the parent process to exit while the child process continues to run. Not sure about zombies.
The start_new_session parameter is supported on all POSIX systems, i.e. Linux, MacOS, etc.
Tested on Python 3.8.1 on macOS 10.15.5

popen on Unix is done using fork. That means you'll be safe with:
you run Popen in your parent process
immediately exit the parent process
When the parent process exits, the child process is inherited by the init process (launchd on OSX) and will still run in the background.
The first two lines of your python program are not needed, this perfectly works:
import subprocess
p = subprocess.Popen(['nc', '-l', '8888'],
cwd="/",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
I was reading about double-forks and not sure if this is necessary
This would be needed if your parent process keeps running and you need to protect your children from dying with the parent. This answer shows how this can be done.
How the double-fork works:
create a child via os.fork()
in this child call Popen() which launches the long running process
exit child: Popen process is inherited by init and runs in the background
Why the parent has to immediately exit? What happens if it doesn't exit immediately?
If you leave the parent running and the user stops the process e.g. via ctrl-C (SIGINT) or ctrl-\ (SIGQUIT) then it would kill both the parent process and the Popen process.
What if it exits one second after forking?
Then, during this 1s period your Popen process is vulnerable to ctrl-c etc. If you need to be 100% sure then use the double forking.