We Keep Coding

How can I start a subprocess in Python that runs independently and continues running if the main process is closed?

I have a small Flask API that is receiving requests from a remote server. Whenever a request is received, a subprocess is started. This subprocess is simply executing a second Python file that is in the same folder. This subprocess can run for several hours and several of these subprocesses can run simultaneously. I am using stdout to write the output of the python file into a text file.
All of this is working fine, but every couple of weeks it happens that the Flask API becomes unresponsive and needs to be restarted. As soon as I stop the Flask server, all running subprocesses stop. I would like to avoid this and run each subprocess independently from the flask API.
This is a small example that illustrates what I am doing (this code is basically inside a method that can be called through the API)
import subprocess
f = open("log.txt","wb")
subprocess.Popen(["python","job.py"],cwd = "./", stdout = f, stderr = f)
I would like to achieve that the subprocess keeps running after I stop the Flask API. This is currently not the case. Somewhere else I read that the reason is that I am using the stdout and stderr parameters, but even after removing those the behavior stays the same.
Any help would be appreciated.

Your sub-processes stop because their parent process dies when you restart your Flask server. You need to completely separate your sub-processes from your Flask process by running your Python call in a new shell:
from subprocess import call
# On Linux:
command = 'gnome-terminal -x bash -l -c "python job.py"'
# On Windows:
# command = 'cmd /c "python job.py"'
call(command, shell=True)
This way your Python call of job.py will run in a separate terminal window, unaffected by your Flask server process.

Use fork() to create a child process of the process in which you are calling this function. On successful fork(), it returns a zero for the child id.
Below is a basic example of fork, which you can easily incorporate in your code.
import os
pid = os.fork()
if pid == 0: # new process
os.system("nohup python ./job.py &")
Hope this helps!

Python kill parent script (subprocess)

Im working on two scripts.
One script is perpetually running.
When it senses an update to itself, it will run the second script as a subprocess.
The second script should kill the first script, implement the changes and run the updated script.
However, I cant find a way to kill the first script. How does the child process kill its parent?

You are doing this backwards, and shouldn't be using the child process to kill the parent process.
Instead, you will want a parent process of your "perpetually running" script (which will now be the subprocess). When an update is detected, the subprocess kills itself, and requests that the parent implement your changes. The parent will then restart the subprocess.

Popen new process group on linux

I am spawning some processes with Popen (Python 2.7, with Shell=True) and then sending SIGINT to them. It appears that the process group leader is actually the Python process, so sending SIGINT to the PID returned by Popen, which is the PID of bash, doesn't do anything.
So, is there a way to make Popen create a new process group? I can see that there is a flag called subprocess.CREATE_NEW_PROCESS_GROUP, but it is only for Windows.
I'm actually upgrading some legacy scripts which were running with Python2.6 and it seems for Python2.6 the default behavior is what I want (i.e. a new process group when I do Popen).

bash does not handle signals while waiting for your foreground child process to complete. This is why sending it SIGINT does not do anything. This behaviour has nothing to do with process groups.
There are a couple of options to let your child process receive your SIGINT:
When spawning a new process with Shell=True try prepending exec to the front of your command line, so that bash gets replaced with your child process.
When spawning a new process with Shell=True append the command line with & wait %-. This will cause bash to react to signals while waiting for your child process to complete. But it won't forward the signal to your child process.
Use Shell=False and specify full paths to your child executables.

python-twisted and SIGKILL

I have a python application that uses twisted framework.
I make use of value stored in the pidfile generated by twistd. A launcher script checks for it's presence and will not spawn a daemon process if the pidfile already exists.
However, twistd does not remove the .pidfile when it gets SIGKILL signal. That makes the launcher script think that the daemon is already running.
I realize the proper way to stop the daemon would be to use SIGTERM signal, but the problem is that when user who started the daemon logs out, the daemon never gets a SIGTERM signal, so apparently it's killed with SIGKILL. That means once a user logs out, he will never be able to start the daemon again, because the pidfile still exists.
Is there any way I could make that file disappear in such situations?

From the signal(2) man page:
The signals SIGKILL and SIGSTOP cannot be caught or ignored.
So there is no way the process can run any cleanup code in response to that signal. Usually you only use SIGKILL to terminate a process that doesn't exit in response to SIGTERM (which can be caught).

You could change your launcher (or wrap it up in another launcher) and remove the pid file before trying to restart twistd.

Kill Windows asynchronous Popen process in 2.4

I have a testing script that needs to open a process (a Pyro server), do some stuff that will call the opened process for information, and when it's all done will need to close the process back down. It's all part of an automated test on a staging server.
In python 2.6 you can do this:
pyro_server = subprocess.Popen(['python', 'pyro_server.py'])
# Do stuff, making remote calls to the Pyro server on occasion
pyro_server.terminate()
Alas, I'm locked into python 2.4 here at work so I don't have access to that function. And if I just let the script end of course the server lives on. What should I be doing to close/kill that process before the script exits?

Consider copying subprocess.py to your python2.4 dist-packages directory. It should just work as it's a simple wrapper around the old popen library.
The Popen object terminate function does nothing more than the following:
import os, signal
os.kill(pid, signal.SIGKILL)
pid is the child process's process id. signal.SIGKILL is the number 9, and is a standard unix kill signal. You can see how to spawn a subprocess and get its pid in python 2.4 with the popen module here:

#BrainCore: Note that os.kill is not available on windows in python24, check the docs.
My solution for killing a subprocess.Popen object on windows when using python24 is this:
import os, subprocess
p = subprocess.Popen(...)
res = os.system('taskkill /PID %d /F' % p.pid)