I need to gain the pid of a program in order to suspend it temporarily. How can I gain the pid of a program using python with just its username in order to use this pid along with psutil to suspend the process? Let's just call it process.exe for now.
I have tried using item for item along with psuti. However, this gives me additional text along with the pid and I am unsure how to remove this unnecesary text.
I have tried using os.getpid but this gives me the pid of Python rather than the process I want to get the pid of.
1.
import psutil
pid = [item for item in psutil.process_iter() if item.name() == 'process.exe']
print(pid)
2.
import os
pid = os.getpid()
print(pid)
For (1) I want the output to just be
pid=x
However, right now it is:
[psutil.Process(pid=x, name='process.exe', started='14:11:40')]
In 1, you are receiving process object which contains all the information about process, you can use the process object to fetch the pid:
pid[0].pid
In 2nd ,
os.getpid()
returns the process id of python interpreter and is of hardly any use under python interpreter, only use it when you are running some python script to get its process id.
I don't know if what I intend to do is even possible or reasonable, so I'm open for any suggestions.
Currently I have a script which starts n subprocesses of some.exe id which I regularly poll() to determine if they terminated and if so, with which errorlevel. The subprocesses are saved in a dict (key = subprocess, value = id). n instances of some.exe are kept running, each with an own id, until every id of a predefined list has been processed.
some.exe has no gui on it's onw, it writes the progress to the stdout, which I do not need. Now, for some reason sometimes, some.exe doesn't continue, as if it would wait - thus poll() never produces an errorlevel and done = proc.poll() is not None is never true. Which sooner or later leads to my dict of n procs, all being inactive and the overall progress is stuck.
If issued manually in a cmd, some.exe with an id, that shows this behaviour in the script - works perfectly fine.
Therefore my idea was to start a new cmd window from the script, which runs some.exe with the id, but I should still be able to poll() said exe.
Here's roughly what I have so far:
while id_list > 0:
if len(proc_dict) < n:
id = next_id()
proc = subprocess.Popen(["some.exe", id], stdout=PIPE)
proc.poll()
proc_dict[proc] = id
else:
done_procs = []
for proc in proc_dict.keys():
done = proc.poll() is not None
if done:
print("returncode: "+proc.returncode)
done_procs.append(proc)
if done_procs:
for p in done_procs:
del proc_dict[p]
time.sleep(2)
edit: if I proc.communicate()[0] in the else: where the sleep is located, some.exe is able to continue/finish, but as communicate waits for the process, it slows down the script way too much.
I believe the problem is that some.exe's output is enough to fill the os pipe buffer, causing a deadlock. There is a warning about this in the docs here
If you want to discard the stdout, instead of sending it to a pipe you could instead send it to devnull, this post explains how to do that
The child process is started with
subprocess.Popen(arg)
Is there a way to ensure it is killed when parent terminates abnormally? I need this to work both on Windows and Linux. I am aware of this solution for Linux.
Edit:
the requirement of starting a child process with subprocess.Popen(arg) can be relaxed, if a solution exists using a different method of starting a process.
Heh, I was just researching this myself yesterday! Assuming you can't alter the child program:
On Linux, prctl(PR_SET_PDEATHSIG, ...) is probably the only reliable choice. (If it's absolutely necessary that the child process be killed, then you might want to set the death signal to SIGKILL instead of SIGTERM; the code you linked to uses SIGTERM, but the child does have the option of ignoring SIGTERM if it wants to.)
On Windows, the most reliable options is to use a Job object. The idea is that you create a "Job" (a kind of container for processes), then you place the child process into the Job, and you set the magic option that says "when no-one holds a 'handle' for this Job, then kill the processes that are in it". By default, the only 'handle' to the job is the one that your parent process holds, and when the parent process dies, the OS will go through and close all its handles, and then notice that this means there are no open handles for the Job. So then it kills the child, as requested. (If you have multiple child processes, you can assign them all to the same job.) This answer has sample code for doing this, using the win32api module. That code uses CreateProcess to launch the child, instead of subprocess.Popen. The reason is that they need to get a "process handle" for the spawned child, and CreateProcess returns this by default. If you'd rather use subprocess.Popen, then here's an (untested) copy of the code from that answer, that uses subprocess.Popen and OpenProcess instead of CreateProcess:
import subprocess
import win32api
import win32con
import win32job
hJob = win32job.CreateJobObject(None, "")
extended_info = win32job.QueryInformationJobObject(hJob, win32job.JobObjectExtendedLimitInformation)
extended_info['BasicLimitInformation']['LimitFlags'] = win32job.JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE
win32job.SetInformationJobObject(hJob, win32job.JobObjectExtendedLimitInformation, extended_info)
child = subprocess.Popen(...)
# Convert process id to process handle:
perms = win32con.PROCESS_TERMINATE | win32con.PROCESS_SET_QUOTA
hProcess = win32api.OpenProcess(perms, False, child.pid)
win32job.AssignProcessToJobObject(hJob, hProcess)
Technically, there's a tiny race condition here in case the child dies in between the Popen and OpenProcess calls, you can decide whether you want to worry about that.
One downside to using a job object is that when running on Vista or Win7, if your program is launched from the Windows shell (i.e., by clicking on an icon), then there will probably already be a job object assigned and trying to create a new job object will fail. Win8 fixes this (by allowing job objects to be nested), or if your program is run from the command line then it should be fine.
If you can modify the child (e.g., like when using multiprocessing), then probably the best option is to somehow pass the parent's PID to the child (e.g. as a command line argument, or in the args= argument to multiprocessing.Process), and then:
On POSIX: Spawn a thread in the child that just calls os.getppid() occasionally, and if the return value ever stops matching the pid passed in from the parent, then call os._exit(). (This approach is portable to all Unixes, including OS X, while the prctl trick is Linux-specific.)
On Windows: Spawn a thread in the child that uses OpenProcess and os.waitpid. Example using ctypes:
from ctypes import WinDLL, WinError
from ctypes.wintypes import DWORD, BOOL, HANDLE
# Magic value from http://msdn.microsoft.com/en-us/library/ms684880.aspx
SYNCHRONIZE = 0x00100000
kernel32 = WinDLL("kernel32.dll")
kernel32.OpenProcess.argtypes = (DWORD, BOOL, DWORD)
kernel32.OpenProcess.restype = HANDLE
parent_handle = kernel32.OpenProcess(SYNCHRONIZE, False, parent_pid)
# Block until parent exits
os.waitpid(parent_handle, 0)
os._exit(0)
This avoids any of the possible issues with job objects that I mentioned.
If you want to be really, really sure, then you can combine all these solutions.
Hope that helps!
The Popen object offers the terminate and kill methods.
https://docs.python.org/2/library/subprocess.html#subprocess.Popen.terminate
These send the SIGTERM and SIGKILL signals for you.
You can do something akin to the below:
from subprocess import Popen
p = None
try:
p = Popen(arg)
# some code here
except Exception as ex:
print 'Parent program has exited with the below error:\n{0}'.format(ex)
if p:
p.terminate()
UPDATE:
You are correct--the above code will not protect against hard-crashing or someone killing your process. In that case you can try wrapping the child process in a class and employ a polling model to watch the parent process.
Be aware psutil is non-standard.
import os
import psutil
from multiprocessing import Process
from time import sleep
class MyProcessAbstraction(object):
def __init__(self, parent_pid, command):
"""
#type parent_pid: int
#type command: str
"""
self._child = None
self._cmd = command
self._parent = psutil.Process(pid=parent_pid)
def run_child(self):
"""
Start a child process by running self._cmd.
Wait until the parent process (self._parent) has died, then kill the
child.
"""
print '---- Running command: "%s" ----' % self._cmd
self._child = psutil.Popen(self._cmd)
try:
while self._parent.status == psutil.STATUS_RUNNING:
sleep(1)
except psutil.NoSuchProcess:
pass
finally:
print '---- Terminating child PID %s ----' % self._child.pid
self._child.terminate()
if __name__ == "__main__":
parent = os.getpid()
child = MyProcessAbstraction(parent, 'ping -t localhost')
child_proc = Process(target=child.run_child)
child_proc.daemon = True
child_proc.start()
print '---- Try killing PID: %s ----' % parent
while True:
sleep(1)
In this example I run 'ping -t localhost' b/c that will run forever. If you kill the parent process, the child process (the ping command) will also be killed.
Since, from what I can tell, the PR_SET_PDEATHSIG solution can result in a deadlock when any threads are running in the parent process, I didn't want to use that and figured out another way. I created a separate auto-terminate process that detects when its parent process is done and kills the other subprocess that is its target.
To accomplish this, you need to pip install psutil, and then write code similar to the following:
def start_auto_cleanup_subprocess(target_pid):
cleanup_script = f"""
import os
import psutil
import signal
from time import sleep
try:
# Block until stdin is closed which means the parent process
# has terminated.
input()
except Exception:
# Should be an EOFError, but if any other exception happens,
# assume we should respond in the same way.
pass
if not psutil.pid_exists({target_pid}):
# Target process has already exited, so nothing to do.
exit()
os.kill({target_pid}, signal.SIGTERM)
for count in range(10):
if not psutil.pid_exists({target_pid}):
# Target process no longer running.
exit()
sleep(1)
os.kill({target_pid}, signal.SIGKILL)
# Don't bother waiting to see if this works since if it doesn't,
# there is nothing else we can do.
"""
return Popen(
[
sys.executable, # Python executable
'-c', cleanup_script
],
stdin=subprocess.PIPE
)
This is similar to https://stackoverflow.com/a/23436111/396373 that I had failed to notice, but I think the way that I came up with is easier for me to use because the process that is the target of cleanup is created directly by the parent. Also note that it is not necessary to poll the status of the parent, though it is still necessary to use psutil and to poll the status of the target subprocess during the termination sequence if you want to try, as in this example, to terminate, monitor, and then kill if terminate didn't work expeditiously.
Hook exit of your process using SetConsoleCtrlHandler, and kill subprocess. I think I do a bit of a overkill there, but it works :)
import psutil, os
def kill_proc_tree(pid, including_parent=True):
parent = psutil.Process(pid)
children = parent.children(recursive=True)
for child in children:
child.kill()
gone, still_alive = psutil.wait_procs(children, timeout=5)
if including_parent:
parent.kill()
parent.wait(5)
def func(x):
print("killed")
if anotherproc:
kill_proc_tree(anotherproc.pid)
kill_proc_tree(os.getpid())
import win32api,shlex
win32api.SetConsoleCtrlHandler(func, True)
PROCESSTORUN="your process"
anotherproc=None
cmdline=f"/c start /wait \"{PROCESSTORUN}\" "
anotherproc=subprocess.Popen(executable='C:\\Windows\\system32\\cmd.EXE', args=shlex.split(cmdline,posix="false"))
...
run program
...
Took kill_proc_tree from:
subprocess: deleting child processes in Windows
I've written a simple wrapper script for repeating commands when they fail called retry.py. However as I want to see the output of child command I've had to pull some pty tricks. This works OK for programs like rsync but others like scp apply additional test for showing things like their progress meter.
The scp code has a test that is broadly:
getpgrp() == tcgetpgrp(STDOUT_FILENO);
Which fails when I run though the wrapper script. As you can see with my simple tty_test.c test case:
./tty_tests
isatty reports 1
pgrps are 13619 and 13619
and:
./retry.py -v -- ./tty_tests
command is ['./tty_tests']
isatty reports 1
pgrps are 13614 and -1
child finished: rc = 0
Ran command 1 times
I've tried using the tcsetpgrp() which ends up as an IOCTL on the pty fd's but that results in an -EINVAL for ptys. I'd prefer to keep using the Python subprocess machinery if at all possible or is manually fork/execve'ing going to be required for this?
I believe you can pare your program down to this, if you don't need to provide a whole new pty to the subprocess:
from argparse import ArgumentParser
import os
import signal
import subprocess
import itertools
# your argumentparser stuff goes here
def become_tty_fg():
os.setpgrp()
hdlr = signal.signal(signal.SIGTTOU, signal.SIG_IGN)
tty = os.open('/dev/tty', os.O_RDWR)
os.tcsetpgrp(tty, os.getpgrp())
signal.signal(signal.SIGTTOU, hdlr)
if __name__ == "__main__":
args = parser.parse_args()
if args.verbose: print "command is %s" % (args.command)
if args.invert and args.limit==None:
sys.exit("You must define a limit if you have inverted the return code test")
for run_count in itertools.count():
return_code = subprocess.call(args.command, close_fds=True,
preexec_fn=become_tty_fg)
if args.test == True: break
if run_count >= args.limit: break
if args.invert and return_code != 0: break
elif not args.invert and return_code == 0: break
print "Ran command %d times" % (run_count)
The setpgrp() call creates a new process group in the same session, so that the new process will receive any ctrl-c/ctrl-z/etc from the user, and your retry script won't. Then the tcsetpgrp() makes the new process group be the foreground one on the controlling tty. The new process gets a SIGTTOU when that happens (because since the setpgrp(), it has been in a background process group), which normally would make the process stop, so that's the reason for ignoring SIGTTOU. We set the SIGTTOU handler back to whatever it was before, to minimize the chance of the subprocess being confused by an unexpected signal table.
Since the subprocess is now in the foreground group for the tty, its tcgetpgrp() and getpgrp() will be the same, and isatty(1) will be true (assuming the stdout it inherits from retry.py actually is a tty). You don't need to proxy traffic between the subprocess and the tty, which lets you ditch all the select event handling and fcntl-nonblocking-setting.
I'm working on a nifty little function:
def startProcess(name, path):
"""
Starts a process in the background and writes a PID file
returns integer: pid
"""
# Check if the process is already running
status, pid = processStatus(name)
if status == RUNNING:
raise AlreadyStartedError(pid)
# Start process
process = subprocess.Popen(path + ' > /dev/null 2> /dev/null &', shell=True)
# Write PID file
pidfilename = os.path.join(PIDPATH, name + '.pid')
pidfile = open(pidfilename, 'w')
pidfile.write(str(process.pid))
pidfile.close()
return process.pid
The problem is that process.pid isn't the correct PID. It seems it's always 1 lower than the correct PID. For instance, it says the process started at 31729, but ps says it's running at 31730. Every time I've tried it's off by 1. I'm guessing the PID it returns is the PID of the current process, not the started one, and the new process gets the 'next' pid which is 1 higher. If this is the case, I can't just rely on returning process.pid + 1 since I have no guarantee that it'll always be correct.
Why doesn't process.pid return the PID of the new process, and how can I achieve the behaviour I'm after?
From the documentation at http://docs.python.org/library/subprocess.html:
Popen.pid The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
If shell is false, it should behave as you expect, I think.
If you were relying on shell being True for resolving executable paths using the PATH environment variable, you can accomplish the same thing using shutil.which instead, then pass the absolute path to Popen instead. (As an aside, if you are using Python 3.5 or newer, you should be using subprocess.run rather than Popen.