We Keep Coding

Executing 2 blocking processes and need the control back in the same ssh session

I'm trying to automate the start up procedure for a Linux Agent machine. I need to run two executable files located in two different directories and get the control back to the prompt so as to proceed with some other task like do a grep to check if the two processes are still running. Here are two different ways using Python in which I tried to do this:
Code snippet 1:(By forking a child process)
import os
import pdb
def child():
cwd = os.getcwd()
os.chdir("THoT")
os.chdir("TH_Node")
print "Executing TH_Node.exe........."
command = "mono TH_Node.exe"
os.system(command)
os._exit(0)
def parent():
i = 0
while i < 1:
i = i + 1
newpid = os.fork()
if newpid == 0:
child()
else:
cwd = os.getcwd()
os.chdir("THoT")
os.chdir("TH_Protocol")
print "Executing TH_Protocol.exe........."
command1 = "mono TH_Protocol.exe"
os.system(command1)
parent()
Code snippet 2:(Using multiprocessing)
import multiprocessing
import time
import sys
import os
import pdb
def TH_Protocol():
os.chdir("THoT")
os.chdir("TH_Protocol")
command = "mono TH_Protocol.exe"
os.system(command)
def TH_Node():
os.chdir("THoT")
os.chdir("TH_Node")
command1 = "mono TH_Node.exe"
os.system(command1)
if __name__ == '__main__':
d = multiprocessing.Process(name='TH_Protocol', target=TH_Protocol)
d.TH_Protocol = True
n = multiprocessing.Process(name='TH_Node', target=TH_Node)
n.TH_Protocol = False
d.start()
n.start()
d.join(1)
n.join()
The problem is although I get both the processes TH_Protocol.exe and TH_Node.exe to run, I need to ssh to another session to run a grep command to check if the two processes are running. I need to get the control back in the same session as the session in which I run my python script. I tried to use the subprocess.Popen as well, but I face the same problem. Is there any way I can solve this issue?

If you just want to run this script in the background, and get control of your ssh session back while it's running… that has nothing to do with Python, or ssh, it's basic shell job control.
For example, assuming your shell on the remote machine is sh/bash/similar, instead of this:
remote_machine$ python script.py
… do this:
remote_machine$ python script.py &
[1] 55341
remote_machine$
Now you've got the prompt back. You can interact with the main interpreter process as %1 or PID 55341. After it finally finishes, the next prompt you get will show something like this:
[1]+ Done python
You can't directly interact with the two child processes this way. You can always grep for them if you want, or search for child processes of PID 55341… but you might find your life easier if you had the child processes do something like print('TH_Protocol on {}'.format(os.getpid())) as soon as they start up, so you don't have to do that.

How to get memory usage of an external program - python

I am trying to get the memory usage of an external program within my python script. I have tried using the script http://code.activestate.com/recipes/286222/ as follows:
m0 = memory()
subprocess.call('My program')
m1 = memory(m0)
print m1
But this seems to be just giving me the memory usage of the python script rather than 'My program'. Is there a way of outputting the memory usage of the program for use within the python script?

Try using Psutil
import psutil
import subprocess
import time
SLICE_IN_SECONDS = 1
p = subprocess.Popen('calling/your/program')
resultTable = []
while p.poll() == None:
resultTable.append(psutil.get_memory_info(p.pid))
time.sleep(SLICE_IN_SECONDS)

If you look at the recipe you will see the line:
_proc_status = '/proc/%d/status' % os.getpid()
I suggest you replace the os.getpid() with the process id of your child process. As #Neal said, as I was typing this you need to use Popen and get the pid attribute of the returned object.
However, you have a possible race condition because you don't know at what state the child process is at, and the memory usage will vary anyway.

You may want to check out the psutil module: http://code.google.com/p/psutil/. The Process Management section on the homepage gives you examples of getting memory usage for a running process specified by the pid.
Do you want to spawn the process you are monitoring in your script as well? If so, you probably don't want to use subprocess.call as this will wait for the program to exit and you won't be able to monitor it while it's running. If you want to spawn the process then monitor it, you probably want to use Popen http://docs.python.org/library/subprocess.html#subprocess.Popen. This will allow you to spawn the process, get the pid, hand the pid to psutil, then monitor the memory usage.

I know this is an older post, but it's the only one that appears when I google this issue, so, I want to add the updated version of this:
import psutil
import humanfriendly
proc = subprocess.Popen("...Your process...")
SLICE_IN_SECONDS = 1
while proc.poll() is None:
p = psutil.Process(proc.pid)
mem_status = "RSS {}, VMS: {}".format(humanfriendly.format_size(p.memory_info().rss),
humanfriendly.format_size(p.memory_info().vms))
time.sleep(SLICE_IN_SECONDS)
print(mem_status)
I used humanfriendly here, to make the values more readable, but it's not required.
The RSS and VMS values are on all os, and there may be other values depending on the os you're using: https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_info

How do I know which python script is running in taskmgr?

It seems that in the task manager all I get is the process of the python/pythonwin. So How can I figure out which python script is running?

The usual answer to such questions is Process Explorer. You can see the full command line for any instance of python.exe or pythonw.exe in the tooltip.
To get the same information in Python, you can use the psutil module.
import psutil
pythons = [[" ".join(p.cmdline), p.pid] for p in psutil.process_iter()
if p.name.lower() in ("python.exe", "pythonw.exe")]
The result, pythons, is a list of lists representing Python processes. The first item of each list is the command line that started the process, including any options. The second item is the process ID.
The psutil Process class has a lot of other stuff in it so if you want all that, you can do this instead:
pythons = [p for p in psutil.process_iter() if p.name.lower() in ("python.exe", "pythonw.exe")]
Now, on my system, iterating all processes with psutil.process_iter() takes several seconds, which seems to me ludicrous. The below is significantly faster, as it does the process filtering before Python sees it, but it relies on the wmic command line tool, which not all versions of Windows have (XP Home lacks it, notably). The result here is the same as the first psutil version (a list of lists, each containing the command line and process ID for one Python process).
import subprocess
wmic_cmd = """wmic process where "name='python.exe' or name='pythonw.exe'" get commandline,processid"""
wmic_prc = subprocess.Popen(wmic_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
wmic_out, wmic_err = wmic_prc.communicate()
pythons = [item.rsplit(None, 1) for item in wmic_out.splitlines() if item][1:]
pythons = [[cmdline, int(pid)] for [cmdline, pid] in pythons]
If wmic is not available, you will get an empty list []. Since you know there's at least one Python process (yours!), you can trap this as an error and display an appropriate message.
To get your own process ID, so you can exclude it from consideration if you're going to e.g. start killing processes, try pywin32's win32process.GetCurrentProcessID()

I had some issues with kindall's answer. With python 3.8:
import psutil
for p in psutil.process_iter():
try:
if p.name().lower() in ["python.exe", "pythonw.exe"]:
print(p.pid, p.cmdline)
except:
continue

With Python 3:
import psutil
pythons = [[" ".join(p.cmdline()), p.pid] for p in psutil.process_iter()
if p.name().lower() in ["python.exe", "pythonw.exe"]]

getting ProcessId within Python code

I am in Windows and Suppose I have a main python code that calls python interpreter in command line to execute another python script ,say test.py .
So test.py is executed as a new process.How can I find the processId for this porcess in Python ?
Update:
To be more specific , we have os.getpid() in os module. It returns the current process id.
If I have a main program that runs Python interpreter to run another script , how can I get the process Id for that executing script ?

If you used subprocess to spawn the shell, you can find the process ID in the pid property:
sp = subprocess.Popen(['python', 'script.py'])
print('PID is ' + str(sp.pid))
If you used multiprocessing, use its pid property:
p = multiprocessing.Process()
p.start()
# Some time later ...
print('PID is ' + str(p.pid))

It all depends on how you're launching the second process.
If you're using os.system or similar, that call won't report back anything useful about the child process's pid. One option is to have your 2nd script communicate the result of os.getpid() back to the original process via stdin/stdout, or write it to a predetermined file location. Another alternative is to use the third-party psutil library to figure out which process it is.
On the other hand, if you're using the subprocess module to launch the script, the resulting "popen" object has an attribute popen.pid which will give you the process id.

You will receive the process ID of the newly created process when you create it. At least, you will if you used fork() (Unix), posix_spawn(), CreateProcess() (Win32) or probably any other reasonable mechanism to create it.
If you invoke the "python" binary, the python PID will be the PID of this binary that you invoke. It's not going to create another subprocess for itself (Unless your python code does that).

Another option is that the process you execute will set a console window title for himself.
And the searching process will enumerate all windows, find the relevant window handle by name and use the handle to find PID. It works on windows using ctypes.

Cross-platform way to get PIDs by process name in python

Several processes with the same name are running on host. What is the cross-platform way to get PIDs of those processes by name using python or jython?
I want something like pidof but in python. (I don't have pidof anyway.)
I can't parse /proc because it might be unavailable (on HP-UX).
I do not want to run os.popen('ps') and parse the output because I think it is ugly (field sequence may be different in different OS).
Target platforms are Solaris, HP-UX, and maybe others.

You can use psutil (https://github.com/giampaolo/psutil), which works on Windows and UNIX:
import psutil
PROCNAME = "python.exe"
for proc in psutil.process_iter():
if proc.name() == PROCNAME:
print(proc)
On my machine it prints:
<psutil.Process(pid=3881, name='python.exe') at 140192133873040>
EDIT 2017-04-27 - here's a more advanced utility function which checks the name against processes' name(), cmdline() and exe():
import os
import psutil
def find_procs_by_name(name):
"Return a list of processes matching 'name'."
assert name, name
ls = []
for p in psutil.process_iter():
name_, exe, cmdline = "", "", []
try:
name_ = p.name()
cmdline = p.cmdline()
exe = p.exe()
except (psutil.AccessDenied, psutil.ZombieProcess):
pass
except psutil.NoSuchProcess:
continue
if name == name_ or cmdline[0] == name or os.path.basename(exe) == name:
ls.append(p)
return ls

There's no single cross-platform API, you'll have to check for OS. For posix based use /proc. For Windows use following code to get list of all pids with coresponding process names
from win32com.client import GetObject
WMI = GetObject('winmgmts:')
processes = WMI.InstancesOf('Win32_Process')
process_list = [(p.Properties_("ProcessID").Value, p.Properties_("Name").Value) for p in processes]
You can then easily filter out processes you need.
For more info on available properties of Win32_Process check out Win32_Process Class

import psutil
process = filter(lambda p: p.name() == "YourProcess.exe", psutil.process_iter())
for i in process:
print i.name,i.pid
Give all pids of "YourProcess.exe"

A note on ThorSummoner's comment
process = [proc for proc in psutil.process_iter() if proc.name == "YourProcess.exe"].
I have tried it on Debian with Python 3, I think it has to be proc.name() instead of proc.name.

First, Windows (in all it's incarnations) is a non-standard OS.
Linux (and most proprietary unixen) are POSIX-compliant standard operating systems.
The C libraries reflect this dichotomy. Python reflects the C libraries.
There is no "cross-platform" way to do this. You have to hack up something with ctypes for a particular release of Windows (XP or Vista)

I don't think you will be able to find a purely python-based, portable solution without using /proc or command line utilities, at least not in python itself. Parsing os.system is not ugly - someone has to deal with the multiple platforms, be it you or someone else. Implementing it for the OS you are interested in should be fairly easy, honestly.

There isn't, I'm afraid. Processes are uniquely identified by pid not by name. If you really must find a pid by name, then you will have use something like you have suggested, but it won't be portable and probably will not work in all cases.
If you only have to find the pids for a certain application and you have control over this application, then I'd suggest changing this app to store its pid in files in some location where your script can find it.

For jython, if Java 5 is used, then you can get the Java process id as following:
from java.lang.management import *
pid = ManagementFactory.getRuntimeMXBean().getName()

Just use:
def get_process_by_name(name):
import re, psutil
ls = list()
for p in psutil.process_iter():
if hasattr(p, 'name'):
if re.match(".*" + name + ".*", p.name()):
ls.append(p)
return ls
returns Process object