It seems that in the task manager all I get is the process of the python/pythonwin. So How can I figure out which python script is running?
The usual answer to such questions is Process Explorer. You can see the full command line for any instance of python.exe or pythonw.exe in the tooltip.
To get the same information in Python, you can use the psutil module.
import psutil
pythons = [[" ".join(p.cmdline), p.pid] for p in psutil.process_iter()
if p.name.lower() in ("python.exe", "pythonw.exe")]
The result, pythons, is a list of lists representing Python processes. The first item of each list is the command line that started the process, including any options. The second item is the process ID.
The psutil Process class has a lot of other stuff in it so if you want all that, you can do this instead:
pythons = [p for p in psutil.process_iter() if p.name.lower() in ("python.exe", "pythonw.exe")]
Now, on my system, iterating all processes with psutil.process_iter() takes several seconds, which seems to me ludicrous. The below is significantly faster, as it does the process filtering before Python sees it, but it relies on the wmic command line tool, which not all versions of Windows have (XP Home lacks it, notably). The result here is the same as the first psutil version (a list of lists, each containing the command line and process ID for one Python process).
import subprocess
wmic_cmd = """wmic process where "name='python.exe' or name='pythonw.exe'" get commandline,processid"""
wmic_prc = subprocess.Popen(wmic_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
wmic_out, wmic_err = wmic_prc.communicate()
pythons = [item.rsplit(None, 1) for item in wmic_out.splitlines() if item][1:]
pythons = [[cmdline, int(pid)] for [cmdline, pid] in pythons]
If wmic is not available, you will get an empty list []. Since you know there's at least one Python process (yours!), you can trap this as an error and display an appropriate message.
To get your own process ID, so you can exclude it from consideration if you're going to e.g. start killing processes, try pywin32's win32process.GetCurrentProcessID()
I had some issues with kindall's answer. With python 3.8:
import psutil
for p in psutil.process_iter():
try:
if p.name().lower() in ["python.exe", "pythonw.exe"]:
print(p.pid, p.cmdline)
except:
continue
With Python 3:
import psutil
pythons = [[" ".join(p.cmdline()), p.pid] for p in psutil.process_iter()
if p.name().lower() in ["python.exe", "pythonw.exe"]]
Related
I'm trying to port a shell script to the much more readable python version. The original shell script starts several processes (utilities, monitors, etc.) in the background with "&". How can I achieve the same effect in python? I'd like these processes not to die when the python scripts complete. I am sure it's related to the concept of a daemon somehow, but I couldn't find how to do this easily.
While jkp's solution works, the newer way of doing things (and the way the documentation recommends) is to use the subprocess module. For simple commands its equivalent, but it offers more options if you want to do something complicated.
Example for your case:
import subprocess
subprocess.Popen(["rm","-r","some.file"])
This will run rm -r some.file in the background. Note that calling .communicate() on the object returned from Popen will block until it completes, so don't do that if you want it to run in the background:
import subprocess
ls_output=subprocess.Popen(["sleep", "30"])
ls_output.communicate() # Will block for 30 seconds
See the documentation here.
Also, a point of clarification: "Background" as you use it here is purely a shell concept; technically, what you mean is that you want to spawn a process without blocking while you wait for it to complete. However, I've used "background" here to refer to shell-background-like behavior.
Note: This answer is less current than it was when posted in 2009. Using the subprocess module shown in other answers is now recommended in the docs
(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)
If you want your process to start in the background you can either use system() and call it in the same way your shell script did, or you can spawn it:
import os
os.spawnl(os.P_DETACH, 'some_long_running_command')
(or, alternatively, you may try the less portable os.P_NOWAIT flag).
See the documentation here.
You probably want the answer to "How to call an external command in Python".
The simplest approach is to use the os.system function, e.g.:
import os
os.system("some_command &")
Basically, whatever you pass to the system function will be executed the same as if you'd passed it to the shell in a script.
I found this here:
On windows (win xp), the parent process will not finish until the longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.
The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen([sys.executable, "longtask.py"],
creationflags=DETACHED_PROCESS).pid
Use subprocess.Popen() with the close_fds=True parameter, which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits.
https://gist.github.com/yinjimmy/d6ad0742d03d54518e9f
import os, time, sys, subprocess
if len(sys.argv) == 2:
time.sleep(5)
print 'track end'
if sys.platform == 'darwin':
subprocess.Popen(['say', 'hello'])
else:
print 'main begin'
subprocess.Popen(['python', os.path.realpath(__file__), '0'], close_fds=True)
print 'main end'
Both capture output and run on background with threading
As mentioned on this answer, if you capture the output with stdout= and then try to read(), then the process blocks.
However, there are cases where you need this. For example, I wanted to launch two processes that talk over a port between them, and save their stdout to a log file and stdout.
The threading module allows us to do that.
First, have a look at how to do the output redirection part alone in this question: Python Popen: Write to stdout AND log file simultaneously
Then:
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
import threading
def output_reader(proc, file):
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
file.buffer.write(byte)
else:
break
with subprocess.Popen(['./sleep.py', '0'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc1, \
subprocess.Popen(['./sleep.py', '10'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc2, \
open('log1.log', 'w') as file1, \
open('log2.log', 'w') as file2:
t1 = threading.Thread(target=output_reader, args=(proc1, file1))
t2 = threading.Thread(target=output_reader, args=(proc2, file2))
t1.start()
t2.start()
t1.join()
t2.join()
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(4):
print(i + int(sys.argv[1]))
sys.stdout.flush()
time.sleep(0.5)
After running:
./main.py
stdout get updated every 0.5 seconds for every two lines to contain:
0
10
1
11
2
12
3
13
and each log file contains the respective log for a given process.
Inspired by: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
Tested on Ubuntu 18.04, Python 3.6.7.
You probably want to start investigating the os module for forking different threads (by opening an interactive session and issuing help(os)). The relevant functions are fork and any of the exec ones. To give you an idea on how to start, put something like this in a function that performs the fork (the function needs to take a list or tuple 'args' as an argument that contains the program's name and its parameters; you may also want to define stdin, out and err for the new thread):
try:
pid = os.fork()
except OSError, e:
## some debug output
sys.exit(1)
if pid == 0:
## eventually use os.putenv(..) to set environment variables
## os.execv strips of args[0] for the arguments
os.execv(args[0], args)
You can use
import os
pid = os.fork()
if pid == 0:
Continue to other code ...
This will make the python process run in background.
I haven't tried this yet but using .pyw files instead of .py files should help. pyw files dosen't have a console so in theory it should not appear and work like a background process.
I currently run:
ps -f -u vinish | sort -k5
but how can I do this in Python without writing the same quoted command in my code?
For executing bash commands I use os.system('<cmd>') or will call through subprocess, but I am looking something that won't hold any bash commands in it.
You can just sort the output from subprocess.check_output():
import subprocess
processes = subprocess.check_output(['ps', '-f', '-u', 'vinish'])
for line in sorted(processes.splitlines(), key=lambda l: int(l.split()[1])):
print(line)
This goes one better on the sort -k5 output in that it extracts the second column and sorts it numerically.
Note that no sub-shell is created (no bash commands are being executed here); Python executes the ps command directly. This is the easiest way to get process information without installing third-party libraries.
If you really wanted to avoid executing external commands, you'll have to use an add-on library. I recommend the psutil library here:
import psutil
matching = [proc for proc in psutil.process_iter() if proc.username() == 'vinish']
for proc in sorted(matching, key=lambda p: p.pid):
print(proc)
See the psutil.Process() class documentation for details on what each proc object offers in the way of information and functionality.
I am trying to get the memory usage of an external program within my python script. I have tried using the script http://code.activestate.com/recipes/286222/ as follows:
m0 = memory()
subprocess.call('My program')
m1 = memory(m0)
print m1
But this seems to be just giving me the memory usage of the python script rather than 'My program'. Is there a way of outputting the memory usage of the program for use within the python script?
Try using Psutil
import psutil
import subprocess
import time
SLICE_IN_SECONDS = 1
p = subprocess.Popen('calling/your/program')
resultTable = []
while p.poll() == None:
resultTable.append(psutil.get_memory_info(p.pid))
time.sleep(SLICE_IN_SECONDS)
If you look at the recipe you will see the line:
_proc_status = '/proc/%d/status' % os.getpid()
I suggest you replace the os.getpid() with the process id of your child process. As #Neal said, as I was typing this you need to use Popen and get the pid attribute of the returned object.
However, you have a possible race condition because you don't know at what state the child process is at, and the memory usage will vary anyway.
You may want to check out the psutil module: http://code.google.com/p/psutil/. The Process Management section on the homepage gives you examples of getting memory usage for a running process specified by the pid.
Do you want to spawn the process you are monitoring in your script as well? If so, you probably don't want to use subprocess.call as this will wait for the program to exit and you won't be able to monitor it while it's running. If you want to spawn the process then monitor it, you probably want to use Popen http://docs.python.org/library/subprocess.html#subprocess.Popen. This will allow you to spawn the process, get the pid, hand the pid to psutil, then monitor the memory usage.
I know this is an older post, but it's the only one that appears when I google this issue, so, I want to add the updated version of this:
import psutil
import humanfriendly
proc = subprocess.Popen("...Your process...")
SLICE_IN_SECONDS = 1
while proc.poll() is None:
p = psutil.Process(proc.pid)
mem_status = "RSS {}, VMS: {}".format(humanfriendly.format_size(p.memory_info().rss),
humanfriendly.format_size(p.memory_info().vms))
time.sleep(SLICE_IN_SECONDS)
print(mem_status)
I used humanfriendly here, to make the values more readable, but it's not required.
The RSS and VMS values are on all os, and there may be other values depending on the os you're using: https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_info
I am in Windows and Suppose I have a main python code that calls python interpreter in command line to execute another python script ,say test.py .
So test.py is executed as a new process.How can I find the processId for this porcess in Python ?
Update:
To be more specific , we have os.getpid() in os module. It returns the current process id.
If I have a main program that runs Python interpreter to run another script , how can I get the process Id for that executing script ?
If you used subprocess to spawn the shell, you can find the process ID in the pid property:
sp = subprocess.Popen(['python', 'script.py'])
print('PID is ' + str(sp.pid))
If you used multiprocessing, use its pid property:
p = multiprocessing.Process()
p.start()
# Some time later ...
print('PID is ' + str(p.pid))
It all depends on how you're launching the second process.
If you're using os.system or similar, that call won't report back anything useful about the child process's pid. One option is to have your 2nd script communicate the result of os.getpid() back to the original process via stdin/stdout, or write it to a predetermined file location. Another alternative is to use the third-party psutil library to figure out which process it is.
On the other hand, if you're using the subprocess module to launch the script, the resulting "popen" object has an attribute popen.pid which will give you the process id.
You will receive the process ID of the newly created process when you create it. At least, you will if you used fork() (Unix), posix_spawn(), CreateProcess() (Win32) or probably any other reasonable mechanism to create it.
If you invoke the "python" binary, the python PID will be the PID of this binary that you invoke. It's not going to create another subprocess for itself (Unless your python code does that).
Another option is that the process you execute will set a console window title for himself.
And the searching process will enumerate all windows, find the relevant window handle by name and use the handle to find PID. It works on windows using ctypes.
Several processes with the same name are running on host. What is the cross-platform way to get PIDs of those processes by name using python or jython?
I want something like pidof but in python. (I don't have pidof anyway.)
I can't parse /proc because it might be unavailable (on HP-UX).
I do not want to run os.popen('ps') and parse the output because I think it is ugly (field sequence may be different in different OS).
Target platforms are Solaris, HP-UX, and maybe others.
You can use psutil (https://github.com/giampaolo/psutil), which works on Windows and UNIX:
import psutil
PROCNAME = "python.exe"
for proc in psutil.process_iter():
if proc.name() == PROCNAME:
print(proc)
On my machine it prints:
<psutil.Process(pid=3881, name='python.exe') at 140192133873040>
EDIT 2017-04-27 - here's a more advanced utility function which checks the name against processes' name(), cmdline() and exe():
import os
import psutil
def find_procs_by_name(name):
"Return a list of processes matching 'name'."
assert name, name
ls = []
for p in psutil.process_iter():
name_, exe, cmdline = "", "", []
try:
name_ = p.name()
cmdline = p.cmdline()
exe = p.exe()
except (psutil.AccessDenied, psutil.ZombieProcess):
pass
except psutil.NoSuchProcess:
continue
if name == name_ or cmdline[0] == name or os.path.basename(exe) == name:
ls.append(p)
return ls
There's no single cross-platform API, you'll have to check for OS. For posix based use /proc. For Windows use following code to get list of all pids with coresponding process names
from win32com.client import GetObject
WMI = GetObject('winmgmts:')
processes = WMI.InstancesOf('Win32_Process')
process_list = [(p.Properties_("ProcessID").Value, p.Properties_("Name").Value) for p in processes]
You can then easily filter out processes you need.
For more info on available properties of Win32_Process check out Win32_Process Class
import psutil
process = filter(lambda p: p.name() == "YourProcess.exe", psutil.process_iter())
for i in process:
print i.name,i.pid
Give all pids of "YourProcess.exe"
A note on ThorSummoner's comment
process = [proc for proc in psutil.process_iter() if proc.name == "YourProcess.exe"].
I have tried it on Debian with Python 3, I think it has to be proc.name() instead of proc.name.
First, Windows (in all it's incarnations) is a non-standard OS.
Linux (and most proprietary unixen) are POSIX-compliant standard operating systems.
The C libraries reflect this dichotomy. Python reflects the C libraries.
There is no "cross-platform" way to do this. You have to hack up something with ctypes for a particular release of Windows (XP or Vista)
I don't think you will be able to find a purely python-based, portable solution without using /proc or command line utilities, at least not in python itself. Parsing os.system is not ugly - someone has to deal with the multiple platforms, be it you or someone else. Implementing it for the OS you are interested in should be fairly easy, honestly.
There isn't, I'm afraid. Processes are uniquely identified by pid not by name. If you really must find a pid by name, then you will have use something like you have suggested, but it won't be portable and probably will not work in all cases.
If you only have to find the pids for a certain application and you have control over this application, then I'd suggest changing this app to store its pid in files in some location where your script can find it.
For jython, if Java 5 is used, then you can get the Java process id as following:
from java.lang.management import *
pid = ManagementFactory.getRuntimeMXBean().getName()
Just use:
def get_process_by_name(name):
import re, psutil
ls = list()
for p in psutil.process_iter():
if hasattr(p, 'name'):
if re.match(".*" + name + ".*", p.name()):
ls.append(p)
return ls
returns Process object