Apparently I can't get the process resources usage in Mac OS X with psutil after the process got reaped, i.e. after p.wait() where p is a psutil.Popen() instance. So for example, if I try ps.cpu_times().system where ps is a psutil.Process() instance, I get a raise of no such process. What are the other options for measuring the resources usage in a mac (elapsed time, memory and cpu usage)?
If the process "dies" or gets reaped how are you supposed to interact with it? Of course you can't, 'cause it's gone. If on the other hand the process is a zombie, then in that case you might be able to extract some info off of it, like the parent PID, but not CPU or memory stats.
Related
I'm running a script like
def test0():
start = time()
for i in range(int(1e8)):
i += 1
print(time() - start)
When I run this on my machine which has 4 CPUs I get the following trace for CPU usage using the Ubuntu 20.04 system monitor.
In the image you can see I ran two experiments separated by some time. But in each experiment, the activity of 2 of the CPUs peaks. Why?
This seems normal to me. The process that is running your python code is not, at least by default, pinned to a specific core. This means that the process can be switched between different cores, which is what is happening in this case. Those spikes are not simultaneous, it indicates that the process was switched from one core to another.
On Linux, you can observe this using
watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p 172810
where 172810 is PID of the python process (which you can get, for example, from the output of top)
If you want to pin the process to a particular core, you can use psutils in your code.
import psutil
p = psutil.Process()
p.cpu_affinity([0]) # pinning the process to cpu 0
Now, you should see only one core spiking. (but avoid doing this if you don't have a good reason for it).
I would like to try to make a process, memory and network resource tracer similar to the one that comes by default in ubuntu for any operating system. But being new in python I don't know how to get these values to be displayed (in principle by console, then I'll do them as graphics). Which library would be easier to do it with?
On linux, you could leverage the /proc filesystem to read the information you need for such a task.
the /proc filesystem is a window into the kernel with a lot of data on each process running. It is displayed as a virtual filesystem, meaning you can access all that info simply by reading and parsing files.
For instance,
from pathlib import Path
proc = Path('/proc')
for proc in proc.iterdir():
if not proc.name.isnumeric():
continue # ignore directories that aren't processes
pid = proc.name
cmdline = (proc / 'cmdline').read_text()
print(f'PROCESS : {pid} : {cmdline}')
This will list all processes running, along with their commandline.
You have a lot of information you can gather in there.
more info on /proc here
I spent a lot of time searching for the answer for my question, but I could not find it.
I run xdm-open, using subrocess, to play a video (I do not want to know what applications are available)
I'm waiting for a while
I want to kill the video
import os
import subprocess
import psutil
from time import sleep;
from signal import SIGTERM
file = "test.mkv"
out = subprocess.Popen(['xdg-open', file])
pid = out.pid
print('sleeping...')
sleep(20.0)
print('end of sleep...')
os.kill(pid, SIGTERM) #alternatively: out.terminate()
Unfortunatelly the last line is killing only the xdg-open process. The mplayer process (which was started by xdg) is still running.
I tried to get the sub-processes of the xdg by using the following code:
main_process = psutil.Process(pid)
children_processes = main_process.children(recursive=True)
for child in children_processes:
print("child process: ", child.pid, child.name())
but it did not help either. The list was empty.
Has anybody an idea how to kill the player process?
Programs like xdg-open typically look for a suitable program to open a file with, start that program with the file as argument and then exit.
By the time you get around to checking for child processes, xdg-open has probably already exited.
What happens then is OS dependant. In what follows, I'll be talking about UNIX-like operating systems.
The processes launched by xdg-open will usually get PID 1 as their parent process id (PPID) after xdg-open exits, so it will be practically impossible to find out for certain who started them by looking at the PPID.
But, there will probably be a relatively small number of processes running under your user-ID with PPID 1, so if you list those before and after calling xdg-open and remove all the programs that were in the before-list from the after-list, the program you seek will be in the after-list. Unless your machine is very busy, chances are that there will be only one item in the after-list; the one started by xdg-open.
Edit 1:
You commented:
I want to make the app OS independent.
All operating systems that support xdg-open are basically UNIX-like operating systems. If you use the psutil Python module to get process information, you can run your "app" on all the systems that psutil supports:
Linux
macOS
FreeBSD, OpenBSD, NetBSD
Sun Solaris
AIX
(psutil even works on ms-windows, but I kind of doubt you will find xdg-open there...)
I'm trying to share objects between the parent and child process in Python. To play around with the idea, I've created a simple Python script:
from multiprocessing import Process
from os import getpid
import psutil
shared = list(range(20000000))
def shared_printer():
mem = psutil.Process(getpid()).memory_info().rss / (1024 ** 2)
print(getpid(), len(shared), '{}MB'.format(mem))
if __name__ == '__main__':
p = Process(target=shared_printer)
p.start()
shared_printer()
p.join()
The code snippet uses the excellent psutil library to print the RSS (Resident Set Size). When I run this on OSX with Python 2.7.15, I get the following output:
(33101, 20000000, '1MB')
(33100, 20000000, '626MB')
When I run the exact same snippet on Ubuntu (Linux 4.15.0-1029-aws #30-Ubuntu SMP x86_64 GNU/Linux), I get the following output:
(4077, 20000000, '632MB')
(4078, 20000000, '629MB')
Notice that the child process' RSS is basicall 0MB on OSX and about the same size as the parent process' RSS in Linux. I had assumed that copy-on-write behavior would work the same way in Linux and allow the child process to refer to the parent process' memory for most pages (perhaps except the one storing the head of the object).
So I'm guessing that there's some difference in the copy-on-write behavior in the 2 systems. My question is: is there anything I can do in Linux to get that OSX-like copy-on-write behavior?
So I'm guessing that there's some difference in the copy-on-write behavior >in the 2 systems. My question is: is there anything I can do in Linux to >get that OSX-like copy-on-write behavior?
The answer is NO. Behind the command psutil.Process(getpid()).memory_info().rss / (1024 ** 2) the OS uses the UNIX command $top [PID] and search for the field RES. Which contains the non-swapped physical memory a task has used in kb. i.e. RES = CODE + DATA.
IMHO, these means that both OS uses different memory managers. So that, it's almost impossible to constrain how much memory a process uses/needs. This is a intern issue of the OS.
In Linux the child process has the same size of the parent process. Indeed, they copy the same stack, code and data. But different PCB (Process Control Block). Therefore, it is impossible to get close to 0 as OSX does. It smells that OSX does not literally copy the code and data. If they are the same code, it will make pointer to the data of the parent process.
PD: I hope that would help you!
I have written a python program to detect faces of a video input (webcam) using Haar Cascade. I would like to know how much CPU, GPU and RAM are being utilized by this specific program and not the overall CPU, GPU, RAM usage.
I came across psutil package (https://pypi.org/project/psutil/) which enables to profile system resources being used but I could not manage to get system resources being used by a specific program. How can I achieve that?
I have also seen this How to get current CPU and RAM usage in Python? but its not what I want.
My python code as follows:
def main():
#Get current Process ID
pid = os.getpid()
#Get resources used by current process
p = psutil.Process(pid)
with p.oneshot():
cpu_percent = p.cpu_percent()
# Get Memory and GPU usage of this PID as well
You can get CPU/RAM metrics (not GPU) for a particular process given its PID:
import psutil
psutil.Process(1234).cpu_percent()
psutil.Process(1234).memory_info()