I'm using keras to make and test different types of Neural Nets and need data to compare them. I need data on the cpu and memory used during the training and testing. This is in python and as I looked around I found lot of suggestions for psutil. However everything I see seems to grab the current usage.
What is current usage? Like the amount of memory used at that specific moment? How do I use it to get the total cpu and memory used by the entire program, or at least the portion of the code where the NN is training and testing. Thanks for any help!
psutil is a good recommendation to collect that type of information. If you incorporate this code into your existing keras code, you can collect information about the cpu usage of your process at the time the cpu_times() method is called
import psutil
process = psutil.Process()
print(process.cpu_times())
The meaning of the value returned by cpu_times() is explained here. It is cumulative, so if you want to know how much CPU time your keras code used altogether, just run it before you exit the python script.
To get the memory usage information for your process, at the particular time you make the call to memory_info() you can run this on the same process object we declared before
print(process.memory_info())
The exact meaning of the cpu and memory results depend on what platform you're using. The memory info structure is explained here
A more comprehensive example shows how you could use the Advanced Python Scheduler to take cpu and memory measurements in the background as you run your keras training
import psutil
import time
import os
from apscheduler.schedulers.background import BackgroundScheduler
process = psutil.Process()
def get_info():
print(process.cpu_times(), process.memory_info())
if __name__ == '__main__':
scheduler = BackgroundScheduler()
scheduler.add_job(get_info, 'interval', seconds=3)
scheduler.start()
# run the code you want to measure here
# replace this nonsense loop
now = time.time()
finish = now + 60
while time.time() < finish:
print("Some progress message: {}".format(time.time()))
time.sleep(10)
Related
How could we clear up the GPU memory after finishing a deep learning model training with Jupyter notebook. The problem is, no matter what framework I am sticking to (tensorflow, pytorch) the memory stored in the GPU do not get released except I kill the process manually or kill the kernel and restart the Jupyter. Do you have any idea how we can possible get rid of this problem by automating the steps?
The only walkaround I found was to use threading. Executing the Training with a subprocess.
An example:
def Training(arguments):
....
....
return model
if __name__=='__main__':
Subprocess = Process(# The complete function defined above
target = Training,
# Pass the arguments defined in the complete function
# above.
# Note - The comma after the arguments : In order for
# Python
# to understand this is a tuple
args = (arguments, ))
# Starting the defined subprocess
Subprocess.start()
# Wait for the subprocess to get completed
Subprocess.join()
I'm running a python script that handles and processes data using Pandas functions inside an infinite loop. But the program seems to be leaking memory over time.
This is the graph produced by the memory-profiler package:
Sadly, I cannot identify the source of the increasing memory usage. To my knowledge, all data (pandas timeseries) are stored in the object Obj, and I track the memory usage of this object using the pandas function .memory_usage and the objsize function get_deep_size(). According to their output, the memory usage should be stable around 90-100 MB. Other than this, I don't see where memory can ramp up.
It may be useful to know that the python program is running inside a docker container.
Below is a simplified version of the script which should illuminate the basic working principle.
from datetime import datetime
from time import sleep
import objsize
from dateutil import relativedelta
def update_data(Obj, now_utctime):
# attaining the newest timeseries data
new_data = requests.get(host, start=Obj.data[0].index, end=now_utctime)
Obj.data.append(new_data)
# cut off data older than 1 day
Obj.data.truncate(before=now_utctime-relativedelta.relativedelta(days=1))
class ExampleClass():
def __init__(self):
now_utctime = datetime.utcnow()
data = requests.get(host, start=now_utctime-relativedelta.relativedelta(days=1), end=now_utctime)
Obj = ExampleClass()
while True:
update_data(Obj, datetime.utcnow())
logger.info(f"Average at {datetime.utcnow()} is at {Obj.data.mean()}")
logger.info(f"Stored timeseries memory usage at {Obj.data.memory_usage(deep=True)* 10 ** -6} MB")
logger.info(f"Stored Object memory usage at {objsize.get_deep_size(Obj) * 10 ** -6} MB")
time.sleep(60)
Any advice into where memory could ramp up, or how to further investigate, would be appreciated.
EDIT: Looking at the chart, it makes sense that there will be spikes before I truncate, but since the data ingress is steady I don't know why it wouldn't normalize, but remain at a higher point. Then there is this sudden drop after every 4th cycle, even though the process does not have another, broader cycle that could explain this ...
As suggested by moooeeeep, the increase of memory usage was related to a memory leak, the exact source of which remains to be identified. However, I was able to resolve the issue by manually calling the garbage collector after every loop, via gc.collect().
I have written a python program to detect faces of a video input (webcam) using Haar Cascade. I would like to know how much CPU, GPU and RAM are being utilized by this specific program and not the overall CPU, GPU, RAM usage.
I came across psutil package (https://pypi.org/project/psutil/) which enables to profile system resources being used but I could not manage to get system resources being used by a specific program. How can I achieve that?
I have also seen this How to get current CPU and RAM usage in Python? but its not what I want.
My python code as follows:
def main():
#Get current Process ID
pid = os.getpid()
#Get resources used by current process
p = psutil.Process(pid)
with p.oneshot():
cpu_percent = p.cpu_percent()
# Get Memory and GPU usage of this PID as well
You can get CPU/RAM metrics (not GPU) for a particular process given its PID:
import psutil
psutil.Process(1234).cpu_percent()
psutil.Process(1234).memory_info()
I'm working in Jupyter (Anaconda) with Python 2.7.
I'm trying to get an odeint function I wrote to run multiple times, however it takes an incredible amount of time.
While trying to figure out how to decrease the run time, I realized that when I ran it only took up about 12% of my CPU.
I operate off of an Intel Core i7-3740QM # 2.70GHz:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-3740QM+%40+2.70GHz&id=1481
So, I'm assuming this is because of Python's GIL causing my script to only run off of one core.
After some research on parallel processing in Python, I thought I found the answer by using this code:
import sys
import multiprocessing as mp
Altitude = np.array([[550],[500],[450],[400],[350],[300]])
if __name__ == "__main__":
processes = 4
p = mp.Pool(processes)
mp_solutions = p.map(calc, Altitude)
This doesn't seem to work though. Once I run it, Jupyter just becomes constantly busy. My first thought was that it was just a high computation level so it was taking a long time, but then I looked at my CPU usage and although there were multiple instances of Python processes, none of them were using any CPU.
I can't figure out what the reasoning for this is. I found this post as well and tried using their code but it simply did the same thing:
Multiple scipy.integrate.ode instances
Any help would be much appreciated.
Thanks!
I'm trying to get the percent of CPU usage for an external process in python. I've seen some other posts on this topic, but haven't helped me too much. When I run the following function I get values that aren't consistent with what I'm seeing in task manager. For example, if I'm monitoring a chrome process I get values that oscillate between 1 and 2, but task manager shows values oscillating between 25 and 30. Any suggestions? Thanks.
def monitor(pid):
cpu_table = []
p = psutil.Process(pid)
while p.is_running():
cpu_table.append(p.get_cpu_percent())
time.sleep(1)
return cpu_table
There are several chrome processes and you might be monitoring the wrong one
cpu_percent() "compares system CPU times elapsed since last call or module import". Pass the same interval that task manager uses (in case, it is not 1 second). Make sure to start both your monitor() function and the task manager at the same time.