/proc/stat shows ticks for user, nice, sys, idle, iowait, irq and sirq like this:
cpu 6214713 286 1216407 121074379 260283 253506 197368 0 0 0
How can I calculate the individual utilizations (in %) for user, nice etc with these values? Like the values that shows in 'top' or 'vmstat'.
This code calculates user utilization spread over all cores.
import os
import time
import multiprocessing
def main():
jiffy = os.sysconf(os.sysconf_names['SC_CLK_TCK'])
num_cpu = multiprocessing.cpu_count()
stat_fd = open('/proc/stat')
stat_buf = stat_fd.readlines()[0].split()
user, nice, sys, idle, iowait, irq, sirq = ( float(stat_buf[1]), float(stat_buf[2]),
float(stat_buf[3]), float(stat_buf[4]),
float(stat_buf[5]), float(stat_buf[6]),
float(stat_buf[7]) )
stat_fd.close()
time.sleep(1)
stat_fd = open('/proc/stat')
stat_buf = stat_fd.readlines()[0].split()
user_n, nice_n, sys_n, idle_n, iowait_n, irq_n, sirq_n = ( float(stat_buf[1]), float(stat_buf[2]),.
float(stat_buf[3]), float(stat_buf[4]),
float(stat_buf[5]), float(stat_buf[6]),
float(stat_buf[7]) )
stat_fd.close()
print ((user_n - user) * 100 / jiffy) / num_cpu
if __name__ == '__main__':
main()
From Documentation/filesystems/proc.txt:
(...) These numbers identify the amount of time the CPU has spent performing
different kinds of work. Time units are in USER_HZ (typically hundredths of a second).
So to figure out utilization in terms of percentages you need to:
Find out what USER_HZ is on the machine
Find out how long it's been since the system booted.
The second one is easy: there is a btime line in that same file which you can use for that. For USER_HZ, check out How to get number of mili seconds per jiffy.
Related
This is a question for the astronomy-minded folks on here.
I am an amateur astrophotographer looking to develop a personal script to aid my photography of next year's total solar eclipse. I am developing a Python script to automate my photography, so that I may enjoy the eclipse with my own eyes while my DSLR clicks away. Here's the script I've developed so far. The script uses digicamcontrol to control the camera.
Right now in the script I have just develop automation based on the partial phase of the eclipse (first contact, C1) and the timing of the eclipse in UTC (as well as my own PC). But a thought occurred to me: What if I can't connect to the internet and get the exact timing of the solar eclipse based on my location? I'd like to be able to generate those times. Is there a more efficient method to utilize astropy for this task? Thanks in advance.
import digiCamControlPython as dccp
import time
from datetime import datetime
from astropy.time import Time
local_time = Time.now()
utc_time_now = local_time.utc
def PartialEclipse(start_time:str, end_time:str):
camera = dccp.Camera()
camera.setIso(100)
camera.setShutterspeed("1/50")
camera.setFolder(r"C:\Users\My_Name\Pictures\digiCamControl")
# Set the target capture time in astropy time format
partial_eclipse_start = Time(start_time, format='isot')
partial_eclipse_end = Time(end_time, format='isot')
# Wait until the capture time
while utc_time_now < partial_eclipse_start:
time.sleep(1)
# Start capturing images
while utc_time_now < partial_eclipse_end:
camera.capture()
time.sleep(30) # Capture an image every 30 seconds
PartialEclipse("2024-04-08T17:12:13", "2024-04-08T18:29:24") #times of partial eclipse start and T-15s before totality
EDIT: In the event anyone ever looks at this question, I did make some progress on this.
import numpy as np
import astropy.units as u
from astropy.coordinates import solar_system_ephemeris, AltAz, EarthLocation, SkyCoord
from astropy.coordinates import get_body, get_moon, get_sun
from astropy.time import Time
myLocation = EarthLocation(lat=26*u.deg, lon=-80*u.deg, height=0*u.m)
# set the time step (how often to check for a solar eclipse in seconds)
time_step = 3600 # 1 hour
# set the number of days to check for a solar eclipse
num_days = 365
# set the start and end times to check for a solar eclipse
start_time = Time.now()
end_time = start_time + num_days * u.day
# initialize a list to store the times of a solar eclipse
eclipse_times = []
# loop over the desired time range, checking for a solar eclipse every time_step seconds
with solar_system_ephemeris.set('jpl'):
for t in np.arange(start_time.unix, end_time.unix, time_step):
time = Time(t, format='unix')
moon = get_body('moon', time, myLocation)
sun = get_body('sun', time, myLocation)
sun_coord = SkyCoord(sun.ra, sun.dec, sun.distance, frame='icrs')
moon_coord = SkyCoord(moon.ra, moon.dec, moon.distance, frame='icrs')
# check if the angular separation between the moon and sun is close to zero
angular_separation = moon_coord.separation(sun_coord)
if angular_separation < 0.6 * u.deg: #elongation where the partial eclipse begins
eclipse_times.append(time)
# print the times of the next solar eclipse
if len(eclipse_times) > 0:
print("The next solar eclipse is at: ", eclipse_times[0].iso)
else:
print("No solar eclipses found in the specified time range.")
I think your approach is really good! If you want to increase the accuracy of your start time prediction without using a lot more computational power, you can use scipy.optimize.root_scalar to refine the start time you found.
In my solution below, I've defined a function called distance_contact() whose root represents the start of the eclipse. This function is zero if the Sun and Moon are barely touching, positive if they are separated, and negative if they are overlapping. Then I define a grid of times with a timestep of 1 hour similar to your code, and pass it into this function to search for eclipses. It then finds the first time where distance_contanct is negative and uses that time and the timestep before as the search space for scipy.optimize.root_scalar.
Also, instead of using 0.6 * u.deg as the separation distance for an eclipse to occur, I've calculated the angular radius of the Sun and Moon for the time argument to distance_contact to make the prediction as accurate as possible.
import numpy as np
import scipy.optimize
import astropy.units as u
import astropy.time
import astropy.constants
import astropy.coordinates
def distance_contact(
location: astropy.coordinates.EarthLocation,
time: astropy.time.Time,
eclipse_type: str,
) -> u.Quantity:
radius_sun = astropy.constants.R_sun
radius_moon = 1737.4 * u.km
coordinate_sun = astropy.coordinates.get_sun(time)
coordinate_moon = astropy.coordinates.get_moon(time)
frame_local = astropy.coordinates.AltAz(obstime=time, location=location)
alt_az_sun = coordinate_sun.transform_to(frame_local)
alt_az_moon = coordinate_moon.transform_to(frame_local)
angular_radius_sun = np.arctan2(radius_sun, alt_az_sun.distance).to(u.deg)
angular_radius_moon = np.arctan2(radius_moon, alt_az_moon.distance).to(u.deg)
if eclipse_type == 'total':
separation_max = angular_radius_moon - angular_radius_sun
elif eclipse_type == 'partial':
separation_max = angular_radius_moon + angular_radius_sun
else:
raise ValueError("Unknown eclipse type")
return (alt_az_moon.separation(alt_az_sun).deg * u.deg) - separation_max
def calc_time_start(
location: astropy.coordinates.EarthLocation,
time_search_start: astropy.time.Time,
time_search_stop: astropy.time.Time,
eclipse_type: str = 'partial'
) -> astropy.time.Time:
astropy.coordinates.solar_system_ephemeris.set("de430")
# If we're only looking for a partial eclipse, we can accept a coarser search grid
if eclipse_type == "partial":
step = 1 * u.hr
elif eclipse_type == "total":
step = 1 * u.min
else:
raise ValueError("Unknown eclipse type")
# Define a grid of times to search for eclipses
time = astropy.time.Time(np.arange(time_search_start, time_search_stop, step=step))
# Find the times that are during an eclipse
mask_eclipse = distance_contact(location=location, time=time, eclipse_type=eclipse_type) < 0
# Find the index of the first time that an eclipse is occuring
index_start = np.argmax(mask_eclipse)
# Search around that time to find when the eclipse actually starts
time_eclipse_start = scipy.optimize.root_scalar(
f=lambda t: distance_contact(location, astropy.time.Time(t, format="unix"), eclipse_type=eclipse_type).value,
bracket=[time[index_start - 1].unix, time[index_start].unix],
).root
time_eclipse_start = astropy.time.Time(time_eclipse_start, format="unix")
return time_eclipse_start
def test_calc_time_start():
location = astropy.coordinates.EarthLocation(lat=26 * u.deg, lon=-80 * u.deg, height=0 * u.m)
eclipse_type = 'partial'
time_start = calc_time_start(
location=location,
time_search_start=astropy.time.Time.now(),
time_search_stop=astropy.time.Time.now() + 0.9 * u.yr,
eclipse_type=eclipse_type,
)
print(time_start.isot)
which outputs:
2023-10-14T15:57:38.068
I am trying to run a job on an HPC using multiprocessing. Each process has a peak memory usage of ~44GB. The job class I can use allows 1-16 nodes to be used, each with 32 CPUs and a memory of 124GB. Therefore if I want to run the code as quickly as possible (and within the max walltime limit) I should be able to run 2 CPUs on each node up to a maximum of 32 across all 16 nodes. However, when I specify mp.Pool(32) the job quickly exceeds the memory limit, I assume because more than two CPUs were used on a node.
My natural instinct was to specify 2 CPUs as the maximum in the pbs script I run my python script from, however this configuration is not permitted on the system. Would really appreciate any insight, having been scratching my head on this one for most of today - and have faced and worked around similar problems in the past without addressing the fundamentals at play here.
Simplified versions of both scripts below:
#!/bin/sh
#PBS -l select=16:ncpus=32:mem=124gb
#PBS -l walltime=24:00:00
module load anaconda3/personal
source activate py_env
python directory/script.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import numpy as np
import pandas as pd
import multiprocessing as mp
def df_function(df, arr1, arr2):
df['col3'] = some_algorithm(df, arr1, arr2)
return df
def parallelize_dataframe(df, func, num_cores):
df_split = np.array_split(df, num_cores)
with mp.Pool(num_cores, maxtasksperchild = 10 ** 3) as pool:
df = pd.concat(pool.map(func, df_split))
return df
def main():
# Loading input data
direc = '/home/dir1/dir2/'
file = 'input_data.csv'
a_file = 'array_a.npy'
b_file = 'array_b.npy'
df = pd.read_csv(direc + file)
a = np.load(direc + a_file)
b = np.load(direc + b_file)
# Globally defining function with keyword defaults
global f
def f(df):
return df_function(df, arr1 = a, arr2 = b)
num_cores = 32 # i.e. 2 per node if evenly distributed.
# Running the function as a multiprocess:
df = parallelize_dataframe(df, f, num_cores)
# Saving:
df.to_csv(direc + 'outfile.csv', index = False)
if __name__ == '__main__':
main()
To run your job as-is, you could simply request ncpu=32 and then in your python script set num_cores = 2. Obviously this has you paying for 32 cores and then leaving 30 of them idle, which is wasteful.
The real problem here is that your current algorithm is memory-bound, not CPU-bound. You should be going to great lengths to read only chunks of your files into memory, operating on the chunks, and then writing the result chunks to disk to be organized later.
Fortunately Dask is built to do exactly this kind of thing. As a first step, you can take out the parallelize_dataframe function and directly load and map your some_algorithm with a dask.dataframe and dask.array:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import dask.dataframe as dd
import dask.array as da
def main():
# Loading input data
direc = '/home/dir1/dir2/'
file = 'input_data.csv'
a_file = 'array_a.npy'
b_file = 'array_b.npy'
df = dd.read_csv(direc + file, blocksize=25e6)
a_and_b = da.from_np_stack(direc)
df['col3'] = df.apply(some_algorithm, args=(a_and_b,))
# dask is lazy, this is the only line that does any work
# Saving:
df.to_csv(
direc + 'outfile.csv',
index = False,
compute_kwargs={"scheduler": "threads"}, # also "processes", but try threads first
)
if __name__ == '__main__':
main()
That will require some tweaks to some_algorithm, and to_csv and from_np_stack work a bit differently, but you will be able to reasonably run this thing just on your own laptop and it will scale to your cluster hardware. You can level up from here by using the distributed scheduler or even deploy it directly to your cluster with dask-jobqueue.
I have been trying to exploit Numba to speed up large array calculations. I have been measuring the calculation speed in GFLOPS, and it consistently falls far short of my expectations for my CPU.
My processor is i9-9900k, which according to float32 benchmarks should be capable of over 200 GFLOPS. In my tests I have never exceeded about 50 GFLOPS. This is running on all 8 cores.
On a single core I achieve about 17 GFLOPS, which (I believe) is 50% of the theoretical performance. I'm not sure if this is improvable, but the fact that it doesn't extend well to multi-core is a problem.
I am trying to learn this because I am planning to write some image processing code that desperately needs every speed boost possible. I also feel I should understand this first, before I dip my toes into GPU computing.
Here is some example code with a few of my attempts at writing fast functions. The operation I am testing, is multiplying an array by a float32 then summing the whole array, i.e. a MAC operation.
How can I get better results?
import os
# os.environ["NUMBA_ENABLE_AVX"] = "1"
import numpy as np
import timeit
from timeit import default_timer as timer
import numba
# numba.config.NUMBA_ENABLE_AVX = 1
# numba.config.LOOP_VECTORIZE = 1
# numba.config.DUMP_ASSEMBLY = 1
from numba import float32, float64
from numba import jit, njit, prange
from numba import vectorize
from numba import cuda
lengthY = 16 # 2D array Y axis
lengthX = 2**16 # X axis
totalops = lengthY * lengthX * 2 # MAC operation has 2 operations
iters = 100
doParallel = True
#njit(fastmath=True, parallel=doParallel)
def MAC_numpy(testarray):
output = (float)(0.0)
multconst = (float)(.99)
output = np.sum(np.multiply(testarray, multconst))
return output
#njit(fastmath=True, parallel=doParallel)
def MAC_01(testarray):
lengthX = testarray.shape[1]
lengthY = testarray.shape[0]
output = (float)(0.0)
multconst = (float)(.99)
for y in prange(lengthY):
for x in prange(lengthX):
output += multconst*testarray[y,x]
return output
#njit(fastmath=True, parallel=doParallel)
def MAC_04(testarray):
lengthX = testarray.shape[1]
lengthY = testarray.shape[0]
output = (float)(0.0)
multconst = (float)(.99)
for y in prange(lengthY):
for x in prange(int(lengthX/4)):
xn = x*4
output += multconst*testarray[y,xn] + multconst*testarray[y,xn+1] + multconst*testarray[y,xn+2] + multconst*testarray[y,xn+3]
return output
# ======================================= TESTS =======================================
testarray = np.random.rand(lengthY, lengthX)
# ==== MAC_numpy ====
time = 1000
for n in range(iters):
start = timer()
output = MAC_numpy(testarray)
end = timer()
if((end-start) < time): #get shortest time
time = end-start
print("\nMAC_numpy")
print("output = %f" % (output))
print(type(output))
print("fastest time = %16.10f us" % (time*10**6))
print("Compute Rate = %f GFLOPS" % ((totalops/time)/10**9))
# ==== MAC_01 ====
time = 1000
lengthX = testarray.shape[1]
lengthY = testarray.shape[0]
for n in range(iters):
start = timer()
output = MAC_01(testarray)
end = timer()
if((end-start) < time): #get shortest time
time = end-start
print("\nMAC_01")
print("output = %f" % (output))
print(type(output))
print("fastest time = %16.10f us" % (time*10**6))
print("Compute Rate = %f GFLOPS" % ((totalops/time)/10**9))
# ==== MAC_04 ====
time = 1000
for n in range(iters):
start = timer()
output = MAC_04(testarray)
end = timer()
if((end-start) < time): #get shortest time
time = end-start
print("\nMAC_04")
print("output = %f" % (output))
print(type(output))
print("fastest time = %16.10f us" % (time*10**6))
print("Compute Rate = %f GFLOPS" % ((totalops/time)/10**9))
Q : How can I get better results?
1st : Learn how to avoid doing useless work - you can straight eliminate HALF of the FLOP-s not speaking about also the half of all the RAM-I/O-s avoided, each one being at a cost of +100~350 [ns] per writeback
Due to the distributive nature of MUL and ADD ( a.C + b.C ) == ( a + b ).C, better first np.sum( A ) and only after that then MUL the sum by the (float) constant.
#utput = np.sum(np.multiply(testarray, multconst)) # AWFULLY INEFFICIENT
output = np.sum( testarray)*multconst #######################
2nd : Learn how to best align data along the order of processing ( cache-line reuses get you ~100x faster re-use of pre-fetched data. Not aligning vectorised-code along these already pre-fetched data side-effects just let your code pay many times the RAM-access latencies, instead of smart re-using the already paid for data-blocks. Designing work-units aligned according to this principle means a few SLOCs more, but the rewards are worth that - who gets ~100x faster CPUs+RAMs for free and right now or about a ~100x speedup for free, just from not writing a badly or naively designed looping iterators?
3rd : Learn how to efficiently harness vectorised (block-directed) operations inside numpy or numba code-blocks and avoid pressing numba to spend time on auto-analysing the call-signatures ( you pay an extra time for this auto-analyses per call, while you have designed the code and knew exactly what data-types are going to go there, so why to pay an extra time for auto-analysis each time a numba-block gets called???)
4th : Learn where the extended Amdahl's Law, having all the relevant add-on costs and processing atomicity put into the game, supports your wish to get speedups, not to ever pay way more than you will get back (to at least justify the add-on costs... ) - paying extra costs for not getting any reward is possible, yet has no beneficial impact on your code's performance ( rather the opposite )
5th : Learn when and how the manually created inline(s) may save your code, once the steps 1-4 are well learnt and routinely excersised with proper craftmanship ( Using popular COTS frameworks is fine, yet these may deliver results after a few days of work, while a hand-crafted single purpose smart designed assembly code was able to get the same results in about 12 minutes(!), not several days without any GPU/CPU tricks etc - yes, that faster - just by not doing a single step more than what was needed for the numerical processing of the large matrix data )
Did I mention float32 may surprise at being processed slower on small scales than float64, while on larger data-scales ~ n [GB] the RAM I/O-times grow slower for more efficient float32 pre-fetches? This never happens here, as float64 array gets processed here. Sure, unless one explicitly instructs the constructor(s) to downconvert the default data type, like this: np.random.rand( lengthY, lengthX ).astype( dtype = np.float32 )>>> np.random.rand( 10, 2 ).dtypedtype('float64')Avoiding extensive memory allocations is another performance trick, supported in numpy call-signatures. Using this option for large arrays will save you a lot of extra time wasted on mem-allocs for large interim arrays. Reusing already pre-allocated memory-zones and wisely controlled gc-policing are another signs of a professional, focused on low-latency & design-for-performance
I ran SimpleSpeedTest.py from the PyCuda examples, producing the following output:
Using nbr_values == 8192
Calculating 100000 iterations
SourceModule time and first three results:
0.058294s, [ 0.005477 0.005477 0.005477]
Elementwise time and first three results:
0.102527s, [ 0.005477 0.005477 0.005477]
Elementwise Python looping time and first three results:
2.398071s, [ 0.005477 0.005477 0.005477]
GPUArray time and first three results:
8.207257s, [ 0.005477 0.005477 0.005477]
CPU time measured using :
0.000002s, [ 0.005477 0.005477 0.005477]
The first four time measurements are reasonable, the last one (0.000002s) however is way off. The CPU result should be the slowest one but it is orders of magnitude faster than the fastest GPU method. So obviously the measured time must be wrong. This is strange since the same timing method seems to work fine for the first four results.
So I took some code from SimpleSpeedTest.py and made a small test file [2], which produced:
time measured using option 1:
0.000002s
time measured using option 2:
5.989620s
Option 1 measures the duration using pycuda.driver.Event.record() (as in SimpleSpeedTest.py), option 2 uses time.clock(). Again, option 1 is off while option 2 gives a reasonable result (the time it takes to run the test file is around 6s).
Does anyone have an idea as to why this is happening?
Since using option 1 is endorsed in SimpleSpeedTest.py, could it be my setup that is causing the problem? I am running a GTX 470, Display Driver 301.42, CUDA 4.2, Python 2.7 64, PyCuda 2012.1, X5650 Xeon
[2] Test file:
import numpy
import time
import pycuda.driver as drv
import pycuda.autoinit
n_iter = 100000
nbr_values = 8192 # = 64 * 128 (values as used in SimpleSpeedTest.py)
start = drv.Event() # option 1 uses pycuda.driver.Event
end = drv.Event()
a = numpy.ones(nbr_values).astype(numpy.float32) # test data
start.record() # start option 1 (inserting recording points into GPU stream)
tic = time.clock() # start option 2 (using CPU time)
for i in range(n_iter):
a = numpy.sin(a) # do some work
end.record() # end option 1
toc = time.clock() # end option 2
end.synchronize()
events_secs = start.time_till(end)*1e-3
time_secs = toc - tic
print "time measured using option 1:"
print "%fs " % events_secs
print "time measured using option 2:"
print "%fs " % time_secs
I contacted Andreas Klöckner and he suggested to synchronize on the start event, too.
...
start.record()
start.synchronize()
...
And this seems to solve the issue!
time measured using option 1:
5.944461s
time measured using option 2:
5.944314s
Apparently CUDA's behaviour changed in the last two years. I updated SimpleSpeedTest.py.
How to retrieve the process start time (or uptime) in python in Linux?
I only know, I can call "ps -p my_process_id -f" and then parse the output. But it is not cool.
By using psutil https://github.com/giampaolo/psutil:
>>> import psutil, os, time
>>> p = psutil.Process(os.getpid())
>>> p.create_time()
1293678383.0799999
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(p.create_time()))
'2010-12-30 04:06:23'
>>>
...plus it's cross platform, not only Linux.
NB: I am one of the authors of this project.
If you are doing it from within the python program you're trying to measure, you could do something like this:
import time
# at the beginning of the script
startTime = time.time()
# ...
def getUptime():
"""
Returns the number of seconds since the program started.
"""
# do return startTime if you just want the process start time
return time.time() - startTime
Otherwise, you have no choice but to parse ps or go into /proc/pid. A nice bashy way of getting the elapsed time is:
ps -eo pid,etime | grep $YOUR_PID | awk '{print $2}'
This will only print the elapsed time in the following format, so it should be quite easy to parse:
days-HH:MM:SS
(if it's been running for less than a day, it's just HH:MM:SS)
The start time is available like this:
ps -eo pid,stime | grep $YOUR_PID | awk '{print $2}'
Unfortunately, if your process didn't start today, this will only give you the date that it started, rather than the time.
The best way of doing this is to get the elapsed time and the current time and just do a bit of math. The following is a python script that takes a PID as an argument and does the above for you, printing out the start date and time of the process:
import sys
import datetime
import time
import subprocess
# call like this: python startTime.py $PID
pid = sys.argv[1]
proc = subprocess.Popen(['ps','-eo','pid,etime'], stdout=subprocess.PIPE)
# get data from stdout
proc.wait()
results = proc.stdout.readlines()
# parse data (should only be one)
for result in results:
try:
result.strip()
if result.split()[0] == pid:
pidInfo = result.split()[1]
# stop after the first one we find
break
except IndexError:
pass # ignore it
else:
# didn't find one
print "Process PID", pid, "doesn't seem to exist!"
sys.exit(0)
pidInfo = [result.split()[1] for result in results
if result.split()[0] == pid][0]
pidInfo = pidInfo.partition("-")
if pidInfo[1] == '-':
# there is a day
days = int(pidInfo[0])
rest = pidInfo[2].split(":")
hours = int(rest[0])
minutes = int(rest[1])
seconds = int(rest[2])
else:
days = 0
rest = pidInfo[0].split(":")
if len(rest) == 3:
hours = int(rest[0])
minutes = int(rest[1])
seconds = int(rest[2])
elif len(rest) == 2:
hours = 0
minutes = int(rest[0])
seconds = int(rest[1])
else:
hours = 0
minutes = 0
seconds = int(rest[0])
# get the start time
secondsSinceStart = days*24*3600 + hours*3600 + minutes*60 + seconds
# unix time (in seconds) of start
startTime = time.time() - secondsSinceStart
# final result
print "Process started on",
print datetime.datetime.fromtimestamp(startTime).strftime("%a %b %d at %I:%M:%S %p")
man proc says that the 22nd item in /proc/my_process_id/stat is:
starttime %lu
The time in jiffies the process started after system boot.
Your problem now is, how to determine the length of a jiffy and how to determine when the system booted.
The answer for the latter comes still from man proc: it's in /proc/stat, on a line of its own like this:
btime 1270710844
That's a measurement in seconds since Epoch.
The answer for the former I'm not sure about. man 7 time says:
The Software Clock, HZ, and Jiffies
The accuracy of many system calls and timestamps is limited by the resolution of the software clock, a clock maintained by the kernel which measures time in jiffies. The size of a jiffy is determined by the value of the kernel constant HZ. The value of HZ varies across kernel versions and hardware platforms. On x86 the situation is as follows: on kernels up to and including 2.4.x, HZ was 100, giving a jiffy value of 0.01 seconds; starting with 2.6.0, HZ was raised to 1000, giving a jiffy of 0.001 seconds; since kernel 2.6.13, the HZ value is a kernel configuration parameter and can be 100, 250 (the default) or 1000, yielding a jiffies value of, respectively, 0.01, 0.004, or 0.001 seconds.
We need to find HZ, but I have no idea on how I'd go about that from Python except for hoping the value is 250 (as Wikipedia claims is the default).
ps obtains it thus:
/* sysinfo.c init_libproc() */
if(linux_version_code > LINUX_VERSION(2, 4, 0)){
Hertz = find_elf_note(AT_CLKTCK);
//error handling
}
old_Hertz_hack(); //ugh
This sounds like a job well done by a very small C module for Python :)
Here's code based on badp's answer:
import os
from time import time
HZ = os.sysconf(os.sysconf_names['SC_CLK_TCK'])
def proc_age_secs():
system_stats = open('/proc/stat').readlines()
process_stats = open('/proc/self/stat').read().split()
for line in system_stats:
if line.startswith('btime'):
boot_timestamp = int(line.split()[1])
age_from_boot_jiffies = int(process_stats[21])
age_from_boot_timestamp = age_from_boot_jiffies / HZ
age_timestamp = boot_timestamp + age_from_boot_timestamp
return time() - age_timestamp
I'm not sure if it's right though. I wrote a test program that calls sleep(5) and then runs it and the output is wrong and varies over a couple of seconds from run to run. This is in a vmware workstation vm:
if __name__ == '__main__':
from time import sleep
sleep(5)
print proc_age_secs()
The output is:
$ time python test.py
6.19169998169
real 0m5.063s
user 0m0.020s
sys 0m0.036s
def proc_starttime(pid=os.getpid()):
# https://gist.github.com/westhood/1073585
p = re.compile(r"^btime (\d+)$", re.MULTILINE)
with open("/proc/stat") as f:
m = p.search(f.read())
btime = int(m.groups()[0])
clk_tck = os.sysconf(os.sysconf_names["SC_CLK_TCK"])
with open("/proc/%d/stat" % pid) as f:
stime = int(f.read().split()[21]) / clk_tck
return datetime.fromtimestamp(btime + stime)
you can parse /proc/uptime
>>> uptime, idletime = [float(f) for f in open("/proc/uptime").read().split()]
>>> print uptime
29708.1
>>> print idletime
26484.45
for windows machines, you can probably use wmi
import wmi
c = wmi.WMI()
secs_up = int([uptime.SystemUpTime for uptime in c.Win32_PerfFormattedData_PerfOS_System()][0])
hours_up = secs_up / 3600
print hours_up