My question was not specific enough last time, and so this is second question about this topic.
I'm running some experiments and I need to precisely measure participants' response time to questions in millisecond unit.
I know how to do this with the time module, but I was wondering if this is reliable enough or I should be careful using it. I was wondering if there are possibilities of some other random CPU load will interfere with the measuring of time.
So my question is, will the response time measure with time module be very accurate or there will be some noise associate with it?
Thank you,
Joon
CPU load will affect timing. If your application is startved of a slice of CPU time, then timing would get affected. You can not help that much. You can be as precise and no more. Ensure that your program gets a health slice of cpu time and the result will be accurate. In most cases, the results should be accurate to milliseconds.
If you benchmark on a *nix system (Linux most probably), time.clock() will return CPU time in seconds. On its own, it's not very informative, but as a difference of results (i.e. t0 = time.clock(); some_process(); t = time.clock() - t0), you'd have a much more load-independent timing than with time.time().
Related
I'm fairly new to python (and CS in general), and I've been reading some docs regarding the "time" library in python. There are quite a lot of time measuring methods, and I'm trying to find the most suitable one that will enable me to compare the performance of 2 versions of an algorithm.
I understand that time.time() is wall time, and time.process_time() is either user-cpu time or system-cpu time (I'm not quite sure), but which one of these two would be a better (more accurate) measure of performance?
Thank you!!!
I would suggest you use time.perf_counter() as it is the recommended function for this kind of tasks (it auto selects the method with highest precision available).
It returns a float, that means just nothing on its own (unlike the result of time.time()), but computing the difference between two time.perf_counter()'s measurements tells you how much time elapsed.
For more info, read the time.perf_counter()'s docs.
Recently, when creating a loop with a very short wait at the end, I ran into an unexpected behaviour of time.sleep() when used in quick succession.
I used this piece of code to look further into my problem
import time
import statistics
def average_wait(func):
waits=[]
loops=0
while loops<1000:
start=time.time()
func(1/1000)
waits.append(time.time()-start)
loops+=1
print(waits)
print("Average wait for 0.001: {}".format(statistics.mean(waits)))
average_wait(time.sleep)
This function usually returns something around 0.0013 which is many many times less accurate than just calling time.sleep() once, upon further inspection of this problem by looking at the waits list, I found that the amount of time time.sleep() was actually sleeping for was either almost exactly the right amount of time or almost exactly double the amount of time.
Here is a sample from waits:
[0.0010008811950683594, 0.0020041465759277344, 0.0009999275207519531, 0.0019621849060058594, 0.0010418891906738281]
Is there any reason for this behaviour and anything that can be done to avoid it?
From the time.time() documentation:
Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second.
The precision is platform dependent. Moreover, it produces wall-clock time, and your process is never the only thing running on a modern OS, other processes also are given time to process and you'll see variation in timings in your own process because of that.
The module offers different clocks, with more precision and some are per-process. See the time.get_clock_info() function to see what precision they offer. Note time.process_time() offers per-process time but excludes sleep time.
Next, time.sleep() is also not going to sleep in exact time spans; again from the relevant documentation:
[T]he suspension time may be longer than requested by an arbitrary amount because of the scheduling of other activity in the system.
It too is subject to OS scheduling.
Together, these effects can easily add up to the millisecond variation in timings you see in your experiments. So this is not a doubling of time slept; even if you used different values for time.sleep() you'd still see a similar deviation from the requested time.
This question already has answers here:
Python's time.clock() vs. time.time() accuracy?
(16 answers)
Closed 6 years ago.
I thought both measure the amount of time? But they return very different numbers and I don't understand what the documentation is saying. Can anyone elaborate?
time.clock() gives you an elapsed amount of time. time.time() gives you the wall clock time.
You can use time.time() to communicate with others (including humans) about when something happened. time.clock() only lets you measure how long something takes.
Generally speaking, you'd use time.clock() when you want to measure timings, time.time() to schedule something. To that end time.time() has to be set correctly on your computer (to agree with the rest of your region as to what time it is now), but time.clock() doesn't, it just counts seconds from an arbitrary point in time (usually when your computer started or when your process first used the function).
The exact behaviour of time.clock() depends on your OS (it could just measure process time, excluding time sleeping, or it could measure time elapsed even when the process is inactive, it could go backwards if your system time is adjusted, etc).
For some use-cases this variability in exact behaviour isn't good enough, and as such it is deprecated in Python 3. There better options are available for either measuring performance or process time, see time.perf_counter() and time.process_time().
I am writing a 3D engine in Python using Pygame. For the first 300 or so frames, it takes from about 0.004 to 0.006 seconds to render 140 polygons. But after that, it suddenly takes an average of about 0.020 seconds to do the same task. This is concerning to me because this is a small-scale test, and even though 50 FPS is decent, it cannot be sustained at 1000 polygons, for instance.
I have already done a lot of streamlining to my code. I have also done some slightly deeper profiling, and it appears that the increased time is more or less proportionately distributed, which suggests that the problem is not specific to a single piece of code.
I assume that the problem has something to do with memory usage, but I do not know exactly why this problem is happening. What specific issue is causing this to happen, and how can I optimize my code to fix it, as well as some more general practices? Since my code is very long, it is posted here.
Although I can't answer your question exactly, I would use a task manager and watch the "python" (or "pygame" depending on your OS) process, and view it's memory consumption. If that turns out to be the issue, you could check to see what variables you don't need after a certain time, and then you could clear those variables.
Edit: Some CPUs have data loss prevention systems. What I mean by this is this:
If Application X takes up 40ish % of the CPU (it doesn't have to be that high). After a certain amount of time, the CPU will throttle the amount of CPU that Application X is allowed to use. This can cause slowdown for things such as this. This doesn't happen with (most) games because they're set up to tell the CPU to expect that amount of strain.
I am trying to sample cpu registers every millisecond and calculate the frequency. To have an accurate measurement, I require that the sampling time to be very accurate. I have been using the time.sleep() to achieve this but sleep is not very accurate past 1 second.
What I would like to do is set up a counter and sample when that counter reaches a certain value and where the counter is incremented at an accurate rate. I am running Python 2.6. Does anyone have any suggestions?
I suspect there are likely several Python packages out there that -help- with what you want. I also suspect Python is not the right tool for that surpose.
There is the timeit module
There is time module with clock() which is NOT a wall clock but a CPU usage clock (for the application initializing the time.clock() object. It is a floating point value which shows some 12+ digits below the ones place ie 1.12345678912345. Python floats are not know for there accuracy and the return value from time.clock() is not something I personally trust as accurate.
There are other Python introspection tools that you can google for, like inspect, itertools, and others, that time processes. however I suspect their accuracy is dependant on running averages of many iterations of measuring the same thing.