I want to be able to easily set a benchmarking program's file write rate. It's a python program that I'm using to test another system. I'd like to be able to control the rate of file creation. One method I have thought of is to have a function with an argument for the number of files to create. This could be called in a loop which keeps track of the clock and only calls the function every second. This would fulfill the requirement of creating a certain number of files every second. The problem with this is that there could be a chunk of dead time (milliseconds, but still). I'd like a continuous load.
You'll have to somehow keep track of the time it takes to actually perform the file I/O calls, and adjust the sleep times between the operations. Adjustment needs to be continuous, as the sleeps and IO calls might take different amount of time depending on system load.
If you'd like to do N operations per second on average, you could run loops of few seconds (or longer), and after every round see if you're running too fast or slow, and adjust the sleep() time done between each operation upwards or downwards based on that. If you're running much too fast, increment the sleep time more, if you're only a little bit too fast, increment less.
import time
# target rate: 100 ops / 1 second
target = 100.0
round_time = 1.0
# at first, assume the writes are immediate
sleepTime = round_time/target
ops = 0
t_start = time.time()
while True:
#doYourIOoperationHere()
ops += 1
time.sleep(sleepTime)
# adjust sleep time periodically
if ops == target:
t_end = time.time()
elapsed = t_end - t_start
difference = round_time - elapsed
# print out the vars here to debug adjustment
print "%d ops done, elapsed %.3f, difference %.3f" % (ops, elapsed, difference)
# increase or decrease the sleep time, approach the target time slowly
sleepTime += difference/target/2
t_start = time.time()
ops = 0
Or something along those lines (simplistic code untested). This might not work well for very high IO rates or system loads, you might have to start doing multiple write operations per single sleep call. Also, a longer averaging than 1 second is likely to be necessary.
Related
I need to meassure the time certain parts of my code take. While executing my code on a powerfull server, I get 10 diffrent results
I tried comparing time measured with time.time(), time.perf_counter(), time.perf_counter_ns(), time.process_time() and time.process_time_ns().
import time
for _ in range(10):
start = time.perf_counter()
i = 0
while i < 100000:
i = i + 1
time.sleep(1)
end = time.perf_counter()
print(end - start)
I'm expecting when executing the same code 10 times, to be the same (the results to have a resolution of at least 1ms) ex. 1.041XX and not 1.030sec - 1.046sec.
When executing my code on a 16 cpu, 32gb memory server I'm receiving this result:
1.045549364
1.030857833
1.0466020120000001
1.0309665050000003
1.0464690349999994
1.046397238
1.0309525370000001
1.0312070380000007
1.0307592159999999
1.046095523
Im expacting the result to be:
1.041549364
1.041857833
1.0416020120000001
1.0419665050000003
1.0414690349999994
1.041397238
1.0419525370000001
1.0412070380000007
1.0417592159999999
1.041095523
Your expectations are wrong. If you want to measure code average time consumption use the timeit module. It executes your code multiple times and averages over the times.
The reason your code has different runtimes lies in your code:
time.sleep(1) # ensures (3.5+) _at least_ 1000ms are waited, won't be less, might be more
You are calling it in a tight loop,resulting in accumulated differences:
Quote from time.sleep(..) documentation:
Suspend execution of the calling thread for the given number of seconds. The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal’s catching routine. Also, the suspension time may be longer than requested by an arbitrary amount because of the scheduling of other activity in the system.
Changed in version 3.5: The function now sleeps at least secs even if the sleep is interrupted by a signal, except if the signal handler raises an exception (see PEP 475 for the rationale).
Emphasis mine.
Perfoming a code do not take the same time at each loop iteration because of the scheduling of the system (system puts on hold your process to perform another process then back to it...).
I am trying to sample a signal at 10Khz in Python. There is no problem when try to run this code(at 1KHz):
import sched, time
i = 0
def f(): # sampling function
s.enter(0.001, 1, f, ())
global i
i += 1
if i == 1000:
i = 0
print "one second"
s = sched.scheduler(time.time, time.sleep)
s.enter(0.001, 1, f, ())
s.run()
When I try to make the time less, it starts to exceed one second(in my computer, 1.66s at 10e-6).
It it possible to run a sampling function at a specific frequency in Python?
You didn't account for the code's overhead. Each iteration, this error adds up and skews the "clock".
I'd suggest to use a loop with time.sleep() instead (see comments to https://stackoverflow.com/a/14813874/648265) and count the time to sleep from the next reference moment so the inevitable error doesn't add up:
period=0.001
t=time.time()
while True:
t+=period
<...>
time.sleep(max(0,t-time.time())) #max is needed in Windows due to
#sleep's behaviour with negative argument
Note that the OS scheduling will not allow you to reach precisions beyond a certain level since other processes have to preempt yours from time to time. In this case, you'll need to use some OS-specific facilities for multimedia applications or work out a solution that doesn't need this level of accuracy (e.g. sample the signal with a specialized app and work with its saved output).
I'm currently reading physics in the university, and im learning python as a little hobby.
To practise both at the same time, i figured I'll write a little "physics engine" that calculates the movement of an object based on x,y and z coordinates. Im only gonna return the movement in text (at least for now!) but i want the position updates to be real-time.
To do that i need to update the position of an object, lets say a hundred times a second, and print it back to the screen. So every 10 ms the program prints the current position.
So if the execution of the calculations take 2 ms, then the loop must wait 8ms before it prints and recalculate for the next position.
Whats the best way of constructing a loop like that, and is 100 times a second a fair frequency or would you go slower, like 25 times/sec?
The basic way to wait in python is to import time and use time.sleep. Then the question is, how long to sleep? This depends on how you want to handle cases where your loop misses the desired timing. The following implementation tries to catch up to the target interval if it misses.
import time
import random
def doTimeConsumingStep(N):
"""
This represents the computational part of your simulation.
For the sake of illustration, I've set it up so that it takes a random
amount of time which is occasionally longer than the interval you want.
"""
r = random.random()
computationTime = N * (r + 0.2)
print("...computing for %f seconds..."%(computationTime,))
time.sleep(computationTime)
def timerTest(N=1):
repsCompleted = 0
beginningOfTime = time.clock()
start = time.clock()
goAgainAt = start + N
while 1:
print("Loop #%d at time %f"%(repsCompleted, time.clock() - beginningOfTime))
repsCompleted += 1
doTimeConsumingStep(N)
#If we missed our interval, iterate immediately and increment the target time
if time.clock() > goAgainAt:
print("Oops, missed an iteration")
goAgainAt += N
continue
#Otherwise, wait for next interval
timeToSleep = goAgainAt - time.clock()
goAgainAt += N
time.sleep(timeToSleep)
if __name__ == "__main__":
timerTest()
Note that you will miss your desired timing on a normal OS, so things like this are necessary. Note that even with asynchronous frameworks like tulip and twisted you can't guarantee timing on a normal operating system.
Since you cannot know in advance how long each iteration will take, you need some sort of event-driven loop. A possible solution would be using the twisted module, which is based on the reactor pattern.
from twisted.internet import task
from twisted.internet import reactor
delay = 0.1
def work():
print "called"
l = task.LoopingCall(work)
l.start(delay)
reactor.run()
However, as has been noted, don't expect a true real-time responsiveness.
A piece of warning. You may not expect a real time on a non-realtime system. The sleep family of calls guarantees at least a given delay, but may well delay you for more.
Therefore, once you returned from sleep, query current time, and make the calculations into the "future" (accounting for the calculation time).
How can I best create a high frequency, 100 times a second, loop in python on Linux? It does not need to be highly accurate, just good enough so on average does actually loop about 100 times a second.
I tried:
import time
count = 0
start = time.time()
while count <= 300:
time.sleep(0.01)
count = count + 1
end = time.time()
print('avg. {0}'.format((end - start) / count))
And it works pretty well! Getting
avg. 0.0103...
output.
But is there a better way to do such tight loops?
(This is for a server that needs to frequently read incoming network packets and process them, and perform periodic processing whether there are new packets or not).
Since you are on UNIX, you can also use the signal module.
def handler(signum, frame):
print 'do it'
# This will fire an ITIMER_REAL signal every 0.01 seconds
signal.setitimer(signal.ITIMER_REAL, 0, 0.01)
# Tell the signal module to execute handler when upon signal
# ITIMER_REAL
signal.signal(signal.ITIMER_REAL, handler)
For more info see: https://docs.python.org/2/library/signal.html. I have used signals in my own code. I found it highly robust and accurate but somewhat limited in scope (you can only set a limited number of signals). A big advantage is that you do not need threads for concurrency. Be aware, that when you combine multi-threaded code with signals, you must take good care that you set the signal handler (signal.signal) in the main thread.
Here's a simple ad-hock solution without drift:
import time
INTERVAL = 0.01
last = time.time()
while True:
next = last + INTERVAL
time.sleep(next - time.time()) # it's ok to sleep negative time
last = next
do_your_thing_here()
This is for Pygame only but at the end of the loop you could do...
mainClock = pygame.time.Clock()
mainClock.tick(100)
100 or 1000 or whatever really.
I need to wait for about 25ms in one of my functions. Sometimes this function is called when the processor is occupied with other things and other times it has the processor all to itself.
I've tried time.sleep(.25) but sometimes its actually 25ms and other times it takes much longer. Is there a way to sleep for an exact amount of time regardless of processor availability?
Because you're working with a preemptive operating system, there's no way you can guarantee that your process will be able to have control of the CPU in 25ms.
If you'd still like to try, it would be better to have a busy loop that polls until 25ms has passed. Something like this might work:
import time
target_time = time.clock() + 0.025
while time.clock() < target_time:
pass
0.25 seconds are 250 ms, not 25. Apart from this, there is no way to wait for exactly 25 ms on common operating systems – you would need some real-time operating system.
What system are you on? If you're on Windows you may want to do something like this for exact timing:
import ctypes
kernel32 = ctypes.windll.kernel32
# This sets the priority of the process to realtime--the same priority as the mouse pointer.
kernel32.SetThreadPriority(kernel32.GetCurrentThread(), 31)
# This creates a timer. This only needs to be done once.
timer = kernel32.CreateWaitableTimerA(ctypes.c_void_p(), True, ctypes.c_void_p())
# The kernel measures in 100 nanosecond intervals, so we must multiply .25 by 10000
delay = ctypes.c_longlong(.25 * 10000)
kernel32.SetWaitableTimer(timer, ctypes.byref(delay), 0, ctypes.c_void_p(), ctypes.c_void_p(), False)
kernel32.WaitForSingleObject(timer, 0xffffffff)
This code will pretty much guarentee your process will sleep .25 seconds. Watch out though- you may want to lower the priority to 2 or 3 unless it's absolutely critical that this sleeps for .25 seconds. Certainly don't change the priority too high for a user-end product.
Edit: in Windows 10 this nonsense seems unnecessary. Try it like so:
>>> from time import sleep
>>> import timeit
>>> '%.2f%% overhead' % (timeit.timeit('sleep(0.025)', number=100, globals=globals()) / 0.025 - 100)
'0.29% overhead'
.29%, or thereabout, is fairly low overhead, and usually more than accurate enough.
Previous Windows versions will by default have a sleep resolution of 55 msecs, which means your sleep call will take somewhere between 25 and 55 msecs. To get the sleep resolution down to 1 millisecond you need to set the resolution used by Windows by calling timeBeginPeriod:
import ctypes
winmm = ctypes.WinDLL('winmm')
winmm.timeBeginPeriod(1)
Another solution for accurate timings and delay is to use the perf_counter() function from module time. Especially useful in windows as time.sleep is not accurate in milliseconds. See below example where function accurate_delay creates a delay in millisecond.
import time
def accurate_delay(delay):
''' Function to provide accurate time delay in millisecond
'''
_ = time.perf_counter() + delay/1000
while time.perf_counter() < _:
pass
delay = 10
t_start = time.perf_counter()
print('Wait for {:.0f} ms. Start: {:.5f}'.format(delay, t_start))
accurate_delay(delay)
t_end = time.perf_counter()
print('End time: {:.5f}. Delay is {:.5f} ms'.
format(t_end, 1000*(t_end - t_start)))
sum = 0
ntests = 1000
for _ in range(ntests):
t_start = time.perf_counter()
accurate_delay(delay)
t_end = time.perf_counter()
print('Test completed: {:.2f}%'.format(_/ntests * 100), end='\r', flush=True)
sum = sum + 1000*(t_end - t_start) - delay
print('Average difference in time delay is {:.5f} ms.'.format(sum/ntests))
What you intend to do is a real time application. Python (and probably the OS you are using) is not intended to program this kind of applications, where time restriction is so strict.
In order for you to achieve what you are looking for you need a RTOS (Real Time Operating System) and develop your application using a suitable programming language (usually C) following RT best practises.
From the docs of the sleep method:
Suspend execution for the given number of seconds. The argument may be
a floating point number to indicate a more precise sleep time. The
actual suspension time may be less than that requested because any
caught signal will terminate the sleep() following execution of that
signal’s catching routine. Also, the suspension time may be longer
than requested by an arbitrary amount because of the scheduling of
other activity in the system.
The fact is that it depends on your underlying OS.