We Keep Coding

Running Python function at defined intervals

I've got a Python loop that should run every minute, do some data processing and sleep until the next minute is up. However the processing takes variable amount of time, sometimes it's close to zero when there is not much to do, sometimes it takes even 10 or 20 seconds.
To compensate for that I measure the time it takes to run the processing like this:
while True:
time_start = time.time()
do_something() # <-- This takes unknown time
time_spent = time.time() - time_start
time.sleep(60 - time_spent)
It kind of works but over a couple of days it still drifts away by a number of seconds. I guess it happens when the computer (small Raspberry Pi) is busy and delays the start of the loop, then it all starts slipping away.
I don't need do_something() executed exactly every minute, so no need for real-time OS or anything like that, but I don't want one delayed start affect all the subsequent ones either.
Is there some kind of scheduler that can run my function at a predefined rate? Or some more clever way to compensate for the occasional delays?

Playing with the loop a little this seems to work quite well. The trick is to record the start time once before the loop starts, not on every iteration. That way one delayed start won't affect any future ones.
rate_sec = 60
time_start = time.time()
while True:
print("{}".format(datetime.now()))
# Emulate processing time 0s to 20s
time.sleep(random.randint(0, 20))
# Sleep until the next 'rate_sec' multiple
delay = rate_sec - (time.time() - time_start) % rate_sec
time.sleep(delay)

Is sleeping a pre-requisite of your project? I mean, you don't need to have your processing blocked if you want to run the task every ~1 minute.
Since you are on a Raspberry Pi, you can (and probably should) use crontab.
This will give you the most flexibility and allow you to don't have the computer sleeping idle.
To do this, open your cron tab with
crontab -e
And add the following entry
* * * * * /usr/bin/env python3 /path/to/script.py

python: run tasks at different precise times without busy loop

Input: array of float time values (in seconds) relative to program start. [0.452, 0.963, 1.286, 2.003, ... ]. They are not evenly spaced apart.
Desired Output: Output text to console at those times (i.e. printing '#')
My question is what is the best design principle to go about this. Below is my naive solution using time.time.
times = [0.452, 0.963, 1.286, 2.003]
start_time = time.time()
for event_time in times:
while 1:
if time.time() - start_time >= event_time:
print '#'
break
The above feels intuitively wrong using that busy loop (even if its in its own thread).
I'm leaning towards scheduling but want to make sure there aren't better design options: Executing periodic actions in Python
There is also the timer object: timers
Edit: Events only need 10ms precision, so +/- 10ms from exact event time.

A better pattern than busy waiting might be to use time.sleep(). This suspends execution rather than using the CPU.
time_diffs = [0.452, 0.511, 0.323, 0.716]
for diff in time_diffs:
time.sleep(diff)
print '#'
Threading can also be used to similar effect. However both of these solutions only work if the action you want to perform each time the program 'restarts' takes negligible time (perhaps not true of printing).
That being said no pattern is going to work if you are after 10ms precision and want to use Python on a standard OS. I recommend this question on Real time operating via Python which explains both that GUI events (i.e. printing to a screen) are too slow and unreliable for that level of precision, that your typical OSs where Python is run do not guarantee that level of precision and that Python's garbage collection and memory management also play havoc if you want 'real-time' events.

How to advance clock and going through all the events

Reading this answer (point 2) to a question related to Twisted's task.Clock for testing purposes, I found very weird that there is no way to advance the clock from t0 to t1 while catching all the callLater calls within t0 and t1.
Of course, you could solve this problem by doing something like:
clock = task.Clock()
reactor.callLater = clock.callLater
...
def advance_clock(total_elapsed, step=0.01):
elapsed = 0
while elapsed < total_elapsed:
clock.advance(step)
elapsed += step
...
time_to_advance = 10 # seconds
advance_clock(time_to_advance)
But then we have shifted the problem toward choosing a sufficiently small step, which could be very tricky for callLater calls that sample the time from a probability distribution, for instance.
Can anybody think of a solution to this problem?

I found very weird that there is no way to advance the clock from t0 to t1 while catching all the callLater calls within t0 and t1.
Based on what you wrote later in your question, I'm going to suppose that the case you're pointing out is the one demonstrated by the following example program:
from twisted.internet.task import Clock
def foo(reactor, n):
if n == 0:
print "Done!"
reactor.callLater(1, foo, reactor, n - 1)
reactor = Clock()
foo(reactor, 10)
reactor.advance(10)
One might expect this program to print Done! but it does not. If the last line is replaced with:
for i in range(10):
reactor.advance(1)
Then the resulting program does print Done!.
The reason Clock works this way is that it's exactly the way real clocks work. As far as I know, there are no computer clocks that operate with a continuous time system. I won't say it is impossible to implement a timed-event system on top of a clock with discrete steps such that it appears to offer a continuous flow of time - but I will say that Twisted makes no attempt to do so.
The only real difference between Clock and the real reactor implementations is that with Clock you can make the time-steps much larger than you are likely to encounter in typical usage of a real reactor.
However, it's quite possible for a real reactor to get into a situation where a very large chunk of time all passes in one discrete step. This could be because the system clock changes (there's some discussion of making it possible to schedule events independent of the system clock so that this case goes away) or it could be because some application code blocked the reactor for a while (actually, application code always blocks the reactor! But in typical programs it only blocks it for a period of time short enough for most people to ignore).
Giving Clock a way to mimic these large steps makes it possible to write tests for what your program does when one of these cases arises. For example, perhaps you really care that, when the kernel decides not to schedule your program for 2.1 seconds because of a weird quirk in the Linux I/O elevator algorithm, your physics engine nevertheless computes 2.1 seconds of physics even though 420 calls of your 200Hz simulation loop have been skipped.
It might be fair to argue that the default (standard? only?) time-based testing tool offered by Twisted should be somewhat more friendly towards the common case... Or not. Maybe that would encourage people to write programs that only work in the common case and break in the real world when the uncommon (but, ultimately, inevitable) case arises. I'm not sure.
Regarding Mike's suggestion to advance exactly to the next scheduled call, you can do this easily and without hacking any internals. clock.advance(clock.getDelayedCalls()[0].getTime() - clock.seconds()) will do exactly this (perhaps you could argue Clock would be better if it at least offered an obvious helper function for this to ease testing of the common case). Just remember that real clocks do not advance like this so if your code has a certain desirable behavior in your unit tests when you use this trick, don't be fooled into thinking this means that same desirable behavior will exist in real usage.

Given that the typical use-class for Twisted is to mix hardware events and timers I'm confused why you would want to do this, but...
My understanding is that interally Twisted is tracking callLater events via a number of lists that are inside of the reactor object (See: http://twistedmatrix.com/trac/browser/tags/releases/twisted-15.2.0/twisted/internet/base.py#L437 - the xxxTimedCalls lists inside of class ReactorBase)
I haven't done any work to figure out if those lists are exposed anywhere, but if you want to take the reactors life into your own hands I'm sure you could hack your way in.
With access to the timing lists you could simply forward time to whenever the next element of the list is ... though if your trying to test code that interacts with IO events, I can't imagine this is going to do anything but confuse you...
Best of luck

Here's a function that will advance the reactor to the next IDelayedCall by iterating over reactor.getDelayedCalls. This has the problem Mike mentioned of not catching IO events, so you can specify a minimum and maximum time that it should wait, as well as a maximum time step.
def advance_through_delayeds(reactor, min_t=None, max_t=None, max_step=None):
elapsed = 0
while True:
if max_t is not None and elapsed >= max_t:
break
try:
step = min(d.getTime() - reactor.seconds() for d in reactor.getDelayedCalls())
except ValueError:
# nothing else pending
if min_t is not None and elapsed < min_t:
step = min_t - elapsed
else:
break
if max_step is not None:
step = min(step, max_step)
if max_t is not None:
step = min(step, max_t-elapsed)
reactor.advance(step)
elapsed += step
return elapsed
If you need to wait for some I/O to complete, then set min_t and max_step to reasonable values.
# wait at least 10s, advancing the reactor by no more than 0.1s at a time
advance_through_delayeds(reactor, min_t=10, max_step=0.1)
If min_t is set, it will exit once getDelayedCalls returns an empty list after that time is reached.
It's probably a good idea to always set max_t to a sane value to prevent the test suite from hanging. For example, on the above foo function by JPC it does reach the print "Done!" statement, but then would hang forever as the callback chain never completes.

python timing long program/simulation runs?

Using python for starting modelling/simulation runs (tuflow) and logging the runs to db
Currently on windows , python 2.7 and using timeit() .
Is it better to stick with using timeit() or to switch to using time.clock() ?
Simulation/modelling runs can be anything from a couple of minutes to a week+.
Need reasonably accurate runtimes.
I know time.clock has been depreciated on 3.3+ but I can't see this code getting moved to 3 for a long while.
self.starttime = timeit.default_timer()
run sim()
self.endtime = timeit.default_timer()
self.runtime = self.starttime - self.endtime

timeit's `timeit() function performs the code multiple times and takes the best, so it's not a great choice for lengthy runs. But your code just uses the timer, whose documentation warns that it measures elapsed time, so you will need to ensure standard running conditions in so far as you can. If you want to measure actual CPU usage, that's tricker.
When you say "reasonably accurate" times, one percent of two minutes is 2.4 seconds. The precision of time.clock() or the default timer is never going to be less than a second. Is that accurate enough?
To mitigate any possibility of migration to Python 3 you can define your own timing function. On Python 2 it can use time.clock() but at least you will only have one place to alter code if you do migrate.

Python – time.time() vs. bash time

I've been working on some Project Euler problems in Python 3 [osx 10.9], and I like to know how long they take to run.
I've been using the following two approaches to time my programs:
1)
import time
start = time.time()
[program]
print(time.time() - start)
2) On the bash command line, typing time python3 ./program.py
However, these two methods often give wildy different results. In the program I am working on now, the first returns 0.000263 (seconds, truncated) while the second gives
real 0m0.044s
user 0m0.032s
sys 0m0.009s
Clearly there is a huge discrepancy - two orders of magnitude compared to the real time.
My questions are:
a) Why the difference? Is it overhead from the interpreter?
b) Which one should I be using to accurately determine how long the program takes to run? Is time.time() accurate at such small intervals?
I realize these miniscule times are not of the utmost importance; this was more of a curiosity.
Thanks.
[UPDATE:]
Thank-you to all of the answers & comments. You were correct with the overhead. This program:
import time
start = time.time()
print("hello world")
print(time.time() - start)
takes ~0.045 sec, according to bash.
My complicated Project Euler problem took ~0.045 sec, according to bash. Problem solved.
I'll take a look at timeit. Thanks.

The interpreter imports site.py and can touch upon various other files on start-up. This all takes time before your import time line is ever executed:
$ touch empty.py
$ time python3 empty.py
real 0m0.158s
user 0m0.033s
sys 0m0.021s
When timing code, take into account that other processes, disk flushes and hardware interrupts all take time too and influence your timings.
Use timeit.default_timer() to get the most accurate timer for your platform, but preferably use the timeit module itself to time individual snippets of code to eliminate as many variables as possible.

Because when you run the time builtin in bash the real time taken includes the time taken to start up the Python interpreter and import the required modules to run your code, rather than just timing the execution of a single function in your code.
To see this, try for example
import os
import time
start = time.time()
os.system('python <path_to_your_script>')
print time.time() - start
You'll find that this is much closer to what time reports.