Input: array of float time values (in seconds) relative to program start. [0.452, 0.963, 1.286, 2.003, ... ]. They are not evenly spaced apart.
Desired Output: Output text to console at those times (i.e. printing '#')
My question is what is the best design principle to go about this. Below is my naive solution using time.time.
times = [0.452, 0.963, 1.286, 2.003]
start_time = time.time()
for event_time in times:
while 1:
if time.time() - start_time >= event_time:
print '#'
break
The above feels intuitively wrong using that busy loop (even if its in its own thread).
I'm leaning towards scheduling but want to make sure there aren't better design options: Executing periodic actions in Python
There is also the timer object: timers
Edit: Events only need 10ms precision, so +/- 10ms from exact event time.
A better pattern than busy waiting might be to use time.sleep(). This suspends execution rather than using the CPU.
time_diffs = [0.452, 0.511, 0.323, 0.716]
for diff in time_diffs:
time.sleep(diff)
print '#'
Threading can also be used to similar effect. However both of these solutions only work if the action you want to perform each time the program 'restarts' takes negligible time (perhaps not true of printing).
That being said no pattern is going to work if you are after 10ms precision and want to use Python on a standard OS. I recommend this question on Real time operating via Python which explains both that GUI events (i.e. printing to a screen) are too slow and unreliable for that level of precision, that your typical OSs where Python is run do not guarantee that level of precision and that Python's garbage collection and memory management also play havoc if you want 'real-time' events.
Related
I'd like to know how'd you measure the amount of clock cycles per instruction say copy int from one place to another?
I know you can time it down to nano seconds but with today's cpu's that resolution is too low to get a correct reading for the oprations that take just a few clock cycles?
It there a way to confirm how many clock cycles per instructions like adding and subing it takes in python? if so how?
This is a very interesting question that can easily throw you into the rabbit's hole. Basically any CPU cycle measurements depends on your processors and compilers RDTSC implementation.
For python there is a package called hwcounter that can be used as follows:
# pip install hwcounter
from hwcounter import Timer, count, count_end
from time import sleep
# Method-1
start = count()
# Do something here:
sleep(1)
elapsed = count_end() - start
print(f'Elapsed cycles: {elapsed:,}')
# Method-2
with Timer() as t:
# Do something here:
sleep(1)
print(f'Elapsed cycles: {t.cycles:,}')
NOTE:
It seem that the hwcounter implementation is currently broken for Windows python builds. A working alternative is to build the pip package using the mingw compiler, instead of MS VS.
Caveats
Using this method, always depend on how your computer is scheduling tasks and threads among its processors. Ideally you'd need to:
bind the test code to one unused processor (aka. processor affinity)
Run the tests over 1k - 1M times to get a good average.
Need a good understanding of not only compilers, but also how python optimize its code internally. Many things are not at all obvious, especially if you come from C/C++/C# background.
Rabbit Hole:
http://en.wikipedia.org/wiki/Time_Stamp_Counter
https://github.com/MicrosoftDocs/cpp-docs/blob/main/docs/intrinsics/rdtsc.md
How to get the CPU cycle count in x86_64 from C++?
__asm
__rdtsc
__cpuid, __cpuidex
Defining __asm Blocks as C Macros
I applied the following code (https://circuitpython.readthedocs.io/projects/ads1x15/en/latest/examples.html) to read voltage.
In the last line of python code, I have set time.sleep command as (1/3300 s)
I have following queries:
In the time column, time-step comes out to be approximately (0.02 s). However expected time-step is (1/3300)s. Why does this occur?
How do I ensure that the time-step i.e sampling frequency between two successive time data points remains exactly at 3300 Hz. ?
How do I ensure that 1st time-data point starts with "0"?
Can somebody please clarify my doubts!
The sampling rate of the ADS1015 is meant to be 3300S/sec only in continuous mode, and sampling one channel at a time.
There are 2 steps here:
Ensure your ADC is in continuous sampling mode.
Putting it in continuous mode would be something like "adc.mode = 0", provided your library supports it. I have used this one https://github.com/adafruit/Adafruit_ADS1X15 and it does support it.
Ensure that the Data rate in the config register is set to 3300. (page 16 on the datasheet at https://cdn-shop.adafruit.com/datasheets/ads1015.pdf)
Purely that would also mostly not be enough, getting to the full potential of the ADC would also need a compatible processor that can handle large amounts of data on its i2c bus. Something like a raspberry pi is mostly not powerful enough.
Using faster languages like C/C++ would also help.
You have at least 3 problems and need to read the time module docs.
time.time is not guaranteed to be accurate to more than a second. In the following, in IDLE Shell on Win 10, multiple time.time() calls give same time.
>>> for i in range(30):
print(time.perf_counter(), time.time(), time.perf_counter())
8572.4002846 1607086901.7035756 8572.4002855
8572.4653746 1607086901.756807 8572.4653754
8572.4706208 1607086901.7724454 8572.4706212
8572.4755909 1607086901.7724454 8572.4755914
8572.4806756 1607086901.7724454 8572.4806759
... # time.time continues repeating 3 or 4 times.
time.sleep(t) has a minimum system-dependent interval, even if t is much smaller. On Windows, it is about .015 seconds. There is no particular upper limit if there is other system activity.
>>> for i in range(5):
print(time.perf_counter())
time.sleep(.0000001)
9125.1041623
9125.1188101
9125.134417
9125.1565579
9125.1722012
Printing to IDLE's shell is slower than running a program direct with Python (from command line) and printing to the system console. For one thing, IDLE runs user code in a separate process, adding interprocess overhead. For another, IDLE is a GUI program and the GUI framework, tk via tkinter, add more overhead. IDLE is designed for learning Python and developing Python programs. It is not optimized for running Python programs.
If user code outputs to a tkinter GUI it creates in the same process, avoiding the interprocess delay, the minimum interval is much shorter, about .0012 seconds in this particular example.
>>> import tkinter as tk
>>> r = tk.Tk()
>>> t = tk.Text(r)
>>> t.pack()
>>> for i in range(5):
t.insert('insert', f'{time.perf_counter()}\n')
r.update()
# In the text widget...
9873.6484271
9873.6518752
9873.6523338
9873.6527421
9873.6532307
I'm trying to get a looping call to run every 2 seconds. Sometimes, I get the desired functionality, but othertimes I have to wait up to ~30 seconds which is unacceptable for my applications purposes.
I reviewed this SO post and found that looping call might not be reliable for this by default. Is there a way to fix this?
My usage/reason for needing a consistent ~2 seconds:
The function I am calling scans an image (using CV2) for a dollar value and if it finds that amount it sends a websocket message to my point of sale client. I can't have customers waiting 30 seconds for the POS terminal to ask them to pay.
My source code is very long and not well commented as of yet, so here is a short example of what I'm doing:
#scan the image for sales every 2 seconds
def scanForSale():
print ("Now Scanning for sale requests")
#retrieve a new image every 2 seconds
def getImagePreview():
print ("Loading Image From Capture Card")
lc = LoopingCall(scanForSale)
lc.start(2)
lc2 = LoopingCall(getImagePreview)
lc2.start(2)
reactor.run()
I'm using a Raspberry Pi 3 for this application, which is why I suspect it hangs for so long. Can I utilize multithreading to fix this issue?
Raspberry Pi is not a real time computing platform. Python is not a real time computing language. Twisted is not a real time computing library.
Any one of these by itself is enough to eliminate the possibility of a guarantee that you can run anything once every two seconds. You can probably get close but just how close depends on many things.
The program you included in your question doesn't actually do much. If this program can't reliably print each of the two messages once every two seconds then presumably you've overloaded your Raspberry Pi - a Linux-based system with multitasking capabilities. You need to scale back your usage of its resources until there are enough available to satisfy the needs of this (or whatever) program.
It's not clear whether multithreading will help - however, I doubt it. It's not clear because you've only included an over-simplified version of your program. I would have to make a lot of wild guesses about what your real program does in order to think about making any suggestions of how to improve it.
Reading this answer (point 2) to a question related to Twisted's task.Clock for testing purposes, I found very weird that there is no way to advance the clock from t0 to t1 while catching all the callLater calls within t0 and t1.
Of course, you could solve this problem by doing something like:
clock = task.Clock()
reactor.callLater = clock.callLater
...
def advance_clock(total_elapsed, step=0.01):
elapsed = 0
while elapsed < total_elapsed:
clock.advance(step)
elapsed += step
...
time_to_advance = 10 # seconds
advance_clock(time_to_advance)
But then we have shifted the problem toward choosing a sufficiently small step, which could be very tricky for callLater calls that sample the time from a probability distribution, for instance.
Can anybody think of a solution to this problem?
I found very weird that there is no way to advance the clock from t0 to t1 while catching all the callLater calls within t0 and t1.
Based on what you wrote later in your question, I'm going to suppose that the case you're pointing out is the one demonstrated by the following example program:
from twisted.internet.task import Clock
def foo(reactor, n):
if n == 0:
print "Done!"
reactor.callLater(1, foo, reactor, n - 1)
reactor = Clock()
foo(reactor, 10)
reactor.advance(10)
One might expect this program to print Done! but it does not. If the last line is replaced with:
for i in range(10):
reactor.advance(1)
Then the resulting program does print Done!.
The reason Clock works this way is that it's exactly the way real clocks work. As far as I know, there are no computer clocks that operate with a continuous time system. I won't say it is impossible to implement a timed-event system on top of a clock with discrete steps such that it appears to offer a continuous flow of time - but I will say that Twisted makes no attempt to do so.
The only real difference between Clock and the real reactor implementations is that with Clock you can make the time-steps much larger than you are likely to encounter in typical usage of a real reactor.
However, it's quite possible for a real reactor to get into a situation where a very large chunk of time all passes in one discrete step. This could be because the system clock changes (there's some discussion of making it possible to schedule events independent of the system clock so that this case goes away) or it could be because some application code blocked the reactor for a while (actually, application code always blocks the reactor! But in typical programs it only blocks it for a period of time short enough for most people to ignore).
Giving Clock a way to mimic these large steps makes it possible to write tests for what your program does when one of these cases arises. For example, perhaps you really care that, when the kernel decides not to schedule your program for 2.1 seconds because of a weird quirk in the Linux I/O elevator algorithm, your physics engine nevertheless computes 2.1 seconds of physics even though 420 calls of your 200Hz simulation loop have been skipped.
It might be fair to argue that the default (standard? only?) time-based testing tool offered by Twisted should be somewhat more friendly towards the common case... Or not. Maybe that would encourage people to write programs that only work in the common case and break in the real world when the uncommon (but, ultimately, inevitable) case arises. I'm not sure.
Regarding Mike's suggestion to advance exactly to the next scheduled call, you can do this easily and without hacking any internals. clock.advance(clock.getDelayedCalls()[0].getTime() - clock.seconds()) will do exactly this (perhaps you could argue Clock would be better if it at least offered an obvious helper function for this to ease testing of the common case). Just remember that real clocks do not advance like this so if your code has a certain desirable behavior in your unit tests when you use this trick, don't be fooled into thinking this means that same desirable behavior will exist in real usage.
Given that the typical use-class for Twisted is to mix hardware events and timers I'm confused why you would want to do this, but...
My understanding is that interally Twisted is tracking callLater events via a number of lists that are inside of the reactor object (See: http://twistedmatrix.com/trac/browser/tags/releases/twisted-15.2.0/twisted/internet/base.py#L437 - the xxxTimedCalls lists inside of class ReactorBase)
I haven't done any work to figure out if those lists are exposed anywhere, but if you want to take the reactors life into your own hands I'm sure you could hack your way in.
With access to the timing lists you could simply forward time to whenever the next element of the list is ... though if your trying to test code that interacts with IO events, I can't imagine this is going to do anything but confuse you...
Best of luck
Here's a function that will advance the reactor to the next IDelayedCall by iterating over reactor.getDelayedCalls. This has the problem Mike mentioned of not catching IO events, so you can specify a minimum and maximum time that it should wait, as well as a maximum time step.
def advance_through_delayeds(reactor, min_t=None, max_t=None, max_step=None):
elapsed = 0
while True:
if max_t is not None and elapsed >= max_t:
break
try:
step = min(d.getTime() - reactor.seconds() for d in reactor.getDelayedCalls())
except ValueError:
# nothing else pending
if min_t is not None and elapsed < min_t:
step = min_t - elapsed
else:
break
if max_step is not None:
step = min(step, max_step)
if max_t is not None:
step = min(step, max_t-elapsed)
reactor.advance(step)
elapsed += step
return elapsed
If you need to wait for some I/O to complete, then set min_t and max_step to reasonable values.
# wait at least 10s, advancing the reactor by no more than 0.1s at a time
advance_through_delayeds(reactor, min_t=10, max_step=0.1)
If min_t is set, it will exit once getDelayedCalls returns an empty list after that time is reached.
It's probably a good idea to always set max_t to a sane value to prevent the test suite from hanging. For example, on the above foo function by JPC it does reach the print "Done!" statement, but then would hang forever as the callback chain never completes.
Using python for starting modelling/simulation runs (tuflow) and logging the runs to db
Currently on windows , python 2.7 and using timeit() .
Is it better to stick with using timeit() or to switch to using time.clock() ?
Simulation/modelling runs can be anything from a couple of minutes to a week+.
Need reasonably accurate runtimes.
I know time.clock has been depreciated on 3.3+ but I can't see this code getting moved to 3 for a long while.
self.starttime = timeit.default_timer()
run sim()
self.endtime = timeit.default_timer()
self.runtime = self.starttime - self.endtime
timeit's `timeit() function performs the code multiple times and takes the best, so it's not a great choice for lengthy runs. But your code just uses the timer, whose documentation warns that it measures elapsed time, so you will need to ensure standard running conditions in so far as you can. If you want to measure actual CPU usage, that's tricker.
When you say "reasonably accurate" times, one percent of two minutes is 2.4 seconds. The precision of time.clock() or the default timer is never going to be less than a second. Is that accurate enough?
To mitigate any possibility of migration to Python 3 you can define your own timing function. On Python 2 it can use time.clock() but at least you will only have one place to alter code if you do migrate.