Python, Raspberry pi, call a task every 10 milliseconds precisely - python

I'm currently trying to have a function called every 10ms to acquire data from a sensor.
Basically I was triggering the callback from a gpio interrupt but I changed my sensor and the one I'm currently using doesn't have a INT pin to drive the callback.
So my goal is to have the same behavior but with an internal interrupt generated by a timer.
I tried this from this topic
import threading
def work ():
threading.Timer(0.25, work).start ()
print(time.time())
print "stackoverflow"
work ()
But when I run it I can see that the timer is not really precise and it's deviating over time as you can see.
1494418413.1584847
stackoverflow
1494418413.1686869
stackoverflow
1494418413.1788757
stackoverflow
1494418413.1890721
stackoverflow
1494418413.1992736
stackoverflow
1494418413.2094712
stackoverflow
1494418413.2196639
stackoverflow
1494418413.2298684
stackoverflow
1494418413.2400634
stackoverflow
1494418413.2502584
stackoverflow
1494418413.2604961
stackoverflow
1494418413.270702
stackoverflow
1494418413.2808678
stackoverflow
1494418413.2910736
stackoverflow
1494418413.301277
stackoverflow
So the timer is deviating by 0.2 milliseconds every 10 milliseconds which is quite a big bias after few seconds.
I know that python is not really made for "real-time" but I think there should be a way to do it.
If someone already have to handle time constraints with python I would be glad to have some advices.
Thanks.

This code works on my laptop - logs the delta between target and actual time - main thing is to minimise what is done in the work() function because e.g. printing and scrolling screen can take a long time.
Key thing is to start the next timer based on difference between the time when that call is made and the target.
I slowed down the interval to 0.1s so it is easier to see the jitter which on my Win7 x64 can exceed 10ms which would cause problems with passing a negative value to thte Timer() call :-o
This logs 100 samples, then prints them - if you redirect to a .csv file you can load into Excel to display graphs.
from multiprocessing import Queue
import threading
import time
# this accumulates record of the difference between the target and actual times
actualdeltas = []
INTERVAL = 0.1
def work(queue, target):
# first thing to do is record the jitter - the difference between target and actual time
actualdeltas.append(time.clock()-target+INTERVAL)
# t0 = time.clock()
# print("Current time\t" + str(time.clock()))
# print("Target\t" + str(target))
# print("Delay\t" + str(target - time.clock()))
# print()
# t0 = time.clock()
if len(actualdeltas) > 100:
# print the accumulated deltas then exit
for d in actualdeltas:
print d
return
threading.Timer(target - time.clock(), work, [queue, target+INTERVAL]).start()
myQueue = Queue()
target = time.clock() + INTERVAL
work(myQueue, target)
Typical output (i.e. don't rely on millisecond timing on Windows in Python):
0.00947008617187
0.0029628920052
0.0121824719378
0.00582923077099
0.00131316206917
0.0105631524709
0.00437298744466
-0.000251418553351
0.00897956530515
0.0028528821332
0.0118192949105
0.00546301269675
0.0145723546788
0.00910063698529

I tried your solution but I got strange results.
Here is my code :
from multiprocessing import Queue
import threading
import time
def work(queue, target):
t0 = time.clock()
print("Target\t" + str(target))
print("Current time\t" + str(t0))
print("Delay\t" + str(target - t0))
print()
threading.Timer(target - t0, work, [queue, target+0.01]).start()
myQueue = Queue()
target = time.clock() + 0.01
work(myQueue, target)
And here is the output
Target 0.054099
Current time 0.044101
Delay 0.009998
Target 0.064099
Current time 0.045622
Delay 0.018477
Target 0.074099
Current time 0.046161
Delay 0.027937999999999998
Target 0.084099
Current time 0.0465
Delay 0.037598999999999994
Target 0.09409899999999999
Current time 0.046877
Delay 0.047221999999999986
Target 0.10409899999999998
Current time 0.047211
Delay 0.05688799999999998
Target 0.11409899999999998
Current time 0.047606
Delay 0.06649299999999997
So we can see that the target is increasing per 10ms and for the first loop, the delay for the timer seems to be good.
The point is instead of starting again at current_time + delay it start again at 0.045622 which represents a delay of 0.001521 instead of 0.01000
Did I missed something? My code seems to follow your logic isn't it?
Working example for #Chupo_cro
Here is my working example
from multiprocessing import Queue
import RPi.GPIO as GPIO
import threading
import time
import os
INTERVAL = 0.01
ledState = True
GPIO.setmode(GPIO.BCM)
GPIO.setup(2, GPIO.OUT, initial=GPIO.LOW)
def work(queue, target):
try:
threading.Timer(target-time.time(), work, [queue, target+INTERVAL]).start()
GPIO.output(2, ledState)
global ledState
ledState = not ledState
except KeyboardInterrupt:
GPIO.cleanup()
try:
myQueue = Queue()
target = time.time() + INTERVAL
work(myQueue, target)
except KeyboardInterrupt:
GPIO.cleanup()

Related

Change the Scheduling Time - Python

I am using schedule module to automatically run a function...
I am thinking of changing the scheduling time dynamically, but the solution is not success
Code -
import schedule
import pandas
from time import gmtime, strftime, sleep
import time
import random
time = 0.1
def a():
global time
print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
index = random.randint(1, 9)
print(index, time)
if(index==2):
time = 1
print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
schedule.every(time).minutes.do(a) #specify the minutes to automatically run the api
while True:
schedule.run_pending()
In this program, I scheduled the program to run every 6 seconds. And if the random integer - index value becomes 2, then the time variable is assigned as 1(1 minute). I checked, the time variable is changed to 1 after the random integer index becomes 2. The issue - After changing the time variable to 1, the scheduling still runs the function a() every 6 seconds not 1 minute.
How to change the scheduling time dynamically?
Thank you
After changing the time variable to 1, the scheduling still runs the function a() every 6 seconds not 1 minute.
This is because schedule.every(time).minutes.do(a) # specify the minutes to automatically run the api sets time to 6 seconds at beginning which does not change even if you change the value of that variable because that line has executed just once where value of time was 6 seconds at that execution.
How to change the scheduling time dynamically?
After reading DOCUMENTATION, I found nothing(I think) regarding changing time manually(when certain condition becomes satisfies) but it has built in Random Interval function where that function itself specifies random time within the range.
In your case you could do:
schedule.every(5).to(10).seconds.do(a)
The problem is that you cannot change time when certain condition satisfies.
Maybe there might be some way to fix that issue but could not figure out. And these information may help to investigate further to solve your problem.
I usually use custom schedulers, as they allow greater control and are also less memory intensive. The variable "time" needs to be shared between processes. This is where Manager().Namespace() comes to rescue. It talks 'between' processes.
import time
import random
from multiprocessing import Process, Manager
ns = Manager().Namespace()
ns.time = 0.1
processes = []
def a():
print(time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime()))
index = random.randint(1, 4)
if(index==2):
ns.time = 1
print(index, ns.time)
while True:
try:
s = time.time() + ns.time*60
for x in processes:
if not x.is_alive():
x.join()
processes.remove(x)
print('Sleeping :',round(s - time.time()))
time.sleep(round(s - time.time()))
p = Process(target = a)
p.start()
processes.append(p)
except:
print('Killing all to prevent orphaning ...')
[p.terminate() for p in processes]
[processes.remove(p) for p in processes]
break

Python - Accurate time.sleep

I am working on a project which accurate timer is really crucial. I am working on python and am using timer.sleep() function.
I noticed that timer.sleep() function will add additional delay because of the scheduling problem (refer to timer.sleep docs). Due to that issue, the longer my program runs, the more inaccurate the timer is.
Is there any more accurate timer/ticker to sleep the program or solution for this problem?
Any help would be appreciated. Cheers.
I had a solution similar to above, but it became processor heavy very quickly. Here is a processor-heavy idea and a workaround.
def processor_heavy_sleep(ms): # fine for ms, starts to work the computer hard in second range.
start = time.clock()
end = start + ms /1000.
while time.clock() < end:
continue
return start, time.clock()
def efficient_sleep(secs, expected_inaccuracy=0.5): # for longer times
start = time.clock()
end = secs + start
time.sleep(secs - expected_inaccuracy)
while time.clock() < end:
continue
return start, time.clock()
output of efficient_sleep(5, 0.5) 3 times was:
(3.1999303695151594e-07, 5.0000003199930365)
(5.00005983869791, 10.00005983869791)
(10.000092477987678, 15.000092477987678)
This is on windows. I'm running it for 100 loops right now. Here are the results.
(485.003749358414, 490.003749358414)
(490.0037919174879, 495.0037922374809)
(495.00382903668014, 500.00382903668014)
The sleeps remain accurate, but the calls are always delayed a little. If you need a scheduler that accurately calls every xxx secs to the millisecond, that would be a different thing.
the longer my program runs, the more inaccurate the timer is.
So, for example by expecting 0.5s delay, it will be time.sleep(0.5 - (start-end)). But still didn't solve the issue
You seem to be complaining about two effects, 1) the fact that timer.sleep() may take longer than you expect, and 2) the inherent creep in using a series of timer.sleep() calls.
You can't do anything about the first, short of switching to a real-time OS. The underlying OS calls are defined to sleep for at least as long as requested. They only guarantee that you won't wake early; they make no guarantee that you won't wake up late.
As for the second, you ought to figure your sleep time according to an unchanging epoch, not from your wake-up time. For example:
import time
import random
target = time.time()
def myticker():
# Sleep for 0.5s between tasks, with no creep
target += 0.5
now = time.time()
if target > now:
time.sleep(target - now)
def main():
previous = time.time()
for _ in range(100):
now = time.time()
print(now - previous)
previous = now
# simulate some work
time.sleep(random.random() / 10) # Always < tick frequency
# time.sleep(random.random()) # Not always < tick frequency
myticker()
if __name__ == "__main__":
main()
Working on Linux with zero knowledge of Windows, I may be being naive here but is there some reason that writing your own sleep function, won't work for you?
Something like:
import time
def sleep_time():
start_time = time.time()
while (time.time() - start_time) < 0.0001:
continue
end_time = time.time() + 60 # run for a minute
cnt = 0
while time.time() < end_time:
cnt += 1
print('sleeping',cnt)
sleep_time()
print('Awake')
print("Slept ",cnt," Times")

Accurate sleep/delay within Python while loop

I have a while True loop which sends variables to an external function, and then uses the returned values. This send/receive process has a user-configurable frequency, which is saved and read from an external .ini configuration file.
I've tried time.sleep(1 / Frequency), but am not satisfied with the accuracy, given the number of threads being used elsewhere. E.g. a frequency of 60Hz (period of 0.0166667) is giving an 'actual' time.sleep() period of ~0.0311.
My preference would be to use an additional while loop, which compares the current time to the start time plus the period, as follows:
EndTime = time.time() + (1 / Frequency)
while time.time() - EndTime < 0:
sleep(0)
This would fit into the end of my while True function as follows:
while True:
A = random.randint(0, 5)
B = random.randint(0, 10)
C = random.randint(0, 20)
Values = ExternalFunction.main(Variable_A = A, Variable_B = B, Variable_C = C)
Return_A = Values['A_Out']
Return_B = Values['B_Out']
Return_C = Values['C_Out']
#Updated other functions with Return_A, Return_B and Return_C
EndTime = time.time() + (1 / Frequency)
while time.time() - EndTime < 0:
time.sleep(0)
I'm missing something, as the addition of the while loop causes the function to execute once only. How can I get the above to function correctly? Is this the best approach to 'accurate' frequency control on a non-real time operating system? Should I be using threading for this particular component? I'm testing this function on both Windows 7 (64-bit) and Ubuntu (64-bit).
If I understood your question correctly, you want to execute ExternalFunction.main at a given frequency. The problem is that the execution of ExternalFunction.main itself takes some time. If you don't need very fine precision -- it seems that you don't -- my suggestion is doing something like this.
import time
frequency = 1 # Hz
period = 1.0/frequency
while True:
time_before = time.time()
[...]
ExternalFunction.main([...])
[...]
while (time.time() - time_before) < period:
time.sleep(0.001) # precision here
You may tune the precision to your needs. Greater precision (smaller number) will make the inner while loop execute more often.
This achieves decent results when not using threads. However, when using Python threads, the GIL (Global Interpreter Lock) makes sure only one thread runs at a time. If you have a huge number of threads it may be that it is taking way too much time for the program to go back to your main thread. Increasing the frequency Python changes between threads may give you more accurate delays.
Add this to the beginning of your code to increase the thread switching frequency.
import sys
sys.setcheckinterval(1)
1 is the number of instructions executed on each thread before switching (the default is 100), a larger number improves performance but will increase the threading switching time.
You may want to try python-pause
Pause until a unix time, with millisecond precision:
import pause
pause.until(1370640569.7747359)
Pause using datetime:
import pause, datetime
dt = datetime.datetime(2013, 6, 2, 14, 36, 34, 383752)
pause.until(dt)
You may use it like:
freqHz=60.0
td=datetime.timedelta(seconds=1/freqHz)
dt=datetime.now()
while true:
#Your code here
dt+=td
pause.until(dt)
Another solution for an accurate delay is to use the perf_counter() function from module time. Especially useful in windows as time.sleep is not accurate in milliseconds. See below example where function accurate_delay creates a delay in milliseconds.
import time
def accurate_delay(delay):
''' Function to provide accurate time delay in millisecond
'''
_ = time.perf_counter() + delay/1000
while time.perf_counter() < _:
pass
delay = 10
t_start = time.perf_counter()
print('Wait for {:.0f} ms. Start: {:.5f}'.format(delay, t_start))
accurate_delay(delay)
t_end = time.perf_counter()
print('End time: {:.5f}. Delay is {:.5f} ms'.
format(t_end, 1000*(t_end - t_start)))
sum = 0
ntests = 1000
for _ in range(ntests):
t_start = time.perf_counter()
accurate_delay(delay)
t_end = time.perf_counter()
print('Test completed: {:.2f}%'.format(_/ntests * 100), end='\r', flush=True)
sum = sum + 1000*(t_end - t_start) - delay
print('Average difference in time delay is {:.5f} ms.'.format(sum/ntests))`

Python numpy.fft very slow (10x slower) when run in subprocess

I've found that numpy.fft.fft (and its variants) very slow when run in the background. Here is an example of what I'm talking about
import numpy as np
import multiprocessing as mproc
import time
import sys
# the producer function, which will run in the background and produce data
def Producer(dataQ):
numFrames = 5
n = 0
while n < numFrames:
data = np.random.rand(3000, 200)
dataQ.put(data) # send the datta to the consumer
time.sleep(0.1) # sleep for 0.5 second, so we dont' overload CPU
n += 1
# the consumer function, which will run in the backgrounnd and consume data from the producer
def Consumer(dataQ):
while True:
data = dataQ.get()
t1 = time.time()
fftdata = np.fft.rfft(data, n=3000*5)
tDiff = time.time() - t1
print("Elapsed time is %0.3f" % tDiff)
time.sleep(0.01)
sys.stdout.flush()
# the main program if __name__ == '__main__': is necessary to prevent this code from being run
# only when this program is started by user
if __name__ == '__main__':
data = np.random.rand(3000, 200)
t1 = time.time()
fftdata = np.fft.rfft(data, n=3000*5, axis=0)
tDiff = time.time() - t1
print("Elapsed time is %0.3f" % tDiff)
# generate a queue for transferring data between the producedr and the consumer
dataQ = mproc.Queue(4)
# start up the processoso
producerProcess = mproc.Process(target=Producer, args=[dataQ], daemon=False)
consumerProcess = mproc.Process(target=Consumer, args=[dataQ], daemon=False)
print("starting up processes")
producerProcess.start()
consumerProcess.start()
time.sleep(10) # let program run for 5 seconds
producerProcess.terminate()
consumerProcess.terminate()
The output it produes on my machine:
Elapsed time is 0.079
starting up processes
Elapsed time is 0.859
Elapsed time is 0.861
Elapsed time is 0.878
Elapsed time is 0.863
Elapsed time is 0.758
As you can see, it is roughly 10x slower when run in the background, and I can't figure out why this would be the case. The time.sleep() calls should ensure that the other process (the main process and producer process) aren't doing anything when the FFT is being computed, so it should use all the cores. I've checked CPU utilization through Windows Task Manager and it seems to use up about 25% when numpy.fft.fft is called heavily in both the single process and multiprocess cases.
Anyone have an idea whats going on?
The main problem is that your fft call in the background thread is:
fftdata = np.fft.rfft(data, n=3000*5)
rather than:
fftdata = np.fft.rfft(data, n=3000*5, axis=0)
which for me made all the difference.
There are a few other things worth noting. Rather than having the time.sleep() everywhere, why not just let the processor take care of this itself? Further more, rather than suspending the main thread, you can use
consumerProcess.join()
and then have the producer process run dataQ.put(None) once it has finished loading the data, and break out of the loop in the consumer process, i.e.:
def Consumer(dataQ):
while True:
data = dataQ.get()
if(data is None):
break
...

Implement sub millisecond processing in python without busywait

How would i implement processing of an array with millisecond precision using python under linux (running on a single core Raspberry Pi).
I am trying to parse information from a MIDI file, which has been preprocessed to an array where each millisecond i check if the array has entries at the current timestamp and trigger some functions if it does.
Currently i am using time.time() and employ busy waiting (as concluded here). This eats up all the CPU, therefor i opt for a better solution.
# iterate through all milliseconds
for current_ms in xrange(0, last+1):
start = time()
# check if events are to be processed
try:
events = allEvents[current_ms]
# iterate over all events for this millisecond
for event in events:
# check if event contains note information
if 'note' in event:
# check if mapping to pin exists
if event['note'] in mapping:
pin = mapping[event['note']]
# check if event contains on/off information
if 'mode' in event:
if event['mode'] == 0:
pin_off(pin)
elif event['mode'] == 1:
pin_on(pin)
else:
debug("unknown mode in event:"+event)
else:
debug("no mapping for note:" + event['note'])
except:
pass
end = time()
# fill the rest of the millisecond
while (end-start) < (1.0/(1000.0)):
end = time()
where last is the millisecond of the last event (known from preprocessing)
This is not a question about time() vs clock() more about sleep vs busy wait.
I cant really sleep in the "fill rest of millisecond" loop, because of the too low accuracy of sleep(). If i were to use ctypes, how would i go about it correctly?
Is there some Timer library which would call a callback each millisecond reliably?
My current implementation is on GitHub. With this approach i get a skew of around 4 or 5ms on the drum_sample, which is 3.7s total (with mock, so no real hardware attached). On a 30.7s sample, the skew is around 32ms (so its at least not linear!).
I have tried using time.sleep() and nanosleep() via ctypes with the following code
import time
import timeit
import ctypes
libc = ctypes.CDLL('libc.so.6')
class Timespec(ctypes.Structure):
""" timespec struct for nanosleep, see:
http://linux.die.net/man/2/nanosleep """
_fields_ = [('tv_sec', ctypes.c_long),
('tv_nsec', ctypes.c_long)]
libc.nanosleep.argtypes = [ctypes.POINTER(Timespec),
ctypes.POINTER(Timespec)]
nanosleep_req = Timespec()
nanosleep_rem = Timespec()
def nsleep(us):
#print('nsleep: {0:.9f}'.format(us))
""" Delay microseconds with libc nanosleep() using ctypes. """
if (us >= 1000000):
sec = us/1000000
us %= 1000000
else: sec = 0
nanosleep_req.tv_sec = sec
nanosleep_req.tv_nsec = int(us * 1000)
libc.nanosleep(nanosleep_req, nanosleep_rem)
LOOPS = 10000
def do_sleep(min_sleep):
#print('try: {0:.9f}'.format(min_sleep))
total = 0.0
for i in xrange(0, LOOPS):
start = timeit.default_timer()
nsleep(min_sleep*1000*1000)
#time.sleep(min_sleep)
end = timeit.default_timer()
total += end - start
return (total / LOOPS)
iterations = 5
iteration = 1
min_sleep = 0.001
result = None
while True:
result = do_sleep(min_sleep)
#print('res: {0:.9f}'.format(result))
if result > 1.5 * min_sleep:
if iteration > iterations:
break
else:
min_sleep = result
iteration += 1
else:
min_sleep /= 2.0
print('FIN: {0:.9f}'.format(result))
The result on my i5 is
FIN: 0.000165443
while on the RPi it is
FIN: 0.000578617
which suggest a sleep period of about 0.1 or 0.5 milliseconds, with the given jitter (tend to sleep longer) that at most helps me reduce the load a little bit.
One possible solution, using the sched module:
import sched
import time
def f(t0):
print 'Time elapsed since t0:', time.time() - t0
s = sched.scheduler(time.time, time.sleep)
for i in range(10):
s.enterabs(t0 + 10 + i, 0, f, (t0,))
s.run()
Result:
Time elapsed since t0: 10.0058200359
Time elapsed since t0: 11.0022959709
Time elapsed since t0: 12.0017120838
Time elapsed since t0: 13.0022599697
Time elapsed since t0: 14.0022521019
Time elapsed since t0: 15.0015859604
Time elapsed since t0: 16.0023040771
Time elapsed since t0: 17.0023028851
Time elapsed since t0: 18.0023078918
Time elapsed since t0: 19.002286911
Apart from some constant offset of about 2 millisecond (which you could calibrate), the jitter seems to be on the order of 1 or 2 millisecond (as reported by time.time itself). Not sure if that is good enough for your application.
If you need to do some useful work in the meantime, you should look into multi-threading or multi-processing.
Note: a standard Linux distribution that runs on a RPi is not a hard real-time operating system. Also Python can show non-deterministic timing, e.g. when it starts a garbage collection. So your code might run fine with low jitter most of the time, but you might have occasional 'hickups', where there is a bit of delay.

Categories