We Keep Coding

Accurate timer with PyQt

I'm using pyqtgraph to plot a huge number of data that I receive from sensors.
To do so, I made one thread that acquire the data and put in a queue. To plot the data, I check periodically with a timer if the queue is not empty.
The problem is that the accuracy of the timer (QTimer) seems to be really bad. I mean when the load is low (sleep for 1000/100 ms) in the measuring thread, the accuracy is pretty good but when the load increase (sleep for 10ms), my update function used to plot data is not called back with the same period.
Here is an example code:
import sys
import time
from queue import Queue
from random import random
import numpy as np
import pyqtgraph as pg
from pyqtgraph.Qt import QtCore, QtWidgets
data_queue = Queue()
class WorkerThread(QtCore.QThread):
def __init__(self, parent):
super(WorkerThread, self).__init__(parent=parent)
def run(self):
t_init = time.time()
while True:
# Generating random data
values = [(time.time()-t_init, random()) for _ in range(200)]
data_queue.put(values)
print("adding data")
self.msleep(10)
class GraphPlot(QtWidgets.QMainWindow):
def __init__(self, parent=None):
super(GraphPlot, self).__init__(parent)
self.mainbox = QtWidgets.QWidget()
self.setCentralWidget(self.mainbox)
self.mainbox.setLayout(QtWidgets.QVBoxLayout())
self.canvas = pg.GraphicsLayoutWidget()
self.mainbox.layout().addWidget(self.canvas)
self.analogPlot = self.canvas.addPlot(title='Real-time data')
self.drawplot = self.analogPlot.plot(pen='r')
numPoints = 20000
self.t = np.zeros(numPoints, dtype=int)
self.x = np.zeros(numPoints, dtype=int)
self.worker = WorkerThread(self)
self.worker.start()
self.timer = pg.QtCore.QTimer()
self.timer.setTimerType(QtCore.Qt.PreciseTimer)
self.timer.timeout.connect(self._update)
self.timer.start(1)
def _update(self):
print('start:', time.time())
size = data_queue.qsize()
if size > 0:
for _ in range(size):
values = data_queue.get()
for v in values:
self.t = np.append(self.t[1:], v[0])
self.x = np.append(self.x[1:], v[1])
self.drawplot.setData(self.t, self.x)
print('end:', time.time())
app = QtWidgets.QApplication(sys.argv)
plot = GraphPlot()
plot.show()
sys.exit(app.exec_())
An excerpt of the output:
start: 1572893919.9067862
adding data
end: 1572893919.9217482 <--
adding data
start: 1572893919.9586473 <-- there should be 1ms of difference with last 'end'
actually, there is around 37ms
I want the timer to be synchronous with the same period whatever the load on the measuring thread. I tried to decrease the priority of the former thread but it did not solve the problem.

QTimer documentation partially answers to your issue:
All timer types may time out later than expected if the system is busy
or unable to provide the requested accuracy. In such a case of timeout
overrun, Qt will emit timeout() only once, even if multiple timeouts
have expired, and then will resume the original interval.
The problem is that after you call _update from the timeout, Qt will need some time to process what happens after self.drawplot.setData(), which basically is computing graphical information and actually painting it on the screen.
You're not getting the 1ms delay because Qt just isn't able to work that fast.
Even if a QTimer can work in another thread ("asynchronously", but be careful about the meaning of this word), it always depend on the thread it is created or resides (a QTimer cannot be started or stopped from a thread different than its one). So, since you've created the timer in the window thread (the Qt main event loop), its timeout accuracy depends on the capacity of that loop to handle all its events, and since lot of events are related to GUI painting (which seems fast to our eyes, but is actually slow as it's very demanding for the CPU), you can easily understand why you'll never get that 1ms interval. And don't forget the fact that even if pyqtgraph is very fast we're still talking about Python.
While reaching a better accuracy for a 1ms QTimer is possible (creating a separate thread for it), you wouldn't get any advantage from it anyway: even with a very fast computer, what you're substantially requesting is to update the screen at 1000Hz, while most graphic hardware is not able to go much faster than 100-200Hz; this means that even if you own a high end system, you wouldn't get more than one update each 4ms.
If you need to update the plot each time new data is available, it is probably better to use signals and slots, which also avoids any unnecessary check on the queue and ensures to update the plot as much as needed:
class WorkerThread(QtCore.QThread):
newData = QtCore.pyqtSignal(object)
def __init__(self, parent):
super(WorkerThread, self).__init__(parent=parent)
def run(self):
t_init = time.time()
while True:
# Generating random data
values = [(time.time()-t_init, random()) for _ in range(200)]
print("adding data")
self.newData.emit(values)
self.msleep(10)
class GraphPlot(QtWidgets.QMainWindow):
def __init__(self, parent=None):
super(GraphPlot, self).__init__(parent)
self.mainbox = QtWidgets.QWidget()
self.setCentralWidget(self.mainbox)
self.mainbox.setLayout(QtWidgets.QVBoxLayout())
self.canvas = pg.GraphicsLayoutWidget()
self.mainbox.layout().addWidget(self.canvas)
self.analogPlot = self.canvas.addPlot(title='Real-time data')
self.drawplot = self.analogPlot.plot(pen='r')
numPoints = 20000
self.t = np.zeros(numPoints, dtype=int)
self.x = np.zeros(numPoints, dtype=int)
self.worker = WorkerThread(self)
self.worker.newData.connect(self.newData)
self.worker.start()
def newData(self, data):
print('start:', time.time())
for v in data:
self.t = np.append(self.t[1:], v[0])
self.x = np.append(self.x[1:], v[1])
self.drawplot.setData(self.t, self.x)
print('end:', time.time())
You won't get a 1ms update, but there would be no need for it anyway; also, remember that printing at that rate will always affect the performance in some way.
Finally, there is no advantage in setting PreciseTimer with a 1ms interval, since the timer accuracy is about 1ms on most platforms anyway (as explained at the beginning of the same paragraph in the documentation linked before), and setting the precision is only required for longer intervals (I'd say at least 25-50ms).
There's also an interesting answer about QTimer here, explaining what basically happens whenever you create one and it timeouts.

Run thread in loop only if it's not currently running in Python

I'm trying to separate my logic (function or task) from main loop. This function usually takes long time to run. In following example calculate() function takes a lot of time to calculate self.time variable.
In other words I want calculate() to be called asynchronously. In every iteration first I would check if calculate() is running, if not then call it. Secondly I would call show() function in every iteration, no matter if calculate() changed self.time value.
import time
import datetime
class Clock:
def __init__(self):
self.time = None
def calculate(self):
time.sleep(3)
self.time = datetime.datetime.now()
def show(self):
print(f"{self.time.minute}:{self.time.second}")
def loop(self):
while True:
self.calculate() # this should be asynchronous
self.show() # this should be called every iteration
if __name__ == '__main__':
clock = Clock()
clock.loop()
Output:
36:9
36:12
36:15
36:18
This is not a wanted outcome. What I want:
36:9
...
36:9
36:9
36:12
...
36:12
36:15
36:15
36:15
...
36:15
36:18
36:18
...
36:18
How I've tried to solve this (this probably not the best solution, so future reader please check answers):
import time
import datetime
import threading
class Clock:
def __init__(self):
self.time = datetime.datetime.now()
self.__thread = None
def calculate(self):
time.sleep(3)
self.time = datetime.datetime.now() # race condition?
def show(self):
print(f"{self.time.minute}:{self.time.second}")
def loop(self):
while True:
if self.__thread is None or not self.__thread.is_alive():
self.__thread = threading.Thread(target=self.calculate)
self.__thread.start()
self.show()
if __name__ == '__main__':
clock = Clock()
clock.loop()
Program output is what I wanted. Is there any flaw in this solution? I can think only of race condition.
I am aware that's not the greatest description of this problem. If you
could help me with editing this post with more searchable and specific
vocabulary I would be grateful. If you have idea for better title please
leave it in comments.

I checked your code in my IDE, and it looks like exactly what you wanted using an acceptable approach. The only alternative solution I can think of is: 1) use a temp variable that goes up by one until it gets to 3 then reset. 2) similar idea but use time.nextTime = 3 to jump every 3 seconds and print current time.
Also, your first code solution seems to take some time to run is because it's waiting 3 seconds to show the time. And regarding race condition, that's not an issue since you're not using multiple threads at the same time in your second solution.

PyQt Progress Bar Update with Threads

I have been writing a program which runs a remote script on server. So, I need to show the progress with a bar but somehow when I run my code, GUI start to freeze. I have used QThread and SIGNAL but unfortunately couldnt be succedeed.
Here is my code below;
class dumpThread(QThread):
def __init__(self):
QThread.__init__(self)
def __del__(self):
self.wait()
def sendEstablismentCommands(self, connection):
# Commands are sending sequently with proper delay-timers #
connection.sendShell("telnet localhost 21000")
time.sleep(0.5)
connection.sendShell("admin")
time.sleep(0.5)
connection.sendShell("admin")
time.sleep(0.5)
connection.sendShell("cd imdb")
time.sleep(0.5)
connection.sendShell("dump subscriber")
command = input('$ ')
def run(self):
# your logic here
# self.emit(QtCore.SIGNAL('THREAD_VALUE'), maxVal)
self.sendEstablismentCommands(connection)
class progressThread(QThread):
def __init__(self):
QThread.__init__(self)
def __del__(self):
self.wait()
def run(self):
# your logic here
while 1:
maxVal = 100
self.emit(SIGNAL('PROGRESS'), maxVal)
class Main(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.ui = Ui_MainWindow()
self.ui.setupUi(self)
self.ui.connectButton.clicked.connect(self.connectToSESM)
def connectToSESM(self):
## Function called when pressing connect button, input are being taken from edit boxes. ##
## dumpThread() method has been designed for working thread seperate from GUI. ##
# Connection data are taken from "Edit Boxes"
# username has been set as hardcoded
### Values Should Be Defined As Global ###
username = "ntappadm"
password = self.ui.passwordEdit.text()
ipAddress = self.ui.ipEdit.text()
# Connection has been established through paramiko shell library
global connection
connection = pr.ssh(ipAddress, username, password)
connection.openShell()
pyqtRemoveInputHook() # For remove unnecessary items from console
global get_thread
get_thread = dumpThread() # Run thread - Dump Subscriber
self.progress_thread = progressThread()
self.progress_thread.start()
self.connect(self.progress_thread, SIGNAL('PROGRESS'), self.updateProgressBar)
get_thread.start()
def updateProgressBar(self, maxVal):
for i in range(maxVal):
self.ui.progressBar.setValue(self.ui.progressBar.value() + 1)
time.sleep(1)
maxVal = maxVal - 1
if maxVal == 0:
self.ui.progressBar.setValue(100)
def parseSubscriberList(self):
parsing = reParser()
def done(self):
QtGui.QMessageBox.information(self, "Done!", "Done fetching posts!")
if __name__ == "__main__":
app = QtGui.QApplication(sys.argv)
main = Main()
main.show()
sys.exit(app.exec_())
I am expecting to see updateProgressBar method has called with SIGNAL, so process goes through seperate thread. I coudlnt find where I am missing.
Thanks for any help

There are really two problems. One thing I have noticed is that Python threads are greedy if they are not used for IO operations like reading from a serial port. If you tell a thread to run a calculation or something that is not IO related a thread will take up all of the processing and doesn't like to let the main thread / event loop run. The second problem is that signals are slow ... very slow. I've noticed that if you emit a signal from a thread and do it very fast it can drastically slow down a program.
So at the heart of the issue, the thread is taking up all of the time and you are emitting a signal very very fast which will cause slow downs.
For clarity and ease of use I would use the new style signal and slots.
http://pyqt.sourceforge.net/Docs/PyQt4/new_style_signals_slots.html
class progressThread(QThread):
progress_update = QtCore.Signal(int) # or pyqtSignal(int)
def __init__(self):
QThread.__init__(self)
def __del__(self):
self.wait()
def run(self):
# your logic here
while 1:
maxVal = 100
self.progress_update.emit(maxVal) # self.emit(SIGNAL('PROGRESS'), maxVal)
# Tell the thread to sleep for 1 second and let other things run
time.sleep(1)
To connect to the new style signal
...
self.progress_thread.start()
self.process_thread.progress_update.connect(self.updateProgressBar) # self.connect(self.progress_thread, SIGNAL('PROGRESS'), self.updateProgressBar)
...
EDIT
Sorry there is another problem. When a signal calls a function you cannot stay in that function forever. That function is not running in a separate thread it is running on the main event loop and the main event loop waits to run until you exit that function.
Update progress sleeps for 1 second and keeps looping. The hanging is coming from staying in this function.
def updateProgressBar(self, maxVal):
for i in range(maxVal):
self.ui.progressBar.setValue(self.ui.progressBar.value() + 1)
time.sleep(1)
maxVal = maxVal - 1
if maxVal == 0:
self.ui.progressBar.setValue(100)
It would be better to write the progress bar like
class progressThread(QThread):
progress_update = QtCore.Signal(int) # or pyqtSignal(int)
def __init__(self):
QThread.__init__(self)
def __del__(self):
self.wait()
def run(self):
# your logic here
while 1:
maxVal = 1 # NOTE THIS CHANGED to 1 since updateProgressBar was updating the value by 1 every time
self.progress_update.emit(maxVal) # self.emit(SIGNAL('PROGRESS'), maxVal)
# Tell the thread to sleep for 1 second and let other things run
time.sleep(1)
def updateProgressBar(self, maxVal):
self.ui.progressBar.setValue(self.ui.progressBar.value() + maxVal)
if maxVal == 0:
self.ui.progressBar.setValue(100)

Getting the time remaining before event.wait is done in Python

I'm writing a Python script in which i have a thread running that calculates some values and creates a graph every hour. What I would like to do is have a function in that thread that tells me how much time there is remaining before the next update happens. My current implementation is as follows:
class StatsUpdater(threading.Thread):
def __init__(self, updateTime):
threading.Thread.__init__(self)
self.event = threading.Event()
self.updateTime = updateTime
def run(self):
while not self.event.is_set():
self.updateStats()
self.event.wait(self.updateTime)
def updateStats(self):
print "Updating Stats"
tables = SQLInterface.listTables()
for table in tables:
PlotTools.createAndSave(table)
def stop(self):
self.event.set()
So what i would like is adding another function in that class that gives me back the time remaining gefore self.event.wait(self.updateTime) times out, something like this:
def getTimeout(self):
return self.event.timeRemaining()
Is this possible somehow?

There's no support for getting the remaining time directly but you can sleep several times and keep track of how much time remains.
def __init__(self, updateTime):
threading.Thread.__init__(self)
self.event = threading.Event()
self.updateTime = updateTime
self.wait_time=None
def run(self):
while not self.event.is_set():
self.updateStats()
try:
self.wait_time=self.updateTime
inttime=int(self.updateTime)
remaining=inttime-self.updateTime
self.event.wait(remaining)
for t in reversed(range(inttime)):
self.wait_time=t+1
self.event.wait(1)
finally:
self.wait_time=0
And then use
def getTimeout(self):
return self.wait_time

Alright, i have a compromis to my problem. I implemented a variable in StatsUpdater.run:
self.lastUpdateTime = int(time.time())
right before i do the update function.
Now when I call getTimeout(), I do:
def getTimeout(self):
timePassed = int(time.time() - self.lastUpdateTime
return self.updateTime - timePassed
This way, I don't have a calculation intensive thread running and calculation
a small sum every second but i still get a pretty good indication of when the next update is since the ammount of time between updates is also known ;)

Equivalent of setInterval in python

I have recently posted a question about how to postpone execution of a function in Python (kind of equivalent to Javascript setTimeout) and it turns out to be a simple task using threading.Timer (well, simple as long as the function does not share state with other code, but that would create problems in any event-driven environment).
Now I am trying to do better and emulate setInterval. For those who are not familiar with Javascript, setInterval allows to repeat a call to a function every x seconds, without blocking the execution of other code. I have created this example decorator:
import time, threading
def setInterval(interval, times = -1):
# This will be the actual decorator,
# with fixed interval and times parameter
def outer_wrap(function):
# This will be the function to be
# called
def wrap(*args, **kwargs):
# This is another function to be executed
# in a different thread to simulate setInterval
def inner_wrap():
i = 0
while i != times:
time.sleep(interval)
function(*args, **kwargs)
i += 1
threading.Timer(0, inner_wrap).start()
return wrap
return outer_wrap
to be used as follows
#setInterval(1, 3)
def foo(a):
print(a)
foo('bar')
# Will print 'bar' 3 times with 1 second delays
and it seems to me it is working fine. My problem is that
it seems overly complicated, and I fear I may have missed a simpler/better mechanism
the decorator can be called without the second parameter, in which case it will go on forever. When I say foreover, I mean forever - even calling sys.exit() from the main thread will not stop it, nor will hitting Ctrl+c. The only way to stop it is to kill python process from the outside. I would like to be able to send a signal from the main thread that would stop the callback. But I am a beginner with threads - how can I communicate between them?
EDIT In case anyone wonders, this is the final version of the decorator, thanks to the help of jd
import threading
def setInterval(interval, times = -1):
# This will be the actual decorator,
# with fixed interval and times parameter
def outer_wrap(function):
# This will be the function to be
# called
def wrap(*args, **kwargs):
stop = threading.Event()
# This is another function to be executed
# in a different thread to simulate setInterval
def inner_wrap():
i = 0
while i != times and not stop.isSet():
stop.wait(interval)
function(*args, **kwargs)
i += 1
t = threading.Timer(0, inner_wrap)
t.daemon = True
t.start()
return stop
return wrap
return outer_wrap
It can be used with a fixed amount of repetitions as above
#setInterval(1, 3)
def foo(a):
print(a)
foo('bar')
# Will print 'bar' 3 times with 1 second delays
or can be left to run until it receives a stop signal
import time
#setInterval(1)
def foo(a):
print(a)
stopper = foo('bar')
time.sleep(5)
stopper.set()
# It will stop here, after printing 'bar' 5 times.

Your solution looks fine to me.
There are several ways to communicate with threads. To order a thread to stop, you can use threading.Event(), which has a wait() method that you can use instead of time.sleep().
stop_event = threading.Event()
...
stop_event.wait(1.)
if stop_event.isSet():
return
...
For your thread to exit when the program is terminated, set its daemon attribute to True before calling start(). This applies to Timer() objects as well because they subclass threading.Thread. See http://docs.python.org/library/threading.html#threading.Thread.daemon

Maybe these are the easiest setInterval equivalent in python:
import threading
def set_interval(func, sec):
def func_wrapper():
set_interval(func, sec)
func()
t = threading.Timer(sec, func_wrapper)
t.start()
return t

Maybe a bit simpler is to use recursive calls to Timer:
from threading import Timer
import atexit
class Repeat(object):
count = 0
#staticmethod
def repeat(rep, delay, func):
"repeat func rep times with a delay given in seconds"
if Repeat.count < rep:
# call func, you might want to add args here
func()
Repeat.count += 1
# setup a timer which calls repeat recursively
# again, if you need args for func, you have to add them here
timer = Timer(delay, Repeat.repeat, (rep, delay, func))
# register timer.cancel to stop the timer when you exit the interpreter
atexit.register(timer.cancel)
timer.start()
def foo():
print "bar"
Repeat.repeat(3,2,foo)
atexit allows to signal stopping with CTRL-C.

this class Interval
class ali:
def __init__(self):
self.sure = True;
def aliv(self,func,san):
print "ali naber";
self.setInterVal(func, san);
def setInterVal(self,func, san):
# istenilen saniye veya dakika aralığında program calışır.
def func_Calistir():
func(func,san); #calışıcak fonksiyon.
self.t = threading.Timer(san, func_Calistir)
self.t.start()
return self.t
a = ali();
a.setInterVal(a.aliv,5);