Optimize Matplotlib plot that dynamically updates and remains interactive - python

Ok, so as the question suggests, I am chasing the best method to create a plot that:
1) Is independent of the main program/script, therefore making changes to the plot will have no adverse (locking or other) effect on the main program/script
2) Can be interacted with via matplotlib's default GUI or other custom GUI
Now, what I have so far for the above criteria:
1) Use the multiprocessing module and its multiprocessing Queue to pass X,Y information to a separate process that appends to current X,Y data and refreshes the plot. This basically addresses criteria 1.
2) Ensuring matplotlib's interactive mode is turned on (ion()), an infinite while True: loop checks the aforementioned queue and if there is no X,Y information, allow's processing of GUI events via pyplot.pause(interval in seconds)
To simplify things, below is my code to accept incoming X,Y data and plot it in pseudocode. This code is run in a different process with a Queue feeding it X,Y data.
from matplotlib import pyplot
def run(self):
p = pyplot
while True:
if there's nothing in the Queue:
redraw and allow GUI events (p.pause(0.0001))
continue
else:
command = get X,Y info from Queue
if command equals X,Y data:
update plots x data (axes.set_ydata())
update plots y data (axes.set_ydata())
update axis data limits (axes.relim())
update axis view limits (axes.autoscale_view())
redraw and allow GUI events i.e. interaction for 0.0001 seconds (p.pause(0.0001))
elif if command equals the "exit loop" value:
break
turn of interactive mode (p.ioff())
keep plot open and block process until closed by user (p.show())
So can the above be optimized to increase interactivity with the figure GUI? potentially by allowing GUI events to be serviced independently of the plot being updated? This method becomes slower the more updates it has to do (i.e. if there are more plots on the same figure to update)
Any clarifications, just ask :)

I would pass the x,y-data via queue (as already planned), but would use the get-command in combination with the block-argument. That way, your drawing-function blocks until there are elements in the queue. If there are not, the function pauses and all other events in your application can be processed.
Something like that:
def process_data(inqueue):
#create your plot
for infile in iter(inqueue.get(block=True), "STOP"):
#update plot until inqueue.get returns "STOP"
#waits for elements in the queue enter code here
def main():
inqueue = Queue()
process = Process(target=process_data, args=(inqueue,)
#can be stopped by putting "STOP" via inqueue.put into the queue
process.start()
#put some data in the queue

Related

Python wait for animation to be completed

I have been using Blender's Python API to animate an object falling from a height. The Python program will start the animation and then pause the animation at frame 250. However, I would like the Python program to wait until this process has finished before continuing.
The snippet of code is below, where I want the final line of code to wait for the condition to be set in the pause_at_frame handler. One solution I experimented with, was to create a threading event. I would then start a thread calling pause_at_frame and set the threading event when the current frame was 250. The program would then wait for the threading event to be set. I could not manage to get this working without the program hanging. Any suggestions or solutions would be appreciated.
bpy.context.scene.frame_current = 0
bpy.ops.screen.animation_play()
def pause_at_frame(scene):
frame_current = bpy.context.scene.frame_current
if frame_current == 250:
bpy.ops.screen.animation_cancel()
bpy.context.scene.frame_current = 250
# condition here
for i in range(len(bpy.app.handlers.frame_change_pre)):
bpy.app.handlers.frame_change_pre.pop()
bpy.app.handlers.frame_change_pre.append(pause_at_frame)
# Wait here for condition
OTHER SOLUTION:
The other solution to this scenario is to bake the animation first.
bpy.ops.nla.bake(frame_start=1, frame_end=250, bake_types={'OBJECT'}) removing the need for the program above.

Updating pyqtgraph BarGraphItem inside a thread

i have some trouble to make a gui responsive while plotting live data. So that the GUI doesn't freeze, I try to thread all of my activities. I want to realize the following:
Recording data through a serial port
Calculations for the later plot in a thread
Plotting in a thread (currently via a QTimer, but when i drag the window, there is always a "small" freeze and the plot does not update on the drag)
1 and 2 is done, but now im not sure how to update my plot in a seperate thread.
My PlotWidget get initiated looks like this:
self.plottingDQ = [queue.deque(maxlen=100), queue.deque(maxlen=100), queue.deque(maxlen=100)]
self.graph = pg.PlotWidget()
self.barItem = pg.BarGraphItem(x0=self.plottingDQ[0], y0=self.plottingDQ[1], width=self.plottingDQ[2], height=1)
self.graph.addItem(self.barItem)
starting my threads is done via a button which is connected to this function. Writer-Thread is not relevant because it has no dependency on the plot. But the Calculator-Thread calculates the data for updating the plot
def startPlotting(self):
# not relevant for the example
self.csvData = queue.Queue()
self.csv = Writer(self.csvData)
self.csv.setDaemon(True)
self.csv.start()
self.calcData = queue.Queue()
self.calcPlot = Calculator(self.calcData, self.plottingDQ)
self.calcPlot.setDaemon(True)
self.calcPlot.start()
# Timer to update plot every x ms
self.timer = QTimer()
self.timer.timeout.connect(self.updatePlot)
self.timer.start(500)
Right now im updating my plot within the Qtimer every 500ms
def updatePlot(self):
print("update")
self.barItem.setOpts()
So every time i get some input from my serial port i just pass the data to my thread and call something like this:
def fromPort(self, data):
self.csvData.put(data)
self.calcData.put(data)
Inside my Calculator-Thread the data will be calculated and handed over to the plottingDQ which is connected to the BarGraphItem
class Calculator(threading.Thread):
def __init__(self, calcData, plottingDQ):
threading.Thread.__init__(self)
self.calcData = calcData
self.plottingDQ = plottingDQ
self.a = threading.Event()
self.a.set()
def run(self):
while self.a.isSet():
# Do some stuff here ...
# After the calculations, i write the data into the plottingDQ
self.plottingDQ[0].append(x)
self.plottingDQ[1].append(y)
self.plottingDQ[2].append(duration)
Is this the correct way to pass my calculated data from my Calcualtor-Thread into the deques which are used in the BarGraphItem? How can i update my BarGraphItem inside a thread?
The way you programmed this looks fine.
The root cause of the "stutter" seems to be an "update block" during the dragging process.
Try to enforce updates by adding pg.QtGui.QApplication.processEvents() to your updatePlot function, as described here:
def updatePlot(self):
print("update")
self.barItem.setOpts()
pg.QtGui.QApplication.processEvents()
It may be better to use Qthread to deal with this, having a worker thread doing the plot update inside the app loop.

Slow FuncAnimation Script

I use python 3.6 (the newest one) with the spider interpreter on a windows 10 PC with more-than-average hardware and wrote a script that enables me to continuously measure and plot the frequencies of two channels on an Agilent frequency counter and saves the data to a txt file. I furthermore have to convert the script to a .exe file (using pyinstaller) in order to distribute it to several measuring PCs.
Everything works fine, even from the .exe file, until the measuring time reaches about 2000 seconds. The software then starts to become very slow until it even show the "window not answering" thing that windows does while plotting.
I tried to activate the bliting-function of FuncAnimate, but when I do so, it only shows a white window.
Therefore I am now looking for options to enhance the speed of my software, especially at high data amounts, without cutting of the lots and lots of data (they are needed to be seen from t=0 to t=whatever).
Why does the blit=True kill my animation? Or is there a better way to plot these amounts of data fast and over and over again?
My code:
#Importing all required additional Python packages and sub-packages/functions
import visa, time, tkinter,sys
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg as FigureCanvas
from tkinter import messagebox, filedialog
from matplotlib.figure import Figure
import matplotlib.animation as animation
#Initialize the tkinter main window (graphical user interface window)
root=tkinter.Tk()
root.geometry('1200x900')
#Initialize the VISA resource manager and define a list object
rm=visa.ResourceManager('C:\\Windows\\System32\\visa32.dll')
liste=[]
#lists all available VISA resources and adds them to the list object
#one entry per VISA instrument
string=str(rm.list_resources())[1:-1]
liste.append(string.split('\'')[1:-1])
#opens a message box for each object in the list (each instrument)
#if user chosses "yes" for a VISA resource, the software tries to access the device
#if the device is present, the for-loop is left, otherwise the whole software quits
for i in liste[0]:
box=messagebox.askyesno('VISA Resources','Is this the correct VISA-Resource?'+'\n'+str(i))
if box==True:
try: inst=rm.open_resource(i)
except:
messagebox.showerror('Wrong Resource','The VISA resource was wrong!')
root.destroy()
sys.exit()
break
elif box==False: continue
#checks if the VISA resource is actually existent and present
try: inst
except:
messagebox.showerror('No Resource found','No VISA Resource was chosen.')
root.destroy()
sys.exit()
#opens a file dialog window and safes the chosen location as "filename"
#furthermore checks if the user specified a valid path or quited by using "cancel"
#if the user clicked "cancel", the software quits
filename=filedialog.asksaveasfilename()
if len(filename)==0:
messagebox.showerror('No File','No file location was specified.')
root.destroy()
sys.exit()
#definition of variables as well as the update function
x_data,y_data1, y_data2, y_data3=[],[],[],[]
def update(frame):
#create X-data, seconds since .clock was first called
x_data.append(time.clock())
#read out current freq on channel1 and float it (originaly returned as string)
value1=float(inst.query('MEAS:FREQ? 10 MHz, 0.1 Hz, (#1)'))
#add data to list
y_data1.append(value1)
#define and automatically adjust subplot limits
subplot1.set_ylim(min(y_data1)-1,max(y_data1)+1)
subplot1.set_xlim(min(x_data)-1,max(x_data)+1)
#define subplot title and labels
subplot1.set_title('Channel 1')
subplot1.set_xlabel('Time')
subplot1.set_ylabel('Frequency')
#same as above for second channel
value2=float(inst.query('MEAS:FREQ? 10 MHz, 0.1 Hz, (#2)'))
y_data2.append(value2)
subplot2.set_ylim(min(y_data2)-1,max(y_data2)+1)
subplot2.set_xlim(min(x_data)-1,max(x_data)+1)
subplot2.set_title('Channel 2')
subplot2.set_xlabel('Time')
subplot2.set_ylabel('Frequency')
#calculates and plots the difference of the upper two channels
y_data3.append(value1-value2)
subplot3.set_ylim(min(y_data3)-1,max(y_data3)+1)
subplot3.set_xlim(min(x_data)-1,max(x_data)+1)
subplot3.set_title('Difference')
subplot3.set_xlabel('Time')
subplot3.set_ylabel('Frequency')
#plots the subplots in the main plot frame
subplot1.plot(x_data,y_data1,'b')
subplot2.plot(x_data, y_data2,'r')
subplot3.plot(x_data, y_data3,'g')
#writes all data do a new file defined before
newfile.write(str(time.clock())+', '+str(value1)+', ' +str(value2)+'\n')
#enables the code to make use of the defined variables/data
return x_data, y_data1, y_data2, y_data3
#create a global boolean variable and set it to "True"
global boolean
boolean=True
#define a Pause function using the global boolean variable
def Pause():
global boolean
#if the boolean is True, the animation stops and the variable is set to False
if boolean==True:
anim.event_source.stop()
boolean=False
#if the boolean is False, the animation continues and the variable is set to True
elif boolean==False:
anim.event_source.start()
boolean=True
#define a Quit function that quits the application and closes the created file
def Quit():
newfile.close()
root.destroy()
#define a function that applies the user input data aquisition time to the animation
def SpeedApply():
anim.event_source.interval=int(float(Interv.get())*1000)
#create and place different buttons that call the defined functions upon buttonclick
QuitBut=tkinter.Button(text='Quit', command=Quit)
QuitBut.place(x=15,y=15)
StartBut=tkinter.Button(text='Pause/Resume',command=Pause)
StartBut.place(x=55, y=15)
Interv=tkinter.Spinbox(root,values=(0.1,0.2,0.5,1,1.5,2), width=8)
Interv.place(x=160, y=17)
Interv.delete(0,'end')
Interv.insert(0,1)
Speedbut=tkinter.Button(text='Apply Speed', command=SpeedApply)
Speedbut.place(x=250, y=15)
#create the figure needed to plot the animated data
figure=Figure(figsize=(8,8), dpi=100)
subplot1=figure.add_subplot(311)
subplot2=figure.add_subplot(312)
subplot3=figure.add_subplot(313)
figure.subplots_adjust(hspace=0.6)
#create a tkinter canvas, needed to embedd the figure into a tkinter root window
canvas=FigureCanvas(figure,root)
canvas.draw()
#canvas.start_event_loop(0.001)
canvas.get_tk_widget().place(x=25,y=50, height=850, width=1150)
#create the newfile where the data will be stored lateron
newfile=open(filename+time.strftime('%d')+time.strftime('%m')+time.strftime('%y')+'.txt','w')
newfile.write('Time, Channel 1, Channel 2\n')
#animation calling the update function upon the figure in the canvas with an interval of 1 second
anim=animation.FuncAnimation(figure,update, blit=False, interval=1000)
#tkinter mainloop, needed to react to user input in tkinter GUI
root.mainloop()
Okay, as requested, a simpler script showing my problem:
import time, tkinter
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg as FigureCanvas
from matplotlib.figure import Figure
import matplotlib.animation as animation
#Initialize the tkinter main window (graphical user interface window)
root=tkinter.Tk()
root.geometry('1200x900')
value1=1
value2=2
#definition of variables as well as the update function
x_data,y_data1, y_data2, y_data3=[],[],[],[]
def update(frame):
#create X-data, seconds since .clock was first called
x_data.append(time.clock())
global value1, value2, value3
value1=value1+2
#add data to list
y_data1.append(value1)
#define and automatically adjust subplot limits
subplot1.set_ylim(min(y_data1)-1,max(y_data1)+1)
subplot1.set_xlim(min(x_data)-1,max(x_data)+1)
#define subplot title and labels
subplot1.set_title('Channel 1')
subplot1.set_xlabel('Time')
subplot1.set_ylabel('Frequency')
#same as above for second channel
value2=value2+1
y_data2.append(value2)
subplot2.set_ylim(min(y_data2)-1,max(y_data2)+1)
subplot2.set_xlim(min(x_data)-1,max(x_data)+1)
subplot2.set_title('Channel 2')
subplot2.set_xlabel('Time')
subplot2.set_ylabel('Frequency')
#calculates and plots the difference of the upper two channels
y_data3.append(value1-value2)
subplot3.set_ylim(min(y_data3)-1,max(y_data3)+1)
subplot3.set_xlim(min(x_data)-1,max(x_data)+1)
subplot3.set_title('Difference')
subplot3.set_xlabel('Time')
subplot3.set_ylabel('Frequency')
#plots the subplots in the main plot frame
subplot1.plot(x_data,y_data1,'b')
subplot2.plot(x_data, y_data2,'r')
subplot3.plot(x_data, y_data3,'g')
#enables the code to make use of the defined variables/data
return x_data, y_data1, y_data2, y_data3
#create a global boolean variable and set it to "True"
global boolean
boolean=True
#define a Quit function that quits the application and closes the created file
def Quit():
root.destroy()
#create and place different buttons that call the defined functions upon buttonclick
QuitBut=tkinter.Button(text='Quit', command=Quit)
QuitBut.place(x=15,y=15)
#create the figure needed to plot the animated data
figure=Figure(figsize=(8,8), dpi=100)
subplot1=figure.add_subplot(311)
subplot2=figure.add_subplot(312)
subplot3=figure.add_subplot(313)
figure.subplots_adjust(hspace=0.6)
#create a tkinter canvas, needed to embedd the figure into a tkinter root window
canvas=FigureCanvas(figure,root)
canvas.draw()
#canvas.start_event_loop(0.001)
canvas.get_tk_widget().place(x=25,y=50, height=850, width=1150)
#animation calling the update function upon the figure in the canvas with an interval of 1 second
anim=animation.FuncAnimation(figure,update, blit=False, interval=100)
#tkinter mainloop, needed to react to user input in tkinter GUI
root.mainloop()
I left the "beautyful-makings" like titles in the code.
To actually see the problem I am having you would have to run that script for at least 600 seconds, the problem becomes "stronger" after 2000 seconds.
It seems that a large part of your issues is having to recompute the entire lists min and max each time,
e.g. your subplot x and y min/max values, rather than processing your entire appended list, you should use a variable to hold the lowest and highest values so far compare the current piece of input data to them, and if it meets condition update them,
subplot1.set_ylim(min(y_data1)-1,max(y_data1)+1)
subplot1.set_xlim(min(x_data)-1,max(x_data)+1)
You also appear to be resetting up the subplot labels every single update, now I am not as familiar with GUI's in TKinter, but i feel this could be covered in an initilisation definition,
subplot1.set_title('Channel 1')
subplot1.set_xlabel('Time')
subplot1.set_ylabel('Frequency')
The first suggestion should keep your program running in a more consistent time, rather than incrementally slowing down, the other should be able to reduce how much your doing per update.

Runtime progress status (like statusbar) in python

I'm trying to create simple progress status label for my Pyqt5 Python code and update it after every iteration of a loop in which a function does a bunch of stuff. The label I want to be updated is "status_label_counter". The below code only shows the part for creating labels and the exact place where I want to use the functionality I've mentioned.
#initialisation code, etc...
self.status_label_counter = QLabel()
self.status_label_from = QLabel(' from: ')
self.status_label_total = QLabel()
status_hbox = QHBoxLayout()
status_hbox.addStretch()
status_hbox.addWidget(self.status_label_counter)
status_hbox.addWidget(self.status_label_from)
status_hbox.addWidget(self.status_label_total)
status_hbox.addStretch()
#bunch of other code...
def create_ics(self):
counter = 0
self.status_label_total.setText(str(len(self.groups)))
for group in self.groups:
#does a bunch of stuff inside
group_manager.create_calendar_for(self.rows, group, self.term)
counter += 1
#for console output
print('iteration: ', counter)
#trying to update status counter
self.status_label_counter.setText(str(counter))
The problem is that I only see the update of both labels when the loop is done with the nested function. When I click a button that calls for "create_ics" function window becomes inactive for about 5 seconds, I see logs on console with count of iterations, but nothing happens in view.
The view (Qt) is locked in your main thread and never gets its chance to process its event loop and thus redraw itself. If you really want to do it this way, call:
self.status_label_counter.repaint()
After you set the text (and if you have some complex layout measuring call QApplication.processEvents() instead).
However, much better option would be to run your create_ics() function in a separate thread leaving your main thread to deal with the view and Qt's event processing. You can do it either through standard Python's threading module, or using Qt's own QThread: https://nikolak.com/pyqt-threading-tutorial/ .

Memory leak when embedding and updating a matplotlib graph in a PyQt GUI

I am trying to embed a matplotlib graph that updates every second into a PyQt GUI main window.
In my program I call an update function every second using threading.Timer via the timer function shown below. I have a problem: my program grows bigger every second - at a rate of about 1k every 4 seconds. My initial thoughts are that the append function (that returns a new array in update_figure) does not delete the old array? Is it possible this is the cause of my problem?
def update_figure(self):
self.yAxis = np.append(self.yAxis, (getCO22()))
self.xAxis = np.append(self.xAxis, self.i)
# print(self.xAxis)
if len(self.yAxis) > 10:
self.yAxis = np.delete(self.yAxis, 0)
if len(self.xAxis) > 10:
self.xAxis = np.delete(self.xAxis, 0)
self.axes.plot(self.xAxis, self.yAxis, scaley=False)
self.axes.grid(True)
self.i = self.i + 1
self.draw()
This is my timer function - this is triggered by the click of a button in my PyQt GUI and then calls itself as you can see:
def timer(self):
getCH4()
getCO2()
getConnectedDevices()
self.dc.update_figure()
t = threading.Timer(1.0, self.timer)
t.start()
EDIT: I cant post my entire code because it requires a lot of .dll includes. So i'll try to explain what this program does.
In my GUI I want to show the my CO2 value over time. My get_co22 function just returns a float value and I'm 100% sure this works fine. With my timer, shown above, I want to keep append a value to a matplotlib graph - the Axes object is available to me as self.axes. I try to plot the last 10 values of the data.
EDIT 2: After some discussion in chat, I tried putting the call to update_figure() in a while loop and using just one thread to call it and was able to make this minimal example http://pastebin.com/RXya6Zah. This changed the structure of the code to call update_figure() to the following:
def task(self):
while True:
ui.dc.update_figure()
time.sleep(1.0)
def timer(self):
t = Timer(1.0, self.task())
t.start()
but now the program crashes after 5 iterations or so.
The problem is definitely not with how you are appending to your numpy array, or truncating it.
The problem here is with your threading model. Integrating calculation loops with a GUI control loop is difficult.
Fundamentally, you need your GUI threading to have control of when your update code is called (spawning a new thread to handle it if necessary) - so that
your code does not block the GUI updating,
the GUI updating does not block your code executing and
you don't spawn loads of threads holding multiple copies of objects (which might be where your memory leak comes from).
In this case, as your main window is controlled by PyQt4, you want to use a QTimer (see a simple example here)
So - alter your timer code to
def task(self):
getCH4()
getCO2()
getConnectedDevices()
self.dc.update_figure()
def timer(self):
self.t = QtCore.QTimer()
self.t.timeout.connect(self.task)
self.t.start(1000)
and this should work. Keeping the reference to the QTimer is essential - hence self.t = QtCore.QTimer() rather than t = QtCore.QTimer(), otherwise the QTimer object will be garbage collected.
Note:
This is a summary of a long thread in chat clarifying the issue and working through several possible solutions. In particular - the OP managed to mock up a simpler runnable example here: http://pastebin.com/RXya6Zah
and the fixed version of the full runnable example is here: http://pastebin.com/gv7Cmapr
The relevant code and explanation is above, but the links might help anyone who wants to replicate / solve the issue. Note that they require PyQt4 to be installed
if you are creating a new figure for every time this is quite common.
matplotlib do not free the figures you create, unless you ask it, somethink like:
pylab.close()
see How can I release memory after creating matplotlib figures

Categories