We Keep Coding

Way to wait until user deletes Matplotlib figure before adding more?

So I have a function that scatter-plots some data and does so by creating new figures. The maximum amount of figures allowed at a time is 20 to avoid memory overload. If the user wants to plot a data-set with 6 variables to be exact, then there would be 30 different figures. Is there a way to wait until the user deletes the necessary amount of figures before adding more?
This is what I've though of:
import matplolib.pyplot as plt
... # some code
# this below is inside a loop structure
f = plt.figure
# add some stuff to the figure
plt.show(block=False)
Halt() # checks to see if there are too many figures
Where Halt() is defined as such:
def halt():
first = True
while plt.gcf().number > 20: # are there more than 20 figures
if first:
# show message
first = False
# time.sleep(100)
The only problem with this is that it "freezes" the program, not allowing the user to exit out of any of the figures, as it is "not responding". I've also tried the time.sleep() but that does not seem work either.
Does anyone know of a good way to loop until a condition is met?

https://matplotlib.org/api/_as_gen/matplotlib.pyplot.show.html says:
If False ensure that all windows are displayed and return immediately. In this case, you are responsible for ensuring that the event loop is running to have responsive figures.
How to do this, you ask? Well, the documentation is at https://matplotlib.org/users/interactive_guide.html#explicitly-spinning-the-event-loop .
After some fiddling around, I made the following which plots 20 figures with maximum 5 at the same time:
import matplotlib.pyplot as plt
import numpy as np
from time import sleep
def plot_stuff(exponent, titlenum):
x = np.linspace(0.0, 1.0)
f = plt.figure()
ax = f.add_subplot(1, 1, 1)
ax.set_title('{} - {}'.format(titlenum, exponent))
ax.plot(x, x**exponent)
def get_fighandles():
fignumbers = plt.get_fignums()
return [plt.figure(fign) for fign in fignumbers]
N_figs_eventually_plotted = 20
N_figs_max_simultaneous = 5
N=0
while N < N_figs_eventually_plotted:
if len(get_fighandles()) < N_figs_max_simultaneous:
N += 1
# put here whichever update is needed when you can add new figures
plot_stuff(np.random.random(), N)
plt.show(block=False)
print('hi')
for fig in get_fighandles():
print(fig.canvas)
fig.canvas.flush_events()
fig.canvas.draw_idle() # might not be needed, but here it's fast
sleep(0.1)
# note: solution terminates when the last figure is plotted, you might want something to prevent this (for instance a plt.show(block=True) when the last figure is plotted)
There might be some subtle concurrency bugs (for instance, if you close a figure after the loop reads the figure handles but before it flushes the events), but I do not see how you can avoid that for your use case.

How to plot in real time with VisPy library?

I wrote a script for modeling the evolution of a pandemic (with graphs and scatter plots).
I tried several libraries to display results in real-time (8 countries x 500 particles):
Matplotlib (not fast enough)
PyQtGraph (better but still not fast enough)
OpenGL (good, but I did not find how to use it in 2D efficiently, using subplots, titles, legends...)
Bokeh (good, but the scatter plots "blink" each time their particles turn color. Code is here if you are interested)
That is why I am turning now to VisPy.
I am using a class Visualizer to display the results, with the method app.Timer().connect to manage the real-time side. Pandemic code is here.
from Pandemic import *
from vispy.plot import Fig
from vispy import app
class Visualizer:
def __init__(self, world):
self.fig = Fig()
self.world = world
self.traces = {}
#Scatter plots
for idx, c in world.countries.items():
pos_x = idx % self.world.nb_cols
pos_y = idx // self.world.nb_cols
subplot = self.fig[pos_y, pos_x]
data = np.array([c.x_coord, c.y_coord]).reshape(-1,2)
self.traces[idx] = subplot.plot(data, symbol='o', width=0, face_color=c.p_colors, title='Country {}'.format(idx+1))
def display(self):
for idx, c in self.world.countries.items():
data = np.array([c.x_coord, c.y_coord]).reshape(-1,2)
self.traces[idx].set_data(data, face_color=c.p_colors)
def update(self, event):
self.world.update(quarantine=False)
self.display()
def animation(self):
self.timer = app.Timer()
self.timer.connect(self.update)
self.timer.start(0)
self.start()
def start(self):
if (sys.flags.interactive != 1):
self.status = app.run()
if __name__ == '__main__':
w = World(move=0.001)
for i in range(8):
w.add_country(nb_S=500)
v = Visualizer(w)
v.animation()
The scatter plots "blink" each time their particles turn color, as with Bokeh. Am I doing something wrong?
Is there a more efficient way for real-time display, maybe using vispy.gloo or vispy.scene? (It is slower than pyqtgraph.opengl for the moment)

We can efficiently plot in real time by using vispy.gloo module to leverage the power of GPU. Here is one way of doing it :
1) Build a class that inherits vispy.app.Canvas class.
2) Create an OpenGL Program whose inputs are shaders. This object allows us to link our data to shader variables. Each dot on the canvas depends on these variable values (describing its coordinate, color, etc). For example, it is way harder for displaying text (titles, labels, etc) than with Matplotlib library. Here is a deeper explanation of the process.
3) Set a timer connected to the function we want to call repeatedly (real-time side).
The vispy.scene module, dedicated to the high-level visualization interfaces for scientists, is still experimental. Maybe this is the reason why my first code got some bugs.
Here is my new code.

Live data plotting lags after a while

I've been writing a program for a workstation automation in a laboratory. One of the instruments I communicate is called beam profiler, it basically reads light inputs from two ortogonal directions (x,y). Once the input is read, I need to convert it to a 2D image, for that I use the numpy meshgrid and I'm able to obtain my desired output.
For better clarity, see image bellow. The two Gaussian lines in the x and y axis are my raw input and the colored figure is after processed with meshgrid.
I divide my software in two parts for this. First I create another QT thread that initializes my device and runs in a loop getting the data and processing it. Then this thread sends a signal to the main thread with the values.
In the main thread I get the values, plot the graph and update the gui screen.
It is already working, the problem is that when I start the beam profiler readings the software starts getting slower as time passes. At first I thought it was because of the data processing but it doesn't make sense because it is running in the second thread and when I start the device there is no lag.
It seems like if it were "saving" the data in memory and getting slower, which is weird since I'm using the set_data and draw methods for plotting.
Note: if I close the device readings inside my software the lags stops and if I start it again, it starts good but then lags as time passes.
Any incoming help is much appreciated!
Data acquisition thread code:
class ThreadGraph(QtCore.QThread):
_signalValues = QtCore.pyqtSignal(float, float, float, float, float, float, float, float)
_signalGraph = QtCore.pyqtSignal(np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray)
_signalError = QtCore.pyqtSignal(str)
BEAMstatus = QtCore.pyqtSignal(str)
def __init__(self, parent=None):
super(ThreadGraph, self).__init__(parent)
self.slit = 0
self.state = False
#Thread starts
def run(self):
self.init() #Device initialization (Not relevant, therefore omitted)
time.sleep(0.1)
while self.state == True: #Thread loop (data acquisition)
self.emitValues() #Fun to get the data and emit
time.sleep(0.016)
self.emitGraph() #Process data into 2D and emit
try: #When while is over, terminate the thread
self.beam.close(self.session)
except RuntimeError as err:
print err
self.quit()
def emitGraph(self): #Use the data acquired to to generate 2D image and emit
xx, yy = np.meshgrid(self.slit_data_int[self.slit][0::10], self.slit_data_int[self.slit+1][0::10])
zz = xx * yy
self._signalGraph.emit(
self.slit_data_pos[self.slit][0::10],
self.slit_data_int[self.slit][0::10],
self.slit_data_pos[self.slit + 1][0::10],
self.slit_data_int[self.slit + 1][0::10],
zz
)
def emitValues(self):
try: #Try to get data from device (data is stored in calculation_result)
self.slit_data_pos, self.slit_data_int, self.calculation_result, self.power, self.power_saturation, self.power_intensities = self.beam.get_slit_scan_data(self.session)
except RuntimeError as err:
self._signalError.emit(str(err))
return
else: #emit data to gui main thread
self._signalValues.emit(
self.calculation_result[self.slit].peakPosition,
self.calculation_result[self.slit + 1].peakPosition,
self.calculation_result[self.slit].peakIntensity,
self.calculation_result[self.slit + 1].peakIntensity,
self.calculation_result[self.slit].centroidPosition,
self.calculation_result[self.slit + 1].centroidPosition,
self.calculation_result[self.slit].gaussianFitDiameter,
self.calculation_result[self.slit + 1].gaussianFitDiameter
)
Main Gui code:
class BP209_class(QtGui.QWidget):
def __init__(self, vbox, slit25, slit5, peakposx, peakposy, peakintx, peakinty, centroidposx, centroidposy, mfdx, mfdy):
QtGui.QWidget.__init__(self)
#Initialize a bunch of gui variables
self.matplotlibWidget = MatplotlibWidget('2d')
self.vboxBeam = vbox
self.vboxBeam.addWidget(self.matplotlibWidget)
self.vboxBeam.addWidget(self.matplotlibWidget.canvastoolbar)
#Create the thread and connects
self.thread = ThreadGraph(self)
self.thread._signalError.connect(self.Error_Handling)
self.thread._signalValues.connect(self.values_update)
self.thread._signalGraph.connect(self.graph_update)
self.thread.BEAMstatus.connect(self.Status)
#Initialize variables for plots
self.zz = zeros([750, 750])
self.im = self.matplotlibWidget.axis.imshow(self.zz, cmap=cm.jet, origin='upper', vmin=0, vmax=1, aspect='auto', extent=[-5000,5000,-5000,5000])
self.pv, = self.matplotlibWidget.axis.plot(np.zeros(750) , np.zeros(750) , color="white" , alpha=0.6, lw=2)
self.ph, = self.matplotlibWidget.axis.plot(np.zeros(750) , np.zeros(750), color="white" , alpha=0.6, lw=2)
self.matplotlibWidget.figure.subplots_adjust(left=0.00, bottom=0.01, right=0.99, top=1, wspace=None, hspace=None)
self.matplotlibWidget.axis.set_xlim([-5000, 5000])
self.matplotlibWidget.axis.set_ylim([-5000,5000])
def __del__(self): #stop thread
self.thread.state = False
self.thread.wait()
def start(self): #start thread
if self.thread.state == False:
self.thread.state = True
self.thread.start()
else:
self.thread.state = False
self.thread.wait()
#Slot that receives data from device and plots it
def graph_update(self, slit_samples_positionsX, slit_samples_intensitiesX, slit_samples_positionsY, slit_samples_intensitiesY, zz):
self.pv.set_data(np.divide(slit_samples_intensitiesX, 15)-5000, slit_samples_positionsX)
self.ph.set_data(slit_samples_positionsY, np.divide(slit_samples_intensitiesY, 15)-5000)
self.im.set_data(zz)
self.im.autoscale()
self.matplotlibWidget.canvas.draw()
Edit: I also have a camera attached to my system and I display it also in the gui using opencv. I noticed that if I start the cam the beam profiler's fps reduce to almost a half. So, maybe a QT paint optimization would be the way to go?

Calls to canvas.draw() are expensive. You are likely acquiring data faster than drawing commands can complete. This will cause paint events to get queued up and your plot will appear to lag. This blog post details a method that avoids calling canvas.draw() and can be used to speed up matplotlib realtime plotting.
If this is still not fast enough you may have to lower the acquisition rate, implement some form of frame skipping mechanism or use a different plotting library better optimised for speed.

How to speed up a Python Basemap choropleth animation

Taking ideas from various sources, and combining with my own, I sought to create an animated maps showing the shading of countries based on some value in my data.
The basic process is this:
Run DB query to get dataset, keyed by country and time
Use pandas to do some data manipulation (sums, avgs, etc)
Initialize basemap object, then load Load external shapefile
Using the animation library, color the countries, one frame for each distinct "time" in the dataset.
Save as gif or mp4 or whatever
This works just fine. The problem is that it is extremely slow. I have potentially over 100k time intervals (over several metrics) I want to animate, and I'm getting an average time of 15s to generate each frame, and it gets worse the more frames there are. At this rate, it will potentially take weeks of maxing out the cpu and memory on my computer to generate a single animation.
I know that matplotlib isn't known for being very fast (examples: 1 and 2) But I read stories of people generating animations at 5+ fps and wonder what I'm doing wrong.
Some optimizations that I have done:
Only recolor the countries in the animate function. This takes on average ~3s per frame, so while it could be improved, it's not what takes the most time.
I use the blit option.
I tried using a smaller plots size and less detailed basemap, but the results were marginal.
Perhaps a less detailed shapefile would speed up the coloring of the shapes, but as I said before, that's only a 3s per frame improvement.
Here is the code (minus a few identifiable features)
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import time
from math import pi
from sqlalchemy import create_engine
from mpl_toolkits.basemap import Basemap
from matplotlib.patches import Polygon
from matplotlib.collections import PatchCollection
from geonamescache import GeonamesCache
from datetime import datetime
def get_dataset(avg_interval, startTime, endTime):
### SQL query
# Returns a dataframe with fields [country, unixtime, metric1, metric2, metric3, metric4, metric5]]
# I use unixtime so I can group by any arbitrary interval to get sums and avgs of the metrics (hence the param avg_interval)
return df
# Initialize plot figure
fig=plt.figure(figsize=(11, 6))
ax = fig.add_subplot(111, axisbg='w', frame_on=False)
# Initialize map with Robinson projection
m = Basemap(projection='robin', lon_0=0, resolution='c')
# Load and read shapefile
shapefile = 'countries/ne_10m_admin_0_countries'
m.readshapefile(shapefile, 'units', color='#dddddd', linewidth=0.005)
# Get valid country code list
gc = GeonamesCache()
iso2_codes = list(gc.get_dataset_by_key(gc.get_countries(), 'fips').keys())
# Get dataset and remove invalid countries
# This one will get daily aggregates for the first week of the year
df = get_dataset(60*60*24, '2016-01-01', '2016-01-08')
df.set_index(["country"], inplace=True)
df = df.ix[iso2_codes].dropna()
num_colors = 20
# Get list of distinct times to iterate over in the animation
period = df["unixtime"].sort_values(ascending=True).unique()
# Assign bins to each value in the df
values = df["metric1"]
cm = plt.get_cmap('afmhot_r')
scheme= cm(1.*np.arange(num_colors)/num_colors)
bins = np.linspace(values.min(), values.max(), num_colors)
df["bin"] = np.digitize(values, bins) - 1
# Initialize animation return object
x,y = m([],[])
point = m.plot(x, y,)[0]
# Pre-zip country details and shap objects
zipped = zip(m.units_info, m.units)
tbegin = time.time()
# Animate! This is the part that takes a long time. Most of the time taken seems to happen between frames...
def animate(i):
# Clear the axis object so it doesn't draw over the old one
ax.clear()
# Dynamic title
fig.suptitle('Num: {}'.format(datetime.utcfromtimestamp(int(i)).strftime('%Y-%m-%d %H:%M:%S')), fontsize=30, y=.95)
tstart = time.time()
# Get current frame dataset
frame = df[df["unixtime"]==i]
# Loop through every country
for info, shape in zipped:
iso2 = info['ISO_A2']
if iso2 not in frame.index:
# Gray if not in dataset
color = '#dddddd'
else:
# Colored if in dataset
color = scheme[int(frame.ix[iso2]["bin"])]
# Get shape info for country, then color on the ax subplot
patches = [Polygon(np.array(shape), True)]
pc = PatchCollection(patches)
pc.set_facecolor(color)
ax.add_collection(pc)
tend = time.time()
#print "{}%: {} of {} took {}s".format(str(ind/tot*100), str(ind), str(tot), str(tend-tstart))
print "{}: {}s".format(datetime.utcfromtimestamp(int(i)).strftime('%Y-%m-%d %H:%M:%S'), str(tend-tstart))
return None
# Initialize animation object
output = animation.FuncAnimation(fig, animate, period, interval=150, repeat=False, blit=False)
filestring = time.strftime("%Y%m%d%H%M%S")
# Save animation object as m,p4
#output.save(filestring + '.mp4', fps=1, codec='ffmpeg', extra_args=['-vcodec', 'libx264'])
# Save animation object as gif
output.save(filestring + '.gif', writer='imagemagick')
tfinish = time.time()
print "Total time: {}s".format(str(tfinish-tbegin))
print "{}s per frame".format(str((tfinish-tbegin)/len(df["unixtime"].unique())))
P.S. I know the code is sloppy and could use some cleanup. I'm open to any suggestions, especially if that cleanup would improve performance!
Edit 1: Here is an example of the output
2016-01-01 00:00:00: 3.87843298912s
2016-01-01 00:00:00: 4.08691620827s
2016-01-02 00:00:00: 3.40868711472s
2016-01-03 00:00:00: 4.21187019348s
Total time: 29.0233821869s
9.67446072896s per frame
The first first few lines represent the date being processed, and the runtime of each frame. I have no clue why the first one is repeated. The final line it the total runtime of the program divided by the number of frames. Note that the average time is 2-3x the individual times. This makes me think that there is something happening "between" the frames that is eating up a lot of time.
Edit 2: I ran some performance tests and determined that average time to generate each additional frame is greater than the last, proportional to the number of frames, indicating that this is an quadratic-time process. (or would it be exponential?) Either way, I'm very confused why this wouldn't be linear. If the dataset is already generated, and the maps take a constant time to regenerate, what variable is causing each extra frame to take longer than the previous?
Edit 3: I just made the realization that I have no idea how the animation function works. The (x,y) and point variables were taken from an example that was just plotting moving dots, so it makes sense in that context. A map... not so much. I tried returning something map related from the animate function and got better performance. Returning the ax object (return ax,) makes the procedure run in linear time... but doesn't write anything to the gif. Anybody have any idea what I need to return from the animate function to make this work?
Edit 4: Clearing the axis every frame lets the frames generate at a constant rate! Now I just have to work on general optimizations. I'll start with ImportanceOfBeingErnest's suggestion first. The previous edits are obsolete now.

How to use matplotlib.animation to get a video of program output?

I wrote a python script that uses a heuristic to cluster 2D points in space. I'm representing each cluster using different cluster.
Presently,
the structure of my program is:
def cluster():
while True:
<do_some_work>
if <certain_condition_is_met>:
print "ADDED a new cluster:",cluster_details
if <breaking_condition_is_met>:
break
return Res
def plot_cluster(result):
<chooses a unique color for each cluster, and calls
pyplot.plot(x_coods,y_coods)
for each cluster>
def driver_function():
result = cluster()
plot_cluster(result)
pyplot.show()
That is, presently, I just obtain the final image of clustered points, where each cluster is represented by a different color.
However, I need to create an animation of how the program proceeds, i.e., something like:
"Initially, all points should be of same color, say blue. Then, as is cluster() function, instead of simply printing "ADDED a new cluster", the color of those points in the new cluster, should be changed in the image already present on screen.
Is there any way I can generate a video of such a program using matplotlib?
I saw an example of
`matplotlib.animation.FuncAnimation( ..., animate, ...)`
but it repeatedly calls the animate function that should return plottable values, which I think my program cannot.
Is there any way to obtain such a video of how this program proceeds?

To get this to work like you want will require a bit of refactoring, but I think something like this will work:
class cluster_run(object):
def __init__(self, ...):
# what ever set up you want
self.Res = None
def reset(self):
# clears all work and starts from scratch
pass
def run(self):
self.reset()
while True:
#<do_some_work>
if <certain_condition_is_met>:
print "ADDED a new cluster:",cluster_details
yield data_to_plot
if <breaking_condition_is_met>:
break
self.Res = Res
class culster_plotter(object):
def __init__(self):
self.fig, self.ax = plt.subplots(1, 1)
def plot_cluster(self, data_to_plot):
# does what ever plotting you want.
# fold in and
x_coords, y_coords)
ln = self.ax.plot(x_coords, y_coords)
return ln
cp = cluster_plotter()
cr = cluster_run()
writer = animation.writers['ffmpeg'](fps=30, bitrate=16000, codec='libx264')
ani = animation.FuncAnimation(cp.fig, cp.plot_cluster, cr.run())
ani.save('out.mp4', writer=writer)
plot_cluster(cr.Res)

Would it be sufficient to use pyplot.savefig('[frame].png'), where [frame] is the frame number of your plot in sequence, and then stitch these images together using a codec such as ffmpeg?