Python matploblib memory usage and speed up options? [duplicate] - python

It seems that the standard way of creating a figure in matplotlib doesn't behave as I'd expect in python: by default calling fig = matplotlib.figure() in a loop will hold on to all the figures created, and eventually run out of memory.
There are quite a few posts which deal with workarounds, but requiring explicit calls to matplotlib.pyplot.close(fig) seems a bit hackish. What I'd like is a simple way to make fig reference counted, so I won't have to worry about memory leaks. Is there some way to do this?

If you create the figure without using plt.figure, then it should be reference counted as you expect. For example (This is using the non-interactive Agg backend, as well.)
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
# The pylab figure manager will be bypassed in this instance.
# This means that `fig` will be garbage collected as you'd expect.
fig = Figure()
canvas = FigureCanvas(fig)
ax = fig.add_subplot(111)

If you're only going to be saving figures rather than displaying them, you can use:
def savefig(*args, **kwargs):
plt.savefig(*args, **kwargs)
plt.close(plt.gcf())
This is arguably no less hacky, but whatever.

If you want to profit from the use of pyplot and have the possibility to wrap the figure in a class of your own, you can use the __del__ method of your class to close the figure.
Something like:
import matplotlib.pyplot as plt
class MyFigure:
"""This is my class that just wraps a matplotlib figure"""
def __init__(self):
# Get a new figure using pyplot
self.figure = plt.figure()
def __del__(self):
# This object is getting deleted, close its figure
plt.close(self.figure)
Whenever the garbage collector decides to delete your figure because it is inaccessible, the matplotlib figure will be closed.
However note that if someone has grabbed the figure (but not your wrapper) they might be annoyed that you closed it. It probably can be solved with some more thought or imposing some restrictions on usage.

Related

How to save animations with tight layout, transparency and in high quality with matplotlib?

I am trying to implement an option in my GUI to save an image sequence displayed using matplotlib. The code looks something like this:
import matplotlib.pyplot as plt
from matplotlib.backends.backend_qt5agg import \
FigureCanvasQTAgg as FigureCanvas
from matplotlib.animation import FuncAnimation
from PIL import Image
plt.rcParams['savefig.bbox'] = 'tight'
class Printer:
def __init__(self, data):
self.fig, self.ax = plt.subplots()
self.canvas = FigureCanvas(self.fig)
# some irrelevant color adjustment here
#self.ax.spines['bottom'].set_color('#f9f2d7')
#self.ax.spines['top'].set_color('#f9f2d7')
#self.ax.spines['right'].set_color('#f9f2d7')
#self.ax.spines['left'].set_color('#f9f2d7')
#self.ax.tick_params(axis='both', colors='#f9f2d7')
#self.ax.yaxis.label.set_color('#f9f2d7')
#self.ax.xaxis.label.set_color('#f9f2d7')
#self.fig.subplots_adjust(left=0.1, right=0.975, bottom=0.09, top=0.98)
self.fig.patch.set_alpha(0)
self.fig.patch.set_visible(False)
self.canvas.setStyleSheet("background-color:transparent;")
self.fig.set_size_inches(10, 10, True)
self.fig.tight_layout()
self.data = data
self.image_artist = self.ax.imshow(data[0])
def animate(self, i):
self.image_artist.set_data(self.data[i])
self.canvas.draw()
def save_animation():
data = [
Image.open("test000.png"),
Image.open("test001.png"),
]
file = 'test.gif'
printer = Printer(data)
ani = FuncAnimation(
printer.fig, printer.animate, interval=100, frames=len(data),
)
# writer = animation.writers['pillow'](bitrate=1000)
ani.save(
file, writer='pillow', savefig_kwargs={'transparent': True, 'bbox_inches': 'tight'}
)
save_animation()
Transparency:
As you can see I have already tried several different approaches as suggested elsewhere (1, 2), but didn't manage to find a solution. All of the settings and arguments patch.set_alpha(0), patch.set_visible(False), canvas.setStyleSheet("background-color:transparent;"), savefig_kwargs={'transparent': True} seem to have no effect at all on the transparency. I found this post but I didn't get the code to work (for one I had to comment out this %matplotlib inline, but then I ended up getting some error during the MovieWriter.cleanup out = TextIOWrapper(BytesIO(out)).read() TypeError: a bytes-like object is required, not 'str'). Here, it was suggested that this is actually a bug, but the proposed workaroud doesn't work for me since I would have to rely on third-party software. There also exists this bug report which was supposedly solved, so maybe it is unrelated.
Tight layout
I actually couldn't really find much on this, but all the things I tried (plt.rcParams['savefig.bbox'] = 'tight', fig.tight_layout(), savefig_kwargs={'bbox_inches': 'tight'}) don't have any effect or are even actively discarded in the case of the bbox_inches argument. How does this work?
High quality
Since I cannot use ImageMagick and can't get ffmpeg to work (more on this below), I rely on pillow to save my animation. But the only argument in terms of quality that I can pass on seems to be the bitrate, which doesn't have any effect. The files still have the same size and the animation still looks like mush. The only way that I found to increase the resolution was to use fig.set_size_inches(10, 10, True), but this still doesn't improve the overall quality of the animation. It still looks bad. I saw that you can pass on codec and extra_args so maybe that is something that might help, but I have no idea how to use these because I couldn't find a list with allowed arguments.
ffmpeg
I can't get ffmpeg to work. I installed the python package from here and can import it into a python session but I don't know how I can get matplotlib to use that. I also got ffmpeg from here (Windows 64-bit version) and set the plt.rcParams['animation.ffmpeg_path'] to where I saved the files (there was no intaller to run, not sure if I did it correctly). But this didn't help either. Also this is of course also third-party software, so if somebody else were to use my code/program it wouldn't work.

Python: Why does this command work when I didn't import command

I'm writing a Python GUI for the first time to plot some data, and have imported the following modules/commands to do so
from tkinter import *
from tkinter.filedialog import askopenfilename
from tkinter.ttk import *
import matplotlib
matplotlib.use('TkAgg')
from matplotlib.cm import get_cmap
from matplotlib.pyplot import figure
from scipy.interpolate import griddata
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
Searching the internet, I found some code that worked to get my plot embedded in the GUI:
fig = figure()
fig.add_subplot(111).pcolormesh(X, Y, Z, cmap = get_cmap('BuPu'))
canvas = FigureCanvasTkAgg(fig, master = window) # A tk.DrawingArea.
canvas.draw()
canvas.get_tk_widget().pack(side = TOP, fill = BOTH, expand = 1)
My question is why does pcolormesh work in the second line above when I didn't import that command from matplotlib.pyplot? I'm new to Python, and I'm sure I'm missing something, but if I wanted to use the pcolormesh command in the past, I had to import it. Thanks.
Your function calls created objects and those objects have methods. You don't need to import those methods as they are attached directly to the object. They are different from importing a function as you don't have direct access to those methods: they can only be accessed from the object.
when you set fig = figure(), you created an object. If you print(repr(fig)) you'll see something like this.
'<matplotlib.figure.Figure object at 0x000000000784F208>'
so fig is a Figure object. This objects comes with some methods and properties attached to it. You can get a list of all the methods using dir(fig) (it's a long list, so I won't print it here).
fig.subplot(111) calls the subplot method attached to your fig object. It returns a new object (an AxesSubplot). That object has a method attached to it called pcolormesh. If you want to play around with that object, you can do this.
ax = fig.subplot(111)
ax.pcolormesh(X, Y, Z, cmap = get_cmap('BuPu'))
print(repr(ax))
In addition to doing your stuff, this will print <matplotlib.axes._subplots.AxesSubplot at 0x8996f28>.
I remember being similarly confused by matplot.pyplot when I first started out because you can often do nearly identical things with a function (matplotlib.pyplot.pcolomesh) and as a method of an object (in the above example, ax.pcolormesh). They mention it in the API explanation here and here.

How do pyplot functions (show, savefig, etc) work without "object" being passed in?

How do pyplot functions such as show() and savefig() not require a "plot object" to work?
For example, the following code works, but somehow I expect to use a file handler or pass a "plot object" into plt.show() and plot.savefig("venn3.pdf").
from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles
# Subset sizes
s = (2,3,4,3,1,0.5,4)
v = venn3(subsets=s, set_labels=('A', 'B', 'C'))
# Subset labels
v.get_label_by_id('100').set_text('Abc')
v.get_label_by_id('010').set_text('aBc')
v.get_label_by_id('110').set_text('ABc')
v.get_label_by_id('001').set_text('Abc')
v.get_label_by_id('101').set_text('aBc')
v.get_label_by_id('011').set_text('ABc')
v.get_label_by_id('111').set_text('ABC')
# Subset colors
v.get_patch_by_id('100').set_color('c')
v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('110').set_color('blue')
# Subset alphas
v.get_patch_by_id('101').set_alpha(0.4)
v.get_patch_by_id('011').set_alpha(1.0)
v.get_patch_by_id('111').set_alpha(0.7)
# Border styles
c = venn3_circles(subsets=s, linestyle='solid')
c[0].set_ls('dotted') # Line style
c[1].set_ls('dashed')
c[2].set_lw(1.0) # Line width
plt.show() # For show() to work without using variable v seems counter-intuitive to me.
plt.savefig("venn3.pdf") # For savefig() to work without using variable v seems counter-intuitive to me.
2[]
matplotlib.pyplot is often called "statemachine". This means that the function it provides do certain things depending on the internal state of pyplot.
In your code, you have created a figure and this is stored as an object; pyplot knows it has one figure.
If you then call other commands, it is assumend that they apply to that one figure which has been created previously, like plt.savefig.
plt.show() would work on all previously created figures (all of them would be shown).
Pyplot uses a global variable to hold the figure object. All pyplot functions work with that variable(s). If you are working interactively, pyplot is perfectly fine since only you will modify that variable. If you are writing multi-threaded or multi-user code pyplot will not work, and you would have to use the layer benath it, which needs the figure object passed in (and is a terrible interface).

Python Matplotlib: Clear figure when figure window is not open

I'm working with matplotlib plotting and use ioff() to switch interactive mode off to suppress the automatic opening of the plotting window on figrue creation. I want to have full control over the figure and only see it when explicitely using the show() command.
Now apparently the built-in commands to clear figures and axes do not work properly anymore.
Example:
import numpy as np
import matplotlib.pyplot as mpp
class PlotTest:
def __init__(self,nx=1,ny=1):
# Switch off interactive mode:
mpp.ioff()
# Create Figure and Axes:
self.createFigure(nx, ny)
def createFigure(self,nx=1,ny=1):
self.fig, self.axes = mpp.subplots(nx,ny)
if nx*ny == 1:
self.axes = np.array([self.axes])
def linePlot(self):
X = np.linspace(0,20,21)
Y = np.random.rand(21)
self.axes[0].plot(X,Y)
P = PlotTest()
P.linePlot()
P.fig.show()
Now I was thinking I could use P.fig.clear() any time to simply clear P.fig, but apparently that's not the case.
Writing P.fig.clear() directly into the script and execute it together it works and all I see is an empty figure. However that's rather pointless as I never get to see the actual plot like that.
Doing P.fig.clear() manually in the console does not do anything, regardless if the plot window is open or not, all other possible commands fail as well:
P.fig.clf()
P.axes[0].clear()
P.axes[0].cla()
mpp.clf()
mpp.cla()
mpp.close(P.fig)
Wrapping the command into a class method doesn't work either:
def clearFig(self):
self.fig.clear()
EDIT ================
After a clear() fig.axes is empty, yet show() still shows the old plot with the axes still being plotted.
/EDIT ================
Is it because I switched off interactive mode?
If you add a call to plt.draw() after P.fig.clear() it clears the figure. From the docs,
This is used in interactive mode to update a figure that has been altered, but not automatically re-drawn. This should be only rarely needed, but there may be ways to modify the state of a figure with out marking it as stale. Please report these cases as bugs.
I guess this is not a bug as you have switched off interactive mode so it is now your responsibility to explicitly redraw when you want to.
You can also use P.fig.canvas.draw_idle() which could be wrapper in the class as clearFigure method.

Why does matplotlib.figure.Figure behave so different than matplotlib.pyplot.figure

A fellow programmer alerted me to a problem where matplotlib.pyplot and Tkinter don't behave well together, as demonstrated by this question Tkinter/Matplotlib backend conflict causes infinite mainloop
We changed our code to prevent potential problems as mentioned in the linked question, as follows:
Old
import matplotlib.pyplot as plt
self.fig = plt.figure(figsize=(8,6))
if os.path.isfile('./UI.png'):
image = plt.imread('./UI.png')
plt.axis('off')
plt.tight_layout()
im = plt.imshow(image)
# The Canvas
self.canvas = FigureCanvasTkAgg(self.fig, master = master)
self.toolbar = NavigationToolbar2TkAgg(self.canvas, root)
self.canvas.get_tk_widget().pack(fill=BOTH,expand=YES)
self.canvas.draw()
Intermediate (UI.png not being shown)
import matplotlib.pyplot as plt
import matplotlib
self.fig = matplotlib.figure.Figure(figsize=(8, 6))
if os.path.isfile('./UI.png'):
image = matplotlib.image.imread('./UI.png')
plt.axis('off')
plt.tight_layout()
plt.imshow(image)
# The Canvas
self.canvas = FigureCanvasTkAgg(self.fig, master=master)
self.toolbar = NavigationToolbar2TkAgg(self.canvas, root)
self.canvas.get_tk_widget().pack(fill=BOTH, expand=YES)
self.canvas.draw()
The changed code did not display the 'background' image anymore and I have been mostly just trying random things (as I am quite lost in the difference between the two options) to get the figure displaying again. The changes involved switching from tight_layout to set_tight_layout to avoid a warning, as mentioned on https://github.com/matplotlib/matplotlib/issues/1852 . The resulting code is as follows:
Potential Fix
import matplotlib.pyplot as plt
import matplotlib
self.fig = matplotlib.figure.Figure(figsize=(8, 6))
background_image = self.fig.add_subplot(111)
if os.path.isfile('./UI.png'):
image = matplotlib.image.imread('./UI.png')
background_image.axis('off')
#self.fig.tight_layout() # This throws a warning and falls back to Agg renderer, 'avoided' by using the line below this one.
self.fig.set_tight_layout(True)
background_image.imshow(image)
# The Canvas
self.canvas = FigureCanvasTkAgg(self.fig, master=master)
self.toolbar = NavigationToolbar2TkAgg(self.canvas, root)
self.canvas.get_tk_widget().pack(fill=BOTH, expand=YES)
self.canvas.draw()
The question therefore is, why do we need to use a subplot now (using matplotlib.figure.Figure) while before we did not (using matplotlib.pyplot)?
PS: I am sorry if this is a silly question but almost everything that I can find on the subject seems to use the matplotlib.pyplot variant. Therefore, I am having trouble finding any good documentation for the matplotlib.figure.Figure variant.
TL;DR
The question therefore is, why do we need to use a subplot now (using matplotlib.figure.Figure) while before we did not (using matplotlib.pyplot)?
subplot creates an Axes object. You did have one before, but the pyplot API "hid" it from you under the covers so you didn't realise it. You are now trying to use the objects directly, so have to handle it yourself.
More detailed reason
The reason you see this behaviour is because of how matplotlib.pyplot works. To quote the tutorial a little:
matplotlib.pyplot is a collection of command style functions that make matplotlib work like MATLAB.... matplotlib.pyplot is stateful, in that it keeps track of the current figure and plotting area, and the plotting functions are directed to the current axes
The crucial bit is that pyplot is stateful. It is keeping track of state "under the covers" and hiding the object model from you to some extent. It also does some implicit things. So - if you simply call, e.g., plt.axis(), under the covers pyplot calls plt.gca() and that in turn calls gcf() which will return a new figure, because you haven't set up a figure via pyplot yet. This is true for most calls to plt.some_function() - if pyplot doesn't have a figure object in it's own state yet, it will create one.
So, in your intermediate example, you've created your own Figure object - fiven it a name self.fig (I'm not sure what your class structure is, so I don't know what self is, but I'm guessing it's your tk.Frame object or something similar).
The punchline
pyplot doesn't know anything about self.fig. So in your intermediate code, you're calling imshow() on the Figure object in pyplot state, but displaying a different figure (self.fig) on your canvas.
The problem is not that you need to use subplot as such, but that you need to change the background image on the correct Figure object. The way you've used subplot in your potential fix code will do that - although I suggest an alternative below which maybe makes the intent clearer.
How to fix
Change
plt.axis('off')
plt.tight_layout()
plt.imshow(image)
to
self.fig.set_tight_layout(True)
ax = self.fig.gca() # You could use subplot here to get an Axes object instead
ax.axis('off')
ax.imshow(image)
A note on root cause: pyplot API vs direct use of objects
This a bit of an opinion, but might help. I tend to use the pyplot interface when I need to quickly get things prototyped and want to use one of the fairly standard cases. Often, that is enough.
As soon as I need to do more complicated things, I start to use the object model directly - maintaining my own named Figure and Axes objects etc.
Mixing the two is possible, but often confusing. You've found this with your intermediate solution. So I recommend doing one or the other.

Categories