Possible Memory Leak with Tkinter Grid that Refreshes - python

I created a tkinter program with a memory leak issue that originally caused it to lock up after about 40 minutes. I tried an initial fix which greatly improved performance, but it still slows down after sometime so I think there is a possible second memory leak and/or other issue that I wanted to check with the community on.
About the program: A back end script updates a data table every sixty seconds which is then pushed to a tkinter script that crafts the data table in a nice layout. (I made some example code below, its not the actual script which is much longer) Every time it is refreshed, the data table can have a different number of rows/columns. Thus, my tkinter script needs to dynamically create the table and bind a button function to each cell. I'm just getting into GUI program and choose Tkinter as the first library to test out. I wanted to use a GUI library instead of a charting library because: 1) I want to learn how to build basic GUIs and figure this would be a fun application; 2) I want to be able to click on any portion of my data table and have a window pop-up that will adjust inputs for the back-end script on its next refresh.
Evolution of my code to solve memory leak problem and potential additional memory leak: So originally, I assumed that the cells would just get erased when you write a new box over the grid position. Version 1.0 below:
import pandas as pd
import numpy as np
from tkinter import *
root = Tk()
root.configure(background='black')
#Placeholder for example code
def popupwindow():
pass
def build():
mydf = pd.DataFrame([np.arange(1, np.random.randint(3, 7)) * np.random.randint(1,10) for x in np.arange(1, np.random.randint(3, 7))])
rowindex = 1
for row in mydf.iterrows():
colindex = 1
for i in row[1]:
label = Label(root, text=str(i), width=7)
label.bind('<Button-1>', popupwindow)
label.grid(row=rowindex,column=colindex)
colindex += 1
rowindex += 1
#If grid is smaller than previous grid, remove old widgets
for label in root.grid_slaves():
if int(label.grid_info()['row']) > rowindex-1 or int(label.grid_info()['column']) > colindex-1:
label.grid_forget()
def refresh():
build()
#For purpose of example code, I made refresh rate faster than my actual program's 60 seconds
root.after(5000, refresh)
refresh()
root.mainloop()
As I found out that is not the case, and that was causing the first memory leak problem. So I created Version 2.0 which ‘forgot’ all the grid slaves before recreating the grid (See the two new lines of code in build()). This substantially improved the performance but I still see a partial slowdown in Tkinter responsiveness overtime. Although, it has brought a second issue where after 30-40 minutes, the screen will go partially or entirely black (background for my frame is black) and will stop refreshing after:
def build():
mydf = pd.DataFrame([np.arange(1, np.random.randint(3, 7)) * np.random.randint(1,10) for x in np.arange(1, np.random.randint(3, 7))])
# I ADDED THESE TWO LINES OF CODE
for label in root.grid_slaves():
label.grid_forget()
rowindex = 1
for row in mydf.iterrows():
colindex = 1
for i in row[1]:
label = Label(root, text=str(i), width=7)
label.bind('<Button-1>', popupwindow)
label.grid(row=rowindex,column=colindex)
colindex += 1
rowindex += 1
# REMOVED THESE 3 LINES OF CODE AS NOW REDUNDANT WITH ADDED CODE ABOVE
# for label in root.grid_slaves():
# if int(label.grid_info()['row']) > rowindex-1 or int(label.grid_info()['column']) > colindex-1:
# label.grid_forget()
After perusing the forums more I saw this post on Overstack (Tkinter - memory leak with canvas) , which possibly signaled that widget labels may never be garbage collected even if forgotten. Not sure if this is an accurate interpretation, but if it is then this could be another possible reason why my V2.0 slows down overtime as i'm always forgetting and never rewriting my labels. Therefore, my proposed solution for V3 would be to use an if else function to see if a label already exists at a given position, if it doesn’t create a new label, if it does exist then adjust it. My question is, is this how you would approach it? Is there another memory leak/performance issue you can see from my basic example? If you have additional proposed adjustments to my code, such as how I dynamically created the data table, feel free to provide any input/improvements! Since i'm new to programming, I am very open to different ideas and more efficient methods.
Thank you in advance for your help!

The bug isn't in grid, it's in your code. Every five seconds you are creating a new dataframe, and new labels for every row in that dataframe, and you never delete them. Instead, you just keep stacking them on top of each other.
Calling grid_forget or grid_remove only removes them from the view, the objects aren't deleted.
You need to either delete all of the old widgets on each call to refresh, or reuse the existing labels rather than creating new labels.
Therefore, my proposed solution for V3 would be to use an if else function to see if a label already exists at a given position, if it doesn’t create a new label, if it does exist then adjust it. My question is, is this how you would approach it?
Yes. That, and destroy the old widgets that are no longer being used, rather than just removing them from the grid.

Related

Resetting generated widget count in Python with Tkinter

Using Python 3.7 - Working in Pycharm
I am currently working on a project where I am constantly generating new widgets and removing them to show different things, and I came across a fairly annoying problem that I can't seem to figure out, let alone find information on.
When the top widget(frame) in a row of frames has been made 255 times it crashes. I guess it has something to do with how it stores the information in bytes.
Edit*
I realize that my initial explaination was more complicated than it need to so here is a simplified version:
from tkinter import *
list = []
for x in range(1):
e = Frame()
list.append(e)
print(list) #Prints .!frame
list.clear()
print(list) #Prints [], the list is now empty
for x in range(1):
e = Frame()
list.append(e)
print(list) #Prints .!frame2, which is one step up from the previous one even the list started empty
list[0].destroy()
list[0].forget()
print(list) #.!frame2 is still there
list.clear()
print(list) #list is now empty again
for x in range(1):
e = Frame()
list.append(e)
print(list) #Prints .!frame3, and it still remembers.
As you can see, it keeps adding to the .!frame number.
What I am looking for is a way to keep it from reaching .!frame255
When the top widget(frame) in a row of frames has been made 255 times it crashes. I guess it has something to do with how it stores the information in bytes.
No, it has nothing to do with that. I think your assertion that the program crashes when the number reaches 255 is likely wrong. It's very easy to create a tkinter program that continues to work even when that number reaches into the thousands. If you look in the tkinter source code you'll see that it's just a plain integer that is appended to a string.
You seem to misunderstand the relationship between your list variable and the widgets that are stored in it. Clearing the list will not destroy the widgets in the list. You must delete each one individually, or destroy their parent. In this case their parent is the root window so that's not a viable solution.
If you want to destroy all of the frames, use a small loop before you clear the list:
for frame in list:
frame.destroy()
The number that tkinter assigns to the widget is inconsequential. It doesn't matter what the number is, and it may or may not reflect how many widgets actually exist. It's an internal detail that is unimportant.
If you want to see how many widgets actually exist, you can call winfo_children on the root window. For that you need a reference to the root window. The easiest and best way to do that is to explicitly create the root window at the start of your code:
root = Tk()
At the end of your script you can print out all of the windows that actually exist:
print("all windows:", root.winfo_children())
In your example code it shows exactly two widgets, which is what is expected. You create a widget, then you create a second widget, then you delete the first widget, and then you create a third widget.

Tkinter: Update frequency of Listbox

Is it possible to control the update frequency of the Listbox widget? Right now I do a lot of insert and delete operations at a high frequency and the Listbox doesn’t refresh very well. Maybe there is a way to override some draw function of the Listbox to fix this issue?
I am not able to find a way to disable visual updates of your listbox so I had to build a work around. If someone knows if you can disable the visual update of listbox please let me know.
My workaround will involve a list and 2 functions.
My first function will take the data that is going to be added to the listbox and instead add it to a list. This function simply simulates new values being added faster than what we want to update for a good visual on the method. You can adapt this code to yours to see how it will work with your inserts.
My second function will run once a second and take all the new values of this list and add them to the listbox by index.
This is a simple example but it should be a good starting point for you.
import tkinter as tk
root = tk.Tk()
add_tracker = 1
new_lb_items = []
lb = tk.Listbox(root)
lb.pack()
def add_to_listbox():
global add_tracker, new_lb_items, root
new_lb_items.append([add_tracker, "Number {}".format(add_tracker)])
add_tracker += 1
root.after(250, add_to_listbox)
def update_listbox_display():
global lb, new_lb_items, root
for item in new_lb_items:
lb.insert(item[0], item[1])
new_lb_items = [] # resets the list so only new values are added next time.
root.after(1000, update_listbox_display)
add_to_listbox()
update_listbox_display()
root.mainloop()

tkinter Label updating but not correctly

Using Python 3.5. I have a tkinter form. The user clicks a button to import many files into a Listbox. Another button loops thru the files and reads and extracts data from them.
I have a Label on the form that indicates the status of the loop. The status, for the most part, works as expected except that extra characters are added on the end. I'm not sure where the characters come from. I also print() the same content as the Label back to the screen and the print() displays exactly what it should.
My question is why is my Label not displaying the correct string?
My code, greatly shortened:
class tk_new_db:
def __init__(self, master):
self.master = master # sets 'root' to the instance variable 'master'
self.var_text_2 = StringVar()
self.var_text_2.set('STATUS: Active')
self.label_6 = Label(master, textvariable=self.var_text_2,
font=self.font_10)
self.label_6.grid(row=15, sticky=W, padx=15)
def execute_main(self): # extract data from files
file_num = 0
nf = len(self.listbox_1.get(0, END))
for fr in li:
file_num += 1
print('STATUS: Extracting Loads from File '
'{} in {}'.format(file_num, nf))
self.var_text_2.set('STATUS: Extracting Loads from File '
'{} in {}'.format(file_num, nf))
self.master.update_idletasks()
The print() is writing the following:
STATUS: Extracting Loads from File 1 in 5
The Label is writing the following:
STATUS: Extracting Loads from File 1 in 5 nce...
It always adds ' nce...' on the Form.
EDIT:
I do use self.var_text_2 earlier in the program. The ' nce...' looks like it is a fragment of the previous string. I've since tried resetting the variable in two different ways, but I'm still getting the same result.
self.var_text_2.set('STATUS: Checking .F06 Files for Convergence...')
self.master.update_idletasks()
self.var_text_2.__del__()
self.var_text_2.set('STATUS: Checking .F06 Files for Convergence...')
self.master.update_idletasks()
self.var_text_2.set('')
How do you properly delete the StringVar() for reuse?
The only explanation I can think of is that you're stacking labels on top of each other in the same row and column, so when you add a long string that ends in "nce..." and later update the screen by adding a shorter label in the same grid cell, the trailing text of the longer label underneath is showing through.
The reasons I draw this conclusion are:
this is a common mistake I've seen several times
there are no bugs in tkinter that would cause this
there is nothing in the code that you've shown that could possibly add "nce..." to the end of the string
the fact that you use sticky="w" rather than sticky="ew", which will cause a shorter label to not completely overlay a longer label if placed in the same row and column
I think I can explain. Root.mainloop repeatedly calls root.update. Root.update executes tasks on the 'main' event queue. In brief, it only executes idletasks when the event queue is empty after checking for file events (on non-Windows), window-gui events, and timer events. Idletasks, which are not well documented but seem to primarily be screen update tasks, are never added to the main event queue. It is therefore possible that idletasks will remain undone indefinitely. Root.update_idletasks exists to force execution of idletasks anyway.
When mainloop is running, calling update within a task run by update is usually unnecessary and possibly worse. It is something only for adventurous experts. Hence the warnings you have read (which assume that you are running mainloop).
When mainloop is not running and update is not being called repeatedly, either because you never called mainloop or because you block it with a long running loop, then you likely must call update yourself. What you seem to have discovered is that part of completely handling a StringVar update is a main event and not just an idletask.
I sympathize with your desire to do repeated tasks with a for loop, but doing so currently means taking responsibility for event handling yourself. I hope in the future that it will be possible to have 'async for' (new in 3.5) interoperate with Tk.mainloop, but it is not as easy as I hoped.
In the meanwhile, you could put the body of the for loop into a callback and loop with root.after.

Tkinter canvas create_image and create_oval optimization

Background
I am trying - and succeeding - in creating a simple plot using using the Canvas object within tkinter. I am trying to use as many tools that are installed with Python3 as possible. Matplotlib and others are great, but they are pretty large installs for something that I'm trying to keep a bit smaller.
The plots are updated every 0.5s based on input from a hardware device. The previous 128 points are deleted and the current 128 points are drawn. See my most recent blog post for a couple of screenshots. I have successfully created the plots using canvas.create_oval(), but as I was running it, I heard my PC fans ramp up a bit (I have them on an aggressive thermal profile) and realized that I was using 15% of the CPU, which seemed odd.
The Problem
After running cProfile, I found that the canvas.create_oval() was taking more cumulative time than I would have expected.
After reading a bit about optimization in the tkinter canvas (there isn't much out there except 'use something else'), I came across a post that suggested that one might use an image of a dot and use canvas.create_images() instead of a canvas.create_oval(). I tried that and the time in create_image() was a bit less, but still quite significant.
For completeness, I will include the code fragment. Note that this method is part of a class called Plot4Q which is a subclass of tk.Canvas:
def plot_point(self, point, point_format=None, fill='green', tag='data_point'):
x, y = point
x /= self.x_per_pixel
y /= self.y_per_pixel
x_screen, y_screen = self.to_screen_coords(x, y)
if fill == 'blue':
self.plot.create_image((x_screen, y_screen), image=self.blue_dot, tag=tag)
else:
self.plot.create_image((x_screen, y_screen), image=self.green_dot, tag=tag)
The Profile
I am a profiling newb, so it would be prudent to include some portion of the output of that profiler. I have sorted by 'cumtime' and highlighted the relevant methods.
update_plots calls scatter
scatter calls plot_point (above)
Note that scatter consumes 11.6% of the total run time.
The Question
Is there a more efficient method of creating points (and deleting them, though that doesn't take very long in tkinter) on a canvas?
If not, is there a more efficient way of creating the plot and embedding it into the tkinter interface?
I am somewhat open to using a different library, but I would like to keep it small and fast. I had thought that the tk canvas would be small and fast since it was functioning competently on machines with 1/10th of the power that a modern PC has.
More Info
After running a helpful answer below (Brian Oakley), I have updated results.
To explain the updated code a bit, I am using ovals again (I like the color control). I check to see if the tag exists. If it does not exist, then the new oval is created at the point specified. If the tag does exist, then the new coordinate is calculated and the move function is called.
def plot_point(self, point, fill='green', tag='data_point'):
if not fill:
fill = self.DEFAULT_LINE_COLOR
point_width = 2
# find the location of the point on the canvas
x, y = point
x /= self.x_per_pixel
y /= self.y_per_pixel
x_screen, y_screen = self.to_screen_coords(x, y)
x0 = x_screen - point_width
y0 = y_screen - point_width
x1 = x_screen + point_width
y1 = y_screen + point_width
# if the tag exists, then move the point, else create the point
point_ids = self.plot.find_withtag(tag)
if point_ids != ():
point_id = point_ids[0]
location = self.plot.coords(point_id)
current_x = location[0]
current_y = location[1]
move_x = x_screen - current_x
move_y = y_screen - current_y
self.plot.move(point_id, move_x, move_y)
else:
point = self.plot.create_oval(x0,
y0,
x1,
y1,
outline=fill,
fill=fill,
tag=tag)
The improvement is only slight, 10.4% vs. 11.6%.
The canvas has performance problems when many items are created (more specifically, when new object ids are created). Deleting objects doesn't help, the problem is in the ever increasing object ids which are never reused. This problem usually doesn't appear until you have 10's of thousands of items. If you're creating 256/second, you'll start to bump into that problem in just a minute or two.
You can completely eliminate this overhead if you create 128 objects off screen once, and then simply move them around rather than destroying and recreating them.

Memory leak when embedding and updating a matplotlib graph in a PyQt GUI

I am trying to embed a matplotlib graph that updates every second into a PyQt GUI main window.
In my program I call an update function every second using threading.Timer via the timer function shown below. I have a problem: my program grows bigger every second - at a rate of about 1k every 4 seconds. My initial thoughts are that the append function (that returns a new array in update_figure) does not delete the old array? Is it possible this is the cause of my problem?
def update_figure(self):
self.yAxis = np.append(self.yAxis, (getCO22()))
self.xAxis = np.append(self.xAxis, self.i)
# print(self.xAxis)
if len(self.yAxis) > 10:
self.yAxis = np.delete(self.yAxis, 0)
if len(self.xAxis) > 10:
self.xAxis = np.delete(self.xAxis, 0)
self.axes.plot(self.xAxis, self.yAxis, scaley=False)
self.axes.grid(True)
self.i = self.i + 1
self.draw()
This is my timer function - this is triggered by the click of a button in my PyQt GUI and then calls itself as you can see:
def timer(self):
getCH4()
getCO2()
getConnectedDevices()
self.dc.update_figure()
t = threading.Timer(1.0, self.timer)
t.start()
EDIT: I cant post my entire code because it requires a lot of .dll includes. So i'll try to explain what this program does.
In my GUI I want to show the my CO2 value over time. My get_co22 function just returns a float value and I'm 100% sure this works fine. With my timer, shown above, I want to keep append a value to a matplotlib graph - the Axes object is available to me as self.axes. I try to plot the last 10 values of the data.
EDIT 2: After some discussion in chat, I tried putting the call to update_figure() in a while loop and using just one thread to call it and was able to make this minimal example http://pastebin.com/RXya6Zah. This changed the structure of the code to call update_figure() to the following:
def task(self):
while True:
ui.dc.update_figure()
time.sleep(1.0)
def timer(self):
t = Timer(1.0, self.task())
t.start()
but now the program crashes after 5 iterations or so.
The problem is definitely not with how you are appending to your numpy array, or truncating it.
The problem here is with your threading model. Integrating calculation loops with a GUI control loop is difficult.
Fundamentally, you need your GUI threading to have control of when your update code is called (spawning a new thread to handle it if necessary) - so that
your code does not block the GUI updating,
the GUI updating does not block your code executing and
you don't spawn loads of threads holding multiple copies of objects (which might be where your memory leak comes from).
In this case, as your main window is controlled by PyQt4, you want to use a QTimer (see a simple example here)
So - alter your timer code to
def task(self):
getCH4()
getCO2()
getConnectedDevices()
self.dc.update_figure()
def timer(self):
self.t = QtCore.QTimer()
self.t.timeout.connect(self.task)
self.t.start(1000)
and this should work. Keeping the reference to the QTimer is essential - hence self.t = QtCore.QTimer() rather than t = QtCore.QTimer(), otherwise the QTimer object will be garbage collected.
Note:
This is a summary of a long thread in chat clarifying the issue and working through several possible solutions. In particular - the OP managed to mock up a simpler runnable example here: http://pastebin.com/RXya6Zah
and the fixed version of the full runnable example is here: http://pastebin.com/gv7Cmapr
The relevant code and explanation is above, but the links might help anyone who wants to replicate / solve the issue. Note that they require PyQt4 to be installed
if you are creating a new figure for every time this is quite common.
matplotlib do not free the figures you create, unless you ask it, somethink like:
pylab.close()
see How can I release memory after creating matplotlib figures

Categories