We Keep Coding

Confusion with matplotlib timers

I tried to implement the workaround mentioned here to the problem of the figure not updating properly when a draw_event is triggered once a user zooms into a plot. This was meant to improve this answer. In the solution, a Timer is added to the Canvas of the Figure, which delays the draw function for a short while.
In the linked answer this is done for a single figure and a reference to the timer is stored as self.timer to the class that contains the update function. I would like to have an a bit more general behaviour and allow this for many Figure instances. Therefore I tried not to save the Timer reference (I don't need it anymore). But when I do so, the script crashes with a Segmentation Fault (no traceback). Here is my code:
from matplotlib import pyplot as plt
import numpy as np
##plt.switch_backend('TkAgg')
class DrawEventHandler:
def __init__(self):
x = np.linspace(0,1,10)
y = np.sin(x)
self.fig, self.ax = plt.subplots()
self.ax.plot(x,y)
self.ax.figure.canvas.mpl_connect('draw_event', self.update)
def update(self, event = None):
self._redraw_later_ok()
##self._redraw_later_not_ok(self.fig)
def _redraw_later_ok(self):
self.timer = self.fig.canvas.new_timer(interval=10)
self.timer.single_shot = True
self.timer.add_callback(lambda : self.fig.canvas.draw_idle())
self.timer.start()
def _redraw_later_not_ok(self, fig):
print('start')
timer = fig.canvas.new_timer(interval=10)
timer.single_shot = True
timer.add_callback(lambda : fig.canvas.draw_idle())
timer.start()
print('stop')
if __name__ == '__main__':
my_handler = DrawEventHandler()
plt.show()
The original solution is implemented as _redraw_later_ok(), which works fine for me. The problematic solution is called `_redraw_later_not_ok()', which produces this output:
start
stop
Segmentation fault: 11
I use a Mac with High Sierra, Python 3.6.4 or Python 2.7.14 and Matplotlib 2.1.1 with the MacOSX backend. When I switch to the TkAgg backend (which is terribly slow), the code works fine. Can anybody explain what is going on?

As usual with interactive features which use the event loop of a GUI you need to keep a reference to the object that handles the callback.
The problem with the approach of not storing the timer in this code
def _redraw_later_not_ok(self, fig):
print('start')
timer = fig.canvas.new_timer(interval=10)
timer.single_shot = True
timer.add_callback(lambda : fig.canvas.draw_idle())
timer.start()
print('stop')
is that timer gets garbage collected once the _redraw_later_not_ok function terminates (i.e. directly after stop is printed). At this point there would be a pointer to a place in memory, which may or may not store the callback (any more). An attempt by the python interpreter to call the callback would hence (most probably) fail.
The solution is indeed to always keep a reference to the object that steers the callback. This is the solution shown in _redraw_later_ok, where the timer is made a class attribute, self.timer. Such that it can later be used at the time the callback is called.
I do not understand in how far using this working approach would prevent the use of several figures, so it may make sense to create a MWE with two figures that shows the problem clearly.

trying to draw over sprite or change picture pyglet

I am trying to learn pyglet and practice some python coding with a questionnaire thing, but I am unable to find a way to make the background picture be removed or drawn on top of or something for 10 seconds. I am new and am lacking in a lot of the knowledge I would need, thank you for helping!
import pyglet
from pyglet.window import Window
from pyglet.window import key
from pyglet import image
import time
card1 = False
cat_image = pyglet.image.load("cat.png")
dog_image = pyglet.image.load("dog.png")
image = pyglet.image.load("backg.png")
background_sprite = pyglet.sprite.Sprite(image)
cat = pyglet.sprite.Sprite(cat_image)
dog = pyglet.sprite.Sprite(dog_image)
window = pyglet.window.Window(638, 404, "Life")
mouse_pos_x = 0
mouse_pos_y = 0
catmeme = pyglet.image.load("catmeme.png")
sprite_catmeme = pyglet.sprite.Sprite(catmeme)
#window.event
def on_draw():
window.clear()
background_sprite.draw()
card_draw1(63, 192, 385, 192)
def card1():
while time.time() < (time.time() + 10):
window.clear()
sprite_catmeme.draw()
#window.event
def card_draw1(x1, y1, x2, y2):
cat.set_position(x1, y1)
dog.set_position(x2, y2)
cat.draw()
dog.draw()
def card_draw2():
pass
#window.event
def on_mouse_press(x, y, button, modifiers):
if x > cat.x and x < (cat.x + cat.width):
if y > cat.y and y < (cat.y + cat.height):
card1()
game = True
while game:
on_draw()
pyglet.app.run()

There's a few flaws in the order and in which you do things.
I will try my best to describe them and give you a piece of code that might work better for what your need is.
I also think your description of the problem is a bit of an XY Problem which is quite common when asking for help on complex matters where you think you're close to a solution, so you're asking for help on the solution you've come up with and not the problem.
I'm assuming you want to show a "Splash screen" for 10 seconds, which happens to be your background? And then present the cat.png and dog.png ontop of it, correct?
If that's the case, here's where you probably need to change things in order for it to work:
The draw() function
It doesn't really update the screen much, it simply adds things to the graphical memory. What updates the screen is you or something telling the graphics library that you're done adding things to the screen and it's time to update everything you've .draw()'n. So the last thing you need in the loop would be window.flip() in order for the things you've drawn to actually show.
Your things might show if you try to wiggle the window around, it should trigger a re-draw of the scene because of how the internal mechanics of pyglet work..
If you don't call .flip() - odds are probable that the redraw() call will never occur - which again, is a internal mechanism of Pyglet/GL that tells the graphics card that something has been updated, we're done updating and it's time to redraw the scene.
a scene
This is the word most commonly used for what the user is seeing.
I'll probably throw this around a lot in my text, so it's good to know that this is what the user is seeing, not what you've .draw()'n or what's been deleted, it's the last current rendering of the graphics card to the monitor.
But because of how graphical buffers work we've might have removed or added content to the memory without actually drawing it yet. Keep this in mind.
The pyglet.app.run() call
This is a never ending loop in itself, so having that in a while game: loop doesn't really make sense because .run() will "hang" your entire application, any code you want to execute needs to be in def on_draw or an event that is generated from within the graphical code itself.
To better understand this, have a look at my code, i've pasted it around a couple of times here on SO over the years and it's a basic model of two custom classes that inherits the behavior of Pyglet but lets you design your own classes to behave slightly differently.
And most of the functionality is under on_??? functions, which is almost always a function used to catch Events. Pyglet has a lot of these built in, and we're going to override them with our own (but the names must be the same)
import pyglet
from pyglet.gl import *
key = pyglet.window.key
class CustomSprite(pyglet.sprite.Sprite):
def __init__(self, texture_file, x=0, y=0):
## Must load the texture as a image resource before initializing class:Sprite()
self.texture = pyglet.image.load(texture_file)
super(CustomSprite, self).__init__(self.texture)
self.x = x
self.y = y
def _draw(self):
self.draw()
class MainScreen(pyglet.window.Window):
def __init__ (self):
super(MainScreen, self).__init__(800, 600, fullscreen = False)
self.x, self.y = 0, 0
self.bg = CustomSprite('bg.jpg')
self.sprites = {}
self.alive = 1
def on_draw(self):
self.render()
def on_close(self):
self.alive = 0
def on_key_press(self, symbol, modifiers):
if symbol == key.ESCAPE: # [ESC]
self.alive = 0
elif symbol == key.C:
print('Rendering cat')
self.sprites['cat'] = CustomSprite('cat.png', x=10, y=10)
elif symbol == key.D:
self.sprites['dog'] = CustomSprite('dog.png', x=100, y=100)
def render(self):
self.clear()
self.bg.draw()
for sprite_name, sprite_obj in self.sprites.items():
sprite_obj._draw()
self.flip()
def run(self):
while self.alive == 1:
self.render()
# -----------> This is key <----------
# This is what replaces pyglet.app.run()
# but is required for the GUI to not freeze
#
event = self.dispatch_events()
x = MainScreen()
x.run()
Now, this code is kept simple on purpose, the full code I usually paste on SO can be found at Torxed/PygletGui, the gui.py is where most of this comes from and it's the main loop.
What I do here is simply replace the Decorators by using "actual" functions inside a class. The class itself inherits the functions from a traditional pyglet.window.Window, and as soon as you name the functions the same as the inherited onces, you replace the core functionality of Window() with whatever you decide.. In this case, i mimic the same functions but add a few of my own.
on_key_press
One such example is on_key_press(), which normally just contain a pass call and does nothing, here, we check if key.C is pressed, and if so - we add a item to self.sprites.. self.sprites just so happen to be in our render() loop, anything in there will be rendered ontop of a background.
Here's the pictures I used:
(named bg.jpg, cat.png, dog.png - note the different file endings)
class:CustomSprite
CustomSprite is a very simple class designed to make your life easier at this point, nothing else. It's very limited in functionality but the little it do is awesome.
It's soul purpose is to take a file name, load it as an image and you can treat the object like a traditional pyglet.sprite.Sprite, meaning you can move it around and manipulate it in many ways.
It saves a few lines of code having to load all the images you need and as you can see in gui_classes_generic.py you can add a heap of functions that's "invisible" and normally not readily availbale to a normal sprite class.
I use this a bunch! But the code gets complicated real fast so I kept this post simple on purpose.
the flip function
Even in my class, I still need to use flip() in order to update the contents of the screen. This is because .clear() clears the window as you would expect, that also triggers a redraw of the scene.
bg.draw() might in some cases trigger a redraw if the data is big enough or if something else happens, for instance you move the window.
but calling .flip() will tell the GL backend to force a redraw.
Further optimizations
There's a thing called batched rendering, basically the graphic card is designed to take enormous ammounts of data and render it in one go, so calling .draw() on several items will only clog the CPU before the GPU even gets a chance to shine. Read more about Batched rendering and graphics! It will save you a lot of frame rates.
Another thing is to keep as little functionality as possible in the render() loop and use the event triggers as your main source of coding style.
Pyglet does a good job of being fast, especially if you only do things on event driven tasks.
Try to avoid timers, but if you really do need to use time for things, such as removing cat.png after a certain ammount of time, use the clock/time event to call a function that removes the cat. Do not try to use your own t = time() style of code unless you know where you're putting it and why. There's a good timer, I rarely use it.. But you should if you're starting off.
This has been one hell of a wall of text, I hope it educated you some what in the life of graphics and stuff. Keep going, it's a hurdle to get into this kind of stuff but it's quite rewarding once you've mastered it (I still haven't) :)

Memory leak when embedding and updating a matplotlib graph in a PyQt GUI

I am trying to embed a matplotlib graph that updates every second into a PyQt GUI main window.
In my program I call an update function every second using threading.Timer via the timer function shown below. I have a problem: my program grows bigger every second - at a rate of about 1k every 4 seconds. My initial thoughts are that the append function (that returns a new array in update_figure) does not delete the old array? Is it possible this is the cause of my problem?
def update_figure(self):
self.yAxis = np.append(self.yAxis, (getCO22()))
self.xAxis = np.append(self.xAxis, self.i)
# print(self.xAxis)
if len(self.yAxis) > 10:
self.yAxis = np.delete(self.yAxis, 0)
if len(self.xAxis) > 10:
self.xAxis = np.delete(self.xAxis, 0)
self.axes.plot(self.xAxis, self.yAxis, scaley=False)
self.axes.grid(True)
self.i = self.i + 1
self.draw()
This is my timer function - this is triggered by the click of a button in my PyQt GUI and then calls itself as you can see:
def timer(self):
getCH4()
getCO2()
getConnectedDevices()
self.dc.update_figure()
t = threading.Timer(1.0, self.timer)
t.start()
EDIT: I cant post my entire code because it requires a lot of .dll includes. So i'll try to explain what this program does.
In my GUI I want to show the my CO2 value over time. My get_co22 function just returns a float value and I'm 100% sure this works fine. With my timer, shown above, I want to keep append a value to a matplotlib graph - the Axes object is available to me as self.axes. I try to plot the last 10 values of the data.
EDIT 2: After some discussion in chat, I tried putting the call to update_figure() in a while loop and using just one thread to call it and was able to make this minimal example http://pastebin.com/RXya6Zah. This changed the structure of the code to call update_figure() to the following:
def task(self):
while True:
ui.dc.update_figure()
time.sleep(1.0)
def timer(self):
t = Timer(1.0, self.task())
t.start()
but now the program crashes after 5 iterations or so.

The problem is definitely not with how you are appending to your numpy array, or truncating it.
The problem here is with your threading model. Integrating calculation loops with a GUI control loop is difficult.
Fundamentally, you need your GUI threading to have control of when your update code is called (spawning a new thread to handle it if necessary) - so that
your code does not block the GUI updating,
the GUI updating does not block your code executing and
you don't spawn loads of threads holding multiple copies of objects (which might be where your memory leak comes from).
In this case, as your main window is controlled by PyQt4, you want to use a QTimer (see a simple example here)
So - alter your timer code to
def task(self):
getCH4()
getCO2()
getConnectedDevices()
self.dc.update_figure()
def timer(self):
self.t = QtCore.QTimer()
self.t.timeout.connect(self.task)
self.t.start(1000)
and this should work. Keeping the reference to the QTimer is essential - hence self.t = QtCore.QTimer() rather than t = QtCore.QTimer(), otherwise the QTimer object will be garbage collected.
Note:
This is a summary of a long thread in chat clarifying the issue and working through several possible solutions. In particular - the OP managed to mock up a simpler runnable example here: http://pastebin.com/RXya6Zah
and the fixed version of the full runnable example is here: http://pastebin.com/gv7Cmapr
The relevant code and explanation is above, but the links might help anyone who wants to replicate / solve the issue. Note that they require PyQt4 to be installed

if you are creating a new figure for every time this is quite common.
matplotlib do not free the figures you create, unless you ask it, somethink like:
pylab.close()
see How can I release memory after creating matplotlib figures

Python realtime mousedata with pyqtgraph

In my epic struggle to process raw mouse data on my ubuntu (14.04) OS with python, with alot help from here i am stuck again. It seems very hard for me to understand the "easyness" of pyqtgraph. all i want to do is to wrap the code i have now into a nice little gui with a start/pause/stop button, a list widget to show the numbers and a plot to let me see whats happening. I guess the main problem for me is, that i dont quite understand this whole event thing in pyqt.
anyway taking a "easy" example which has the widgets i want, i fail to implement my code(edited more minimalistic):
#!/usr/bin/python
import threading
import struct
import time
import numpy as np
from PyQt4 import QtGui # (the example applies equally well to PySide)
from PyQt4 import QtCore
import pyqtgraph as pg
##
data =[(0,0)]
sum_data = [(0,0)]
file = open("/dev/input/mouse2", "rb")
def getMouseEvent():
buf = file.read(3);
#python 2 & 3 compatibility
button = buf[0] if isinstance(buf[0], int) else ord(buf[0])
x,y = struct.unpack( "bb", buf[1:] );
print x,y
return x, y
def mouseCollect():
while True:
data.append(getMouseEvent())
sum_data.append(tuple(map(sum,zip(sum_data[-1],data[-1]))))
plot.plot(sum_data[0], clear=True)
pg.QtGui.QApplication.processEvents()
print sum_data[-1]
## Always start by initializing Qt (only once per application)
app = QtGui.QApplication([])
## Define a top-level widget to hold everything
w = QtGui.QWidget()
## Create some widgets to be placed inside
btn1 = QtGui.QPushButton('Start')
listw = QtGui.QListWidget()
plot = pg.PlotWidget()
def start_btn():
print 'test'
threading.Thread(target=mouseCollect).start()
btn1.clicked.connect(start_btn)
## Create a grid layout to manage the widgets size and position
layout = QtGui.QGridLayout()
w.setLayout(layout)
## Add widgets to the layout in their proper positions
layout.addWidget(btn1, 0, 0) # button goes in upper-left
layout.addWidget(plot, 0, 1, 4, 1)
## Display the widget as a new window
w.show()
## Start the Qt event loop
app.exec_()
##------------------------------------------------------------------
when i press the start button, the window just freezes and nothing happens. my thought was, if i press the button, it connects to the methot state there and it just doing its stuff. okay i have an infinite loop, but at least i thought i should see something. any help is apreciated, also any tips for a good reading about the matter is very welcome.
regards
Edit: inserted a thread as suggested by echocage

The best approach for repetitive updates is to use a QTimer. In this way you allow program control to go back to the Qt event loop after every update (no infinite loops allowed), and Qt periodically invokes your update function for you.
For example, see one of the many the updating plot examples included with pyqtgraph: https://github.com/pyqtgraph/pyqtgraph/blob/develop/examples/Plotting.py#L58
Threading is very difficult to do correctly and when it's done wrong you usually end up with crashing that is difficult to debug. I recommend to avoid threading until you are very confident with the event system and familiar with the pitfalls of threading.

I finally found it. I somehow never realized, that you HAVE to use numpy arrays. also i didnt use curve.setdata for plotting.
the more or less final code(not full code) now looks like this:
class mouseCollect(QtCore.QThread):
def __init__(self):
QtCore.QThread.__init__(self)
def run (self):
global e, curve, curve2, data1, data2
while e.wait():
tmp = getMouseEvent() #returns tuples in form of (x,y)
data1 = np.append(data1, tmp[0] )
data2 = np.append(data2, tmp[1] )
sum1= np.cumsum(data1)
sum2= np.cumsum(data2)
curve.setData(y=sum1)
curve2.setData(y=sum2)
guiPE # process event thingy
def stop(self):
e.clear()
well, it is a not really written efficiently, but it works :)

wxpython interface becomes unresponsive when running live matplotlib draw() function

So I am using wxpython to make a GUI for a program. I have also embedded matplotlib graphs in this program.
My problem is when I try to use draw() to update the plot continuously, my program becomes unresponsible, however the matplotlib graph is still updating.
Here is part of my code, and how I execute all this,
def update(self, e):
if self.liveMode:
self.tune.plot(); # new plot
self.canvas.draw();
self.startLive(e);
def startLive(self, e):
self.liveMode = False;
refFreq = 500e6 / 80;
qx = self.pv1.get();
qy = self.pv2.get();
self.tune.newWorkingpoint(refFreq/qx, refFreq/qy);
self.tune.liveUpdate();
time.sleep(0.3);
self.update(e);
When the 'live mode' button is pressed in my program the 'update' method gets called.
I plot all the background lines, and then repeatedly get new point values from elsewhere and plot this as a circle on the graph and update with the draw() method. Of course since I am essentially in a continuous loop, nothing else will work until this process is finished. My goal was to just put a 'stop' button to stop this live plotting loop, but since everything becomes unresponsive when matplotlib is drawing, no buttons are clickable, and I have to essentially stop compiling to stop the program.
Is there anyway around this? I have looked into threading, but I get the same problems with this.
EDIT: ----------------------------------------------------------------------------------------------------------------------------I have got it working using this page as a guide. However, after a few seconds I get this fatal error and my program crashes.
'python2.7: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.'
Here is the method I implemented. When a start button is pressed, 'startlive' starts, and when a stop button is pressed, 'stoplive' starts. Everything works fine, and my matplotlib graphs are updated properly without my wxpython GUI becoming unresponsive.
The problem is after a few seconds I get the fatal error and my program crashes. I believe this is most likely tied to the canvas.draw() method updating matplotlib inside a thread. Any ideas on how to get around this problem?
def startLive(self, e):
self.livemode.change(True);
self.worker = CountingThread(self.panel,self.tune,self.canvas,self.livemode, self.pv1, self.pv2);
self.worker.start();
def stopLive(self, evt):
self.livemode.change(False);
class CountingThread(threading.Thread):
def __init__(self, parent, tune, canvas, livemode, pv1, pv2):
threading.Thread.__init__(self)
self._parent = parent;
self._tune = tune;
self._canvas = canvas;
self._livemode = livemode;
self._pv1 = pv1;
self._pv2 = pv2;
def run(self):
refFreq = 500e6 / 80;
i=0;
while self._livemode.get():
self._tune.newWorkingpoint(refFreq / self._pv1.get(), refFreq / self._pv2.get());
self._tune.liveUpdate();
self._canvas.draw();
time.sleep(0.2);
i+=1;
print(i)
class livemode:
livemode=False;
def __init__(self):
self.livemode = False;
def change(self, livemode):
self.livemode = livemode;
def get(self):
return self.livemode;
--------------------------------EDIT---------------------------------
I solved the problem if anyone is interested,
In the threading run method, I removed the loop and simply had one instance of 'updating'. After the update, wx.PostEvent is called which calls a method back in the main class containing the wx.App which updates the plot by calling draw(). After this, the threading is restarted again and the same process repeats. My GUI continues to work and the plotting speed is still really fast. The important methods are below,
Threading.start() is called and the below exectures
def run(self):
refFreq = 500e6 / 80;
self._tune.newWorkingpoint(refFreq / self._pv1.get(), refFreq /self._pv2.get());
self._tune.liveUpdate();
evt = CountEvent(myEVT_PLOT, -1);
wx.PostEvent(self._parent, evt);
wx.PostEvent calls the following new method in main wx.App class,
def continueLive(self, evt):
if self.livemode.get():
self.canvas.draw();
self.startLive(evt)
The plot is updated and the whole process starts over.
This methods keeps the GUI responsive and the plotting speed does not slow down.

I think it might be a stack issue. Your update loop is recursive and the stack has a limited size (every function call gets pushed to the stack and removed once the function returns).
You should change it to a iterative version.
def update(self, e):
if self.liveMode:
self.tune.plot(); # new plot
self.canvas.draw();
#self.startLive(e); # causes endless recursion
def startLive(self, e):
while (self.isInLiveMode): # some flag here that gets set to False once the mode is deactivated
self.liveMode = False;
refFreq = 500e6 / 80;
qx = self.pv1.get();
qy = self.pv2.get();
self.tune.newWorkingpoint(refFreq/qx, refFreq/qy);
self.tune.liveUpdate();
time.sleep(0.3);
self.update(e);
Well i assume self.liveMode should be that flag.
Hope this helps, LG
Daniel