We Keep Coding

Dash running in a PyQt QThread() - most appropriate way to re-initialize with new data

At the moment I have a PyQt5 app running Dash in a QThread like so:
class MapView(QObject):
message = pyqtSignal(str)
shutting_down = pyqtSignal(bool)
def __init__(self, data):
super().__init__()
self.app = dash.Dash(__name__)
self.data = data
#manual callbacks
self.app.callback(
Output('hover-data', 'children'),
Input('basic-interactions', 'hoverData'))(self.display_hover_data)
self.app.callback(
Output('page-content', 'children'),
Input('url', 'pathname'))(self.shutdown)
#...more callbacks
def shutdown(self, pathname):
if pathname != '/shutdown':
return
print("Trying to shutdown map dash")
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Not running with the Werkzeug Server')
func()
self.shutting_down.emit(True)
#.....lots of stuff like graphs, the layout, a function to start the server etc
and then the threading code in my main pyqt5 app is:
def show_map(self):
self.mapthread = QThread()
self.map = map_dash.MapView(self.data)
self.map.moveToThread(self.mapthread)
self.map.message.connect(self.update_output)
self.map.shutting_down.connect(self.close_map_thread)
self.mapthread.finished.connect(self.open_project)
self.mapthread.started.connect(self.map.run)
self.mapthread.start()
self.browser.setUrl(QUrl('http://127.0.0.1:8050'))
self.update_output("Map plotted.")
My initial problem here was that if I tried to run show_map() with new data when Dash was already running (the user goes to File -> Open when a project is already running) , I would get the fatal Qthread: destroyed while thread is still running. So I embarked along the path of shutting down the flask server, then closing the thread, then going back to the function that opens a new project. It's like this:
User goes to File -> Open
def open_project() checks mapthread.isRunning()
It's not running so it opens a new project, creates a new QThread and new MapView instance
User goes to File -> Open again
The check in (2) returns True so the flask server is asked to shut down
After the server has shut down, the shutting_down signal causes the thread to be asked to quit() (I'm not sure how robust this is because it isn't a signal from flask, just a line after I've asked it to shut down. But it seems to work for now).
Once the thread has finished, the thread emits finished() which calls open_project() again
open_project this time sees that the thread is not running and allows the user to open a new file.
4 to 8 doesn't take too long to run but it all seems a bit complicated and the Dash layout glitches a bit for whatever reason. As it stands, when the QThread finishes under any other circumstances it would call open_project (although I could probably work around that). Is there a better way to do this? Can I feed the existing map instance new data somehow? I have read the documentation on dcc.Interval but that doesn't seem a great way to do it either...
Update (as per comments below):
Now what I'm doing is passing new data to the thread self.map.data = new_data, then using a callback on url to re-draw the map and refresh the layout. Exactly like this:
elif pathname == '/refresh':
self.draw_map()
self.set_layout()
But here is the problem. self.set_layout() refreshes dash with the OLD figure. I have verified that self.draw_map() is drawing a new figure. But calling /refresh a second time does cause dash to use the new figure. So is dash storing the old figure in a cache that hasn't been updated in the fraction of a second it takes to set the layout again? How can I make dash wait for the cache to be updated?

One way to do this is to pass the new data into the dash instance
self.map.data = new_data
but this does not refresh the layout and you can't simply call
self.map.set_layout() from the main thread. Instead you have to use the url callback like this:
def shutdown(self, pathname):
if pathname == '/shutdown':
print("Trying to shutdown map dash")
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Not running with the Werkzeug Server')
func()
self.shutting_down.emit(True)
elif pathname == '/refresh':
self.draw_map()
self.set_layout()
of course it would be more appropriate to call that method something like handle_url() now.
But there is a further problem. I'm guessing due to some caching issue, dash displays the old version of the figure unless the user manually calls refresh again. The solution is to add a dcc.Interval callback that updates the figure (it calls draw_map()). It doesn't need to be doing this every second, you can set the interval to 999999, for example. In fact you can set 'max_intervals=1' and it still works. As long as the callback is there, it will be called the first time the page refreshes and the figure will be updated.
In fact, with that in place, the draw_map() and set_layout() functions don't even need to be called in the url callback. So the code now looks like this:
def shutdown(self, pathname):
if pathname == '/shutdown':
print("Trying to shutdown map dash")
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Not running with the Werkzeug Server')
func()
self.shutting_down.emit(True)
elif pathname == '/refresh':
print("I'm doing nothing")
And it works. But the url callback itself IS necessary.

how to set timer properly in wxpython

My problem is: how to use Timer properly? Sometimes i missed the data.
part of the code:
import wx
import serial
store=[]
class all(wx.panel):
def __init__(self, parent):
…
self.SetSizer( bSizer17 )
self.Layout()
self.timer1 = wx.Timer()
self.timer1.SetOwner( self, 1 )
self.timer2 = wx.Timer()
self.timer2.SetOwner( self, 2 )
self.timer2.Start( 500 ) ### running when app begins
self.timer3 = wx.Timer()
self.timer3.SetOwner( self, 3 )
self.timer3.Start( 401) ### running when app begins
def timer1(self,event):
if self.timer1.IsRunning():
self.timer1.Stop()
else:
self.timer1.Start( 100 )
def timer2(self, event):
event.Skip()
def timer3(self, event):
event.Skip()
def timer1_plot(self, event)
…
plt.plot(x,y)
def timer2_store( self, event ):
for line in ser:
store.append(line)
with open("C:\\Users\\Desktop\\saved_data.txt","a") as f:
for line in store:
f.writelines(str(line)+ "\n")
def timer3_del( self, event ):
del store[:]
I have coming data from serial port per second. I use data for plotting and saving background(independent from plotting). For this i used wxTimer.
I have 3 buttons(bind with timers), 2 of them are hidden(2 timers run automatically).
1.button is for plotting. When i press, timer run.
2.button is for storing and writing to .txt file and delete from the list
3.button is for deleting the stored data in the list
self.timer1.Start( 100 ) # running when button press
self.timer2.Start( 500 ) # running when app begins
self.timer3.Start( 401) # running when app begins
range between timer is good? Should i run 2 of them automatically? What can you suggest? Any help would be appreciated.

This is pretty old, so you may have already found a solution... but here is my advice.
My project monitors 2 serial ports, one has a scale that sends a data packet every second, the other has a barcode scanner - very random when data arrives.
I use 1 thread per serial port, that just timer.sleep's waiting for serial input, and stores it in a shared object (1 object per thread). Then I have one timer that updates the state machine, it looks at each shared object, and updates the GUI accordingly.
If this was my project, I would suggest the following changes:
Don't call timer event .Skip() nothing else should be chained after your callback.
Create a thread for each serial port or other async communication source.
Be sure to mark the threads as daemon's or explicitly kill them when shutting down.
Use one timer that updates everything, use the lowest frequency that gives your users acceptable response.
Good luck.

Listening for a threading Event in python

first time SO user, please excuse any etiquette errors. I'm trying to implement a multithreaded program in python and am having troubles. This is no doubt due to a lack of understanding of how threading is implemented, but hopefully you can help me figure it out.
I have a basic program that continually listens for messages on a serial port and can then print/save/process/etc them, which works fine. It basically looks like this:
import serial
def main():
usb = serial.Serial('/dev/cu.usbserial-A603UBRB', 57600) #open serial w\ baud rate
while True:
line = usb.readline()
print(line)
However what I want to do is continually listen for the messages on a serial port, but not necessarily do anything with them. This should run in the background, and meanwhile in the foreground I want to have some kind of interface where the user can command the program to read/use/save these data for a while and then stop again.
So I created the following code:
import time
import serial
import threading
# this runs in the background constantly, reading the serial bus input
class serial_listener(threading.Thread):
def __init__(self, line, event):
super(serial_listener, self).__init__()
self.event = threading.Event()
self.line = ''
self.usb = serial.Serial('/dev/cu.usbserial-A603UBRB', 57600)
def run(self):
while True:
self.line = self.usb.readline()
self.event.set()
self.event.clear()
time.sleep(0.01)
# this lets the user command the software to record several values from serial
class record_data(threading.Thread):
def __init__(self):
super(record_data, self).__init__()
self.line = ''
self.event = threading.Event()
self.ser = serial_listener(self.line,self.event)
self.ser.start() #run thread
def run(self):
while(True):
user_input = raw_input('Record data: ')
if user_input == 'r':
event_counter = 0
while(event_counter < 16):
self.event.wait()
print(self.line)
event_counter += 1
# this is going to be the mother function
def main():
dat = record_data()
dat.start()
# this makes the code behave like C code.
if __name__ == '__main__':
main()
It compiles and runs, but when I order the program to record by typing r into the CLI, nothing happens. It doesn't seem to be receiving any events.
Any clues how to make this work? Workarounds are also fine, the only thing is that I can't constantly open and close the serial interface, it has to remain open the whole time, or else the device stops working until un/replugged.

Instead of using multiple threads, I would suggest using multiple processes. When you use threads, you have to think about the global interpreter lock. So you either listen to events or do something in your main thread. Both at the same time will not work.
When using multiple processes I would then use a queue to forward the events from your watchdog that you would like to handle. Or you could code your own event handler. Here you can find an example for multiprocess event handlers

pyqt main window keeps crashing after receiving data from a thread

i have a QMainWindow that starts a QThread and waits for data from the network. updates UI when it receive any data.
the problem is : it sometimes crash. and sometimes doesn't , all i do i start it and wait for data.
here is the thread class :
class ListenerThread(QtCore.QThread):
def __init__(self,host,port,window):
super(ListenerThread,self).__init__(window)
self.host = host
self.port = port
self.window = window
def run(self):
soc = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
soc.bind((self.host, self.port))
while True:
data, address = soc.recvfrom(9999)
print address
if data:
dataList = data.split("\\")
company = dataList[1]
projectName = dataList[2]
assets = dataList[3]
assetType = dataList[4]
assetName = dataList[5]
# parent here is the main window(the main thread) : updateCombo is a function that updates combo box inside the main window
self.parent().updateCombo(self.window.comboBoxCompany,company)
self.parent().updateCombo(self.window.dropDownProjects,projectName)
self.parent().select(assets,assetName)
why is this happening ?? put in mind that the main Window by itself works fine.
the function (updateCombo) is working fine also ( when you call it from it's class).
but main window keeps crashing when i send data ! any idea why ?

GUI widgets may be accessed only from main thread, meaning the thread that calls QApplication.exec(). Access to GUI widgets from any other thread – what you're doing with your calls to self.parent() – is undefined behaviour, in your case this means crashes.
You signals and slots to communicate between background threads and the GUI in a safe manner.
And please read the documentation about Qt's threading functionality, because the above is actually essential knowledge when dealing with multi-threaded GUI applications, not only in Qt, but in any other GUI framework, too.

Showing data in a GUI where the data comes from an outside source

I'm kind of lost on how to approach this problem, I'd like to write a GUI ideally using Tkinter with python, but I initially started with Qt and found that the problem extends either with all GUI frameworks or my limited understanding.
The data in this case is coming from a named pipe, and I'd like to display whatever comes through the pipe into a textbox. I've tried having one thread listen on the pipe and another create the GUI, but in both cases one thread always seems to hang or the GUI never gets created.
Any suggestions?

Here is the way I would do it (on windows):
import wx, wx.lib.newevent, threading
import win32event, win32pipe, win32file, pywintypes, winerror
NewMessage, EVT_NEW_MESSAGE = wx.lib.newevent.NewEvent()
class MessageNotifier(threading.Thread):
pipe_name = r"\\.\pipe\named_pipe_demo"
def __init__(self, frame):
threading.Thread.__init__(self)
self.frame = frame
def run(self):
open_mode = win32pipe.PIPE_ACCESS_DUPLEX | win32file.FILE_FLAG_OVERLAPPED
pipe_mode = win32pipe.PIPE_TYPE_MESSAGE
sa = pywintypes.SECURITY_ATTRIBUTES()
sa.SetSecurityDescriptorDacl(1, None, 0)
pipe_handle = win32pipe.CreateNamedPipe(
self.pipe_name, open_mode, pipe_mode,
win32pipe.PIPE_UNLIMITED_INSTANCES,
0, 0, 6000, sa
)
overlapped = pywintypes.OVERLAPPED()
overlapped.hEvent = win32event.CreateEvent(None, 0, 0, None)
while 1:
try:
hr = win32pipe.ConnectNamedPipe(pipe_handle, overlapped)
except:
# Error connecting pipe
pipe_handle.Close()
break
if hr == winerror.ERROR_PIPE_CONNECTED:
# Client is fast, and already connected - signal event
win32event.SetEvent(overlapped.hEvent)
rc = win32event.WaitForSingleObject(
overlapped.hEvent, win32event.INFINITE
)
if rc == win32event.WAIT_OBJECT_0:
try:
hr, data = win32file.ReadFile(pipe_handle, 64)
win32file.WriteFile(pipe_handle, "ok")
win32pipe.DisconnectNamedPipe(pipe_handle)
wx.PostEvent(self.frame, NewMessage(data=data))
except win32file.error:
continue
class Messages(wx.Frame):
def __init__(self):
wx.Frame.__init__(self, None)
self.messages = wx.TextCtrl(self, style=wx.TE_MULTILINE | wx.TE_READONLY)
self.Bind(EVT_NEW_MESSAGE, self.On_Update)
def On_Update(self, event):
self.messages.Value += "\n" + event.data
app = wx.PySimpleApp()
app.TopWindow = Messages()
app.TopWindow.Show()
MessageNotifier(app.TopWindow).start()
app.MainLoop()
Test it by sending some data with:
import win32pipe
print win32pipe.CallNamedPipe(r"\\.\pipe\named_pipe_demo", "Hello", 64, 0)
(you also get a response in this case)

When I did something like this I used a separate thread listening on the pipe. The thread had a pointer/handle back to the GUI so it could send the data to be displayed.
I suppose you could do it in the GUI's update/event loop, but you'd have to make sure it's doing non-blocking reads on the pipe. I did it in a separate thread because I had to do lots of processing on the data that came through.
Oh and when you're doing the displaying, make sure you do it in non-trivial "chunks" at a time. It's very easy to max out the message queue (on Windows at least) that's sending the update commands to the textbox.

In the past when I've had GUI's reading data off of external things (eg: ethernet sockets), I've had a separate thread that handles servicing the external thing, and a timed callback (generally set to something like half a second) to update the GUI widget that displays the external data.