We Keep Coding

How to listen for a file change in urwid?

I want to remote control a python application which uses urwid for the user interface.
My idea was to create a file, pass it's name as command line argument to the application and whenever I write to the file the application reads from that file.
Urwid's event loop has a method watch_file(fd, callback).
This method is described as "Call callback() when fd has some data to read."
This sounds exactly like what I want to have, but it causes an infinite loop.
callback is executed as often as possible, despite the fact that the file is empty.
Even if I delete the file, callback is still called.
#!/usr/bin/env python3
import urwid
import atexit
def onkeypress(key, size=None):
if key == 'q':
raise urwid.ExitMainLoop()
text.set_text(key)
def onfilechange():
text.set_text(cmdfile.read())
# clear file so that I don't read already executed commands again
# and don't run into an infinite loop - but I am doing that anyway
with open(cmdfile.name, 'w') as f:
pass
cmdfile = open('/tmp/cmd', 'rt')
atexit.register(cmdfile.close)
text = urwid.Text("hello world")
filler = urwid.Filler(text)
loop = urwid.MainLoop(filler, unhandled_input=onkeypress)
loop.watch_file(cmdfile, onfilechange)
if __name__ == '__main__':
loop.run()
(My initial idea was to open the file only for reading instead of keeping it open all the time but fd has to be a file object, not a path.)
Urwid offers several different event loops.
By default, SelectEventLoop is used.
GLibEventLoop has the same behaviour, it runs into an infinite loop.
AsyncioEventLoop instead throws an "operation not permitted" exception.
TwistedEventLoop and TornadoEventLoop would need additional software to be installed.
I have considered using the independent watchdog library but it seems accessing the user interface from another thread would require to write a new loop, see this stack overflow question.
The answer to that question recommends polling instead which I would prefer to avoid.
If urwid specifically provides a method to watch a file I cannot believe that it does not work in any implementation.
So what am I doing wrong?
How do I react to a file change in a python/urwid application?
EDIT:
I have tried using named pipes (and removed the code to clear the file) but visually it has the same behaviour: the app does not start.
Audibly, however, there is a great difference: It does not go into the infinite loop until I write to the file.
Before I write to the file callback is not called but the app is not started either, it just does nothing.
After I write to the file, it behaves as described above for regular files.

I have found the following work around: read a named pipe in another thread, safe each line in a queue and poll in the UI thread to see if something is in the queue.
Create the named pipe with mkfifo /tmp/mypipe.
Then write to it with echo >>/tmp/mypipe "some text".
#!/usr/bin/env python3
import os
import threading
import queue
import urwid
class App:
POLL_TIME_S = .5
def __init__(self):
self.text = urwid.Text("hello world")
self.filler = urwid.Filler(self.text)
self.loop = urwid.MainLoop(self.filler, unhandled_input=self.onkeypress)
def watch_pipe(self, path):
self._cmd_pipe = path
self.queue = queue.Queue()
threading.Thread(target=self._read_pipe_thread, args=(path,)).start()
self.loop.set_alarm_in(0, self._poll_queue)
def _read_pipe_thread(self, path):
while self._cmd_pipe:
with open(path, 'rt') as pipe:
for ln in pipe:
self.queue.put(ln)
self.queue.put("!! EOF !!")
def _poll_queue(self, loop, args):
while not self.queue.empty():
ln = self.queue.get()
self.text.set_text(ln)
self.loop.set_alarm_in(self.POLL_TIME_S, self._poll_queue)
def close(self):
path = self._cmd_pipe
# stop reading
self._cmd_pipe = None
with open(path, 'wt') as pipe:
pipe.write("")
os.remove(path)
def run(self):
self.loop.run()
def onkeypress(self, key, size=None):
if key == 'q':
raise urwid.ExitMainLoop()
self.text.set_text(key)
if __name__ == '__main__':
a = App()
a.watch_pipe('/tmp/mypipe')
a.run()
a.close()

GUI not responding after child processes finishes due to redirection of Stdout

I've spent the past few days reading various threads about making tkinter thread-safe and running children without blocking the main thread. I thought I had arrived at a solution that allowed my code to run as I wanted it to, but now my main thread becomes non-responsive when my child process finishes. I can move the window around but the GUI part shows a loading cursor, whites out, and says "Not Responding" in the title of the window. I can let it sit like that forever and nothing will happen. I know what part of the code is causing the problem but I am not sure why it's causing the GUI to freeze. I'm using Windows.
I want my GUI to run another process using multiprocess. I have sys.stdout and sys.stderr routed to a queue and I use threading to create a thread that holds an automatic queue checker that updates the GUI every 100 ms so my GUI updates in "real time". I have tried every way of sending the child's stdout/stderr to the GUI and this is the only way that works the way I want it to (except for the freezing bit), so I would like to find out why it's freezing. Or I would like help setting up a proper way of sending the child's output to the GUI. I have tried every method I could find and I could not get them to work.
My main thread:
#### _______________IMPORT MODULES_________________###
import Tkinter
import multiprocessing
import sys
from threading import Thread
import qBMPchugger
###____________Widgets__________________###
class InputBox(Tkinter.Tk):
def __init__(self,parent):
Tkinter.Tk.__init__(self, parent)
self.parent = parent
self.initialize()
def initialize(self):
# Styles
self.grid()
# Approval
self.OKbutton = Tkinter.Button(self, text=u"OK", command=self.OKgo, anchor="e")
self.OKbutton.pack(side="right")
self.view = Tkinter.Text(self)
self.view.pack(side="left")
self.scroll = Tkinter.Scrollbar(self, orient=Tkinter.VERTICAL)
self.scroll.config(command=self.view.yview)
self.view.config(yscrollcommand=self.scroll.set)
self.scroll.pack(side="left")
def write(self, text):
self.view.insert("end", text)
def OKgo(self):
sys.stdout = self
sys.stderr = self
checker = Thread(target=self._update)
checker.daemon = True
checker.start()
self.view.delete(1.0, "end")
self.update_idletasks()
print("Loading user-specified inputs...")
path = "C:/"
inarg = (q, path)
print("Creating the program environment and importing modules...")
# Starts the text monitor to read output from the child process, BMPchugger
p = multiprocessing.Process(target=qBMPchugger.BMPcode, args=inarg)
p.daemon = 1
p.start()
def _update(self):
msg = q.get()
self.write(msg)
self.update_idletasks()
self.after(100, self._update)
if __name__ == "__main__":
app = InputBox(None)
app.title("File Inputs and Program Settings")
q = multiprocessing.Queue()
app.mainloop()
My child process (qBMPchugger):
#### _______________INITIALIZE_________________###
import os
import sys
import tkMessageBox
import Tkinter
class BadInput(Exception):
pass
def BMPcode(q, path):
# Create root for message boxes
boxRoot = Tkinter.Tk()
boxRoot.withdraw()
# Send outputs to the queue
class output:
def __init__(self, name, queue):
self.name = name
self.queue = queue
def write(self, msg):
self.queue.put(msg)
def flush(self):
sys.__stdout__.flush()
class error:
def __init__(self, name, queue):
self.name = name
self.queue = queue
def write(self, msg):
self.queue.put(msg)
def flush(self):
sys.__stderr__.flush()
sys.stdout = output(sys.stdout, q)
sys.stderr = error(sys.stderr, q)
print("Checking out the Spatial Analyst extension from GIS...")
# Check out extension and overwrite outputs
### _________________VERIFY INPUTS________________###
print("Checking validity of specified inputs...")
# Check that the provided file paths are valid
inputs = path
for i in inputs:
if os.path.exists(i):
pass
else:
message = "\nInvalid file path: {}\nCorrect the path name and try again.\n"
tkMessageBox.showerror("Invalid Path", message.format(i))
print message.format(i)
raise BadInput
print("Success!")
It's the part under # Send outputs to the queue (starting with the output class and ending with sys.stderr = error(sys.stderr, q)) that is causing my program to freeze. Why is that holding up my main thread when the child process finishes executing? EDIT: I think the freezing is being caused by the queue remaining open when the child process closes... or something. It's not the particular snippet of code like I thought it was. It happens even when I change the print statements to q.put("text") in either the parent or the child.
What is a better way to send the output to the queue? If you link me to a topic that answers my question, PLEASE show me how to implement it within my code. I have not been successful with anything I've found so far and chances are that I've already tried that particular solution and failed.

Use a manager list or dictionary to communicate between processes https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes . You can have a process update the dictionary and send it to the GUI/some code outside the processes, and vice versa. The following is a simple, and a little sloppy, example of doing it both ways.
import time
from multiprocessing import Process, Manager
def test_f(test_d):
""" frist process to run
exit this process when dictionary's 'QUIT' == True
"""
test_d['2'] = 2 ## add as a test
while not test_d["QUIT"]:
print "P1 test_f", test_d["QUIT"]
test_d["ctr"] += 1
time.sleep(1.0)
def test_f2(test_d):
""" second process to run. Runs until the for loop exits
"""
for j in range(0, 10):
## print to show that changes made anywhere
## to the dictionary are seen by this process
print " P2", j, test_d
time.sleep(0.5)
print "second process finished"
if __name__ == '__main__':
##--- create a dictionary via Manager
manager = Manager()
test_d = manager.dict()
test_d["ctr"] = 0
test_d["QUIT"] = False
##--- start first process and send dictionary
p = Process(target=test_f, args=(test_d,))
p.start()
##--- start second process
p2 = Process(target=test_f2, args=(test_d,))
p2.start()
##--- sleep 2 seconds and then change dictionary
## to exit first process
time.sleep(2.0)
print "\nterminate first process"
test_d["QUIT"] = True
print "test_d changed"
print "dictionary updated by processes", test_d
##--- may not be necessary, but I always terminate to be sure
time.sleep(5.0)
p.terminate()
p2.terminate()

For my particular problem, the main thread was trying to read from the queue when the queue was empty and not having anything else put into it. I don't know the exact details as to why the main loop got hung up on that thread (self._update in my code) but changing _update to the following stopped making the GUI non-responsive when the child finished:
def _update(self):
if q.empty():
pass
else:
msg = q.get()
self.write(msg)
self.update_idletasks()

pygtk: asynchronous output implemented with io_add_watch blocks when printing large output dataset

I'm writing a GTK+ GUI program with a python command line emulator. My python command line is implemented as a gtk.TextView, which can be used to output results of prints (and to read commands from TextView and exec them, but I don't post the input part here as it has nothing to do with the question).
I use the following technique to tee the stdout stream between the real terminal and my python command line:
r_out, w_out = os.pipe() # create a pipe, cause sys.stdout is unreadable, thus we can't poll it for available output
w_out_stream = os.fdopen(w_out, "w", 0) # open write-end of pipe with 0-length buffer
sys.stdout = w_out_stream # now prints and sys.stdout.writes would go to the pipe
watch = gobject.io_add_watch(r_out, gobject.IO_IN, stdout_callback) # poll the read-end of pipe for data and call stdout_callback
def stdout_callback(stream, condition):
data = os.read(stream, 1) # read the pipe byte-by-byte
iter = textbuf.get_end_iter() # textbuf is buffer of TextView, the GUI widget, I use as python command line
textbuf.insert(iter, data) # output stdout to GUI
sys.__stdout__.write(data) # output stdout to "real" terminal stdout
gtk.main()
This works pretty nice with small output. But unfortunately, when the output becomes relatively large (like thousands of bytes), my application hangs and doesn't display any output.
However, if I send a SIGINT, my output appears both in GUI and in real terminal. Obviously, I want it there without SIGINT. Any ideas, what causes such a block?

The external process is blocking the UI while using os.read, you should spawn the process using glib.spawn_async and use a IOChannel to read the input something like:
from gi.repository import Gtk, GLib
class MySpawned(Gtk.Window):
def __init__(self):
Gtk.Window.__init__(self)
vb = Gtk.VBox(False, 5)
self.tw = Gtk.TextView()
bt = Gtk.Button('Run')
bt.connect('clicked', self.process)
vb.pack_start(self.tw, True, True, 0)
vb.pack_start(bt, False, False, 0)
self.add(vb)
self.set_size_request(200, 300)
self.connect('delete-event', Gtk.main_quit)
self.show_all()
def run(self):
Gtk.main()
def process(self, widget, data=None):
params = ['python', '-h']
def write_to_textview(io, condition):
print condition
if condition is GLib.IO_IN:
line = io.readline()
self.tw.props.buffer.insert_at_cursor(line)
return True
elif condition is GLib.IO_HUP|GLib.IO_IN:
GLib.source_remove(self.source_id)
return False
self.source_id = None
pid, stdin, stdout, stderr = GLib.spawn_async(params,
flags=GLib.SpawnFlags.SEARCH_PATH,
standard_output=True)
io = GLib.IOChannel(stdout)
self.source_id = io.add_watch(GLib.IO_IN|GLib.IO_HUP,
write_to_textview,
priority=GLib.PRIORITY_HIGH)
if __name__ == '__main__':
s = MySpawned()
s.run()
There are like a gazillion threads out there telling you that you could use threads or other stuff, please don't do that, the above example will not block you UI even on a long process and the output will be printed on the textview, works on windows too (but it will open an ugly console window until a bug in GLib will get fixed).

Changing an appindicator's icon ocurrs after other process has finished

I'm writing an appindicator for Grive, a daemon for syncing Google Drive files. Since I have little programming experience, I decided to write a Python script that calls Grive as a subprocess instead of integrating it in its C++ source code.
I've adapted Stefaan Lippens' code for asynchronously reading subprocess pipes to both show a notification and change the indicator's icon when something important happens (e.g. a new file is added, or a network error). Notifications work well; however, the indicator's icon changes only when the whole process has finished, which is useless because I need to change it many times after it finishes.
Here is the code I'm using:
async.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import subprocess
import time
import threading
import Queue
class AsynchronousFileReader(threading.Thread):
'''
Helper class to implement asynchronous reading of a file
in a separate thread. Pushes read lines on a queue to
be consumed in another thread.
'''
def __init__(self, fd, queue):
assert isinstance(queue, Queue.Queue)
assert callable(fd.readline)
threading.Thread.__init__(self)
self._fd = fd
self._queue = queue
def run(self):
'''The body of the tread: read lines and put them on the queue.'''
for line in iter(self._fd.readline, ''):
self._queue.put(line)
def eof(self):
'''Check whether there is no more content to expect.'''
return not self.is_alive() and self._queue.empty()
def run(command, show):
'''
Main function to consume the output of a command.
command = The command to be run
show = Function that will process output
'''
# Launch the command as subprocess.
process = subprocess.Popen(command, shell=True, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
# Launch the asynchronous readers of the process' stdout and stderr.
stdout_queue = Queue.Queue()
stdout_reader = AsynchronousFileReader(process.stdout, stdout_queue)
stdout_reader.start()
stderr_queue = Queue.Queue()
stderr_reader = AsynchronousFileReader(process.stderr, stderr_queue)
stderr_reader.start()
# Check the queues if we received some output (until there is nothing more to get).
while not stdout_reader.eof() or not stderr_reader.eof():
# Show what we received from standard output.
while not stdout_queue.empty():
line = stdout_queue.get()
show(line)
# Show what we received from standard error.
while not stderr_queue.empty():
line = stderr_queue.get()
show(line)
# Sleep a bit before asking the readers again.
time.sleep(.1)
# Let's be tidy and join the threads we've started.
stdout_reader.join()
stderr_reader.join()
# Close subprocess' file descriptors.
process.stdout.close()
process.stderr.close()
return True
grive.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import subprocess
import time
import async
from gi.repository import Gtk
from gi.repository import Gio
from gi.repository import GObject
from gi.repository import Notify
from gi.repository import AppIndicator3 as AppIndicator
GRIVE_FOLDER = "/home/ilhuitemoc/Publike/Google Drive"
def show(line):
line = line.replace("\n", "")
print line
if line.startswith("[gr::expt::MsgTag*] = "):
line = line.replace("[gr::expt::MsgTag*] = ","",1)
n = Notify.Notification.new("Error", line, icon)
n.show()
indicator.set_icon("ubuntuone-client-offline")
if line.startswith("sync "):
line = line.replace("sync ","",1)
line = line.replace('"','<b>',1)
line = line.replace('"','</b>',1)
n = Notify.Notification.new("Sync in progress", line, icon)
n.show()
indicator.set_icon("ubuntuone-client-updating")
if "Finished!" in line:
#n = Notify.Notification.new(line, line, icon)
#n.show()
indicator.set_icon("ubuntuone-client-idle")
def openinfolder(obj):
subprocess.call(["xdg-open",GRIVE_FOLDER])
def openinbrowser(obj):
subprocess.call(["xdg-open","http://drive.google.com/"])
if __name__ == '__main__':
subprocess.call(["killall","pantheon-notify"])
time.sleep(1)
indicator = AppIndicator.Indicator.new('grive', 'ubuntuone-client-offline', AppIndicator.IndicatorCategory.APPLICATION_STATUS)
indicator.set_status(AppIndicator.IndicatorStatus.ACTIVE)
menu = Gtk.Menu()
status = Gtk.MenuItem("Connecting...") #Not finished yet
status.set_sensitive(False)
menu.append(status)
sp = Gtk.SeparatorMenuItem()
menu.append(sp)
mi = Gtk.MenuItem("Open Google Drive folder")
mi.connect('activate',openinfolder)
menu.append(mi)
mi = Gtk.MenuItem("Go to Google Drive webpage")
mi.connect('activate',openinbrowser)
menu.append(mi)
sp = Gtk.SeparatorMenuItem()
menu.append(sp)
mi = Gtk.ImageMenuItem("Quit")
img = Gtk.Image.new_from_stock(Gtk.STOCK_QUIT, Gtk.IconSize.MENU)
mi.set_image(img)
mi.connect('activate',Gtk.main_quit)
menu.append(mi)
menu.show_all()
indicator.set_menu(menu)
Notify.init('grive')
icon = 'google-drive'
#async.run('cd "%s" && grive' % GRIVE_FOLDER, show)
GObject.timeout_add(5*60000, async.run, 'cd "%s" && grive' % GRIVE_FOLDER, show)
#GObject.timeout_add(5000, async.run, "test.sh", show)
Gtk.main()
I think I'm doing something wrong, but it's not obvious to me. Is it right to modify the indicator using a subprocess? Or is any other way I can correctly do this?

Redirecting processes from multiprocessing to separate wx.TextCtrl

I have four text boxes and up to four processes that I will be starting using the multiprocessing module. I can get the processes to execute properly, but I would really like to redirect all the output of each process to a different wx.TextCtrl so I can see what is going on throughout the solution process. I have done this successfully with a single thread and sys.stdout redirection as in
http://www.velocityreviews.com/forums/t515815-wxpython-redirect-the-stdout-to-a-textctrl.html
but a similar idea doesn't work with processes. Can someone hack me a simple solution for this problem? I can't imagine I am the only person to have ever run into this problem.

As I understand it, in wxPython land you normally want to run processes from a thread. So you communicate from the processes to the thread and from the thread back to wxPython. I would use some kind of naming scheme to associate each potential process with a text control (maybe 1-4?) and pass that back to the thread which will use wx.CallAfter or wx.PostEvent to tell wx to update.
You might be able to use a simple Python socket server to accomplish this too. Post a message to the server with a header that says which text control it belongs to. In the wx part you could have a wx.Timer check the socket server for new messages and update as appropriate.

For those that are interested, here is a working code snippet that does what my original question asked. It works a treat.
This code creates a thread that wraps a process. Within the run() function of the process the stderr and stdout are redirected to a pipe using a redirection class. Pipes can be pickled which is a pre-requisite to using them within the run() function of the process.
The Thread then just sits and pulls output from the pipe as long as there is output waiting. The text pulled from the pipe is written to a wx.TextCtrl using the wx.CallAfter function. Note that this is a non-blocking call, and in fact all the code here is non-blocking which makes for a responsive GUI. Note the flush() function in the redirection class to redirect stderr as well.
Note: One thing to be cautious of is that if you are trying to read and write with the pipe at too high a throughput, the GUI will lock up. But as long as you have reasonably slow output, there is no problem
import wx
import sys
import time
from multiprocessing import Pipe, Process
from threading import Thread
class RedirectText2Pipe(object):
def __init__(self, pipe_inlet):
self.pipe_inlet = pipe_inlet
def write(self, string):
self.pipe_inlet.send(string)
def flush(self):
return None
class Run1(Process):
def __init__(self, pipe_inlet):
Process.__init__(self)
self.pipe_std = pipe_inlet
def run(self):
redir = RedirectText2Pipe(self.pipe_std)
sys.stdout = redir
sys.stderr = redir
for i in range(100):
time.sleep(0.01)
print i,'Hi'
class RedirectedWorkerThread(Thread):
"""Worker Thread Class."""
def __init__(self, stdout_target):
"""Init Worker Thread Class."""
Thread.__init__(self)
self.stdout_target_ = stdout_target
def run(self):
"""
In this function, actually run the process and pull any output from the
pipes while the process runs
"""
pipe_outlet, pipe_inlet = Pipe(duplex = False)
p = Run1(pipe_inlet)
p.daemon = True
p.start()
while p.is_alive():
#Collect all display output from process
while pipe_outlet.poll():
wx.CallAfter(self.stdout_target_.WriteText, pipe_outlet.recv())
class MainFrame(wx.Frame):
def __init__(self):
wx.Frame.__init__(self,None)
self.txt1 = wx.TextCtrl(self, style = wx.TE_MULTILINE|wx.TE_READONLY)
self.txt2 = wx.TextCtrl(self, style = wx.TE_MULTILINE|wx.TE_READONLY)
self.txt3 = wx.TextCtrl(self, style = wx.TE_MULTILINE|wx.TE_READONLY)
self.btn = wx.Button(self, label='Run')
self.btn.Bind(wx.EVT_BUTTON, self.OnStart)
sizer = wx.BoxSizer(wx.VERTICAL)
sizer.AddMany([(self.txt1,1,wx.EXPAND),(self.txt2,1,wx.EXPAND),(self.txt3,1,wx.EXPAND),self.btn])
self.SetSizer(sizer)
def OnStart(self, event):
t1 = RedirectedWorkerThread(self.txt1)
t1.daemon = True
t1.start()
t2 = RedirectedWorkerThread(self.txt2)
t2.daemon = True
t2.start()
t3 = RedirectedWorkerThread(self.txt3)
t3.daemon = True
t3.start()
if __name__ == '__main__':
app = wx.App(False)
frame = MainFrame()
frame.Show(True)
app.MainLoop()