We Keep Coding

How to listen for a file change in urwid?

I want to remote control a python application which uses urwid for the user interface.
My idea was to create a file, pass it's name as command line argument to the application and whenever I write to the file the application reads from that file.
Urwid's event loop has a method watch_file(fd, callback).
This method is described as "Call callback() when fd has some data to read."
This sounds exactly like what I want to have, but it causes an infinite loop.
callback is executed as often as possible, despite the fact that the file is empty.
Even if I delete the file, callback is still called.
#!/usr/bin/env python3
import urwid
import atexit
def onkeypress(key, size=None):
if key == 'q':
raise urwid.ExitMainLoop()
text.set_text(key)
def onfilechange():
text.set_text(cmdfile.read())
# clear file so that I don't read already executed commands again
# and don't run into an infinite loop - but I am doing that anyway
with open(cmdfile.name, 'w') as f:
pass
cmdfile = open('/tmp/cmd', 'rt')
atexit.register(cmdfile.close)
text = urwid.Text("hello world")
filler = urwid.Filler(text)
loop = urwid.MainLoop(filler, unhandled_input=onkeypress)
loop.watch_file(cmdfile, onfilechange)
if __name__ == '__main__':
loop.run()
(My initial idea was to open the file only for reading instead of keeping it open all the time but fd has to be a file object, not a path.)
Urwid offers several different event loops.
By default, SelectEventLoop is used.
GLibEventLoop has the same behaviour, it runs into an infinite loop.
AsyncioEventLoop instead throws an "operation not permitted" exception.
TwistedEventLoop and TornadoEventLoop would need additional software to be installed.
I have considered using the independent watchdog library but it seems accessing the user interface from another thread would require to write a new loop, see this stack overflow question.
The answer to that question recommends polling instead which I would prefer to avoid.
If urwid specifically provides a method to watch a file I cannot believe that it does not work in any implementation.
So what am I doing wrong?
How do I react to a file change in a python/urwid application?
EDIT:
I have tried using named pipes (and removed the code to clear the file) but visually it has the same behaviour: the app does not start.
Audibly, however, there is a great difference: It does not go into the infinite loop until I write to the file.
Before I write to the file callback is not called but the app is not started either, it just does nothing.
After I write to the file, it behaves as described above for regular files.

I have found the following work around: read a named pipe in another thread, safe each line in a queue and poll in the UI thread to see if something is in the queue.
Create the named pipe with mkfifo /tmp/mypipe.
Then write to it with echo >>/tmp/mypipe "some text".
#!/usr/bin/env python3
import os
import threading
import queue
import urwid
class App:
POLL_TIME_S = .5
def __init__(self):
self.text = urwid.Text("hello world")
self.filler = urwid.Filler(self.text)
self.loop = urwid.MainLoop(self.filler, unhandled_input=self.onkeypress)
def watch_pipe(self, path):
self._cmd_pipe = path
self.queue = queue.Queue()
threading.Thread(target=self._read_pipe_thread, args=(path,)).start()
self.loop.set_alarm_in(0, self._poll_queue)
def _read_pipe_thread(self, path):
while self._cmd_pipe:
with open(path, 'rt') as pipe:
for ln in pipe:
self.queue.put(ln)
self.queue.put("!! EOF !!")
def _poll_queue(self, loop, args):
while not self.queue.empty():
ln = self.queue.get()
self.text.set_text(ln)
self.loop.set_alarm_in(self.POLL_TIME_S, self._poll_queue)
def close(self):
path = self._cmd_pipe
# stop reading
self._cmd_pipe = None
with open(path, 'wt') as pipe:
pipe.write("")
os.remove(path)
def run(self):
self.loop.run()
def onkeypress(self, key, size=None):
if key == 'q':
raise urwid.ExitMainLoop()
self.text.set_text(key)
if __name__ == '__main__':
a = App()
a.watch_pipe('/tmp/mypipe')
a.run()
a.close()

Passing Variables to a process python

Need help with how to modify/fix code to allow me to control what is occurring in a process. I have looked around and read I need to either make a global variable which the process can read or use an event function to trigger the process. Problem though is I don't know how to implement them in a class function. I thought that if I followed pyimagesearch code that it would work but it appears that it only works with the threading module and not the multiprocessing module.
import RPi.GPIO as GPIO
from RPI.GPIO import LOW,OUT,HIGH,BCM
import multiprocessing as mp
import time
class TestClass():
def __init__(self,PinOne=22,PinTwo=27):
self.PinOne = PinOne
self.PinTwo = PinTwo
self.RunningSys = True
GPIO.setmode(BCM)
GPIO.setup(PinOne,OUT)
GPIO.output(PinOne,LOW)
GPIO.setup(PinTwo,OUT)
GPIO.output(PinTwo,LOW)
def Testloop(self):
while self.RunningSys:
GPIO.output(PinOne,HIGH)
GPIO.output(PinTwo,HIGH)
time.sleep(1)
GPIO.output(PinOne,LOW)
GPIO.output(PinTwo,LOW)
GPIO.output(PinOne,LOW)
GPIO.output(PinTwo,LOW)
def StopPr(self):
self.RunningSys = False
def MProc(self):
MPGP = mp.process(target=TestClass().Testloop())
MPGP.start()
MPGP.join()
In a separate script
From testfile import TestClass
import time
TestClass().MProc()
time.sleep(4)
TestClass().StopPr()

python parallel processes return exit code

lets see if I can make this clear... I'm a total Python beginner so bear with me, this is my first python program (though I'm familiar with basic scripting in a few other languages). I've been searching around for hours and I'm sure the answer to this is fairly simple but I have yet to get it to work properly.
I'm writing a code that should launch multiple commandline processes, and when each one finishes I want to update a cell in a QTableWidget. The table has a row for each process to run, and each row has a cell for the "status" of the process.
I can run this no problem if I just do a for loop, spawning one process per row using subprocess.call() however this is too linear and I would like to fire them all off at the same time and not hang the program for each loop cycle. I've been digging through the subprocess documentation and am having a really hard time with it. I understand that I need to use subprocess.Popen (which will prevent my program from hanging while the process runs, and thus I can spawn multiple instances). Where I run into trouble is getting the exit code back so that I can update my table, without hanging the program - for instance using subprocess.wait() followed by a subprocess.returncode still just sticks until the process completes. I need a sort of "when process completes, check the exit code and run a function that updates the QTableWidget."
I did find these two posts that seemed to get me going in the right direction, but didn't quite get me there:
Understanding Popen.communicate
How to get exit code when using Python subprocess communicate method?
Hopefully that made sense. Here's a simplified version of my code, I realize it is half-baked and half-broken but I've been screwing around with it for over an hour and I've lost track of a few things...
import os, subprocess
ae_app = 'afterfx'
ae_path = os.path.join('C:/Program Files/Adobe/Adobe After Effects CC 2015/Support Files', ae_app + ".exe")
filename = "E:/Programming/Python/Archive tool/talk.jsx"
commandLine = 'afterfx -noui -r ' + filename
processList = [commandLine]
processes = []
for process in processList:
f = os.tmpfile()
aeProcess = subprocess.Popen(process, executable=ae_path, stdout=f)
processes.append((aeProcess, f))
for aeProcess, f in processes:
# this is where I need serious help...
aeProcess.wait()
print "the line is:"
print aeProcess.returncode
Spencer

You mentioned PyQt, so you can use PyQt's QProcess class.
def start_processes(self, process_list):
for cmd, args in process_list:
proc = QProcess(self)
proc.finished.connect(self.process_finished)
proc.start(cmd, args)
def process_finished(self, code, status):
# Do something
UPDATE: Added fully working example. Works properly for both PyQt4 and PyQt5 (to switch just comment line 3 and uncomment line 4)
sleeper.py
import sys
from time import sleep
from datetime import datetime as dt
if __name__ == '__main__':
x = int(sys.argv[1])
started = dt.now().time()
sleep(x)
ended = dt.now().time()
print('Slept for: {}, started: {}, ended: {}'.format(x, started, ended))
sys.exit(0)
main.py
import sys
from PyQt5 import QtCore, QtWidgets
# from PyQt4 import QtCore, QtGui as QtWidgets
class App(QtWidgets.QMainWindow):
cmd = r'python.exe C:\_work\test\sleeper.py {}'
def __init__(self):
super(App, self).__init__()
self.setGeometry(200, 200, 500, 300)
self.button = QtWidgets.QPushButton('Start processes', self)
self.button.move(20, 20)
self.editor = QtWidgets.QTextEdit(self)
self.editor.setGeometry(20, 60, 460, 200)
self.button.clicked.connect(self.start_proc)
def start_proc(self):
for x in range(5):
proc = QtCore.QProcess(self)
proc.finished.connect(self.finished)
proc.start(self.cmd.format(x))
def finished(self, code, status):
self.editor.append(str(self.sender().readAllStandardOutput()))
if __name__ == '__main__':
app = QtWidgets.QApplication(sys.argv)
gui = App()
gui.show()
app.exec_()

Python multiprocessing module do not work

i am trying to write a spider with multiprocessing module
here is my python code:
# -*- coding:utf-8 -*-
import multiprocessing
import requests
class SpiderWorker(object):
def __init__(self, q):
self._q = q
def run(self):
def _crawl_item(url):
requests.get("http://www.baidu.com")
if respon.ok:
print respon.url
while True:
rst = self._q.get()
_crawl_item(rst)
def general_worker():
q = multiprocessing.Queue()
CPU_COUNT = multiprocessing.cpu_count()
worker_processes = [
multiprocessing.Process(target=SpiderWorker(q).run)
for i in range(CPU_COUNT)
]
map( lambda process: process.start(), worker_processes )
return q, worker_processes
maybe it is my process way wrong
every time i run this code, my process tell me
<Process(Process-1, stopped[SIGSEGV])>
hope love it

The major problem here is that you don't have any information on why your processes fail. It could be gevent, but it could just as easily be something else. So learning the actual reason why your processes get terminated is the first step before doing anything else.
What you need is multiprocessing.log_to_stderr():
class SpiderWorker(object):
# ...
def run(self):
logger = multiprocessing.log_to_stderr()
logger.setLevel(multiprocessing.SUBDEBUG)
try:
# Here goes your original run() code
except Exception:
logger.exception('whoopsie')
What this code does:
Creates a special logger which will transmit it's information to the main process and dump it to stderr (console by default).
Configures this logger to report everything, including some internal multiprocessing module events (just in case as you probably don't need them).
Wraps your entire code in catch-all statement so whatever happens there cannot escape your notice.
Runs .exception() method on the logger, which not only logs the message (it's meaningless anyway as we don't know what actually happens) but most importantly logs the entire error traceback - which we actually need.

How to keep function called with python ctypes running

I'm trying to create a minimal python script that calls the winsparkle C library using ctypes. The code only works if I run it line-by-line, win_sparkle_check_update_without_ui() pops up a window to download an update as expected. But running the script normally this function just gets called and skipped instantly, it doesn't remain open, unless I add the hacky time.sleep option.
What is the proper pythonic way to run this function and leave it open until the user closes the popup?
from ctypes import CDLL
import logging, time
class UpdateAgent(object):
def __init__(self, appcast_url):
self.appcast_url = appcast_url
def check_for_update(self):
winsparkle = CDLL("WinSparkle.dll")
url = self.appcast_url
winsparkle.win_sparkle_set_appcast_url(url.encode('ascii', 'ignore'))
winsparkle.win_sparkle_set_app_details(
unicode('Company'),
unicode('myapp'),
unicode(9))
winsparkle.win_sparkle_init()
winsparkle.win_sparkle_check_update_without_ui()
#time.sleep(10)
def run(self):
try:
self.check_for_update()
except Exception as e:
logging.info('%s' % (e))