In my project on windows, I would like to start the mirroring of two directories.
I know that I can use python watchdog to do that, but I though that using robocopy would be easier and faster.
To simplify the situation, let's assume I have a GUI with two buttons: start and stop mirroring.
Here below is a snippet with the relevant code:
class MirrorDemon(Thread):
def __init__(self, src, dest) :
self.threading_flag = Event()
self.src = src
self.dest = dest
self.opt = ' /MIR /MON:1 /MOT:1'
self.mirror = None
Thread.__init__(self)
def run(self):
command = 'robocopy {} {} {}'.format(str(self.src),str(self.dest), self.opt)
self.p = subprocess.Popen(command.split(), shell=True)
print(command)
print('start robocopy with PID {}'.format(self.p.pid))
class Window(QMainWindow, Ui_MainWindow):
def __init__(self, parent=None):
super().__init__(parent)
self.setupUi(self)
def stop_demon(self):
self.mirror.threading_flag.set()
self.mirror.p.kill()
self.mirror.join()
print('stop demon')
def start_demon(self):
self.mirror = MirrorDemon(Path('./src'), Path('./dest'))
self.mirror.setDaemon(True)
self.mirror.start()
print('start demon')
if __name__ == "__main__":
app = QApplication(sys.argv)
win = Window()
win.show()
sys.exit(app.exec())
When you click on the start button, you get a PID print out on the console and if I check this PID in the tasklist it corresponds to 'cmd.exe' process and the robocopy starts its job.
When you click on stop, the cmd.exe process corresponding to the PID disappers, but the background robocopy continues!!
I have tried several variations, but no luck.
Do you have some advises? Do you know if somebody has found a solution? Or maybe implemented a mirroring watchdog?
thanks
Update
Following the the suggestion of #Ulrich, setting shell=False is actually doing the trick and killing the robocopy process.
Thanks!
By changing this:
self.p = subprocess.Popen(command.split(), shell=True)
To this:
self.p = subprocess.Popen(command.split(), shell=False)
... you're ensuring that the process will be started directly from the current process, without starting a new shell process to start it in.
The PID you were getting back was for the shell process, and you can kill the shell without killing processes launched from that shell. By not starting it in a new shell, the PID you're getting back is the PID of the actual process and you'll be able to kill it as expected.
As the documentation states: "The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable."
Related
lets see if I can make this clear... I'm a total Python beginner so bear with me, this is my first python program (though I'm familiar with basic scripting in a few other languages). I've been searching around for hours and I'm sure the answer to this is fairly simple but I have yet to get it to work properly.
I'm writing a code that should launch multiple commandline processes, and when each one finishes I want to update a cell in a QTableWidget. The table has a row for each process to run, and each row has a cell for the "status" of the process.
I can run this no problem if I just do a for loop, spawning one process per row using subprocess.call() however this is too linear and I would like to fire them all off at the same time and not hang the program for each loop cycle. I've been digging through the subprocess documentation and am having a really hard time with it. I understand that I need to use subprocess.Popen (which will prevent my program from hanging while the process runs, and thus I can spawn multiple instances). Where I run into trouble is getting the exit code back so that I can update my table, without hanging the program - for instance using subprocess.wait() followed by a subprocess.returncode still just sticks until the process completes. I need a sort of "when process completes, check the exit code and run a function that updates the QTableWidget."
I did find these two posts that seemed to get me going in the right direction, but didn't quite get me there:
Understanding Popen.communicate
How to get exit code when using Python subprocess communicate method?
Hopefully that made sense. Here's a simplified version of my code, I realize it is half-baked and half-broken but I've been screwing around with it for over an hour and I've lost track of a few things...
import os, subprocess
ae_app = 'afterfx'
ae_path = os.path.join('C:/Program Files/Adobe/Adobe After Effects CC 2015/Support Files', ae_app + ".exe")
filename = "E:/Programming/Python/Archive tool/talk.jsx"
commandLine = 'afterfx -noui -r ' + filename
processList = [commandLine]
processes = []
for process in processList:
f = os.tmpfile()
aeProcess = subprocess.Popen(process, executable=ae_path, stdout=f)
processes.append((aeProcess, f))
for aeProcess, f in processes:
# this is where I need serious help...
aeProcess.wait()
print "the line is:"
print aeProcess.returncode
Spencer
You mentioned PyQt, so you can use PyQt's QProcess class.
def start_processes(self, process_list):
for cmd, args in process_list:
proc = QProcess(self)
proc.finished.connect(self.process_finished)
proc.start(cmd, args)
def process_finished(self, code, status):
# Do something
UPDATE: Added fully working example. Works properly for both PyQt4 and PyQt5 (to switch just comment line 3 and uncomment line 4)
sleeper.py
import sys
from time import sleep
from datetime import datetime as dt
if __name__ == '__main__':
x = int(sys.argv[1])
started = dt.now().time()
sleep(x)
ended = dt.now().time()
print('Slept for: {}, started: {}, ended: {}'.format(x, started, ended))
sys.exit(0)
main.py
import sys
from PyQt5 import QtCore, QtWidgets
# from PyQt4 import QtCore, QtGui as QtWidgets
class App(QtWidgets.QMainWindow):
cmd = r'python.exe C:\_work\test\sleeper.py {}'
def __init__(self):
super(App, self).__init__()
self.setGeometry(200, 200, 500, 300)
self.button = QtWidgets.QPushButton('Start processes', self)
self.button.move(20, 20)
self.editor = QtWidgets.QTextEdit(self)
self.editor.setGeometry(20, 60, 460, 200)
self.button.clicked.connect(self.start_proc)
def start_proc(self):
for x in range(5):
proc = QtCore.QProcess(self)
proc.finished.connect(self.finished)
proc.start(self.cmd.format(x))
def finished(self, code, status):
self.editor.append(str(self.sender().readAllStandardOutput()))
if __name__ == '__main__':
app = QtWidgets.QApplication(sys.argv)
gui = App()
gui.show()
app.exec_()
I am trying to start the Arduino IDE in command line mode and get the standard output back into my PyQt application like this:
def recvConsoleOutput(self, output):
print(output)
def upload(self):
[...]
cmd = [ARDUINO_PATH,
'--verbose',
'--upload', os.path.join(os.getcwd(), 'sketch', 'sketch.ino'),
'--port', self.serialPort.currentText(),
'--board', 'arduino:avr:leonardo']
cmd = ' '.join(cmd)
proc = AsyncProcess(cmd, self.recvConsoleOutput, parent=self)
proc.start()
It is called within a QMainWindow function and works flawlessly (Arduino IDE uploads the sketch!) except the fact, that it does not output any messages like it does when called by the regular windows command prompt.
The AsyncProcess class looks like this:
from PyQt5.QtCore import QObject
from PyQt5.Qt import QProcess
class AsyncProcess(QObject):
def __init__(self, path, slot, parent=None):
super().__init__(parent)
self.path = path
self.proc = None
self.slot = slot
def start(self):
self.proc = QProcess(self)
self.proc.readyReadStandardOutput.connect(self.readStandardOutput)
self.proc.readyReadStandardError.connect(self.readStandardError)
self.proc.start(self.path)
def readStandardOutput(self):
procOutput = self.proc.readAllStandardOutput()
self.slot(procOutput)
def readStandardError(self):
procOutput = self.proc.readAllStandardError()
self.slot(procOutput)
It works for some reason if I start "ping 127.0.0.1" instead of Arduino.
Any ideas?
Okay, mea culpa. At some point the developers of the IDE decided to output the information I needed only in the designated debugging version... I was actually calling the wrong executable...
My problem is with os.system. Until it finishes formatting, GUI freezes, and I can't fix it.
class ImageDialog(QtGui.QMainWindow):
def __init__(self):
QtGui.QDialog.__init__(self)
self.ui = uic.loadUi("Recursos/main.ui",self)
self.connect(self.ui.proteger_Button, QtCore.SIGNAL("clicked()"),self,
QtCore.SLOT("protec()"))
#QtCore.pyqtSlot()
def protec(self):
self.USB = "G:"
comando = "format %s /fs:ntfs /q /v:test /y" %(self.USB)
os.system(comando)`
if __name__ == '__main__':
app = QtGui.QApplication(sys.argv)
window = ImageDialog()
window.show()
sys.exit(app.exec_())
The simplest fix in your case is to add:
from threading import Thread
to your imports list, and then:
#QtCore.pyqtSlot()
def protec(self):
self.USB = "G:"
comando = "format %s /fs:ntfs /q /v:test /y" %(self.USB)
t = Thread(target = lambda: os.system(comando))
t.start()
This way the GUI thread will not get stuck waiting for the external process to finish.
You could probably remember the t's value and manage it in some way, so the number of threads running won't increase without limit, in case the external commands will hang, or run for a long time.
Note that it is not a good way to handle external processes. subprocess module is recommended for this.
I need to stop the service(runs at the background in another thread) that I issued through Popen in python after I got the result, but the following approach failed(just use ping for the sake of explanation):
class sample(threading.Thread):
def __init__(self, command, queue):
threading.Thread.__init__(self)
self.command = command;
self.queue = queue
def run(self):
result = Popen(self.command, shell=True, stdout=PIPE, stderr=STDOUT)
while True:
output = result.stdout.readline()
if not self.queue.empty():
result.kill()
break
if output != "":
print output
else:
break
def main():
q = Queue()
command = sample("ping 127.0.0.1", q)
command.start()
time.sleep(10)
q.put("stop!")
command.join()
if __name__ == "__main__":
main()
After running above program, when I pgrep for ping, it's still there. How can I kill the subprocess opened by Popen? Thanks.
PS: I also tried result.terminate(), but doesn't really solve the problem either.
You don't really need to run a subprocess from a thread. Try running the subprocess without a thread. Also, you specified shell=True, so it's running the command in a shell. So there are two new processes, the shell and the command. You can also remove the shell by making shell=False.
I want to create a popup window using wxPython that acts like a bash shell. I don't want a terminal emulator, I don't need job control, I just want a REPL (Read, Eval, Print Loop) based on a bash process.
Is there an easy way to do that with wxPython? I know the basic concept from my days as a tcl/tk programmer but my wxPython fu is weak and I don't want to have to reinvent the wheel if I don't have to. I've read a little about py.shell. Shell but that looks like it creates a python shell and I want one to run bash commands instead.
ok here is another try, which reads all output and errors too, in a separate thread and communicates via Queue.
I know it is not perfect(e.g. command with delayed output will not work and there output will get into next commnd for example tryr sleep 1; date) and replicating whole bash not trivial but for few commands i tested it seems to work fine
Regarding API of wx.py.shell I just implemented those method which Shell class was calling for Interpreter, if you go thru source code of Shell you will understand.
basically
push is where user entered command is sent to interpreter
getAutoCompleteKeys returns keys
which user can user for
auto completing commands e.g. tab key
getAutoCompleteList return list of
command matching given text
getCallTip "Display argument spec and
docstring in a popup window. so for
bash we may show man page :)
here is the source code
import threading
import Queue
import time
import wx
import wx.py
from subprocess import Popen, PIPE
class BashProcessThread(threading.Thread):
def __init__(self, readlineFunc):
threading.Thread.__init__(self)
self.readlineFunc = readlineFunc
self.outputQueue = Queue.Queue()
self.setDaemon(True)
def run(self):
while True:
line = self.readlineFunc()
self.outputQueue.put(line)
def getOutput(self):
""" called from other thread """
lines = []
while True:
try:
line = self.outputQueue.get_nowait()
lines.append(line)
except Queue.Empty:
break
return ''.join(lines)
class MyInterpretor(object):
def __init__(self, locals, rawin, stdin, stdout, stderr):
self.introText = "Welcome to stackoverflow bash shell"
self.locals = locals
self.revision = 1.0
self.rawin = rawin
self.stdin = stdin
self.stdout = stdout
self.stderr = stderr
self.more = False
# bash process
self.bp = Popen('bash', shell=False, stdout=PIPE, stdin=PIPE, stderr=PIPE)
# start output grab thread
self.outputThread = BashProcessThread(self.bp.stdout.readline)
self.outputThread.start()
# start err grab thread
self.errorThread = BashProcessThread(self.bp.stderr.readline)
self.errorThread.start()
def getAutoCompleteKeys(self):
return [ord('\t')]
def getAutoCompleteList(self, *args, **kwargs):
return []
def getCallTip(self, command):
return ""
def push(self, command):
command = command.strip()
if not command: return
self.bp.stdin.write(command+"\n")
# wait a bit
time.sleep(.1)
# print output
self.stdout.write(self.outputThread.getOutput())
# print error
self.stderr.write(self.errorThread.getOutput())
app = wx.PySimpleApp()
frame = wx.py.shell.ShellFrame(InterpClass=MyInterpretor)
frame.Show()
app.SetTopWindow(frame)
app.MainLoop()
I found the solution to my problem. It is funny how it never turned up in Google searches before now. It's not production-ready code, but ultimately, I was looking for a way to run a bash shell in a wxPython window.
http://sivachandran.blogspot.com/2008/04/termemulator-10-released.html on webarchive
https://sourceforge.net/projects/termemulator/files/TermEmulator/1.0/
i searched but there doesn't seem to be any exiting bash shell for wxPython
though wx.py module has Shell module which is for python interpretor
good thing is you can pass your own interpretor to it,so I have come with very simple bash interpreter.
example currently reads only one line from bash stdout, otherwise it will get stuck,
in real code you must read output in thread or use select
import wx
import wx.py
from subprocess import Popen, PIPE
class MyInterpretor(object):
def __init__(self, locals, rawin, stdin, stdout, stderr):
self.introText = "Welcome to stackoverflow bash shell"
self.locals = locals
self.revision = 1.0
self.rawin = rawin
self.stdin = stdin
self.stdout = stdout
self.stderr = stderr
#
self.more = False
# bash process
self.bp = Popen('bash', shell=False, stdout=PIPE, stdin=PIPE, stderr=PIPE)
def getAutoCompleteKeys(self):
return [ord('\t')]
def getAutoCompleteList(self, *args, **kwargs):
return []
def getCallTip(self, command):
return ""
def push(self, command):
command = command.strip()
if not command: return
self.bp.stdin.write(command+"\n")
self.stdout.write(self.bp.stdout.readline())
app = wx.PySimpleApp()
frame = wx.py.shell.ShellFrame(InterpClass=MyInterpretor)
frame.Show()
app.SetTopWindow(frame)
app.MainLoop()
Going to see what i can come up with.
But if you change your mind and decide to use pygtk instead, here it is:
enjoy!!
EDIT
I started making a poor man's version of a terminal using the text control widget.
I stopped because there are flaws that can't be fixed, such as when you use the sudo command.
import wx
import subprocess
class MyFrame(wx.Frame):
def __init__(self, *args, **kwds):
# begin wxGlade: MyFrame.__init__
kwds["style"] = wx.DEFAULT_FRAME_STYLE
wx.Frame.__init__(self, *args, **kwds)
self.prompt = "user#stackOvervlow:~ "
self.textctrl = wx.TextCtrl(self, -1, '', style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
self.default_txt = self.textctrl.GetDefaultStyle()
self.textctrl.AppendText(self.prompt)
self.__set_properties()
self.__do_layout()
self.__bind_events()
def __bind_events(self):
self.Bind(wx.EVT_TEXT_ENTER, self.__enter)
def __enter(self, e):
self.value = (self.textctrl.GetValue())
self.eval_last_line()
e.Skip()
def __set_properties(self):
self.SetTitle("Poor Man's Terminal")
self.SetSize((800, 600))
self.textctrl.SetFocus()
def __do_layout(self):
sizer_1 = wx.BoxSizer(wx.VERTICAL)
sizer_1.Add(self.textctrl, 1, wx.EXPAND, 0)
self.SetSizer(sizer_1)
self.Layout()
def eval_last_line(self):
nl = self.textctrl.GetNumberOfLines()
ln = self.textctrl.GetLineText(nl-1)
ln = ln[len(self.prompt):]
args = ln.split(" ")
proc = subprocess.Popen(args, stdout=subprocess.PIPE)
retvalue = proc.communicate()[0]
c = wx.Colour(239, 177, 177)
tc = wx.TextAttr(c)
self.textctrl.SetDefaultStyle(tc)
self.textctrl.AppendText(retvalue)
self.textctrl.SetDefaultStyle(self.default_txt)
self.textctrl.AppendText(self.prompt)
self.textctrl.SetInsertionPoint(GetLastPosition() - 1)
if __name__ == "__main__":
app = wx.PySimpleApp(0)
wx.InitAllImageHandlers()
frame_1 = MyFrame(None, -1, "")
app.SetTopWindow(frame_1)
frame_1.Show()
app.MainLoop()
If really wanted, this could be worked upon.