We Keep Coding

Can't kill robocopy subprocess from python

In my project on windows, I would like to start the mirroring of two directories.
I know that I can use python watchdog to do that, but I though that using robocopy would be easier and faster.
To simplify the situation, let's assume I have a GUI with two buttons: start and stop mirroring.
Here below is a snippet with the relevant code:
class MirrorDemon(Thread):
def __init__(self, src, dest) :
self.threading_flag = Event()
self.src = src
self.dest = dest
self.opt = ' /MIR /MON:1 /MOT:1'
self.mirror = None
Thread.__init__(self)
def run(self):
command = 'robocopy {} {} {}'.format(str(self.src),str(self.dest), self.opt)
self.p = subprocess.Popen(command.split(), shell=True)
print(command)
print('start robocopy with PID {}'.format(self.p.pid))
class Window(QMainWindow, Ui_MainWindow):
def __init__(self, parent=None):
super().__init__(parent)
self.setupUi(self)
def stop_demon(self):
self.mirror.threading_flag.set()
self.mirror.p.kill()
self.mirror.join()
print('stop demon')
def start_demon(self):
self.mirror = MirrorDemon(Path('./src'), Path('./dest'))
self.mirror.setDaemon(True)
self.mirror.start()
print('start demon')
if __name__ == "__main__":
app = QApplication(sys.argv)
win = Window()
win.show()
sys.exit(app.exec())
When you click on the start button, you get a PID print out on the console and if I check this PID in the tasklist it corresponds to 'cmd.exe' process and the robocopy starts its job.
When you click on stop, the cmd.exe process corresponding to the PID disappers, but the background robocopy continues!!
I have tried several variations, but no luck.
Do you have some advises? Do you know if somebody has found a solution? Or maybe implemented a mirroring watchdog?
thanks
Update
Following the the suggestion of #Ulrich, setting shell=False is actually doing the trick and killing the robocopy process.
Thanks!

By changing this:
self.p = subprocess.Popen(command.split(), shell=True)
To this:
self.p = subprocess.Popen(command.split(), shell=False)
... you're ensuring that the process will be started directly from the current process, without starting a new shell process to start it in.
The PID you were getting back was for the shell process, and you can kill the shell without killing processes launched from that shell. By not starting it in a new shell, the PID you're getting back is the PID of the actual process and you'll be able to kill it as expected.
As the documentation states: "The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable."

How & where to best retrieve sudo password via a native GUI on a macOS Python-based app - (while maintaining an interactive output stream (stdout))

Ok, so the situation is this: I am building a macOS GUI App using Python and wx (wxphoenix). The user can use the GUI (say: script1) to launch a file-deletion process (contained in script2). In order to run successfully script2 needs to run with sudo rights.
script2 will itterate over a long list of files and delete them. But I need it to communicate with the GUI contained in script1 after each round so that script1 can update the progressbar.
In it's absolute most basic form my current working setup looks like this:
Script1:
import io
from threading import Thread
import subprocess
import wx
# a whole lot of wx GUI stuff
def get_password():
"""Retrieve user password via a GUI"""
# A wx solution using wx.PasswordEntryDialog()
# Store password in a variable
return variable
class run_script_with_sudo(Thread):
"""Launch a script with administrator privileges"""
def __init__(self, path_to_script, wx_pubsub_sendmessage):
"""Set variables to self"""
self.path = path_to_script
self.sender = wx_pubsub_sendmessage
self.password = get_password()
Thread.__init__(self)
self.start()
def run(self):
"""Run thread"""
prepare_script = subprocess.Popen(["echo", password], stdout=subprocess.PIPE)
prepare_script.wait()
launch_script = subprocess.Popen(['sudo', '-S', '/usr/local/bin/python3.6', '-u', self.path], stdin=prepare_script.stdout, stdout=subprocess.PIPE)
for line in io.TextIOWrapper(launch_script.stdout, encoding="utf-8"):
print("Received line: ", line.rstrip())
# Tell progressbar to add another step:
wx.CallAfter(self.sender, "update", msg="")
Script2:
import time
# This is a test setup, just a very simple loop that produces an output.
for i in range(25):
time.sleep(1)
print(i)
The above setup works in that script1 receives the output of script2 in real-time and acts on it. (So in the given example: after each second script1 adds another step to the progress bar until it reaches 25 steps).
What I want to achieve = not storing the password in a variable and using macOS it's native GUI to retrieve the password.
However when I change:
prepare_script = subprocess.Popen(["echo", password], stdout=subprocess.PIPE)
prepare_script.wait()
launch_script = subprocess.Popen(['sudo', '-S', '/usr/local/bin/python3.6', '-u', self.path], stdin=prepare_script.stdout, stdout=subprocess.PIPE)
for line in io.TextIOWrapper(launch_script.stdout, encoding="utf-8"):
print("Received line: ", line.rstrip())
# Tell progressbar to add another step:
wx.CallAfter(self.sender, "update", msg="")
Into:
command = r"""/usr/bin/osascript -e 'do shell script "/usr/local/bin/python3.6 -u """ + self.path + """ with prompt "Sart Deletion Process " with administrator privileges'"""
command_list = shlex.split(command)
launch_script = subprocess.Popen(command_list, stdout=subprocess.PIPE)
for line in io.TextIOWrapper(launch_script.stdout, encoding="utf-8"):
print("Received line: ", line.rstrip())
# Tell progressbar to add another step:
wx.CallAfter(self.sender, "update", msg="")
It stops working because osascript apparently runs in a non-interactive shell. This means script2 doesn't sent any output until it is fully finished, causing the progress bar in script1 to stall.
My question thus becomes: How can I make sure to use macOS native GUI to ask for the sudo password, thus preventing having to store it in a variable, while still maintaining the possibility to catch the stdout from the privileged script in an interactive / real-time stream.
Hope that makes sense.
Would appreciate any insights!

My question thus becomes: How can I make sure to use macOS native GUI
to ask for the sudo password, thus preventing having to store it in a
variable, while still maintaining the possibility to catch the stdout
from the privileged script in an interactive / real-time stream.
I have found a solution myself, using a named pipe (os.mkfifo()).
That way, you can have 2 python scripts communicate with each other while 1 of them is launched with privileged rights via osascript (meaning: you get a native GUI window that asks for the users sudo password).
Working solution:
mainscript.py
import os
from pathlib import Path
import shlex
import subprocess
import sys
from threading import Thread
import time
class LaunchDeletionProcess(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
launch_command = r"""/usr/bin/osascript -e 'do shell script "/usr/local/bin/python3.6 -u /path/to/priviliged_script.py" with prompt "Sart Deletion Process " with administrator privileges'"""
split_command = shlex.split(launch_command)
print("Thread 1 started")
testprogram = subprocess.Popen(split_command)
testprogram.wait()
print("Thread1 Finished")
class ReadStatus(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
while not os.path.exists(os.path.expanduser("~/p1")):
time.sleep(0.1)
print("Thread 2 started")
self.wfPath = os.path.expanduser("~/p1")
rp = open(self.wfPath, 'r')
response = rp.read()
self.try_pipe(response)
def try_pipe(self, response):
rp = open(self.wfPath, 'r')
response = rp.read()
print("Receiving response: ", response)
rp.close()
if response == str(self.nr_of_steps-1):
print("Got to end")
os.remove(os.path.expanduser("~/p1"))
else:
time.sleep(1)
self.try_pipe(response)
if __name__ == "__main__":
thread1 = LaunchDeletionProcess()
thread2 = ReadStatus()
thread1.start()
thread2.start()
priviliged_script.py
import os
import time
import random
wfPath = os.path.expanduser("~/p1")
try:
os.mkfifo(wfPath)
except OSError:
print("error")
pass
result = 10
nr = 0
while nr < result:
random_nr = random.random()
wp = open(wfPath, 'w')
print("writing new number: ", random_nr)
wp.write("Number: " + str(random_nr))
wp.close()
time.sleep(1)
nr += 1
wp = open(wfPath, 'w')
wp.write("end")
wp.close()

Introduce a text in a lineEdit of PyQt from a thread

How can I introduce a text in a lineEdit from a thread that are getting the data whithout colapse the program? The important line is in the class "fil" where it shows Principal.self.aplicacio.actual_lineEdit.setText(self.temp)
# !/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import serial
import threading
from time import sleep
from PyQt4 import QtCore, QtGui
from temperaturaUI import Ui_Form
class Principal(QtGui.QWidget):
def __init__(self):
QtGui.QWidget.__init__(self)
self.aplicacio = Ui_Form()
self.aplicacio.setupUi(self)
self.aplicacio.sortir_Button.clicked.connect(exit)
self.aplicacio.connectar_Button.clicked.connect(self.connectar)
def connectar(self):
try:
arduino = serial.Serial('/dev/ttyACM0', 9600)
print "Connectat amb èxit"
temperatura = fil(0, arduino, self.aplicacio.actual_lineEdit)
temperatura.start()
except:
print "Impossible connectar a l'Arduino"
class fil(threading.Thread):
def __init__(self, temp, serie, line):
threading.Thread.__init__(self)
self.temp = temp
self.serie = serie
self.line = line
def run(self):
try:
while 1:
self.temp = self.serie.readline()
if self.temp != 0:
**Principal.self.aplicacio.actual_lineEdit.setText(self.temp)**
sleep(0.2)
except:
print "Error al llegir de l'Arduino"
def main():
app = QtGui.QApplication(sys.argv)
aplicacio = Principal()
aplicacio.show()
sys.exit(app.exec_())
if __name__ == '__main__':
main()

You can use signals. You would add a signal to the fil class that emits the new text:
class fil(threading.Thread):
update_line_edit = pyqtSignal(str)
def __init__(self, temp, serie, line):
...
def run(self):
try:
while True:
self.temp = self.serie.readline()
if not self.temp:
update_line_edit.emit(self.temp)
...
Then, simply connect that signal to a slot function in your Principal class:
class Principal(QtGui.QWidget):
def __init__(self):
...
def connectar(self):
try:
arduino = serial.Serial('/dev/ttyACM0', 9600)
print "Connectat amb èxit"
temperatura = fil(0, arduino, self.aplicacio.actual_lineEdit)
temperatura.change_line_edit.connect(self.update_line_edit)
...
def update_line_edit(self, text):
self.aplicacio.actual_lineEdit.setText(text)

There are a few ways to do this correctly.
The first is to use a QThread instead of a python thread. You can then use Qt signals to pass a message back from the fil thread to the Qt MainThread and append the message to the QLineEdit there. Another similar approach is to continue using a Python thread, but place your message in a Python Queue.Queue() object. This Queue is then read by a secondary QThread, whose sole purpose is to read messages out of the Queue and emit a signal back to the MainThread.
The common feature of these two methods is that you only access Qt GUI objects from the MainThread and use signals/slots to communicate between threads. Here are some other questions where I've answered similar questions (you should be able to adapt them to your program):
Redirecting stdout and stderr to a PyQt4 QTextEdit from a secondary thread
Syncing activity in PyQt QThreads
However, since answering those questions, my colleagues and I have created a project that helps simplify writing multi-threaded Qt applications. The project is called qtutils and is on PyPi so it can be installed with pip or easy_install (just run pip install qtutils or easy_install qtutils from a commandline/terminal window).
This library has (among others) some functions inmain and inmain_later which will run a specified method in the Qt MainThread (regardless of the thread the call is made from) synchronously or asynchronously. Documentation on how to use these methods is here. I've modified your example code to use my inmain method and put the code here: http://pastebin.com/QM1Y6zBx -- obviously you need to install qtutils for it to work!

how to implement GUI window or a frame for bash shell with wxwidget or wxpython [duplicate]

I want to create a popup window using wxPython that acts like a bash shell. I don't want a terminal emulator, I don't need job control, I just want a REPL (Read, Eval, Print Loop) based on a bash process.
Is there an easy way to do that with wxPython? I know the basic concept from my days as a tcl/tk programmer but my wxPython fu is weak and I don't want to have to reinvent the wheel if I don't have to. I've read a little about py.shell. Shell but that looks like it creates a python shell and I want one to run bash commands instead.

ok here is another try, which reads all output and errors too, in a separate thread and communicates via Queue.
I know it is not perfect(e.g. command with delayed output will not work and there output will get into next commnd for example tryr sleep 1; date) and replicating whole bash not trivial but for few commands i tested it seems to work fine
Regarding API of wx.py.shell I just implemented those method which Shell class was calling for Interpreter, if you go thru source code of Shell you will understand.
basically
push is where user entered command is sent to interpreter
getAutoCompleteKeys returns keys
which user can user for
auto completing commands e.g. tab key
getAutoCompleteList return list of
command matching given text
getCallTip "Display argument spec and
docstring in a popup window. so for
bash we may show man page :)
here is the source code
import threading
import Queue
import time
import wx
import wx.py
from subprocess import Popen, PIPE
class BashProcessThread(threading.Thread):
def __init__(self, readlineFunc):
threading.Thread.__init__(self)
self.readlineFunc = readlineFunc
self.outputQueue = Queue.Queue()
self.setDaemon(True)
def run(self):
while True:
line = self.readlineFunc()
self.outputQueue.put(line)
def getOutput(self):
""" called from other thread """
lines = []
while True:
try:
line = self.outputQueue.get_nowait()
lines.append(line)
except Queue.Empty:
break
return ''.join(lines)
class MyInterpretor(object):
def __init__(self, locals, rawin, stdin, stdout, stderr):
self.introText = "Welcome to stackoverflow bash shell"
self.locals = locals
self.revision = 1.0
self.rawin = rawin
self.stdin = stdin
self.stdout = stdout
self.stderr = stderr
self.more = False
# bash process
self.bp = Popen('bash', shell=False, stdout=PIPE, stdin=PIPE, stderr=PIPE)
# start output grab thread
self.outputThread = BashProcessThread(self.bp.stdout.readline)
self.outputThread.start()
# start err grab thread
self.errorThread = BashProcessThread(self.bp.stderr.readline)
self.errorThread.start()
def getAutoCompleteKeys(self):
return [ord('\t')]
def getAutoCompleteList(self, *args, **kwargs):
return []
def getCallTip(self, command):
return ""
def push(self, command):
command = command.strip()
if not command: return
self.bp.stdin.write(command+"\n")
# wait a bit
time.sleep(.1)
# print output
self.stdout.write(self.outputThread.getOutput())
# print error
self.stderr.write(self.errorThread.getOutput())
app = wx.PySimpleApp()
frame = wx.py.shell.ShellFrame(InterpClass=MyInterpretor)
frame.Show()
app.SetTopWindow(frame)
app.MainLoop()

I found the solution to my problem. It is funny how it never turned up in Google searches before now. It's not production-ready code, but ultimately, I was looking for a way to run a bash shell in a wxPython window.
http://sivachandran.blogspot.com/2008/04/termemulator-10-released.html on webarchive
https://sourceforge.net/projects/termemulator/files/TermEmulator/1.0/

i searched but there doesn't seem to be any exiting bash shell for wxPython
though wx.py module has Shell module which is for python interpretor
good thing is you can pass your own interpretor to it,so I have come with very simple bash interpreter.
example currently reads only one line from bash stdout, otherwise it will get stuck,
in real code you must read output in thread or use select
import wx
import wx.py
from subprocess import Popen, PIPE
class MyInterpretor(object):
def __init__(self, locals, rawin, stdin, stdout, stderr):
self.introText = "Welcome to stackoverflow bash shell"
self.locals = locals
self.revision = 1.0
self.rawin = rawin
self.stdin = stdin
self.stdout = stdout
self.stderr = stderr
#
self.more = False
# bash process
self.bp = Popen('bash', shell=False, stdout=PIPE, stdin=PIPE, stderr=PIPE)
def getAutoCompleteKeys(self):
return [ord('\t')]
def getAutoCompleteList(self, *args, **kwargs):
return []
def getCallTip(self, command):
return ""
def push(self, command):
command = command.strip()
if not command: return
self.bp.stdin.write(command+"\n")
self.stdout.write(self.bp.stdout.readline())
app = wx.PySimpleApp()
frame = wx.py.shell.ShellFrame(InterpClass=MyInterpretor)
frame.Show()
app.SetTopWindow(frame)
app.MainLoop()

Going to see what i can come up with.
But if you change your mind and decide to use pygtk instead, here it is:
enjoy!!
EDIT
I started making a poor man's version of a terminal using the text control widget.
I stopped because there are flaws that can't be fixed, such as when you use the sudo command.
import wx
import subprocess
class MyFrame(wx.Frame):
def __init__(self, *args, **kwds):
# begin wxGlade: MyFrame.__init__
kwds["style"] = wx.DEFAULT_FRAME_STYLE
wx.Frame.__init__(self, *args, **kwds)
self.prompt = "user#stackOvervlow:~ "
self.textctrl = wx.TextCtrl(self, -1, '', style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
self.default_txt = self.textctrl.GetDefaultStyle()
self.textctrl.AppendText(self.prompt)
self.__set_properties()
self.__do_layout()
self.__bind_events()
def __bind_events(self):
self.Bind(wx.EVT_TEXT_ENTER, self.__enter)
def __enter(self, e):
self.value = (self.textctrl.GetValue())
self.eval_last_line()
e.Skip()
def __set_properties(self):
self.SetTitle("Poor Man's Terminal")
self.SetSize((800, 600))
self.textctrl.SetFocus()
def __do_layout(self):
sizer_1 = wx.BoxSizer(wx.VERTICAL)
sizer_1.Add(self.textctrl, 1, wx.EXPAND, 0)
self.SetSizer(sizer_1)
self.Layout()
def eval_last_line(self):
nl = self.textctrl.GetNumberOfLines()
ln = self.textctrl.GetLineText(nl-1)
ln = ln[len(self.prompt):]
args = ln.split(" ")
proc = subprocess.Popen(args, stdout=subprocess.PIPE)
retvalue = proc.communicate()[0]
c = wx.Colour(239, 177, 177)
tc = wx.TextAttr(c)
self.textctrl.SetDefaultStyle(tc)
self.textctrl.AppendText(retvalue)
self.textctrl.SetDefaultStyle(self.default_txt)
self.textctrl.AppendText(self.prompt)
self.textctrl.SetInsertionPoint(GetLastPosition() - 1)
if __name__ == "__main__":
app = wx.PySimpleApp(0)
wx.InitAllImageHandlers()
frame_1 = MyFrame(None, -1, "")
app.SetTopWindow(frame_1)
frame_1.Show()
app.MainLoop()
If really wanted, this could be worked upon.

Open a program with python minimized or hidden

What I'm trying to do is to write a script which would open an application only in process list. Meaning it would be "hidden". I don't even know if its possible in python.
If its not possible, I would settle for even a function that would allow for a program to be opened with python in a minimized state maybe something like this:
import subprocess
def startProgram():
subprocess.Hide(subprocess.Popen('C:\test.exe')) # I know this is wrong but you get the idea...
startProgram()
Someone suggested to use win32com.client but the thing is that the program that i want to launch doesn't have a COM server registered under the name.
Any ideas?

It's easy :)
Python Popen Accept STARTUPINFO Structure...
About STARTUPINFO Structure: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686331(v=vs.85).aspx
Run Hidden:
import subprocess
def startProgram():
SW_HIDE = 0
info = subprocess.STARTUPINFO()
info.dwFlags = subprocess.STARTF_USESHOWWINDOW
info.wShowWindow = SW_HIDE
subprocess.Popen(r'C:\test.exe', startupinfo=info)
startProgram()
Run Minimized:
import subprocess
def startProgram():
SW_MINIMIZE = 6
info = subprocess.STARTUPINFO()
info.dwFlags = subprocess.STARTF_USESHOWWINDOW
info.wShowWindow = SW_MINIMIZE
subprocess.Popen(r'C:\test.exe', startupinfo=info)
startProgram()

You should use win32api and hide your window e.g. using win32gui.EnumWindows you can enumerate all top windows and hide your window
Here is a small example, you may do something like this:
import subprocess
import win32gui
import time
proc = subprocess.Popen(["notepad.exe"])
# lets wait a bit to app to start
time.sleep(3)
def enumWindowFunc(hwnd, windowList):
""" win32gui.EnumWindows() callback """
text = win32gui.GetWindowText(hwnd)
className = win32gui.GetClassName(hwnd)
#print hwnd, text, className
if text.find("Notepad") >= 0:
windowList.append((hwnd, text, className))
myWindows = []
# enumerate thru all top windows and get windows which are ours
win32gui.EnumWindows(enumWindowFunc, myWindows)
# now hide my windows, we can actually check process info from GetWindowThreadProcessId
# http://msdn.microsoft.com/en-us/library/ms633522(VS.85).aspx
for hwnd, text, className in myWindows:
win32gui.ShowWindow(hwnd, False)
# as our notepad is now hidden
# you will have to kill notepad in taskmanager to get past next line
proc.wait()
print "finished."

What is the purpose?
if you want a hidden(no window) process working in background, best way would be to write a windows service and start/stop it using usual window service mechanism. Windows service can be easily written in python e.g. here is part of my own service (it will not run without some modifications)
import os
import time
import traceback
import pythoncom
import win32serviceutil
import win32service
import win32event
import servicemanager
import jagteraho
class JagteRahoService (win32serviceutil.ServiceFramework):
_svc_name_ = "JagteRaho"
_svc_display_name_ = "JagteRaho (KeepAlive) Service"
_svc_description_ = "Used for keeping important services e.g. broadband connection up"
def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.stop = False
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.log('stopping')
self.stop = True
def log(self, msg):
servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
servicemanager.PYS_SERVICE_STARTED,
(self._svc_name_,msg))
def SvcDoRun(self):
self.log('folder %s'%os.getcwd())
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.start()
def shouldStop(self):
return self.stop
def start(self):
try:
configFile = os.path.join(jagteraho.getAppFolder(), "jagteraho.cfg")
jagteraho.start_config(configFile, self.shouldStop)
except Exception,e:
self.log(" stopped due to eror %s [%s]" % (e, traceback.format_exc()))
self.ReportServiceStatus(win32service.SERVICE_STOPPED)
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(AppServerSvc)
and you can install it by
python svc_jagteraho.py--startup auto install
and run it by
python python svc_jagteraho.py start
I will be also be seen in services list e.g. services.msc will show it and you can start/stop it else you can use commandline
sc stop jagteraho

Run Hidden:
from subprocess_maximize import Popen
Popen("notepad.exe",show='hidden', priority=0)
Before the code above, use the following command:
pip install subprocess-maximize

If what is appearing is a terminal, redirect the process's stdout.