A couple of answers (first, second) have mentioned that subprocess.Popen is a non blocking call.
What can be a simple example which can validate it or can be used to explain it to a beginner.
I tried the following code. It shows that "Finish" is printed before printing output of ls -lrt but as soon as I add sleep 10 before ls -lrt in command, it waits for command to finish.
import logging
import os
import subprocess
import signal
import time
log = logging.getLogger(__name__)
class Utils(object):
#staticmethod
def run_command(cmnd, env=None, cwd=None, timeout=0):
p = subprocess.Popen(cmnd, shell=True, stdin=None, bufsize=-1, env=env,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
close_fds=True, cwd=cwd, preexec_fn=os.setsid)
#stdout_val = p.communicate()[0]
stdout_val = p.stdout.read()
return p.returncode, stdout_val.strip()
if __name__ == '__main__':
print "Start"
print "Invoke command"
status, output = Utils.run_command("ls -lrt") # line - 10
#status, output = Utils.run_command("sleep 10;ls -lrt") # line - 11
for i in xrange(10):
print "Finish"
print status
print output
EDIT 1: Replacing call p.communicate() with p.stdout.read() after suggestion.
Related
I want to utilize subprocess Popen to call strace on Linux.
I also want to catch every line of output strace gives, in realtime if possible.
I came up with the following code for that, but for some reason I can't get it working. I'll only get the output AFTER I terminate the program.
from threading import Thread
from queue import Queue, Empty
pid = 1
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(["strace", "-p", pid], stdout=subprocess.PIPE, bufsize=1)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()
try:
line = q.get_nowait()
print("Got it! "+line)
except Empty:
pass
Here is a short working example:
Please note that:
strace writes to stderr (unless -o filename is given)
all arguments must be strings (or bytes), i.e. pid must be given as "1"
line buffering works only with universal newlines
you must be root to trace process 1
import subprocess
PID = 1
p = subprocess.Popen(
["strace", "-p", str(PID)],
stdin=subprocess.DEVNULL, stderr=subprocess.PIPE,
universal_newlines=True, bufsize=1)
for line in p.stderr:
line = line.rstrip()
print(line)
The script below (a python process is called, which calls a waitfor cmd with a timeout of 4 seconds) is supposed to end after 4 seconds.
But instead proc.communicate stops after the 20 seconds timeout.
Why???
if __name__ == "__main__":
proc_str = ["C:\\Program Files (x86)\\Anaconda3\\Python.exe",
"-c", "import
subprocess;subprocess.run('cmd /S /C waitfor g /t 200', shell=False, timeout=4)"]
proc = subprocess.Popen(proc_str,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=False,
universal_newlines=True)
(proc_out, proc_err) = proc.communicate(timeout=20)
if (proc.returncode == 0):
pass
Everything works 100% ok if I remove the stdin/stderr/stdout parameters
I want to interact with a process.
I can start the process and print out the first two lines (something like 'process successfully started').
Now I want to send a new command to the process which should return again something like 'command done' but nothing happens.
Please help me.
import subprocess
def PrintAndPraseOutput(output, p):
print(output)
if 'sucessfully' in output:
p.stdin.write('command')
cmd = ["./programm"]
p = subprocess.Popen(cmd, universal_newlines=True, shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
while p.poll() is None:
output = p.stdout.readline()
PrintAndPraseOutput(output, p)
Update:
same problem, no output after 'process successfully started'
import subprocess
def print_and_parse_output(output, p):
print(output)
if 'successfully' in output:
p.stdin.write('command\n')
with subprocess.Popen(["./programm"], universal_newlines=True, shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:
while proc.poll() is None:
output = proc.stdout.readline()
print_and_parse_output(output, proc)
Your I/O should be line buffered, so PrintAndPraseOutput should send a '\n' at the end of the string.
BTW, you have a couple of spelling errors. That function should be named print_and_parse_output to conform to PEP-0008, and "successfully" has 2 c's.
def print_and_parse_output(output, p):
print(output)
if 'successfully' in output:
p.stdin.write('command\n')
When using subprocess like this it's a good idea to put it in a with statement. From the subprocess.Popen` docs:
Popen objects are supported as context managers via the with
statement: on exit, standard file descriptors are closed, and the
process is waited for.
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
I'm running into some difficulties getting output from a subprocess stdout pipe. I'm launching some third party code via it, in order to extract log output. Up until a recent update of the third party code, everything worked fine. After the update, python has started blocking indefinitely, and not actually showing any output. I can manually launch the third party app fine and see output.
A basic version of the code I'm using:
import subprocess, time
from threading import Thread
def enqueue_output(out):
print "Hello from enqueue_output"
for line in iter(out.readline,''):
line = line.rstrip("\r\n")
print "Got %s" % line
out.close()
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
thread = Thread(target=enqueue_output, args=(proc.stdout,))
thread.daemon = True
thread.start()
time.sleep(30)
This works perfectly if I substitute third_party.exe for this script:
import time, sys
while True:
print "Test"
sys.stdout.flush()
time.sleep(1)
So I'm unclear as to magic needs to be done to get this working with the original command.
These are all variants of the subprocess.Popen line I've tried with no success:
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=0)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, shell=True)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_CONSOLE)
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, startupinfo=si)
Edit 1:
I can't actually use .communicate() in this case. The app I'm launching remains running for long periods of time (days to weeks). The only way I could actually test .communicate() would be to kill the app shortly after it launches, which I don't feel would give me valid results.
Even the non-threaded version of this fails:
import subprocess, time
from threading import Thread
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print "App started, reading output..."
for line in iter(proc.stdout.readline,''):
line = line.rstrip("\r\n")
print "Got: %s" % line
Edit 2:
Thanks to jdi, the following works okay:
import tempfile, time, subprocess
w = "test.txt"
f = open("test.txt","a")
p = subprocess.Popen("third_party.exe", shell=True, stdout=f,
stderr=subprocess.STDOUT, bufsize=0)
time.sleep(30)
with open("test.txt", 'r') as r:
for line in r:
print line
f.close()
First I would recommend that you simplify this example to make sure you can actually read anything. Remove the complication of the thread from the mix:
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
print proc.communicate()
If that works, great. Then you are having problems possibly with how you are reading the stdout directly or possibly in your thread.
If this does not work, have you tried piping stderr to stdout as well?
proc = subprocess.Popen("third_party.exe",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, bufsize=1)
Update
Since you say communicate() is deadlocking, here is another approach you can try to see if its a problem with the internal buffer of subprocess...
import tempfile
import subprocess
w = tempfile.NamedTemporaryFile()
p = subprocess.Popen('third_party.exe', shell=True, stdout=w,
stderr=subprocess.STDOUT, bufsize=0)
with open(w.name, 'r') as r:
for line in r:
print line
w.close()
args = ['svn','log','-v']
def foo(info=''):
import logging
import subprocess
import tempfile
try:
pipe = subprocess.Popen(args,bufsize = 0,\
stdout = subprocess.PIPE,\
stderr=subprocess.STDOUT)
except Exception as e:
logging.error(str(e))
return False
while 1:
s = pipe.stdout.read()
if s:
print s,
if pipe.returncode is None:
pipe.poll()
else:
break
if not 0 == pipe.returncode:
return False
return True
print foo()
This one should works,not thread,temp file magic.
I'm trying to launch an 'rsync' using subprocess module and Popen inside of a thread. After I call the rsync I need to read the output as well. I'm using the communicate method to read the output. The code runs fine when I do not use a thread. It appears that when I use a thread it hangs on the communicate call. Another thing I've noticed is that when I set shell=False I get nothing back from the communicate when running in a thread.
You didn't supply any code for us to look at, but here's a sample that does something similar to what you describe:
import threading
import subprocess
class MyClass(threading.Thread):
def __init__(self):
self.stdout = None
self.stderr = None
threading.Thread.__init__(self)
def run(self):
p = subprocess.Popen('rsync -av /etc/passwd /tmp'.split(),
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
self.stdout, self.stderr = p.communicate()
myclass = MyClass()
myclass.start()
myclass.join()
print myclass.stdout
Here's a great implementation not using threads:
constantly-print-subprocess-output-while-process-is-running
import subprocess
def execute(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = ''
# Poll process for new output until finished
for line in iter(process.stdout.readline, ""):
print line,
output += line
process.wait()
exitCode = process.returncode
if (exitCode == 0):
return output
else:
raise Exception(command, exitCode, output)
execute(['ping', 'localhost'])