I want to interact with a process.
I can start the process and print out the first two lines (something like 'process successfully started').
Now I want to send a new command to the process which should return again something like 'command done' but nothing happens.
Please help me.
import subprocess
def PrintAndPraseOutput(output, p):
print(output)
if 'sucessfully' in output:
p.stdin.write('command')
cmd = ["./programm"]
p = subprocess.Popen(cmd, universal_newlines=True, shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
while p.poll() is None:
output = p.stdout.readline()
PrintAndPraseOutput(output, p)
Update:
same problem, no output after 'process successfully started'
import subprocess
def print_and_parse_output(output, p):
print(output)
if 'successfully' in output:
p.stdin.write('command\n')
with subprocess.Popen(["./programm"], universal_newlines=True, shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:
while proc.poll() is None:
output = proc.stdout.readline()
print_and_parse_output(output, proc)
Your I/O should be line buffered, so PrintAndPraseOutput should send a '\n' at the end of the string.
BTW, you have a couple of spelling errors. That function should be named print_and_parse_output to conform to PEP-0008, and "successfully" has 2 c's.
def print_and_parse_output(output, p):
print(output)
if 'successfully' in output:
p.stdin.write('command\n')
When using subprocess like this it's a good idea to put it in a with statement. From the subprocess.Popen` docs:
Popen objects are supported as context managers via the with
statement: on exit, standard file descriptors are closed, and the
process is waited for.
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
Related
I'm using python 3.6 in Windows, and my aim is to run a cmd command and save the output as a string in a variable.
I'm using subprocess and its objects like check_output, Popen and Communicate, and getoutput. But here is my problem with these:
subprocess.check_output the problem is if the code returns non-zero it raises an exception and I can't read the output, for example, executing the netstat -abcd.
stdout_value = (subprocess.check_output(command, shell=True, stdin=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=self.timeout)).decode()
subprocess.Popen and communicate() the problem is some commands like netstat -abcd returns empty from communicate().
self.process = subprocess.Popen(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
stdin=subprocess.PIPE)
try:
self.process.wait(timeout=5)
stdout_value = self.process.communicate()[0]
except:
self.process.kill()
self.process.wait()
subprocess.getoutput(Command) is ok but there is no timeout so my code would block forever on executing some commands like netstat. I also tried to run it as a thread but the code is blocking and I can't stop the thread itself.
stdout_value = subprocess.getoutput(command)
What I want is to run any cmd commands (blocking like netstat or nonblocking like dir) with timeout for example if the user executes netstat it only shows the lines generated in timeout and then kills it.
Thanks.
EDIT------
According to Jean's answer, I rewrote the code but the timeout doesn't work in running some commands like netstat.
# command = "netstat"
command = "test.exe" # running an executable program can't be killed after timeout
self.process = subprocess.run(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
timeout=3,
universal_newlines=True
)
stdout_value = self.process.stdout
subprocess.run() with timeout doesn't seem to run properly on Windows.
You can try running the subprocess within a Timer-thread or in case of you dont need communicate(), you can do something like this:
import time
import subprocess
#cmd = 'cmd /c "dir c:\\ /s"'
#cmd = ['calc.exe']
cmd = ['ping', '-n', '25', 'www.google.com']
#_stdout = open('C:/temp/stdout.txt', 'w')
#_stderr = open('C:/temp/stderr.txt', 'w')
_stdout = subprocess.PIPE
_stderr = subprocess.PIPE
proc = subprocess.Popen(cmd, bufsize=0, stdout=_stdout, stderr=_stderr)
_startTime = time.time()
while proc.poll() is None and proc.returncode is None:
if (time.time() - _startTime) >= 5:
print ("command ran for %.6f seconds" % (time.time() - _startTime))
print ("timeout - killing process!")
proc.kill()
break
print (proc.stdout.read())
It works for all three commands on Win7/py3.6, but not for the 'killed-netstat' issue!
When I use stdout in my Popen with stdin my process does not run.
def program():
p = subprocess.Popen(link, shell=True, stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
out = p.communicate(longquery)
print out
When I run this my out comes out as (None, None) although I do have an output that shows up on the console. Why is this happening and how can I save the output that is coming out?
You need to pipe the output data per the manual:
"to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too."
def program():
p = subprocess.Popen(link, shell=True, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
out = p.communicate(longquery)
print out
You need to set stdout=PIPE as well.
Change your code to:
from subprocess import Popen, PIPE, STDOUT
def program():
p = Popen(link, shell=True, stdin=PIPE, stderr=STDOUT, stdout=PIPE)
out = p.communicate(longquery)
print out
I'm running into some difficulties getting output from a subprocess stdout pipe. I'm launching some third party code via it, in order to extract log output. Up until a recent update of the third party code, everything worked fine. After the update, python has started blocking indefinitely, and not actually showing any output. I can manually launch the third party app fine and see output.
A basic version of the code I'm using:
import subprocess, time
from threading import Thread
def enqueue_output(out):
print "Hello from enqueue_output"
for line in iter(out.readline,''):
line = line.rstrip("\r\n")
print "Got %s" % line
out.close()
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
thread = Thread(target=enqueue_output, args=(proc.stdout,))
thread.daemon = True
thread.start()
time.sleep(30)
This works perfectly if I substitute third_party.exe for this script:
import time, sys
while True:
print "Test"
sys.stdout.flush()
time.sleep(1)
So I'm unclear as to magic needs to be done to get this working with the original command.
These are all variants of the subprocess.Popen line I've tried with no success:
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=0)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, shell=True)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_CONSOLE)
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, startupinfo=si)
Edit 1:
I can't actually use .communicate() in this case. The app I'm launching remains running for long periods of time (days to weeks). The only way I could actually test .communicate() would be to kill the app shortly after it launches, which I don't feel would give me valid results.
Even the non-threaded version of this fails:
import subprocess, time
from threading import Thread
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print "App started, reading output..."
for line in iter(proc.stdout.readline,''):
line = line.rstrip("\r\n")
print "Got: %s" % line
Edit 2:
Thanks to jdi, the following works okay:
import tempfile, time, subprocess
w = "test.txt"
f = open("test.txt","a")
p = subprocess.Popen("third_party.exe", shell=True, stdout=f,
stderr=subprocess.STDOUT, bufsize=0)
time.sleep(30)
with open("test.txt", 'r') as r:
for line in r:
print line
f.close()
First I would recommend that you simplify this example to make sure you can actually read anything. Remove the complication of the thread from the mix:
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
print proc.communicate()
If that works, great. Then you are having problems possibly with how you are reading the stdout directly or possibly in your thread.
If this does not work, have you tried piping stderr to stdout as well?
proc = subprocess.Popen("third_party.exe",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, bufsize=1)
Update
Since you say communicate() is deadlocking, here is another approach you can try to see if its a problem with the internal buffer of subprocess...
import tempfile
import subprocess
w = tempfile.NamedTemporaryFile()
p = subprocess.Popen('third_party.exe', shell=True, stdout=w,
stderr=subprocess.STDOUT, bufsize=0)
with open(w.name, 'r') as r:
for line in r:
print line
w.close()
args = ['svn','log','-v']
def foo(info=''):
import logging
import subprocess
import tempfile
try:
pipe = subprocess.Popen(args,bufsize = 0,\
stdout = subprocess.PIPE,\
stderr=subprocess.STDOUT)
except Exception as e:
logging.error(str(e))
return False
while 1:
s = pipe.stdout.read()
if s:
print s,
if pipe.returncode is None:
pipe.poll()
else:
break
if not 0 == pipe.returncode:
return False
return True
print foo()
This one should works,not thread,temp file magic.
I'm looking for a Python solution that will allow me to save the output of a command in a file without hiding it from the console.
FYI: I'm asking about tee (as the Unix command line utility) and not the function with the same name from Python intertools module.
Details
Python solution (not calling tee, it is not available under Windows)
I do not need to provide any input to stdin for called process
I have no control over the called program. All I know is that it will output something to stdout and stderr and return with an exit code.
To work when calling external programs (subprocess)
To work for both stderr and stdout
Being able to differentiate between stdout and stderr because I may want to display only one of the to the console or I could try to output stderr using a different color - this means that stderr = subprocess.STDOUT will not work.
Live output (progressive) - the process can run for a long time, and I'm not able to wait for it to finish.
Python 3 compatible code (important)
References
Here are some incomplete solutions I found so far:
http://devlishgenius.blogspot.com/2008/10/logging-in-real-time-in-python.html (mkfifo works only on Unix)
http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html (doesn't work at all)
Diagram http://blog.i18n.ro/wp-content/uploads/2010/06/Drawing_tee_py.png
Current code (second try)
#!/usr/bin/python
from __future__ import print_function
import sys, os, time, subprocess, io, threading
cmd = "python -E test_output.py"
from threading import Thread
class StreamThread ( Thread ):
def __init__(self, buffer):
Thread.__init__(self)
self.buffer = buffer
def run ( self ):
while 1:
line = self.buffer.readline()
print(line,end="")
sys.stdout.flush()
if line == '':
break
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutThread = StreamThread(io.TextIOWrapper(proc.stdout))
stderrThread = StreamThread(io.TextIOWrapper(proc.stderr))
stdoutThread.start()
stderrThread.start()
proc.communicate()
stdoutThread.join()
stderrThread.join()
print("--done--")
#### test_output.py ####
#!/usr/bin/python
from __future__ import print_function
import sys, os, time
for i in range(0, 10):
if i%2:
print("stderr %s" % i, file=sys.stderr)
else:
print("stdout %s" % i, file=sys.stdout)
time.sleep(0.1)
Real output
stderr 1
stdout 0
stderr 3
stdout 2
stderr 5
stdout 4
stderr 7
stdout 6
stderr 9
stdout 8
--done--
Expected output was to have the lines ordered. Remark, modifying the Popen to use only one PIPE is not allowed because in the real life I will want to do different things with stderr and stdout.
Also even in the second case I was not able to obtain real-time like out, in fact all the results were received when the process finished. By default, Popen should use no buffers (bufsize=0).
I see that this is a rather old post but just in case someone is still searching for a way to do this:
proc = subprocess.Popen(["ping", "localhost"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
with open("logfile.txt", "w") as log_file:
while proc.poll() is None:
line = proc.stderr.readline()
if line:
print "err: " + line.strip()
log_file.write(line)
line = proc.stdout.readline()
if line:
print "out: " + line.strip()
log_file.write(line)
If requiring python 3.6 isn't an issue there is now a way of doing this using asyncio. This method allows you to capture stdout and stderr separately but still have both stream to the tty without using threads. Here's a rough outline:
class RunOutput:
def __init__(self, returncode, stdout, stderr):
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
async def _read_stream(stream, callback):
while True:
line = await stream.readline()
if line:
callback(line)
else:
break
async def _stream_subprocess(cmd, stdin=None, quiet=False, echo=False) -> RunOutput:
if isWindows():
platform_settings = {"env": os.environ}
else:
platform_settings = {"executable": "/bin/bash"}
if echo:
print(cmd)
p = await asyncio.create_subprocess_shell(
cmd,
stdin=stdin,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
**platform_settings
)
out = []
err = []
def tee(line, sink, pipe, label=""):
line = line.decode("utf-8").rstrip()
sink.append(line)
if not quiet:
print(label, line, file=pipe)
await asyncio.wait(
[
_read_stream(p.stdout, lambda l: tee(l, out, sys.stdout)),
_read_stream(p.stderr, lambda l: tee(l, err, sys.stderr, label="ERR:")),
]
)
return RunOutput(await p.wait(), out, err)
def run(cmd, stdin=None, quiet=False, echo=False) -> RunOutput:
loop = asyncio.get_event_loop()
result = loop.run_until_complete(
_stream_subprocess(cmd, stdin=stdin, quiet=quiet, echo=echo)
)
return result
The code above was based on this blog post: https://kevinmccarthy.org/2016/07/25/streaming-subprocess-stdin-and-stdout-with-asyncio-in-python/
This is a straightforward port of tee(1) to Python.
import sys
sinks = sys.argv[1:]
sinks = [open(sink, "w") for sink in sinks]
sinks.append(sys.stderr)
while True:
input = sys.stdin.read(1024)
if input:
for sink in sinks:
sink.write(input)
else:
break
I'm running on Linux right now but this ought to work on most platforms.
Now for the subprocess part, I don't know how you want to 'wire' the subprocess's stdin, stdout and stderr to your stdin, stdout, stderr and file sinks, but I know you can do this:
import subprocess
callee = subprocess.Popen(
["python", "-i"],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
Now you can access callee.stdin, callee.stdout and callee.stderr like normal files, enabling the above "solution" to work. If you want to get the callee.returncode, you'll need to make an extra call to callee.poll().
Be careful with writing to callee.stdin: if the process has exited when you do that, an error may be rised (on Linux, I get IOError: [Errno 32] Broken pipe).
This is how it can be done
import sys
from subprocess import Popen, PIPE
with open('log.log', 'w') as log:
proc = Popen(["ping", "google.com"], stdout=PIPE, encoding='utf-8')
while proc.poll() is None:
text = proc.stdout.readline()
log.write(text)
sys.stdout.write(text)
If you don't want to interact with the process you can use the subprocess module just fine.
Example:
tester.py
import os
import sys
for file in os.listdir('.'):
print file
sys.stderr.write("Oh noes, a shrubbery!")
sys.stderr.flush()
sys.stderr.close()
testing.py
import subprocess
p = subprocess.Popen(['python', 'tester.py'], stdout=subprocess.PIPE,
stdin=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
print stdout, stderr
In your situation you can simply write stdout/stderr to a file first. You can send arguments to your process with communicate as well, though I wasn't able to figure out how to continually interact with the subprocess.
On Linux, if you really need something like the tee(2) syscall, you can get it like this:
import os
import ctypes
ld = ctypes.CDLL(None, use_errno=True)
SPLICE_F_NONBLOCK = 0x02
def tee(fd_in, fd_out, length, flags=SPLICE_F_NONBLOCK):
result = ld.tee(
ctypes.c_int(fd_in),
ctypes.c_int(fd_out),
ctypes.c_size_t(length),
ctypes.c_uint(flags),
)
if result == -1:
errno = ctypes.get_errno()
raise OSError(errno, os.strerror(errno))
return result
To use this, you probably want to use Python 3.10 and something with os.splice (or use ctypes in the same way to get splice). See the tee(2) man page for an example.
My solution isn't elegant, but it works.
You can use powershell to gain access to "tee" under WinOS.
import subprocess
import sys
cmd = ['powershell', 'ping', 'google.com', '|', 'tee', '-a', 'log.txt']
if 'darwin' in sys.platform:
cmd.remove('powershell')
p = subprocess.Popen(cmd)
p.wait()
I'm trying to launch an 'rsync' using subprocess module and Popen inside of a thread. After I call the rsync I need to read the output as well. I'm using the communicate method to read the output. The code runs fine when I do not use a thread. It appears that when I use a thread it hangs on the communicate call. Another thing I've noticed is that when I set shell=False I get nothing back from the communicate when running in a thread.
You didn't supply any code for us to look at, but here's a sample that does something similar to what you describe:
import threading
import subprocess
class MyClass(threading.Thread):
def __init__(self):
self.stdout = None
self.stderr = None
threading.Thread.__init__(self)
def run(self):
p = subprocess.Popen('rsync -av /etc/passwd /tmp'.split(),
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
self.stdout, self.stderr = p.communicate()
myclass = MyClass()
myclass.start()
myclass.join()
print myclass.stdout
Here's a great implementation not using threads:
constantly-print-subprocess-output-while-process-is-running
import subprocess
def execute(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = ''
# Poll process for new output until finished
for line in iter(process.stdout.readline, ""):
print line,
output += line
process.wait()
exitCode = process.returncode
if (exitCode == 0):
return output
else:
raise Exception(command, exitCode, output)
execute(['ping', 'localhost'])