We Keep Coding

Difficulties in pipes between Python subprocesses in Linux

The following script (which should take the output from p1 and pipe it to p2, and then output the result in the terminal) doesn't seem to work as expected.
Code as follows :
#!/binr/bin/python
# Script to lauch mosquitto_sub to monitor a mqtt topic -- SYNTAX : mosk topic
import sys
import subprocess
total = len(sys.argv)
cmdargs = str(sys.argv)
print ("The total numbers of args passed to the script: %d " % total)
print ("Args list: %s " % cmdargs)
# Pharsing args one by one
print ("Script name: %s" % str(sys.argv[0]))
print ("First argument: %s" % str(sys.argv[1]))
path = str(sys.argv[1])
print (path)
p1 = subprocess.Popen(['mosquitto_sub','-h','192.168.1.100','-t',path,'-v'], shell=False, stdout=subprocess.PIPE)
p2 = subprocess.Popen(['ts'], stdin=p1.stdout, shell=False, stdout=subprocess.PIPE)
for line in p2.stdout:
sys.stdout.write(line)
with an input as follows "./mosk2.py test/+" and whilst publishing MQTT via mosquitto on the relevant topics, I never get the expected output in the terminal

Solved - I ended up stepping neatly around the problem (cheating) as follows :
cmd = "mosquitto_sub -h 192.168.1.100 -v -t " + topic + " | ts"
print "The command which was executed is : " , cmd
def run_command(command):
process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
rc = process.poll()
return rc
run_command(cmd) #This is the lauch of the actual command

How can a call to suprocess.checkoutput() return nothing when I just echo a string

In python, I try to run some command but I can't get it work, to sum up here is a script that should execute four simple commands via subproscess, store theirs respective outputs and then display them.
#!/usr/bin/python
import subprocess
output_1 = None
output_2 = None
output_3 = None
output_4 = None
print "Start"
try:
output_1 = subprocess.check_output(["ls", "-al"], shell=True, stderr=subprocess.STDOUT)
output_2 = subprocess.check_output(["echo", "'Hi'"], shell=True, stderr=subprocess.STDOUT)
output_3 = subprocess.check_output(["echo", "\"Hi\""], shell=True, stderr=subprocess.STDOUT)
output_4 = subprocess.check_output(["echo", "Hi"], shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as ex:
print "Command '%s' return exit code %s:\n\n%s" % (ex.cmd, ex.returncode, ex.output)
print "Out 1: '%s'" % output_1
print "Out 2: '%s'" % output_2
print "Out 3: '%s'" % output_3
print "Out 4: '%s'" % output_4
I do not undestand why there is no 'Hi' in my outputs... Here is the output I get:
Start
Out 1: '
test
undefined10_error.png
undefined20_error.png
undefined35_error.png
undefined40_error.png
'
Out 2: '
'
Out 3: '
'
Out 4: '
'

By passing the shell=True argument, you are telling the interpreter to execute the command as a single string. This means that the first argument needs to be a string. If you look at the output of your first command you'll notice that you're not getting the flags am, but just a standard ls.
If you want to use shell=True, pass the argument as a str:
> subprocess.check_output("echo 'Hello World!'", shell=True, stderr=subprocess.STDOUT)
b'Hello World!\n'
If you're planning on using variables, you'll need to escape them yourself with shelex.quote.
To use a local variable:
>>> import shlex
>>> local_variable = 'Hi \'world"'
>>> command = 'echo {}'.format(shlex.quote(local_variable))
>>> subprocess.check_output(command,shell=True, stderr=subprocess.STDOUT)
b'Hi \'world"\n'

how to use python popen to print and also store communicate and return code?

this is the function i am creating, i have one argument that can tell to print real time or not since some of the process take like an hour. and since i am subprocesing several at the same time, another argument to raise an error and stop everything, or just let the main script run.
but if i do print_real_time True, i loose the p.communicate()
i could store all the prints from the iter in a variable and return that, but how do i put in order the std out, and the stderr, and get the return value to see if did fail or not?
def launch_subprocess_cmd(command_to_lunch, cwd=None, print_real_time=False, raise_errors=True):
"""
for a given command line will lunch that as a subprocess
:param command_to_lunch: string
:param print_real_time: boolean
:param cwd: the folder path from where the command should be run.
:param raise_errors: boolean if the return code of the subprocess is different than 0 raise an error an stop all scripts.
else the main script will keep running and can access the third return value of this function and decide what to do with it.
:return: list com return the stdout and the stderr of the Popen subprocess.
"""
if cwd is None:
p = subprocess.Popen(command_to_lunch, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
else:
p = subprocess.Popen(command_to_lunch, cwd=cwd, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
if print_real_time is True:
for i in iter(p.stdout.readline, b''):
print i
com = p.communicate()
if raise_errors is True:
if p.returncode != 0:
raise ValueError("\n\nSubprocess fail: \n" + "Error captures: \n" + "stdout:\n" + com[0] + "\nstderr:\n" + com[1] + "\n")
# com[0] is std_out, com[1] is std_err and p.return code is if the subprocess was successful or not with a int number
return com[0], com[1], p.returncode
thanks guys =)

A possible solution to your problem is to store the stdout stream in a list when print_real_time is True and then use the content of the list to generate the stdout data string. If print_real_time is not True, then use the content from com[0] instead.
def launch_subprocess_cmd(cmd, cwd=None, print_real_time=False, raise_errors=True):
"""
for a given command line will lunch that as a subprocess
:param cmd: string
:param print_real_time: boolean
:param cwd: the folder path from where the command should be run.
:param raise_errors: boolean if the return code of the subprocess is different
than 0 raise an error an stop all scripts else
the main script will keep running and can access the third
return value of this function and decide what to do with it.
:return: list com return the stdout and the stderr of the Popen subprocess.
"""
if cwd is None:
p = subprocess.Popen(cmd, stderr=subprocess.PIPE,
stdout=subprocess.PIPE, shell=True)
else:
p = subprocess.Popen(cmd, cwd=cwd, stderr=subprocess.PIPE,
stdout=subprocess.PIPE, shell=True)
stdout_list = []
if print_real_time is True:
for i in iter(p.stdout.readline, b''):
stdout_list.append(i)
print i
com = p.communicate()
stdout_data = "".join(stdout_list) if print_real_time is True else com[0]
if raise_errors is True:
if p.returncode != 0:
raise ValueError("\n\nSubprocess fail: \n" + "Error captures: \n" +
"stdout:\n" + stdout_data + "\nstderr:\n" +
com[1] + "\n")
# stdout_data is stdout, com[1] is stderr and
# p.return code is if the subprocess was successful or not with a int number
return stdout_data, com[1], p.returncode
As a side note, I would also urge you to try to rewrite the program to not use shell=True in your Popen calls. It may require that you preprocess the input cmd into a list of base command and arguments, but it is generally considered a bad idea to use shell=True.

launch_subprocess_cmd(command_to_lunch, cwd=None, print_real_time=False, raise_errors=True):
if cwd is None:
p = subprocess.Popen(command_to_lunch, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
else:
p = subprocess.Popen(command_to_lunch, cwd=cwd, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
stdout_list = []
errout_list = []
if print_real_time is True:
for i in iter(p.stdout.readline, b''):
stdout_list.append(i)
print i
for j in iter(p.stderr.readline, b''):
errout_list.append(j)
print j
com = p.communicate()
if print_real_time is True:
stdout_data = "".join(stdout_list)
std_err_data = "".join(errout_list)
else:
stdout_data = com[0]
std_err_data = com[1]
if raise_errors is True:
if p.returncode != 0:
raise ValueError("\n\npopen fail:\n" + command_to_lunch + "\nError:\n" + "Error captures:\n" + "stdout:\n" + stdout_data + "\nstderr:\n" + std_err_data + "\n")
# com[0] is out, com[1] is errors and p.return code is if it fail or not
return stdout_data, std_err_data, p.returncode

Print output of external command in realtime and have it in a string at the same time in python

For example:
#!/usr/bin/env python3
# cmd.py
import time
for i in range(10):
print("Count %d" % i)
time.sleep(1)
#!/usr/bin/env python3
import subprocess
# useCmd.py
p = subprocess.Popen(['./cmd.py'], stdout=subprocess.PIPE)
out, err = p.communicate()
out = out.decode()
print(out)
In useCmd.py I can print out the output of cmd.py, but only after it's finished outputting. How can I print out it in realtime and still have it stored in a string? (sort of like tee in bash.)

If you don't have to deal with stdin, you could avoid using communicate that is blocking, and read directly from the process stdout until your stdout ends:
p = subprocess.Popen(['python', 'cmd.py'], stdout=subprocess.PIPE)
# out, err = p.communicate()
while True:
line = p.stdout.readline()
if line != '':
print line,
else:
break
related

How to get the last N lines of a subprocess' stderr stream output?

I am a Python newbie writing a Python (2.7) script that needs to exec a number of external applications, one of which writes a lot of output to its stderr stream. What I am trying to figure out is a concise and succinct way (in Python) to get the last N lines from that subprocess' stderr output stream.
Currently, I am running that external application from my Python script like so:
p = subprocess.Popen('/path/to/external-app.sh', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode != 0:
print "ERROR: External app did not complete successfully (error code is " + str(p.returncode) + ")"
print "Error/failure details: ", stderr
status = False
else:
status = True
I'd like to capture the last N lines of output from its stderr stream so that they can be written to a log file or emailed, etc.

N = 3 # for 3 lines of output
p = subprocess.Popen(['/path/to/external-app.sh'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode != 0:
print ("ERROR: External app did not complete successfully "
"(error code is %s)" % p.returncode)
print "Error/failure details: ", '\n'.join(stderr.splitlines()[-N:])
status = False
else:
status = True

If the whole output can't be stored in RAM then:
import sys
from collections import deque
from subprocess import Popen, PIPE
from threading import Thread
ON_POSIX = 'posix' in sys.builtin_module_names
def start_thread(func, *args):
t = Thread(target=func, args=args)
t.daemon = True
t.start()
return t
def consume(infile, output):
for line in iter(infile.readline, ''):
output(line)
infile.close()
p = Popen(['cat', sys.argv[1]], stdout=PIPE, stderr=PIPE,
bufsize=1, close_fds=ON_POSIX)
# preserve last N lines of stdout, print stderr immediately
N = 100
queue = deque(maxlen=N)
threads = [start_thread(consume, *args)
for args in (p.stdout, queue.append), (p.stderr, sys.stdout.write)]
for t in threads: t.join() # wait for IO completion
print ''.join(queue), # print last N lines
retcode = p.wait()