None in redirected stdout - python

I have this part of following code:
p = subprocess.Popen(['C:/Python27/python.exe', '-m', 'robot', '-d', logs_directory, input_file], stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
output = sys.stdout.write(line)
#sys.stdout.write(line)
print "\n\n"
print "************************************************"
print output
print "\n\n"
print "************************************************"
print "\n\n"
But 'output' variable in console shows 'none'
************************************************
None
************************************************
What i am doing wrong here?

sys.stdout.write(line) returns the number of characters written on stdout.
You could write your loop like this:
output = ""
for line in iter(p.stdout.readline, ''):
output += line
sys.stdout.write(line)
imho you can also remove the sys.stdout.write(line) part. But I don't know what you are meant to do with it so...

Try output += line instead of output = sys.stdout.write(line)

You can also try function check_output.
Here is an example:
>>> subprocess.check_output(["echo", "Hello World!"])
'Hello World!\n'
You can try:
cmd = ['C:/Python27/python.exe', '-m', 'robot', '-d', logs_directory, input_file]
output = subprocess.check_output(cmd)
print output
In most of the cases, it can do the job simply.
For details, you may want to reference to subprocess.check_output

Related

'list index out of range' from subprocess stdout

Whetever I do, I get this error when trying to print anything from 'ip_macAddr' using an index.
Type is 'list' so I don't get why. The only explanation I have is that there's some type of caracter coming from the stdout of that subprocess that's messing things up.
Also I tried converting to string with no different result.
At this point I have ne clue.
Thanks for your help!
Code
#!/usr/bin/env python3.7
import sys
import subprocess
import os
IP_NETWORK = '192.168.254.10'
IP_DEVICE = '192.168.254.194'
proc = subprocess.Popen(['ping', IP_NETWORK], stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
# print(line)
if not line:
break
connected_ip = line.decode('utf-8').split()[3].replace(':','')
proc2 = subprocess.Popen(['ip', 'neigh', 'show', 'to', connected_ip], stdout=subprocess.PIPE)
ip_macAddr = proc2.stdout.readline().decode('utf-8').split()
print(connected_ip)
# print(type(ip_macAddr))
print(ip_macAddr[0])
Error
IndexError: list index out of range
edit 1
The output of print(ip_macAddr) would be: ['192.168.254.10', 'dev', 'wlp61s0', 'lladdr', '88:88:a1:b2:c3:88', 'REACHABLE']
edit 2
print(type(ip_macAddr)) returns <class 'list'>
edit 3
Traceback
Traceback (most recent call last):
File "./device-connected-to-network.py", line 34, in <module>
print(ip_macAddr[0])
IndexError: list index out of range
This would happen using ANY index and I also tried using:
print(ip_macAddr[0]) or ip_macAddr = proc2.stdout.split()[0]
Again; same result using ANY index # and print(type(ip_macAddr)) returns type 'list'.
Another exemple;
The output of print(ip_macAddr)is ['192.168.254.10', 'dev', 'wlp61s0', 'lladdr', '74:83:c2:d2:a4:12', 'REACHABLE'].
So, if I do (in another file) - using index #4:
a = ['192.168.254.10', 'dev', 'wlp61s0', 'lladdr', '11:22:a1:b2:c3:33', 'REACHABLE']
print(type(a))
print(a[4])
The output is:
<class 'list'>
11:22:a1:b2:c3:33
So works as expected in this context, but not within the actual program.
split()[3] will fail with an IndexError on any line which has less than four whitespace-separated fields.
A common fix is to simply skip lines with fewer fields.
Similarly, if ip produces an empty string, splitting it produces an empty list, which you can't get the first element of.
You can also simplify your code by passing in text=True, and use subprocess.run() where you don't particularly need Popen(). I have also removed unused imports.
#!/usr/bin/env python3.7
import subprocess
IP_NETWORK = '192.168.254.10'
IP_DEVICE = '192.168.254.194'
proc = subprocess.Popen(['ping', IP_NETWORK],
stdout=subprocess.PIPE, text=True)
while True:
line = proc.stdout.readline()
# print(line)
if not line:
break
fields = line.split()
if len(fields) < 4:
continue
connected_ip = fields[3].replace(':','')
proc2 = subprocess.run(
['ip', 'neigh', 'show', 'to', connected_ip],
capture_output=True, text=True)
result = proc2.stdout
if not result:
continue
ip_macAddr = result[0]
print(connected_ip)
# print(type(ip_macAddr))
print(ip_macAddr)
Even better, trap and report any exceptions.
try:
proc2 = subprocess.run(
['ip', 'neigh', 'show', 'to', connected_ip],
capture_output=True, text=True,
check=True) # important
result = proc2.stdout
except CalledProcessError as exc:
print('`ip\' for', connected_ip, 'failed:', exc)
continue

How to remove \n\r or [] in stdout.readlines() in paramiko python

stdin, stdout, stderr = ssh.exec_command(f'vol create {vol} -aggregate {aggregate} -size {size}')
print("The output is: ")
print(stdout.readlines())
print("The error is: ")
print(stderr.readlines())
Output:
The output is:
['\r\r\n', 'Last login time: 9/11/2020 13:54:24\r\r\n', '\x\r\n', 'Error: command failed: Duplicate volume name vol.\r\n', '\r\n']
The error is:
[]
Process finished with exit code 0
I suggest that you print each line of the list:
lines = stdout.readlines()
for l in lines:
print(l)
Or, alternatively, join the list and print it:
print(''.join(stdout.readlines()))

Custom Popen.communicate method gives wrong output

Let's start by considering this code:
proc_stdin.py
import sys
if __name__ == '__main__':
for i, line in enumerate(sys.stdin):
sys.stdout.write(line)
test.py
import subprocess
def run_bad(target, input=None):
proc = subprocess.Popen(
target,
universal_newlines=True,
shell=True,
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE if input else subprocess.DEVNULL,
stdout=subprocess.PIPE,
)
if input:
proc.stdin.write(input)
proc.stdin.flush()
proc.stdin.close()
lines = []
for line in iter(proc.stdout.readline, ""):
line = line.rstrip("\n")
lines.append(line)
proc.stdout.close()
ret_code = proc.wait()
return "\n".join(lines)
def run_good(target, input):
return subprocess.Popen(
target,
universal_newlines=True,
shell=True,
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
).communicate(input=input)[0]
if __name__ == '__main__':
lst = [
"",
"token1",
"token1\n",
"token1\r\n",
"token1\n\n",
"token1\r\n\ntoken2",
"token1 token2",
"token1\ntoken2",
"token1\r\ntoken2",
"token1\n\ntoken2",
"token1\r\n\ntoken2",
"token1 \ntoken2\ntoken2\n"
]
cmd = "python proc_stdin.py"
for inp in lst:
a, b = run_bad(cmd, inp), run_good(cmd, inp)
if a != b:
print("Error: {} vs {}".format(repr(a), repr(b)))
else:
print("ok: {}".format(repr(a)))
Output:
ok: ''
ok: 'token1'
Error: 'token1' vs 'token1\n'
Error: 'token1\n' vs 'token1\n\n'
Error: 'token1\n' vs 'token1\n\n'
ok: 'token1\n\n\ntoken2'
ok: 'token1 token2'
ok: 'token1\ntoken2'
ok: 'token1\n\ntoken2'
ok: 'token1\n\ntoken2'
ok: 'token1\n\n\ntoken2'
Error: 'token1 \ntoken2\ntoken2' vs 'token1 \ntoken2\ntoken2\n'
My question is, why is the output of both run_bad & run_good not equal in all cases? How would you change the run_bad function so the output becomes equal than run_good?
You also may wonder, why are you not using directly Popen.communicate for this particular case or other helpers from subprocess module? Well, in the real world case I'm creating a plugin for SublimeText3 which is forcing me to stick to python3.3 (can't use many of the modern subprocess goodies) plus I'd like to inject some callbacks while reading the lines from stdout and that's something I can't do by using the Popen.communicate method (as far as I know).
Thanks in advance.
If you strip newlines from every line and then add them back between the lines, what happens to the last newline (if any)? (There’s no final, empty line after a final newline because your iter discards it.) This is why Python’s readline (or line iteration) function includes the newlines: they’re necessary to represent the end of the file accurately.

grep: write error: Broken pipe with subprocess

I get couple of grep:write errors when I run this code.
What am I missing?
This is only part of it:
while d <= datetime.datetime(year, month, daysInMonth[month]):
day = d.strftime("%Y%m%d")
print day
results = [day]
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+"*.txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt", shell=True, stdout=subprocess.PIPE, )
output1=first.communicate()[0]
d += delta
day = d.strftime("%Y%m%d")
second=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+"*.txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt", shell=True, stdout=subprocess.PIPE, )
output2=second.communicate()[0]
articleList = (output1.split('\n'))
articleList2 = (output2.split('\n'))
results.append( len(articleList)+len(articleList2))
w.writerow(tuple(results))
d += delta
When you do
A | B
in a shell, process A's output is piped into process B as input. If process B shuts down before reading all of process A's output (e.g. because it found what it was looking for, which is the function of the -l option), then process A may complain that its output pipe was prematurely closed.
These errors are basically harmless, and you can work around them by redirecting stderr in the subprocesses to /dev/null.
A better approach, though, may simply be to use Python's powerful regex capabilities to read the files:
def fileContains(fn, pat):
with open(file) as f:
for line in f:
if re.search(pat, line):
return True
return False
first = []
for file in glob.glob(monthDir +"/"+day+"*.txt"):
if fileContains(file, 'Algeria|Bahrain') and fileContains(file, 'Protest|protesters'):
file.append(first)
To find the files matching two patterns, the command structure should be:
grep -l pattern1 $(grep -l pattern2 files)
$(command) substitutes the output of the command into the command line.
So your script should be:
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' $("+ grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt)", shell=True, stdout=subprocess.PIPE, )
and similarly for second
If you are just looking for whole words, you could use the count() member function;
# assuming names is a list of filenames
for fn in names:
with open(fn) as infile:
text = infile.read().lower()
# remove puntuation
text = text.replace(',', '')
text = text.replace('.', '')
words = text.split()
print "Algeria:", words.count('algeria')
print "Bahrain:", words.count('bahrain')
print "protesters:", words.count('protesters')
print "protest:", words.count('protest')
If you want more powerful filtering, use re.
Add stderr args in the Popen function based on the python version the stderr value will change. This will support if the python version is less than 3
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+".txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+".txt", shell=True, stdout=subprocess.PIPE, stderr = subprocess.STDOUT)

How to print output using python?

When this .exe file runs it prints a screen full of information and I want to print a particular line out to the screen, here on line "6":
cmd = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
outputlist = outputstring.splitlines()
Output = outputlist[5]
print cmd
This works fine:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
This doesn't work:
cmd = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
I get an error saying Output isn't defined. But when I cut and paste:
outputstring = process.communicate()[0]
outputlist = outputstring.splitlines()
Output = outputlist[5]
before the cmd statement it tells me the process isn't defined. str(Output) should be what is printed on line 6 when the .exe is ran.
You're trying to append the result of a call into the call itself. You have to run the command once without the + str(Output) part to get the output in the first place.
Think about it this way. Let's say I was adding some numbers together.
z = 5 + b
b = z + 2
I have to define either z or b before the statements, depending on the order of the two statements. I can't use a variable before I know what it is. You're doing the same thing, using the Output variable before you define it.
It's not supposed to be a "dance" to move things around. It's a matter of what's on the left side of the "=". If it's on the left side, it's getting created; if it's on the right side it's being used.
As it is, your example can't work even a little bit because line one wants part of output, which isn't created until the end.
The easiest way to understand this is to work backwards. You want to see as the final result?
print output[5]
Right? So to get there, you have to get this from a larger string, right?
output= outputstring.splitlines()
print output[5]
So where did outputstring come from? It was from some subprocess.
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
So where did process come from? It was created by subprocess Popen
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
So where did cmd come from? I can't tell. Your example doesn't make sense on what command is being executed.
cmd = ?
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
Just change your first line to:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
and the print statement at the end to:
print cmd + str(Output)
This is without knowing exactly what it is you want to print...
It -seems- as if your problem is trying to use Output before you actually define what the Output variable is (as the posts above)
Like you said, a variable has to be declared before you can use it. Therefore when you call str(Output) ABOVE Output = outputlist[5], Output doesn't exist yet. You need the actually call first:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
then you can print the output of that command:
cmd_return = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
should be the line directly above print cmd_return.

Categories