We Keep Coding

'list index out of range' from subprocess stdout

Whetever I do, I get this error when trying to print anything from 'ip_macAddr' using an index.
Type is 'list' so I don't get why. The only explanation I have is that there's some type of caracter coming from the stdout of that subprocess that's messing things up.
Also I tried converting to string with no different result.
At this point I have ne clue.
Thanks for your help!
Code
#!/usr/bin/env python3.7
import sys
import subprocess
import os
IP_NETWORK = '192.168.254.10'
IP_DEVICE = '192.168.254.194'
proc = subprocess.Popen(['ping', IP_NETWORK], stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
# print(line)
if not line:
break
connected_ip = line.decode('utf-8').split()[3].replace(':','')
proc2 = subprocess.Popen(['ip', 'neigh', 'show', 'to', connected_ip], stdout=subprocess.PIPE)
ip_macAddr = proc2.stdout.readline().decode('utf-8').split()
print(connected_ip)
# print(type(ip_macAddr))
print(ip_macAddr[0])
Error
IndexError: list index out of range
edit 1
The output of print(ip_macAddr) would be: ['192.168.254.10', 'dev', 'wlp61s0', 'lladdr', '88:88:a1:b2:c3:88', 'REACHABLE']
edit 2
print(type(ip_macAddr)) returns <class 'list'>
edit 3
Traceback
Traceback (most recent call last):
File "./device-connected-to-network.py", line 34, in <module>
print(ip_macAddr[0])
IndexError: list index out of range
This would happen using ANY index and I also tried using:
print(ip_macAddr[0]) or ip_macAddr = proc2.stdout.split()[0]
Again; same result using ANY index # and print(type(ip_macAddr)) returns type 'list'.
Another exemple;
The output of print(ip_macAddr)is ['192.168.254.10', 'dev', 'wlp61s0', 'lladdr', '74:83:c2:d2:a4:12', 'REACHABLE'].
So, if I do (in another file) - using index #4:
a = ['192.168.254.10', 'dev', 'wlp61s0', 'lladdr', '11:22:a1:b2:c3:33', 'REACHABLE']
print(type(a))
print(a[4])
The output is:
<class 'list'>
11:22:a1:b2:c3:33
So works as expected in this context, but not within the actual program.

split()[3] will fail with an IndexError on any line which has less than four whitespace-separated fields.
A common fix is to simply skip lines with fewer fields.
Similarly, if ip produces an empty string, splitting it produces an empty list, which you can't get the first element of.
You can also simplify your code by passing in text=True, and use subprocess.run() where you don't particularly need Popen(). I have also removed unused imports.
#!/usr/bin/env python3.7
import subprocess
IP_NETWORK = '192.168.254.10'
IP_DEVICE = '192.168.254.194'
proc = subprocess.Popen(['ping', IP_NETWORK],
stdout=subprocess.PIPE, text=True)
while True:
line = proc.stdout.readline()
# print(line)
if not line:
break
fields = line.split()
if len(fields) < 4:
continue
connected_ip = fields[3].replace(':','')
proc2 = subprocess.run(
['ip', 'neigh', 'show', 'to', connected_ip],
capture_output=True, text=True)
result = proc2.stdout
if not result:
continue
ip_macAddr = result[0]
print(connected_ip)
# print(type(ip_macAddr))
print(ip_macAddr)
Even better, trap and report any exceptions.
try:
proc2 = subprocess.run(
['ip', 'neigh', 'show', 'to', connected_ip],
capture_output=True, text=True,
check=True) # important
result = proc2.stdout
except CalledProcessError as exc:
print('`ip\' for', connected_ip, 'failed:', exc)
continue

Python - Reading Powershell script output, using subprocess, want to receive stdout line by line

Been Trying to read stdout from a Powershell script that runs for a little bit, generating output based on the number of computers it's pinging.
Trying to get the data to stream into the text box, but after all I've tried, I only seem to be able to get it to give all the output at once.
Been trying not to use subprocess.communicate(), as it also seems to give all the output at once.
Here is the Code:
from tkinter import *
import os
from subprocess import Popen, PIPE
window = Tk()
window.title( 'PowerShell Script Temp' )
frame = Frame(window)
fldrPath = r"C:/Users/firstname.lastname/Downloads/Powershell Development/Monthly Scans/"
listbox = Listbox(frame)
listbox.configure(width=50)
for name in os.listdir(fldrPath):
listbox.insert('end', name)
def selection():
fileList = listbox.curselection()
for file in fileList:
os.chdir(fldrPath)
# Right here is the problematic section
with Popen(["powershell.exe", '-File', fldrPath + '\\' + listbox.get(file)], stdout=PIPE, bufsize=1,
universal_newlines=True) as p:
for line in p.stdout:
output.insert('end', line)
print(line, end='')
output = Text(window, width=75, height=6, wrap=WORD, background="white")
btn = Button(frame, text='Run Script', command=selection)
btn.pack(side=RIGHT, padx=5)
listbox.pack(side=LEFT)
frame.pack(padx=30, pady=30)
output.pack(fill=BOTH, expand=1, padx=5, pady=5)
window.mainloop()
Doesn't filter line-by-line - just vomits it all after the PS script has finished.
I've tried quite a few other things:
Attempt 1
proc = subprocess.Popen(["powershell.exe", '-File', fldrPath + '\\' + listbox.get(file)], shell=True, stdout=PIPE)
while proc.poll() is None:
data = proc.stdout.readline() # Alternatively proc.stdout.read(1024)
print(data)
text_box.insert('end', data)
Attempt 2
outpipe = subprocess.Popen(["powershell.exe", '-File', fldrPath + '\\' + listbox.get(file)],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()[0]
text_box.insert("end", outpipe)
Attempt 3
with Popen(["powershell.exe", '-File', fldrPath + '\\' + listbox.get(file)], stdout=PIPE, bufsize=1,
universal_newlines=True) as p, StringIO() as buf:
for line in p.stdout:
print(line, end='')
buf.write(line)
outpipe = buf.getvalue()
output.insert('end', outpipe)
Attempt 4
proc = subprocess.Popen(["powershell.exe", '-File', fldrPath + '\\' + listbox.get(file)], shell=True,
stdout=subprocess.PIPE)
while proc.poll() is None:
p = proc.stdout.readline()
output.insert('end', p)
I've actually attempted far more, but can't seem to remember them. Lots of fors and withs...I'm a bit tired of it at this point 😔. I can get this thing to Print() in a manner that I desire, but not stream into the text box; almost like .insert() is too slow.
If someone could lead me in the right direction, I'd greatly appreciate it. Tried almost every similar thread on here that I could manage, just doesn't seem to be doing the trick.

Trying to get the data to stream into the text box, but after all I've
tried, I only seem to be able to get it to give all the output at
once.
This code here:
def selection():
fileList = listbox.curselection()
for file in fileList:
os.chdir(fldrPath)
# Right here is the problematic section
with Popen(["powershell.exe", '-File', fldrPath + '\\' + listbox.get(file)], stdout=PIPE, bufsize=1,
universal_newlines=True) as p:
for line in p.stdout:
output.insert('end', line)
print(line, end='')
is executed and after its finished you can see what it has done. To update it you could use the after method of tkinter and check for changes in the given time.
Like in this exampel here.
The tkinter mainloop works like this
Start
|
|<----------------------------------------------------------+
v ^
Do I have No[*] Calculate how Sleep for at |
work to do? -----> long I may sleep -----> most that much --->|
| time |
| Yes |
| |
v |
Do one callback |
| |
+-----------------------------------------------------------+
Another suggestion would be to use another thread.
Also you should read that answer here for .communicate().

Please try following :
command1 = "please type command whose output you want to achieve"
output1 = os.popen(command1).read()
print(output1)
Not sure man but i hope it works out for you

subprocess open file and pipe awk command [duplicate]

This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 5 years ago.
This is my input file format:
#SRR2056440.1 1 length=100
TGTAGGTCTGAGCAGCTTGTCCTGGCTGTGTCCATGTCAGAGCAACGGCCCAAGTCTGGGTCTGGGGGGGAAGGTGTCATGGAGCCCCCTACGATTCCCA
+SRR2056440.1 1 length=100
BCBFFFEFHHHHHJJJJJJIJJJJJJJJIJHHIJJIIJJJJJIJJIJJJJJJJJFHIJJJHHHHHHFDDDBDDD>>ACDEDDDDDDDDDDDDDDDDDEDD
#SRR2056440.2 2 length=100
CTGCCGCCACCGCAGCAGCCACAGGCAGAGGAGGACGAGGACGACTGGGAATCGTAGGGGGCTCCATGACACCTTCCCCCCCAGACCCAGACTTGGGCCA
+SRR2056440.2 2 length=100
CCCFFFFFHHHHHJJJJJJJJJJJIJIJIGJGGIGGJIJJEHFEDDDDDDDDDDABDDDDDDDDDDDDDDADDDDDDDDDDDCDDDDDDBBDDCDDBDD#
#SRR2056440.3 3 length=100
TCTGCCGCCACCGCAGCAGCCACAGGCAGAGGAGGACGAGGACGACTGGGAATCGTAGGGGGCTCCATGACACCTTCCCCCCCAGACCCAGACTTGGGCC
+SRR2056440.3 3 length=100
CCCFFFFFHGHHHJJJJJIJJJJJJIJJIJJJIJJIIIGIJ<CDBCDDDDDDDDDDDDDDDDDDDDDDDDDDDDDCDDDDDDDDDDDDDDDDDDCDCBDD
This is the command I want to execute:
cat input.fq | awk 'NR%4==2{sum+=length($0);nr++;sumsq+=length($0)*length($0)}END{printf"%.1f\t%.1f\n",sum/nr,sqrt(sumsq/nr-(sum/nr)**2)}'
And the output of the command:
100.0 0.0
I want to execute that command inside a python script using subprocess. I have done several attempts but I can't figure out, this is my last try:
awk_comm = r"""'NR%4==2{sum+=length($0);nr++;sumsq+=length($0)*length($0)}END{printf"%.1f\t%.1f\n",sum/nr,sqrt(sumsq/nr-(sum/nr)**2)}'"""
cmd = ['cat', 'input.fq', '|', 'awk', awk_comm]
p2 = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
out1, err = p2.communicate()
EDIT:
I can't see any error in the output. It gets stuck, running forever.

The following works for me.
>>> awk_comm = r"""cat input.fq | awk 'NR%4==2{sum+=length($0);nr++;sumsq+=length($0)*length($0)}END{printf"%.1f\t%.1f\n",sum/nr,sqrt(sumsq/nr-(sum/nr)**2)}'"""
>>> p2 = subprocess.Popen(awk_comm, stdout=subprocess.PIPE,shell=True)
>>> res = p2.communicate()
>>> res
('100.0\t0.0\n', None)

There's no point to shell=True here. Just set up your subprocess.Popen object to do everything you'd otherwise use the shell for:
# the original awk code, with whitespace added for readability
awk_command = r"""
NR%4==2 {
sum+=length($0);
nr++;
sumsq+=length($0)*length($0)
}
END {
printf "%.1f\t%.1f\n", sum/nr, sqrt(sumsq/nr-(sum/nr)**2)
}
"""
p2 = subprocess.Popen(
['awk', awk_command],
stdin=open('input.fq', 'r'), # pass a file handle to input.fq directly on awk's stdin
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out1, err = p2.communicate()

By default, Python doesn't use the shell to run commands...but pipes are evaluated by the shell!! You need to pass shell=True:
p2 = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

You can use the commands module to achieve this:
import commands
awk_comm = r"""'NR%4==2{sum+=length($0);nr++;sumsq+=length($0)*length($0)}END{printf"%.1f\t%.1f\n",sum/nr,sqrt(sumsq/nr-(sum/nr)**2)}'"""
p1 = commands.getoutput('cat input.fq | awk ' + awk_comm)
print p1
Hope this helps

None in redirected stdout

I have this part of following code:
p = subprocess.Popen(['C:/Python27/python.exe', '-m', 'robot', '-d', logs_directory, input_file], stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
output = sys.stdout.write(line)
#sys.stdout.write(line)
print "\n\n"
print "************************************************"
print output
print "\n\n"
print "************************************************"
print "\n\n"
But 'output' variable in console shows 'none'
************************************************
None
************************************************
What i am doing wrong here?

sys.stdout.write(line) returns the number of characters written on stdout.
You could write your loop like this:
output = ""
for line in iter(p.stdout.readline, ''):
output += line
sys.stdout.write(line)
imho you can also remove the sys.stdout.write(line) part. But I don't know what you are meant to do with it so...

Try output += line instead of output = sys.stdout.write(line)

You can also try function check_output.
Here is an example:
>>> subprocess.check_output(["echo", "Hello World!"])
'Hello World!\n'
You can try:
cmd = ['C:/Python27/python.exe', '-m', 'robot', '-d', logs_directory, input_file]
output = subprocess.check_output(cmd)
print output
In most of the cases, it can do the job simply.
For details, you may want to reference to subprocess.check_output

grep: write error: Broken pipe with subprocess

I get couple of grep:write errors when I run this code.
What am I missing?
This is only part of it:
while d <= datetime.datetime(year, month, daysInMonth[month]):
day = d.strftime("%Y%m%d")
print day
results = [day]
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+"*.txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt", shell=True, stdout=subprocess.PIPE, )
output1=first.communicate()[0]
d += delta
day = d.strftime("%Y%m%d")
second=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+"*.txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt", shell=True, stdout=subprocess.PIPE, )
output2=second.communicate()[0]
articleList = (output1.split('\n'))
articleList2 = (output2.split('\n'))
results.append( len(articleList)+len(articleList2))
w.writerow(tuple(results))
d += delta

When you do
A | B
in a shell, process A's output is piped into process B as input. If process B shuts down before reading all of process A's output (e.g. because it found what it was looking for, which is the function of the -l option), then process A may complain that its output pipe was prematurely closed.
These errors are basically harmless, and you can work around them by redirecting stderr in the subprocesses to /dev/null.
A better approach, though, may simply be to use Python's powerful regex capabilities to read the files:
def fileContains(fn, pat):
with open(file) as f:
for line in f:
if re.search(pat, line):
return True
return False
first = []
for file in glob.glob(monthDir +"/"+day+"*.txt"):
if fileContains(file, 'Algeria|Bahrain') and fileContains(file, 'Protest|protesters'):
file.append(first)

To find the files matching two patterns, the command structure should be:
grep -l pattern1 $(grep -l pattern2 files)
$(command) substitutes the output of the command into the command line.
So your script should be:
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' $("+ grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt)", shell=True, stdout=subprocess.PIPE, )
and similarly for second

If you are just looking for whole words, you could use the count() member function;
# assuming names is a list of filenames
for fn in names:
with open(fn) as infile:
text = infile.read().lower()
# remove puntuation
text = text.replace(',', '')
text = text.replace('.', '')
words = text.split()
print "Algeria:", words.count('algeria')
print "Bahrain:", words.count('bahrain')
print "protesters:", words.count('protesters')
print "protest:", words.count('protest')
If you want more powerful filtering, use re.

Add stderr args in the Popen function based on the python version the stderr value will change. This will support if the python version is less than 3
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+".txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+".txt", shell=True, stdout=subprocess.PIPE, stderr = subprocess.STDOUT)