I'm trying to execute a command using Popen and search through the output using a regex to find a UserID. When I run the code below and have if reg_ex.search(line) code block commented out, the command runs successfully (although I don't have my userID). As soon as I uncomment out the regex code, the command itself fails.
Any ideas why this regex if statement makes the command itself fail when I'm just iterating through output and any idea how to get the userId I need from the output? I've already confirmed the regex itself works when run separately (although for those that want it, the output line for regex will look like this u'userId': u'userID_in–question'}.
import re
import subprocess
#command I want to run
command = "add-user " + FirstName + " " + LastName + " " + Email + " " + Phone
#Regex to find userId in output
reg_ex_find = re.compile('userId')
reg_ex_replace = re.compile("(""u'userId""|\s|:|}|')")
#call command
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
for line in iter(process.stdout.readline, b''):
print line
if reg_ex_find.search(line):
UserId = line
reg_ex_replace.sub('',UserId)
process.stdout.close()
process.wait()
print process.returncode
Related
I have written a Python script which will execute a shell command and print the output to stdout.
The shell command itself connects to a backend service and prints the output (line by line)
Issue: Only the first line of the output is printed by the script. The expected output is a multi line output as shown below:
//+This is line one of output (/something/something)
This is line 2 of output…
This is line 3 of output
…
…
…
Completed in 946 miliseconds
I have tried using the following ways to get the output of the command:
Method 1:
import os
import sys
something = sys.argv[1]
command = "here is the command part 1" + something + "command part 2"
os.system(command)
Output of the command (only the first line is printed):
//+This is line one of output (/something/something)
Method 2:
import sys
import subprocess
something = sys.argv[1]
command = "here is the command part 1" + something + "command part 2"
print subprocess.check_output(command, shell=True)
Output of the command:
//+This is line one of output (/something/something)
Method 3:
import sys
import subprocess
something = sys.argv[1]
command = "here is the command part 1" + something + "command part 2"
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
print output
How can I print the complete output?
Update: Per the suggestions in the comments below, I redirected the output of the command to a file from within the Python script as shown below:
import os
import sys
something = sys.argv[1]
ofile = sys.argv[2]
command = "here is the command part 1" + something + "command part 2 >> " + ofile
os.system(command)
f = open(ofile, "r")
data = f.read()
f.close()
print data
It still prints only the first line of the output.
However, if I redirect the output of the command from the command line to an output file, the complete output is captured.
Thanks.
I want to run a python script file with this ping -t www.google.com command.
So far I did one with the ping www.google.com command which works, but I didn't suceed to do on with the ping -t looping indefinitely.
You can find below my ping.py script:
import subprocess
my_command="ping www.google.com"
my_process=subprocess.Popen(my_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result, error = my_process.communicate()
result = result.strip()
error = error.strip()
print ("\nResult of ping command\n")
print("-" *22)
print(result.decode('utf-8'))
print(error.decode('utf-8'))
print("-" *22)
input("Press Enter to finish...")
I want the command box to stay open when finished. I am using Python 3.7.
If you want to keep the process open and communicate with it all the time, you can use my_process.stdout as input and e.g. iterate over it's lines. With "communicate" you wait until the process completes, which would be bad for an indefinitely running process :)
import subprocess
my_command=["ping", "www.google.com"]
my_process=subprocess.Popen(my_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
print( my_process.stdout.readline() )
EDIT
In this version we use re to only get the "time=xxms" part from the output:
import subprocess
import re
my_command=["ping", "-t", "www.google.com"]
my_process=subprocess.Popen(my_command, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
while True:
line = my_process.stdout.readline() #read a line - one ping in this case
line = line.decode("utf-8") #decode the byte literal to string
line = re.sub("(?s).*?(time=.*ms).*", "\\1", line) #find time=xxms in the string and use only that
print(line)
The following code does not work properly in Python. The problem is that the output does not get redirected to output by using >:
command = toolpath + " " + query + " " + number + " > " + output;
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE);
output = process.communicate()[0];
The command is actually the following if you print it:
./subsetA.pl ./remainingqueries.fasta 100 > ./tmpQuery.fasta
So the perl script subsetA.pl takes two arguments and writes it to stdout which gets redirected to tmpQuery.fasta. But tmpQuery.fasta is empty after the command gets called.
If I run it direct on CLI then it works perfectly.
You don't need the shell's output redirection operator with Popen; that's what the stdout argument is for.
command = [toolpath, query, number]
with open(output) as output_fh:
process = subprocess.Popen(command, stdout=output_fh)
Since you are calling communicate to get the standard output, though, you don't want to redirect the output at all:
command = [toolpath, query, number]
process = subprocess.Popen(command, stdout=subprocess.PIPE)
output = process.communicate()[0]
You can try
command = toolpath + " " + query + " " + number + " > " + output;
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE,Shell=True);
output = process.communicate()[0];
Both current answers don't work!
This works (tested on Python 3.7):
subprocess.Popen(['./my_script.sh arg1 arg2 > "output.txt"'],
stdout=subprocess.PIPE, shell=True)
Note:
You do not need split or array in Popen.
BOTH stdout and shell parameters are needed.
I have a python script which tries to run an external command and look for result of the command. And it needs to use the value 'count=' from the output of the external command
COUNT_EXP = re.compile("count=(.*)")
cmd = [] # external command
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
result = COUNT_EXP.match(line)
if result:
print "count= " + result.group(1)
return int(result.group(1))
When I try to run my script, my external command ("cmd") got execute and I see count=10 in the shell. But why my python can't find that and print out "count= 10? in the 'if' clause above?
p = subprocess.Popen(['python','blah.py'],stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if len(line) != 0:
print "success" #if this code works, expanding to your regex should also work
I wrote the following C program:
#include "stdio.h"
int main() {
printf ("count=199");
return 0;
}
... which I called countOutput.c and the following Python script, modified from yours:
import subprocess, re
COUNT_EXP = re.compile("count=(.*)")
cmd = "./countOutput" # external command
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
result = COUNT_EXP.match(line)
if result:
print "count is equal to " + result.group(1)
... which I called countTest.py, and then ran:
$ python countTest.py
count is equal to 199
... which all works as expected. I'd therefore tend to agree with #kichik in thinking that the external command that you're using may be writing to stderr rather than stdout.
It might be printing that to stderr. Try redirecting that one to PIPE as well and read data from there. You can also append 2>&1 to the end of the command to get stderr redirected to stdout by the shell. You might have to add shell=True for that.
I have a list of IP's that I want to run a whois (using the linux tool whois) against and only see the Country option.
Here is my script:
import os
import time
iplist = open('ips.txt').readlines()
for i in iplist:
time.sleep(2)
print "Country: IP {0}".format(i)
print os.system("whois -h whois.arin.net + {0} | grep Country ".format(i))
So I want to display what IP is being ran, then I just want to see the Country info using grep. I see this error when I run it and the grep is not ran:
sh: -c: line 1: syntax error near unexpected token `|'
sh: -c: line 1: ` | grep Country '
this code below works so it must be an issue with my for loop:
print os.system("whois -h whois.arin.net + {0} | grep Country ".format('8.8.8.8'))
What am I doing wrong? Thank you!!!!
You're not stripping trailing newlines from the lines you read from the file. As a result, you are passing to os.system a string like "whois -h whois.arin.net + a.b.c.d\n | grep Country". The shell parses the string as two commands and complains of "unexpected token |" at the beginning of the second one. This explains why there is no error when you use a hand-made string such as "8.8.8.8".
Add i = i.strip() after the sleep, and the problem will go away.
user4815162342 is correct about the issue you are having, but might I suggest you replace os.system with subprocess.Popen? Capturing the output from the system call is not intuitive.. should you want to result to go anywhere but your screen, you'll likely going to have issues
from subprocess import Popen, PIPE
server = 'whois.arin.net'
def find_country(ip):
proc = Popen(['whois', '-h', server, ip], stdout = PIPE, stderr = PIPE)
stdout, stderr = proc.communicate()
if stderr:
raise Exception("Error with `whois` subprocess: " + stderr)
for line in stdout.split('\n'):
if line.startswith('Country:'):
return line.split(':')[1].strip() # Good place for regex
for ip in [i.strip() for i in open('ips.txt').readlines()]:
print find_country(ip)
Python is awesome at string handling- there should be no reason to create a grep subprocess to pattern match the output of a separate subprocess.
Try sh:
import os
import time
import re
import sh
iplist = open('ips.txt').readlines()
for i in iplist:
time.sleep(2)
print "Country: IP {0}".format(i)
print sh.grep(sh.whois(i, h="whois.arin.net"), "Country")