I try to get the output of a curl command by using the python subprocess. But the output is empty. Below is my source code:
def validateURL(url):
p = subprocess.Popen("curl",
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
shell = False)
p.stdin.write("http://validator.w3.org/check?uri=" + url + "\n")
p.stdin.close()
stdout_data = p.stdout.read()
print stdout_data
result = re.findall("Error", stdout_data)
print result # empty here
if (len(result) != 0):
return 'ERR'
else:
return 'OK'
Why?
PS: I run this piece of code on my mac os and I use Python 2.7.
Drop the stderr = subprocess.PIPE,, and see the error message printed by curl. Act accordingly to fix it.
One possible reason is that the URL should be specified as a command-line argument, and not on stdin:
p = subprocess.Popen(("curl", "http://..."), stdout=subprocess.PIPE)
You were passing data to Popen after it executed the command.
Try this:
def validateURL(url):
p = subprocess.Popen(["curl", "http://validator.w3.org/check?uri=" + url + "\n"],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
shell = False)
stdout_data = p.stdout.read()
print stdout_data
result = re.findall("Error", stdout_data)
print result # empty here
if (len(result) != 0):
return 'ERR'
else:
return 'OK'
You're not specifying the URL on the command line, so curl is printing an error message and exiting. Thus, there is no output on stdout. You're trying to send the URL on standard input, but curl does not work that way.
Instead, try:
p = subprocess.Popen(["curl", "http://validator.w3.org/check?uri=" + url],
stdout=subprocess.PIPE, shell=False)
Or, you know, just use urllib2 (or requests) and do it in native Python instead of shelling out to curl and dealing with all that plumbing.
Related
I want to check the DNS value from my system.
If the command goes wrong, the error should be stored in a different variable.
This is what I have so far:
proc = subprocess.Popen(['echo', '"to stdout"'], stdout=subprocess.PIPE,)
stdout_value = proc.communicate()
print '\tstdout:', repr(stdout_value)
subprocess.call('echo #user', shell=True)
#subprocess.check_call('echo #HOME', shell=True)
You should try this :
It captures errorcode, stdout and stderr from a command you passed as an argument :
import shlex
from subprocess import Popen, PIPE
def get_exitcode_stdout_stderr(cmd):
"""
Execute the external command and get its exitcode, stdout and stderr.
"""
args = shlex.split(cmd)
proc = Popen(args, stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
exitcode = proc.returncode
#
return exitcode, out, err
cmd = "..." # arbitrary external command, e.g. "python mytest.py"
exitcode, out, err = get_exitcode_stdout_stderr(cmd)
For your need, I think you can use a python module to get what you want instead of using the bash cmd line. For example, to get your fully qualified domain name you can use :
socket.getfqdn()
I have a working code to retrieve the data from stdout of an ssh connection in chunks, check for a pattern match at the end of each chunk and send appropriate response back via stdin.
ssh = paramiko.SSHClient()
...
transport = ssh.get_transport()
session = transport.open_session()
session.set_combine_stderr(True)
session.get_pty()
stdin = session.makefile('wb', -1)
stdout = session.makefile('rb', -1)
session.exec_command(cmd)
for chunk in iter(lambda: session.recv(9999), ""):
if re.search('Password: $', chunk):
stdin.write(sudo_pw + '\n')
stdin.flush()
output += chunk
Now I have a subprocess using which I execute commands locally like below:
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
(output, err) = p.communicate()
How can I implement the exact same logic of analyzing the output in chunks and send appropriate response via stdin? I am looking for a solution without using the pexpect.
Found a solution which works the same way. In Linux, sudo prompt still can't be caught because it reads and writes directly to the detected terminal device. sudo -S works just fine.
master, slave = pty.openpty()
p = Popen(cmd, shell=True, stdout=slave, stdin=slave, stderr=STDOUT)
q = select.poll()
q.register(master,select.POLLIN)
output = ""
while True:
if not q.poll(0):
time.sleep(0.1)
if p.poll() is not None:
time.sleep(0.1)
while q.poll(0):
chunk = os.read(master, 9999)
output += chunk
break
else:
chunk = os.read(master, 9999)
output += chunk
if re.search('Password: $', chunk):
os.write(master, sudo_pw + '\n')
rc = p.returncode
Trying to write a script that checks a directory for files which then uses the names of the files found to insert in to a subprocess command as shown below:
for filename in os.listdir('/home/dross/python/scripts/var/running/'):
print(str(filename))
cmd = 'app_query --username=dross --password=/home/dross/dross.txt "select row where label = \'Id: ' + filename + '\' SHOW status"'
print(cmd)
query = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
query.wait()
If I run the command manually from the command line there are 2 possible values returned "Error:No result" or "True"
When the "Error: No result" condition is true the script returns the same however when the "True" condition is present nothing is returned.
If the result of the print statement is copied and pasted in to the os command line it runs and returns "True"
What could be the deception I am seeing here ?
Is there a better approach to achieve what I am trying to do ?
You seem to be missing a call to .communicate(), to read the results of the command through the pipe.
In your original query = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) then anything send to stderr will be displayed on the screen, which seems to be what's happening for your error message. Anything sent to stdout will be sent to the pipe, ready for reading with communicate()
Some experimenting, showing that you won't see what's written to the subprocess.PIPE channels unless you communicate with the command you've run, and that stderr will display to the terminal if it's not redirected:
>>> import subprocess
>>> query = subprocess.Popen('echo STDERR 1>&2', shell=True, stdout=subprocess.PIPE)
STDERR
>>> query.wait()
0
>>> print(query.communicate())
('', None)
>>> query = subprocess.Popen('echo STDERR 1>&2', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> query.wait()
0
>>> print(query.communicate())
('', 'STDERR\n')
>>> query = subprocess.Popen('echo STDOUT', shell=True, stdout=subprocess.PIPE)
>>> query.wait()
0
>>> print(query.communicate())
('STDOUT\n', None)
So, to use your code from the question, you want something like this:
for filename in os.listdir('/home/dross/python/scripts/var/running/'):
print(filename) # print can convert to a string, no need for str()
cmd = 'app_query --username=dross --password=/home/dross/dross.txt "select row where label = \'Id: ' + filename + '\' SHOW status"'
print(cmd)
query = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
query.wait()
output, error = query.communicate()
print("stdout: {}".format(output))
print("stderr: {}".format(error))
I am using subprocess module to interact with output of the linux commands. below is my code.
import subprocess
import sys
file_name = 'myfile.txt'
p = subprocess.Popen("grep \"SYSTEM CONTROLLER\" "+ file_name, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
print output.strip()
p = subprocess.Popen("grep \"controller\|worker\" "+ file_name, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
lines = output.rstrip().split("\n")
print lines
My program hangs while executing second subprocess i.e.
p = subprocess.Popen("grep \"controller\|worker\""+ file_name,stdout=subprocess.PIPE, shell=True)
I got to know that the reason of process hang is buffer redirected to subprocess.PIPE is getting filled, which blocks the process from writing further.
I want to know if there is any way to avoid the buffer full situation so that my program keeps on executing without any hang issue ?
The actual issue is that there is a whitespace missing between the pattern and the filename and therefore grep waits for input on the standard input (stdin).
"buffer full" (.communicate() is not susceptible) or p.stdout.read() (it fixes nothing: it loads the output into memory and unlike .communicate() it fails if more than one pipe is used) are a red herring here.
Drop shell=True and use a list argument for the command:
#!/usr/bin/env python
from subprocess import Popen, PIPE
p = Popen(["grep", r"controller\|worker", file_name], stdout=PIPE)
output = p.communicate()[0]
if p.returncode == 0:
print('found')
elif p.returncode == 1:
print('not found')
else:
print('error')
As it says at https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate:
Note: The data read is buffered in memory, so do not use this method
if the data size is large or unlimited.
Instead, use the file objects to read the text as it is produced:
output = p.stdout.read()
As long as no other pipes (e.g. stderr) fill up while you are reading, the process shouldn't be blocked.
I am trying to use subprocess module with Popen to fetch log from a specified URL, However, I am not able to fetch the log and the program returns me a blank.
I have been using the below mentioned code:
import subprocess
url = r'C:\project\dummy\pro'
mycmd = ['svn', 'log', url]
log = subprocess.Popen(mycmd, shell=True,
stdin = subprocess.PIPE, stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
result = log.wait()
out1, err = log.communicate()
print out1
I need the output string to use as next part of the program. Any help would be appreciated.
Try without shell=True:
log = subprocess.Popen(mycmd,
stdin = subprocess.PIPE, stdout = subprocess.PIPE,
stderr = subprocess.PIPE)