I have the following:
cmd = "ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'".split(' ')
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print out
When I run the command in the console (outside of python), I get the desired output. Running this above code in python prints a blank line. I am assuming there is something up with the cmd (specifically the | operator) but I can't be sure.
I need to achieve this with the standard Python 2.6.6 install (no additional modules)
You need to use a single call to Popen() for each piece of the original command, as connected by the pipe, as in
import subprocess
p1 = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "java -jar"], stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p3 = subprocess.Popen(["grep", "-v", "grep"], stdin=p2.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p4 = subprocess.Popen(["awk", "//{print $2}"], stdin=p3.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p4.communicate()
print out
The subprocess documentation has an in-depth discussion.
Popen by default only executes executables, not shell command lines.
When you pass the list of arguments to Popen they should call one executable with its arguments:
import subprocess
proc = subprocess.Popen(['ps', 'aux'])
Also note that you should not use str.split to split a command, because:
>>> "ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'".split(' ')
['ps', 'aux', '|', 'grep', "'java", "-jar'", '', '', '', '', '|', 'grep', '-v', 'grep', '|', 'awk', "'//{print", "$2}'"]
Note how:
The arguments that were quoted (e.g. 'java -jar') are splitted.
If there is more than one consecutive space you get some empty arguments.
Python already provides a module that knows how to split a command line in a reasonable manner, it's shlex:
>>> shlex.split("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'")
['ps', 'aux', '|', 'grep', 'java -jar', '|', 'grep', '-v', 'grep', '|', 'awk', '//{print $2}']
Note how quoted arguments were preserved, and multiple spaces are handled gracefully. Still you cannot pass the result to Popen, because Popen will not interpret the | as a pipe by default.
If you want to run a shell command line (i.e. use any shell feature such as pipes, path expansion, redirection etc.) you must pass shell=True. In this case you should not pass a list of strings as argumento to Popen, but only a string that is the complete command line:
proc = subprocess.Popen("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True)
If you pass a list of strings with shell=True its meaning is different: the first element should be the complete command line, while the other elements are passed as options to the shell used. For example on my machine the default shell (sh) has an -x option that will display on stderr all the processes that gets executed:
>>> from subprocess import Popen
>>> proc = Popen(['ps aux | grep python3', '-x'], shell=True)
>>>
username 7301 0.1 0.1 39440 7408 pts/9 S+ 12:57 0:00 python3
username 7302 0.0 0.0 4444 640 pts/9 S+ 12:58 0:00 /bin/sh -c ps aux | grep python3 -x
username 7304 0.0 0.0 15968 904 pts/9 S+ 12:58 0:00 grep python3
Here you can see that a /bin/sh was started that executed the command ps aux | python3 and with an option of -x.
(This is all documented in the documentation for Popen).
This said, one way to achieve what you want is to use subprocess.check_output:
subprocess.check_output("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True)
However this isn't available in python<2.7 so you have to use Popen and communicate():
proc = subprocess.Popen("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True, stdout=subprocess.PIPE)
out, err = proc.communicate()
The alternative is to avoid using shell=True (which is generally a very good thing, since shell=True introduces some security risks) and manually write the pipe using multiple processes:
from subprocess import Popen, PIPE
ps = Popen(['ps', 'aux'], stdout=PIPE)
grep_java = Popen(['grep', 'java -jar'], stdin=ps.stdout, stdout=PIPE)
grep_grep = Popen(['grep', '-v', 'grep'], stdin=grep_java.stdout, stdout=PIPE)
awk = Popen(['awk', '//{print $2}'], stdin=grep_grep.stdout, stdout=PIPE)
out, err = awk.communicate()
grep_grep.wait()
grep_java.wait()
ps.wait()
Note that if you don't care for the standard error you can avoid specifying it. It will then inherit the one of the current process.
There is just one command in the shell pipeline which can't be easily replaced by Python code. So you can start several external processes and connect their inputs and outputs, but I would just start the ps aux and add some Python code to filter and extract the desired data:
from subprocess import PIPE, Popen
def main():
process = Popen(['ps', 'aux'], stdout=PIPE)
pids = [
line.split(None, 2)[1] for line in process.stdout if 'java -jar' in line
]
process.wait()
print '\n'.join(pids)
if __name__ == '__main__':
main()
Related
Take a simple command in bash
cmd='ls -l | wc -l'
I understand that we can run this command several ways using subprocess call/check_output/ communicate. The problem arise for me if initial commands do not work or fail for some reason. Like [replace ls with lsx].
cmd='lsx -l | wc -l'
In this case how can we capture error or we just have to process the output to figure out? Here is what I tried.
import subprocess
>>> subprocess.call('lsx -l | wc -l', shell=True)
/bin/sh: lsx: command not found
0
0
>>> subprocess.check_output('lsx -l | wc -l', shell=True)
/bin/sh: lsx: command not found
b' 0\n'
It seems that error code are still 0 in above two commands.
I also tried https://docs.python.org/3.5/library/subprocess.html#replacing-shell-pipeline but cannot figure out how to get error code for first process.
If you specify bash instead of sh, you can set the pipefail option to return a nonzero exit status if any part of a pipeline fails:
subprocess.check_output(['bash', '-c', 'set -o pipefail; lsx -l | wc -l'])
That said, you can certainly avoid using shell=True altogether:
import subprocess
try:
p1 = subprocess.Popen(['ls', '--invalid-argument'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['wc', '-l'], stdin=p1.stdout, stdout=subprocess.PIPE)
wc_stdout = p2.communicate()[0]
if p1.wait() != 0 or p2.wait() != 0:
raise RuntimeError("Something failed!")
except FileNotFoundError as ex:
raise RuntimeError("Something failed, because we couldn't find an executable!") from ex
You can assign stderr to PIPE. Consider this example:
>>> from subprocess import PIPE, Popen
>>> sub = Popen('lsx -l | wc -l', shell=True, stderr=PIPE, stdout=PIPE)
>>> output, error_output = sub.communicate()
>>> error_output
b'/bin/sh: 1: lsx: not found\n'
>>> output
b'0\n'
>>> sub = Popen('ls -l | wc -l', shell=True, stderr=PIPE, stdout=PIPE)
>>> output, error_output = sub.communicate()
>>> error_output
b''
I have a code so far that filters out everything except the gateway IP (route -n | awk '{if($4=="UG")print $2}'), but I'm trying to figure out how to pipe this to a variable in Python. Here's what I got:
import shlex;
from subprocess import Popen, PIPE;
cmd = "route -n | grep 'UG[ \t]' | awk '{print $2}'";
gateway = Popen(shlex.split(cmd), stdout=PIPE);
gateway.communicate();
exit_code = gateway.wait();
Any ideas?
NOTE: I'm new at this.
For better or worse, your cmd uses a shell pipeline. To use shell features in subprocess, one must set shell=True:
from subprocess import Popen, PIPE
cmd = "/sbin/route -n | grep 'UG[ \t]' | awk '{print $2}'"
gateway = Popen(cmd, shell=True, stdout=PIPE)
stdout, stderr = gateway.communicate()
exit_code = gateway.wait()
Alternatively, one could keep shell=False, eliminate the pipeline, and do all the string processing in python:
from subprocess import Popen, PIPE
cmd = "/sbin/route -n"
gateway = Popen(cmd.split(), stdout=PIPE)
stdout, stderr = gateway.communicate()
exit_code = gateway.wait()
gw = [line.split()[1] for line in stdout.decode().split('\n') if 'UG' in line][0]
Because of the vagaries of shell processing, and unless there is a specific need, it is probably best to avoid shell=True.
I try to measure CPU usage via top and python:
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
proc = subprocess.Popen("top -c -b -n 1 | grep /usr/local/bin/wineserver | grep -v grep | awk '{print $9}'", shell=True, stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
print len(stdout)
print len(stderr)
Output:
0
0
If I run the cmd via shell I get:
54
It seems the piping is the issue but I am not sure.
Solution:
os.system("top -c -b -n 1 | grep /usr/local/bin/wineserver | grep -v grep | awk '{print $9}' > top")
stdout = open("top").read().strip("\n")
When run interactively, top limits its display to your screen width. When run through Popen with stdout=PIPE, top is not running in a terminal and reverts to its default column width. This can be changed with an environment variable.
You could ditch the shell completely and process with python:
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
import os
myenv = os.environ.copy()
myenv["COLUMNS"] = "512"
proc = subprocess.Popen(["top", "-c", "-b", "-n", "1"], stdout=PIPE, stderr=PIPE, env=myenv)
for line in proc.stdout:
columns = print line.strip().split()
if columns[-1] == '/usr/local/bin/wineserver':
print columns
proc.wait()
Alternately, you can get CPU information through ps and use its filtering and output format specifiers to just grab the information you want. Below I use filters to display CPU and command line for "wineserver" only.
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
proc = subprocess.Popen(["ps", "-ww", "--no-headers", "-C", "wineserver", "-o", "pcpu args"],
stdout=PIPE, stderr=PIPE, env=myenv)
for line in proc.stdout:
pcpu, command = print line.strip().split(" ", 1)
print pcpu, command
proc.wait()
ip = subprocess.Popen(["/sbin/ifconfig $(/sbin/route | awk '/default/ {print $8}') | grep \"inet addr\" | awk -F: '{print $2}' | awk \'{print $1}\'"], stdout=subprocess.PIPE)
I am not sure where to put the commas to separate them to use this command using subprocess.Popen. Does anyone know?
You are using shell features (the pipe) so instead of splitting the command, you should pass it as a single string (not a list) with shell=True
ip = subprocess.Popen("/sbin/ifconfig $(/sbin/route | awk '/default/ {print $8}') | grep \"inet addr\" | awk -F: '{print $2}' | awk \'{print $1}\'",
shell=True,
stdout=subprocess.PIPE)
Here's what I would recommend.
Create a file with this contents - call it 'route-info' and make it executable:
#!/bin/sh
/sbin/ifconfig $(/sbin/route | awk '/default/ {print $8}') |
grep "inet addr" |
awk -F: '{print $2}' |
awk '{print $1}'
In your python program, use:
ip = subprocess.Popen(["/path/to/route-info"], stdout=subprocess.PIPE)
Then you don't have to worry about quoting characters and you can independently test the route-info script to make sure it is working correctly.
The script route-info doesn't take any command line arguments, but if it did this is how you would pass them:
ip = subprocess.Popen(["/path/to/route-info", arg1, arg2, ...], stdout=subprocess.PIPE)
Quoting the official documentation of subprocess.Popen here
It may not be obvious how to break a shell command into a sequence of
arguments, especially in complex cases. shlex.split() can illustrate
how to determine the correct tokenization for args:
import shlex, subprocess
command_line = input()
args = shlex.split(command_line)
print(args)
p = subprocess.Popen(args) # Success!
shlex is included in standard library so you need not to install it.
Writing it in a single line like str.split() should look like:
import shlex
import subprocess
command = "ls -l"
proc = subprocess.Popen(shlex.split(command) , stdout = subprocess.PIPE , stderr = subprocess.PIPE)
output , errors = proc.communicate()
print(output , errors)
How could I run this code using subprocess module?
commands.getoutput('sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"')
I've tried this but it doesn't work at all
out_1 = subprocess.Popen(('sudo', 'blkid'), stdout=subprocess.PIPE)
out_2 = subprocess.Popen(('grep', 'uuid'), stdin=out_1.stdout, stdout=subprocess.PIPE)
out_3 = subprocess.Popen(('cut', '-d', '" "', '-f', '1'), stdin=out_2.stdout, stdout=subprocess.PIPE)
main_command = subprocess.check_output(('tr', '-d', '":"'), stdin=out_3.stdout)
main_command
Error: cut: the delimiter must be a single character
from subprocess import check_output, STDOUT
shell_command = '''sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"'''
output = check_output(shell_command, shell=True, stderr=STDOUT,
universal_newlines=True).rstrip('\n')
btw, it returns nothing on my system unless grep -i is used. In the latter case it returns devices. If it is your intent then you could use different command:
from subprocess import check_output
devices = check_output(['sudo', 'blkid', '-odevice']).split()
I'm trying not to use shell=True
It is ok to use shell=True if you control the command i.e., if you don't use user input to construct the command. Consider the shell command as a special language that allows you to express your intent concisely (like regex for string processing). It is more readable then several lines of code that do not use shell:
from subprocess import Popen, PIPE
blkid = Popen(['sudo', 'blkid'], stdout=PIPE)
grep = Popen(['grep', 'uuid'], stdin=blkid.stdout, stdout=PIPE)
blkid.stdout.close() # allow blkid to receive SIGPIPE if grep exits
cut = Popen(['cut', '-d', ' ', '-f', '1'], stdin=grep.stdout, stdout=PIPE)
grep.stdout.close()
tr = Popen(['tr', '-d', ':'], stdin=cut.stdout, stdout=PIPE,
universal_newlines=True)
cut.stdout.close()
output = tr.communicate()[0].rstrip('\n')
pipestatus = [cmd.wait() for cmd in [blkid, grep, cut, tr]]
Note: there are no quotes inside quotes here (no '" "', '":"'). Also unlike the previous command and commands.getoutput(), it doesn't capture stderr.
plumbum provides some syntax sugar:
from plumbum.cmd import sudo, grep, cut, tr
pipeline = sudo['blkid'] | grep['uuid'] | cut['-d', ' ', '-f', '1'] | tr['-d', ':']
output = pipeline().rstrip('\n') # execute
See How do I use subprocess.Popen to connect multiple processes by pipes?
pass your command as one string like this:
main_command = subprocess.check_output('tr -d ":"', stdin=out_3.stdout)
if you have multiple commands and if you want to execute one by one, pass them as list:
main_command = subprocess.check_output([comand1, command2, etc..], shell=True)