How could I run this code using subprocess module?
commands.getoutput('sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"')
I've tried this but it doesn't work at all
out_1 = subprocess.Popen(('sudo', 'blkid'), stdout=subprocess.PIPE)
out_2 = subprocess.Popen(('grep', 'uuid'), stdin=out_1.stdout, stdout=subprocess.PIPE)
out_3 = subprocess.Popen(('cut', '-d', '" "', '-f', '1'), stdin=out_2.stdout, stdout=subprocess.PIPE)
main_command = subprocess.check_output(('tr', '-d', '":"'), stdin=out_3.stdout)
main_command
Error: cut: the delimiter must be a single character
from subprocess import check_output, STDOUT
shell_command = '''sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"'''
output = check_output(shell_command, shell=True, stderr=STDOUT,
universal_newlines=True).rstrip('\n')
btw, it returns nothing on my system unless grep -i is used. In the latter case it returns devices. If it is your intent then you could use different command:
from subprocess import check_output
devices = check_output(['sudo', 'blkid', '-odevice']).split()
I'm trying not to use shell=True
It is ok to use shell=True if you control the command i.e., if you don't use user input to construct the command. Consider the shell command as a special language that allows you to express your intent concisely (like regex for string processing). It is more readable then several lines of code that do not use shell:
from subprocess import Popen, PIPE
blkid = Popen(['sudo', 'blkid'], stdout=PIPE)
grep = Popen(['grep', 'uuid'], stdin=blkid.stdout, stdout=PIPE)
blkid.stdout.close() # allow blkid to receive SIGPIPE if grep exits
cut = Popen(['cut', '-d', ' ', '-f', '1'], stdin=grep.stdout, stdout=PIPE)
grep.stdout.close()
tr = Popen(['tr', '-d', ':'], stdin=cut.stdout, stdout=PIPE,
universal_newlines=True)
cut.stdout.close()
output = tr.communicate()[0].rstrip('\n')
pipestatus = [cmd.wait() for cmd in [blkid, grep, cut, tr]]
Note: there are no quotes inside quotes here (no '" "', '":"'). Also unlike the previous command and commands.getoutput(), it doesn't capture stderr.
plumbum provides some syntax sugar:
from plumbum.cmd import sudo, grep, cut, tr
pipeline = sudo['blkid'] | grep['uuid'] | cut['-d', ' ', '-f', '1'] | tr['-d', ':']
output = pipeline().rstrip('\n') # execute
See How do I use subprocess.Popen to connect multiple processes by pipes?
pass your command as one string like this:
main_command = subprocess.check_output('tr -d ":"', stdin=out_3.stdout)
if you have multiple commands and if you want to execute one by one, pass them as list:
main_command = subprocess.check_output([comand1, command2, etc..], shell=True)
Related
Take a simple command in bash
cmd='ls -l | wc -l'
I understand that we can run this command several ways using subprocess call/check_output/ communicate. The problem arise for me if initial commands do not work or fail for some reason. Like [replace ls with lsx].
cmd='lsx -l | wc -l'
In this case how can we capture error or we just have to process the output to figure out? Here is what I tried.
import subprocess
>>> subprocess.call('lsx -l | wc -l', shell=True)
/bin/sh: lsx: command not found
0
0
>>> subprocess.check_output('lsx -l | wc -l', shell=True)
/bin/sh: lsx: command not found
b' 0\n'
It seems that error code are still 0 in above two commands.
I also tried https://docs.python.org/3.5/library/subprocess.html#replacing-shell-pipeline but cannot figure out how to get error code for first process.
If you specify bash instead of sh, you can set the pipefail option to return a nonzero exit status if any part of a pipeline fails:
subprocess.check_output(['bash', '-c', 'set -o pipefail; lsx -l | wc -l'])
That said, you can certainly avoid using shell=True altogether:
import subprocess
try:
p1 = subprocess.Popen(['ls', '--invalid-argument'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['wc', '-l'], stdin=p1.stdout, stdout=subprocess.PIPE)
wc_stdout = p2.communicate()[0]
if p1.wait() != 0 or p2.wait() != 0:
raise RuntimeError("Something failed!")
except FileNotFoundError as ex:
raise RuntimeError("Something failed, because we couldn't find an executable!") from ex
You can assign stderr to PIPE. Consider this example:
>>> from subprocess import PIPE, Popen
>>> sub = Popen('lsx -l | wc -l', shell=True, stderr=PIPE, stdout=PIPE)
>>> output, error_output = sub.communicate()
>>> error_output
b'/bin/sh: 1: lsx: not found\n'
>>> output
b'0\n'
>>> sub = Popen('ls -l | wc -l', shell=True, stderr=PIPE, stdout=PIPE)
>>> output, error_output = sub.communicate()
>>> error_output
b''
I have a code so far that filters out everything except the gateway IP (route -n | awk '{if($4=="UG")print $2}'), but I'm trying to figure out how to pipe this to a variable in Python. Here's what I got:
import shlex;
from subprocess import Popen, PIPE;
cmd = "route -n | grep 'UG[ \t]' | awk '{print $2}'";
gateway = Popen(shlex.split(cmd), stdout=PIPE);
gateway.communicate();
exit_code = gateway.wait();
Any ideas?
NOTE: I'm new at this.
For better or worse, your cmd uses a shell pipeline. To use shell features in subprocess, one must set shell=True:
from subprocess import Popen, PIPE
cmd = "/sbin/route -n | grep 'UG[ \t]' | awk '{print $2}'"
gateway = Popen(cmd, shell=True, stdout=PIPE)
stdout, stderr = gateway.communicate()
exit_code = gateway.wait()
Alternatively, one could keep shell=False, eliminate the pipeline, and do all the string processing in python:
from subprocess import Popen, PIPE
cmd = "/sbin/route -n"
gateway = Popen(cmd.split(), stdout=PIPE)
stdout, stderr = gateway.communicate()
exit_code = gateway.wait()
gw = [line.split()[1] for line in stdout.decode().split('\n') if 'UG' in line][0]
Because of the vagaries of shell processing, and unless there is a specific need, it is probably best to avoid shell=True.
I try to measure CPU usage via top and python:
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
proc = subprocess.Popen("top -c -b -n 1 | grep /usr/local/bin/wineserver | grep -v grep | awk '{print $9}'", shell=True, stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
print len(stdout)
print len(stderr)
Output:
0
0
If I run the cmd via shell I get:
54
It seems the piping is the issue but I am not sure.
Solution:
os.system("top -c -b -n 1 | grep /usr/local/bin/wineserver | grep -v grep | awk '{print $9}' > top")
stdout = open("top").read().strip("\n")
When run interactively, top limits its display to your screen width. When run through Popen with stdout=PIPE, top is not running in a terminal and reverts to its default column width. This can be changed with an environment variable.
You could ditch the shell completely and process with python:
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
import os
myenv = os.environ.copy()
myenv["COLUMNS"] = "512"
proc = subprocess.Popen(["top", "-c", "-b", "-n", "1"], stdout=PIPE, stderr=PIPE, env=myenv)
for line in proc.stdout:
columns = print line.strip().split()
if columns[-1] == '/usr/local/bin/wineserver':
print columns
proc.wait()
Alternately, you can get CPU information through ps and use its filtering and output format specifiers to just grab the information you want. Below I use filters to display CPU and command line for "wineserver" only.
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
proc = subprocess.Popen(["ps", "-ww", "--no-headers", "-C", "wineserver", "-o", "pcpu args"],
stdout=PIPE, stderr=PIPE, env=myenv)
for line in proc.stdout:
pcpu, command = print line.strip().split(" ", 1)
print pcpu, command
proc.wait()
I have the following:
cmd = "ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'".split(' ')
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print out
When I run the command in the console (outside of python), I get the desired output. Running this above code in python prints a blank line. I am assuming there is something up with the cmd (specifically the | operator) but I can't be sure.
I need to achieve this with the standard Python 2.6.6 install (no additional modules)
You need to use a single call to Popen() for each piece of the original command, as connected by the pipe, as in
import subprocess
p1 = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "java -jar"], stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p3 = subprocess.Popen(["grep", "-v", "grep"], stdin=p2.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p4 = subprocess.Popen(["awk", "//{print $2}"], stdin=p3.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p4.communicate()
print out
The subprocess documentation has an in-depth discussion.
Popen by default only executes executables, not shell command lines.
When you pass the list of arguments to Popen they should call one executable with its arguments:
import subprocess
proc = subprocess.Popen(['ps', 'aux'])
Also note that you should not use str.split to split a command, because:
>>> "ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'".split(' ')
['ps', 'aux', '|', 'grep', "'java", "-jar'", '', '', '', '', '|', 'grep', '-v', 'grep', '|', 'awk', "'//{print", "$2}'"]
Note how:
The arguments that were quoted (e.g. 'java -jar') are splitted.
If there is more than one consecutive space you get some empty arguments.
Python already provides a module that knows how to split a command line in a reasonable manner, it's shlex:
>>> shlex.split("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'")
['ps', 'aux', '|', 'grep', 'java -jar', '|', 'grep', '-v', 'grep', '|', 'awk', '//{print $2}']
Note how quoted arguments were preserved, and multiple spaces are handled gracefully. Still you cannot pass the result to Popen, because Popen will not interpret the | as a pipe by default.
If you want to run a shell command line (i.e. use any shell feature such as pipes, path expansion, redirection etc.) you must pass shell=True. In this case you should not pass a list of strings as argumento to Popen, but only a string that is the complete command line:
proc = subprocess.Popen("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True)
If you pass a list of strings with shell=True its meaning is different: the first element should be the complete command line, while the other elements are passed as options to the shell used. For example on my machine the default shell (sh) has an -x option that will display on stderr all the processes that gets executed:
>>> from subprocess import Popen
>>> proc = Popen(['ps aux | grep python3', '-x'], shell=True)
>>>
username 7301 0.1 0.1 39440 7408 pts/9 S+ 12:57 0:00 python3
username 7302 0.0 0.0 4444 640 pts/9 S+ 12:58 0:00 /bin/sh -c ps aux | grep python3 -x
username 7304 0.0 0.0 15968 904 pts/9 S+ 12:58 0:00 grep python3
Here you can see that a /bin/sh was started that executed the command ps aux | python3 and with an option of -x.
(This is all documented in the documentation for Popen).
This said, one way to achieve what you want is to use subprocess.check_output:
subprocess.check_output("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True)
However this isn't available in python<2.7 so you have to use Popen and communicate():
proc = subprocess.Popen("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True, stdout=subprocess.PIPE)
out, err = proc.communicate()
The alternative is to avoid using shell=True (which is generally a very good thing, since shell=True introduces some security risks) and manually write the pipe using multiple processes:
from subprocess import Popen, PIPE
ps = Popen(['ps', 'aux'], stdout=PIPE)
grep_java = Popen(['grep', 'java -jar'], stdin=ps.stdout, stdout=PIPE)
grep_grep = Popen(['grep', '-v', 'grep'], stdin=grep_java.stdout, stdout=PIPE)
awk = Popen(['awk', '//{print $2}'], stdin=grep_grep.stdout, stdout=PIPE)
out, err = awk.communicate()
grep_grep.wait()
grep_java.wait()
ps.wait()
Note that if you don't care for the standard error you can avoid specifying it. It will then inherit the one of the current process.
There is just one command in the shell pipeline which can't be easily replaced by Python code. So you can start several external processes and connect their inputs and outputs, but I would just start the ps aux and add some Python code to filter and extract the desired data:
from subprocess import PIPE, Popen
def main():
process = Popen(['ps', 'aux'], stdout=PIPE)
pids = [
line.split(None, 2)[1] for line in process.stdout if 'java -jar' in line
]
process.wait()
print '\n'.join(pids)
if __name__ == '__main__':
main()
I want to run a bash command from python shell.
my bash is:
grep -Po "(?<=<cite>).*?(?=</cite>)" /tmp/file1.txt | awk -F/ '{print $1}' | awk '!x[$0]++' > /tmp/file2.txt
what I tried is:
#!/usr/bin/python
import commands
commands.getoutput('grep ' + '-Po ' + '\"\(?<=<dev>\).*?\(?=</dev>\)\" ' + '/tmp/file.txt ' + '| ' + 'awk \'!x[$0]++\' ' + '> ' + '/tmp/file2.txt')
But I don't have any result.
Thank you
If you want to avoid splitting your arguments and worrying about pipes, you can use the shell=True option:
cmd = "grep -Po \"(?<=<dev>).*?(?=</dev>)\" /tmp/file.txt | awk -F/ '{print $1}' | awk '!x[$0]++' > file2.txt"
out = subprocess.check_output(cmd, shell=True)
This will run a subshell which will understands all your directives, including "|" for piping, ">" for redirection. If you do not do this, these symbols normally parsed by the shell will just be passed to grep program.
Otherwise, you have to create the pipes yourself. For example (untested code below):
grep_p = subprocess.Popen(["grep", "-Po", "(?<=<dev>).*?(?=</dev>)", "/tmp/file.txt"], stdout=subprocess.PIPE)
awk_p = subprocess.Popen(["awk", "-F/", "'{print $1}'"], stdin = grep_p.stdout)
file2_fh = open("file2.txt", "w")
awk_p_2 = subprocess.Popen(["awk", "!x[$0]++", stdout = file2_fh, stdin = awk_p.stdout)
awk_p_2.communicate()
However, you're missing the point of python if you are doing this. You should instead look into the re module: re.match, re.sub, re.search, though I'm not familiar enough with awk to translate your commands.
The recommend way to run system commands in python is to use the module subprocess.
import subprocess
a=['grep' ,'-Po', '"(?<=<dev>).*?(?=</dev>)"','/tmp/file.txt']
b=['awk', '-F/', '"{print $1}"']
c=["awk", '"!x[$0]++"']
p1 = subprocess.Popen(a,stdout=subprocess.PIPE)
p2 = subprocess.Popen(b,stdin=p1.stdout,stdout=subprocess.PIPE)
p3 = subprocess.Popen(c,stdin=p2.stdout,stdout=subprocess.PIPE)
p1.stdout.close()
p2.stdout.close()
out,err=p3.communicate()
print out
The point of creating pipes between each subprocess is for security and debugging reasons. Also it makes the code much clearer in terms, which process gets input and sends output to.
Let us write a simple function to easily deal with these messy pipes for us:
def subprocess_pipes (pipes, last_pipe_out = None):
import subprocess
from subprocess import PIPE
last_p = None
for cmd in pipes:
out_pipe = PIPE if not (cmd==pipes[-1] and last_pipe_out) else open(last_pipe_out, "w")
cmd = cmd if isinstance(cmd, list) else cmd.split(" ")
in_pipe = last_p.stdout if last_p else None
p = subprocess.Popen(cmd, stdout = out_pipe, stdin = in_pipe)
last_p = p
comm = last_p.communicate()
return comm
Then we run,
subprocess_pipes(("ps ax", "grep python"), last_pipe_out = "test.out.2")
The result is a "test.out.2" file with the contents of piping "ps ax" into "grep python".
In your case,
a = ["grep", "-Po", "(?<=<cite>).*?(?=</cite>)", "/tmp/file1.txt"]
b = ["awk", "-F/", "{print $1}"]
c = ["awk", "!x[$0]++"]
subprocess_pipes((a, b, c), last_pipe_out = "/tmp/file2.txt")
The commands module is obsolete now.
If you don't actually need the output of your command you can use
import os
exit_status = os.system("your-command")
Otherwise you can use
import suproccess
out, err = subprocess.Popen("your | commands", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell = True).communicate()
Note: for your command you send stdout to file2.txt so I wouldn't expect to see anything in out you will however still see error messages on stderr which will go into err
you must use
import os
os.system(command)
I think what you are looking for is something like:
ubprocess.check_output(same as popen arguments, **kwargs) , use it the same way you would use a popen command , it should show you the output of the program that's being called.
For more details here is a link: http://freefilesdl.com/how-to-call-a-shell-command-from-python/