I am using Python to do sed delete / replace with the help of the subprocess. Somehow I am not getting the number of escapes correct. Here is my code:
from subprocess import call
remover = ["sed", "-i", "'1d'", "file"]
call(remover)
removeq = ["sed", "-i", "'s/\"//g'", "file"]
call(removeq)
Both of these tasks produce the error message:
sed: -e expression #1, char 1: unknown command: `''
How many times does the ' sign need to be escaped in the first list and how many times does the " have to be escaped in the last one? I tried once, twice and three times, but to no avail. There are also no hidden characters that can potentially mess something up.
Any suggestions?
Fix the quoting mechanism:
sed -i 's/\"//g' file
Should be just:
sed -i 's/"//g' file
You can also take adventage of shlex library.
Example from interpreter:
>>> import shlex
>>> cmd = "sed -i '1d' file"
>>> shlex.split(cmd)
['sed', '-i', '1d', 'file']
>>> cmd = """sed -i 's/"//g' file"""
>>> shlex.split(cmd)
['sed', '-i', 's/"//g', 'file']
Related
This question already has answers here:
How to parse a command line with regular expressions?
(15 answers)
Closed 3 years ago.
I want to split text into list, where file name with spaces should be treated as a single item: example
s = 'cmd -a -b -c "file with spaces.mp4" -e -f'.split()
print(s)
output:
['cmd', '-a', '-b', '-c', '"file', 'with', 'spaces.mp4"', '-e', '-f']
desired output:
['cmd', '-a', '-b', '-c', '"file with spaces.mp4"', '-e', '-f']
I tried using some for loops but it gets nasty, is there a decent way using regex or anything else which doesn't looks ugly
Actually, in this case I won't use regex. This is what shlex.split() is for:
import shlex
s = shlex.split( 'cmd -a -b -c "file with spaces.mp4" -e -f' )
print(s)
Prints:
['cmd', '-a', '-b', '-c', 'file with spaces.mp4', '-e', '-f']
Try shlex
import shlex
data=('cmd -a -b -c "file with spaces.mp4" -e -f')
new=shlex.split(data)
print(new)
yields,
['cmd', '-a', '-b', '-c', 'file with spaces.mp4', '-e', '-f']
This can be accomplished with the built-in shlex module, as such:
import shlex
s = shlex.split('cmd -a -b -c "file with spaces.mp4" -e -f', posix=False)
print(s)
The purpose of posix=False passed into split is to preserve the quotation marks around the multi-word file name, since your desired output formats it like that. If you don't want to preserve the quotes, you can remove the posix argument.
Use a regular expression to match either:
" eventually followed by another " ("[^"]*"), or
any non-space characters (\S+):
input = 'cmd -a -b -c "file with spaces.mp4" -e -f'
output = re.findall('"[^"]*"|\S+', input)
I am trying to find out orphan files on the node in python.
Below is the code snippet
#!/usr/bin/python
import subprocess
try:
s = subprocess.check_output(["find", "/", "-fstype", "proc", "-prune", "-o", "\( -nouser -o -nogroup \)", "-print"])
except subprocess.CalledProcessError as e:
print e.output
else:
if len(s) > 0:
print ("List of Orphan Files are \n%s\n" % s)
else:
print ("Orphan Files does not Exists on the NE")
When I try to run this python code
> python test.py
find: paths must precede expression: \( -nouser -o -nogroup \)
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
When I run the same command on CLI it is working fine.
> find / -fstype proc -prune -o \( -nouser -o -nogroup \) -print
/root/a
Few of you suggested to use shell=true, as per the python doc for subprocess it is security hazard.
Warning Using shell=True can be a security hazard.
Just add shell=True to your check_output
s = subprocess.check_output(["find", "/", "-fstype", "proc", "-prune", "-o", "\( -nouser -o -nogroup \)", "-print"], shell=True)
You should split the command on every whitespace. The easiest way to do this is with shlex.split:
import shlex
import subprocess
cmd = shlex.split('find / -fstype proc -prune -o \( -nouser -o -nogroup \) -print')
subprocess.check_output(cmd)
I tried your script and got the same error as you did. I was trying different things and found something that worked for me. I changed
-print
to
-exec
and it worked. But I am not sure why this was the behavior.
You can set the shell=True option and then just pass the entire shell command. I think the whitespace is causing the issue.
s = subprocess.check_output("find / -fstype proc -prune -o \( -nouser -o -nogroup \) -print", shell=True)
Note this warning relating to setting shell=True. Basically don't do it if the input comes from an external source (like user input). In this case it should therefore be fine.
Each command line parameter must be passed as a separate list item, including the parentheses and their contents:
s = subprocess.check_output(["find", "/", "-fstype", "proc", "-prune", "-o",
"(", "-nouser", "-o", "-nogroup", ")",
"-print"])
I need to fire this query which runs perfectly on the terminal:
sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' <PATH_TO_FILE>
This adds a line below where I find "default]" string.
Using the python code:
query = r""" sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' %s """ % (PATH_T)_FILE)
proc = subprocess.Popen(query.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
However, the command fails in python with error:
Error executing the query sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' /Users/hshah/tmpFile . output = , error = sed: 1: "'/default\]/a\'$'\n''": invalid command code '
What could be the problem here?
You split on every whitespace. This causes query.split() to be
['sed',
'-i',
"''",
"'/default\\]/a\\'$'\\n''",
'Hello',
"world'$'\\n'",
'/tmp/foo']
which is not what you want. Build up the parameters for subprocess.Popen by hand, not by splitting a string.
How could I run this code using subprocess module?
commands.getoutput('sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"')
I've tried this but it doesn't work at all
out_1 = subprocess.Popen(('sudo', 'blkid'), stdout=subprocess.PIPE)
out_2 = subprocess.Popen(('grep', 'uuid'), stdin=out_1.stdout, stdout=subprocess.PIPE)
out_3 = subprocess.Popen(('cut', '-d', '" "', '-f', '1'), stdin=out_2.stdout, stdout=subprocess.PIPE)
main_command = subprocess.check_output(('tr', '-d', '":"'), stdin=out_3.stdout)
main_command
Error: cut: the delimiter must be a single character
from subprocess import check_output, STDOUT
shell_command = '''sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"'''
output = check_output(shell_command, shell=True, stderr=STDOUT,
universal_newlines=True).rstrip('\n')
btw, it returns nothing on my system unless grep -i is used. In the latter case it returns devices. If it is your intent then you could use different command:
from subprocess import check_output
devices = check_output(['sudo', 'blkid', '-odevice']).split()
I'm trying not to use shell=True
It is ok to use shell=True if you control the command i.e., if you don't use user input to construct the command. Consider the shell command as a special language that allows you to express your intent concisely (like regex for string processing). It is more readable then several lines of code that do not use shell:
from subprocess import Popen, PIPE
blkid = Popen(['sudo', 'blkid'], stdout=PIPE)
grep = Popen(['grep', 'uuid'], stdin=blkid.stdout, stdout=PIPE)
blkid.stdout.close() # allow blkid to receive SIGPIPE if grep exits
cut = Popen(['cut', '-d', ' ', '-f', '1'], stdin=grep.stdout, stdout=PIPE)
grep.stdout.close()
tr = Popen(['tr', '-d', ':'], stdin=cut.stdout, stdout=PIPE,
universal_newlines=True)
cut.stdout.close()
output = tr.communicate()[0].rstrip('\n')
pipestatus = [cmd.wait() for cmd in [blkid, grep, cut, tr]]
Note: there are no quotes inside quotes here (no '" "', '":"'). Also unlike the previous command and commands.getoutput(), it doesn't capture stderr.
plumbum provides some syntax sugar:
from plumbum.cmd import sudo, grep, cut, tr
pipeline = sudo['blkid'] | grep['uuid'] | cut['-d', ' ', '-f', '1'] | tr['-d', ':']
output = pipeline().rstrip('\n') # execute
See How do I use subprocess.Popen to connect multiple processes by pipes?
pass your command as one string like this:
main_command = subprocess.check_output('tr -d ":"', stdin=out_3.stdout)
if you have multiple commands and if you want to execute one by one, pass them as list:
main_command = subprocess.check_output([comand1, command2, etc..], shell=True)
I have this commands in bash:
ACTIVE_MGMT_1=ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;
I was trying to do it in Python like this:
active_mgmgt_1 = os.popen("""ssh -n MGMT_IP_1 ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2""") ACTIVE_MGMT_1 = active_mgmgt_1.read().replace('\n', '')
It doesn't work; any advice please?
Your popen call needs be set to communicate via a pipe.
Also stop trying to put everything on one line - python doesn't require it and places a lot of empasis on readable code.
I would strongly suggest doing the string processing in python rather than egrep, (use find or re in python), awk (find or egrep) and cut (string split).
It is also recommended to use subprocess.Popen rather than os.popen functions. There is a suggestion to use shlex.spilt to clear up this sort of issue.
untested code
import subprocess
import re
import os
MGMT_IP_1 = os.getenv('MGMT_IP_1')
sp = subprocess.Popen(
['ssh', '-n', MGMT_IP_1, '. .bash_profile; xms sho proc TRAF.*'],
stdout=PIPE, stderr=None)
(result, outtext) = sp.communicate()
# Proceed to process outtext from here using re, find and split
# to the equivalent of egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;