I am trying to find out orphan files on the node in python.
Below is the code snippet
#!/usr/bin/python
import subprocess
try:
s = subprocess.check_output(["find", "/", "-fstype", "proc", "-prune", "-o", "\( -nouser -o -nogroup \)", "-print"])
except subprocess.CalledProcessError as e:
print e.output
else:
if len(s) > 0:
print ("List of Orphan Files are \n%s\n" % s)
else:
print ("Orphan Files does not Exists on the NE")
When I try to run this python code
> python test.py
find: paths must precede expression: \( -nouser -o -nogroup \)
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
When I run the same command on CLI it is working fine.
> find / -fstype proc -prune -o \( -nouser -o -nogroup \) -print
/root/a
Few of you suggested to use shell=true, as per the python doc for subprocess it is security hazard.
Warning Using shell=True can be a security hazard.
Just add shell=True to your check_output
s = subprocess.check_output(["find", "/", "-fstype", "proc", "-prune", "-o", "\( -nouser -o -nogroup \)", "-print"], shell=True)
You should split the command on every whitespace. The easiest way to do this is with shlex.split:
import shlex
import subprocess
cmd = shlex.split('find / -fstype proc -prune -o \( -nouser -o -nogroup \) -print')
subprocess.check_output(cmd)
I tried your script and got the same error as you did. I was trying different things and found something that worked for me. I changed
-print
to
-exec
and it worked. But I am not sure why this was the behavior.
You can set the shell=True option and then just pass the entire shell command. I think the whitespace is causing the issue.
s = subprocess.check_output("find / -fstype proc -prune -o \( -nouser -o -nogroup \) -print", shell=True)
Note this warning relating to setting shell=True. Basically don't do it if the input comes from an external source (like user input). In this case it should therefore be fine.
Each command line parameter must be passed as a separate list item, including the parentheses and their contents:
s = subprocess.check_output(["find", "/", "-fstype", "proc", "-prune", "-o",
"(", "-nouser", "-o", "-nogroup", ")",
"-print"])
Related
This question already has answers here:
How to parse a command line with regular expressions?
(15 answers)
Closed 3 years ago.
I want to split text into list, where file name with spaces should be treated as a single item: example
s = 'cmd -a -b -c "file with spaces.mp4" -e -f'.split()
print(s)
output:
['cmd', '-a', '-b', '-c', '"file', 'with', 'spaces.mp4"', '-e', '-f']
desired output:
['cmd', '-a', '-b', '-c', '"file with spaces.mp4"', '-e', '-f']
I tried using some for loops but it gets nasty, is there a decent way using regex or anything else which doesn't looks ugly
Actually, in this case I won't use regex. This is what shlex.split() is for:
import shlex
s = shlex.split( 'cmd -a -b -c "file with spaces.mp4" -e -f' )
print(s)
Prints:
['cmd', '-a', '-b', '-c', 'file with spaces.mp4', '-e', '-f']
Try shlex
import shlex
data=('cmd -a -b -c "file with spaces.mp4" -e -f')
new=shlex.split(data)
print(new)
yields,
['cmd', '-a', '-b', '-c', 'file with spaces.mp4', '-e', '-f']
This can be accomplished with the built-in shlex module, as such:
import shlex
s = shlex.split('cmd -a -b -c "file with spaces.mp4" -e -f', posix=False)
print(s)
The purpose of posix=False passed into split is to preserve the quotation marks around the multi-word file name, since your desired output formats it like that. If you don't want to preserve the quotes, you can remove the posix argument.
Use a regular expression to match either:
" eventually followed by another " ("[^"]*"), or
any non-space characters (\S+):
input = 'cmd -a -b -c "file with spaces.mp4" -e -f'
output = re.findall('"[^"]*"|\S+', input)
I want to start the testrunner.bat of ReadyApi with parameters. I try to pass some XML parts (PeriodEnd in the code below) as arguments to subprocess.Popen:
argslist = ['C:/Program Files/SmartBear/ReadyAPI-1.9.0/bin/testrunner.bat',
'-a', '-s', 'TestSuite', '-c', 'TestCase', '-f', 'C:/temp/', '-P',
'PeriodEnd=<PeriodEnd>2017-04-11T00:00:00.000Z</PeriodEnd>',
'C:/temp/soapui-project.xml']
proc = Popen(argslist, stdout=PIPE, stderr=PIPE)
This produces the following error:
The system cannot find the file specified.
I found out, that the "<" and ">" are the problems. How can I escape them or pass them to Popen?
The escape character in CMD is ^.
C:\> echo asdf<123>
The syntax of the command is incorrect.
C:\> echo asdf^<123^>
asdf<123>
\ is used to escape character, try \<
https://docs.python.org/2.0/ref/strings.html
possibly use " instead of the '
I am using Python to do sed delete / replace with the help of the subprocess. Somehow I am not getting the number of escapes correct. Here is my code:
from subprocess import call
remover = ["sed", "-i", "'1d'", "file"]
call(remover)
removeq = ["sed", "-i", "'s/\"//g'", "file"]
call(removeq)
Both of these tasks produce the error message:
sed: -e expression #1, char 1: unknown command: `''
How many times does the ' sign need to be escaped in the first list and how many times does the " have to be escaped in the last one? I tried once, twice and three times, but to no avail. There are also no hidden characters that can potentially mess something up.
Any suggestions?
Fix the quoting mechanism:
sed -i 's/\"//g' file
Should be just:
sed -i 's/"//g' file
You can also take adventage of shlex library.
Example from interpreter:
>>> import shlex
>>> cmd = "sed -i '1d' file"
>>> shlex.split(cmd)
['sed', '-i', '1d', 'file']
>>> cmd = """sed -i 's/"//g' file"""
>>> shlex.split(cmd)
['sed', '-i', 's/"//g', 'file']
I have this commands in bash:
ACTIVE_MGMT_1=ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;
I was trying to do it in Python like this:
active_mgmgt_1 = os.popen("""ssh -n MGMT_IP_1 ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2""") ACTIVE_MGMT_1 = active_mgmgt_1.read().replace('\n', '')
It doesn't work; any advice please?
Your popen call needs be set to communicate via a pipe.
Also stop trying to put everything on one line - python doesn't require it and places a lot of empasis on readable code.
I would strongly suggest doing the string processing in python rather than egrep, (use find or re in python), awk (find or egrep) and cut (string split).
It is also recommended to use subprocess.Popen rather than os.popen functions. There is a suggestion to use shlex.spilt to clear up this sort of issue.
untested code
import subprocess
import re
import os
MGMT_IP_1 = os.getenv('MGMT_IP_1')
sp = subprocess.Popen(
['ssh', '-n', MGMT_IP_1, '. .bash_profile; xms sho proc TRAF.*'],
stdout=PIPE, stderr=None)
(result, outtext) = sp.communicate()
# Proceed to process outtext from here using re, find and split
# to the equivalent of egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;
When I run my Python application (that synchronizes a remote directory locally) I have a problem if the directory that contains my app has one or more spaces in its name.
Directory name appears in ssh options like "-o UserKnownHostsFile=<path>" and "-i <path>".
I try to double quote paths in my function that generates the command string, but nothing. I also try to replace spaces like this: path.replace(' ', '\\ '), but it doesn't work.
Note that my code works with dirnames without spaces.
The error returned by ssh is "garbage at the end of line" (code 12)
The command line generated seems ok..
rsync -rztv --delete --stats --progress --timeout=900 --size-only --dry-run \
-e 'ssh -o BatchMode=yes \
-o UserKnownHostsFile="/cygdrive/C/Users/my.user/my\ app/.ssh/known_hosts" \
-i "/cygdrive/C/Users/my.user/my\ app/.ssh/id_rsa"'
user#host:/home/user/folder/ "/cygdrive/C/Users/my.user/my\ app/folder/"
What am I doing wrong? Thank you!
Have you tried building your command as a list of arguments - I just had a similar problem passing a key file for the ssh connection:
command = [
"rsync",
"-rztv",
"--delete",
"--stats",
"--progress",
"--timeout=900",
"--size-only",
"--dry-run",
"-e",
"ssh -o BatchMode=yes -o UserKnownHostsFile='/cygdrive/C/Users/my.user/my\ app/.ssh/known_hosts' -i '/cygdrive/C/Users/my.user/my\ app/.ssh/id_rsa'",
"user#host:/home/user/folder/",
"/cygdrive/C/Users/my.user/my\ app/folder/"
]
sp = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = sp.communicate()[0]
The returned data from the sub-command is including spaces as delimiters. Try updating the Internal Field Separator (IFS) list like:
# store a copy of the current IFS
SAVEIFS=$IFS;
# set IFS to be newline-only
IFS=$(echo -en "\n\b");
# execute your command(s)
rsync -rztv --delete --stats --progress --timeout=900 --size-only --dry-run -e 'ssh -o BatchMode=yes -o UserKnownHostsFile="/cygdrive/C/Users/my.user/my\ app/.ssh/known_hosts" -i "/cygdrive/C/Users/my.user/my\ app/.ssh/id_rsa"' user#host:/home/user/folder/ "/cygdrive/C/Users/my.user/my\ app/folder/"
# put the original IFS back
IFS=$SAVEIFS;
I haven't tested using your command, though it has worked in all cases I've tried in the past.
To avoid escaping issues, use a raw string,
raw_string = r'''rsync -rztv --delete --stats --progress --timeout=900 --size-only --dry-run -e 'ssh -o BatchMode=yes -o UserKnownHostsFile="/cygdrive/C/Users/my.user/my app/.ssh/known_hosts" -i "/cygdrive/C/Users/my.user/my app/.ssh/id_rsa"' user#host:/home/user/folder/ "/cygdrive/C/Users/my.user/my app/folder/"'''
sp = subprocess.Popen(raw_string, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = sp.communicate()[0]