I'm using subprocess to call a program within python and I'm passing a string to it, which can contain quotation marks.
This is the piece of code that is giving me troubles
import subprocess
text = subprocess.Popen("""awk 'BEGIN { print "%s"}' | my_program """ % sentence, stdout=subprocess.PIPE, shell=True)
When sentence = "I'm doing this" I get the following error message
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
I guess this has to do with the way quotes are escaped in python and linux. Is there a way to fix it?
you're confusing awk and underlying shell because there's a quote in your quoted awk expression. First part is equivalent to:
awk 'BEGIN { print "I'm doing this"}'
Which is incorrect, even in pure shell.
Quickfix, escape the quotes in your sentence:
text = subprocess.Popen("""awk 'BEGIN { print "%s"}' | my_program """ % sentence.replace("'","\\'"), stdout=subprocess.PIPE, shell=True)
Proper fix: don't use awk at all just to print something, just feed input to your subprocess:
text = subprocess.Popen(my_program, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output,error = text.communicate(sentence.encode())
(and you can get rid of the shell=True in the process)
Last point: you seem to have trouble because my_program is some program plus arguments. To pass a command such as aspell -a you can do:
my_program = "aspell -a"
or:
my_program = ['aspell','-a']
but not
my_program = ['aspell -a']
which is probably what you've done here, so Python tries to literally execute the program "aspell -a" instead of splitting into program + argument.
Related
I'm trying to run the following command in a Python script:
sudo sed -i 's/auth-user-pass/auth-user-pass \/etc\/openvpn\/credentials/g' /etc/openvpn/US-East.ovpn
The command above runs fine in a terminal.
My Python script looks like this;
import subprocess
subprocess.Popen(["sudo", "sed", "-i", "'s/auth-user-pass/auth-user-pass", "\/etc\/openvpn\/credentials/g'", "/etc/openvpn/US-East.ovpn"], stdout=subprocess.PIPE, text=True)
But I get the following error when I run the script
sed: -e expression #1, char 1: unknown command: `''
I thought some characters (like the single quote) might need escaping but I've been trying and none of them work.
I'm quite lost; can anyone help?
Since your command line is:
sudo sed -i 's/auth-user-pass/auth-user-pass \/etc\/openvpn\/credentials/g' /etc/openvpn/US-East.ovpn
You need to remove the single quotes (the shell does that for you) and you need to keep the whole of the single-quoted argument as one argument, not splitting it at spaces (the shell doesn't split at spaces inside a quoted string):
subprocess.Popen(["sudo", "sed", "-i", "s/auth-user-pass/auth-user-pass \/etc\/openvpn\/credentials/g", "/etc/openvpn/US-East.ovpn"], stdout=subprocess.PIPE, text=True)
The sed command does not expect to see the single quotes.
Can you test this without the commas in the sed command parameter?
import subprocess
subprocess.Popen(["sudo", "sed", "-i", "'s/auth-user-pass/auth-user-pass \/etc\/openvpn\/credentials/g'", "/etc/openvpn/US-East.ovpn"], stdout=subprocess.PIPE, text=True)
My goal is to execute the following bash command in Python and store its output:
echo 'sudo ./run_script.sh -dates \\{\\'2017-11-16\\',\\'2017-11-29\\'\\}'|sed 's;\\\\;\\;'
When I run this command in bash, the output is: sudo ./run_script.sh -dates \{\'2019-10-05\',\'2019-10-04\'\}
My initial idea was to replace the double backslash by a single backslash in Python. As ridiculous as it seems, I couldn't do it in Python (only when using print() the output is as I would like but I can't store the output of print() and str() doesn't convert \ to . So I decided to do it in bash.
import subprocess
t= 'some \\ here'
cmd = "echo \'"+ t+"\'|sed 's;\\\\;\\;'"
ps = subprocess.run(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
ps.stdout
Out[6]: b"sed: -e expression #1, char 7: unterminated `s' command\n"
Running Python 3.6.8 on Ubuntu 18
Try using subprocess.check_output instead. You're also forgetting an extra backslash for every backslash in your command.
import subprocess
command = "echo 'some \\\\here'|sed 's;\\\\\\\\;\\\\;'"
output = subprocess.check_output(command, shell=True).decode()
print(output) # prints your expect result "some \here"
After re-reading your question I kinda understood what you wanted.
a = r'some \here'
print(a) #some \here
Again, raw string literals...
I am trying to escape the following, so I can grab the version of iDevice attached via USB:
system_profiler SPUSBDataType | sed -n -e 's/ */ /g' -e '/iPad/,/Version/p' -e '/iPhone/,/Version/p' | grep 'iPad\|iPhone\|Version' | awk 'NR%2{printf $0;next;}1'
So I can run it via Popen, however everytime I always get an issue on iPad\|iPhone\|Version, my code is the following, in an attempt to escape the single quotes:
cmd1 = Popen([r'system_profiler', 'SPUSBDataType'], stdout=subprocess.PIPE)
cmd2 = Popen([r'sed','-n','-e','\'s/ */ /g\'','-e','\'/iPad/,/Version/p\'', '-e', '\'/iPhone/,/Version/p\''], stdin=cmd1.stdout, stdout=subprocess.PIPE)
cmd3 = Popen([r'grep', '\'iPad\|iPhone\|Version\''], stdin=cmd2.stdout, stdout=subprocess.PIPE)
cmd4 = Popen([r'awk', '\'NR%2{printf $0;next;}1\''], stdin=cmd3.stdout, stdout=subprocess.PIPE)
cmd1.stdout.close()
ver = cmd4.communicate()[0]
Use a raw string literal, or double the backslashes; \| has a meaning in a Python string definition syntax too, resulting in no backslash being present in the resulting value. You don't need those quotes either (the shell would have removed them too):
cmd3 = Popen([r'grep', r"iPad\|iPhone\|Version"], stdin=cmd2.stdout, stdout=subprocess.PIPE)
It'd be much easier to apply the string filtering and replacements in Python code, in my opinion.
Played around with grep and managed to extract what I needed from system_profiler. However Martijn's answer is more suitable if you cannot grep for the necessary string.
prof = Popen(['system_profiler', 'SPUSBDataType'], stdout=subprocess.PIPE)
grep1 = Popen(['grep','-e','iPhone','-e','iPad','-e','iPod', '-A', '4'], stdin=prof.stdout, stdout=subprocess.PIPE)
grep2 = Popen(['grep', 'Version'], stdin=grep1.stdout, stdout=subprocess.PIPE)
prof.stdout.close() # Allow ps_process to receive a SIGPIPE if grep_process exits.
stdoutver = grep2.communicate()[0]
I need to write a python script where I need to call a few awk commands inside of it.
#!/usr/bin/python
import os, sys
input_dir = '/home/abc/data'
os.chdir(input_dir)
#wd=os.getcwd()
#print wd
os.system ("tail -n+2 ./*/*.tsv|cat|awk 'BEGIN{FS="\t"};{split($10,arr,"-")}{print arr[1]}'|sort|uniq -c")
It gives an error in line 8: SyntaxError: unexpected character after line continuation character
Is there a way I can get the awk command get to work within the python script?
Thanks
You have both types of quotes in that string, so use triple quotes around the whole thing
>>> x = '''tail -n+2 ./*/*.tsv|cat|awk 'BEGIN{FS="\t"};{split($10,arr,"-")}{print arr[1]}'|sort|uniq -c'''
>>> x
'tail -n+2 ./*/*.tsv|cat|awk \'BEGIN{FS="\t"};{split($10,arr,"-")}{print arr[1]}\'|sort|uniq -c'
You should use subprocess instead of os.system:
import subprocess
COMMAND = "tail -n+2 ./*/*.tsv|cat|awk 'BEGIN{FS=\"\t\"};{split($10,arr,\"-\")}{print arr[1]}'|sort|uniq -c"
subprocess.call(COMMAND, shell=True)
As TehTris has pointed out, the arrangement of quotes in the question breaks the command string into multiple strings. Pre-formatting the command and escaping the double-quotes fixes this.
I'm trying to write an svn pre-commit hook in python. Part of this involves checking the diff file to see if there are any actual file changes (as opposed to just property changes).
I have a working grep command which I can execute fine on the shell
grep "^\(Added: \|Modified: \|Deleted: \)" diff filename | grep -v 'svn:'
However when I put it through subprocess.POpen it escapes all my backslashes, which knackers the regexp.
Executing command: ['grep', '"^\\Added: \\|Modified: \\|Deleted: \\)", ...]
How do I avoid this?
NB: I'm aware that I can pipe results between subprocesses and I can do the two greps that way. I need help getting the first one working first though :/
NB2: I also tried using filterdiff --clean instead and couldn't get it to work. Searching for Added, Modified or Deleted lines, removing those with 'svn:' in and checking I had some results seemed to work though.
Python code:
command = ['grep', '"^\(Added: \|Modified: \|Deleted: \)"', filename]
sys.stdout.write('Executing command: %s\n' % (command))
p = subprocess.Popen(command,
stdin = subprocess.PIPE
stdout = subprocess.PIPE
stderr = subprocess.STDOUT
shell = True)
data = p.stdout.read()
if len(data) == 0:
sys.stdout.write("Diff does not contain any file modifications./n")
exit(0)
You need to consider what you want grep to see in its command line arguments.
The first argument needs to be the literal string "^\(Added: \|Modified: \|Deleted: \)", so that means that it shouldn't include the double quotes but should include the backslashes.
The way to express this kind of string is to use Python raw strings:
command = ['grep', r'^\(Added: \|Modified: \|Deleted: \)', filename]
A good way to check what you're actually running is to replace grep by echo so you can at least see what you're passing to the command.