I've got a problem with executing a shell command in python. Here is some part of my code which is causing the error:
p = subprocess.Popen(["cat input.txt |apertium -d. kaz-morph|\
sed -e 's/\$\W*\^/$\n^/g'| cut -f2 -d'/'|cut -f1 -d '<'|\
awk '{print tolower($0)}'|sort -u>output.txt"], shell=True, stdout=f1)
Still getting the error: unterminated 's' command.
Hope you will help me because I couldn't solve it for 10 days :(
p.s. sorry for my english
'\n' must be '\\n', or else it is interpreted as a line break, which results in an unterminated string "cat input.txt |apertium -d. kaz-morph|sed -e 's/\$\W*\^/$".
Alternatively, mark the string as raw: r"cat input.txt |apertium ....".
Related
I am trying to write a bash script to use in python code.
Multi-line bash command (this works perfectly when run directly from terminal)
mydatefile="/var/tmp/date"
while IFS= read line
do
echo $line
sh /opt/setup/Script/EPS.sh $(echo $line) | grep "WORD" | awk -F ',' '{print $6}'
sleep 1
done <"$mydatefile"
My single line conversion
mydatefile="/var/tmp/date;" while IFS= read line do echo $line; sh /opt/setup/Script/EPS.sh $(echo $line) | grep "WORD" | awk -F ',' '{print $6}'; sleep 1; done <"$mydatefile";
ERROR
-bash: syntax error near unexpected token `done'
Missing a ; (fatal syntax error):
while IFS= read line; do echo ...
# ^
# here
More in depth :
combined grep+awk in a single command
mydatefile="/var/tmp/date"
while IFS= read line; do
echo "$line"
sh /opt/setup/Script/EPS.sh "$line" |
awk -F ',' '/WORD/{print $6}'
sleep 1
done < "$mydatefile"
use more quotes !
Learn how to quote properly in shell, it's very important :
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
finally:
mydatefile="/var/tmp/date;" while IFS= read line; do echo $line; sh /opt/setup/Script/EPS.sh "$line" | awk -F ',' '/WORD/{print $6}'; sleep 1; done < "$mydatefile";
One way to do this conversion might be to paste the script onto the command-line, then look up in the history - though this might depend on the version of command-line editor you have. Note that you do need a semicolon before do, but NOT after. You are punished for too many semicolons as well as too few.
Another way would be to line-by-line fold each line in your script and keep testing it.
The binary chop approach is do the first half, test and undo or continue.
Once you have it down to 1 line that works you can pasted it into python.
I have a shell command which parses a certain content and gives the required output. I need to implement this in python but the shell command has a new line character "\n" which is not getting getting executed when run through python command.
Of the many lines in the output log, the required line looks like - configurationFile=/app/log/conf/the_jvm_name.4021.logback.xml
I would only need the_jvm_name from the above. The syntax will always be the same. The shell command works fine.
Shell Command -
ps -ef | grep 12345 | tr " " "\n" | grep logback.configurationFile | awk -F"/" '{print $NF}'| cut -d. -f1
Python (escaped all the required double quotes) -
import subprocess
pid_arr = "12345"
sh_command = "ps -ef | grep "+pid_arr+" | tr \" \" \"\n\" | grep configurationFile | awk -F \"/\" '{print $NF}' | cut -d. -f1"
outpt = subprocess.Popen(sh_command , shell=True,stdout=subprocess.PIPE).communicate()[0].decode('utf-8').strip()
With python, I'm not getting the desired output. It just prints configurationFile as it is in the command.
what am I missing here. Any other better way for getting this details?
You can achieve what you want using a regex substitution in Python:
output = subprocess.check_output(["ps", "-ef"])
for line in output.splitlines():
if re.search("12345", line):
output = re.sub(r".*configurationFile=.*/([^.]+).*", r"\1", line)
This captures the part after the last / in the configuration file path, up to the next ..
You could make it slightly more robust by checking only the second column (the PID) for 12345, either by splitting each line on white space:
cols = re.split("\s+", line)
if len(cols) > 1 and cols[1] == "12345":
or by using a better regex, like:
if re.match(r"\S+\s+12345\s", line):
Note that you could also shorten your pipe considerable by just doing something like:
ps -ef | sed -nE '/12345/ { s/.*configurationFile=.*\/([^.]*).*/\1/; p }'
Your shell command works, but it has to deal with too many lines of output and too many fields per line. An easier solution is to tell the ps command to just give you 1 line and on that line, just one field that you care about. For example, on my system:
ps -o cmd h 979
will output:
/usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
The -o cmd flag will output only the CMD column of the output, while the h parameter represents a command to tell ps to omit the header. Finally, the 979 is the process ID, which tells ps to output information just for this process.
This output is not exactly what you have in your problem, but similar enough. Once we limited the output, we eliminate the need for other commands such as grep, awk, ... At this point, we can use regular expression to extract what we want:
from __future__ import print_function
import re
import subprocess
pid = '979'
command = ['ps', '-o', 'cmd', 'h', pid]
output = subprocess.check_output(command)
pattern = re.compile(r"""
config-file= # Literal string search
.+\/ # Everything up to the last forward slash
([^.]+) # Non-dot chars, this is what we want
""", re.VERBOSE)
matched = pattern.search(output)
if matched:
print(matched.group(1))
Notes
For the regular expression, I am using a verbose form, allowing me to use comment to annotate my pattern. I like this way as regular expression can be difficult to read
On your system, please adjust the "configuration-file" part to work with your output.
Command to run
['ssh', 'tara#ds0', 'sudo', 'cat', '/etc/shadow', '|', 'awk', '-F":"', "'{print $1}'", '|', 'uniq']
<><><><><>
Successfuly fetched from ds0 /etc/shadow
['sudo', 'cat', '/etc/shadow', '|', 'awk', '-F":"', "'{print $1}'", '|', 'uniq']
<><><><><>
cat: invalid option -- 'F'
As you can see above I have kept the command to run in a list which is than run by python subprocess module
The problem here is the first command with ssh is working fine but when running the command in localhost is throwing the invalid option -F error
I have used -F to filter the result to remove password from the result.
I have tested the code manually running sudo cat /etc/shadow | awk -F':' '{print $1}' | uniq and it is running and giving me the expected result
Why am I getting that error: cat invalid option -- 'F' . Where might have the things gone wrong. Why is the subprocess not able to handle it.
Subprocess will consider as | or -F is the cat prameters. Cat dont have this parameters. If you have list and want to use subprocess use like this.
p = ['sudo', 'cat', '/etc/shadow', '|', 'awk', '-F":"', "'{print $1}'", '|', 'uniq']
output = subprocess.check_output(' '.join(p),shell=True)
print output # you will get expected output
OR ....... If you want to give string
p = 'sudo cat /etc/shadow | awk -F: '{print $1}' | uniq'
output = subprocess.check_output(' '.join(p),shell=True)
print output # you will get expected output
| is a pipe operator, interpreted by shell. To have it interpreted, you need to tell subprocess that your command is to be run in a shell - you do it by adding keyword parameter shell=True. Without that, you're executing cat command with several parameters, | and -F":" among them. And -F is not a valid option for cat (as the error message says).
I am trying to escape the following, so I can grab the version of iDevice attached via USB:
system_profiler SPUSBDataType | sed -n -e 's/ */ /g' -e '/iPad/,/Version/p' -e '/iPhone/,/Version/p' | grep 'iPad\|iPhone\|Version' | awk 'NR%2{printf $0;next;}1'
So I can run it via Popen, however everytime I always get an issue on iPad\|iPhone\|Version, my code is the following, in an attempt to escape the single quotes:
cmd1 = Popen([r'system_profiler', 'SPUSBDataType'], stdout=subprocess.PIPE)
cmd2 = Popen([r'sed','-n','-e','\'s/ */ /g\'','-e','\'/iPad/,/Version/p\'', '-e', '\'/iPhone/,/Version/p\''], stdin=cmd1.stdout, stdout=subprocess.PIPE)
cmd3 = Popen([r'grep', '\'iPad\|iPhone\|Version\''], stdin=cmd2.stdout, stdout=subprocess.PIPE)
cmd4 = Popen([r'awk', '\'NR%2{printf $0;next;}1\''], stdin=cmd3.stdout, stdout=subprocess.PIPE)
cmd1.stdout.close()
ver = cmd4.communicate()[0]
Use a raw string literal, or double the backslashes; \| has a meaning in a Python string definition syntax too, resulting in no backslash being present in the resulting value. You don't need those quotes either (the shell would have removed them too):
cmd3 = Popen([r'grep', r"iPad\|iPhone\|Version"], stdin=cmd2.stdout, stdout=subprocess.PIPE)
It'd be much easier to apply the string filtering and replacements in Python code, in my opinion.
Played around with grep and managed to extract what I needed from system_profiler. However Martijn's answer is more suitable if you cannot grep for the necessary string.
prof = Popen(['system_profiler', 'SPUSBDataType'], stdout=subprocess.PIPE)
grep1 = Popen(['grep','-e','iPhone','-e','iPad','-e','iPod', '-A', '4'], stdin=prof.stdout, stdout=subprocess.PIPE)
grep2 = Popen(['grep', 'Version'], stdin=grep1.stdout, stdout=subprocess.PIPE)
prof.stdout.close() # Allow ps_process to receive a SIGPIPE if grep_process exits.
stdoutver = grep2.communicate()[0]
i have a bash script, that extracts the bugs from a csv file and imports it into bugzilla using PyBugz.
The following sequences are used:
description=$(echo "$line" |cut -f5 -d ';')
bugz -d 3 -b http://bugzilla/bugzilla/xmlrpc.cgi -u "$user" -p "$pass" post --product "$prod" --component "$compo" --title "$title" --description "$description" --op-sys "$ops" --platform "$platf" --priority ""$prio"" --severity "$sever" --alias "$alias" --assigned-to "$assign" --cc "$ccl" --version "$ver" --url "$url" --append-command "$appen" --default-confirm "y"
but the description line containing "blablabla \n blablabla" including the newline is beeing recognized as
"Description : blablabla n blablabla"
If I export a bug and dump the output into a textfile, pybugz puts a 0x0a as newline character. how can I make pybugz recognize my \n character as 0x0a??
If description contains the characters \n, and you want to convert that into an actual newline, then you'll have to do some work:
bugz ... --description "$(echo -e "$description")" ...
That will expose other escape sequences as well, see https://www.gnu.org/software/bash/manual/bashref.html#index-echo
I got it.
The way to catch the data was done in the following way:
while read line ; do
description=$(echo "$line" |cut -f5 -d ';')
done <csvfile
however, the read already changed the \n string to n
so whatever I did after that was obviously a failure.
I did it in a very unnice way now but it works like a charm
lines=$(cat csvexport |wc -l)
for (( lineno=1 ; $lineno<=$lines ; lineno++ )); do
description=$(cat csvexport |sed -n "$lineno"p |cut -f5 -d ';')
done
and everything is fine ;-)
Thanks anyway for the help.