We Keep Coding

Invalid argument/option - '|' [duplicate]

This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 1 year ago.
When trying to run the tasklist command with grep by using subprocess:
command = ("tasklist | grep edpa.exe | gawk \"{ print $2 }\"")
p = subprocess.Popen(command, stdout=subprocess.PIPE)
text = p.communicate(timeout=600)[0]
print(text)
I get this error:
ERROR: Invalid argument/option - '|'.
Type "TASKLIST /?" for usage.
It works fine when i run the command directly from cmd, but when using subprocess something goes wrong.
How can it be fixed? I need to use the output of the command so i can not use os.system
.

Two options:
Use the shell=True option of the Popen(); this will pass it through the shell, which is the part that interprets things like the |
Just run tasklist in the Popen(), then do the processing in Python rather than invoking grep and awk
Of the two, the latter is probably the better approach in this particular instance, since these grep and awk commands are easily translated into Python.
Your linters may also complain that shell=True is prone to security issues, although this particular usage would be OK.

In the absence of shell=True, subprocess runs a single subprocess. In other words, you are passing | and grep etc as arguments to tasklist.
The simplest fix is to add shell=True; but a much better fix is to do the trivial text processing in Python instead. This also coincidentally gets rid of the useless grep.
for line in subprocess.check_output(['tasklist'], timeout=600).splitlines():
if 'edpa.exe' in line:
text = line.split()[1]
print(text)
I have assumed you really want to match edpa.exe literally, anywhere in the output line; your regex would match edpa followed by any character followed by exe. The code could be improved by doing the split first and then look for the search string only in the process name field (if that is indeed your intent).
Perhaps notice also how you generally want to avoid the low-level Popen whenever you can use one of the higher-level functions.

Python subprocess using perl for formatting is giving incomplete output

I'm having an issue reading output from a python subprocess command.
The bash command from whose output I want to read:
pacmd list-sink-inputs | tr '\n' '\r' | perl -pe 's/ *index: ([0-9]+).+?application\.process\.id = "([^\r]+)"\r.+?(?=index:|$)/\2:\1\r/g' | tr '\r' '\n'
When I run this via bash I get the intended output:
4 sink input(s) available.
6249:72
20341:84
20344:86
20350:87
When I try to get it's output via python's subprocess running either one :
subprocess.Popen(cmnd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0].decode('UTF-8')
check_output(cmnd,shell=True).decode('UTF-8')
subprocess.run(cmnd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.decode('utf-8')
where cmnd = """pacmd list-sink-inputs | tr '\n' '\r' | perl -pe 's/ *index: ([0-9]+).+?application\.process\.id = "([^\r]+)"\r.+?(?=index:|$)/\2:\1\r/g' | tr '\r' '\n'"""
It gives the following output:
'4 sink input(s) available.\n\x02:\x01\n\x02:\x01\n\x02:\x01\n\x02:\x01\n'
Which is unintended as it doesn't have the 6249:72 ,etc. numbers I want. Even stderr is blank and returncode is 0 as intended.
The only workaround, I could find was to redirect the bash output to a text file and then read the text file via python which I don't want to use because that's unnecessary file IO.
I've already gone through Missing output from subprocess command, Python Subprocess Grep, Python subprocess run() is giving abnormal output [duplicate] and many others but can't wrap my head around what's going wrong.

You have a quoting issue. """\1""" means chr(0o1). To produce the string \1, you could use """\\1""". The other instances of \ should be \\ as well.
Since all instances of \ need to be escaped, you could also use r"""\1""".
Other issues:
\1 and \2 outside of a regular expression is wrong anyways. You should be using $1 and $2.
There's no use for a mutliline literal here. "..." or r"..." would suffice.
The whole tr business can be avoided by using -0777 to cause perl to treat the entire file as one line.
This gives us:
cmnd = "pacmd list-sink-inputs | perl -0777pe's/ *index: (\\d+).+?application\\.process\\.id = "([^\\n]+)"\\n.+?(?=index:|$)/$2:$1\\n/sag'"
or
cmnd = r"pacmd list-sink-inputs | perl -0777pe's/ *index: (\d+).+?application\.process\.id = "([^\n]+)"\n.+?(?=index:|$)/$2:$1\n/sag'"
But why is Perl being used at all here? You could easily do the same thing in Python!

Convert python list to bash array

I have a python list as a string with the following structure:
var="["127.0.0.1:14550","127.0.0.1:14551"]"
I would like to turn the string into a bash array to be able to loop through it with bash:
for ip in ${var[#]}; do
something
done

Use Perl to parse the Python output, like so (note single quotes around the string, which contains double quotes inside):
array=( $( echo '["127.0.0.1:14550","127.0.0.1:14551"]' | perl -F'[^\d.:]' -lane 'print for grep /./, #F;' ) )
echo ${array[*]}
Output:
127.0.0.1:14550 127.0.0.1:14551
Alternatively, use jq as in the answer by 0stone0, or pipe its output through xargs, which removes quotes, like so:
array=( $( echo '["127.0.0.1:14550","127.0.0.1:14551"]' | jq -c '.[]' | xargs ) )
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-n : Loop over the input one line at a time, assigning it to $_ by default.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-a : Split $_ into array #F on whitespace or on the regex specified in -F option.
-F'[^\d.:]' : Split into #F on any chars other than digit, period, or colon, rather than on whitespace.
print for grep /./, #F; : take the line split into array of strings #F, select with grep only non-empty strings, print one per line.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches

One option is to treat the string as json, and use jq to parse it:
jq -rc '.[]' <<< '["127.0.0.1:14550","127.0.0.1:14551"]' | while read i; do
echo $i
done
127.0.0.1:14550
127.0.0.1:14551

passing text from python to shell | unicode | applying cut on it

I have a python script that essentially parses an xml file, uses the package re and prints text as follows:
string = str(search_compiled.groups(0)[0].encode('utf-8')) + "%" + str(text.encode('utf-8'))
print string
I receive the text in the shell script as follows:
string="$($file.py $arg1 $arg2 $arg3)"
varA="$(echo "$string" | cut -d'%' -f1)"
varB="$(echo "$string" | cut -d'%' -f2)"
echo "$string"
So, in summary, I need the passed string to be cut into two by the delimiter '%' and store the results in varA and varB.
The splitting does not happen.
string shows the entire thingy: part A plus the part B. Here's the catch, the '%' I added in the python script does not get printed though.
Could anyone please help me in understanding what is going wrong?

You can use the pipe and cut commands as you have in the question but without the quotes on the delimiter character use -d% instead of -d'%'
varA=$(echo $string | cut -f1 -d%)
varB=$(echo $string | cut -f2 -d%)

[root#test /tmp]$ eval `echo "aaa%bbb%ccc" | awk -F '%' '{print "a="$1" b="$2}'`
[root#test /tmp]$ echo $a
aaa
[root#test /tmp]$ echo $b
bbb
Explanation
Use awk -F '%' '{print "a="$1" b="$2}' get like this a=aaa b=bbb
eval a=aaa b=bbb Equivalent to the input terminal
$ a=aaa
$ b=bbb

I re-read this for a 3rd time, and I think this is the basic problem (from your description):
string shows the entire thingy: part A plus the part B. Here's the catch, the '%' I added in the python script does not get printed though.
The conversion of data to utf-8 then back to string seems suspect to me. Can you change the string creation line in your python program to this:
string = u'{}%{}'.format(search_compiled.groups(0)[0].encode('utf-8'), text.encode('utf-8'))
You might be double encoding, so this could be what you need:
string = u'{}%{}'.format(search_compiled.groups(0)[0], text)
Add this in the shell script before it calls the python script:
export PYTHONIOENCODING=UTF-8

Need to use a variable in an os.system command in python [duplicate]

This question already has answers here:
How to use python variable in os.system? [duplicate]
(2 answers)
Closed 1 year ago.
I am new to python and I need to use a variable in the os.system command, here's my code so far
import os , sys
s = raw_input('test>')
then i want to use the varable s in an os.system command so I was thinking something like os.system("shutdown -s -t 10 -c" 's')
I don't want answers for that specific command I just want to know in general but if you are going to use an example use shutdown

then i want to use the varable s in an os.system
Use string.format function.
os.system("shutdown -s -t 10 -c {}".format(s))

You can use os.system to execute the particular command, in which case you can join the two strings either by using the + operator, string formatting (.format()), string substitution or some other method.
However, consider the case when the user enters the command 5; rm -rf / or some other malicious command. Instead of using os.system you might want to take a look at subprocess
If you use subprocess you might find the following example handy:
import subprocess
s = raw_input('test>')
subprocess.call(["shutdown", "-s", "-t", "10", "-c", s])

Use subprocess.check_call, pass a list of args and you can add the variable wherever you like:
from subprocess import check_call
check_call(["shutdown", some_var ,"-s", "-t" "10", "-c"])

os.system takes a string as the argument, so you can use anything that modifies a string. For example, string formatting:
os.system('shutdown -s -t 10 -c {0}'.format(s))

Passing place-holder "%s string alongside os.system" should work.
os.system() will execute the command in shell.
import os
var = raw_input('test>') # changed "s" variable to "var" to keep clean
os.system("shutdown -%s -t 10 -c", %(var)) # "var" will be passed to %s place holder

os.system("shutdown -s -t 10 -c" + s)
The plus sign joins two strings
No quotes are used around s. That way you get the value the user entered and not just a literal "s".