We Keep Coding

Python subprocess using perl for formatting is giving incomplete output

I'm having an issue reading output from a python subprocess command.
The bash command from whose output I want to read:
pacmd list-sink-inputs | tr '\n' '\r' | perl -pe 's/ *index: ([0-9]+).+?application\.process\.id = "([^\r]+)"\r.+?(?=index:|$)/\2:\1\r/g' | tr '\r' '\n'
When I run this via bash I get the intended output:
4 sink input(s) available.
6249:72
20341:84
20344:86
20350:87
When I try to get it's output via python's subprocess running either one :
subprocess.Popen(cmnd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0].decode('UTF-8')
check_output(cmnd,shell=True).decode('UTF-8')
subprocess.run(cmnd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.decode('utf-8')
where cmnd = """pacmd list-sink-inputs | tr '\n' '\r' | perl -pe 's/ *index: ([0-9]+).+?application\.process\.id = "([^\r]+)"\r.+?(?=index:|$)/\2:\1\r/g' | tr '\r' '\n'"""
It gives the following output:
'4 sink input(s) available.\n\x02:\x01\n\x02:\x01\n\x02:\x01\n\x02:\x01\n'
Which is unintended as it doesn't have the 6249:72 ,etc. numbers I want. Even stderr is blank and returncode is 0 as intended.
The only workaround, I could find was to redirect the bash output to a text file and then read the text file via python which I don't want to use because that's unnecessary file IO.
I've already gone through Missing output from subprocess command, Python Subprocess Grep, Python subprocess run() is giving abnormal output [duplicate] and many others but can't wrap my head around what's going wrong.

You have a quoting issue. """\1""" means chr(0o1). To produce the string \1, you could use """\\1""". The other instances of \ should be \\ as well.
Since all instances of \ need to be escaped, you could also use r"""\1""".
Other issues:
\1 and \2 outside of a regular expression is wrong anyways. You should be using $1 and $2.
There's no use for a mutliline literal here. "..." or r"..." would suffice.
The whole tr business can be avoided by using -0777 to cause perl to treat the entire file as one line.
This gives us:
cmnd = "pacmd list-sink-inputs | perl -0777pe's/ *index: (\\d+).+?application\\.process\\.id = "([^\\n]+)"\\n.+?(?=index:|$)/$2:$1\\n/sag'"
or
cmnd = r"pacmd list-sink-inputs | perl -0777pe's/ *index: (\d+).+?application\.process\.id = "([^\n]+)"\n.+?(?=index:|$)/$2:$1\n/sag'"
But why is Perl being used at all here? You could easily do the same thing in Python!

passing text from python to shell | unicode | applying cut on it

I have a python script that essentially parses an xml file, uses the package re and prints text as follows:
string = str(search_compiled.groups(0)[0].encode('utf-8')) + "%" + str(text.encode('utf-8'))
print string
I receive the text in the shell script as follows:
string="$($file.py $arg1 $arg2 $arg3)"
varA="$(echo "$string" | cut -d'%' -f1)"
varB="$(echo "$string" | cut -d'%' -f2)"
echo "$string"
So, in summary, I need the passed string to be cut into two by the delimiter '%' and store the results in varA and varB.
The splitting does not happen.
string shows the entire thingy: part A plus the part B. Here's the catch, the '%' I added in the python script does not get printed though.
Could anyone please help me in understanding what is going wrong?

You can use the pipe and cut commands as you have in the question but without the quotes on the delimiter character use -d% instead of -d'%'
varA=$(echo $string | cut -f1 -d%)
varB=$(echo $string | cut -f2 -d%)

[root#test /tmp]$ eval `echo "aaa%bbb%ccc" | awk -F '%' '{print "a="$1" b="$2}'`
[root#test /tmp]$ echo $a
aaa
[root#test /tmp]$ echo $b
bbb
Explanation
Use awk -F '%' '{print "a="$1" b="$2}' get like this a=aaa b=bbb
eval a=aaa b=bbb Equivalent to the input terminal
$ a=aaa
$ b=bbb

I re-read this for a 3rd time, and I think this is the basic problem (from your description):
string shows the entire thingy: part A plus the part B. Here's the catch, the '%' I added in the python script does not get printed though.
The conversion of data to utf-8 then back to string seems suspect to me. Can you change the string creation line in your python program to this:
string = u'{}%{}'.format(search_compiled.groups(0)[0].encode('utf-8'), text.encode('utf-8'))
You might be double encoding, so this could be what you need:
string = u'{}%{}'.format(search_compiled.groups(0)[0], text)
Add this in the shell script before it calls the python script:
export PYTHONIOENCODING=UTF-8

python multiline command running from bash

I'm trying to run this:
python -c "for i in range(10):\n print i"
but I get an error:
File "<string>", line 1
for i in range(10):\n print i
^
SyntaxError: unexpected character after line continuation character
According to this I assume that bash should have processed (namely, newline symbol) command line arguments but the returned error shows the opposite case. Where am I wrong, and why does this happen?
P.S. python-2.7
EDIT
Let me explain my motivation a bit.
This code example is definitely pretty silly. Since the doc says that "command can be one or more statements separated by newlines, with significant leading whitespace as in normal module code", I was interested in how should I bring those mentioned newlines to the command properly.
The proposed solutions here are:
Use ; to distinguish several commands inside the loop. Yes, that works but it still is a one-liner, I can not use it If I want to run some commands after the loop. ; is not a replacement for a newline.
Type ^M where newline is needed. This hits the goal more precisely but unfortunately, to my point of view, this basically ruins the whole idea of running a python code from the command line because it requires interactive mode. As I understand it's the same as entering a command ant hitting Enter key. So no difference to typing python and working in its shell. That said, I cannot write this in a bash script. Or may I?
Probably the question really should have been splitted into two ones:
Bash escaping:
Enclosing characters in double quotes (‘"’) preserves the literal value of all characters within the quotes, with the exception of ‘$’, ‘’, ‘\’, and, when history expansion is enabled, ‘!’. The characters ‘$’ and ‘’ retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: ‘$’, ‘`’, ‘"’, ‘\’, or newline.
How does this correspond to the case described? How does bash handles newlines? I found that putting the command into unary quotes makes no change.
How to pass a newline to python in a non-interactive way. (You may say -- why don't you write an ordinary python file with all newlines you want -- you are right but I'm interested in what is exactly meant in the documentation since it quotes newline)

You actually would need to transform the \n part into an actual newline. That can be done with the $'' syntax:
python -c $'for i in range(10):\n print i'
0
1
2
3
4
5
6
7
8
9
You can also reach that result with echo -e or printf
$ python -c "$(echo -e "for i in range(10):\n print i")"
You could also use a here string:
$ python <<< $(echo -e "for i in range(10):\n print i")
See section 3.1.2.4 ANSI-C Quoting of the Bash Manpage for more information.

Remove \n
python -c "for i in range(10): print i"
Or
You can use ; for using multiple line in for loop
python -c "for i in range(10): print '1st newline';print '2nd newline';print i"

You can run a multi-line python -c statement by adding CR characters in your line:
python -c "for i in range(10):^M print (i)^M print ('Hello:' + str(i*i))"
where ^M is not actually ^ followed by M, it is actually the character you get when you type [CTRL-v][CTRL-m]. Notice the space after this character, which means there are two print statements in the for loop, and it should print:
0
Hello:0
1
Hello:1
....
9
Hello:81
You can do this in a bash script too:
#!/bin/bash
A="python -c \"for i in range(10):^M print (i)^M print ('Hello:' + str(i*i))\""
eval $A

How to run multiple lines using python -c

I need to use python -c to remotely run some code, it works when I use:
python -c "a=4;print a"
4
Or
python -c "if True:print 'ye'"
But python -c "a=4;if a<5:print 'ye'" will generate an error:
File "<string>", line 1
a=4;if a<5:print 'ye'
SyntaxError: invalid syntax
What should I do to make it work, any advice?

Enclose it in single quotes and use multiple lines:
python -c '
a = 4
if a < 5:
print "ye"
'
If you need a single quote in the code, use this horrible construct:
python -c '
a = 4
if a < 5:
print '\''ye'\''
'
This works because most UNIX shells will not interpret anything between single quotes—so we have some Python code right up until where we need a single quote. Since it’s completely uninterpreted, we can’t just escape the quote; rather, we end the quotation, then insert a literal quotation mark (escaped, so it won’t be interpreted as the start of a new quote), and finally another quote to put us back into the uninterpreted-string state. It’s a little ugly, but it works.

symbols in command line argument.. python, bash

I am writing a python script on Linux for twitter post using API, Is it possible to pass symbols like "(" ")" etc in clear text without apostrophes....
% ./twitterupdate this is me #works fine
% ./twitterupdate this is bad :(( #this leaves a error on bash.
Is the only alternative is to enclose the text into --> "" ?? like..
% ./twitterupdate "this is bad :((" #this will reduce the ease of use for the script
Is there any workaround?

Yes, quoting the string is the only way. Bash has its syntax and and some characters have special meaning. Btw, using "" is not enough, use apostrophes instead. Some characters will still get interpretted with normal quotation marks:
$ echo "lots of $$"
lots of 15570
$ echo 'lots of $$'
lots of $$

http://www.gnu.org/software/bash/manual/bashref.html#Quoting