We Keep Coding

Python Subprocess with double quotes

I've got a Docker Service Log that takes in NiFi Actions and I want to capture only Log Entries that include "Successfully sent" and "Failed to process session" (and nothing more). They should be captured in a directory called "nifi_logs" in the present working directory. I need to do all of this using Python.
This is what I got so far:
docker_log = 'docker service logs nifi | grep -e "Successfully sent" -e "Failed to process session" >> $PWD/nifi_logs/nifi1.log'
subprocess.Popen(docker_log, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I believe subprocess.Popen() is having difficulty with the double quotes used in the grep, as nifi1.log is completely empty. If the first command looks like the following:
docker_log = 'docker service logs nifi | grep session >> $PWD/nifi_logs/nifi1.log'
The Python code works just fine and captures all log entries with "session" in nifi1.log. As I explained above though, I need to grep for 2 kinds of Log Entires and both include multiple words, meaning I need to use quotes.
If I were to just run this command on the Terminal without Python:
docker service logs nifi | grep -e "Successfully sent" -e "Failed to process session" >> $PWD/nifi_logs/nifi1.log
The log generates the entries just fine, so I know the Docker Service command is written correctly.
I've tried switching the single and double quotes around, I've tried using \" instead of " within the single quotes ... nifi1.log continues to be empty.
I also tried using os.system() instead of subprocess.Popen(), but I run into the same problem (and I believe os.system() is somewhat deprecated).
Any ideas what I'd need to do to change what docker_log equals so that it will properly grep for the 2 search criteria? So you're aware: this question is not asking HOW I generate the log entries (I know what Docker Services I'm looking for, they generate properly), just what I need to do to get Python Subprocess Popen to accept a command with quotes in it.

Thank you for your assistance #David. Looking at your example, I found a solution: I removed stdout=subprocess.PIPE from subprocess.Popen and now it accepts double quotes just fine!
docker_log = 'docker service logs nifi | grep -e "Successfully sent" -e "Failed to process session" >> $PWD/nifi_logs/nifi1.log'
subprocess.Popen(docker_log, shell=True, stderr=subprocess.STDOUT)

How to replicate bash's escaping on commands passed as arguments to other commands

I'm passing the result of the execution of a command to python as input, like so:
$ python parse_ips.py "$(kubectl get configmap ...)"
This works fine when executing from the command line, however I'm now trying to edit the file using PyCharm. Therefore I need the escaped version of the result of this command which I can paste into PyCharm's debug configuration, as I can't execute the command in real-time like I can do on the command line.
However, I am struggling to find a way to replicate the escaping bash does behind the scenes, so I can use the result as an argument within the PyCharm configuration. Running the above kubectl command results in a multi-line string which includes spaces and quotes. When I paste this into PyCharm it just interprets it as multiple arguments. I'm looking for the escaped result, which I could paste directly into the command line, or into PyCharm's debug configuration, to achieve the same result with a fixed parameter for testing.
Any help would be greatly appreciated!
Edit: To clarify, I mean on the command line the result of the $(kubectl ...) command is passed into the python program as a single command line argument when it is surrounded by quotes ("$(kubectl ...)"). So in the python program, you can access sys.argv[1] and it will contain the entire execution output of $(kubectl get configmap ...). However, if I execute that command myself on the command line, the result is a multi-line string.
If I then copy the result of that into PyCharm (or even on the command line again), it is interpreted as many command line arguments. E.g. it would look something like this:
$ python parse_ips.py apiVersion: v1
data:
item1: ifconfig-push 127.0.0.0 255.255.0.0
item2: ifconfig-push 127.0.0.1 255.255.0.0
item3: ifconfig-push 127.0.0.2 255.255.0.0
...
And so on. This obviously doesn't work in the same way as it did before. So I am unable to test my program without making the kubectl call from the command line each time. I was looking to replicate what "$(kubectl ...)" gets converted into so it is able to pass the entire output as a single command line entry.

I am struggling to find a way to replicate the escaping bash does behind the scenes
Typically use printf "%q" to escape stuff.
printf "%q" "$(kubectl get configmap ....)"
This is printf as the bash builtin command. It differs from coreutils printf, and newest ones also support %q with different quoting style:
/usr/bin/printf "%q" "$(kubectl get configmap ....)"
Modern bash also has quoting expansion:
var="$(kubectl get configmap ....)"
echo "${var#Q}"
And there is also the quoting style outputted by set -x.
I would suggest to use a file:
kubectl get configmap ... > /tmp/tempfile
python parse_ips.py "$(cat /tmp/tempfile)"
With xclip you can copy command output straight to the X server clipboard, which is handy:
printf "%q" "$(kubectl get configmap ...)" | xclip -selection clipboard
# then in another window:
python parse_ips.py <right mouse click><select paste>

Save output of a command execution with subprocess

I am trying to get output of subprocess.Popen in variable.
It is working fine for pwd command, but not working for pwdx $(pgrep -U $USER -f SimpleHTTPServer) command.
This works:
(Pdb++) p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
(Pdb++) result = p.communicate()[0]
(Pdb++) result
'xyz'
This is not working:
(Pdb++) subprocess.Popen("pwdx $(pgrep -U $USER -f SimpleHTTPServer)", stdout=subprocess.PIPE)
*** OSError: [Errno 2] No such file or directory
Can someone please let me know how can I save the output of it to a variable?

If you want to pass a command with arguments to Popen(), you have to pass it as a list, like so:
subprocess.Popen(['/bin/ls', '-lat'])
If you just pass a single string as in your example, it assumes the entire thing is the command name, and obviously there is no command literally named pwdx $(pgrep -U $USER -f SimpleHTTPServer).

As the docs and previous answers already state the command you want to execute via subprocess.Popen() needs to be passed as a list.
From the docs:
Note in particular that options (such as -input) and arguments (such
as eggs.txt) that are separated by whitespace in the shell go in
separate list elements, while arguments that need quoting or backslash
escaping when used in the shell (such as filenames containing spaces) are single list elements.
Another useful tip you can get from the docs is to use shlex.split() to help you to properly split your command into a list.
Beware though that the use of special shell parameters (e.g. $USER) might not work well with subprocess.Popen() unless you would set the shell=True option, which you shouldn't do without reading the doc's Security Considerations.

Python linux script (whois error)

I'm trying to create a python script on linux that does a 'whois' command on every connected/connecting IP Address that is parsed from the 'netstat' command.
I am get an error saying "sh: 1: Syntax error: Unterminated quoted string"
and the whois usage options posted below that.
Can anyone explain to me what's wrong the script? I believe it's something to do with the for loop and the way it executes the whois command I just cant seem to find a solution. Below is the script in question:
#!/usr/bin/python
from os import system
answer = [system("netstat -alpntu46 |grep 'ESTABLISHED\|SYN_RECV' | awk '{print $5 }' |cut -d: -f1'")]
for i in answer:
system('whois')
EDIT So my original problem is completely fixed, I'm getting no errors. However, now all the script does is list the IP Addresses and underneath that it lists the whois usage examples:
-h HOST, --host HOST connect to server HOST
-p PORT, --port PORT connect to PORT
-H hide legal disclaimers
--verbose explain what is being done
--help display this help and exit
--version output version information and exit"
So it seems to be running the answer variable but not being able to run the whois command on each address.

Your command string (inside de system() command) has one ' more than needed (at the end of the string). Here it is corrected:
#!/usr/bin/python
from os import system
answer = [system("netstat -alpntu46 |grep 'ESTABLISHED\|SYN_RECV' | awk '{print $5 }' |cut -d: -f1")]
for i in answer:
system('whois')
EDIT (your second question):
When you do for i in answer in python you are looping through all items in your answer, that is correct, however for each IP address you are looping on you are executing only a 'whois' command, without passing any parameters. You should add the parameter to the string, as in:
for i in answer:
system('whois %s' % i)
that is assuming the variable i holds the ip string.

Please check the edit on my first answer (posting this just so you get notified.)

multiple argument from terminal using python

I am able to run this properly using os.system. It is writing pcap file into text.
os.system("tshark -z 'proto,colinfo,tcp.srcport,tcp.srcport' -r filename.pcap > testfile")
But when I tried to give input file from termimal, I got following error:
tshark: -z invalid argument
host = raw_input("Enter file name: ")
test = subprocess.Popen(["tshark","-z","'proto,colinfo,tcp.srcport,tcp.srcport'","-r",host,">","testfile"], stdout=subprocess.PIPE)
output = test.communicate()[0]
Can anybody please figure out where I am making mistake?

To emulate the os.system command, use the shell=True parameter to subprocess.Popen and provide the same command string (not an array of strings):
subprocess.Popen("tshark -z 'proto,colinfo,tcp.srcport,tcp.srcport' -r "
+ host + "> testfile", stdout=subprocess.PIPE, shell=True)
You need a shell to interpret your command line as you are using output redirection to a file ("> testfile").
In your example, you are passing each element of the string list to the execve() system call and hence as parameters to the tshark command (which gets 'proto,colinfo,tcp.srcport,tcp.srcport' as the argument to the -z option instead of proto,colinfo,tcp.srcport,tcp.srcport and which won't know what to do with the > and testfile arguments).
As wnnmaw points out in his comment, using os.system or subprocess.Popen with shell=True with command lines built from user input (the host variable in your case) allows a user to pass arbitrary data to the shell. This can be used to execute (potentially nasty) commands on your system.
For instance, setting host in your example to ; /bin/rm -rf / would delete every file on a UNIX system (assuming the user running the process had enough privilege).
It is therefore very important to validate an user input before adding it to the command string.