We Keep Coding

Save output of a command execution with subprocess

I am trying to get output of subprocess.Popen in variable.
It is working fine for pwd command, but not working for pwdx $(pgrep -U $USER -f SimpleHTTPServer) command.
This works:
(Pdb++) p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
(Pdb++) result = p.communicate()[0]
(Pdb++) result
'xyz'
This is not working:
(Pdb++) subprocess.Popen("pwdx $(pgrep -U $USER -f SimpleHTTPServer)", stdout=subprocess.PIPE)
*** OSError: [Errno 2] No such file or directory
Can someone please let me know how can I save the output of it to a variable?

If you want to pass a command with arguments to Popen(), you have to pass it as a list, like so:
subprocess.Popen(['/bin/ls', '-lat'])
If you just pass a single string as in your example, it assumes the entire thing is the command name, and obviously there is no command literally named pwdx $(pgrep -U $USER -f SimpleHTTPServer).

As the docs and previous answers already state the command you want to execute via subprocess.Popen() needs to be passed as a list.
From the docs:
Note in particular that options (such as -input) and arguments (such
as eggs.txt) that are separated by whitespace in the shell go in
separate list elements, while arguments that need quoting or backslash
escaping when used in the shell (such as filenames containing spaces) are single list elements.
Another useful tip you can get from the docs is to use shlex.split() to help you to properly split your command into a list.
Beware though that the use of special shell parameters (e.g. $USER) might not work well with subprocess.Popen() unless you would set the shell=True option, which you shouldn't do without reading the doc's Security Considerations.

Running bash command on server

I am trying to run the bash command pdfcrack in Python on a remote server. This is my code:
bashCommand = "pdfcrack -f pdf123.pdf > myoutput.txt"
import subprocess
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I, however, get the following error message:
Non-option argument myoutput2.txt
Error: file > not found
Can anybody see my mistake?

The first argument to Popen is a list containing the command name and its arguments. > is not an argument to the command, though; it is shell syntax. You could simply pass the entire line to Popen and instruct it to use the shell to execute it:
process = subprocess.Popen(bashCommand, shell=True)
(Note that since you are redirecting the output of the command to a file, though, there is no reason to set its standard output to a pipe, because there will be nothing to read.)
A better solution, though, is to let Python handle the redirection.
process = subprocess.Popen(['pdfcrack', '-f', 'pdf123.pdf'], stdout=subprocess.PIPE)
with open('myoutput.txt', 'w') as fh:
for line in process.stdout:
fh.write(line)
# Do whatever else you want with line
Also, don't use str.split as a replacement for the shell's word splitting. A valid command line like pdfcrack -f "foo bar.pdf" would be split into the incorrect list ['pdfcrack', '-f', '"foo', 'bar.pdf"'], rather than the correct list ['pdfcrack', '-f', 'foo bar.pdf'].

> is interpreted by shell, but not valid otherwise.
So, that would work (don't split, use as-is):
process = subprocess.Popen(bashCommand, shell=True)
(and stdout=subprocess.PIPE isn't useful since all output is redirected to the output file)
But it could be better with native python for redirection to output file and passing arguments as list (handles quote protection if needed)
with open("myoutput.txt","w") as f:
process = subprocess.Popen(["pdfcrack","-f","pdf123.pdf"], stdout=subprocess.PIPE)
f.write(process.read())
process.wait()

Your mistake is > in command.
It doesn't treat this as redirection to file because normally bash does it and now you run it without using bash.
Try with shell=True if you whan to use bash. And then you don't have to split command into list.
subprocess.Popen("pdfcrack -f pdf123.pdf > myoutput.txt", shell=True)

multiple argument from terminal using python

I am able to run this properly using os.system. It is writing pcap file into text.
os.system("tshark -z 'proto,colinfo,tcp.srcport,tcp.srcport' -r filename.pcap > testfile")
But when I tried to give input file from termimal, I got following error:
tshark: -z invalid argument
host = raw_input("Enter file name: ")
test = subprocess.Popen(["tshark","-z","'proto,colinfo,tcp.srcport,tcp.srcport'","-r",host,">","testfile"], stdout=subprocess.PIPE)
output = test.communicate()[0]
Can anybody please figure out where I am making mistake?

To emulate the os.system command, use the shell=True parameter to subprocess.Popen and provide the same command string (not an array of strings):
subprocess.Popen("tshark -z 'proto,colinfo,tcp.srcport,tcp.srcport' -r "
+ host + "> testfile", stdout=subprocess.PIPE, shell=True)
You need a shell to interpret your command line as you are using output redirection to a file ("> testfile").
In your example, you are passing each element of the string list to the execve() system call and hence as parameters to the tshark command (which gets 'proto,colinfo,tcp.srcport,tcp.srcport' as the argument to the -z option instead of proto,colinfo,tcp.srcport,tcp.srcport and which won't know what to do with the > and testfile arguments).
As wnnmaw points out in his comment, using os.system or subprocess.Popen with shell=True with command lines built from user input (the host variable in your case) allows a user to pass arbitrary data to the shell. This can be used to execute (potentially nasty) commands on your system.
For instance, setting host in your example to ; /bin/rm -rf / would delete every file on a UNIX system (assuming the user running the process had enough privilege).
It is therefore very important to validate an user input before adding it to the command string.

Python 2.7 - Quotes added to command when using Popen to execute from variable

I'm trying to send a command through a socket from client to server to execute a command on the server and send the output back to me. Everything works fine if the command is one word with no options. However if I use an option such as netstat -an or dir c:\ then the command isn't recognized and from the output it looks like quotes are put around the command before being executed ('"netstat -an"' is not recognized as an internal or external command). I know they aren't saved in the variable this way because I printed it before being executed to error check. Please help. Here is what my code looks like:
commout = subprocess.Popen([data], stdout=subprocess.PIPE, shell=True)
(out, err) = commout.communicate()

Try make data an array of arguments (the first being the actual command).
For example:
commout = subprocess.Popen(['netstat', '-an'], stdout=subprocess.PIPE, shell=True)
The first element is a string representing the actual command (netstat), the next element is a string representing the first argument (-an).
To clarify, Popen(['echo', 'a', 'b'] is equivalent to echo a b on the command line, whereas Popen(['echo', 'a b'] would be equivalent to echo "a b" instead (ie. a single argument with a space between a and b.

If you pass a list, each item in it will be quoted separately. In this case, just pass a string:
subprocess.Popen(data, stdout=subprocess.PIPE, shell=True)

Python subprocess library: Running grep command from Python

I am trying to run grep command from my Python module using the subprocess library. Since, I am doing this operation on the doc file, I am using Catdoc third party library to get the content in a plan text file. I want to store the content in a file. I don't know where I am going wrong but the program fails to generate a plain text file and eventually to get the grep result. I have gone through the error log but its empty. Thanks for all the help.
def search_file(name, keyword):
#Extract and save the text from doc file
catdoc_cmd = ['catdoc', '-w' , name, '>', 'testing.txt']
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
output = catdoc_process.communicate()[0]
grep_cmd = []
#Search the keyword through the text file
grep_cmd.extend(['grep', '%s' %keyword , 'testing.txt'])
print grep_cmd
p = subprocess.Popen(grep_cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
stdoutdata = p.communicate()[0]
print stdoutdata

On UNIX, specifying shell=True will cause the first argument to be treated as the command to execute, with all subsequent arguments treated as arguments to the shell itself. Thus, the > won't have any effect (since with /bin/sh -c, all arguments after the command are ignored).
Therefore, you should actually use
catdoc_cmd = ['catdoc -w "%s" > testing.txt' % name]
A better solution, though, would probably be to just read the text out of the subprocess' stdout, and process it using re or Python string operations:
catdoc_cmd = ['catdoc', '-w' , name]
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for line in catdoc_process.stdout:
if keyword in line:
print line.strip()

I think you're trying to pass the > to the shell, but that's not going to work the way you've done it. If you want to spawn a process, you should arrange for its standard out to be redirected. Fortunately, that's really easy to do; all you have to do is open the file you want the output to go to for writing and pass it to popen using the stdout keyword argument, instead of PIPE, which causes it to be attached to a pipe which you can read with communicate().