Using the subprocess module how do I get the following command to work?
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
This command works perfectly when I run it in a bash shell but I can't get it to work when running from within Python. I'm trying to do this from within Python because I need to be able to modify the query and get different result sets back and then process them in Python. I can't use one of the nice Python database connectors for various reasons which leaves me trying to pipe output from isql.
My code currently looks similar to the following:
bash_command = '''
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
'''
process = subprocess.Popen(bash_command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, error = process.communicate()
However I have tried lots of variations:
Using the entire command as a string, or as a list of strings.
Using check_output vs Popen.
Using communicate() to try and send the query to the isql command or having the query be part of the command string using a heredoc.
Using shell = True or not.
Specifying /bin/bash or using the default /bin/sh.
Lots of different quoting and escaping patterns.
And pretty much every permutation of the above.
In no case do I receive the output of the query that I'm looking for. I'm pretty sure that the command isn't being sent to the shell as is but I can't tell what is being sent to the shell.
I feel like this should be pretty simple, send a command to the shell and get the output back, but I just can't make it work. I can't even see what command is being sent to the shell, even using pdb.
shell=True makes subprocess use /bin/sh by default. <<< "here-string" is a bash-ism; pass executable='/bin/bash':
>>> import subprocess
>>> subprocess.call(u'cat <<< "\u0061"', shell=True)
/bin/sh: 1: Syntax error: redirection unexpected
2
>>> subprocess.call(u'cat <<< "\u0061"', shell=True, executable='/bin/bash')
a
0
You should also use raw-string literals to avoid escaping backslashes: "\\u0061" == r"\u0061" != u"\u0061":
>>> subprocess.call(r'cat <<< "\u0061"', shell=True, executable='/bin/bash')
\u0061
0
Though you don't need shell=True here. You could pass the input as a string using process.communicate(input=input_string):
>>> process = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.communicate(br"\u0061")
('\\u0061', None)
The result could look like:
#!/usr/bin/env python
import shlex
from subprocess import Popen, PIPE
cmd = shlex.split(r'isql -v -b -d, DSN_NAME "DOMAIN\username" password')
process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, errors = process.communicate(
b"SELECT column_name, data_type "
b"FROM database_name.information_schema.columns "
b"WHERE table_name = 'some_table';")
Try giving this a shot:
import shlex
from subprocess import Popen, PIPE, STDOUT
sql_statement = '''"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"'''
isqlcommand = 'isql -v -b -d, DSN_NAME "DOMAIN\username" password'
isqlcommand_args = shlex.split(isqlcommand)
process = Popen(isqlcommand_args, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
output = process.communicate(input=sql_statement)[0]
print output
The idea here is to separate the here-string redirection from the isql command execution. This example will pipe the here-string into the stdin of process via process.communicate(). I'm also using shlex.split() to tokenize the command and its arguments.
Edit: Removed Shell=True after reviewing comment from J.F. Sebastian
Related
I have this command which I can get the external IP address returned from Unix shell, so I can use it in my server:
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
on my mac, terminal returns:
192.168.1.3
How do I get this outputted in the python script? I have tried:
import subprocess
command = ['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p']
p = subprocess.Popen(command, stdout=subprocess.PIPE)
text = p.stdout.read()
retcode = p.wait()
and I got error says
ifconfig: interface | does not exist
Thanks in advance!
Your shell command is invoking two commands, the output of ifconfig is used as input to sed. You could emulate this using subprocess but the sed call is just doing some text manipulation, so a cleaner approach would just be to use Python for that step. For example:
import re
import subprocess
pattern = r'inet (?:addr:)?(?!127\.0\.0\.1)((?:\d+\.){3}\d+)'
p = subprocess.Popen(['ifconfig'], stdout=subprocess.PIPE)
text = re.search(pattern, p.stdout.read()).group(1)
retcode = p.wait()
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
you are trying to envoke 2 shell commands, ifconfig and sed, which is okay. but, these are shell commands and the shell keyword argument must be set to true when calling the subprocess.Poen.
use the communictae method, it better to use here. and dispatch the command as a string not list.
import subprocess
command = ' '.join(['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'])
p = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
stdoutdata, stderrdata = p.communicate() #this is blocking
for line in stdoutdata:
#do some thing with line
Popen.communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
The optional input argument should be a string to be sent to the child
process, or None, if no data should be sent to the child.
I am trying to call an executable called foo, and pass it some command line arguments. An external script calls into the executable and uses the following command:
./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log
The script uses Popen to execute this command as follows:
from subprocess import Popen
from subprocess import PIPE
def run_command(command, returnObject=False):
cmd = command.split(' ')
print('%s' % cmd)
p = None
print('command : %s' % command)
if returnObject:
p = Popen(cmd)
else:
p = Popen(cmd)
p.communicate()
print('returncode: %s' % p.returncode)
return p.returncode
return p
command = "./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log
"
run_command(command)
However, this passes extra arguments ['2>&1', '|', '/usr/bin/tee', 'temp.log'] to the foo executable.
How can I get rid of these extra arguments getting passed to foo while maintaining the functionality?
I have tried shell=True but read about avoiding it for security purposes (shell injection attack). Looking for a neat solution.
Thanks
UPDATE:
- Updated the file following the tee command
The string
./main/foo --config config_file 2>&1 | /usr/bin/tee >temp.log
...is full of shell constructs. These have no meaning to anything without a shell in play. Thus, you have two options:
Set shell=True
Replace them with native Python code.
For instance, 2>&1 is the same thing as passing stderr=subprocess.STDOUT to Popen, and your tee -- since its output is redirected and it's passed no arguments -- could just be replaced with stdout=open('temp.log', 'w').
Thus:
p = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=open('temp.log', 'w'))
...or, if you really did want the tee command, but were just using it incorrectly (that is, if you wanted tee temp.log, not tee >temp.log):
p1 = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['tee', 'temp.log'], stdin=p1.stdout)
p1.stdout.close() # drop our own handle so p2's stdin is the only handle on p1.stdout
stdout, _ = p2.communicate()
Wrapping this in a function, and checking success for both ends might look like:
def run():
p1 = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['tee', 'temp.log'], stdin=p1.stdout)
p1.stdout.close() # drop our own handle so p2's stdin is the only handle on p1.stdout
# True if both processes were successful, False otherwise
return (p2.wait() == 0 && p1.wait() == 0)
By the way -- if you want to use shell=True and return the exit status of foo, rather than tee, things get a bit more interesting. Consider the following:
p = subprocess.Popen(['bash', '-c', 'set -o pipefail; ' + command_str])
...the pipefail bash extension will force the shell to exit with the status of the first pipeline component to fail (and 0 if no components fail), rather than using only the exit status of the final component.
Here's a couple of "neat" code examples in addition to the explanation from #Charles Duffy answer.
To run the shell command in Python:
#!/usr/bin/env python
from subprocess import check_call
check_call("./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log",
shell=True)
without the shell:
#!/usr/bin/env python
from subprocess import Popen, PIPE, STDOUT
tee = Popen(["/usr/bin/tee", "temp.log"], stdin=PIPE)
foo = Popen("./main/foo --config config_file".split(),
stdout=tee.stdin, stderr=STDOUT)
pipestatus = [foo.wait(), tee.wait()]
Note: don't use "command arg".split() with non-literal strings.
See How do I use subprocess.Popen to connect multiple processes by pipes?
You may combine answers to two StackOverflow questions:
1. piping together several subprocesses
x | y problem
2. Merging a Python script's subprocess' stdout and stderr (while keeping them distinguishable)
2>&1 problem
I am new to python and working on trying to make a script which checks if a specified host as for example sensu-client exist. I use a deployment software called NSO and run it by: nso status and it shows me this information:
nagios-client host nagios-client down
test host test down
Is there any possibility to make a script to check if for example nagios-Client exist with a script ?
In shell I do it by:
nso status | awk '{ print $1 }'
In this case I would suggest using subprocess' check_output function. The documentation is here. check_output can return, as a string the shell output of a command. So you would have something like this:
import subprocess
foo=subprocess.check_output(['nso', 'status', '|', 'awk', '\'{ print $1 }\''], shell=True)
#Thanks bereal for shell=True
print foo
Of course, if your only targeting linux, you could use the much easier sh module. It allows you to import programs as if they were libraries.
you can use subprocess to run this command and parse the output
import subprocess
command = ['nso', 'status', '|', 'awk', '\'{ print $1 }\'']
p1 = subprocess.Popen(command, stdout=subprocess.PIPE)
You don't have to run awk, since you're already in Python:
import subprocess
proc = subprocess.Popen(['nso', 'status'], stdout=subprocess.PIPE)
# get stdout as a EOL-separated string, ignore stderr for now
out, _ = proc.communicate()
# parse the output, line.split()[0] is awk's $1
items = [line.split()[0] for line in out.split('\n')]
Im trying to get a linux binary to send its standard output to a variable by using subprocess. But just keep getting tracebacks.
>>> import subprocess
>>>nmap -sn -Pn todd.ns.cloudflare.com --script dns-check-zone --script-args='dns-check-zone.domain=www.macmonster.com
Any Ideas (oh and Im using Python2.7).
Ideally I would like to avoid using Shell=true to avoid any security concerns.
Thanks,
shlex to the rescue!
The module shlex will take a string containing the whole shell command and split it up exactly how Popen and check_output expect it. Like this:
import shlex, subprocess
cmd = "/usr/bin/nmap -sn -Pn todd.ns.cloudflare.com --script dns-check-zone --script-args='dns-check-zone.domain=www.macmonster.com'"
args = shlex.split(cmd)
output = subprocess.check_output(args)
When you look at contents of args you'll see:
>>> print args
['/usr/bin/nmap', '-sn', '-Pn', 'todd.ns.cloudflare.com', '--script', 'dns-check-zone', '--script-args=dns-check-zone.domain=www.macmonster.com']
Note that shlex split up the option "--script dns-check-zone" into two tokens. On the other hand it kept "--script-args='dns-check-zone.domain=www.macmonster.com'", but removed the single-quotes.
import subprocess
output = subprocess.check_output(["/usr/bin/nmap", "-sP", "-n", "172.16.1.0/24"])
output = subprocess.check_output(["/usr/bin/nmap", "-sP", "-n", "172.16.1.0/24"], stderr=subprocess.STDOUT)
Have you tried this:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
var = p.stdout.read()
print var
I would have used communicate, but it returns an odd list-type thing.
I am trying to spawn a process using Popen and send it a particular string to its stdin.
I have:
pipe = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
pipe.communicate( my_stdin_str.encode(encoding='ascii') )
pipe.stdin.close()
However, the second line actually escapes the whitespace in my_stdin_str. For example, if I have:
my_stdin_str="This is a string"
The process will see:
This\ is\ a\ string
How can I prevent this behaviour?
I can't reproduce it on Ubuntu:
from subprocess import Popen, PIPE
shell_cmd = "perl -pE's/.\K/-/g'"
p = Popen(shell_cmd, shell=True, stdin=PIPE)
p.communicate("This $PATH is a string".encode('ascii'))
In this case shell=True is unnecessary:
from subprocess import Popen, PIPE
cmd = ["perl", "-pE" , "s/.\K/-/g"]
p = Popen(cmd, stdin=PIPE)
p.communicate("This $PATH is a string".encode('ascii'))
Both produce the same output:
T-h-i-s- -$-P-A-T-H- -i-s- -a- -s-t-r-i-n-g-
Unless you know you need it for some reason, don't run with "shell=True" in general (which, without testing, sounds like what's going on here).