I need to fire this query which runs perfectly on the terminal:
sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' <PATH_TO_FILE>
This adds a line below where I find "default]" string.
Using the python code:
query = r""" sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' %s """ % (PATH_T)_FILE)
proc = subprocess.Popen(query.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
However, the command fails in python with error:
Error executing the query sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' /Users/hshah/tmpFile . output = , error = sed: 1: "'/default\]/a\'$'\n''": invalid command code '
What could be the problem here?
You split on every whitespace. This causes query.split() to be
['sed',
'-i',
"''",
"'/default\\]/a\\'$'\\n''",
'Hello',
"world'$'\\n'",
'/tmp/foo']
which is not what you want. Build up the parameters for subprocess.Popen by hand, not by splitting a string.
Related
I have the following code to receive list of process with sudo:
sudoPass = 'mypass'
command = "launchctl list | grep -v com.apple"
x = os.system('echo %s|sudo -S %s' % (sudoPass, command))
But, I receive answer in int. I need in str. Is it possible to convert it to str without loosing data?
os.system returns (in most cases, see https://docs.python.org/3/library/os.html#os.system) the exit value of the process. Meaning most of the time 0 is everything went fine.
What you look for is the subprocess module (https://docs.python.org/3/library/subprocess.html) that allow you to capture output like so :
import subprocess
sudoPass = 'mypass\n' #Note the new line
command = "launchctl list | grep -v com.apple"
x = subprocess.Popen('echo %s|sudo -S %s' % (sudoPass, command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdout, stderr = x.communicate()
print(stdout)
Using the subprocess module how do I get the following command to work?
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
This command works perfectly when I run it in a bash shell but I can't get it to work when running from within Python. I'm trying to do this from within Python because I need to be able to modify the query and get different result sets back and then process them in Python. I can't use one of the nice Python database connectors for various reasons which leaves me trying to pipe output from isql.
My code currently looks similar to the following:
bash_command = '''
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
'''
process = subprocess.Popen(bash_command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, error = process.communicate()
However I have tried lots of variations:
Using the entire command as a string, or as a list of strings.
Using check_output vs Popen.
Using communicate() to try and send the query to the isql command or having the query be part of the command string using a heredoc.
Using shell = True or not.
Specifying /bin/bash or using the default /bin/sh.
Lots of different quoting and escaping patterns.
And pretty much every permutation of the above.
In no case do I receive the output of the query that I'm looking for. I'm pretty sure that the command isn't being sent to the shell as is but I can't tell what is being sent to the shell.
I feel like this should be pretty simple, send a command to the shell and get the output back, but I just can't make it work. I can't even see what command is being sent to the shell, even using pdb.
shell=True makes subprocess use /bin/sh by default. <<< "here-string" is a bash-ism; pass executable='/bin/bash':
>>> import subprocess
>>> subprocess.call(u'cat <<< "\u0061"', shell=True)
/bin/sh: 1: Syntax error: redirection unexpected
2
>>> subprocess.call(u'cat <<< "\u0061"', shell=True, executable='/bin/bash')
a
0
You should also use raw-string literals to avoid escaping backslashes: "\\u0061" == r"\u0061" != u"\u0061":
>>> subprocess.call(r'cat <<< "\u0061"', shell=True, executable='/bin/bash')
\u0061
0
Though you don't need shell=True here. You could pass the input as a string using process.communicate(input=input_string):
>>> process = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.communicate(br"\u0061")
('\\u0061', None)
The result could look like:
#!/usr/bin/env python
import shlex
from subprocess import Popen, PIPE
cmd = shlex.split(r'isql -v -b -d, DSN_NAME "DOMAIN\username" password')
process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, errors = process.communicate(
b"SELECT column_name, data_type "
b"FROM database_name.information_schema.columns "
b"WHERE table_name = 'some_table';")
Try giving this a shot:
import shlex
from subprocess import Popen, PIPE, STDOUT
sql_statement = '''"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"'''
isqlcommand = 'isql -v -b -d, DSN_NAME "DOMAIN\username" password'
isqlcommand_args = shlex.split(isqlcommand)
process = Popen(isqlcommand_args, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
output = process.communicate(input=sql_statement)[0]
print output
The idea here is to separate the here-string redirection from the isql command execution. This example will pipe the here-string into the stdin of process via process.communicate(). I'm also using shlex.split() to tokenize the command and its arguments.
Edit: Removed Shell=True after reviewing comment from J.F. Sebastian
I am using Python to do sed delete / replace with the help of the subprocess. Somehow I am not getting the number of escapes correct. Here is my code:
from subprocess import call
remover = ["sed", "-i", "'1d'", "file"]
call(remover)
removeq = ["sed", "-i", "'s/\"//g'", "file"]
call(removeq)
Both of these tasks produce the error message:
sed: -e expression #1, char 1: unknown command: `''
How many times does the ' sign need to be escaped in the first list and how many times does the " have to be escaped in the last one? I tried once, twice and three times, but to no avail. There are also no hidden characters that can potentially mess something up.
Any suggestions?
Fix the quoting mechanism:
sed -i 's/\"//g' file
Should be just:
sed -i 's/"//g' file
You can also take adventage of shlex library.
Example from interpreter:
>>> import shlex
>>> cmd = "sed -i '1d' file"
>>> shlex.split(cmd)
['sed', '-i', '1d', 'file']
>>> cmd = """sed -i 's/"//g' file"""
>>> shlex.split(cmd)
['sed', '-i', 's/"//g', 'file']
How could I run this code using subprocess module?
commands.getoutput('sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"')
I've tried this but it doesn't work at all
out_1 = subprocess.Popen(('sudo', 'blkid'), stdout=subprocess.PIPE)
out_2 = subprocess.Popen(('grep', 'uuid'), stdin=out_1.stdout, stdout=subprocess.PIPE)
out_3 = subprocess.Popen(('cut', '-d', '" "', '-f', '1'), stdin=out_2.stdout, stdout=subprocess.PIPE)
main_command = subprocess.check_output(('tr', '-d', '":"'), stdin=out_3.stdout)
main_command
Error: cut: the delimiter must be a single character
from subprocess import check_output, STDOUT
shell_command = '''sudo blkid | grep 'uuid' | cut -d " " -f 1 | tr -d ":"'''
output = check_output(shell_command, shell=True, stderr=STDOUT,
universal_newlines=True).rstrip('\n')
btw, it returns nothing on my system unless grep -i is used. In the latter case it returns devices. If it is your intent then you could use different command:
from subprocess import check_output
devices = check_output(['sudo', 'blkid', '-odevice']).split()
I'm trying not to use shell=True
It is ok to use shell=True if you control the command i.e., if you don't use user input to construct the command. Consider the shell command as a special language that allows you to express your intent concisely (like regex for string processing). It is more readable then several lines of code that do not use shell:
from subprocess import Popen, PIPE
blkid = Popen(['sudo', 'blkid'], stdout=PIPE)
grep = Popen(['grep', 'uuid'], stdin=blkid.stdout, stdout=PIPE)
blkid.stdout.close() # allow blkid to receive SIGPIPE if grep exits
cut = Popen(['cut', '-d', ' ', '-f', '1'], stdin=grep.stdout, stdout=PIPE)
grep.stdout.close()
tr = Popen(['tr', '-d', ':'], stdin=cut.stdout, stdout=PIPE,
universal_newlines=True)
cut.stdout.close()
output = tr.communicate()[0].rstrip('\n')
pipestatus = [cmd.wait() for cmd in [blkid, grep, cut, tr]]
Note: there are no quotes inside quotes here (no '" "', '":"'). Also unlike the previous command and commands.getoutput(), it doesn't capture stderr.
plumbum provides some syntax sugar:
from plumbum.cmd import sudo, grep, cut, tr
pipeline = sudo['blkid'] | grep['uuid'] | cut['-d', ' ', '-f', '1'] | tr['-d', ':']
output = pipeline().rstrip('\n') # execute
See How do I use subprocess.Popen to connect multiple processes by pipes?
pass your command as one string like this:
main_command = subprocess.check_output('tr -d ":"', stdin=out_3.stdout)
if you have multiple commands and if you want to execute one by one, pass them as list:
main_command = subprocess.check_output([comand1, command2, etc..], shell=True)
I have a list of IP's that I want to run a whois (using the linux tool whois) against and only see the Country option.
Here is my script:
import os
import time
iplist = open('ips.txt').readlines()
for i in iplist:
time.sleep(2)
print "Country: IP {0}".format(i)
print os.system("whois -h whois.arin.net + {0} | grep Country ".format(i))
So I want to display what IP is being ran, then I just want to see the Country info using grep. I see this error when I run it and the grep is not ran:
sh: -c: line 1: syntax error near unexpected token `|'
sh: -c: line 1: ` | grep Country '
this code below works so it must be an issue with my for loop:
print os.system("whois -h whois.arin.net + {0} | grep Country ".format('8.8.8.8'))
What am I doing wrong? Thank you!!!!
You're not stripping trailing newlines from the lines you read from the file. As a result, you are passing to os.system a string like "whois -h whois.arin.net + a.b.c.d\n | grep Country". The shell parses the string as two commands and complains of "unexpected token |" at the beginning of the second one. This explains why there is no error when you use a hand-made string such as "8.8.8.8".
Add i = i.strip() after the sleep, and the problem will go away.
user4815162342 is correct about the issue you are having, but might I suggest you replace os.system with subprocess.Popen? Capturing the output from the system call is not intuitive.. should you want to result to go anywhere but your screen, you'll likely going to have issues
from subprocess import Popen, PIPE
server = 'whois.arin.net'
def find_country(ip):
proc = Popen(['whois', '-h', server, ip], stdout = PIPE, stderr = PIPE)
stdout, stderr = proc.communicate()
if stderr:
raise Exception("Error with `whois` subprocess: " + stderr)
for line in stdout.split('\n'):
if line.startswith('Country:'):
return line.split(':')[1].strip() # Good place for regex
for ip in [i.strip() for i in open('ips.txt').readlines()]:
print find_country(ip)
Python is awesome at string handling- there should be no reason to create a grep subprocess to pattern match the output of a separate subprocess.
Try sh:
import os
import time
import re
import sh
iplist = open('ips.txt').readlines()
for i in iplist:
time.sleep(2)
print "Country: IP {0}".format(i)
print sh.grep(sh.whois(i, h="whois.arin.net"), "Country")