We Keep Coding

Which is the error inside the quotes Python 2.7

I am trying to configure Ansible in an automatic way with the following def:
def configure_ansible():
with open('/etc/hosts', 'r') as f:
valid_ips=[line.split(None, 1)[0] for line in f]
if os.path.isfile('/etc/ansible/hosts'):
open('/etc/ansible/hosts', 'w').close()
os.system('cp /etc/hosts /etc/ansible/hosts')
for valid_ip in valid_ips:
os.system("sudo sed -i '14 s/^#//g' /etc/ansible/ansible.cfg")
os.system("sudo sed -i '22 s/^#//g' /etc/ansible/ansible.cfg")
if valid_ip == "localhost":
os.system("su - ansible -c 'echo -e '\n\n\n' | ssh-keygen -t rsa'")
os.system("su - ansible -c 'ssh-copy-id ansible#"+valid_ip)
Looks like the problem is inside the quotes of the last "if". Any idea how I can solve it?
UPDATE
I have followed chepner's recommendation, but the last line is not working properly. If I am using the code as below, the ssh-copy-id is not performed correctly and the ssh keys are not exchanged. I would need to introduce also the password to fully automate this process. Any idea how I can accomplish this?
Here is what I have tried:
def create_user():
users=["dante", "ansible"]
with open('/etc/hosts', 'r') as f:
valid_ips=[line.split(None, 1)[0] for line in f]
for valid_ip in valid_ips:
for user in users:
subprocess.call(["sudo", "useradd", user])
passwd_users = subprocess.Popen(["sudo", "passwd", user], stdin = subprocess.PIPE)
passwd_users.communicate(input = "test123\ntest123")
sudoers = open("/etc/sudoers", 'a')
sudoers.write(user + " ALL=(ALL) NOPASSWD: ALL \n")
sudoers.close()
def configure_ansible():
with open('/etc/hosts', 'r') as f:
valid_ips=[line.split(None, 1)[0] for line in f]
if os.path.isfile('/etc/ansible/hosts'):
open('/etc/ansible/hosts', 'w').close()
os.system('cp /etc/hosts /etc/ansible/hosts')
config = "/etc/ansible/ansible.cfg"
for valid_ip in valid_ips:
subprocess.call(["sudo", "sed", "-i", "14 s/^#//g", config])
subprocess.call(["sudo", "sed", "-i", "22 s/^#//g", config])
if valid_ip == "localhost":
keygen = subprocess.Popen(["sudo", "-u", "ansible", "ssh-keygen", "-t", "rsa"], stdin = subprocess.PIPE)
keygen.communicate(input = "\n\n\n")
copy_keygen = subprocess.Popen(["sudo", "-u", "ansible", "ssh-copy-id", "-o StrictHostKeyChecking=no", valid_ip], stdin = subprocess.PIPE)
copy_keygen.stdin.write('test123\n')

You are missing the closing single quote in that call to os.system:
os.system("su - ansible -c 'ssh-copy-id ansible#"+valid_ip+"'")
However, shell doesn't allow you to nest single quotes; the previous call should look something like:
os.system("su - ansible -c 'echo -e \"\n\n\n\" | ssh-keygen -t rsa'")
Even better, prefer subprocess.call to os.system in all cases:
config = "/etc/ansible/ansible.cfg"
for valid_ip in valid_ips:
subprocess.call(["sudo", "sed", "-i", "14 s/^#//g", config])
subprocess.call(["sudo", "sed", "-i", "22 s/^#//g", config])
if valid_ip == "localhost":
p = subprocess.Popen(["sudo", "-u", "ansible", "ssh-keygen", "-t", "rsa"])
p.communicate("\n\n\n")
subprocesss.call(["sudo", "-u", "ansible", "ssh-copy-id", "ansible#" + valid_ip])

Subprocess run/call not working

I want to use the "raster2pgsql" utility in my Python code. When I use it in a Linux terminal, it works fine. This is the command:
$ raster2pgsql -a "/mnt/c/Users/Jan/path/to/raster/dem.tiff" test_schema.raster2 | psql -h localhost -d pisl -U pisl
Then I use subprocess.run (I have also tried subprocess.call) to use this same tool in my Python code. This is my code:
from subprocess import run
command = ["raster2pgsql", "-a", '"' + file_name + '"', self.schema_name + "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command)
I get this error:
ERROR: Unable to read raster file: "/mnt/c/Users/Jan/path/to/raster/dem.tiff"
Printing command gives this which I think is correct (equivalent to what worked in the terminal):
['raster2pgsql', '-a', '"/mnt/c/Users/Jan/path/to/raster/dem.tiff"', 'test_schema.raster2', '|', 'psql', '-h', 'localhost', '-p', '5432', '-d', 'pisl']
I have double checked that the path to the raster file is correct, tried single quotes, double quotes but nothing helps. I have looked at a number of similar question (here, here or here ) but did not find anything helpful.
I use Python 3.5 and Linux Bash Shell in Windows 10.
Question: What is wrong with the way I use subprocess?

2 issues here:
no need to extra-quote the filename. It's passed to the system literally and since there's no file called "/tmp/something" the command fails.
second, to be able to pass a pipe, you need shell=True
so quickfix:
command = ["raster2pgsql", "-a", file_name, self.schema_name + "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command,shell=True)
or using a command string (because shell=True is picky with argument list):
command = "raster2pgsql -a "+ file_name + " " + self.schema_name + "." + identifier + " | psql -h localhost -p 5432 -d" + self.dbname
run(command,shell=True)
(ugly, isn't it?)
It's much better to run 2 processes without shell=True and pipe them together using python, more portable & secure (not sure how shell=True reacts with an argument list on Linux):
from subprocess import *
command1 = ["raster2pgsql", "-a", file_name, self.schema_name + "." + identifier]
p = Popen(command1,stdout=PIPE)
command2 = ["psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command2,stdin=p.stdout)
The first Popen object created writes its output to a pipe (thanks to stdout=PIPE argument). The run function can take an input as a pipe too (thanks to stdin=p.stout). It consumes the output of the first command, creating a native piped chain of commands, without the need of the shell (and the caveats of quoting, spaces, special character interpretation and such...)

How to use subprocess Popen?

I'm trying to execute a command using Popen.
The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:
"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres
When using within Python, I make it a string with the ' codes:
command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'
attempting to execute it results in an error:
p = subprocess.Popen(command)
ERROR: Unable to read raster file: test
The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)
Am I using Popen wrong?

Your command uses pipe |. It requires a shell:
p = subprocess.Popen(command, shell=True)
The command itself as far as I can tell looks ok.

It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.
raster2pgsql_ps = subprocess.Popen([
'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
'C:\\temp\\SampleDTM\\SampleDTM.tif',
'test'
], stdout=subprocess.PIPE)
# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
'psql',
'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)

The following worked for me on Windows, while avoiding shell=True
One can make use of Python's fstring formatting to make sure the commands will work in windows.
Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.
Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.
import os
import subprocess
shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")
command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'
try:
shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
sys.stderr.write("An error occurred while importing data into the database, you might want to \
check the SQL command below:")
sys.stderr.write(command)
raise
To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.

PIPE = subprocess.PIPE
pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
stdout=PIPE, stderr=PIPE)
stdout, stderr = pd.communicate()

It will be better to use subprocess.Popen in this way:
proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()
B.T.W, it's good to format the path first, use:
path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")
this will avoid some path problems for different OS platform.
The shell = True is important if you want to execute your command just like executing it in local shell.
Hope will help you.

python subprocess.popen tshark works with some args, but not others

When I execute a python script with this:
tsharkCall = ["tshark", "-a", "duration:6", "-i", "2", "-w", "thsark.pcap"]
tsharkProc = subprocess.Popen(tsharkCall, bufsize=0, executable="C:\\Program Files\\Wireshark\\tshark.exe")
A pcap file with the expected contents duly appears in the same folder as the script.
A second procedure to create a text file from the pcap does not work:
tsharkCall = ["tshark", "-i", "-", "<", "tshark.pcap", ">", "tshark.txt", "-V"]
tsharkProc = subprocess.Popen(tsharkCall, bufsize=0, executable="C:\\Program Files\\Wireshark\\tshark.exe")
I see "Capturing on Standard input" in the cmd window, but no "x packets captured", and no tshark.txt file appears in the folder.
From a command prompt in the same location, this does the job I am hoping for from the script:
>"C:\Program Files\Wireshark\tshark.exe" -i - < "tshark.pcap" > "tshark.txt" -V
It seems odd that one call works and the other doesn't. Any ideas as to what I'm missing?

subprocess.Popen by default bypasses CMD.EXE / sh, therefore command line I/O redirections (<, >) will not work. You can get a similar effect like this:
tsharkCall = ["tshark", "-i", "-", "-V"]
tsharkIn = open("tshark.pcap", "rb")
tsharkOut = open("tshark.txt", "wb")
tsharkProc = subprocess.Popen(tsharkCall,
stdin=tsharkIn,
stdout=tsharkOut,
executable="C:\\Program Files\\Wireshark\\tshark.exe")

This also works
tsharkCall = ["C:\\Program Files\\Wireshark\\tshark.exe", "-P", "-V", "-x", "-r", "C:\\Data\\PCAP_TEST_FILES\\test.pcap"]
tsharkOut = open("tshark.txt", "wb")
tsharkProc = subprocess.call(tsharkCall, stdout=tsharkOut)

Run shell command with input redirections from python 2.4?

What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.

You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)

The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename

As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.