I'm trying to execute a command using Popen.
The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:
"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres
When using within Python, I make it a string with the ' codes:
command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'
attempting to execute it results in an error:
p = subprocess.Popen(command)
ERROR: Unable to read raster file: test
The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)
Am I using Popen wrong?
Your command uses pipe |. It requires a shell:
p = subprocess.Popen(command, shell=True)
The command itself as far as I can tell looks ok.
It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.
raster2pgsql_ps = subprocess.Popen([
'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
'C:\\temp\\SampleDTM\\SampleDTM.tif',
'test'
], stdout=subprocess.PIPE)
# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
'psql',
'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)
The following worked for me on Windows, while avoiding shell=True
One can make use of Python's fstring formatting to make sure the commands will work in windows.
Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.
Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.
import os
import subprocess
shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")
command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'
try:
shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
sys.stderr.write("An error occurred while importing data into the database, you might want to \
check the SQL command below:")
sys.stderr.write(command)
raise
To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.
PIPE = subprocess.PIPE
pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
stdout=PIPE, stderr=PIPE)
stdout, stderr = pd.communicate()
It will be better to use subprocess.Popen in this way:
proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()
B.T.W, it's good to format the path first, use:
path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")
this will avoid some path problems for different OS platform.
The shell = True is important if you want to execute your command just like executing it in local shell.
Hope will help you.
Related
Based on this tutorial to build a TF image classifier, I have a bash shell in which I run a Docker image with the following command:
docker run --name fooldocker -it -v $HOME/tf_files:/tf_files/ gcr.io/tensorflow/tensorflow:latest-devel
And then in this docker image I run my Python script:
python /tf_files/label_image.py /tf_files/myimages
exit
It works.
But now, I need to automate these commands in a Python script. I tried :
p = Popen(['docker', 'run', '--rm', '--name', 'fooldocker','-it', '-v', '$HOME/tf_files:/tf_files/', 'gcr.io/tensorflow/tensorflow:latest-devel'], stdout=PIPE)
p = Popen(['docker', 'exec', 'fooldocker', 'python', '/tf_files/label_NES.py', '/tf_files/NES/WIP'])
p = Popen(['docker', 'kill', 'fooldocker'], shell=True, stdout=PIPE, stderr=PIPE)
p = Popen(['docker', 'rm', 'fooldocker'], shell=True, stdout=PIPE, stderr=PIPE)
Leading to this error after Popen #2 is run :
docker: Error response from daemon: create $HOME/tf_files: "$HOME/tf_files" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
The problem is that $HOME cannot be evaluated in this single quotes string. Either try doublequotes, or evaluate the variable beforehand and put it into the command string.
Also: If you set shell=True, you don't split your command into a list:
p = Popen('docker kill fooldocker', shell=True, stdout=PIPE, stderr=PIPE)
it is because Popen didn't interpret $HOME to your home path.
and it is the string $HOME and pass to docker command which not allow $ in a volume name.
maybe you can use subprocess module for convenience, for example:
import subprocess
subprocess.call("echo $HOME", shell=True)
it interpreted $HOME if shell=True specified.
Using the subprocess module how do I get the following command to work?
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
This command works perfectly when I run it in a bash shell but I can't get it to work when running from within Python. I'm trying to do this from within Python because I need to be able to modify the query and get different result sets back and then process them in Python. I can't use one of the nice Python database connectors for various reasons which leaves me trying to pipe output from isql.
My code currently looks similar to the following:
bash_command = '''
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
'''
process = subprocess.Popen(bash_command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, error = process.communicate()
However I have tried lots of variations:
Using the entire command as a string, or as a list of strings.
Using check_output vs Popen.
Using communicate() to try and send the query to the isql command or having the query be part of the command string using a heredoc.
Using shell = True or not.
Specifying /bin/bash or using the default /bin/sh.
Lots of different quoting and escaping patterns.
And pretty much every permutation of the above.
In no case do I receive the output of the query that I'm looking for. I'm pretty sure that the command isn't being sent to the shell as is but I can't tell what is being sent to the shell.
I feel like this should be pretty simple, send a command to the shell and get the output back, but I just can't make it work. I can't even see what command is being sent to the shell, even using pdb.
shell=True makes subprocess use /bin/sh by default. <<< "here-string" is a bash-ism; pass executable='/bin/bash':
>>> import subprocess
>>> subprocess.call(u'cat <<< "\u0061"', shell=True)
/bin/sh: 1: Syntax error: redirection unexpected
2
>>> subprocess.call(u'cat <<< "\u0061"', shell=True, executable='/bin/bash')
a
0
You should also use raw-string literals to avoid escaping backslashes: "\\u0061" == r"\u0061" != u"\u0061":
>>> subprocess.call(r'cat <<< "\u0061"', shell=True, executable='/bin/bash')
\u0061
0
Though you don't need shell=True here. You could pass the input as a string using process.communicate(input=input_string):
>>> process = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.communicate(br"\u0061")
('\\u0061', None)
The result could look like:
#!/usr/bin/env python
import shlex
from subprocess import Popen, PIPE
cmd = shlex.split(r'isql -v -b -d, DSN_NAME "DOMAIN\username" password')
process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, errors = process.communicate(
b"SELECT column_name, data_type "
b"FROM database_name.information_schema.columns "
b"WHERE table_name = 'some_table';")
Try giving this a shot:
import shlex
from subprocess import Popen, PIPE, STDOUT
sql_statement = '''"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"'''
isqlcommand = 'isql -v -b -d, DSN_NAME "DOMAIN\username" password'
isqlcommand_args = shlex.split(isqlcommand)
process = Popen(isqlcommand_args, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
output = process.communicate(input=sql_statement)[0]
print output
The idea here is to separate the here-string redirection from the isql command execution. This example will pipe the here-string into the stdin of process via process.communicate(). I'm also using shlex.split() to tokenize the command and its arguments.
Edit: Removed Shell=True after reviewing comment from J.F. Sebastian
I am trying to automate the process of generating an SSH RSA key.When the command below is run on the command line it generates the key correctly:
ssh-keygen -f /root/.ssh/id_rsa -t rsa -N ''
The issue arises when I try to replicate this in a python 2 script.
if not os.path.isfile('/root/.ssh/id_rsa.pub'):
fullcmd = ['ssh-keygen', '-f', '/root/.ssh/id_rsa', '-t', 'rsa', '-N', '\'\'']
sshcmd = subprocess.Popen(fullcmd,
shell= False,
stdout= subprocess.PIPE,
stderr= subprocess.STDOUT)
out = sshcmd.communicate()[0].split('\n')
for lin in out:
print lin
When I run this script I get this error:
> passphrase too short: have 2 bytes, need >4
> Generating public/private
> rsa key pair. Saving the key failed: /root/.ssh/id_rsa.
Why dose it work when I execute it on the command line and not through the python script?
fullcmd = ['ssh-keygen', '-f', '/root/.ssh/id_rsa', '-t', 'rsa', '-N', '\'\'']
'' is shell syntax for passing an empty string. You're invoking the command directly, not via a shell, so don't pass single quotes. Just pass the empty string itself.
fullcmd = ['ssh-keygen', '-f', '/root/.ssh/id_rsa', '-t', 'rsa', '-N', '']
I have a rsync command like this:
rsync --recursive source | grep "\."
I want to call this in subprocess like this:
subprocess.Popen(['sshpass', '-p', password, 'rsync', '--recursive', source, '|', 'grep', '"\."'],
stdout=subprocess.PIPE).communicate()[0]
But it won't work. What's the correct way?
You need to invoke the shell with shell=True. However, the documentation discourages using shell=True.
In order to avoid using shell=True, you can first create rsync process and pipe it's output to grep:
rsync_out = subprocess.Popen(['sshpass', '-p', password, 'rsync', '--recursive', source], stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', '\.'), stdin=rsync_out.stdout)
I'm trying to use wget with subprocess.
my attempts worked until I tried to download the page to a specified directory with this code:
url = 'google.com'
location = '/home/patrick/downloads'
args = ['wget', 'r', 'l 1' 'p' 'P %s' % location, url]
output = Popen(args, stdout=PIPE)
if I run this code in /home/patrick I get index.html in /home/patrick and not in /home/patrick/downloads.
Can you help me?
Thanks ;)
You need to have hyphens and location should be just another argument:
args = ['wget', '-r', '-l', '1', '-p', '-P', location, url]
Edit: popen from os intends to replace os.popen module. Hence, using os.popen is not recommended
Initially I thought it was popen from os.
If you are using popen from os
#wget 'http://google.com/' -r -l 1 -p -P /Users/abhinay/Downloads
from os import popen
url = 'google.com'
location = '/Users/abhinay/Downloads'
args = ['wget %s', '-r', '-l 1', '-p', '-P %s' % location, url]
output = popen(' '.join(args))
and using Popen from subprocess
#wget 'http://google.com/' -r -l 1 -p -P Downloads/google
from subprocess import Popen
url = 'google.com'
location = '/Users/abhinay/Downloads'
#as suggested by #SilentGhost the `location` and `url` should be separate argument
args = ['wget', '-r', '-l', '1', '-p', '-P', location, url]
output = Popen(args, stdout=PIPE)
Let me know if I'm missing something.
Thx!