I want to use the "raster2pgsql" utility in my Python code. When I use it in a Linux terminal, it works fine. This is the command:
$ raster2pgsql -a "/mnt/c/Users/Jan/path/to/raster/dem.tiff" test_schema.raster2 | psql -h localhost -d pisl -U pisl
Then I use subprocess.run (I have also tried subprocess.call) to use this same tool in my Python code. This is my code:
from subprocess import run
command = ["raster2pgsql", "-a", '"' + file_name + '"', self.schema_name + "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command)
I get this error:
ERROR: Unable to read raster file: "/mnt/c/Users/Jan/path/to/raster/dem.tiff"
Printing command gives this which I think is correct (equivalent to what worked in the terminal):
['raster2pgsql', '-a', '"/mnt/c/Users/Jan/path/to/raster/dem.tiff"', 'test_schema.raster2', '|', 'psql', '-h', 'localhost', '-p', '5432', '-d', 'pisl']
I have double checked that the path to the raster file is correct, tried single quotes, double quotes but nothing helps. I have looked at a number of similar question (here, here or here ) but did not find anything helpful.
I use Python 3.5 and Linux Bash Shell in Windows 10.
Question: What is wrong with the way I use subprocess?
2 issues here:
no need to extra-quote the filename. It's passed to the system literally and since there's no file called "/tmp/something" the command fails.
second, to be able to pass a pipe, you need shell=True
so quickfix:
command = ["raster2pgsql", "-a", file_name, self.schema_name + "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command,shell=True)
or using a command string (because shell=True is picky with argument list):
command = "raster2pgsql -a "+ file_name + " " + self.schema_name + "." + identifier + " | psql -h localhost -p 5432 -d" + self.dbname
run(command,shell=True)
(ugly, isn't it?)
It's much better to run 2 processes without shell=True and pipe them together using python, more portable & secure (not sure how shell=True reacts with an argument list on Linux):
from subprocess import *
command1 = ["raster2pgsql", "-a", file_name, self.schema_name + "." + identifier]
p = Popen(command1,stdout=PIPE)
command2 = ["psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command2,stdin=p.stdout)
The first Popen object created writes its output to a pipe (thanks to stdout=PIPE argument). The run function can take an input as a pipe too (thanks to stdin=p.stout). It consumes the output of the first command, creating a native piped chain of commands, without the need of the shell (and the caveats of quoting, spaces, special character interpretation and such...)
Related
I want to send a certificate to the remote server automatically with ssh-copy-id. I chose the python subprocess library for this, but somehow it does not send the password to the terminal.
I am aware that I can do this with sshpass or paramiko, but I don't want to choose it unless I have to. Can you help me with this? My code is below.
from subprocess import run,PIPE
send_cert = run(['ssh-copy-id', '-i', '~/.ssh/id_rsa.pub','pardus'], stdout=PIPE, input=input_cert, encoding='utf-8')
input_cert = '1'
pardus is my remote host's name. You can replace user#IP .
~ is replaced with the home directory by the shell, but you're not executing the command through a shell, so it's being interpreted literally.
You can use the os.path.expanduser() function to perform this substitution.
import os
from subprocess import run,PIPE
send_cert = run(['ssh-copy-id', '-i', os.path.expanduser('~/.ssh/id_rsa.pub'),'pardus'], stdout=PIPE, input=input_cert, encoding='utf-8')
I solved.
send_pass = 'PASSWORD' + '\n'
send_cert = 'ssh-copy-id -i ' + 'CERT_PATH' + ' ' + 'USER#HOSTNAME'
child = pexpect.spawn(send_cert,encoding='utf-8')
child.expect('password:')
child.sendline(send_pass)
time.sleep(2)
Thanks all.
I was trying to run the next command in my python code:
iostat -d 7 7 -p sda | awk '!/^Device/' | awk '!/^Linux/'
The way I tried it so far was this:
command = ["iostat", "-d", "7", "7", "-p", "sda", "|", "awk", "'!/^Device/'", "|", "awk", "'!/^Linux/'"]
device = subprocess.Popen(command, stdout=subprocess.PIPE)
devicetr = device.stdout.read()
It seems that the code won't handle the quotes marks in "'!/^Device/'" and in "'!/^Linux/'" like '!/^Device/' and '!/^Linux/' as intended.
When there is no USB connected, instead of blank I get this:
Linux 4.14.98-v7+ 01/28/2020 _armv7l_ (4 CPU)
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
Tried already adding '\' and this:
Passing double quote shell commands in python to subprocess.Popen()?
You don't need to add additional quotes because subprocess will automatically add them when needed.
command = ["iostat", "-d", "7", "7", "-p", "sda", "|", "awk", "!/^Device/", "|", "awk", "!/^Linux/"]
device = subprocess.Popen(command, stdout=subprocess.PIPE)
devicetr = device.stdout.read() # also here was a typo red -> read
As #noufal-ibrahim pointer out | doesn't work this way. You need to pipe your commands:
ps1 = subprocess.Popen(["iostat"], stdout=subprocess.PIPE)
ps2 = subprocess.Popen(["awk", "!/^ *KB/"], stdin=ps1.stdout, stdout=subprocess.PIPE) # filter out line with KB and extra space
print(ps2.stdout.read().decode())
^ This code works on my machine, you can adjust it to your use case.
Appendix. And instead of manually creating command list you can use shlex lib (from standard python lib):
>>> import shlex
>>> shlex.split("python foo.py 'multi word arg'")
['python', 'foo.py', 'multi word arg']
I'm trying to execute a command using Popen.
The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:
"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres
When using within Python, I make it a string with the ' codes:
command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'
attempting to execute it results in an error:
p = subprocess.Popen(command)
ERROR: Unable to read raster file: test
The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)
Am I using Popen wrong?
Your command uses pipe |. It requires a shell:
p = subprocess.Popen(command, shell=True)
The command itself as far as I can tell looks ok.
It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.
raster2pgsql_ps = subprocess.Popen([
'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
'C:\\temp\\SampleDTM\\SampleDTM.tif',
'test'
], stdout=subprocess.PIPE)
# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
'psql',
'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)
The following worked for me on Windows, while avoiding shell=True
One can make use of Python's fstring formatting to make sure the commands will work in windows.
Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.
Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.
import os
import subprocess
shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")
command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'
try:
shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
sys.stderr.write("An error occurred while importing data into the database, you might want to \
check the SQL command below:")
sys.stderr.write(command)
raise
To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.
PIPE = subprocess.PIPE
pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
stdout=PIPE, stderr=PIPE)
stdout, stderr = pd.communicate()
It will be better to use subprocess.Popen in this way:
proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()
B.T.W, it's good to format the path first, use:
path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")
this will avoid some path problems for different OS platform.
The shell = True is important if you want to execute your command just like executing it in local shell.
Hope will help you.
I have this commands in bash:
ACTIVE_MGMT_1=ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;
I was trying to do it in Python like this:
active_mgmgt_1 = os.popen("""ssh -n MGMT_IP_1 ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2""") ACTIVE_MGMT_1 = active_mgmgt_1.read().replace('\n', '')
It doesn't work; any advice please?
Your popen call needs be set to communicate via a pipe.
Also stop trying to put everything on one line - python doesn't require it and places a lot of empasis on readable code.
I would strongly suggest doing the string processing in python rather than egrep, (use find or re in python), awk (find or egrep) and cut (string split).
It is also recommended to use subprocess.Popen rather than os.popen functions. There is a suggestion to use shlex.spilt to clear up this sort of issue.
untested code
import subprocess
import re
import os
MGMT_IP_1 = os.getenv('MGMT_IP_1')
sp = subprocess.Popen(
['ssh', '-n', MGMT_IP_1, '. .bash_profile; xms sho proc TRAF.*'],
stdout=PIPE, stderr=None)
(result, outtext) = sp.communicate()
# Proceed to process outtext from here using re, find and split
# to the equivalent of egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;
What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.
You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)
The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename
As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.