We Keep Coding

Subprocess run/call not working

I want to use the "raster2pgsql" utility in my Python code. When I use it in a Linux terminal, it works fine. This is the command:
$ raster2pgsql -a "/mnt/c/Users/Jan/path/to/raster/dem.tiff" test_schema.raster2 | psql -h localhost -d pisl -U pisl
Then I use subprocess.run (I have also tried subprocess.call) to use this same tool in my Python code. This is my code:
from subprocess import run
command = ["raster2pgsql", "-a", '"' + file_name + '"', self.schema_name + "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command)
I get this error:
ERROR: Unable to read raster file: "/mnt/c/Users/Jan/path/to/raster/dem.tiff"
Printing command gives this which I think is correct (equivalent to what worked in the terminal):
['raster2pgsql', '-a', '"/mnt/c/Users/Jan/path/to/raster/dem.tiff"', 'test_schema.raster2', '|', 'psql', '-h', 'localhost', '-p', '5432', '-d', 'pisl']
I have double checked that the path to the raster file is correct, tried single quotes, double quotes but nothing helps. I have looked at a number of similar question (here, here or here ) but did not find anything helpful.
I use Python 3.5 and Linux Bash Shell in Windows 10.
Question: What is wrong with the way I use subprocess?

2 issues here:
no need to extra-quote the filename. It's passed to the system literally and since there's no file called "/tmp/something" the command fails.
second, to be able to pass a pipe, you need shell=True
so quickfix:
command = ["raster2pgsql", "-a", file_name, self.schema_name + "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command,shell=True)
or using a command string (because shell=True is picky with argument list):
command = "raster2pgsql -a "+ file_name + " " + self.schema_name + "." + identifier + " | psql -h localhost -p 5432 -d" + self.dbname
run(command,shell=True)
(ugly, isn't it?)
It's much better to run 2 processes without shell=True and pipe them together using python, more portable & secure (not sure how shell=True reacts with an argument list on Linux):
from subprocess import *
command1 = ["raster2pgsql", "-a", file_name, self.schema_name + "." + identifier]
p = Popen(command1,stdout=PIPE)
command2 = ["psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command2,stdin=p.stdout)
The first Popen object created writes its output to a pipe (thanks to stdout=PIPE argument). The run function can take an input as a pipe too (thanks to stdin=p.stout). It consumes the output of the first command, creating a native piped chain of commands, without the need of the shell (and the caveats of quoting, spaces, special character interpretation and such...)

redircet stdout to file using variable python 3

I want to redirect o/p of shell commands to file using variable "path" but it is not working
import os, socket, shutil, subprocess
host = os.popen("hostname -s").read().strip()
path = "/root/" + host
if os.path.exists(path):
print(path, "Already exists")
else:
os.mkdir("Directory", path , "Created")
os.system("uname -a" > path/'uname') # I want to redirect o/p of shell commands to file using varibale "path" but it is not working
os.system("df -hP"> path/'df')

I think the problem is the bare > and / symbols in the os.system command...
Here is a python2.7 example with os.system that does what you want
import os
path="./test_dir"
command_str="uname -a > {}/uname".format(path)
os.system(command_str)

Here's a very minimal example using subprocess.run. Also, search StackOverflow for "python shell redirect", and you'll get this result right away:
Calling an external command in Python
import subprocess
def run(filename, command):
with open(filename, 'wb') as stdout_file:
process = subprocess.run(command, stdout=subprocess.PIPE, shell=True)
stdout_file.write(process.stdout)
return process.returncode
run('test_out.txt', 'ls')

Download files via curl using python's subprocess

How is it that when I write this in my terminal the file is downloaded,
curl "http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p50.pl?file=gfs.t00z.pgrb2full.0p50.f000&lev_10_m_above_ground=on&var_UGRD=on&var_VGRD=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&dir=%2Fgfs.2016121900" -o "tmp_folder/gfs.t00z.pgrb2full.0p50.f000"
but using python's subprocess module, the download just hangs?
import subprocess
URL = "http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p50.pl?file=gfs.t00z.pgrb2full.0p50.f000&lev_10_m_above_ground=on&var_UGRD=on&var_VGRD=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&dir=%2Fgfs.2016121900"
pipe = subprocess.Popen("curl " + URL + " -o" + " my_file", shell = True)
pipe.communicate()
What am I missing?
Thank you

The URL is probably not properly quoted, so it is interpreted by the shell (with all the & chars...)
Better run subprocess with explicit arguments as a list:
pipe = subprocess.Popen(["curl",URL,"-o","my_file"])
shell=True can probably be omitted. Since you're not using any shell capabilities, it will save you trouble.

rather than creating a Popen instance, you can just use the call method. Don't pass shell=True... just split the command with shlex so the args can be passed as a list.
import shlex
import subprocess
cmd = 'curl "http://foo.com" -o "foo.txt"'
subprocess.call(shlex.split(cmd))

subprocess.popen seems to fail when run from crontab

I'm running a script from crontab that will just ssh and run a command and store the results in a file.
The function that seems to be failing is subprocess.popen.
Here is the python function:
def _executeSSHCommand(sshcommand,user,node):
'''
Simple function to execute an ssh command on a remote node.
'''
sshunixcmd = '/usr/bin/ssh %s#%s \'%s\'' % (user,node,sshcommand)
process = subprocess.Popen([sshunixcmd],
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
process.wait()
result = process.stdout.readlines()
return result
When it's run from the command line, it executes correctly, from cron it seems to fail with the error message below.
Here are the crontab entries:
02 * * * * /home/matt/scripts/check-diskspace.py >> /home/matt/logs/disklog.log
Here are the errors:
Sep 23 17:02:01 timmy CRON[13387]: (matt) CMD (/home/matt/scripts/check-diskspace.py >> /home/matt/logs/disklog.log)
Sep 23 17:02:01 timmy CRON[13386]: (CRON) error (grandchild #13387 failed with exit status 2)
I'm going blind trying to find exactly where I have gone so wrong. Any ideas?

The cron PATH is very limited. You should either set absolute path to your ssh /usr/bin/ssh or set the PATH as a first line in your crontab.

You probably need to pass ssh the -i argument to tell ssh to use a specific key file. The problem is that your environment is not set up to tell ssh which key to use.
The fact that you're using python here is a bit of a red herring.

For everything ssh-related in python, you might consider using paramiko. Using it, the following code should do what you want.
import paramiko
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect(node, username=user)
stdout = client.exec_command(ssh_command)[0]
return stdout.readlines()

When running python scripts from cron, the environment PATH can be a hangup, as user1652558 points out.
To expand on this answer with example code to add custom PATH values to the environment for a subprocess call:
import os
import subprocess
#whatever user PATH values you need
my_path = "/some/custom/path1:/some/custom/path2"
#append the custom values to the current PATH settings
my_env = os.environ.copy()
my_env["PATH"] = my_path + ":" + my_env["PATH"]
#subprocess call
resp = subprocess.check_output([cmd], env=my_env, shell=True)

Run shell command with input redirections from python 2.4?

What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.

You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)

The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename

As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.