I have a rsync command like this:
rsync --recursive source | grep "\."
I want to call this in subprocess like this:
subprocess.Popen(['sshpass', '-p', password, 'rsync', '--recursive', source, '|', 'grep', '"\."'],
stdout=subprocess.PIPE).communicate()[0]
But it won't work. What's the correct way?
You need to invoke the shell with shell=True. However, the documentation discourages using shell=True.
In order to avoid using shell=True, you can first create rsync process and pipe it's output to grep:
rsync_out = subprocess.Popen(['sshpass', '-p', password, 'rsync', '--recursive', source], stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', '\.'), stdin=rsync_out.stdout)
Related
I would like to run a command in python using subprocess.run
I would like to switch the working directory JUST for the execution of this command.
Also, I need to record the output and the return code.
Here is the code I have:
import subprocess
result = subprocess.run("echo \"blah\"", cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
but this only returns
FileNotFoundError: [Errno 2] No such file or directory: 'echo "Running ls -la" && ls -la'
I also tried using the following arguments:
subprocess.run(["echo", "\"blah\""], cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Like Jean-François Fabre said, the solution is to add "shell=True" to the call
import subprocess
result = subprocess.run("echo \"blah\"", cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
shell=True seems to tell subprocess to use the string as a command.
I have a file: "abc", it's a executable file.
I want to execute it by Phthon or windows CMD.
If I write code:
subprocess.Popen('-a -b -c', creationflags=0x08, shell=True, executable="C:\\abc")
Then, abc executed, but params(-a -b -c) been ignored
So...How Can I reslove it?
Why not change your code to the following version
args = ['C:\\abc', '-a', '-b', '-c']
p = subprocess.Popen(' '.join(args), shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
or without shell=True you can further reduce it to
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
or you can use another method to get output from your exe and pass arguments to it
Output = subprocess.check_output(' '.join(args), shell=True).decode()
Finally, I found the answer myself, which is to use win32process.CreateProcess
Thanks to other authors, but their answers are wrong.
The correct answer is:
win32process.CreateProcess(r"C:\abc", "-a -b -c", None, None, 0, 0, None, None, win32process.STARTUPINFO())
The following answer does not work:
subprocess.Popen('-a -b -c', creationflags=0x08, shell=True, executable="C:\\abc")
#or
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Maybe I really don’t know how to use subprocess, but Windows is really special. Executable files without a suffix cannot be executed directly on the command line.
In PowerShell
& "C:\abc.exe"
I cant figure out how to start the server using a python command.
s = subprocess.Popen('"D:\MC SERVER 2k19\server.jar" -jar server.jar java', stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
This code runs without error but doesn't start the server in cmd.
Thanks.
It's got to do with how you're passing your arguments.
subprocess.Popen(['java', '-jar', 'server.jar'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True, cwd='D:\MC SERVER 2k19')
If you need to start a Java program using CMD process from Python and show the window, you can use subprocess to call open another CMD terminal and run the command.
In Windows you will need to CMD-escape spaces in the path you passing to the secondary CMD process. This is done with the carrot ^
proc = subprocess.Popen(
['start', 'cmd', '/k', "D:\\MC^ SERVER^ 2k19\\server.jar",
'-jar', 'server.jar', 'java'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
Keep in mind you will NOT be able to retrieve any output from the secondary CMD process from Python.
I.e. the process will return nothing.
proc.communicate()
# returns:
(b'', b'')
I am trying to call an executable called foo, and pass it some command line arguments. An external script calls into the executable and uses the following command:
./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log
The script uses Popen to execute this command as follows:
from subprocess import Popen
from subprocess import PIPE
def run_command(command, returnObject=False):
cmd = command.split(' ')
print('%s' % cmd)
p = None
print('command : %s' % command)
if returnObject:
p = Popen(cmd)
else:
p = Popen(cmd)
p.communicate()
print('returncode: %s' % p.returncode)
return p.returncode
return p
command = "./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log
"
run_command(command)
However, this passes extra arguments ['2>&1', '|', '/usr/bin/tee', 'temp.log'] to the foo executable.
How can I get rid of these extra arguments getting passed to foo while maintaining the functionality?
I have tried shell=True but read about avoiding it for security purposes (shell injection attack). Looking for a neat solution.
Thanks
UPDATE:
- Updated the file following the tee command
The string
./main/foo --config config_file 2>&1 | /usr/bin/tee >temp.log
...is full of shell constructs. These have no meaning to anything without a shell in play. Thus, you have two options:
Set shell=True
Replace them with native Python code.
For instance, 2>&1 is the same thing as passing stderr=subprocess.STDOUT to Popen, and your tee -- since its output is redirected and it's passed no arguments -- could just be replaced with stdout=open('temp.log', 'w').
Thus:
p = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=open('temp.log', 'w'))
...or, if you really did want the tee command, but were just using it incorrectly (that is, if you wanted tee temp.log, not tee >temp.log):
p1 = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['tee', 'temp.log'], stdin=p1.stdout)
p1.stdout.close() # drop our own handle so p2's stdin is the only handle on p1.stdout
stdout, _ = p2.communicate()
Wrapping this in a function, and checking success for both ends might look like:
def run():
p1 = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['tee', 'temp.log'], stdin=p1.stdout)
p1.stdout.close() # drop our own handle so p2's stdin is the only handle on p1.stdout
# True if both processes were successful, False otherwise
return (p2.wait() == 0 && p1.wait() == 0)
By the way -- if you want to use shell=True and return the exit status of foo, rather than tee, things get a bit more interesting. Consider the following:
p = subprocess.Popen(['bash', '-c', 'set -o pipefail; ' + command_str])
...the pipefail bash extension will force the shell to exit with the status of the first pipeline component to fail (and 0 if no components fail), rather than using only the exit status of the final component.
Here's a couple of "neat" code examples in addition to the explanation from #Charles Duffy answer.
To run the shell command in Python:
#!/usr/bin/env python
from subprocess import check_call
check_call("./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log",
shell=True)
without the shell:
#!/usr/bin/env python
from subprocess import Popen, PIPE, STDOUT
tee = Popen(["/usr/bin/tee", "temp.log"], stdin=PIPE)
foo = Popen("./main/foo --config config_file".split(),
stdout=tee.stdin, stderr=STDOUT)
pipestatus = [foo.wait(), tee.wait()]
Note: don't use "command arg".split() with non-literal strings.
See How do I use subprocess.Popen to connect multiple processes by pipes?
You may combine answers to two StackOverflow questions:
1. piping together several subprocesses
x | y problem
2. Merging a Python script's subprocess' stdout and stderr (while keeping them distinguishable)
2>&1 problem
I'm trying to execute a command using Popen.
The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:
"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres
When using within Python, I make it a string with the ' codes:
command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'
attempting to execute it results in an error:
p = subprocess.Popen(command)
ERROR: Unable to read raster file: test
The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)
Am I using Popen wrong?
Your command uses pipe |. It requires a shell:
p = subprocess.Popen(command, shell=True)
The command itself as far as I can tell looks ok.
It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.
raster2pgsql_ps = subprocess.Popen([
'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
'C:\\temp\\SampleDTM\\SampleDTM.tif',
'test'
], stdout=subprocess.PIPE)
# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
'psql',
'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)
The following worked for me on Windows, while avoiding shell=True
One can make use of Python's fstring formatting to make sure the commands will work in windows.
Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.
Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.
import os
import subprocess
shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")
command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'
try:
shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
sys.stderr.write("An error occurred while importing data into the database, you might want to \
check the SQL command below:")
sys.stderr.write(command)
raise
To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.
PIPE = subprocess.PIPE
pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
stdout=PIPE, stderr=PIPE)
stdout, stderr = pd.communicate()
It will be better to use subprocess.Popen in this way:
proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()
B.T.W, it's good to format the path first, use:
path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")
this will avoid some path problems for different OS platform.
The shell = True is important if you want to execute your command just like executing it in local shell.
Hope will help you.