Based on this tutorial to build a TF image classifier, I have a bash shell in which I run a Docker image with the following command:
docker run --name fooldocker -it -v $HOME/tf_files:/tf_files/ gcr.io/tensorflow/tensorflow:latest-devel
And then in this docker image I run my Python script:
python /tf_files/label_image.py /tf_files/myimages
exit
It works.
But now, I need to automate these commands in a Python script. I tried :
p = Popen(['docker', 'run', '--rm', '--name', 'fooldocker','-it', '-v', '$HOME/tf_files:/tf_files/', 'gcr.io/tensorflow/tensorflow:latest-devel'], stdout=PIPE)
p = Popen(['docker', 'exec', 'fooldocker', 'python', '/tf_files/label_NES.py', '/tf_files/NES/WIP'])
p = Popen(['docker', 'kill', 'fooldocker'], shell=True, stdout=PIPE, stderr=PIPE)
p = Popen(['docker', 'rm', 'fooldocker'], shell=True, stdout=PIPE, stderr=PIPE)
Leading to this error after Popen #2 is run :
docker: Error response from daemon: create $HOME/tf_files: "$HOME/tf_files" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
The problem is that $HOME cannot be evaluated in this single quotes string. Either try doublequotes, or evaluate the variable beforehand and put it into the command string.
Also: If you set shell=True, you don't split your command into a list:
p = Popen('docker kill fooldocker', shell=True, stdout=PIPE, stderr=PIPE)
it is because Popen didn't interpret $HOME to your home path.
and it is the string $HOME and pass to docker command which not allow $ in a volume name.
maybe you can use subprocess module for convenience, for example:
import subprocess
subprocess.call("echo $HOME", shell=True)
it interpreted $HOME if shell=True specified.
Related
I would like to run a command in python using subprocess.run
I would like to switch the working directory JUST for the execution of this command.
Also, I need to record the output and the return code.
Here is the code I have:
import subprocess
result = subprocess.run("echo \"blah\"", cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
but this only returns
FileNotFoundError: [Errno 2] No such file or directory: 'echo "Running ls -la" && ls -la'
I also tried using the following arguments:
subprocess.run(["echo", "\"blah\""], cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Like Jean-François Fabre said, the solution is to add "shell=True" to the call
import subprocess
result = subprocess.run("echo \"blah\"", cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
shell=True seems to tell subprocess to use the string as a command.
I cant figure out how to start the server using a python command.
s = subprocess.Popen('"D:\MC SERVER 2k19\server.jar" -jar server.jar java', stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
This code runs without error but doesn't start the server in cmd.
Thanks.
It's got to do with how you're passing your arguments.
subprocess.Popen(['java', '-jar', 'server.jar'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True, cwd='D:\MC SERVER 2k19')
If you need to start a Java program using CMD process from Python and show the window, you can use subprocess to call open another CMD terminal and run the command.
In Windows you will need to CMD-escape spaces in the path you passing to the secondary CMD process. This is done with the carrot ^
proc = subprocess.Popen(
['start', 'cmd', '/k', "D:\\MC^ SERVER^ 2k19\\server.jar",
'-jar', 'server.jar', 'java'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
Keep in mind you will NOT be able to retrieve any output from the secondary CMD process from Python.
I.e. the process will return nothing.
proc.communicate()
# returns:
(b'', b'')
Using the subprocess module how do I get the following command to work?
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
This command works perfectly when I run it in a bash shell but I can't get it to work when running from within Python. I'm trying to do this from within Python because I need to be able to modify the query and get different result sets back and then process them in Python. I can't use one of the nice Python database connectors for various reasons which leaves me trying to pipe output from isql.
My code currently looks similar to the following:
bash_command = '''
isql -v -b -d, DSN_NAME "DOMAIN\username" password <<<
"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"
'''
process = subprocess.Popen(bash_command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, error = process.communicate()
However I have tried lots of variations:
Using the entire command as a string, or as a list of strings.
Using check_output vs Popen.
Using communicate() to try and send the query to the isql command or having the query be part of the command string using a heredoc.
Using shell = True or not.
Specifying /bin/bash or using the default /bin/sh.
Lots of different quoting and escaping patterns.
And pretty much every permutation of the above.
In no case do I receive the output of the query that I'm looking for. I'm pretty sure that the command isn't being sent to the shell as is but I can't tell what is being sent to the shell.
I feel like this should be pretty simple, send a command to the shell and get the output back, but I just can't make it work. I can't even see what command is being sent to the shell, even using pdb.
shell=True makes subprocess use /bin/sh by default. <<< "here-string" is a bash-ism; pass executable='/bin/bash':
>>> import subprocess
>>> subprocess.call(u'cat <<< "\u0061"', shell=True)
/bin/sh: 1: Syntax error: redirection unexpected
2
>>> subprocess.call(u'cat <<< "\u0061"', shell=True, executable='/bin/bash')
a
0
You should also use raw-string literals to avoid escaping backslashes: "\\u0061" == r"\u0061" != u"\u0061":
>>> subprocess.call(r'cat <<< "\u0061"', shell=True, executable='/bin/bash')
\u0061
0
Though you don't need shell=True here. You could pass the input as a string using process.communicate(input=input_string):
>>> process = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.communicate(br"\u0061")
('\\u0061', None)
The result could look like:
#!/usr/bin/env python
import shlex
from subprocess import Popen, PIPE
cmd = shlex.split(r'isql -v -b -d, DSN_NAME "DOMAIN\username" password')
process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, errors = process.communicate(
b"SELECT column_name, data_type "
b"FROM database_name.information_schema.columns "
b"WHERE table_name = 'some_table';")
Try giving this a shot:
import shlex
from subprocess import Popen, PIPE, STDOUT
sql_statement = '''"SELECT column_name, data_type
FROM database_name.information_schema.columns
WHERE table_name = 'some_table';"'''
isqlcommand = 'isql -v -b -d, DSN_NAME "DOMAIN\username" password'
isqlcommand_args = shlex.split(isqlcommand)
process = Popen(isqlcommand_args, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
output = process.communicate(input=sql_statement)[0]
print output
The idea here is to separate the here-string redirection from the isql command execution. This example will pipe the here-string into the stdin of process via process.communicate(). I'm also using shlex.split() to tokenize the command and its arguments.
Edit: Removed Shell=True after reviewing comment from J.F. Sebastian
I'm trying to execute a command using Popen.
The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:
"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres
When using within Python, I make it a string with the ' codes:
command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'
attempting to execute it results in an error:
p = subprocess.Popen(command)
ERROR: Unable to read raster file: test
The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)
Am I using Popen wrong?
Your command uses pipe |. It requires a shell:
p = subprocess.Popen(command, shell=True)
The command itself as far as I can tell looks ok.
It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.
raster2pgsql_ps = subprocess.Popen([
'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
'C:\\temp\\SampleDTM\\SampleDTM.tif',
'test'
], stdout=subprocess.PIPE)
# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
'psql',
'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)
The following worked for me on Windows, while avoiding shell=True
One can make use of Python's fstring formatting to make sure the commands will work in windows.
Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.
Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.
import os
import subprocess
shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")
command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'
try:
shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
sys.stderr.write("An error occurred while importing data into the database, you might want to \
check the SQL command below:")
sys.stderr.write(command)
raise
To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.
PIPE = subprocess.PIPE
pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
stdout=PIPE, stderr=PIPE)
stdout, stderr = pd.communicate()
It will be better to use subprocess.Popen in this way:
proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()
B.T.W, it's good to format the path first, use:
path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")
this will avoid some path problems for different OS platform.
The shell = True is important if you want to execute your command just like executing it in local shell.
Hope will help you.
There's a file named startup.cmd that sets some environment variables, runs some preparation commands, then does:
start "startup" cmd /k
Which opens a command shell named startup. The manual process I'm trying to automate is to then enter the following command into this shell: get startup.xml. I thought the correct way to do this in Python would be something like this:
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
getcommand = 'get startup.xml'
servercommand = 'startserver'
p.stdin.write(getcommand)
p.stdin.write(startserver)
(stdoutdata, stderrdata) = p.communicate()
print stdoutdata
print stderrdata
But those commands don't seem to be executing in the shell. What am I missing? Also, the command shell appears regardless of whether shell is set to True or False.
I found this warning in subprocess's document,
Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
So my suggestion is to use communicate to send your command.
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
command = 'get startup.xml\n'
command += 'startserver\n'
(stdoutdata, stderrdata) = p.communicate(command)
print stdoutdata
print stderrdata
This is a new process, so one cannot communicate directly with Popen.