I am deploying a flask app, in which a multiprocessing.Process will be started. Inside this process I call a shell command trough subprocess.call(). The script runs fine when executed on local host, when deployed with nginx and gunicorn, the flask app behaves as expected until the subprocess starts, then I recieve the following error log:
DEBUG:root:start to run command
DEBUG:root:(<class 'FileNotFoundError'>, FileNotFoundError(2, "No such file or directory: 'java -jar ábsolute/path/to/jar/file')
Process(
target=decode_upload,
args=(
path_to_blf,
path_to_dbc_folder,
path_to_splitted,
path_to_decoder,
system_layout,
dc_doc,
dc_id,
file_type,
)
).start()
Here is the rellevant part of the function.
def decode_file(
path_to_blf,
path_to_dbc_folder,
path_to_splitted,
path_to_decoder,
system_layout=DEFAULT_SYSTEM_LAYOUT):
command = "{} {} --blf={}".format(
SOFTWARE_COMMAND,
path_to_decoder,
path_to_blf
)
for dbc_file_name in DBC_FILE_NAME_LIST:
command += " --dbc={}".format(
os.path.join(
path_to_dbc_folder,
dbc_file_name
)
)
command += " --out={}".format(path_to_splitted)
logging.debug("start to run command")
subprocess.call(command)
logging.debug(f)
logging.debug("run command end")
def decode_upload(
path_to_blf,
path_to_dbc_folder,
path_to_splitted,
path_to_decoder,
system_layout,
dc_doc,
dc_id,
file_type):
logging.basicConfig(filename='flask.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
try:
decode_file(
path_to_blf,
path_to_dbc_folder,
path_to_splitted,
path_to_decoder,
system_layout)
except:
logging.debug(sys.exc_info())
The process fails when it gets to this line.
subprocess.call(command)
if I try to call the "command" from the command line it works without a problem.
from subprocess import Popen
command='your complete command as you paste on cmd'
p1=Popen(command,Shell=True)
This will help you to run the command as complete string
The proper solution is to pass the command as a list of parsed arguments. You need to understand how the shell handles quoting and argument splitting to do this right.
As a quick cheat,
printf "'%s'\n" your command line here
in your shell should give you a good idea of how the shell expands the arguments. For example
bash$ printf "'%s'\n" java -jar "/path/complex path with spaces.jar" \* \>\<
'java'
'-jar'
'/path/complex path with spaces.jar'
'*'
'><'
shows you that you need
subprocess.call(['java', '-jar', '/path/complex path with spaces.jar', '*', '><'])
Adapting your code for this, we obtain
command = [SOFTWARE_COMMAND, path_to_decoder, '--blf={}'.format(path_to_blf)]
for dbc_file_name in DBC_FILE_NAME_LIST:
command.append("--dbc={}".format(
os.path.join(path_to_dbc_folder, dbc_file_name)))
command.append("--out={}".format(path_to_splitted))
Related
Iam trying to execute an exif command using subprocess. The command is :
['exiftool', '-ID3:Picture', '-b', '-ThumbnailImage', '/home/mediaworker/Downloads/Raabta.mp3', '>', '/mnt/share_PROXY/exifData/Raabta.jpg']
Now, the issue is that it returns the status code as 1. But if i execute the same command in the terminal, it executes successfully. The file is written to the location. Is my command going wrong in subprocess ? The error i get when i run my python script is :
Error: File not found - >
Error: File not found - /mnt/share_PROXY/exifData/Raabta.jpg
The code implementation is as follows:
file_name = os.path.basename(file_loc)
file_name = file_name.replace(os.path.splitext(file_name)[1], ".jpg")
dst_loc = os.path.join(dst_loc, file_name)
cmd_ = ["exiftool", "-ID3:Picture", "-b", "-ThumbnailImage", file_loc, ">", dst_loc]
logger.info("Command is {}".format(cmd_))
try:
p = subprocess.Popen(cmd_, stdout=subprocess.PIPE)
p.communicate()
if p.returncode != 0:
logger.error("Failed to write thumbnail artwork")
else:
id3_metadata.append({"file_thumbnail_info_path": dst_loc})
except Exception:
logger.error("[extract_iptc_metadata] Exception : '{}'".format(ex))
The error output refers to the redirection >.
The proper way to redirect using subprocess is using the stdout parameter.
with open(dst_loc, 'wb') as f:
p = subprocess.Popen(cmd_, stdout=f)
p.communicate()
The '>', '/mnt/share_PROXY/exifData/Raabta.jpg' part of your command is shell redirection and is a function of the command line/shell. It is not available when you execute a command from python in this way.
The option you want to look at is the -W (-tagOut) option. This would be the example command you want to work off of. Just replace -preview:all with the tag you want to extract, which would be -ThumbnailImage in this case.
I am in a process of building a simple remote shell tool to communicate with Windows 10. Server sends a "message" through its own shell to the client who runs the message. I need this received message to be run by other process other that default cmd (shell=True) - a specified app.exe. Here is the code that runs on the client:
1)
def work( storage, message ) :
import subprocess
process = subprocess.Popen([message], stdout=subprocess.PIPE, stderr=None, shell=True)
#Launch the shell command:
output = process.communicate()
print output[0]
I tried including "app.exe" or "cmd" to execute the message but with that I get error: TypeError: bufsize must be an integer.
I have also tried pinpointing the issue locally and I can run:
2)
import subprocess
import sys
subprocess.Popen(["C:\\Users\\User\\Desktop\\app.exe", "-switch"] + sys.argv[1:], shell=False)
and pass arguments from a command terminal and it works as it should. Now I am trying to apply the same logic to a remote execution with my program and use either solution 1 or 2.
Update:
3) Trying to implement what I did locally to a remote solution:
def work( storage, message ) :
import subprocess
import sys
process = subprocess.Popen(["C:\\Users\\User\\Desktop\\app.exe", "-switch"] + sys.argv[1:], shell=False)
#Launch the shell command:
output = process.communicate()
print output[0]
I tried replacing sys.argv[1:] with message but I get:
TypeError: can only concatenate list (not "str") to list
shell=True doesn't mean the first argument to Popen is a list of arguments to the shell; it just means the first argument is processed by the shell, rather than being arguments to whatever system call your system would use to execute a new process.
In this case, you appear to want to run app.exe with a given argument; that's simply
cmd = r"C:\Users\User\Desktop\app.exe"
subprocess.Popen([cmd, "-switch", message], stdout=subprocess.PIPE)
#chepner sir, you are a very helpful. That was it! I am so happy, thanks for your help.
Your solution:
Popen(["...\\app.exe", "-switch", message], stdout=subprocess.PIPE, stderr=None)
That was the badger!
I downloaded a python program (PYPDFOCR) that runs through the command line. PYPDFOCR has several dependencies (ghost script, image magick, etc..)
When I execute the program in the command line, it fails. The code executes GhostScript with all its arguments but I get the error that the command is not recognized.
If I go to the command line, I can execute ghost script "C:\Programs.....\gswin64c.exe" and pass the arguments and get the result. However, when I run PYPDFOCR, it fails everytime.
def _run_gs(self, options, output_filename, pdf_filename):
try:
cmd = '%s -q -dNOPAUSE %s -sOutputFile="%s" "%s" -c quit' % (self.binary, options, output_filename, pdf_filename)
logging.info(cmd)
out = subprocess.check_output(cmd, shell=True)
except subprocess.CalledProcessError as e:
print e.output
if "undefined in .getdeviceparams" in e.output:
error(self.msgs['GS_OUTDATED'])
else:
error (self.msgs['GS_FAILED'])
The error I get in the command is "C\Program" is not recognized as an internal or external command, operable program or batch file.
When I print the contents of the command, it shows:
c:\Program File\gs\gs9.16\bin\gswin64c.exe" -q -dNOPAUSE -sDEVICE=j[ecgray -dJPEG=75 -r300 -sOutputFILE="C:\test\a3_%d.jpg "c:\test\a3.pdf" -c quit
Again, I can run the c:...gswin64.exe command without the program.
Any help will be deeply appreciated.
The problem is probably that the space after Program in 'Program Files' is not correctly escaped. Use some function like
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
to escape your command before starting the process.
(Example function taken from here)
I had the same problem on Windows with windres.
It turned out to be a windres problem (MinGW64 4.8.3.20141208 from chocolatey).
I debugged into it and found that subprocess.Popen(cmd,**kw),
cmd being a list with first entry the executable's path,
mapped to _winapi.CreateProcess(executable, cmd, ...), with executable=None.
So the error originated from winapi.
In the Posix branch in subprocess there is
if executable is None:
executable = args[0]
which is missing in the windows branch.
As a try I setexecutable in **kw: Still the same problem.
Then I found https://amindlost.wordpress.com/2012/06/09/mingw-windres-exe-cant-popen-error/.
Indeed it worked with msys2's ming64's windres.
I have an .R file saved locally at the following path:
Rfilepath = "C:\\python\\buyback_parse_guide.r"
The command for RScript.exe is:
RScriptCmd = "C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe --vanilla"
I tried running:
subprocess.call([RScriptCmd,Rfilepath],shell=True)
But it returns 1 -- and the .R script did not run successfully. What am I doing wrong? I'm new to Python so this is probably a simple syntax error... I also tried these, but they all return 1:
subprocess.call('"C:\Program Files\R\R-2.15.2\bin\Rscript.exe"',shell=True)
subprocess.call('"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe"',shell=True)
subprocess.call('C:\Program Files\R\R-2.15.2\bin\Rscript.exe',shell=True)
subprocess.call('C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe',shell=True)
Thanks!
The RScriptCmd needs to be just the executable, no command line arguments. So:
RScriptCmd = "\"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe\""
Then the Rfilepath can actually be all of the arguments - and renamed:
RArguments = "--vanilla \"C:\\python\\buyback_parse_guide.r\""
It looks like you have a similar problem to mine. I had to reinstall RScript to a path which has no spaces.
See: Running Rscript via Python using os.system() or subprocess()
This is how I worked out the communication between Python and Rscript:
part in Python:
from subprocess import PIPE,Popen,call
p = subprocess.Popen([ path/to/RScript.exe, path/to/Script.R, Arg1], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out = p.communicate()
outValue = out[0]
outValue contains the output-Value after executing the Script.R
part in the R-Script:
args <- commandArgs(TRUE)
argument1 <- as.character(args[1])
...
write(output, stdout())
output is the variable to send to Python
I'm running a script from crontab that will just ssh and run a command and store the results in a file.
The function that seems to be failing is subprocess.popen.
Here is the python function:
def _executeSSHCommand(sshcommand,user,node):
'''
Simple function to execute an ssh command on a remote node.
'''
sshunixcmd = '/usr/bin/ssh %s#%s \'%s\'' % (user,node,sshcommand)
process = subprocess.Popen([sshunixcmd],
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
process.wait()
result = process.stdout.readlines()
return result
When it's run from the command line, it executes correctly, from cron it seems to fail with the error message below.
Here are the crontab entries:
02 * * * * /home/matt/scripts/check-diskspace.py >> /home/matt/logs/disklog.log
Here are the errors:
Sep 23 17:02:01 timmy CRON[13387]: (matt) CMD (/home/matt/scripts/check-diskspace.py >> /home/matt/logs/disklog.log)
Sep 23 17:02:01 timmy CRON[13386]: (CRON) error (grandchild #13387 failed with exit status 2)
I'm going blind trying to find exactly where I have gone so wrong. Any ideas?
The cron PATH is very limited. You should either set absolute path to your ssh /usr/bin/ssh or set the PATH as a first line in your crontab.
You probably need to pass ssh the -i argument to tell ssh to use a specific key file. The problem is that your environment is not set up to tell ssh which key to use.
The fact that you're using python here is a bit of a red herring.
For everything ssh-related in python, you might consider using paramiko. Using it, the following code should do what you want.
import paramiko
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect(node, username=user)
stdout = client.exec_command(ssh_command)[0]
return stdout.readlines()
When running python scripts from cron, the environment PATH can be a hangup, as user1652558 points out.
To expand on this answer with example code to add custom PATH values to the environment for a subprocess call:
import os
import subprocess
#whatever user PATH values you need
my_path = "/some/custom/path1:/some/custom/path2"
#append the custom values to the current PATH settings
my_env = os.environ.copy()
my_env["PATH"] = my_path + ":" + my_env["PATH"]
#subprocess call
resp = subprocess.check_output([cmd], env=my_env, shell=True)