I would like to write a simple python script which will be able to clone a git repository into desired directory. I used try...except construction to be able to catch all exceptions however it looks like I am not able to handle 'fatal' properly.
#!/usr/bin/env python
import subprocess
try:
subprocess.check_call(['git', 'clone', 'git clone git#some_repo', '/tmp/some_directory'])
except Exception:
print "There was a problem during repository configuration"
The output of the script above:
fatal: repository 'git clone git#some_repo' does not exist
There was a problem during repository configuration
To be more specific, I was rather expecting to get only the "There was a ..." message. Why do I get a 'fatal' message also?
You need to capture STDERR of your subprocess.check_call() execution. See Catch stderr in subprocess.check_call without using subprocess.PIPE
for details.
The message you are seeing is produced by the git command.
If you want to prevent that message from appearing you should redirect either standard error or all output to /dev/null through a shell, like:
subprocess.check_call(['git', 'clone', 'git clone git#some_repo', '/tmp/some_directory', '2&>/dev/null'], shell=True)
However, I'd recommend against that practice since you lose information on the actual cause of error.
As previously specified you need to capture the standard error. Also, as the documentation specifies, subprocess.check_call() just raises an exception when the return code is non-zero.
So, you could mimic the behavior as follows:
#!/usr/bin/env python
import subprocess
def clone_repository(): # customize your function parameters
# prepare the arguments with your function parameters
arguments = ['git', 'clone', 'git clone git#some_repo', '/tmp/some_directory']
git_proc = subprocess.Popen(arguments)
stdout, stderr = git_proc.communicate()
if git_proc.returncode != 0:
raise subprocess.CalledProcessError(arguments, git_proc.returncode)
return stdout, stderr
try:
stdout, stderr = clone_repository()
except (OSError, ValueError) as e:
# this errors out when the arguments are invalid (ValueError)
# or when there is an underlying file missing, etc (OSError)
# put the print that you require for these errors
pass
except subprocess.CalledProcessError:
# you could use stderr to determine the underlying error
print "There was a problem during repository configuration"
Related
I am trying to create a function that can run any shell command and return the stdout of that command, without worrying about any exceptions that could be thrown. When testing the code written below with incorrect commands like xyz testing, I get a FileNotFoundError rather that the CalledProcessError even though the command xyz testing returns a non-zero return code when I ran it in the shell. But, when I run the same function with a similar invalid command like ls -xyz I get a CalledProcessError as expected. Does anyone know why the exception is not being thrown in a consistent manner.
try:
cmd_result = subprocess.run(
cmd,
capture_output=True,
text=True,
check=True,
)
except subprocess.CalledProcessError as exc:
click.echo(exc.stderr)
click.echo(cmd)
sys.exit(1)
Thank you
I tried putting the shell=True option, and then the error situation swaps (xyz testing) passes and ls -xyz fails.
I have been stuck on what else I can try.
A couple of things are happening. With check=True, subprocess raises CalledProcessError if the command is executed and returns a non-zero exit status. When shell=False, the file to execute couldn't be found, so you got the FileNotFound error code. The program wasn't run, there was no exit code to examine, so CalledProcessError was not an option.
When shell=True, subprocess runs an intermediate shell. Now its the shell return code that matters and that can be different than the called process. Subprocess found the intermediate shell, so there is no FileNotFound error, but couldn't find the program so returned a non-zero value, generating the other exception. Consider the command "xyz testing;exit 0" this would not generate a called process error even though xyz does not exist. That's because the shell returned a 0 exit code.
How do I handle a subprocess.run() error in Python? For example, I want to run cd + UserInput with subprocess.run(). What if the user types in a directory name which does not exist? How do I handle this type of error?
As #match has mentioned, you can't run cd as a subprocess, because cd isn't a program, it's a shell built-in command.
But if you're asking about any subprocess failures, besides cd:
try:
subprocess.run(command_that_might_not_exist) # like ['abcd']
except Exception:
# handle the error
result = subprocess.run(command_that_might_fail) # like ['ls', 'abcd/']
if result.returncode != 0:
# handle the error
There is no way running cd in a subprocess is useful. The subprocess will change its own directory and then immediately exit, leaving no observable change in the parent process or anywhere else.
For the same reason, there is no binary command named cd on most systems; the cd command is a shell built-in.
Generally, if you run subprocess.run() without the check=True keyword argument, any error within the subprocess will simply be ignored. So if /bin/cd or a similar command existed, you could run
# purely theoretical, and utterly useless
subprocess.run(['cd', UserInput])
and simply not know whether it did anything or not.
If you do supply check=True, the exception you need to trap is CalledProcessError:
try:
# pointless code as such; see explanation above
subprocess.run(['cd', UserInput], check=True)
except subprocess.CalledProcessError:
print('Directory name %s misspelled, or you lack the permissions' % UserInput)
But even more fundamentally, allowing users to prod the system by running arbitrary unchecked input in a subprocess is a horrible idea. (Allowing users to run arbitrary shell script with shell=True is a monumentally, catastrophically horrible idea, so let's not even go there. Maybe see Actual meaning of shell=True in subprocess)
A somewhat more secure approach is to run the subprocess with a cwd= keyword argument.
# also vaguely pointless
subprocess.run(['true'], cwd=UserInput)
In this case, you can expect a regular FileNotFoundError if the directory does not exist, or a PermissionError if you lack the privileges.
You should probably still add check=True and be prepared to handle any resulting exception, unless you specifically don't care whether the subprocess succeeded. (There are actually cases where this makes sense, like when you grep for something but are fine with it not finding any matches, which raises an error if you use check=True.)
Perhaps see also Running Bash commands in Python
I want to run a subprocess to check if python-docx is installed similar to this, where the lines
verify_installation = subprocess.run(["pdftotext -v"], shell=True)
if verify_installation.returncode == 127:
checks if pdftotext is installed or not and if it is not installed (returncode ==127), then it raises an exception.
I want to have a similar implementation to check if python-docx is installed, however, while debugging in Colab, even after installing python-docx, the same returncode is returned.
What is the interpretation of (returncode ==127) and how do I raise an exception only when the library is not installed.
Also what exactly does subprocess.run(["pdftotext -v"], shell=True) achieve.
I can recommend for different approach for this, pass PIPEs to the stderr and stdout for the spawned process and check those pipes after child return.
import subprocess
outs=None
errs=None
try:
proc=subprocess.Popen(["pdftotext -v"], shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
outs, errs = proc.communicate(timeout=15) #timing out the execution, just if you want, you dont have to!
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
#parse the stderr and stdoutput of proc:
f_check_if_has_errors(errs, outs)
Also consider to use/look subprocess.check_call method below:
try:
proc = subprocess.check_call(["pdftotext -v"], shell=True)
proc.communicate()
except subprocess.CalledProcessError:
# There was an error - command exited with non-zero code
I found a solution, it doesn't use subprocess as I mentioned, but I am including the answer to make sure someone who faces a similar problem "To check if a module is installed and if not to catch the error" can try it.
try:
import docx
except ImportError as e:
raise Exception(
#do something
)
In case importing the module creates problems, I'm still looking for a solution that runs shell subprocesses without having to import the module.
Is there a doc somewhere which indicates what the different return codes of python's subprocess check_output() command means? I'm seeing the returncode equal 3, and I have no idea what is setting that value and/or what it means.
Example code:
try:
output = subprocess.check_output(cmd,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print e.returncode
This code is printing 3.
The Python subprocess.check_output() doesn't itself return a code, it returns the output of the command being run. The doco can be found here.
If you're seeing an error code in that string, it's almost certainly specific to whatever you're trying to run, not a result of subprocess itself.
If, however, you're capturing the CalledProcessError exception caused by a non-zero return code (which can be extracted from the returncode attribute), that's still specific to whatever you're running.
In both cases, the documentation you need to check is that of whatever tool subprocess is running.
allocating I want to launch a process and retrieve the stdout and stderr.
I don t really care about getting this in real time.
I wanted to use subprocess.check_ouput(), but the process might fail.
After reading StackOverflow and the Python docs I added a try .. catch block:
def execute(cmd,timeinsec=60):
print("execute ",cmd, " with time out ",timeinsec)
try:
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, timeout=timeinsec,universal_newlines=True)
except subprocess.TimeoutExpired:
print("timeout expired")
return "",2
except subprocess.CalledProcessError:
print("called process failed")
return "",1
print ('The command returned the following back to python:'+output)
return output,0
But when I print the output with output.decode('utf-8')
I just get the first line of the output.
Note : I'm running this in the MSys environment distributed with Msysgit 1.8 on windows.
Do you have any idea of what can be wrong?
Do you know any better way to do this?
You must be using python-3.x. Please tag your question accordingly. Also, I am not sure why you are calling read() method of output. The output is a byte string and does not have a read() method. The following code works for me:
#! /usr/bin/env python3
import subprocess
try :
retcode = 0
cmd = ["/bin/ls", "/usr/local"]
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
output = e.output
retcode = e.returncode
print(output.decode('utf-8'))
print(retcode)
The output is:
bin
etc
games
include
lib
man
sbin
share
src
0
If I trigger an error by replacing /usr/local with /usr/localfoo (which does not exist), then the output is:
/bin/ls: cannot access /usr/localfoo: No such file or directory
2
Finally, you can add universal_newlines=True to check_output() call and not have to worry about calling decode() on the output:
...
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT,universal_newlines=True)
...
...
print(output)
Please take the above example and see if you can make it reproduce your problem. If you can reproduce the problem, please post your code, its output, and all error messages (copy, paste, and reformat for SO).
Solution
The problem was that the application in windows launched in the subprocess was allocating a console (built using Visual) and the stdout of the process was already redirected outside, and only one print was done in the original cout before this redirection