I'm on Ubuntu 15.04 (not by choice obviously) and Python 3.4.3 and I'm trying to execute something like the following.
subprocess.check_call("pushd /tmp", shell=True)
I need the shell=True because the actual code I'm trying to execute contains wildcards that need to be interpreted. However, this gives me the following error.
/usr/lib/python3.4/subprocess.py in check_call(*popenargs, **kwargs)
559 if cmd is None:
560 cmd = popenargs[0]
--> 561 raise CalledProcessError(retcode, cmd)
562 return 0
563
CalledProcessError: Command 'pushd /tmp' returned non-zero exit status 127
I've tried doing the same thing on my Mac (El Capitan and Python 3.5.1) and it works perfectly. I've also tried executing subprocess.check_call("ls", shell=True) on the Ubuntu 15.04 with Python 3.4.3 (for sanity check), and it works fine. As a final sanity check, I've tried the command pushd /tmp && popd in Bash on the Ubuntu 15.04 and that works fine too. So somehow, on (my) Ubuntu 15.04 with Python 3.4.3, subprocess.check_call() does not recognise pushd and popd! Why?
You have two problems with your code. The first one is that the shell used by default is /bin/sh which doesn't support pushd and popd.
In your question you failed to provide the whole error output, and at the top of it you should see the line:
/bin/sh: 1: popd: not found
The next time rememeber to post the whole error message, and not just the portion that you (incorrectly) think is relevant.
You can fix this by telling the subprocess module which shell to use via the executable argument:
>>> subprocess.check_call('pushd ~', shell=True, executable='/bin/bash')
~ ~
0
The second problem is that even with this you will get an error if you use multiple check_call calls:
>>> subprocess.check_call('pushd ~', shell=True, executable='/bin/bash')
~ ~
0
>>> subprocess.check_call('popd', shell=True, executable='/bin/bash')
/bin/bash: riga 0: popd: stack delle directory vuoto
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'popd' returned non-zero exit status 1
This is because every call to check_call starts a new subshells, and thus it doesn't matter whether you previously called pushd because the directory stack will always be empty.
Note that if you try to combine pushd and popd in a single call they do work:
>>> subprocess.check_call('pushd ~ && popd', shell=True, executable='/bin/bash')
~ ~
~
0
Now fact is, if you were thinking of using pushd and popd in that way from python... they are useless. That's because you can specify the current working directory via the cwd argument, and so you can keep track of the stack of working directories from python without having to rely on pushd and popd:
current_working_dirs = []
def pushd(dir):
current_working_dirs.append(os.path.realpath(os.path.expanduser(dir)))
def popd():
current_working_dirs.pop()
def run_command(cmdline, **kwargs):
return subprocess.check_call(cmdline, cwd=current_working_dirs[-1], **kwargs)
Replace check_call('pushd xxx') with pushd('xxx') and check_call('popd') with popd and use run_command(...) instead of check_call(...).
As you suggest a more elegant solution would be to use a context manager:
class Pwd:
dir_stack = []
def __init__(self, dirname):
self.dirname = os.path.realpath(os.path.expanduser(self.dirname))
def __enter__(self):
Pwd.dir_stack.append(self.dirname)
return self
def __exit__(self, type, value, traceback):
Pwd.dir_stack.pop()
def run(self, cmdline, **kwargs):
return subprocess.check_call(cmdline, cwd=Pwd.dir_stack[-1], **kwargs)
used as:
with Pwd('~') as shell:
shell.run(command)
with Pwd('/other/directory') as shell:
shell.run(command2) # runs in '/other/directory'
shell.run(command3) # runs in '~'
Related
I'm in the process of implementing a little pre-commit hook that calls gitleaks protect prior to every commit.
This works well in a terminal but when trying to commit from within VSCode, a non-descriptive "Git: O" is returned (I assume this is simply the first line of gitleaks, part of its ascii logo).
As you can tell, I've tried multiple ways to have VSCode's Git module return a proper message upon exit of the submodule. However, nothing seems to work in that regard.
def eprint(*args, **kwargs):
print(*args, file=sys.stderr, **kwargs)
exit_code = subprocess.run("gitleaks protect -v --staged -c gitleaks.toml",shell=True)
if exit_code.returncode == 1:
eprint("This is a test")
sys.exit("TEST")
How do I return an alert window in VSCode that displays a message whenever the subprocess is exiting with exit code 1?
EDIT:
Ok. This works somehow, but it fails in so far as that
subprocess.run("gitleaks version", shell=True, stdout=dev_null, stderr=dev_null) only works with my WSL Bash whereas subprocess.run("gitleaks version", stdout=dev_null, stderr=dev_null) (without the shell=True) only works for my VSCode with Windows Git Bash.
Any way to make this portable, so FileNotFoundError is correctly thrown on both systems?
#!/usr/bin/env python3
# pylint: disable=C0116,W0613
import sys
import warnings
import subprocess
dev_null = subprocess.DEVNULL
def eprint(*args, **kwargs):
print(*args, file=sys.stderr, **kwargs)
def gitleaks_installed():
try:
subprocess.run("gitleaks version", shell=True, stdout=dev_null, stderr=dev_null)
return True
except FileNotFoundError:
return False
if gitleaks_installed():
exit_code = subprocess.run("gitleaks protect -v --staged -c gitleaks.toml", shell=True, stdout=dev_null, stderr=dev_null)
if exit_code.returncode == 1:
eprint("gitleaks has detected sensitive information in your changes. Commit aborted.")
subprocess.run("gitleaks protect -v --staged -c gitleaks.toml", shell=True)
sys.exit(1)
else:
eprint("gitleaks is not installed or in the PATH.")
sys.exit(1)
EDIT2: NVM. The gitleaks_installed part doesn't work at all under WSL Bash. It either always True or always False, depending on whether I include shell=True.
Is there a better way to detect whether gitleaks is installed/in the PATH or not?
The object returned by subprocess.run is a CompletedProcess object, not the return code. You have to access its .returncode attribute to examine what it returned.
However, you might instead add check=True to have Python throw an error on failure.
You should also almost certainly get rid of the superfluous shell=True by parsing the command line into tokens yourself.
try:
subprocess.run(
["gitleaks", "protect", "-v", "--staged", "-c", "gitleaks.toml"],
check=True)
except subprocess.CalledProcessError:
eprint("gitleaks has detected sensitive information in your changes. Commit aborted.")
sys.exit(1)
except FileNotFoundError:
eprint("gitleaks is not installed or in the PATH.")
sys.exit(1)
This also avoids running the process a second time just to obtain the output. If you want to force your own message to go on top, capture the output and print it after your message (add capture_output=True and text=True).
You might also want to replace eprint with logging.warn, and perhaps return different exit codes in the different error scenarios. (Bash conventionally returns 127 when it can't find a binary, but that's just Bash.)
Running subprocess.run on a string, instead of a list, without shell=True weirdly happens to work on Windows, but my recommendation would be to always avoid trying to exploit that for convenience. Perhaps see also Actual meaning of shell=True in subprocess
Not the most elegant solution, but this works.
Instead of the FileNotFoundError exception, we're using the subprocess.run returncode.
#!/usr/bin/env python3
# pylint: disable=C0116,W0613
import sys
import subprocess
dev_null = subprocess.DEVNULL
def eprint(*args, **kwargs):
print(*args, file=sys.stderr, **kwargs)
def gitleaks_installed():
exit_code = subprocess.run("gitleaks version", shell=True, stdout=dev_null, stderr=dev_null)
if exit_code.returncode == 0:
return True
else:
return False
if gitleaks_installed():
exit_code = subprocess.run("gitleaks protect -v --staged -c gitleaks.toml", shell=True, stdout=dev_null, stderr=dev_null)
if exit_code.returncode == 1:
eprint("gitleaks has detected sensitive information in your changes. Commit aborted.")
subprocess.run("gitleaks protect -v --staged -c gitleaks.toml", shell=True)
sys.exit(1)
else:
eprint("gitleaks is not installed or in the PATH.")
sys.exit(1)
I have the code snippet that works with shell=True which isn't secure and when i attempt to remove shell=True and include shell=False the program errors out
The code is below:
cmd = "git clone https://github.com/{} &"
#define a worker function
def worker():
while True:
item = q.get()
subprocess.Popen(cmd.format(item))
q.task_done()
I get the error below:
File "rapid.py", line 56, in worker
subprocess.Popen(cmd.format(item))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git clone https://github.com/laramies/theHarvester.git &': 'git clone https://github.com/laramies/theHarvester.git &'
if i add shell = True to the subprocess line it runs just fine (see below) but then code factor flags it as insecure code. Any way to do this without shell = true?
cmd = "git clone https://github.com/{} &"
#define a worker function
def worker():
while True:
item = q.get()
subprocess.Popen(cmd.format(item), shell = True)
q.task_done()
The command is being passed to subprocess.Popen as a string. This works when using shell=True because the shell can accept a command as a single string. But when shell=False, Popen expects the command as a list of arguments, the first one being the full path of the program to run. (This is assuming you're on a POSIX machine, not Windows.)
Essentially, the code says "Run a program called git clone https://github.com/laramies/theHarvester.git with no arguments" instead of "Run git with arguments clone and https://github.com/laramies/theHarvester.git".
The & should also be removed because that is a shell feature and with shell=False it would be passed to git as an argument it won't understand. You don't need it though, since the process will run in the background anyway.
Something like this should work:
subprocess.Popen(["/usr/bin/git", "clone", "https://github.com/{}".format(item)])
More info here: https://docs.python.org/3.7/library/subprocess.html#popen-constructor
I wrote a script to run mafft module from the terminal:
import subprocess
def linsi_MSA(sequnces_file_path):
cmd = ' mafft --maxiterate 1000 --localpair {seqs} > {out}'.format(seqs=sequnces_file_path, out=sequnces_file_path)
subprocess.call(cmd.split(), shell=True)
if __name__ == '__main__':
import logging
logger = logging.getLogger('main')
from sys import argv
if len(argv) < 2:
logger.error('Usage: MSA <sequnces_file_path> ')
exit()
else:
linsi_MSA(*argv[1:])
for some reason when trying to run the script from the terminal using:
python ./MSA.py ./sample.fa
I get the mafft interactive version opening directly in the trminal (asking for input ..output etc..)
when i'm trying to write the cmd directly in the terminal using:
mafft --maxiterate 1000 --localpair sample.fa > sample.fa
its working as expected and perfoming the command line version as without opening the interactive version.
I want my script to be able to perform the cmd line version on the terminal. what seems to be the problem?
thanks!
If you use shell=True you should pass one string as argument, not a list, e.g.:
subprocess.call("ls > outfile", shell=True)
It's not explained in the docs, but I suspect it has to do with what low-level library function is ultimately called:
call(["ls", "-l"]) --> execlp("ls", "-l")
^^^^^^^^^^ ^^^^^^^^^^
call("ls -l", shell=True) --> execlp("sh", "-c", "ls -l")
^^^^^^^ ^^^^^^^
call(["ls", "-l"], shell=True) --> execlp("sh", "-c", "ls", "-l")
# which can be tried from command line:
sh -c ls -l
# result is a list of files without details, -l was ignored.
# see sh(1) man page for -c string syntax and what happens to further arguments.
I downloaded a python program (PYPDFOCR) that runs through the command line. PYPDFOCR has several dependencies (ghost script, image magick, etc..)
When I execute the program in the command line, it fails. The code executes GhostScript with all its arguments but I get the error that the command is not recognized.
If I go to the command line, I can execute ghost script "C:\Programs.....\gswin64c.exe" and pass the arguments and get the result. However, when I run PYPDFOCR, it fails everytime.
def _run_gs(self, options, output_filename, pdf_filename):
try:
cmd = '%s -q -dNOPAUSE %s -sOutputFile="%s" "%s" -c quit' % (self.binary, options, output_filename, pdf_filename)
logging.info(cmd)
out = subprocess.check_output(cmd, shell=True)
except subprocess.CalledProcessError as e:
print e.output
if "undefined in .getdeviceparams" in e.output:
error(self.msgs['GS_OUTDATED'])
else:
error (self.msgs['GS_FAILED'])
The error I get in the command is "C\Program" is not recognized as an internal or external command, operable program or batch file.
When I print the contents of the command, it shows:
c:\Program File\gs\gs9.16\bin\gswin64c.exe" -q -dNOPAUSE -sDEVICE=j[ecgray -dJPEG=75 -r300 -sOutputFILE="C:\test\a3_%d.jpg "c:\test\a3.pdf" -c quit
Again, I can run the c:...gswin64.exe command without the program.
Any help will be deeply appreciated.
The problem is probably that the space after Program in 'Program Files' is not correctly escaped. Use some function like
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
to escape your command before starting the process.
(Example function taken from here)
I had the same problem on Windows with windres.
It turned out to be a windres problem (MinGW64 4.8.3.20141208 from chocolatey).
I debugged into it and found that subprocess.Popen(cmd,**kw),
cmd being a list with first entry the executable's path,
mapped to _winapi.CreateProcess(executable, cmd, ...), with executable=None.
So the error originated from winapi.
In the Posix branch in subprocess there is
if executable is None:
executable = args[0]
which is missing in the windows branch.
As a try I setexecutable in **kw: Still the same problem.
Then I found https://amindlost.wordpress.com/2012/06/09/mingw-windres-exe-cant-popen-error/.
Indeed it worked with msys2's ming64's windres.
I have an .R file saved locally at the following path:
Rfilepath = "C:\\python\\buyback_parse_guide.r"
The command for RScript.exe is:
RScriptCmd = "C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe --vanilla"
I tried running:
subprocess.call([RScriptCmd,Rfilepath],shell=True)
But it returns 1 -- and the .R script did not run successfully. What am I doing wrong? I'm new to Python so this is probably a simple syntax error... I also tried these, but they all return 1:
subprocess.call('"C:\Program Files\R\R-2.15.2\bin\Rscript.exe"',shell=True)
subprocess.call('"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe"',shell=True)
subprocess.call('C:\Program Files\R\R-2.15.2\bin\Rscript.exe',shell=True)
subprocess.call('C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe',shell=True)
Thanks!
The RScriptCmd needs to be just the executable, no command line arguments. So:
RScriptCmd = "\"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe\""
Then the Rfilepath can actually be all of the arguments - and renamed:
RArguments = "--vanilla \"C:\\python\\buyback_parse_guide.r\""
It looks like you have a similar problem to mine. I had to reinstall RScript to a path which has no spaces.
See: Running Rscript via Python using os.system() or subprocess()
This is how I worked out the communication between Python and Rscript:
part in Python:
from subprocess import PIPE,Popen,call
p = subprocess.Popen([ path/to/RScript.exe, path/to/Script.R, Arg1], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out = p.communicate()
outValue = out[0]
outValue contains the output-Value after executing the Script.R
part in the R-Script:
args <- commandArgs(TRUE)
argument1 <- as.character(args[1])
...
write(output, stdout())
output is the variable to send to Python