Python provides two convenient functions for calling subprocesses that might fail, subprocess.check_call and subprocess.check_output. Basically,
subprocess.check_call(['command', 'arg1', ...])
spawns the specified command as a subprocess, blocks, and verifies that the subprocess terminated successfully (returned zero). If not, it throws an exception. check_output does the same thing, except it captures the subprocess's stdout and returns it as a byte-string.
This is convenient because it is a single Python expression (you don't have to set up and control the subprocess over several lines of code), and there's no risk of forgetting to check the return value.
What are the idiomatic Ruby equivalents to check_call and check_output? I am aware of the $? global that gives the process's return value, but that would be awkward—the point of having exceptions is that you don't have to manually check error codes. There are numerous ways to spawn a subprocess in Ruby, but I don't see any that provide this feature.
Here’s a simple check_call I threw together, and that seems to work.
def check_call(*cmd, **kw)
_, status = Process.waitpid2 Kernel.spawn(*cmd, **kw)
raise "Command #{cmd} #{status}" unless status.success?
end
The basic/in-built methods are supplanted by the POpen4 gem. And the shell-executor gem provides further awesomeness.
It's hard to say what's the most idiomatic solution in Ruby… but the one that's closest to Python is probably Shell.execute! from shell-executer.
From the example on the docs page:
begin
Shell.execute!('ls /not_existing')
rescue RuntimeError => e
print e.message
end
Compare to:
try:
subprocess.check_call('ls /not_existing', shell=True)
except Exception as e:
print e.message
The most notable difference here is that the Ruby equivalent doesn't have a way to do shell=False (and take the args as a list), which Python not only has, but defaults to.
Also, Python's e.message will be a default message or something generated based on the return code, while Ruby's e.message will be the child's stderr.
If you want to do shell=False, as far as I know, you'll have to write your own wrapper around something lower level; all of the Ruby wrappers I know of (shell-executer, Popen4, [open4][4]) are wrappers around or emulators of the POSIX popen functions.
Related
I'm trying to include sequence alignment using muscle or mafft, depending of the user in a pipeline.
To do so, i'm using the subprocess package, but sometimes, the subprocess never terminates and my script doesn't continue. Here is how I call the subprocess:
child = subprocess.Popen(str(muscle_cline), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
child.wait()
The command muscle_cline looks like this:
./tools/muscle/muscle5.1.win64.exe -align C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\genes-fasta\12S_tmp.fasta -output C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\alignement\12S_tmp_muscle_align.fasta
I'm calling this line in a function that just creates the command line and calls the subprocess, and converts the output.
I'm then calling this function in a for loop
for file in getFastaFile(my_dir):
alignSequenceWithMuscle(file)
The issue is that sometimes, for unknown reasons, the subprocess never finishes and get locked...
I tried to check the returncode of the child, or print stuff to see where it gets locked, and it's getting locked when I'm calling the subprocess.
Any ideas?
You generally want to avoid bare Popen, especially if you don't have a good understanding of its requirements. This is precisely why Python offers you subprocess.check_output and other higher-level functions which take care of the nitty-gritty of managing a subprocess.
output = subprocess.check_output(
["./tools/muscle/muscle5.1.win64.exe",
"-align", r"C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\genes-fasta\12S_tmp.fasta",
"-output", r"C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\alignement\12S_tmp_muscle_align.fasta"],
text=True)
Notice also the raw strings r"..." to avoid having to double the backslashes, and the text=True keyword argument to instruct Python to implicitly decode the bytes you receive from the subprocess.
How do I handle a subprocess.run() error in Python? For example, I want to run cd + UserInput with subprocess.run(). What if the user types in a directory name which does not exist? How do I handle this type of error?
As #match has mentioned, you can't run cd as a subprocess, because cd isn't a program, it's a shell built-in command.
But if you're asking about any subprocess failures, besides cd:
try:
subprocess.run(command_that_might_not_exist) # like ['abcd']
except Exception:
# handle the error
result = subprocess.run(command_that_might_fail) # like ['ls', 'abcd/']
if result.returncode != 0:
# handle the error
There is no way running cd in a subprocess is useful. The subprocess will change its own directory and then immediately exit, leaving no observable change in the parent process or anywhere else.
For the same reason, there is no binary command named cd on most systems; the cd command is a shell built-in.
Generally, if you run subprocess.run() without the check=True keyword argument, any error within the subprocess will simply be ignored. So if /bin/cd or a similar command existed, you could run
# purely theoretical, and utterly useless
subprocess.run(['cd', UserInput])
and simply not know whether it did anything or not.
If you do supply check=True, the exception you need to trap is CalledProcessError:
try:
# pointless code as such; see explanation above
subprocess.run(['cd', UserInput], check=True)
except subprocess.CalledProcessError:
print('Directory name %s misspelled, or you lack the permissions' % UserInput)
But even more fundamentally, allowing users to prod the system by running arbitrary unchecked input in a subprocess is a horrible idea. (Allowing users to run arbitrary shell script with shell=True is a monumentally, catastrophically horrible idea, so let's not even go there. Maybe see Actual meaning of shell=True in subprocess)
A somewhat more secure approach is to run the subprocess with a cwd= keyword argument.
# also vaguely pointless
subprocess.run(['true'], cwd=UserInput)
In this case, you can expect a regular FileNotFoundError if the directory does not exist, or a PermissionError if you lack the privileges.
You should probably still add check=True and be prepared to handle any resulting exception, unless you specifically don't care whether the subprocess succeeded. (There are actually cases where this makes sense, like when you grep for something but are fine with it not finding any matches, which raises an error if you use check=True.)
Perhaps see also Running Bash commands in Python
Is there a doc somewhere which indicates what the different return codes of python's subprocess check_output() command means? I'm seeing the returncode equal 3, and I have no idea what is setting that value and/or what it means.
Example code:
try:
output = subprocess.check_output(cmd,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print e.returncode
This code is printing 3.
The Python subprocess.check_output() doesn't itself return a code, it returns the output of the command being run. The doco can be found here.
If you're seeing an error code in that string, it's almost certainly specific to whatever you're trying to run, not a result of subprocess itself.
If, however, you're capturing the CalledProcessError exception caused by a non-zero return code (which can be extracted from the returncode attribute), that's still specific to whatever you're running.
In both cases, the documentation you need to check is that of whatever tool subprocess is running.
I'm writing a rather complex script that is using
asyncio.create_subprocess_exec(sub_cmd, *sub_cmd_args, stdout=PIPE, stderr=PIPE)
to wrap around another Python program -- that I can't modify permanently or otherwise include directly -- to capture its stdout/err for logging. The wrapped Python script is not using the -u (unbuffered) option so the wrapper program tends to log in big buffered blocks. If this were the regular subprocess.Popen I could just pass bufsize=1 to get what I want, namely line buffering. However if I add that to asyncio.create_subprocess_exec() they trap for that specifically and I get:
<snip>
File "/usr/lib64/python3.4/asyncio/subprocess.py", line 193, in create_subprocess_exec
stderr=stderr, **kwds)
File "/usr/lib64/python3.4/asyncio/base_events.py", line 642, in subprocess_exec
raise ValueError("bufsize must be 0")
ValueError: bufsize must be 0
I assume their trap is there for good reason so I wonder if there's some other way I can affect the transport buffering.
I first proved to myself that this was a indeed a pipe buffering issue by adding -u to the wrapped program's shebang line. I couldn't rely on that as a solution though because such a change would eventually get clobbered by OS updates.
I was able to resolve the issue though in a similar fashion though:
The wrapper program is the parent program of the pipe, so it controls the environment of its child programs.
Python should obey the PYTHONUNBUFFERED=1 in its inherited environment.
asyncio.create_subprocess_exec() does support an env= argument and most everything else that can be passed to subprocess.Popen(); perhaps a little under-documented but looking at the code makes this quite obvious.
So I changed my call to:
asyncio.create_subprocess_exec(sub_cmd, *sub_cmd_args, stdout=PIPE, stderr=PIPE, env={'PYTHONUNBUFFERED': '1'})
This worked perfectly and credit goes to my good friend and technical guru.
I have a system() command and I want to catch the exception it may generate. The code that I have is:
def test():
filename = "test.txt"
try:
cmd = "cp /Users/user1/Desktop/Test_Folder/"+filename+" /Users/user1/Desktop/"
output = system(cmd)
except:
print 'In the except'
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
test()
When I execute the above code and say the file that I want to copy is not present then the error is not caught and the code does not enter the except section. How can I catch such errors generated by system() commands?
Note: The above system() command is just an example. There are multiple such system() commands and each of them vary from one another
The system() command doesn't throw an exception on failure; it will simply return the exit status code of the application. If you want an exception thrown on failure, use subprocess.check_call, instead. (And, in general, using the subprocess module is superior in that it gives you greater control over the invocation as well as the ability to redirect the subprocess's standard input/output).
Note, though, that if most of the operations you are doing are simple filesystem operations like copying files from one location to another, that there are Python functions that do the equivalent. For example, shutil provides the ability to copy files from one location to another. Where there are Python functions to do the task, it is generally better to use those rather than invoke a sub process to do it (especially since the Python-provided methods may be able to do it more efficiently without forking a process, and the Python versions will also be more robust to cross-platform considerations).