I am using subprocess.Popen to execute an OS command. Here is what I am trying to emulate in my code:
ssh -T myhost < /path/to/some/file
It works fine like this:
def runWorkerCode(filer, filename):
command = "/usr/bin/ssh -T " + filer + " < /devel/myscript"
try:
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
out, _ = p.communicate()
except Exception:
print "Error: %s" % Exception
sys.exit(1)
return out.rstrip().split('\n')
But the following calls to Popen do not work:
p = subprocess.Popen(["/usr/bin/ssh", "-T", filer, "<", "/devel/myscript"], stdout=subprocess.PIPE, shell=True)
p = subprocess.Popen(["/usr/bin/ssh -T", filer, "< /devel/myscript"], stdout=subprocess.PIPE, shell=True)
I tried a few other combinations but only method I can get to work is defining the command variable and only providing it to Popen(). I've also tried shell=False.
The first method works but the latter approach seems "cleaner" to me.
Why doesn't Popen allow me to specify the arguments in a list?
When you use shell=True on UNIX, you should provide your arguments as a string. When you provide a list, subprocess interprets the first item in the list as your entire command string, and the rest of the items in the list as arguments passed to the shell itself, rather than your command. So in your example above, you're ending up with something like this:
/bin/sh -T filer < /dev/myscript -c "/usr/sbin/ssh"
Definitely not what you meant!
Conversely, when you use shell=False, you can only pass a string if you're running a single command with no arguments. If you do have arguments, have to pass the comamnd as a sequence. You also can't use shell redirection via the < character, because there is no shell involved.
If you want to use shell=False, you can use the stdin keyword argument to pass a file handle to /dev/myscript:
f = open("/dev/myscript")
p = subprocess.Popen(["/usr/bin/ssh", "-T", filer], stdout=subprocess.PIPE, stdin=f, shell=False)
The rules for when to pass a string vs. when to pass a sequence are pretty confusing, especially when you bring Windows into the mix as well. I would read the documentation carefully to try to understand it all. Check out both the section on args and the section on shell.
Related
I am trying to send two simple commands using subprocess.run & trying to store results in a variable then print it but for one arg the output is coming for subprocess.run & for other its empty
Arg are "help" & "adb devices"
command I am sending which returns the output
result = subprocess.run("help", capture_output=True, text=True, universal_newlines=True)
print(result.stdout)
but this command with a different arg is not returning
result = subprocess.run("adb devices", capture_output=True, text=True, universal_newlines=True)
print(result.stdout)
If I try the same command with subprocess.checkoutput it returns the output can anyone explain what exactly is going on here
Is there any specific usage scenario's for these command's like when to use which one ?
c = subprocess.check_output(
"adb devices", shell=True, stderr=subprocess.STDOUT)
print(c)
output - b'List of devices attached\r\n\r\n'
It is because from the python documentation here:
run method
run method accepts the first parameter as arguments and not string.
So you can try passing the arguments in a list as:
result = subprocess.run(['abd', 'devices'], capture_output=True, text=True, universal_newlines=True)
Also,
check_output method accepts args but it has a parameter call "shell = True" Therefore, it works for multi-word args.
If you want to use the run method without a list, add shell=True in the run method parameter. (I tried for "man ls" command and it worked).
I am facing difficulties calling a command line from my script.I run the script but I don't get any result. Through this command line in my script I want to run a tool which produces a folder that has the output files for each line.The inputpath is already defined. Can you please help me?
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
os.system(cmd)
You really want to use the Python standard library module subprocess. Using functions from that module, you can construct you command line as a list of strings, and each would be processed as one file name, option or value. This bypasses the shell's escaping, and eliminates the need to massage you script arguments before calling.
Besides, your code would not work, because the body block of the for statement is not indented. Python would simply not accept this code (could be you pasted into the questiong without the proper indentations).
as mentioned before, executing command vias: os.system(command) is not recomended. please use subprocess (read in python docs about this modulesubprocess_module_docs). see the code here:
for command in input_file:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
# use this if you want to communicate with child process
# p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()
# --- do the rest
I usually do like this for static command
from subprocess import check_output
def sh(command):
return check_output(command, shell=True, universal_newlines=True)
output = sh('echo hello world | sed s/h/H/')
BUT THIS IS NOT SAFE!!! It's vunerable to shell injection you should do
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
output = sh('echo hello world')
The difference is subtle but important. shell=True will create a new shell, so pipes, etc will work. I use this when I have a big command line with pipes and that is static, I mean, it do not depend on user input. This is because this variant is vunerable to shell injection, a user can input something; rm -rf / and it will run.
The second variant only accepts one command, it will not spawn a shell, instead it will run the command directly. So no pipes and such shell things will work, and is safer.
universal_newlines=True is for getting output as string instead of bytes. Use it for text output, if you need binary output just ommit it. The default is false.
So here is the full example
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
sh(cmd)
Ps: I didn't test this
When I run git tag -l v1.1.{[0-9],[0-9][0-9]} in my shell I get results, however
def run_command(cmd_args, number_of_retry=5,
timeout_in_second=90, suppress_output=False):
proc = subprocess.Popen(cmd_args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
timer = threading.Timer(timeout_in_second, proc.kill)
timer.start()
out, err = proc.communicate()
timer.cancel()
if proc.returncode and number_of_retry >= 1:
if not suppress_output:
print('Command "{}" failed/timeout, retrying...'.format(
' '.join(cmd_args)))
return run_command(cmd_args,
number_of_retry - 1,
timeout_in_second + 10,
suppress_output=suppress_output)
return proc.returncode, out, err
return_code, out, _ = run_command(
['git', 'tag', '-l', "v1.1.{[0-9],[0-9][0-9]}"],
number_of_retry=0,
suppress_output=False)
yields no results. Changing the pattern in my code to v1.1.[0-9][0-9] yields the results that match, as does v1.1.[0-9], so its an issue with the brackets. Anyone know how this could be? Is there another way of saying the same expression, or at least seeing the bash subprocess run so I can verify the correct command is being run? (subprocess and threading are python modules)
The curly-brace-and-comma syntax ("brace expansion") in v1.1.{[0-9],[0-9][0-9]} is an instruction to your shell, not to git; it tells the shell to split the argument into two (after which, since they contain unquoted glob characters, the shell will by default try to expand each of those resulting arguments as a glob -- hopefully you don't actually intend that behavior, and thus won't mind it being gone), and then to pass git two separate arguments, one for the content on each side of the commas within the curly braces.
Absent the previously-discussed globbing, the result of this shell expansion is equivalent to the following Python code:
run_command(['git', 'tag', '-l', 'v1.1.[0-9]', 'v1.1.[0-9][0-9]'],
number_of_retry=0, suppress_output=False)
If you actually do rely on the shell to replace v1.1.[0-9] with a list of entries in the current directory matching that pattern should any such entries exist, the native-Python equivalent would be more like:
# to behave like default bash behavior (without running ''shopt -s nullglob''):
import glob
def expandGlob(word):
return glob.glob(word) or [word]
run_command(['git', 'tag', '-l'] + expandGlob('v1.1.[0-9]') + expandGlob('v1.1.[0-9][0-9]'),
number_of_retry=0, suppress_output=False)
On Windows I successfully run this:
cmd = ["gfortran", "test.f90", "-o", "test.exe", "&&", "test.exe"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
print(p.stdout.read())
On Ubuntu I change cmd to:
cmd = ["gfortran", "test.f90", "-o", "test", "&&", "./test"]
and get:
gfortran: fatal error: no input files
compilation terminated.
I want to retain cmd as a list, instead making it a string.
How can I make this work?
It's impossible. When cmd is a list, it has different meaning when shell is True. Quoting docs:
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself.
Use ' '.join(shlex.quote(arg) for arg in cmd) (pipes.quote in Python2) when passing list of arguments to Popen with shell=True for expected behavior. Original list won't be mutated, string will be built before passing to function and garbage collected as soon as it's possible.
This is what the docs state:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself.
If you want to retain the arguments as a list, you can join them while calling Popen:
p = subprocess.Popen(' '.join(cmd), stdout=subprocess.PIPE, shell=True)
It appears that you are using shell=True to allow you to do && on the command line. This is not strictly necessary if you are already using subprocess anyway:
cmd1 = ["gfortran", "test.f90", "-o", "test.exe"]
cmd2 = ["test.exe"]
if not subprocess.call(cmd1):
p = subprocess.Popen(cmd2, ...)
out, err = p.communicate()
...
I'm using Windows 7, and I've tried this under Python 2.6.6 and Python 3.2.
So I'm trying to call this command line from Python:
netstat -ano | find ":80"
under Windows cmd, this line works perfectly fine.
So,
1st attempt:
output = subprocess.Popen(
[r'netstat -ano | find ":80"'],
stdout=subprocess.PIPE,
shell=True
).communicate()
An error is raised that 'find' actually didn't receive correct parameter (e.g. 'find ":80" \'):
Access denied - \
2nd attempt:
#calling netstat
cmd_netstat = subprocess.Popen(
['netstat','-ano'],
stdout = subprocess.PIPE
)
#pipelining netstat result into find
cmd_find = subprocess.Popen(
['find','":80"'],
stdin = cmd_netstat.stdout,
stdout = subprocess.PIPE
)
Again, the same error is raised.
Access denied - \
What did I do wrong? :(
EDIT:
3rd attempt (As #Pavel Repin suggested):
cmd_netstat = subprocess.Popen(
['cmd.exe', '-c', 'netstat -ano | find ":80"'],
stdout=subprocess.PIPE
).communicate()
Unfortunately, subprocess with ['cmd.exe','-c'] results in something resembling deadlock or a blank cmd window. I assume '-c' is ignored by cmd, resulting in communicate() waiting indefinitely for cmd termination. Since this is Windows, my bet bet is cmd only accepts parameter starting with slash (/). So I substituted '-c' with '/c':
cmd_netstat = subprocess.Popen(
['cmd.exe', '/c', 'netstat -ano | find ":80"'],
stdout=subprocess.PIPE
).communicate()
And...back to the same error:
Access denied - \
EDIT:
I gave up, I'll just process the string returned by 'netstat -ano' in Python. Might this be a bug?
What I suggest is that you do the maximum inside Python code. So, you can execute the following command:
# executing the command
import subprocess
output = subprocess.Popen(['netstat', '-ano'], stdout=subprocess.PIPE).communicate()
and then by parsing the output:
# filtering the output
valid_lines = [ line for line in output[0].split('\r\n') if ':80' in line ]
You will get a list of lines. On my computer, the output looks like this for port number 1900 (no html connexion active):
[' UDP 127.0.0.1:1900 *:* 1388', ' UDP 192.xxx.xxx.233:1900 *:* 1388']
In my opinion, this is easier to work with.
Note that :
option shell=True is not mandatory, but a command-line window is opened-closed quickly. See what suits you the most, but take care of command injection;
list of Popen arguments shall be a list of string. Quoting of the list parts is not necessary, subprocess will take care of it for you.
Hope this helps.
EDIT: oops, I missed the last line of the edit. Seems you've already got the idea on your own.
So I revisited this question, and found two solutions (I switched to Python 2.7 sometime ago, so I'm not sure about Python 2.6, but it should be the same.):
Replace find with findstr, and remove doublequotes
output = subprocess.Popen(['netstat','-ano','|','findstr',':80'],
stdout=subprocess.PIPE,
shell=True)
.communicate()
But this doesn't explain why "find" cannot be used, so:
Use string parameter instead of list
output = subprocess.Popen('netstat -ano | find ":80"',
stdout=subprocess.PIPE,
shell=True)
.communicate()
or
pipeout = subprocess.Popen(['netstat', '-ano'],
stdout = subprocess.PIPE)
output = subprocess.Popen('find ":80"',
stdin = pipeout.stdout,
stdout = subprocess.PIPE)
.communicate()
The problem arise from the fact that: ['find','":80"'] is actually translated into ['find,'\":80\"'].
Thus the following command is executed in Windows command shell:
>find \":80\"
Access denied - \
Proof:
Running:
output = subprocess.Popen(['echo','find','":80"'],
stdout=subprocess.PIPE,
shell=True)
.communicate()
print output[0]
returns:
find \":80\"
Running:
output = subprocess.Popen('echo find ":80"',
stdout=subprocess.PIPE,
shell=True)
.communicate()
print output[0]
returns:
find ":80"
New answer, after reading this old question again: this may be due to the two following facts:
The pipe operator executes the following commands in a sub-shell; see for instance this interesting consequence).
Python itself uses the pipe as a way to get the results back:
Note that (...) to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Not sure if this 'conflict' is kind of a bug, or a design choice though.