I wrote a simple piece of code:
import subprocess
p=subprocess.Popen('mkdir -p ./{a,b,c}', shell=True, stderr=subprocess.STDOUT)
p.wait()
Unfortunately, it not always behaves the way I'd expect. I.e, when I run it on my PC, everything is OK (ls -l gives me three dirs: a, b and c). But when my colleague runs it on his desktop, he gets... one dir named: '{a,b,c}' ... We both use Python 2.7.3. Why is that? How would you fix it?
I tried to find the answer by myself. According to manual:
"args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence."
So I tried to execute the code in shell:
python -c "import subprocess; p=subprocess.Popen(['mkdir', '-p', './{ea,fa,ga}'], shell=True, stderr=subprocess.STDOUT); p.wait()"
And I got:
mkdir: missing operand
I will be thankful for any advice
Thanks!
The ./{a,b,c} syntax is bash syntax, not supported by all shells.
The documentation says:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
So your command only works if /bin/sh is symlinked to a shell that supports that syntax, like bash or zsh. Your colleague is probably using dash or another shell that doesn't support this.
You should no be relying in something like a user's default shell. Instead, write the full command with the full expansion:
p = subprocess.Popen('mkdir -p ./a ./b ./c', shell=True, stderr=subprocess.STDOUT)
There's several problems here.
First: if you are using a sequence of arguments, do not set "shell = True" (this is recommended in the Popen manual). Set it to False, and you'll see that your mkdir command will be accepted.
"./{a,b,c}" is AFAIK a specific syntax in bash. If your colleague is using a different shell, it will probably not work, or behave differently.
You should use the python "mkdir" command instead of calling a shell command, it will work whatever the server / shell / OS.
Thank you all for your answers.
It seems, that the best way is simply use /bin/sh syntax. I changed my code to use:
'mkdir -p ./a ./b ./c'
as you suggested.
I avoided to use mkdir() function, because I am writing a scripts with plenty of system calls, and I wanted to provide elegant --dry-run option (so I could list all of the commands).
Problem solved - thank you!
The os.mkdir(path,[mode]) method are as far as I understand safer to use when working on multiplatform projects.
os.mkdir(os.getcwd()/a)
However its not as elegant as taking the subprocess approach.
Related
Im using Ubuntu 20 with zsh. When I using subprocess.call, it always using bash to exec command but not zsh. How should I do to fix this?
No, it uses sh regardless of which shell is your login shell.
There is a keyword argument to select a different shell, but you should generally run as little code as possible in a subshell; mixing nontrivial shell script with Python means the maintainer has to understand both languages.
whatever = subprocess.run(
'echo $SHELL',
shell=True, executable='/usr/bin/zsh',
check=True)
(This will echo your login shell, so the output would be /usr/bin/zsh even if you ran this without executable, or with Bash instead.)
In many situations, you should avoid shell=True entirely if you can.
I am a beginner in Python and can you kindly help me understand the following concept.
If I do the following,
import subprocess
subprocess.run(['ls'])
Here we know that the key word argument, shell is set to False by default so that the 'ls' does not run on the shell. But my question is if it does not run on the shell, on where does it run and how can it give me an output?
I have a windows system but it should work the same.
For getting the output of subprocess you can use check_output.
On windows -
import subprocess
subprocess.check_output(["dir"], shell=True)
running this code without shell=True will result in an error.
If I want to run the code above with shell=False
I would do something like this -
subprocess.check_output(["cmd","/c","dir"], shell=False)
Notice -
On Unix with shell=True, the shell defaults to /bin/sh.
That means when you pass command and use shell=True
it will use /bin/sh to run that command.
I am trying to write a python script that will execute a bash script I have on my Windows machine. Up until now I have been using the Cygwin terminal so executing the bash script RunModels.scr has been as easy as ./RunModels.scr. Now I want to be able to utilize subprocess of Python, but because Windows doesn't have the built in functionality to handle bash I'm not sure what to do.
I am trying to emulate ./RunModels.scr < validationInput > validationOutput
I originally wrote this:
os.chdir(atm)
vin = open("validationInput", 'r')
vout = open("validationOutput", 'w')
subprocess.call(['./RunModels.scr'], stdin=vin, stdout=vout, shell=True)
vin.close()
vout.close()
os.chdir(home)
But after spending a while trying to figure out why my access was denied, I realized my issue wasn't the file permissions but the fact that I was trying to execute a bash file on Windows in general. Can someone please explain how to execute a bash script with directed input/output on windows using a python script?
Edit (Follow up Question):
Thanks for the responses, I needed the full path to my bash.exe as the first param. Now however, command line calls from within RunModels.scr come back in the python output as command not found. For example, ls, cp, make. Any suggestions for this?
Follow up #2:
I updated my call to this:
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
The error I now get is /usr/bin/bash: RunModels.scr: No such file or directory.
Using cwd does not seem to have any effect on this error, either way the subprocess is looking in /usr/bin/bash for RunModels.scr.
SELF-ANSWERED
I needed to specify the path to RunModels.scr in the call as well as using cwd.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'C:\\path\\dir_where_RunModels\\RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
But another problem...
Regardless of specifying cwd, the commands executed by RunModels.scr are throwing errors as if RunModels.scr is in the wrong directory. The script executes, but cp and cd throw the error no such file or directory. If I navigate to where RunModels.scr is through the command line and execute it the old fashioned way I don't get these errors.
Python 3.4 and below
Just put bash.exe in first place in your list of subprocess.call arguments. You can remove shell=True, that's not necessary in this case.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Depending on how bash is installed (is it in the PATH or not), you might have to use the full path to the bash executable.
Python 3.5 and above
subprocess.call() has been effectively replaced by subprocess.run().
subprocess.run(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Edit:
With regard to the second question, you might need to add the -l option to the shell invocation to make sure it reads all the restart command files like /etc/profile. I presume these files contain settings for the $PATH in bash.
Edit 2:
Add something like pwd to the beginning of RunModels.scr so you can verify that you are really in the right directory. Check that there is no cd command in the rc-files!
Edit 3:
The error /usr/bin/bash: RunModels.scr: No such file or directory can also be generated if bash cannot find one of the commands that are called in the script. Try adding the -v option to bash to see if that gives more info.
A better solution than the accepted answer is to use the executable keyword argument to specify the path to your shell. Behind the curtain, Python does something like
exec([executable, '-c`, subprocess_arg_string])
So, concretely, in this case,
subprocess.call(
'./RunModels.scr',
stdin=vin, stdout=vout,
shell=True,
executable="C:/cygwin64/bin/bash.exe")
(Windows thankfully lets you use forward slashes instead of backslashes, so you can avoid the requirement to double the backslashes or use a raw string.)
I know you can run Linux terminal commands through Python scripts using subprocess
subprocess.call(['ls', '-l']) # for linux
But I can't find a way to do the same thing on windows
subprocess.call(['dir']) # for windows
is it possible using Python without heavy tinkering?
Should I stick to good old fashioned batch files?
dir is not a file, it is an internal command, so the shell keyword must be set to True.
subprocess.call(["dir"], shell=True)
Try this
import os
os.system("windows command")
ex: for date
os.system("date")
Almost everyone's answers are right but it seems I can do what I need using os.popen -- varStr = os.popen('dir /b *.py').read()
First of all, to get a directory listing, you should rather use os.listdir(). If you invoke dir instead, you'll have to parse its output to make any use of it, which is lots of unnecessary work and is error-prone.
Now,
dir is a cmd.exe built-in command, it's not a standalone executable. cmd.exe itself is the executable that implements it.
So, you have two options (use check_output instead of check_call if you need to get the output instead of just printing it):
use cmd's /C switch (execute a command and quit):
subprocess.check_call(['cmd','/c','dir','/s'])
use shell=True Popen() option (execute command line through the system shell):
subprocess.check_call('dir /s', shell=True)
The first way is the recommended one. That's because:
In the 2nd case, cmd, will do any shell transformations that it normally would (e.g. splitting the line into arguments, unquoting, environment variable expansion etc). So, your arguments may suddenly become something else and potentially harmful. In particular, if they happen to contain any spaces and cmd special characters and/or keywords.
shell=True uses the "default system shell" (pointed to via COMSPEC environment variable in the case of Windows), so if the user has redefined it, your program will behave unexpectedly.
Consider this command
strings --radix=d --encoding={b,l} abc.exe >> xyz.txt
When I run this on the Ubuntu terminal it works without any problems.
However when I use it through a python code:
import os
os.system("strings --radix=d --encoding={b,l} abc.exe >> xyz.txt")
Its not working.
If I remove the "encoding" then it works fine in both cases.
However, I need to get Unicode strings so that part is necessary.
Anyone have any solutions?
ubuntu uses dash by default as /bin/sh, and bash for login shells.
So in your terminal --encoding={b,l} is propably expanded by bash to --encoding=b --encoding=l, while dash (probably called by os.system as /bin/sh) does no such expansion and it remains --encoding={b,l}
The easiest way is to explicitly expand the encoding parameter and don't leave that to the shell, then it will work with any shell.
And you should use the subprocess module instead of os.system(). Just be aware that when using the shell=True argument, it will also call the default /bin/sh, which isn't guaranteed to be bash.
You don't need shell=True, you can pass a list of args and write the stdout to a file:
from subprocess import check_call
with open('xyz.txt',"a") as out:
check_call(['strings', '--radix=d', '--encoding={b,l}', 'abc.exe'],stdout=out)
A good explanation here of what shell=True does
os.system is outdated, use subprocess instead. u also should use shell=True to have sh like behaviour:
import subprocess
cmd = "strings --radix=d --encoding={b,l} abc.exe >> xyz.txt"
subprocess.check_call(cmd, shell=True)
It also throws exception in case of call failure!