I am a beginner in Python and can you kindly help me understand the following concept.
If I do the following,
import subprocess
subprocess.run(['ls'])
Here we know that the key word argument, shell is set to False by default so that the 'ls' does not run on the shell. But my question is if it does not run on the shell, on where does it run and how can it give me an output?
I have a windows system but it should work the same.
For getting the output of subprocess you can use check_output.
On windows -
import subprocess
subprocess.check_output(["dir"], shell=True)
running this code without shell=True will result in an error.
If I want to run the code above with shell=False
I would do something like this -
subprocess.check_output(["cmd","/c","dir"], shell=False)
Notice -
On Unix with shell=True, the shell defaults to /bin/sh.
That means when you pass command and use shell=True
it will use /bin/sh to run that command.
Related
I am trying to run a shell script using through Python using subprocess.Popen().
The shell script just has the following lines:
#!/bin/sh
echo Hello World
Following is the Python code:
print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
shellscript.wait()
for line in shellscript.stdout.readlines():
print(line)
print("SHELL SCRIPT RUN ENDED")
However, on running this, I am only getting the following output:
RUNNNING SHELL SCRIPT NOW
SHELL SCRIPT RUN ENDED
i.e. I am not getting the shell script output in between these 2 lines.
Moreover, when I remove the stderr=subprocess.PIPE part from the subprocess, I get the following output:
RUNNNING SHELL SCRIPT NOW
'km' is not defined as an internal or external command.
SHELL SCRIPT RUN ENDED
I am not able to understand how to resolve this, and run the shell script properly. Kindly guide. Thanks.
UPDATE:
I also tried the following change:
print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
out, err = shellscript.communicate()
print(out)
print("SHELL SCRIPT RUN ENDED")
I get the following output:
RUNNNING SHELL SCRIPT NOW
b''
SHELL SCRIPT RUN ENDED
The simple and straightforward fix is to not use bare Popen for this.
You also don't need a shell to run a subprocess; if the subprocess is a shell script, that subprocess itself will be a shell, but you don't need the help of the shell to run that script.
proc = subprocess.run(
['km/example/example1/source/test.sh'],
check=True, capture_output=True, text=True)
out = proc.stdout
If you really need to use Popen, you need to understand its processing model. But if you are just trying to get the job done, the simple answer is don't use Popen.
The error message actually looks like you are on Windows, and it tries to run km via cmd which thinks the slashes are option separators, not directory separators. Removing the shell=True avoids this complication, and just starts a process with the requested name. (This of course still requires that the file exists in the relative file name you are specifying. Perhaps see also What exactly is current working directory? and also perhaps switch to native Windows backslashes, with an r'...' string to prevent Python from trying to interpret the backslashes.)
Im using Ubuntu 20 with zsh. When I using subprocess.call, it always using bash to exec command but not zsh. How should I do to fix this?
No, it uses sh regardless of which shell is your login shell.
There is a keyword argument to select a different shell, but you should generally run as little code as possible in a subshell; mixing nontrivial shell script with Python means the maintainer has to understand both languages.
whatever = subprocess.run(
'echo $SHELL',
shell=True, executable='/usr/bin/zsh',
check=True)
(This will echo your login shell, so the output would be /usr/bin/zsh even if you ran this without executable, or with Bash instead.)
In many situations, you should avoid shell=True entirely if you can.
Consider this command
strings --radix=d --encoding={b,l} abc.exe >> xyz.txt
When I run this on the Ubuntu terminal it works without any problems.
However when I use it through a python code:
import os
os.system("strings --radix=d --encoding={b,l} abc.exe >> xyz.txt")
Its not working.
If I remove the "encoding" then it works fine in both cases.
However, I need to get Unicode strings so that part is necessary.
Anyone have any solutions?
ubuntu uses dash by default as /bin/sh, and bash for login shells.
So in your terminal --encoding={b,l} is propably expanded by bash to --encoding=b --encoding=l, while dash (probably called by os.system as /bin/sh) does no such expansion and it remains --encoding={b,l}
The easiest way is to explicitly expand the encoding parameter and don't leave that to the shell, then it will work with any shell.
And you should use the subprocess module instead of os.system(). Just be aware that when using the shell=True argument, it will also call the default /bin/sh, which isn't guaranteed to be bash.
You don't need shell=True, you can pass a list of args and write the stdout to a file:
from subprocess import check_call
with open('xyz.txt',"a") as out:
check_call(['strings', '--radix=d', '--encoding={b,l}', 'abc.exe'],stdout=out)
A good explanation here of what shell=True does
os.system is outdated, use subprocess instead. u also should use shell=True to have sh like behaviour:
import subprocess
cmd = "strings --radix=d --encoding={b,l} abc.exe >> xyz.txt"
subprocess.check_call(cmd, shell=True)
It also throws exception in case of call failure!
I wrote a simple piece of code:
import subprocess
p=subprocess.Popen('mkdir -p ./{a,b,c}', shell=True, stderr=subprocess.STDOUT)
p.wait()
Unfortunately, it not always behaves the way I'd expect. I.e, when I run it on my PC, everything is OK (ls -l gives me three dirs: a, b and c). But when my colleague runs it on his desktop, he gets... one dir named: '{a,b,c}' ... We both use Python 2.7.3. Why is that? How would you fix it?
I tried to find the answer by myself. According to manual:
"args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence."
So I tried to execute the code in shell:
python -c "import subprocess; p=subprocess.Popen(['mkdir', '-p', './{ea,fa,ga}'], shell=True, stderr=subprocess.STDOUT); p.wait()"
And I got:
mkdir: missing operand
I will be thankful for any advice
Thanks!
The ./{a,b,c} syntax is bash syntax, not supported by all shells.
The documentation says:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
So your command only works if /bin/sh is symlinked to a shell that supports that syntax, like bash or zsh. Your colleague is probably using dash or another shell that doesn't support this.
You should no be relying in something like a user's default shell. Instead, write the full command with the full expansion:
p = subprocess.Popen('mkdir -p ./a ./b ./c', shell=True, stderr=subprocess.STDOUT)
There's several problems here.
First: if you are using a sequence of arguments, do not set "shell = True" (this is recommended in the Popen manual). Set it to False, and you'll see that your mkdir command will be accepted.
"./{a,b,c}" is AFAIK a specific syntax in bash. If your colleague is using a different shell, it will probably not work, or behave differently.
You should use the python "mkdir" command instead of calling a shell command, it will work whatever the server / shell / OS.
Thank you all for your answers.
It seems, that the best way is simply use /bin/sh syntax. I changed my code to use:
'mkdir -p ./a ./b ./c'
as you suggested.
I avoided to use mkdir() function, because I am writing a scripts with plenty of system calls, and I wanted to provide elegant --dry-run option (so I could list all of the commands).
Problem solved - thank you!
The os.mkdir(path,[mode]) method are as far as I understand safer to use when working on multiplatform projects.
os.mkdir(os.getcwd()/a)
However its not as elegant as taking the subprocess approach.
I may not at all understand this correctly, but I am trying to allow a Python program to interface with a subprocess that runs commands as if on a Linux shell.
For example, I want to be able to run "cd /" and then "pwd later in the program and get "/".
I am currently trying to use subprocess.Popen and the communicate() method to send and receive data. The first command, sent with the Popen constructor, runs fine and gives proper output. But I cannot send another command via communicate(input="pwd").
My code so far:
from subprocess i
term=Popen("pwd", stdout=PIPE, stdin=PIPE)
print(flush(term.communicate()))
term.communicate(input="cd /")
print(flush(term.communicate(input="pwd")))
Is there a better way to do this? Thanks.
Also, I am running Python 3.
First of all, you need to understand that running a shell command and running a program aren't the same thing.
Let me give you an example:
>>> import subprocess
>>> subprocess.call(['/bin/echo', '$HOME'])
$HOME
0
>>> subprocess.call(['/bin/echo $HOME'], shell=True)
/home/kkinder
0
Notice that without the shell=True parameter, the text of $HOME is not expanded. That's because the /bin/echo program doesn't parse $HOME, Bash does. What's really happening in the second call is something analogous to this:
>>> subprocess.call(['/bin/bash', '-c', '/bin/echo $HOME'])
/home/kkinder
0
Using the shell=True parameter basically says to the subprocess module, go interpret this text using a shell.
So, you could add shell=True, but then the problem is that once the command finishes, its state is lost. Each application in the stack has its own working directory. So what the directory is will be something like this:
bash - /foo/bar
python - /foo
bash via subprocess - /
After your command executes, the python process's path stays the same and the subprocess's path is discarded once the shell finishes your command.
Basically, what you're asking for isn't practical. What you would need to do is, open a pipe to Bash, interactively feed it commands your user types, then read the output in a non-blocking way. That's going to involve a complicated pipe, threads, etc. Are you sure there's not a better way?