I am trying to run a shell script using through Python using subprocess.Popen().
The shell script just has the following lines:
#!/bin/sh
echo Hello World
Following is the Python code:
print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
shellscript.wait()
for line in shellscript.stdout.readlines():
print(line)
print("SHELL SCRIPT RUN ENDED")
However, on running this, I am only getting the following output:
RUNNNING SHELL SCRIPT NOW
SHELL SCRIPT RUN ENDED
i.e. I am not getting the shell script output in between these 2 lines.
Moreover, when I remove the stderr=subprocess.PIPE part from the subprocess, I get the following output:
RUNNNING SHELL SCRIPT NOW
'km' is not defined as an internal or external command.
SHELL SCRIPT RUN ENDED
I am not able to understand how to resolve this, and run the shell script properly. Kindly guide. Thanks.
UPDATE:
I also tried the following change:
print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
out, err = shellscript.communicate()
print(out)
print("SHELL SCRIPT RUN ENDED")
I get the following output:
RUNNNING SHELL SCRIPT NOW
b''
SHELL SCRIPT RUN ENDED
The simple and straightforward fix is to not use bare Popen for this.
You also don't need a shell to run a subprocess; if the subprocess is a shell script, that subprocess itself will be a shell, but you don't need the help of the shell to run that script.
proc = subprocess.run(
['km/example/example1/source/test.sh'],
check=True, capture_output=True, text=True)
out = proc.stdout
If you really need to use Popen, you need to understand its processing model. But if you are just trying to get the job done, the simple answer is don't use Popen.
The error message actually looks like you are on Windows, and it tries to run km via cmd which thinks the slashes are option separators, not directory separators. Removing the shell=True avoids this complication, and just starts a process with the requested name. (This of course still requires that the file exists in the relative file name you are specifying. Perhaps see also What exactly is current working directory? and also perhaps switch to native Windows backslashes, with an r'...' string to prevent Python from trying to interpret the backslashes.)
Related
I have a python script which takes an input, formats it into a command which calls another script on the server, and then executes using subprocess:
import sys, subprocess
thingy = sys.argv[1]
command = 'usr/local/bin/otherscript.pl {0} &'.format(thingy)
command_list = command.split()
subprocess.call(command_list)
I append & to the end because otherscript.pl takes some time to execute, and I prefer to have run in the background. However, the script still seems to execute without giving me back control to the shell, and I have to wait until execution finishes to get back to my prompt. Is there another way to use subprocess to fully run the script in background?
& is a shell feature. If you want it to work with subprocess, you must specify shell=True like:
subprocess.call(command, shell=True)
This will allow you to run command in background.
Notes:
Since shell=True, the above uses command, not command_list.
Using shell=True enables all of the shell's features. Don't do this unless command including thingy comes from sources that you trust.
Safer Alternative
This alternative still lets you run the command in background but is safe because it uses the default shell=False:
p = subprocess.Popen(command_list)
After this statement is executed, the command will run in background. If you want to be sure that it has completed, run p.wait().
If you want to execute it in Background I recommend you to use nohup output that would normally go to the terminal goes to a file called nohup.out
import subprocess
subprocess.Popen("nohup usr/local/bin/otherscript.pl {0} >/dev/null 2>&1 &", shell=True)
>/dev/null 2>&1 & will not create output and will redirect to background
From python, I need to run a python file inside of git bash, while running in Windows.
That is, I have a configuration script written in python that calls other python scripts. Unfortunately, some of them use Unix commands, so they must be run using git bash in Windows.
Currently I'm using this:
cmd = f'{sys.executable} mydependency.py'
pipe = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
# waiting for pipe is handled later...
However, this doesn't work, giving me a cannot execute binary file message. How can I get it to run?
PS: For slightly more context, mydependency.py is actually the amalgamate.py script from the simdjson (https://github.com/simdjson/simdjson) project.
EDIT:
I have also attempted the following:
Switch to run or call instead of subprocess.Popen
Use f'{git_bash_path} {sys.executable} mydependency.py'
Change the shell and executable parameters of Popen,run and call
I found a solution:
cmd = git_bash_path # Found with glob.
pipe = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
pipe.communicate(input=f'{sys.executable} mydependency.py'.encode())
I'm not entirely sure why this works, if anyone has an explanation I'd be glad to hear it.
I am a beginner in Python and can you kindly help me understand the following concept.
If I do the following,
import subprocess
subprocess.run(['ls'])
Here we know that the key word argument, shell is set to False by default so that the 'ls' does not run on the shell. But my question is if it does not run on the shell, on where does it run and how can it give me an output?
I have a windows system but it should work the same.
For getting the output of subprocess you can use check_output.
On windows -
import subprocess
subprocess.check_output(["dir"], shell=True)
running this code without shell=True will result in an error.
If I want to run the code above with shell=False
I would do something like this -
subprocess.check_output(["cmd","/c","dir"], shell=False)
Notice -
On Unix with shell=True, the shell defaults to /bin/sh.
That means when you pass command and use shell=True
it will use /bin/sh to run that command.
I'm currently running an OpenELEC (XBMC) installation on a Raspberry Pi and installed a tool named "Hyperion" which takes care of the connected Ambilight. I'm a total noob when it comes to Python-programming, so here's my question:
How can I run a script that checks if a process with a specific string in its name is running and:
kill the process when it's running
start the process when it's not running
The goal of this is to have one script that toggles the Ambilight. Any idea how to achieve this?
You may want to have a look at the subprocess module which can run shell commands from Python. For instance, have a look at this answer. You can then get the stdout from the shell command to a variable. I suspect you are going to need the pidof shell command.
The basic idea would be along the lines of:
import subprocess
try:
subprocess.check_output(["pidof", "-s", "-x", "hyperiond"])
except subprocess.CalledProcessError:
# spawn the process using a shell command with subprocess.Popen
subprocess.Popen("hyperiond")
else:
# kill the process using a shell command with subprocess.call
subprocess.call("kill %s" % output)
I've tested this code in Ubuntu with bash as the process and it works as expected. In your comments you note that you are getting file not found errors. You can try putting the complete path to pidof in your check_output call. This can be found using which pidof from the terminal. The code for my system would then become
subprocess.check_output(["/bin/pidof", "-s", "-x", "hyperiond"])
Your path may differ. On windows adding shell=True to the check_output arguments fixes this issue but I don't think this is relevant for Linux.
Thanks so much for your help #will-hart, I finally got it working. Needed to change some details because the script kept saying that "output" is not defined. Here's how it now looks like:
#!/usr/bin/env python
import subprocess
from subprocess import call
try:
subprocess.check_output(["pidof", "hyperiond"])
except subprocess.CalledProcessError:
subprocess.Popen(["/storage/hyperion/bin/hyperiond.sh", "/storage/.config/hyperion.config.json"])
else:
subprocess.call(["killall", "hyperiond"])
I'm having trouble using mutilple subprocess calls back to back.
These 2 work fine:
subprocess.call(["gmake", "boot-tilera"], cwd="/home/ecorbett/trn_fp")
p = subprocess.Popen(["gmake", "run-tilera"], stdout=subprocess.PIPE, cwd="/home/ecorbett/trn_fp")
However, I get an error when I try to run this call directly after:
time.sleep(10)
subprocess.call(["./go2.sh"], cwd="/home/ecorbett/trn_fp/kem_ut")
I added sleep in there because I need a few seconds before I run the "./go2.sh" program. Not sure if that is the issue.
Any advice?
A possible reason why your shell script is working on the command-line is that the shebang line was not written correctly (or not written at all). See an example in which the script would work from a command line but not as a Python subprocess: Is this the right way to run a shell script inside Python?
If your shell script did not have a shebang line specified, it would work from command line because $SHELL is set in your environment and the script is taking that as a default. When running from a python subprocess, python does not know what it is and fails with OSError: [Errno 8] Exec format error. The subprocess.call() to gmake worked because it is a binary program and not a shell script. Using the argument shell=True gave an instruction to interpret the argument exactly as it would in a shell.
However, be careful about using shell=True in subprocess.call() as it may be insecure in some cases: subprocess Python docs.