We Keep Coding

Using python subprocess to use the "source" linux command with a bashrc file [duplicate]

This question already has answers here:
Calling the "source" command from subprocess.Popen
(9 answers)
Closed 8 months ago.
I am tasked with automating the process of running bash script using python. Unfortunately I am not responsible for the bash script itself, so I have no idea how it works. When I run the script directly in the terminal using source adastralrc.sh , it works perfectly and gives the desired output.
However when I try to get python to run the file by using subprocess and the exact same command as the argument:
import subprocess
commands = ['source' , 'adastralrc.sh']
p = subprocess.run(commands)
I get the following error:
Traceback (most recent call last):
File "test2.py", line 6, in <module>
p = subprocess.call(commands)
File "/usr/lib64/python3.6/subprocess.py", line 287, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'source adastralrc.sh': 'source adastralrc.sh'
(venv-c3dns) [vivegab#adl20213d1bld01 vivegab] (cth01/dns_mano_dev)
I am fairly inexperienced with subprocess but I thought that it was just an improved version of os.system() and should just enter the commands as if they were being typed by a person. So why am I getting the error above and what can be done to fix this?

def shell_source(script):
"""Sometime you want to emulate the action of "source" in bash,
settings some environment variables. Here is a way to do it."""
import subprocess, os
pipe = subprocess.Popen(". %s; env" % script, stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in output.splitlines()))
os.environ.update(env)
shell_source(adastralrc.sh)
Looks like a duplicate

How to run bash commands using subprocess.run on windows [duplicate]

This question already has answers here:
Python subprocess.run('ls',shell=True) not working on windows
(2 answers)
Closed 1 year ago.
I want to run shell scripts and git-bash commands using subprocess.run(), in python 3.7.4. When I run the simple example on the subprocess documentation page this happens:
import subprocess
subprocess.run(["ls", "-l"])
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\pycharm\project\envs\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\pycharm\project\envs\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\pycharm\project\envs\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
# it also fails with shell=True
subprocess.call(["ls", "-l"], shell=True)
'ls' is not recognized as an internal or external command,
operable program or batch file.
1
The message from shell=True is a message from windows cmd, which suggests subprocess is not sending commands to git-bash.
I am using a conda environment located in the project/envs/ folder for python. I have also installed git-bash.
I also tried setting the env and got the same error.
import os
import subprocess
my_env = os.environ.copy()
my_env["PATH"] = 'C:\Program Files\Git\;' + my_env["PATH"]
subprocess.run(['git-bash.exe', 'ls', '-l'], env=my_env)
Traceback (most recent call last):
File "<input>", line 3, in <module>
File "C:\pycharm\project\envs\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\pycharm\project\envs\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:n\pycharm\project\envs\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I can get it to run by pointing at the git-bash.exe, but it returns an empty string instead of the files in my directory
import subprocess
subprocess.run(['C:\Program Files\Git\git-bash.exe', 'ls', '-l'], capture_output=True)
CompletedProcess(args=['C:\\Program Files\\Git\\git-bash.exe', 'ls', '-l'], returncode=0, stdout=b'', stderr=b'')
I would appreciate any advice on the best way to get this working as shown on the subprocess documentation page.

I found that I can run commands using ...Git\bin\bash.exe instead of the ...\Git\git-bash.exe, like this:
import subprocess
subprocess.run(['C:\Program Files\Git\\bin\\bash.exe', '-c','ls'], stdout=subprocess.PIPE)
CompletedProcess(args=['C:\\Program Files\\Git\\bin\\bash.exe', '-c', 'ls'], returncode=0, stdout=b'README.md\n__pycache__\nconda_create.sh\nenvs\nmain.py\ntest.sh\nzipped\n')

Try this
p = subprocess.Popen(("ls", "-l"), stdout=subprocess.PIPE)
nodes = subprocess.check_output(("grep"), stdin=p.stdout)
p.wait()

ls is Linux shell command for listing files and directories
dir is Windows command line command for listing files and directories
Try to run dir in Windows command line. If it works, try to run the same command using python subprocess:
import subprocess
subprocess.run(["dir"])

For a machine with Windows Operating System, Try the following
import subprocess
subprocess.run(["dir", "/p"], shell=True)
"ls" is replaced with "dir", "-l" is replaced with "/l" and the "shell" is set to true
For a machine with Linux/Mac Operating System, Try the following
import subprocess
subprocess.run(["ls", "-l"])

running separate python programs with subprocess

I am trying to create a script that will run my other python programs. I am new to subprocess module so this is a bit confusing to me.
project structure
/qe-functional
/qe
/tests
cron_functional.py
test_web_events.py
setup.sh
cron_functional.py
print(os.getcwd())
# print(subprocess.check_output('ls'))
runtag = "daily_run_" + datetime.today().strftime("%m_%d_%y")
testrun = "source ../../setup.sh; ./test_web_events.py -n 10 -t prf -E ctg-businessevent -p post {}".format(runtag)
cmd = testrun.split()
print(cmd)
subprocess.check_output(cmd)
output
$ python cron_functional.py
/Users/bli1/Development/QE/qe-functional/qe/tests
['source', '../../setup.sh;', './test_web_events.py', '-n', '10', '-t', 'prf', '-E', 'ctg-businessevent', '-p', 'post', 'daily_run_05_26_15']
Traceback (most recent call last):
File "cron_functional.py", line 11, in <module>
subprocess.check_output(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

source is an internal shell command, not an executable. What you want is not to run one source command with 11 arguments, but a one-liner shell script. You need to pass the whole script as one string to be interpreted by the shell.
subprocess.check_output(testrun, shell=True)
You haven't said what setup.sh does. If it's setting up environment variables and changing the working directory, consider doing that within Python instead. Then you can run
subprocess.check_output(['./test_web_events.py', '-n', '10', …, '-p', 'post', runtag])
… without involving the shell.

How to run bash 'tc' command within python?

I want to us traffic control of the Linux kernel with Python to simulate lost, corrupt and duplicate packages. I'm already able to configure this with the Linux Terminal, but I have to use python.
bash cmd works:
tc filter show dev eth1
python doesn't work:
>>> subprocess.call(["tc", "filter", "show", "dev", "eth1"])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/subprocess.py", line 470, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Thanks.

The python subprocess doesn't know about your shell environment. So provide absolute path to your command, something like:
subprocess.call(["/sbin/tc", "filter", "show", "dev", "eth1"])
find the exact location with command which tc in your shell.

The basic way if you don't need any special control is to use os.system() to launch a command as if you were in your shell command line (no need to specify the full path if in your $PATH):
import os
os.system("tc filter show dev eth1")
This should work exactly as if you did in your cmd:
$ tc filter show dev eth1

Cannot run Python from Cron

I have a python script that will read the temperature of from a probe on the GPIO pins of a Raspberry-Pi, and will append that temperature to a log file. Running the script form terminal with sudo permissions works fine:
sudo python /home/pi/temp.py
I've attempted to run the script every 15 minutes from sudo's crontab file with the line:
*/15 * * * * python /home/pi/temp.py
This fails, with the output being
Traceback (most recent call last):
File "/home/pi/temp.py", line 8, in <module>
subprocess.call(['modprobe', 'w1-gpio'])
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I know the issue is with the modprobe subprocess call, but I can't identify what exactly. In my script, I have the following code related to the issue:
import subprocess
subprocess.call(['modprobe', 'w1-gpio'])
subprocess.call(['modprobe', 'w1-therm'])

This is because cron has its own PATH variable and doesn't use the same path that you do.
For that reason, it would be advisable to call any programs that you use (especially through python's subprocess) with an absolute path to the executable
You could do which modprobe on the commandline to find where modprobe lives (probably in /bin/), and then change your call in subprocess.py to subprocess.call(['/bin/modprobe', 'w1-gpio'])