Im trying to call a shell script from my python with a parameter but the parameter is not being passed
My Shellscript:
echo "Inside shell"
echo $0
echo $1
cd $1
pwd
for file in *.csv
do
split -l 50000 -d -a 4 "$file" "$file"
done
echo "Outside shell"
with shell=True
this_dir = os.path.dirname(os.path.abspath(__file__))
cmd = [os.path.join(this_dir,'split.sh'),fileslocation]
print 'cmd = ', cmd
process = subprocess.Popen(cmd,shell=True)
The parameters are not being passed correctly...
with Shell=True removed
cmd = ['/opt/sw/p3/src/PricesPaidAPI/split.sh', '../cookedData']
Traceback (most recent call last):
File "csv_rename.py", line 23, in <module>
process = subprocess.Popen(cmd)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
Ok, I dunno if this should be a dupe, or not (as per comments). But it is a fact that shebang helps. And here is the confirmation.
popen_test.py:
import subprocess
subprocess.Popen(["./dummy.sh", "test"])
noshebang.sh:
echo "Echoing: $1"
This results with OSError: [Errno 8] Exec format error. It is because OS expects the file to be executable (duh). However, a #! - shebang in a 1st line is a special mark for the system, that the file should be executed in the shell environment specified by it. So it can be treated as an executable, even though it is not. So:
shebang.sh:
#!/bin/sh
echo "Echoing: $1"
Works.
It is essentially the same as python's shell=True. Just the system takes care of it. IMO it is a bit harder to inject random code with this approach. So I would suggest going for it.
Related
I have been testing the stderr with the subprocess module. If I write a simple test with shell=True with the linux shell command ls intentionally bad typed:
p=subprocess.Popen(["lr"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
out, err=p.communicate()
print ("standard error")
print(err)
it outputs the usual from the shell: lr: command not found.
But if shell=False, I don't quite understand why the program has an error executing
Traceback (most recent call last):
File "importInteresantes.py", line 6, in <module>
p=subprocess.Popen(["lr"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I thought that it would give me the same output. Is the code wrong or is the point of view that I should obtain the same stderr?
NOTE: Just in case I also tried with python3
With shell=True, Python launches a shell and tells the shell to run lr. The shell runs just fine, fails to find an lr program, and produces error output reporting this failure.
With shell=False, Python tries to run lr directly. Since there is no lr program to run, Python can't find an executable file corresponding to lr. Python can't launch the subprocess at all, and there are no stdout or stderr streams to read from. Python raises an exception reporting its failure to find the file.
This behavior is normal and expected.
As the title says:
>>> from subprocess import check_output
>>> check_output(['ln', '~/other_folder/src/models/sc_models.py', './src/models/sc_models.py'])
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['ln', '~/other_folder/src/models/sc_models.py', './src/models/sc_models.py']' returned non-zero exit status 1
>>> exit()
$ ln ~/other_folder/src/models/sc_models.py ./src/models/sc_models.py
$
How can this be? How can it succeed from the command line, but fail from the Python subprocess call?
All tips are welcome!
You need to use os.path.expanduser:
On Unix and Windows, return the argument with an initial component of ~ or ~user replaced by that user‘s home directory.
import os
os.path.expanduser('~/other_folder/src/models/sc_models.py')
In [2]: os.path.expanduser("~")
Out[2]: '/home/padraic'
Python is looking for a directory named ~ in your cwd which obviously fails. When you run the code from bash the ~ is expanded, unless you were to use shell=True where the command would be passed to the shell and the shell would expand the tilde then you would need to use os.path.expanduser or pass the whole path i.e /home/user/other_folder...... I would stick to using shell=False withos.path.expanduser("~").
I am trying to create a script that will run my other python programs. I am new to subprocess module so this is a bit confusing to me.
project structure
/qe-functional
/qe
/tests
cron_functional.py
test_web_events.py
setup.sh
cron_functional.py
print(os.getcwd())
# print(subprocess.check_output('ls'))
runtag = "daily_run_" + datetime.today().strftime("%m_%d_%y")
testrun = "source ../../setup.sh; ./test_web_events.py -n 10 -t prf -E ctg-businessevent -p post {}".format(runtag)
cmd = testrun.split()
print(cmd)
subprocess.check_output(cmd)
output
$ python cron_functional.py
/Users/bli1/Development/QE/qe-functional/qe/tests
['source', '../../setup.sh;', './test_web_events.py', '-n', '10', '-t', 'prf', '-E', 'ctg-businessevent', '-p', 'post', 'daily_run_05_26_15']
Traceback (most recent call last):
File "cron_functional.py", line 11, in <module>
subprocess.check_output(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
source is an internal shell command, not an executable. What you want is not to run one source command with 11 arguments, but a one-liner shell script. You need to pass the whole script as one string to be interpreted by the shell.
subprocess.check_output(testrun, shell=True)
You haven't said what setup.sh does. If it's setting up environment variables and changing the working directory, consider doing that within Python instead. Then you can run
subprocess.check_output(['./test_web_events.py', '-n', '10', …, '-p', 'post', runtag])
… without involving the shell.
I want to delete bash history with a python script on my Macbook Pro.
I know two ways to delete bash history with bash shell
1.rm ~/.bash_history
2.history -c
But these command does not work in python script with subprocess:
1.rm ~/.bash_history
import subprocess
subprocess.call([‘rm’, ‘~/.bash_history'])
error:
rm: ~/.bash_history: No such file or directory
2.history -c
import subprocess
subprocess.call(['history', '-c'])
error:
File "test.py", line 8, in
subprocess.call(['history', '-c'])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line >524, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line >711, in init
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line >1308, in _execute_child
raise child_exception
Any ideas?
You have two questions here:
First, python doesn't understand ~, you need to expand it:
subprocess.call(['rm', os.path.expanduser('~/.bash_history')])
Second, history is a shell built-in. Use the shell to invoke it:
subprocess.call(['bash', '-c', 'history -c'])
I put one new alias python3.3 in my .bash_profile so as to launch easily the python3.3 version of pyzo.
I can use this alias in a terminal without any problem, but when I use something like subprocess.check_call(args = ["python3.3", onePyFile]), I have the following error.
Traceback (most recent call last):
...
File "/Library/Frameworks/pyzo2013b/lib/python3.3/subprocess.py", line 540, in check_call
retcode = call(*popenargs, **kwargs)
File "/Library/Frameworks/pyzo2013b/lib/python3.3/subprocess.py", line 521, in call
with Popen(*popenargs, **kwargs) as p:
File "/Library/Frameworks/pyzo2013b/lib/python3.3/subprocess.py", line 818, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/pyzo2013b/lib/python3.3/subprocess.py", line 1416, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'python3.3'
I guess that my alias is not seen everywhere. So how can I fix my problem ? What is the good way to build my own alias ?
If I try subprocess.check_call(args = ["python3.3", onePyFile], shell = True), I have the following error.
onePyFile.py: python3.3: command not found
Traceback (most recent call last):
File "mainFile.py", line 72, in <module>
shell = True
File "/Library/Frameworks/pyzo2013b/lib/python3.3/subprocess.py", line 545, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python3.3', 'onePyFile.py']' returned non-zero exit status 127
If i just use subprocess.check_call(args = ["python3.3", onePyFile]) where the first line of onePyFile is #! /usr/bin/env python3.3, I have the following error.
env: python3.3: No such file or directory
Traceback (most recent call last):
...
I think that my problem is more about the symbolic link than the Python call. But I do not know what is wrong. Indeed this is the first time that I make a personal symbolic link with an alias.
try subprocess.check_call(args = ["python3.3", onePyFile] , shell=True, env={'ENV':path_of_bash_profile})
If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory. However, note that Python itself offers implementations of many shell-like features (in particular, glob, fnmatch, os.walk(), os.path.expandvars(), os.path.expanduser(), and shutil).
That's because subprocess doesn't load a shell by default (see the docs), so it doesn't get what's in your .bash_profile.
Use this:
subprocess.check_call(args = ["python3.3", onePyFile], shell=True)
Edit: Seems like glasslion was faster than me!
Edit 2: I did a bit more digging around, and found something strange. Because shell=True didn't seem to work as expected I went with a more direct approach, calling bash directly.
a.py:
from subprocess import check_call
check_call(['bash', '-c', '. ~/.bash_profile && ABC a bc'])
a.sh (what would be your python3 executable):
echo $*
I first tried with alias ABC='~/Documents/a.sh' in .bash_profile:
$ python ~/Documents/a.py
bash: ABC: command not found
Traceback (most recent call last):
File "Documents/a.py", line 6, in <module>
check_call(['bash', '-c', '. ' + expanduser('~/.bash_profile') + ' && ABC a bc'])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 511, in check_call
subprocess.CalledProcessError: Command '['bash', '-c', '. ~/.bash_profile && ABC a bc']' returned non-zero exit status 127
Then I switched from an alias to a function: ABC() { ~/Documents/a.sh $*; }
And it worked!
$ python ~/Documents/a.py
a bc
Bottom line is I got it to work, but I don't know why! Shells aren't reliable, so the best would be to skip the shell.
We can do that by using the shebang's principle (as Fred Mitchell suggested) in a clever way:
from subprocess import check_call
check_call(['/usr/bin/env', 'python3', onePyFile])
This will work if Python 3 was installed correctly, independently of it's path (what I suppose you wanted to achieve).
Why not just put a shebang to python3.3 in the first line of onePyFile and omit running python explicitly?