We Keep Coding

Subprocess run works from Python, but not when using R and Reticulate

I have a command saved in a text file that I wish to execute using subprocess.run() from R using the reticulate package.
I have a directory with three files:
test_command.txt which contains the command touch foo.txt
run_command.py:
import subprocess
import os
subprocess.check_output('bash test_command.txt')
print(os.path.isfile("foo.txt")) # Check if the command was actually executed properly
run_from_r.R:
library(reticulate)
use_condaenv("my_env") # Same conda environment as used for python
source_python("run_command.py")
When I run run_command.py directly, foo.txt is created, and True is returned.
However, when I run from R using run_from_r.R, I get the following message:
Error in py_run_file_impl(file, local, convert) :
OSError: [WinError 6] The handle is invalid
Detailed traceback:
File "<string>", line 5, in <module>
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 411, in check_output
**kwargs).stdout
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 753, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Users\Danie\miniconda3\envs\wildcats_summer_env\lib\subprocess.py", line 1054, in _get_handles
p2cread = _winapi.GetStdHandle(_winapi.STD_INPUT_HANDLE)
system("bash test_command.txt") runs properly in R.
Any idea what this error message means, and how I can make the command run properly when running using subprocess.check_output/run and reticulate?
Thanks!

I seem to have fixed it by using:
subprocess.run(['bash', 'test_command.txt'], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL)
From Python running as Windows Service: OSError: [WinError 6] The handle is invalid

How to run bash commands using subprocess.run on windows [duplicate]

This question already has answers here:
Python subprocess.run('ls',shell=True) not working on windows
(2 answers)
Closed 1 year ago.
I want to run shell scripts and git-bash commands using subprocess.run(), in python 3.7.4. When I run the simple example on the subprocess documentation page this happens:
import subprocess
subprocess.run(["ls", "-l"])
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\pycharm\project\envs\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\pycharm\project\envs\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\pycharm\project\envs\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
# it also fails with shell=True
subprocess.call(["ls", "-l"], shell=True)
'ls' is not recognized as an internal or external command,
operable program or batch file.
1
The message from shell=True is a message from windows cmd, which suggests subprocess is not sending commands to git-bash.
I am using a conda environment located in the project/envs/ folder for python. I have also installed git-bash.
I also tried setting the env and got the same error.
import os
import subprocess
my_env = os.environ.copy()
my_env["PATH"] = 'C:\Program Files\Git\;' + my_env["PATH"]
subprocess.run(['git-bash.exe', 'ls', '-l'], env=my_env)
Traceback (most recent call last):
File "<input>", line 3, in <module>
File "C:\pycharm\project\envs\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\pycharm\project\envs\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:n\pycharm\project\envs\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I can get it to run by pointing at the git-bash.exe, but it returns an empty string instead of the files in my directory
import subprocess
subprocess.run(['C:\Program Files\Git\git-bash.exe', 'ls', '-l'], capture_output=True)
CompletedProcess(args=['C:\\Program Files\\Git\\git-bash.exe', 'ls', '-l'], returncode=0, stdout=b'', stderr=b'')
I would appreciate any advice on the best way to get this working as shown on the subprocess documentation page.

I found that I can run commands using ...Git\bin\bash.exe instead of the ...\Git\git-bash.exe, like this:
import subprocess
subprocess.run(['C:\Program Files\Git\\bin\\bash.exe', '-c','ls'], stdout=subprocess.PIPE)
CompletedProcess(args=['C:\\Program Files\\Git\\bin\\bash.exe', '-c', 'ls'], returncode=0, stdout=b'README.md\n__pycache__\nconda_create.sh\nenvs\nmain.py\ntest.sh\nzipped\n')

Try this
p = subprocess.Popen(("ls", "-l"), stdout=subprocess.PIPE)
nodes = subprocess.check_output(("grep"), stdin=p.stdout)
p.wait()

ls is Linux shell command for listing files and directories
dir is Windows command line command for listing files and directories
Try to run dir in Windows command line. If it works, try to run the same command using python subprocess:
import subprocess
subprocess.run(["dir"])

For a machine with Windows Operating System, Try the following
import subprocess
subprocess.run(["dir", "/p"], shell=True)
"ls" is replaced with "dir", "-l" is replaced with "/l" and the "shell" is set to true
For a machine with Linux/Mac Operating System, Try the following
import subprocess
subprocess.run(["ls", "-l"])

Python script to run an exe file with multiple parameters

I am a beginner in Python. I wanted to run an exe file with a number of parameters to be passed. The some parameters are path to files and some are just strings. The path for the exe also has spaces in it. I could run it via command prompt as below.
C:\Program Files (x86)\XXX 8.0\bin\xxx.exe -I -c "E:\files" -m ASCII -lib "" -i "E:\Trialtest\input.txt" -t "E:\test\output.txt" -s "E:\Trialtest\test\output.struct"
I tried a lot of posts nothing worked. I found one post which is similar to my query. But didnt work for me. Please help me run this using Python.
The code i tried is
subprocess.check_output(["C:\Program Files (x86)\xxx_x\yyy 8.0\bin\abc.exe", "-I", "-c", "E:\Trialtest.gtp", "-m", "ASCII ", "-lib", "", "-i", "E:\Trialtest\input.txt", "-t", "E:\Trialtest\test\output.txt", "-s", "E:\Trialtest\test\output.struct"])
the error is
Traceback (most recent call last):
File "<pyshell#73>", line 1, in <module>
subprocess.check_output(["C:\Program Files (x86)\xxx_x\yyy 8.0\bin\abc.exe", "-I", "-c", "E:\Trialtest.gtp", "-m", "ASCII ", "-lib", "", "-i", "E:\Trialtest\input.txt", "-t", "E:\Trialtest\test\output.txt", "-s", "E:\Trialtest\test\output.struct"])
File "C:\Python27\lib\subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "C:\Python27\lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Thanks.

All you have to do is to use subprocess
subprocess.check_output(["C:\Program Files (x86)\XXX 8.0\bin\xxx.exe", "-I", "-c", "E:\files", "-m", "ASCII", "-lib","" ,"-i", "E:\Trialtest\input.txt" ,"-t" ,"E:\test\output.txt", "-s", "E:\Trialtest\test\output.struct"])

Python subprocess Popen passing arguments

Im trying to call a shell script from my python with a parameter but the parameter is not being passed
My Shellscript:
echo "Inside shell"
echo $0
echo $1
cd $1
pwd
for file in *.csv
do
split -l 50000 -d -a 4 "$file" "$file"
done
echo "Outside shell"
with shell=True
this_dir = os.path.dirname(os.path.abspath(__file__))
cmd = [os.path.join(this_dir,'split.sh'),fileslocation]
print 'cmd = ', cmd
process = subprocess.Popen(cmd,shell=True)
The parameters are not being passed correctly...
with Shell=True removed
cmd = ['/opt/sw/p3/src/PricesPaidAPI/split.sh', '../cookedData']
Traceback (most recent call last):
File "csv_rename.py", line 23, in <module>
process = subprocess.Popen(cmd)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error

Ok, I dunno if this should be a dupe, or not (as per comments). But it is a fact that shebang helps. And here is the confirmation.
popen_test.py:
import subprocess
subprocess.Popen(["./dummy.sh", "test"])
noshebang.sh:
echo "Echoing: $1"
This results with OSError: [Errno 8] Exec format error. It is because OS expects the file to be executable (duh). However, a #! - shebang in a 1st line is a special mark for the system, that the file should be executed in the shell environment specified by it. So it can be treated as an executable, even though it is not. So:
shebang.sh:
#!/bin/sh
echo "Echoing: $1"
Works.
It is essentially the same as python's shell=True. Just the system takes care of it. IMO it is a bit harder to inject random code with this approach. So I would suggest going for it.

How to delete bash history in python script on Mac?

I want to delete bash history with a python script on my Macbook Pro.
I know two ways to delete bash history with bash shell
1.rm ~/.bash_history
2.history -c
But these command does not work in python script with subprocess:
1.rm ~/.bash_history
import subprocess
subprocess.call([‘rm’, ‘~/.bash_history'])
error:
rm: ~/.bash_history: No such file or directory
2.history -c
import subprocess
subprocess.call(['history', '-c'])
error:
File "test.py", line 8, in
subprocess.call(['history', '-c'])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line >524, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line >711, in init
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line >1308, in _execute_child
raise child_exception
Any ideas?

You have two questions here:
First, python doesn't understand ~, you need to expand it:
subprocess.call(['rm', os.path.expanduser('~/.bash_history')])
Second, history is a shell built-in. Use the shell to invoke it:
subprocess.call(['bash', '-c', 'history -c'])