I am trying to run 2 bash commands sequentially in a for loop in Python. The loop is as given below:
for dataset in data:
subprocess.call("cd " +dataset+'.matrix',shell=True)
subprocess.call("cat part-r-* > combined_output", shell=True)
However, in this way, each command is being considered independently of each other. I need them to execute one after the other. I don't know how to use the subprocess module well (I also tried with os.system). I went through some documentation online but they weren't really useful. Any help in this would be appreciated. thanks in advance!
I don't believe they're running asynchronously (at the same time), just that they're in different subprocesses which each inherit the parent process' working directory. So it is waiting for the cd to complete, but the subprocess which has cd'd then disappears.
Thankfully, in this case you can tell the call to operate in a different directory with the cwd parameter, so you don't need your cd:
for dataset in data:
subprocess.call("cat part-r-* > combined_output",shell=True,cwd=dataset+'.matrix')
An example with os.system:
import os
os.system("echo aaa > ~/SOMEFILE1111; cd ~/; cat SOMEFILE1111; rm SOMEFILE1111")
and one with subprocess.call:
import subprocess
cmds = [ "echo aaa > ~/SOMEFILE1111", "cd ~/",
"cat SOMEFILE1111", "rm SOMEFILE1111"]
subprocess.call(";".join(cmds), shell=True)
Related
I'm using windows 10 and while working on a new project, I need to interact with WSL(Ubuntu on windows) bash from within python (windows python interpreter).
I tried using subprocess python library to execute commands.. what I did looks like this:
import subprocess
print(subprocess.check_call(['cmd','ubuntu1804', 'BashCmdHere(eg: ls)']))#not working
print(subprocess.check_output("ubuntu1804", shell=True).decode())#also not working
The expected behavior is to execute ubuntu1804 command which starts a wsl linux bash on which I want to execute my 'BashCmdHere' and retrieve its results to python but it just freezes. What am I doing wrong ? or how to do this ?
Thank you so much
Found 2 ways to achieve this:
A correct version of my code looks like this
#e.g: To execute "ls -l"
import subprocess
print(subprocess.check_call(['wsl', 'ls','-l','MaybeOtherParamHere']))
I should have used wsl to invoke linux shell from windows aka bash then my command and parameters in separated arguments for the subprocess command.
The other way which I think is cleaner but may be heavier is using PowerShell Scripts:
#script.ps1
param([String]$folderpath, [String]$otherparam)
Write-Output $folderpath
Write-Output $otherparam
wsl ls -l $folderpath $otherparam
Then to execute it in python and get the results:
import subprocess
def callps1():
powerShellPath = r'C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe'
powerShellCmd = "./script.ps1"
#call script with argument '/mnt/c/Users/aaa/'
p = subprocess.Popen([powerShellPath, '-ExecutionPolicy', 'Unrestricted', powerShellCmd, '/mnt/c/Users/aaa/', 'SecondArgValue']
, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = p.communicate()
rc = p.returncode
print("Return code given to Python script is: " + str(rc))
print("\n\nstdout:\n\n" + str(output))
print("\n\nstderr: " + str(error))
# Test
callps1()
Thank you for helping out
What about:
print(subprocess.check_call(['ubuntu1804', 'run', 'BashCmdHere(eg: ls)'])) #also try without "run" or change ubuntu1804 to wsl
Or
print(subprocess.check_call(['cmd', '/c', 'ubuntu1804', 'run', 'BashCmdHere(eg: ls)']))#also try without "run" or change "ubuntu1804" to "wsl"
# I think you need to play with quotes here to produce: cmd /c 'ubuntu1804 run BashCmdHere(eg: ls)'
First, try to call your command from cmd.exe to see the right format and then translate it to Python.
os.system('bash')
I figured this out by accident.
I have a batch file, which I use to load some pre-build binaries to control my device.
It's command is:
cd build
java -classpath .;..\Library\mfz-rxtx-2.2-20081207-win-x86\RXTXcomm.jar -
Djava.library.path=..\Library\mfz-rxtx-2.2-20081207-win-x86 tabotSample/Good1
pause
Now, I want to run the batch file using Python, and I tried os.system(batch,bat), and I tried using Popen
import os
from subprocess import Popen
os.popen("cd TAbot")
r=os.popen("hello.bat")
However, the python console(Anaconda python 2.7) seems like executed the code, but returns nothing, and nothing happens.
I want to run this batch file from python, please help me.
by the way, I tried popen for another batch file like,
echo Hello but nothing happens.
Here is the simple solution.
from subprocess import Popen
import subprocess
def run_batch_file(file_path):
Popen(file_path,creationflags=subprocess.CREATE_NEW_CONSOLE)
run_batch_file('file_name.bat')
file_name.bat
echo .bat file running from python
pause
You can also use this
import subprocess
subprocess.call(["C:\\temp\\test.bat"], shell=False)
test.bat
copy "C:\temp\test.txt" "C:\temp\test2.txt"
I think this should work like this:
batch.py
from subprocess import Popen
p = Popen("test.bat", cwd=r"C:\path\to\batch\folder")
stdout, stderr = p.communicate()
test.bat
echo Hello World!
pause
Here many guys suggested very useful solutions, but I want to point the importance of where is the program located.
(Bat file is usually made for automation task to reduce time and this has high probability to work some task related path)
import subprocess
os.chdir("YOUR TARGET PATH")
exit_code = subprocess.call(FILEPATH)# FILEPATH is from the standpoint on YOUR TARGET PATH
I am invoking the bash script from python script.
I want the bash script to add an element to dictionary "d" in the python script
abc3.sh:
#!/bin/bash
rank=1
echo "plugin"
function reg()
{
if [ "$1" == "what" ]; then
python -c 'from framework import data;data(rank)'
echo "iamin"
else
plugin
fi
}
plugin()
{
echo "i am plugin one"
}
reg $1
python file:
import sys,os,subprocess
from collections import *
subprocess.call(["./abc3.sh what"],shell=True,executable='/bin/bash')
def data(rank,check):
d[rank]["CHECK"]=check
print d[1]["CHECK"]
If I understand correctly, you have a python script that runs a shell script, that in turn runs a new python script. And you'd want the second Python script to update a dictionnary in the first script. That will not work like that.
When you run your first python script, it will create a new python process, which will interpret each instruction from your source script.
When it reaches the instruction subprocess.call(["./abc3.sh what"],shell=True,executable='/bin/bash'), it will spawn a new shell (bash) process which will in turn interpret your shell script.
When the shell script reaches python -c <commands>, it invokes a new python process. This process is independant from the initial python process (even if you run the same script file).
Because each of theses scripts will run in a different process, they don't have access to each other data (the OS makes sure that each process is independant from each other, excepted for specific inter-process communications methods).
What you need to do: use some kind of interprocess mechanism, so that the initial python script gets data from the shell script. You may for example read data from the shell standard output, using https://docs.python.org/3/library/subprocess.html#subprocess.check_output
Let's suppose that you have a shell plugin that echoes the value:
echo $1 12
The mockup python script looks like (I'm on windows/MSYS2 BTW, hence the strange paths for a Linux user):
import subprocess
p = subprocess.Popen(args=[r'C:\msys64\usr\bin\sh.exe',"-c","C:/users/jotd/myplugin.sh myarg"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
o,e= p.communicate()
p.wait()
if len(e):
print("Warning: error found: "+e.decode())
result = o.strip()
d=dict()
d["TEST"] = result
print(d)
it prints the dictionary, proving that argument has been passed to the shell, and went back processed.
Note that stderr has been filtered out to avoid been mixed up with the results, but is printed to the console if occurs.
{'TEST': b'myarg 12'}
I am writing a python tool that makes some processing, data gathering and eventually calling a bash scripts via subprocess.
The tricky part of my tool is: I use nuitka to compile python files to the single binary file. I'm doing this because I don't want my users to add any features plus I'd like my tool to be mysterious.
The problem is of course, what to do with that bash scripts. For now, I store them together with my nuitka-ed binary. However, due to above, I don't want a bash scripts to be that easily accessible.
For now, I can see two options for me:
I can compile *.sh files and link them with nuitka-ed binary.
I can hardcode *.sh files in *.py files.
Option 1 is rather not an option due to its complexity. Option 2 is slightly better, but for now, my best solutions is following:
I make script.py:
"""
#!/bin/bash
ls -alh /var
echo $?
# (...)
"""
And I inject script.__doc__ to the subprocess.Popen(['bash(...).
Is there any better - more elegant way to achieve my goal while maintaining bash script readability ?
To hardcode bash-code within a python file you could do it this way.
import subprocess
import StringIO
# In Python
text = "foobar"
your_bash_cmd = "echo "+text
process = subprocess.Popen([your_bash_cmd], shell=True, stdout=subprocess.PIPE)
process.wait()
your_bash_output = process.stdout.read()
# Process bash output in python
print(your_bash_output)
#!/usr/bin/env bash
I am working on using bash commands in python I found this example in Red hat magazine the tail command does not produce any output. I would like to understand why it is failing and I have also tried to import subprocess but that just hangs.
#Create Commands**strong text**
SPACE=`df -h`
MESSAGES=`tail /var/log/messages`
#Assign to an array(list in Python)
cmds=("$MESSAGES" "$SPACE")
#iteration loop
count=0
for cmd in "${cmds[#]}"; do
count=$((count + 1))
printf "Running Command Number %s \n" $count
echo "$cmd"
done
Printing the command doesn't mean executing it. Look at Python's subprocess library for API and examples for doing what you want.