We Keep Coding

run cmake command via subprocess.Popen or subprocess.run [duplicate]

If I run echo a; echo b in bash the result will be that both commands are run. However if I use subprocess then the first command is run, printing out the whole of the rest of the line.
The code below echos a; echo b instead of a b, how do I get it to run both commands?
import subprocess, shlex
def subprocess_cmd(command):
process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
proc_stdout = process.communicate()[0].strip()
print proc_stdout
subprocess_cmd("echo a; echo b")

You have to use shell=True in subprocess and no shlex.split:
import subprocess
command = "echo a; echo b"
ret = subprocess.run(command, capture_output=True, shell=True)
# before Python 3.7:
# ret = subprocess.run(command, stdout=subprocess.PIPE, shell=True)
print(ret.stdout.decode())
returns:
a
b

I just stumbled on a situation where I needed to run a bunch of lines of bash code (not separated with semicolons) from within python. In this scenario the proposed solutions do not help. One approach would be to save a file and then run it with Popen, but it wasn't possible in my situation.
What I ended up doing is something like:
commands = '''
echo "a"
echo "b"
echo "c"
echo "d"
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands)
print out
So I first create the child bash process and after I tell it what to execute. This approach removes the limitations of passing the command directly to the Popen constructor.

Join commands with "&&".
os.system('echo a > outputa.txt && echo b > outputb.txt')

If you're only running the commands in one shot then you can just use subprocess.check_output convenience function:
def subprocess_cmd(command):
output = subprocess.check_output(command, shell=True)
print output

>>> command = "echo a; echo b"
>>> shlex.split(command);
['echo', 'a; echo', 'b']
so, the problem is shlex module do not handle ";"

Got errors like when I used capture_output=True
TypeError: __init__() got an unexpected keyword argument 'capture_output'
After made changes like as below and its works fine
import subprocess
command = '''ls'''
result = subprocess.run(command, stdout=subprocess.PIPE,shell=True)
print(result.stdout.splitlines())

import subprocess
cmd = "vsish -e ls /vmkModules/lsom/disks/ | cut -d '/' -f 1 | while read diskID ; do echo $diskID; vsish -e cat /vmkModules/lsom/disks/$diskID/virstoStats | grep -iE 'Delete pending |trims currently queued' ; echo '====================' ;done ;"
def subprocess_cmd(command):
process = subprocess.Popen(command,stdout=subprocess.PIPE, shell=True)
proc_stdout = process.communicate()[0].strip()
for line in proc_stdout.decode().split('\n'):
print (line)
subprocess_cmd(cmd)

subprocess.call to run mafft

I wrote a script to run mafft module from the terminal:
import subprocess
def linsi_MSA(sequnces_file_path):
cmd = ' mafft --maxiterate 1000 --localpair {seqs} > {out}'.format(seqs=sequnces_file_path, out=sequnces_file_path)
subprocess.call(cmd.split(), shell=True)
if __name__ == '__main__':
import logging
logger = logging.getLogger('main')
from sys import argv
if len(argv) < 2:
logger.error('Usage: MSA <sequnces_file_path> ')
exit()
else:
linsi_MSA(*argv[1:])
for some reason when trying to run the script from the terminal using:
python ./MSA.py ./sample.fa
I get the mafft interactive version opening directly in the trminal (asking for input ..output etc..)
when i'm trying to write the cmd directly in the terminal using:
mafft --maxiterate 1000 --localpair sample.fa > sample.fa
its working as expected and perfoming the command line version as without opening the interactive version.
I want my script to be able to perform the cmd line version on the terminal. what seems to be the problem?
thanks!

If you use shell=True you should pass one string as argument, not a list, e.g.:
subprocess.call("ls > outfile", shell=True)
It's not explained in the docs, but I suspect it has to do with what low-level library function is ultimately called:
call(["ls", "-l"]) --> execlp("ls", "-l")
^^^^^^^^^^ ^^^^^^^^^^
call("ls -l", shell=True) --> execlp("sh", "-c", "ls -l")
^^^^^^^ ^^^^^^^
call(["ls", "-l"], shell=True) --> execlp("sh", "-c", "ls", "-l")
# which can be tried from command line:
sh -c ls -l
# result is a list of files without details, -l was ignored.
# see sh(1) man page for -c string syntax and what happens to further arguments.

How to call a python script on a new shell window from python code?

I'm trying to execute 10 python scripts from python code and open each of them in a new shell window.
My code :
for i in range(10):
name_of_file = "myscript"+str(i)+".py"
cmd = "python " + name_of_file
os.system("gnome-terminal -e 'bash -c " + cmd + "'")
But each script file are not executing, I get only the live interpreter of python in the new terminal...
Thank you guys

I would suggest using the subprocess module (https://docs.python.org/2/library/subprocess.html).
In this way, you'll write something like the following:
import subprocess
cmd = ['gnome-terminal', '-x', 'bash', '-c']
for i in range(10):
name_of_file = "myscript"+str(i)+".py"
your_proc = subprocess.Popen(cmd + ['python %s' % (name_of_file)])
# or if you want to use the "modern" way of formatting string you can write
# your_proc = subprocess.Popen(cmd + ['python {}'.format(name_of_file)])
...
and you have more control over the processes you start.
If you want to keep using os.system(), build your command string first, then pass it to the function. In your case would be:
cmd = 'gnome-terminal -x bash -c "python {}"'.format(name_of_file)
os.system(cmd)
something along these lines.
Thanks to #anishsane for some suggestions!

I think that it is to do with the string quoting of the argument to os.system. Try this:
os.system("""gnome-terminal -e 'bash -c "{}"'""".format(cmd))

How to redirect command output using os.execvp() in python

I am invoking shell script using os.execvp() in python. my shell script has some echo statements whcih I want to redirect in file.
Here is what I am trying:
cmd = "/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
cmdline = ["/bin/sh", cmd]
os.execvp(cmdline[0], cmdline)
Below is the error I am getting:
Error: /bin/sh: /opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log: No such file or directory
Can any one help?

This is happening because you are passing this entire string as if it were the program name to execute:
"/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
The easy way to fix this is:
cmdline = ["/bin/sh", "/opt/rpm/rpm_upgrade.sh",
">&", "/opt/rpm/upgrader.log"]
os.execvp(cmdline[0], cmdline)
Now sh will receive three arguments rather than one.
Or you can switch to the more full-featured subprocess module, which lets you redirect output in Python:
import subprocess
with open("/opt/rpm/upgrader.log", "wb") as outfile:
subprocess.check_call(["/opt/rpm/rpm_upgrade.sh"], shell=True,
stdout=outfile, stderr=subprocess.STDOUT)

When I run a python code, it says “<< was unexpected at this time"

Basically, it's a python code from our collaborator that used to generate mesh, which is developed under Linux environment. I use Cygwin to run this code on windows. The trouble part is as follows. BiV_temp.geo is also a python script. So the command is to substitute the string <> in the script BiV_temp.geo with a predefined number and the file names.
os.system('cp BiV_fiber.geo BiV_temp.geo')
cmd = "sed -i 's/<<Meshsize>>/"+"%5.2f"%(meshsize)+"/g' BiV_temp.geo"
os.system(cmd)
cmd = "sed -i 's/<<LVfilename>>/"+"\"%s\""%(LVendocutfilename)+"/g' BiV_temp.geo"
os.system(cmd)
cmd = "sed -i 's/<<RVfilename>>/"+"\"%s\""%(RVendocutfilename)+"/g' BiV_temp.geo"
os.system(cmd)
cmd = "sed -i 's/<<Epifilename>>/"+"\"%s\""%(epicutfilename)+"/g' BiV_temp.geo"
os.system(cmd)
cmd = "gmsh -3 BiV_temp.geo -o %s"%(mshfilename)
os.system(cmd)
cmd = "rm BiV_temp.geo"
os.system(cmd)

The sane solution is for your "collaborator" to write Python code which allows you to pass in these things as parameters to a function call.

you are trying to executing command code in python?
and i notice in "%5.2f"%(meshsize) you wrote %5, for adding text you should add %sand also avoid putting plus sign, it make it really hard to read and messy, i would write all in one line like this:
meshsize = 100
cmd = "sed -i 's/<<Meshsize>>/%s5.2f/g' BiV_temp.geo" %meshsize
print cmd
if your meshsize is a list of x,y,z then write it like this:
meshsize = (100,50,20)
cmd = "sed -i 's/<<Meshsize>>/%s,%s,%s5.2f/g' BiV_temp.geo" %meshsize
print cmd
hope that help.