We Keep Coding

Calling a shell script from python

I have a python script that calls a shell scrips, that in turn calls a .exe called iv4_console. I need to print the stdout of iv4_console for debugging purposes. I used this:
Python:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["c.sh", var], shell=True, stdout=subprocess.PIPE)
output = ''
for line in iter(proc.stdout.readline, ""):
print line
output += line
Shell:
start_dir=$PWD
release=$1
echo Release inside shell: $release
echo Directory: $start_dir
cd $start_dir
cd ../../iv_system4/ports/visualC12/Debug
echo Debug dir: $PWD
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.xml
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.xml
exit
But this didn't work, it prints nothing. What do you think?

See my comment, best approach (i.m.o) would be to just use python only.
However, in answer of your question, try:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["/bin/bash", "/full/path/to/c.sh"], stdout=subprocess.PIPE)
# Best to always avoid shell=True because of security vulnerabilities.
proc.wait() # To make sure the shell script does not continue running indefinitely in the background
output, errors = proc.communicate()
print(output.decode())
# Since subprocess.communicate() returns a bytes-string, you can use .decode() to print the actual output as a string.

You can use
import subprocess
subprocess.call(['./c.sh'])
to call the shell script in python file
or
import subprocess
import shlex
subprocess.call(shlex.split('./c.sh var'))

How can I run a os command without getting output to terminal?

When using subprocess or os libraries, executing the command returns the result in terminal. I want to be able to assign the output to a variable without getting any output returned to the terminal.
pid = subprocess.call(['pidof', process])
pid = os.system('pidof ' + process)
I only want to assign the variable pid, not return text to terminal. I was using the 'commands' library earlier, however it is not supported by python3.

You can try check_output.
import subprocess
output = subprocess.check_output("COMMAND_TO_EXECUTE", shell=True)

Have you tried redirect the command output to DEVNULL?
FNULL = open(os.devnull, 'w')
retcode = subprocess.call(['pidof', process], stdout=FNULL, stderr=subprocess.STDOUT)

Python test if client exist

I am new to python and working on trying to make a script which checks if a specified host as for example sensu-client exist. I use a deployment software called NSO and run it by: nso status and it shows me this information:
nagios-client host nagios-client down
test host test down
Is there any possibility to make a script to check if for example nagios-Client exist with a script ?
In shell I do it by:
nso status | awk '{ print $1 }'

In this case I would suggest using subprocess' check_output function. The documentation is here. check_output can return, as a string the shell output of a command. So you would have something like this:
import subprocess
foo=subprocess.check_output(['nso', 'status', '|', 'awk', '\'{ print $1 }\''], shell=True)
#Thanks bereal for shell=True
print foo
Of course, if your only targeting linux, you could use the much easier sh module. It allows you to import programs as if they were libraries.

you can use subprocess to run this command and parse the output
import subprocess
command = ['nso', 'status', '|', 'awk', '\'{ print $1 }\'']
p1 = subprocess.Popen(command, stdout=subprocess.PIPE)

You don't have to run awk, since you're already in Python:
import subprocess
proc = subprocess.Popen(['nso', 'status'], stdout=subprocess.PIPE)
# get stdout as a EOL-separated string, ignore stderr for now
out, _ = proc.communicate()
# parse the output, line.split()[0] is awk's $1
items = [line.split()[0] for line in out.split('\n')]

Executing an R script in python via subprocess.Popen

When I execute the script in R, it is:
$ R --vanilla --args test_matrix.csv < hierarchical_clustering.R > out.txt
In Python, it works if I use:
process = subprocess.call("R --vanilla --args "+output_filename+"_DM_Instances_R.csv < /home/kevin/AV-labels/Results/R/hierarchical_clustering.R > "+output_filename+"_out.txt", shell=True)
But this method doesn't provide the process.wait() function.
So, I would like to use the subprocess.Popen, I tried:
process = subprocess.Popen(['R', '--vanilla', '--args', "\'"+output_filename+"_DM_Instances_R.csv\'", '<', '/home/kevin/AV-labels/Results/R/hierarchical_clustering.R'])
But it didn't work, Python just opened R but didn't execute my script.

Instead of 'R', give it the path to Rscript. I had the same problem. Opens up R but doesn't execute my script. You need to call Rscript (instead of R) to actually execute the script.
retcode = subprocess.call("/Pathto/Rscript --vanilla /Pathto/test.R", shell=True)
This works for me.
Cheers!

I've solved this problem by putting everything into the brackets..
process = subprocess.Popen(["R --vanilla --args "+output_filename+"_DM_Instances_R.csv < /home/kevin/AV-labels/Results/R/hierarchical_clustering.R > "+output_filename+"_out.txt"], shell=True)
process.wait()

A couple of ideas:
You might want to consider using the Rscript frontend, which makes
running scripts easier; you can pass the script filename directly
as a parameter, and do not need to read the script in through standard input.
You don't need the shell for just redirecting standard output to a file, you can
do that directly with subprocess.Popen.
Example:
import subprocess
output_name = 'something'
script_filename = 'hierarchical_clustering.R'
param_filename = '%s_DM_Instances_R.csv' % output_name
result_filename = '%s_out.txt' % output_name
with open(result_filename, 'wb') as result:
process = subprocess.Popen(['Rscript', script_filename, param_filename],
stdout=result);
process.wait()

You never actually execute it fully ^^ try the following
process = subprocess.Popen(['R', '--vanilla', '--args', '\\%s_DM_Instances_R.csv\\' % output_filename, '<', '/home/kevin/AV-labels/Results/R/hierarchical_clustering.R'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, shell=True)
process.communicate()#[0] is stdout

Keven's solution works for my requirement. Just to give another example about #Kevin's solution. You can pass more parameters to the rscript with python-style string:
import subprocess
process = subprocess.Popen(["R --vanilla --args %s %d %.2f < /path/to/your/rscript/transformMatrixToSparseMatrix.R" % ("sparse", 11, 0.98) ], shell=True)
process.wait()
Also, to make things easier you could create an R executable file. For this you just need to add this in the first line of the script:
#! /usr/bin/Rscript --vanilla --default-packages=utils
Reference: Using R as a scripting language with Rscript or this link

What is the multiplatform alternative to subprocess.getstatusoutput (older commands.setstatusoutput() from Python?

The code below is outdated in Python 3.0 by being replaced by subprocess.getstatusoutput().
import commands
(ret, out) = commands.getstatusoutput('some command')
print ret
print out
The real question is what's the multiplatform alternative to this command from Python because the above code does fail ugly under Windows because getstatusoutput is supported only under Unix and Python does not tell you this, instead you get something like:
>test.py
1
'{' is not recognized as an internal or external command,
operable program or batch file.

This would be the multiplatform implementation for getstatusoutput():
def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
"""This new implementation should work on all platforms."""
import subprocess
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True, universal_newlines=True)
output = "".join(pipe.stdout.readlines())
sts = pipe.returncode
if sts is None: sts = 0
return sts, output

I wouldn't really consider this multiplatform, but you can use subprocess.Popen:
import subprocess
pipe = subprocess.Popen('dir', stdout=subprocess.PIPE, shell=True, universal_newlines=True)
output = pipe.stdout.readlines()
sts = pipe.wait()
print sts
print output
Here's a drop-in replacement for getstatusoutput:
def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
"""This new implementation should work on all platforms."""
import subprocess
pipe = subprocess.Popen(cmd, shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = str.join("", pipe.stdout.readlines())
sts = pipe.wait()
if sts is None:
sts = 0
return sts, output
This snippet was proposed by the original poster. I made some changes since getstatusoutput duplicates stderr onto stdout.
The problem is that dir isn't really a multiplatform call but subprocess.Popen allows you to execute shell commands on any platform. I would steer clear of using shell commands unless you absolutely need to. Investigate the contents of the os, os.path, and shutil packages instead.
import os
import os.path
for rel_name in os.listdir(os.curdir):
abs_name = os.path.join(os.curdir, rel_name)
if os.path.isdir(abs_name):
print('DIR: ' + rel_name)
elif os.path.isfile(abs_name):
print('FILE: ' + rel_name)
else:
print('UNK? ' + rel_name)

getstatusoutput docs say it runs the command like so:
{ cmd } 2>&1
Which obviously doesn't work with cmd.exe (the 2>&1 works fine if you need it though).
You can use Popen as above, but also include the parameter 'stderr=subprocess.STDOUT' to get the same behaviour as getstatusoutput.
My tests on Windows had returncode set to None though, which is not ideal if you're counting on the return value.