We Keep Coding

Python program can't find Shellscript File

Hey i'm trying to run a shell Script with python using the Following lines:
import subprocess
shellscript = subprocess.Popen(["displaySoftware.sh"], stdin=subprocess.PIPE)
shellscript.stdin.write("yes\n")
shellscript.stdin.close()
returncode = shellscript.wait()
But when I run the Program it says that it can't find the .sh file.

Your command is missing "sh", you have to pass "shell=True" and "yes\n" has to be encoded.
Your sample code should look like this:
import subprocess
shellscript = subprocess.Popen(["sh displaySoftware.sh"], shell=True, stdin=subprocess.PIPE )
shellscript.stdin.write('yes\n'.encode("utf-8"))
shellscript.stdin.close()
returncode = shellscript.wait()
This method might be better:
import subprocess
shellscript = subprocess.Popen(["displaySoftware.sh"], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
returncode = shellscript.communicate(input='yes\n'.encode())[0]
print(returncode)
When running this on my machine the "displaySoftware.sh" script, that is in the same directory as the python script, is successfully executed.

python subprocess module hangs for spark-submit command when writing STDOUT

I have a python script that is used to submit spark jobs using the spark-submit tool. I want to execute the command and write the output both to STDOUT and a logfile in real time. i'm using python 2.7 on a ubuntu server.
This is what I have so far in my SubmitJob.py script
#!/usr/bin/python
# Submit the command
def submitJob(cmd, log_file):
with open(log_file, 'w') as fh:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
fh.write(output)
rc = process.poll()
return rc
if __name__ == "__main__":
cmdList = ["dse", "spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
log_file = "/tmp/out.log"
exist_status = submitJob(cmdList, log_file)
print "job finished with status ",exist_status
The strange thing is, when I execute the same command direcly in the shell it works fine and produces output on screen as the proggram proceeds.
So it looks like something is wrong in the way I'm using the subprocess.PIPE for stdout and writing the file.
What's the current recommended way to use subprocess module for writing to stdout and log file in real time line by line? I see bunch of options on the internet but not sure which is correct or latest.
thanks

Figured out what the problem was.
I was trying to redirect both stdout n stderr to pipe to display on screen. This seems to block the stdout when stderr is present. If I remove the stderr=stdout argument from Popen, it works fine. So for spark-submit it looks like you don't need to redirect stderr explicitly as it already does this implicitly

To print the Spark log
One can call the commandList given by user330612
cmdList = ["spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
Then it can be printed by using subprocess, remember to use communicate() to prevent deadlocks https://docs.python.org/2/library/subprocess.html
Warning Deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that. Here below is the code to print the log.
import subprocess
p = subprocess.Popen(cmdList,stdout=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
stderr=stderr.splitlines()
stdout=stdout.splitlines()
for line in stderr:
print line #now it can be printed line by line to a file or something else, for the log
for line in stdout:
print line #for the output
More information about subprocess and printing lines can be found at:
https://pymotw.com/2/subprocess/

Capture all output and error, warning of a command in windows by python

In bash shell of Linux, I can read a command (from file), then execute the command and write all the output, error, and return code to a file. Can I do that by using python in windows.

Of course you can. There are many ways to do this.
Assuming you had a text file named commands that contained a command on each line. You could do something like this:
open the input file
read the next command name from the file
execute the command using subprocess
redirect stderr to stdout
capture the combined output
if the command succeeded set return code to 0, otherwise capture the return code from the exception that is thrown.
write the return code and output to file
You will want to use:
https://docs.python.org/2/library/subprocess.html
or
https://docs.python.org/3/library/subprocess.html
for example:
import shlex
import subprocess
with open('commands.txt') as fin:
for command in fin:
try:
proc = subprocess.Popen(
shlex.split(command),
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE
)
returncode = 0
output = proc.communicate()[0]
except subprocess.CalledProcessError as e:
returncode = e.returncode
output = e.output
with open('output.txt', 'w') as fout:
fout.write('{}, {}'.format(returncode, output))

Linux, Write output to the file and terminate it using Python

There are various topics available on this very topic, "How to write output to the text file".
But my issue is different because the output to the command in question is continous.
What I want to do is, write the Output of the command cgps -s to the file aaa.txt
here is the code,
import signal
import os
import subprocess
p = subprocess.Popen(["cgps", "-s", ">> aaa.txt"], stdout=subprocess.PIPE,shell=True, preexec_fn=os.setsid)
os.killpg(p.pid, signal.SIGTERM)
The code doesn't work at all and no file is created with the name aaa.txt
When I execute this command through terminal,
cgps -s > aaa.txt
Then I have to press CTRL+C to terminate the output from being written on the output file because the output is continuous.
Is there any way to just capture one output and write it to the file and terminate it using Python or using Command line ?

So you are trying to create a pipe. Try this:
import subprocess
p = subprocess.Popen(["cgps", "-s"], stdout=subprocess.PIPE)
with open("aaa.txt", "w") as f:
while True:
line = p.stdout.readline()
if not line:
break
if some_condition(): # <-- check whether you have enough output
p.terminate()
break
f.writeline(line)
p.wait()

python subprocess.call output is not interleaved

I have a python (v3.3) script that runs other shell scripts. My python script also prints message like "About to run script X" and "Done running script X".
When I run my script I'm getting all the output of the shell scripts separate from my print statements. I see something like this:
All of script X's output
All of script Y's output
All of script Z's output
About to run script X
Done running script X
About to run script Y
Done running script Y
About to run script Z
Done running script Z
My code that runs the shell scripts looks like this:
print( "running command: " + cmnd )
ret_code = subprocess.call( cmnd, shell=True )
print( "done running command")
I wrote a basic test script and do *not* see this behaviour. This code does what I would expect:
print("calling")
ret_code = subprocess.call("/bin/ls -la", shell=True )
print("back")
Any idea on why the output is not interleaved?

Thanks. This works but has one limitation - you can't see any output until after the command completes. I found an answer from another question (here) that uses popen but also lets me see the output in real time. Here's what I ended up with this:
import subprocess
import sys
cmd = ['/media/sf_git/test-automation/src/SalesVision/mswm/shell_test.sh', '4', '2']
print('running command: "{0}"'.format(cmd)) # output the command.
# Here, we join the STDERR of the application with the STDOUT of the application.
process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
line = line.replace('\n', '')
print(line)
sys.stdout.flush()
process.wait() # Wait for the underlying process to complete.
errcode = process.returncode # Harvest its returncode, if needed.
print( 'Script ended with return code of: ' + str(errcode) )
This uses Popen and allows me to see the progress of the called script.

It has to do with STDOUT and STDERR buffering. You should be using subprocess.Popen to redirect STDOUT and STDERR from your child process into your application. Then, as needed, output them. Example:
import subprocess
cmd = ['ls', '-la']
print('running command: "{0}"'.format(cmd)) # output the command.
# Here, we join the STDERR of the application with the STDOUT of the application.
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process.wait() # Wait for the underlying process to complete.
out, err = process.communicate() # Capture what it outputted on STDOUT and STDERR
errcode = process.returncode # Harvest its returncode, if needed.
print(out)
print('done running command')
Additionally, I wouldn't use shell = True unless it's really required. It forces subprocess to fire up a whole shell environment just to run a command. It's usually better to inject directly into the env parameter of Popen.