There are various topics available on this very topic, "How to write output to the text file".
But my issue is different because the output to the command in question is continous.
What I want to do is, write the Output of the command cgps -s to the file aaa.txt
here is the code,
import signal
import os
import subprocess
p = subprocess.Popen(["cgps", "-s", ">> aaa.txt"], stdout=subprocess.PIPE,shell=True, preexec_fn=os.setsid)
os.killpg(p.pid, signal.SIGTERM)
The code doesn't work at all and no file is created with the name aaa.txt
When I execute this command through terminal,
cgps -s > aaa.txt
Then I have to press CTRL+C to terminate the output from being written on the output file because the output is continuous.
Is there any way to just capture one output and write it to the file and terminate it using Python or using Command line ?
So you are trying to create a pipe. Try this:
import subprocess
p = subprocess.Popen(["cgps", "-s"], stdout=subprocess.PIPE)
with open("aaa.txt", "w") as f:
while True:
line = p.stdout.readline()
if not line:
break
if some_condition(): # <-- check whether you have enough output
p.terminate()
break
f.writeline(line)
p.wait()
Related
I am writing a python script for automation.
I need to run a linux shell command (Program: dvbv5-zap) and wait for specific command output (DVR interface '/dev/dvb/adapter0/dvr0' can now be opened). When command outputs this string python should run another shell program.
I don't know how to capture subprocess cli output, I tried with .stdout.readline(), and I got nothing.
I run a command with subprocess.Popen(['dvbv5-zap', 'args'], stdout=subprocess.PIPE)
I found my answer here: https://fredrikaverpil.github.io/2013/10/11/catching-string-from-stdout-with-python/
Code snippet:
# Imports
import os, sys, subprocess
# Build command
command = [ 'python', os.join.path('/path/to', 'scriptFile.py') ]
# Execute command
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# Read stdout and print each new line
sys.stdout.flush()
for line in iter(p.stdout.readline, b''):
# Print line
sys.stdout.flush()
print(">>> " + line.rstrip())
# Look for the string 'Render done' in the stdout output
if 'Render done' in line.rstrip():
# Write something to stdout
sys.stdout.write('Nice job on completing the render, I am executing myFunction()\n' )
sys.stdout.flush()
# Execute something
myFunction()
I'm trying to display realtime data using subprocess module, But it only display first line only.
I tried to follow some of the tutorials. which are given here
read subprocess stdout line by line
Code i tried
import subprocess
proc = subprocess.Popen(
['aria2c', 'http://cdimage.deepin.com/releases/20Beta/deepin-20Beta-desktop-amd64.iso'], stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line:
break
# the real code does filtering here
print("test:", line.rstrip())
Actual Output
Display only one line
Expected Output
display all the informations
I am trying to run an external application using Popen and print the output in the console or separated console (better) and at the same time save the output to the file. There is no user interaction via console, app.bat just sends (writes) the data and should terminate automatically when the execution is finished.
Running the following command will result only in printing the results in the python console.
p = subprocess.Popen("app.bat --x --y", shell=False)
If I add stdout as file I can redirect the output to the file, but nothing is written in the console, which does not give users any feedback (and the feedback needs to be in real-time, not after the execution because app runs approximately 1-3min).
file_ = open("ouput.txt", "w+")
p = subprocess.Popen("app.bat --x --y", shell=False,stdout=file_)
Therefore, my question is how to run the external app and at the same time write in the console and in the file?
For what you want to do I'd encourage you to use the logging module.
A good starter here is https://docs.python.org/2/howto/logging-cookbook.html
It even describes your usecase almost exactly.
If you want to post-process the output of your Popen() call, you should typically redirect stdout to PIPE and then read the output from there. This will allow you to e.g. both write to file and to screen:
import subprocess
logfile ='output.txt'
command = ['app.bat', '--x', '--y']
p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
with open(logfile, 'w+') as f:
for line in p.stdout:
print(line.rstrip())
f.write(line)
Now, this will block until app.bat finishes, which may be exactly what you want. But, if you want your Python script to continue to run, and have app.bat run in te background, you can start a thread that will handle your subprocess stdout:
import subprocess
import threading
logfile ='output.txt'
command = ['app.bat', '--x', '--y']
def writer(p, logfile):
with open(logfile, 'w+') as f:
for line in p.stdout:
print(line.rstrip())
f.write(line)
p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
t = threading.Thread(target=writer, args=(p,logfile))
t.start()
# Other commands while app.bat runs
t.join()
I have a python script that is used to submit spark jobs using the spark-submit tool. I want to execute the command and write the output both to STDOUT and a logfile in real time. i'm using python 2.7 on a ubuntu server.
This is what I have so far in my SubmitJob.py script
#!/usr/bin/python
# Submit the command
def submitJob(cmd, log_file):
with open(log_file, 'w') as fh:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
fh.write(output)
rc = process.poll()
return rc
if __name__ == "__main__":
cmdList = ["dse", "spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
log_file = "/tmp/out.log"
exist_status = submitJob(cmdList, log_file)
print "job finished with status ",exist_status
The strange thing is, when I execute the same command direcly in the shell it works fine and produces output on screen as the proggram proceeds.
So it looks like something is wrong in the way I'm using the subprocess.PIPE for stdout and writing the file.
What's the current recommended way to use subprocess module for writing to stdout and log file in real time line by line? I see bunch of options on the internet but not sure which is correct or latest.
thanks
Figured out what the problem was.
I was trying to redirect both stdout n stderr to pipe to display on screen. This seems to block the stdout when stderr is present. If I remove the stderr=stdout argument from Popen, it works fine. So for spark-submit it looks like you don't need to redirect stderr explicitly as it already does this implicitly
To print the Spark log
One can call the commandList given by user330612
cmdList = ["spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
Then it can be printed by using subprocess, remember to use communicate() to prevent deadlocks https://docs.python.org/2/library/subprocess.html
Warning Deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that. Here below is the code to print the log.
import subprocess
p = subprocess.Popen(cmdList,stdout=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
stderr=stderr.splitlines()
stdout=stdout.splitlines()
for line in stderr:
print line #now it can be printed line by line to a file or something else, for the log
for line in stdout:
print line #for the output
More information about subprocess and printing lines can be found at:
https://pymotw.com/2/subprocess/
In bash shell of Linux, I can read a command (from file), then execute the command and write all the output, error, and return code to a file. Can I do that by using python in windows.
Of course you can. There are many ways to do this.
Assuming you had a text file named commands that contained a command on each line. You could do something like this:
open the input file
read the next command name from the file
execute the command using subprocess
redirect stderr to stdout
capture the combined output
if the command succeeded set return code to 0, otherwise capture the return code from the exception that is thrown.
write the return code and output to file
You will want to use:
https://docs.python.org/2/library/subprocess.html
or
https://docs.python.org/3/library/subprocess.html
for example:
import shlex
import subprocess
with open('commands.txt') as fin:
for command in fin:
try:
proc = subprocess.Popen(
shlex.split(command),
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE
)
returncode = 0
output = proc.communicate()[0]
except subprocess.CalledProcessError as e:
returncode = e.returncode
output = e.output
with open('output.txt', 'w') as fout:
fout.write('{}, {}'.format(returncode, output))