I am using Popen to call a shell script that is continuously writing its stdout and stderr to a log file. Is there any way to simultaneously output the log file continuously (to the screen), or alternatively, make the shell script write to both the log file and stdout at the same time?
I basically want to do something like this in Python:
cat file 2>&1 | tee -a logfile #"cat file" will be replaced with some script
Again, this pipes stderr/stdout together to tee, which writes it both to stdout and my logfile.
I know how to write stdout and stderr to a logfile in Python. Where I'm stuck is how to duplicate these back to the screen:
subprocess.Popen("cat file", shell=True, stdout=logfile, stderr=logfile)
Of course, I could just do something like this, but is there any way to do this without tee and shell file descriptor redirection?:
subprocess.Popen("cat file 2>&1 | tee -a logfile", shell=True)
You can use a pipe to read the data from the program's stdout and write it to all the places you want:
import sys
import subprocess
logfile = open('logfile', 'w')
proc=subprocess.Popen(['cat', 'file'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout:
sys.stdout.write(line)
logfile.write(line)
proc.wait()
UPDATE
In python 3, the universal_newlines parameter controls how pipes are used. If False, pipe reads return bytes objects and may need to be decoded (e.g., line.decode('utf-8')) to get a string. If True, python does the decode for you
Changed in version 3.3: When universal_newlines is True, the class uses the encoding locale.getpreferredencoding(False) instead of locale.getpreferredencoding(). See the io.TextIOWrapper class for more information on this change.
To emulate: subprocess.call("command 2>&1 | tee -a logfile", shell=True) without invoking the tee command:
#!/usr/bin/env python2
from subprocess import Popen, PIPE, STDOUT
p = Popen("command", stdout=PIPE, stderr=STDOUT, bufsize=1)
with p.stdout, open('logfile', 'ab') as file:
for line in iter(p.stdout.readline, b''):
print line, #NOTE: the comma prevents duplicate newlines (softspace hack)
file.write(line)
p.wait()
To fix possible buffering issues (if the output is delayed), see links in Python: read streaming input from subprocess.communicate().
Here's Python 3 version:
#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE, STDOUT
with Popen("command", stdout=PIPE, stderr=STDOUT, bufsize=1) as p, \
open('logfile', 'ab') as file:
for line in p.stdout: # b'\n'-separated lines
sys.stdout.buffer.write(line) # pass bytes as is
file.write(line)
Write to terminal byte by byte for interactive applications
This method write any bytes it gets to stdout immediately, which more closely simulates the behavior of tee, especially for interactive applications.
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
with subprocess.Popen(sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as proc, \
open('logfile.txt', 'bw') as logfile:
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
logfile.write(byte)
# logfile.flush()
else:
break
exit_status = proc.returncode
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(10):
print(i)
sys.stdout.flush()
time.sleep(1)
First we can do a non-interactive sanity check:
./main.py ./sleep.py
And we see it counting to stdout on real time.
Next, for an interactive test, you can run:
./main.py bash
Then the characters you type appear immediately on the terminal as you type them, which is very important for interactive applications. This is what happens when you run:
bash | tee logfile.txt
Also, if you want the output to show on the ouptut file immediately, then you can also add a:
logfile.flush()
but tee does not do this, and I'm afraid it would kill performance. You can test this out easily with:
tail -f logfile.txt
Related question: live output from subprocess command
Tested on Ubuntu 18.04, Python 3.6.7.
Related
I have a .war file I'd like to launch via python. I want it to run in the background so no log messages appear in my terminal. Also I would love to have the actual log output put into a logfile. This is the python code I use to solve this.
I had no luck yet. The process is not detached because I cannot run other shell commands after executing the script. The logfile is created but no log-output is appended there.
EDIT: To make things more clear. I want to enhance this script to run multiple java processes in the end. Therefore this python script should spawn those java processes and die in the end. How to achieve exactly that including the functionality of redirecting stdout to a file?
#!/usr/bin/env python3
import subprocess
import re
platformDir = "./platform/"
fe = "frontend-webapp-0.5.0.war"
logfile = open("frontend-log", 'w')
process = subprocess.Popen(['java', '-jar', platformDir + fe],
stdout=subprocess.PIPE)
for line in process.stdout:
logFile.write(line)
Here is how you can try it using Python 3:
import sys
from subprocess import Popen, PIPE, STDOUT
platformDir = "./platform/"
fe = "frontend-webapp-0.5.0.war"
logfile = open("frontend-log", 'ab')
p = Popen(['java', '-jar', platformDir + fe], stdout=PIPE, stderr=STDOUT, bufsize=1)
for line in p.stdout:
sys.stdout.buffer.write(line)
logfile.write(line)
I actually just simply had to give stdout the file handle like stdout=logFile. The solution with the for loop through the stdout would leave me in the python script process all the time
import sys
from subprocess import Popen, PIPE, STDOUT
platformDir = "./platform/"
fe = "frontend-webapp-0.5.0.war"
logfile = open("frontend-log", 'ab')
p = Popen(['java', '-jar', platformDir + fe], stdout=logfile, stderr=STDOUT, bufsize=1)
I have created a Subprocess object. The subprocess invokes a shell, I need to send the shell command provided below to it. The code I've tried:
from subprocess import Popen, PIPE
p = Popen(["code.exe","25"],stdin=PIPE,stdout=PIPE,stderr=PIPE)
print p.communicate(input='ping 8.8.8.8')
The command doesn't execute, nothing is being input into the shell. Thanks in advance.
If I simulate code.exe to read the arg and then process stdin:
#!/usr/bin/env bash
echo "arg: $1"
echo "stdin:"
while read LINE
do
echo "$LINE"
done < /dev/stdin
and slightly update your code:
import os
from subprocess import Popen, PIPE
cwd = os.getcwd()
exe = os.path.join(cwd, 'foo.sh')
p = Popen([exe, '25'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(input='aaa\nbbb\n')
for line in out.split('\n'):
print(line)
Then the spawned process outputs:
arg: 25
stdin:
aaa
bbb
If input is changed without a \n though:
out, err = p.communicate(input='aaa')
Then it doesn't appear:
arg: 25
stdin:
Process finished with exit code 0
So you might want to look closely at the protocol between both ends of the pipe. For example this might be enough:
input='ping 8.8.8.8\n'
Hope that helps.
I have a python script that calls a shell scrips, that in turn calls a .exe called iv4_console. I need to print the stdout of iv4_console for debugging purposes. I used this:
Python:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["c.sh", var], shell=True, stdout=subprocess.PIPE)
output = ''
for line in iter(proc.stdout.readline, ""):
print line
output += line
Shell:
start_dir=$PWD
release=$1
echo Release inside shell: $release
echo Directory: $start_dir
cd $start_dir
cd ../../iv_system4/ports/visualC12/Debug
echo Debug dir: $PWD
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.xml
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.xml
exit
But this didn't work, it prints nothing. What do you think?
See my comment, best approach (i.m.o) would be to just use python only.
However, in answer of your question, try:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["/bin/bash", "/full/path/to/c.sh"], stdout=subprocess.PIPE)
# Best to always avoid shell=True because of security vulnerabilities.
proc.wait() # To make sure the shell script does not continue running indefinitely in the background
output, errors = proc.communicate()
print(output.decode())
# Since subprocess.communicate() returns a bytes-string, you can use .decode() to print the actual output as a string.
You can use
import subprocess
subprocess.call(['./c.sh'])
to call the shell script in python file
or
import subprocess
import shlex
subprocess.call(shlex.split('./c.sh var'))
how to use subprocess if my temp-file argument is in the middle of the command? For example a terminal command looks like this:
program subprogram -a -b tmpFILE otherFILE
I tried variations of this:
from subprocess import Popen, PIPE
from tempfile import SpooledTemporaryFile as tempfile
tmpFILE=tempfile()
tmpFILE.write(someList)
tmpFILE.seek(0)
print Popen(['program','subprogram', '-a', '-b', otherFile],stdout=PIPE,stdin=tmpFILE).stdout.read()
f.close()
or
print Popen(['program','subprogram', '-a', '-b', tmpFILE, otherFile],stdout=PIPE,stdin=tmpFILE).stdout.read()
but nothing works... My temporary generated file in python shouldn't be as the last parameter.
Thanks
Is there a reason to use SpooledTemporaryFile instead of other types of temp file? If not, I recommend using NamedTemporaryFile as you can retrieve the name from it. I have tried to retrieve the name from SpooledTemporaryFile and got '<fdopen>' which does not seem to be valid.
Here is the suggested code:
from subprocess import Popen, PIPE
import tempfile
with tempfile.NamedTemporaryFile() as temp_file:
temp_file.write(someList)
temp_file.flush()
process = Popen(['program', 'subprogram', '-a', '-b', temp_file.name, otherFile], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate()
Discussion
Using the with statement, you don't have to worry about closing the file. As soon as the with block is finished, the file is automatically closed.
Instead of calling seek, you should call flush to commit your file buffer to disk before calling program.
i am unable to pass in commands to stdin in python 3.2.5. I have tried with the following 2 approaches
Also: This question is a continuation of a previous question.
from subprocess import Popen, PIPE, STDOUT
import time
p = Popen([r'fileLoc/uploader.exe'],shell = True, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
p.stdin.write('uploader -i file.txt -d outputFolder\n')
print (p.communicate()[0])
p.stdin.close()
i also get numbers such as 96, 0, 85 returned to me when i try the code in the IDLE interpreter, along with errors such as from the print (p.communicate()[0])
Traceback (most recent call last):
File "<pyshell#132>", line 1, in <module>
p.communicate()[0]
File "C:\Python32\lib\subprocess.py", line 832, in communicate
return self._communicate(input)
File "C:\Python32\lib\subprocess.py", line 1060, in _communicate
self.stdin.close()
IOError: [Errno 22] Invalid argument
i have also used:
from subprocess import Popen, PIPE, STDOUT
import time
p = Popen([r'fileLoc/uploader.exe'],shell = True, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
p.communicate(input= bytes(r'uploader -i file.txt -d outputFolder\n','UTF-8'))[0]
print (p.communicate()[0])
p.stdin.close()
but with no luck.
don't use shell=True when passing the arguments as list.
stdin.write needs a bytes object as argument. You try to wire a str.
communicate() writes input to the stdin and returns a tuple with the otput of stdout and sterr, and it waits until the process has finished. You can only use it once, trying to call it a second time will result in an error.
are your sure the line you're writing should be passed to your process on stdin? Shouldn't it be the command you're trying to run?
Pass command arguments as arguments, not as stdin
The command might read username/password from console directly without using subprocess' stdin. In this case you might need winpexpect or SendKeys modules. See my answer to a similar quesiton that has corresponding code examples
Here's an example how to start a subprocess with arguments, pass some input, and write merged subprocess' stdout/stderr to a file:
#!/usr/bin/env python3
import os
from subprocess import Popen, PIPE, STDOUT
command = r'fileLoc\uploader.exe -i file.txt -d outputFolder'# use str on Windows
input_bytes = os.linesep.join(["username#email.com", "password"]).encode("ascii")
with open('command_output.txt', 'wb') as outfile:
with Popen(command, stdin=PIPE, stdout=outfile, stderr=STDOUT) as p:
p.communicate(input_bytes)