We Keep Coding

How to preserve python colorama color output when using tee on Windows

In my Windows 10 scenario, I want to print arbitrary python console output (print(), sys.exit(), etc) both to console and a log file. I don't have control over some parts of the code (external python packages), so I cannot use some dedicated logging mechanism.
After some research I found the tool tee.exe in UnxUtils which does this task almost the way I want.
My problem is to preserve color as generated by python's colorama package. Is there any way to accomplish that? Currently, tee.exe strips away the color.
The answer I am looking does not have to rely on tee.exe, it's just the closest I got to a real solution. What I am looking for should do the following:
any command line output appears both in the command line and the log file (both STOUT and STDERR)
the output appears on the command line in real time. Bonus points if this is also true for the log file.
color is preserved on the command line. Bonus points if the log file does not contain any color-related artifacts.
What I have so far is this:
Python file teetest.py:
import sys
import colorama
print("Test")
print("2nd Test")
colorama.init(autoreset=True)
print(colorama.Fore.RED + 'Color Test')
colorama.deinit(autoreset=True)
sys.exit("Error_Test")
Batch file teetest.bat:
#echo off
python teetest.py 2>&1 | tee log.txt
pause
My output looks like this (command line and log file are identical, no color):
Test
2nd Test
Color Test
Error_Test
The solution I am looking for will print the above to the command line so the words Color Test are red.
Edit:
It seems that tee.exe is not at fault. Instead, colorama strips away the ANSI characters for color control by design, so the color is lost when passed through tee.exe.
From the colorama manual:
Colorama makes this work on Windows, too, by wrapping stdout, stripping ANSI sequences it finds (which would appear as gobbledygook in the output), and converting them into the appropriate win32 calls to modify the state of the terminal.
Colorama's init() function offers the parameter strip, which if False, does cause colorama to not strip away the ANSI characters. This in turn allows to write a custom tee.py that does the same as tee.exe, as outlined by user #martineau below. In it, we can call colorama and handle the color properly.
This might be a workable solution, but it still has the downside that I would have to replace all colorama init() calls with init(strip=False) in my original python code and that in turn would cause ANSI characters to appear in the output if the code was called without redirecting through tee.py.
This might actually be the closest we can get to a proper solution here. If anyone can offer other ideas, I'm all ears but I fear chances are slim.

I don't know how it will work with respect to colorama, but after being unsatisfied with several tee utilities for Windows that I found online, I ended up writing my own in Python (3.x).
You may have to modify it to suit your own needs, but it should be a good start.
mytee.py:
"""
Copies stdin to stdout (screen) *and* the specified file.
"""
import fileinput
import os
from pathlib import Path
import sys
SHOW_FULL_PATH = False
DIVIDER = True
DIV_CH = ' '
if len(sys.argv) != 2:
raise SystemExit('Usage: mytee <filepath>')
try:
inp = fileinput.input(()) # Read from stdin.
path = Path(sys.argv[1])
stdout_write = sys.stdout.write
stdout_flush = sys.stdout.flush
# Assumes .py in same dir as output file.
script = (f'{path.parent/path.stem}{path.suffix}' if SHOW_FULL_PATH else
f'{path.stem}{path.suffix}')
with open(path, 'w') as outp: # Write to specified file.
outp_write = outp.write
outp_flush = outp.flush
def write(line):
stdout_write(line)
outp_write(line)
def writeln(line):
write(line + '\n')
banner = f'"{script}"' if ' ' in script else f'-[{script}]-'
writeln(f'{banner}')
if DIVIDER:
writeln(f'{DIV_CH * len(banner)}')
for line in inp:
write(line)
if DIVIDER:
writeln(f'{DIV_CH * len(banner)}')
writeln('-[done]-')
finally:
inp.close() # Not sure this is really necessary.
sys.exit(0)

How to edit console output

I have the following very simple code
stdout.write("Hello World")
stdout.write("\rBye world")
stdout.write("\rActually hello back")
Which prints as expected 'Actually hello back' however if i were to add a newline
stdout.write("\n")
How can I go back a newline and then to the beginning of the line so I can actually just output "Hi" instead of
Actually hello back
Hi
I tried
stdout.write("\r")
stdout.write("\b")
However none of them seem to do the trick. The end goal is to display a big chunk of text and then update the output in real time without having to write again. How can I achieve this in python?
EDIT
My question is different than the one suggested as I don't want to modify a single line. I want to be able to print 4-5 lines of text and then replace them in real time instead of just one line that is modified.

Well, if You want to gain full control over the terminal, I would suggest to use the curses library.
The curses module provides an interface to the curses library, the
de-facto standard for portable advanced terminal handling.
Using it, You can edit multiple lines in terminal like this:
import curses
import time
stdscr = curses.initscr()
stdscr.addstr("line 1\n")
stdscr.addstr("line 2\n")
stdscr.refresh()
time.sleep(3)
stdscr.erase()
stdscr.addstr("edited line 1\n")
stdscr.addstr("edited line 2\n")
stdscr.refresh()
time.sleep(3)
curses.endwin()
The capabilities of this library are much greater though. Full tutorial here.

Print over Current Line in Python Shell

I'm trying to make a percentage text that displays a progress amount but i'm trying to avoid the percentages printing out like this:
Progress: 10%
Progress: 11%
Progress: 12%
Progress: 13%
How can erase and write over the current line? Iv'e tried using the \r and \b characters but neither seems to work. Every single thing I found before has been for either for Python 2 or Unix so i'm not even sure which of those is the problem (if even one of them) because i'm not using either. Does anyone know how I can do this with Python 3 running Windows 7? This is the unworking code that I have currently, but I've tried plenty of other things.
print('Progress: {}%'.format(solutions//possibleSolutions),flush=True,end="\r")
EDIT:
This is not a problem if I'm executing the program from command prompt so I don't think it is a problem with windows. I tried updating Python from what i was using previously (3.4.1) to the latest v3.4.3 and the issue is the same.
Heres a screenshot of the problem:
This is the best I can do at taking a screenshot of the issue. It appears as if each time I move the cursor farther to the left (passed one of the Progress:'s) that the gray area between the text and the cursor gets larger
EDIT 2: The problem is that IDLE does not support ASCII control codes. Solution: Use a different IDE.

You can use print:
print('Progress: {}%'.format(solutions),flush=True,end="\r")

You can't use '\r' and '\b' in IDLE. If you want to use it, try adding these lines at the start of your program:
import sys
sys.stdout = sys.__stdout__
and running idle with this batch script:
#echo off
echo Running IDLE...
py -m idlelib
then, you see output in cmd window and there are '\r' and '\b'.

Use the character '\r' for the print function. Default is '\n'.
'\r' stands for carriage return, '\n' means new line.

You can create a new class called Printer like this:
class Printer():
def __init__(self, data):
sys.stdout.write("\r\x1b[K"+data.__str__())
sys.stdout.flush()
Then, let's say you want to print the progress of a for loop:
for i in range(0, 100):
p = i * 100
output = "%d%% of the for loop completed" % p
Printer(output)

sys.stdin.readlines() hangs Python script

Everytime I'm executing my Python script, it appears to hang on this line:
lines = sys.stdin.readlines()
What should I do to fix/avoid this?
EDIT
Here's what I'm doing with lines:
lines = sys.stdin.readlines()
updates = [line.split() for line in lines]
EDIT 2
I'm running this script from a git hook so is there anyway around the EOF?

This depends a lot on what you are trying to accomplish. You might be able do:
for line in sys.stdin:
#do something with line
Of course, with this idiom as well as the readlines() method you are using, you need to somehow send the EOF character to your script so that it knows that the file is ready to read. (On unix Ctrl-D usually does the trick).

Unless you are redirecting something to stdin that would be expected behavior. That says to read input from stdin (which would be the console you are running the script from). It is waiting for your input.
See: "How to finish sys.stdin.readlines() input?

If you're running the program in an interactive session, then this line causes Python to read from standard input (i. e. your keyboard) until you send the EOF character (Ctrl-D (Unix/Mac) or Ctrl-Z (Windows)).
>>> import sys
>>> a = sys.stdin.readlines()
Test
Test2
^Z
>>> a
['Test\n', 'Test2\n']

I know this isn't directly answering your question, as others have already addressed the EOF issue, but typically what I've found that works best when reading live output from a long lived subprocess or stdin is the while/if line approach:
while True:
line = sys.stdin.readline()
if not line:
break
process(line)
In this case, sys.stdin.readline() will return lines of text before an EOF is returned. Once the EOF if given, the empty line will be returned which triggers the break from the loop. A hang can still occur here, as long as an EOF isn't provided.
It's worth noting that the ability to process the "live output", while the subprocess/stdin is still running, requires the writing application to flush it's output.

Reading output from os.popen4 before writing writing stdin

I have a script which executes some command using os.popen4. Problem is some time command being executed will require user input ("y" or "n"). I am reading stdout/stderr and printing it, but it seems question from command doesn't got printed and it hangs. To make it work, i had to write "n" to stdin blindly. Can some one please guide on how to handle it?
Code not working:
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
cmd_out = f_p_stdout_stderr.readlines()
print cmd_out
f_p_stdin.write("n")
f_p_stdin.close()
f_p_stdout_stderr.close()
Working Code:
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
cmd_out = f_p_stdout_stderr.readlines()
f_p_stdin.write("n")
f_p_stdin.close()
print cmd_out
f_p_stdout_stderr.close()
NOTE : I am aware that it is depreciated and subprocess module is used, but right now i don't know on how to use it. So i'll appreciate if some one will help me to handle it using os.popen4. I want to capture the question and handle the input from user and execute it.

readlines() : returns a list containing all the lines of data in the file. If reading from a process like in this case, there is a good chance it does not send a newline and/or flush the output. You should read characters from the input and process that to see if the question was posed.
It would help to know what cmd_exec looks like, so others can try and emulate what you tried.
Update:
I wrote a uncheckout command in Python:
#! /usr/bin/env python
# coding: utf-8
import sys
print 'Uncheckout of {} is irreversible'.format(sys.argv[1])
print 'Do you want to proceed? [y/N]',
sys.stdout.flush()
x = raw_input()
if x == 'y':
print sys.argv[1], "no longer checked out"
else:
print sys.argv[1], "still checked out"
I put the prompt string on purpose not as argument to raw_input, to be able to do the flush() explicitly.
Neither of your code snippets work with that (assuming cmd_exec to be ['./uncheckout', 'abc.txt'] or './uncheckout abc.txt', popen4() uses the shell in the latter case to start the program).
Only when I move the readlines() until after the write() and close() will the command continue.
That makes sense to me as the close() flushes the output. You are writing in text mode and that buffers normally until end-of-line, which is not in your .write('n').
To be able to check what the prompt is and test and react on that., the following works with the above uncheckout:
#! /usr/bin/env python
# coding: utf-8
import os
import sys
cmd_exec = ['./uncheckout', 'abc.txt']
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
line = ''
while True:
x = f_p_stdout_stderr.read(1)
if not x:
break
sys.stdout.write(x)
sys.stdout.flush()
if x == '\n':
line = ''
else:
line += x
if line.endswith('[y/N]'):
f_p_stdin.write("n\n")
f_p_stdin.flush()
sys.stdout.write('\n')
Maybe you can work backwards from that to make something that works for you. Make sure to keep flushes at appropriate places.