I have a program depending on a large code base that prints a lot of irrelevant and annoying messages. I would like to clean them up a bit, but since their content is dynamically generated, I can't just grep for them.
Is there a way to place a hook on the print statement? (I use python 2.4, but I would be interested in results for any version). Is there another way to find from which "print" statement the output comes?
For CPython2.5 or older:
import sys
import inspect
import collections
_stdout = sys.stdout
Record = collections.namedtuple(
'Record',
'frame filename line_number function_name lines index')
class MyStream(object):
def __init__(self, target):
self.target = target
def write(self, text):
if text.strip():
record = Record(*inspect.getouterframes(inspect.currentframe())[1])
self.target.write(
'{f} {n}: '.format(f = record.filename, n = record.line_number))
self.target.write(text)
sys.stdout = MyStream(sys.stdout)
def foo():
print('Hi')
foo()
yields
/home/unutbu/pybin/test.py 20: Hi
For CPython2.6+ we can import the print function with
from __future__ import print_function
and then redirect it as we wish:
from __future__ import print_function
import sys
import inspect
import collections
Record = collections.namedtuple(
'Record',
'frame filename line_number function_name lines index')
def myprint(text):
if text.strip():
record = Record(*inspect.getouterframes(inspect.currentframe())[1])
sys.stdout.write('{f} {n}: '.format(f = record.filename, n = record.line_number))
sys.stdout.write(text + '\n')
def foo():
print('Hi')
print = myprint
foo()
Note that inspect.currentframe uses sys._getframe which is not part of all implementations of Python. So the solution above may only work for CPython.
Strictly speaking, code base that you depend on, as in libraries, shouldn't contain any print statements. So, you should really just remove all of them.
Other than that, you can monkey-patch stdout: Adding a datetime stamp to Python print
a very gross hack to make this work:
use your favorite text editor, use your search/find feature.
find all the print statements.
and input into each of them a number, or identifier manually. (or automatically if you do this what a script)
a script to do this would be simple, just have it look for for print with regex, and replace it with print ID, and then it will all be the same, but you will get numbers.
cheers.
edit
barring any strange formatting, the following code should do it for you.
note, this is just an example of a way you could do it. not really an answer.
import re
class inc():
def __init__(self):
self.x = 0
def get(self):
self.x += 1
return self.x
def replacer(filename_in, filename_out):
i = inc()
out = open(filename_out, 'w')
with open(filename_in) as f:
for line in f:
out.write("%s\n" % re.sub(r'print', 'print %d,' % i.get(), line))
i used an basic incrementer class in case you wanted to had some kind of more complex ID, instead of just having a counter.
In harsh circumstances (output done in some weird binary libraries) you could also use strace -e write (and more options). If you do not read strace's output, the straced program waits until you do, so you can send it a signal and see where it dies.
Here is a trick that Jeeeyul came up with for Java: Replace the output stream (i.e. sys.out) with something that notices when a line feed has been written.
If this flag is true, throw an exception when the next byte is being written. Catch the exception in the same place, walk up the stack trace until you find code that doesn't belong to your "debug stream writer".
Pseudocode:
class DebugPrintln:
def __init__(self):
self.wasLF = False
def write(self, x):
if self.wasLF:
self.wasLF = False
frames = traceback.extract_stack()
... find calling code and output it ...
if x == '\n':
self.wasLF = true
super.write(x)
Related
I'm trying to log all my prints to a log file (and add a timestamp before the actual message, only in the log file), but I cannot figure out why sys.stdout.write is called twice.
import sys
from datetime import datetime
class Logger(object):
def __init__(self, filename="Default.log"):
self.stdout = sys.stdout
self.log = open(filename, "w")
def write(self, message):
self.stdout.write(message)
time_stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
self.log.write(f"{time_stamp} - {message}")
self.log.flush()
def flush(self):
pass
sys.stdout = Logger("log_file.txt")
print("Hello world !")
Output from terminal:
Hello World !
Output in log_file.txt:
2021-04-19 18:43:14.800691 - Hello world !2021-04-19 18:43:14.800735 -
What am I missing here? The write method is called again after the self.log.flush() has been called, but here message=''.
If I neglect the time_stamp variable it works like a charm, e.g calling self.log.write(message).
I could of course just check if message is empty, but I would really like to understand my problem here!
My solution
#iBug gave me the answer! I didn't know that print did work this way: one write call for the data and one write call for the end keyword which is \n as default.
My solution was to add a variable self._hidden_end to mthe Logger class that is initiated to 0 and then toggles everytime write is called to check it I should add a time stampe before writing to the log or not.
import sys
from datetime import datetime
class Logger(object):
def __init__(self, filename="Default.log"):
self.stdout = sys.stdout
self.log = open(filename, "w")
self._hidden_end = 0
def write(self, message):
self.stdout.write(message)
if not self._hidden_end:
time_stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
message = f"{time_stamp} - {message}"
self._hidden_end ^= 1
self.log.write(message)
self.log.flush()
def flush(self):
pass
sys.stdout = Logger("log_file.txt")
print("Hello world !")
Any thoughts on my solution above are welcome! :)
Your perceived results indicates that print() calls file.write twice: Once for the data and once for the "end content", which is a newline by default and can be overridden with the end keyword argument. Relevant source code lines confirm this speculation.
However, this is not something you should rely on. It's entirely implementation details, may be different for another implementation (e.g. PyPy) and may change at any time without notice, not to mention that overriding builtins is another bad practice. You should avoid this kind of practice without a compelling reason not to modify other parts of the code to use your custom logging facility.
Should you really need to do monkey-patching, it's safer to override the print() function since it has a defined, reliable interface. You can import the builtins module and assign your new print function to builtins.print, maintaining control over calls to your logging handler.
In C++, I can print debug output like this:
printf(
"FILE: %s, FUNC: %s, LINE: %d, LOG: %s\n",
__FILE__,
__FUNCTION__,
__LINE__,
logmessage
);
How can I do something similar in Python?
There is a module named inspect which provides these information.
Example usage:
import inspect
def PrintFrame():
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename) # __FILE__ -> Test.py
print(info.function) # __FUNCTION__ -> Main
print(info.lineno) # __LINE__ -> 13
def Main():
PrintFrame() # for this line
Main()
However, please remember that there is an easier way to obtain the name of the currently executing file:
print(__file__)
For example
import inspect
frame = inspect.currentframe()
# __FILE__
fileName = frame.f_code.co_filename
# __LINE__
fileNo = frame.f_lineno
There's more here http://docs.python.org/library/inspect.html
Building on geowar's answer:
class __LINE__(object):
import sys
def __repr__(self):
try:
raise Exception
except:
return str(sys.exc_info()[2].tb_frame.f_back.f_lineno)
__LINE__ = __LINE__()
If you normally want to use __LINE__ in e.g. print (or any other time an implicit str() or repr() is taken), the above will allow you to omit the ()s.
(Obvious extension to add a __call__ left as an exercise to the reader.)
You can refer my answer:
https://stackoverflow.com/a/45973480/1591700
import sys
print sys._getframe().f_lineno
You can also make lambda function
I was also interested in a __LINE__ command in python.
My starting point was https://stackoverflow.com/a/6811020 and I extended it with a metaclass object. With this modification it has the same behavior like in C++.
import inspect
class Meta(type):
def __repr__(self):
# Inspiration: https://stackoverflow.com/a/6811020
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
# print(info.filename) # __FILE__ -> Test.py
# print(info.function) # __FUNCTION__ -> Main
# print(info.lineno) # __LINE__ -> 13
return str(info.lineno)
class __LINE__(metaclass=Meta):
pass
print(__LINE__) # print for example 18
wow, 7 year old question :)
Anyway, taking Tugrul's answer, and writing it as a debug type method, it can look something like:
def debug(message):
import sys
import inspect
callerframerecord = inspect.stack()[1]
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename, 'func=%s' % info.function, 'line=%s:' % info.lineno, message)
def somefunc():
debug('inside some func')
debug('this')
debug('is a')
debug('test message')
somefunc()
Output:
/tmp/test2.py func=<module> line=12: this
/tmp/test2.py func=<module> line=13: is a
/tmp/test2.py func=<module> line=14: test message
/tmp/test2.py func=somefunc line=10: inside some func
import inspect
.
.
.
def __LINE__():
try:
raise Exception
except:
return sys.exc_info()[2].tb_frame.f_back.f_lineno
def __FILE__():
return inspect.currentframe().f_code.co_filename
.
.
.
print "file: '%s', line: %d" % (__FILE__(), __LINE__())
Here is a tool to answer this old yet new question!
I recommend using icecream!
Do you ever use print() or log() to debug your code? Of course, you
do. IceCream, or ic for short, makes print debugging a little sweeter.
ic() is like print(), but better:
It prints both expressions/variable names and their values.
It's 40% faster to type.
Data structures are pretty printed.
Output is syntax highlighted.
It optionally includes program context: filename, line number, and parent function.
For example, I created a module icecream_test.py, and put the following code inside it.
from icecream import ic
ic.configureOutput(includeContext=True)
def foo(i):
return i + 333
ic(foo(123))
Prints
ic| icecream_test.py:6 in <module>- foo(123): 456
To get the line number in Python without importing the whole sys module...
First import the _getframe submodule:
from sys import _getframe
Then call the _getframe function and use its' f_lineno property whenever you want to know the line number:
print(_getframe().f_lineno) # prints the line number
From the interpreter:
>>> from sys import _getframe
... _getframe().f_lineno # 2
Word of caution from the official Python Docs:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
In other words: Only use this code for personal testing / debugging reasons.
See the Official Python Documentation on sys._getframe for more information on the sys module, and the _getframe() function / submodule.
Based on Mohammad Shahid's answer (above).
I want to process a live output of a thirdparty script, printing some lines that match a pattern but skipping others:
def thirdparty_code():
from random import choice
stuff = ['keep: important stuff', 'ignore: boring stuff']
while True:
chosen_line = choice(stuff)
print(chosen_line)
I use redirect_stdout (which passes lines to my dummy IO) and an extended StringIO (which serves as the IO but also calls my filtering function). However, when I call print() inside of my processing function I get a RecursionError - which is not unexpected:
from io import StringIO
from contextlib import redirect_stdout
class StringIOWithCallback(StringIO):
def __init__(self, callback, **kwargs):
super().__init__(**kwargs)
self.callback = callback
def write(self, s):
super().write(s)
self.callback(s)
def filter_boring_stuff_out(line):
if line.startswith('ignore'):
return
print(line)
my_io = StringIOWithCallback(filter_boring_stuff_out)
with redirect_stdout(my_io):
thirdparty_code()
I wonder if it is possible to escape from the redirect, e.g. specifying the file parameter in print() function so that it prints to the actual standard output. I know that I can easily use the standard error stream:
import sys
print(line, file=sys.stderr)
But I specifically want to use the standard output. Is there a nice, pythonic way to do that?
After having redirected the stdout, you can easily reset it thanks to __stdout__ which saves the original value.
sys.stdout = redirected_stdout
...
...
sys.stdout = sys.__stdout__
If you continually find yourself switching these output streams, you ought to create a function to output to the redirected stream:
def redirect(*args):
print(*args, file=redirected_file)
redirect(...) # redirect this output
print(...) # use standard stream
Once I have written my question I have realised that one simply needs to save the standard output object sys.stdout to a variable before invoking the redirect:
stdout = sys.stdout
def filter_boring_stuff_out(line):
if line.startswith('ignore'):
return
print(line, file=stdout)
But as always - I would be happy to learn about other possible solutions.
You can redirect stdout and ignore all messages that started with ignore. If you do it this way all prints will be intercepted. This will work better if you are trying to filter messages from code you don't have access to or don't want to change.
import sys
from contextlib import redirect_stdout
class Filter_Out:
def __init__(self, *_, start=None, anywhere=None, end=None):
self.last_ignore = False
self.start = start
self.anywhere = anywhere
self.end = end
self.terminal = sys.stdout
def write(self, txt):
if (self.start and txt.startswith(self.start)) or \
(self.end and txt.endswith(self.end)) or \
(self.anywhere and self.anywhere in txt):
self.last_ignore = True
return
if self.last_ignore and txt == '\n':
self.last_ignore = False
else:
self.terminal.write(txt)
def flush(self):
pass
with redirect_stdout(Filter_Out(start='ignore', anywhere='4')):
print("test")
print("test2")
print("ignore: test2") # will not print because it started with ignore
print("test1")
print("test42") # will not print because it had 4 in it
Is there a way to stop a function from calling print?
I am using the pygame.joystick module for a game I am working on.
I created a pygame.joystick.Joystick object and in the actual loop of the game call its member function get_button to check for user input. The function does everything I need it to do, but the problem is that it also calls print, which slows down the game considerably.
Can I block this call to print?
Python lets you overwrite standard output (stdout) with any file object. This should work cross platform and write to the null device.
import sys, os
# Disable
def blockPrint():
sys.stdout = open(os.devnull, 'w')
# Restore
def enablePrint():
sys.stdout = sys.__stdout__
print 'This will print'
blockPrint()
print "This won't"
enablePrint()
print "This will too"
If you don't want that one function to print, call blockPrint() before it, and enablePrint() when you want it to continue. If you want to disable all printing, start blocking at the top of the file.
Use with
Based on #FakeRainBrigand solution I'm suggesting a safer solution:
import os, sys
class HiddenPrints:
def __enter__(self):
self._original_stdout = sys.stdout
sys.stdout = open(os.devnull, 'w')
def __exit__(self, exc_type, exc_val, exc_tb):
sys.stdout.close()
sys.stdout = self._original_stdout
Then you can use it like this:
with HiddenPrints():
print("This will not be printed")
print("This will be printed as before")
This is much safer because you can not forget to re-enable stdout, which is especially critical when handling exceptions.
Without with — Bad practice
The following example uses enable/disable prints functions that were suggested in previous answer.
Imagine that there is a code that may raise an exception. We had to use finally statement in order to enable prints in any case.
try:
disable_prints()
something_throwing()
enable_prints() # This will not help in case of exception
except ValueError as err:
handle_error(err)
finally:
enable_prints() # That's where it needs to go.
If you forgot the finally clause, none of your print calls would print anything anymore.
It is safer to use the with statement, which makes sure that prints will be reenabled.
Note: It is not safe to use sys.stdout = None, because someone could call methods like sys.stdout.write()
As #Alexander Chzhen suggested, using a context manager would be safer than calling a pair of state-changing functions.
However, you don't need to reimplement the context manager - it's already in the standard library. You can redirect stdout (the file object that print uses) with contextlib.redirect_stdout, and also stderr with contextlib.redirect_stderr.
import os
import contextlib
with open(os.devnull, "w") as f, contextlib.redirect_stdout(f):
print("This won't be printed.")
If you want to block print calls made by a particular function, there is a neater solution using decorators. Define the following decorator:
# decorater used to block function printing to the console
def blockPrinting(func):
def func_wrapper(*args, **kwargs):
# block all printing to the console
sys.stdout = open(os.devnull, 'w')
# call the method in question
value = func(*args, **kwargs)
# enable all printing to the console
sys.stdout = sys.__stdout__
# pass the return value of the method back
return value
return func_wrapper
Then just place #blockPrinting before any function. For example:
# This will print
def helloWorld():
print("Hello World!")
helloWorld()
# This will not print
#blockPrinting
def helloWorld2():
print("Hello World!")
helloWorld2()
If you are using Jupyter Notebook or Colab use this:
from IPython.utils import io
with io.capture_output() as captured:
print("I will not be printed.")
I have had the same problem, and I did not come to another solution but to redirect the output of the program (I don't know exactly whether the spamming happens on stdout or stderr) to /dev/null nirvana.
Indeed, it's open source, but I wasn't passionate enough to dive into the pygame sources - and the build process - to somehow stop the debug spam.
EDIT :
The pygame.joystick module has calls to printf in all functions that return the actual values to Python:
printf("SDL_JoystickGetButton value:%d:\n", value);
Unfortunately you would need to comment these out and recompile the whole thing. Maybe the provided setup.py would make this easier than I thought. You could try this...
A completely different approach would be redirecting at the command line. If you're on Windows, this means a batch script. On Linux, bash.
/full/path/to/my/game/game.py > /dev/null
C:\Full\Path\To\My\Game.exe > nul
Unless you're dealing with multiple processes, this should work. For Windows users this could be the shortcuts you're creating (start menu / desktop).
You can do a simple redirection, this seems a lot safer than messing with stdout, and doesn't pull in any additional libraries.
enable_print = print
disable_print = lambda *x, **y: None
print = disable_print
function_that_has_print_in_it(1) # nothing is printed
print = enable_print
function_that_has_print_in_it(2) # printing works again!
Note: this only works to disable the print() function, and would not disable all output if you're making calls to something else that is producing output. For instance if you were calling a C library that was producing it's own output to stdout, or if you were using intput().
No, there is not, especially that majority of PyGame is written in C.
But if this function calls print, then it's PyGame bug, and you should just report it.
The module I used printed to stderr. So the solution in that case would be:
sys.stdout = open(os.devnull, 'w')
"stop a function from calling print"
# import builtins
# import __builtin__ # python2, not test
printenabled = False
def decorator(func):
def new_func(*args,**kwargs):
if printenabled:
func("print:",*args,**kwargs)
return new_func
print = decorator(print) # current file
# builtins.print = decorator(builtins.print) # all files
# __builtin__.print = decorator(__builtin__.print) # python2
import sys
import xxxxx
def main():
global printenabled
printenabled = True
print("1 True");
printenabled = False
print("2 False");
printenabled = True
print("3 True");
printenabled = False
print("4 False");
if __name__ == '__main__':
sys.exit(main())
#output
print: 1 True
print: 3 True
https://stackoverflow.com/a/27622201
Change value of file object of print() function. By default it's sys.stdout, instead we can write to null device by open(os.devnull, 'w')
import os, sys
mode = 'debug' #'prod'
if mode == 'debug':
fileobj = sys.stdout
else:
fileobj = open(os.devnull,'w')
print('Hello Stackoverflow', file = fileobj)
Based on #Alexander Chzhen solution, I present here the way to apply it on a function with an option to suppress printing or not.
import os, sys
class SuppressPrints:
#different from Alexander`s answer
def __init__(self, suppress=True):
self.suppress = suppress
def __enter__(self):
if self.suppress:
self._original_stdout = sys.stdout
sys.stdout = open(os.devnull, 'w')
def __exit__(self, exc_type, exc_val, exc_tb):
if self.suppress:
sys.stdout.close()
sys.stdout = self._original_stdout
#implementation
def foo(suppress=True):
with SuppressPrints(suppress):
print("It will be printed, or not")
foo(True) #it will not be printed
foo(False) #it will be printed
I hope I can add my solution below answer of Alexander as a comment, but I don`t have enough (50) reputations to do so.
If you want to enable/disable print with a variable, you could call an auxiliary function instead print, something like printe(the name is just for convenience)
def printe(*what_to_print):
if prints_enable:
string = ""
for items in what_to_print:
string += str(items) + " "
print(string)
Define a new Print function where you enable print first. print your output next. And then disable print again.
def Print (*output):
enablePrint()
print (output)
disablePrint()
with one of the above "safe" enable / disable pair of function
In C++, I can print debug output like this:
printf(
"FILE: %s, FUNC: %s, LINE: %d, LOG: %s\n",
__FILE__,
__FUNCTION__,
__LINE__,
logmessage
);
How can I do something similar in Python?
There is a module named inspect which provides these information.
Example usage:
import inspect
def PrintFrame():
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename) # __FILE__ -> Test.py
print(info.function) # __FUNCTION__ -> Main
print(info.lineno) # __LINE__ -> 13
def Main():
PrintFrame() # for this line
Main()
However, please remember that there is an easier way to obtain the name of the currently executing file:
print(__file__)
For example
import inspect
frame = inspect.currentframe()
# __FILE__
fileName = frame.f_code.co_filename
# __LINE__
fileNo = frame.f_lineno
There's more here http://docs.python.org/library/inspect.html
Building on geowar's answer:
class __LINE__(object):
import sys
def __repr__(self):
try:
raise Exception
except:
return str(sys.exc_info()[2].tb_frame.f_back.f_lineno)
__LINE__ = __LINE__()
If you normally want to use __LINE__ in e.g. print (or any other time an implicit str() or repr() is taken), the above will allow you to omit the ()s.
(Obvious extension to add a __call__ left as an exercise to the reader.)
You can refer my answer:
https://stackoverflow.com/a/45973480/1591700
import sys
print sys._getframe().f_lineno
You can also make lambda function
I was also interested in a __LINE__ command in python.
My starting point was https://stackoverflow.com/a/6811020 and I extended it with a metaclass object. With this modification it has the same behavior like in C++.
import inspect
class Meta(type):
def __repr__(self):
# Inspiration: https://stackoverflow.com/a/6811020
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
# print(info.filename) # __FILE__ -> Test.py
# print(info.function) # __FUNCTION__ -> Main
# print(info.lineno) # __LINE__ -> 13
return str(info.lineno)
class __LINE__(metaclass=Meta):
pass
print(__LINE__) # print for example 18
wow, 7 year old question :)
Anyway, taking Tugrul's answer, and writing it as a debug type method, it can look something like:
def debug(message):
import sys
import inspect
callerframerecord = inspect.stack()[1]
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename, 'func=%s' % info.function, 'line=%s:' % info.lineno, message)
def somefunc():
debug('inside some func')
debug('this')
debug('is a')
debug('test message')
somefunc()
Output:
/tmp/test2.py func=<module> line=12: this
/tmp/test2.py func=<module> line=13: is a
/tmp/test2.py func=<module> line=14: test message
/tmp/test2.py func=somefunc line=10: inside some func
import inspect
.
.
.
def __LINE__():
try:
raise Exception
except:
return sys.exc_info()[2].tb_frame.f_back.f_lineno
def __FILE__():
return inspect.currentframe().f_code.co_filename
.
.
.
print "file: '%s', line: %d" % (__FILE__(), __LINE__())
Here is a tool to answer this old yet new question!
I recommend using icecream!
Do you ever use print() or log() to debug your code? Of course, you
do. IceCream, or ic for short, makes print debugging a little sweeter.
ic() is like print(), but better:
It prints both expressions/variable names and their values.
It's 40% faster to type.
Data structures are pretty printed.
Output is syntax highlighted.
It optionally includes program context: filename, line number, and parent function.
For example, I created a module icecream_test.py, and put the following code inside it.
from icecream import ic
ic.configureOutput(includeContext=True)
def foo(i):
return i + 333
ic(foo(123))
Prints
ic| icecream_test.py:6 in <module>- foo(123): 456
To get the line number in Python without importing the whole sys module...
First import the _getframe submodule:
from sys import _getframe
Then call the _getframe function and use its' f_lineno property whenever you want to know the line number:
print(_getframe().f_lineno) # prints the line number
From the interpreter:
>>> from sys import _getframe
... _getframe().f_lineno # 2
Word of caution from the official Python Docs:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
In other words: Only use this code for personal testing / debugging reasons.
See the Official Python Documentation on sys._getframe for more information on the sys module, and the _getframe() function / submodule.
Based on Mohammad Shahid's answer (above).