I have a Python file with contains a few functions. When the "main" function runs, it runs through the functions. Each function needs to write some text to the same single log file.
I have written the following codes. The file handler is declared here as global in each function. I am wondering if there is a better way to achieve this?
LOG_FILE_NAME = 'TestLog.log'
f = None
def log1():
global f
print 'log1 called'
f.write('log1 result' + '\n')
def log2():
global f
print 'log2 called'
f.write('log2 result' + '\n')
logFileDir = LOG_FILE_NAME
f = open(logFileDir, 'w')
log1()
log2()
f.close()
Expanding Tom Dalton's link (docs.python.org/2/howto/logging.html#logging-to-a-file):
import logging
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
will write this too a file
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
in your example, you could use a global logger, and each function could invoke it
import logging
logging.basicConfig(filename='example.log',level=logging.DEBUG)
def log1():
logging.info('log1')
def log2():
logging.info('log2')
if __name__ == '__main__':
log1()
log2()
if you wanted multiple loggers, you can use logging.getLogger(), there is plenty of flexibility with the logging module, check out the documentation (in the link).
One way is to write your own simple logging function as a closure:
def create_logger(log_filename = 'TestLog.log'):
f = open(log_filename, 'w')
def logger(text):
f.write(text + '\n')
return logger,f.close
logwrite,logclose = create_logger()
def log1():
print 'log1 called'
logwrite('log1 result')
def log2():
print 'log2 called'
logwrite('log2 result')
log1()
log2()
logclose()
It avoids having to use global, encapsulates the file object, simplifies the call, and enables you to use several different filenames by calling create_logger for each one required. You might, for example, wish to add a date/time stamp to the log message, simply done in one place by enhancing the logger function.
Related
I'm trying to log all my prints to a log file (and add a timestamp before the actual message, only in the log file), but I cannot figure out why sys.stdout.write is called twice.
import sys
from datetime import datetime
class Logger(object):
def __init__(self, filename="Default.log"):
self.stdout = sys.stdout
self.log = open(filename, "w")
def write(self, message):
self.stdout.write(message)
time_stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
self.log.write(f"{time_stamp} - {message}")
self.log.flush()
def flush(self):
pass
sys.stdout = Logger("log_file.txt")
print("Hello world !")
Output from terminal:
Hello World !
Output in log_file.txt:
2021-04-19 18:43:14.800691 - Hello world !2021-04-19 18:43:14.800735 -
What am I missing here? The write method is called again after the self.log.flush() has been called, but here message=''.
If I neglect the time_stamp variable it works like a charm, e.g calling self.log.write(message).
I could of course just check if message is empty, but I would really like to understand my problem here!
My solution
#iBug gave me the answer! I didn't know that print did work this way: one write call for the data and one write call for the end keyword which is \n as default.
My solution was to add a variable self._hidden_end to mthe Logger class that is initiated to 0 and then toggles everytime write is called to check it I should add a time stampe before writing to the log or not.
import sys
from datetime import datetime
class Logger(object):
def __init__(self, filename="Default.log"):
self.stdout = sys.stdout
self.log = open(filename, "w")
self._hidden_end = 0
def write(self, message):
self.stdout.write(message)
if not self._hidden_end:
time_stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
message = f"{time_stamp} - {message}"
self._hidden_end ^= 1
self.log.write(message)
self.log.flush()
def flush(self):
pass
sys.stdout = Logger("log_file.txt")
print("Hello world !")
Any thoughts on my solution above are welcome! :)
Your perceived results indicates that print() calls file.write twice: Once for the data and once for the "end content", which is a newline by default and can be overridden with the end keyword argument. Relevant source code lines confirm this speculation.
However, this is not something you should rely on. It's entirely implementation details, may be different for another implementation (e.g. PyPy) and may change at any time without notice, not to mention that overriding builtins is another bad practice. You should avoid this kind of practice without a compelling reason not to modify other parts of the code to use your custom logging facility.
Should you really need to do monkey-patching, it's safer to override the print() function since it has a defined, reliable interface. You can import the builtins module and assign your new print function to builtins.print, maintaining control over calls to your logging handler.
Is there a way to stop a function from calling print?
I am using the pygame.joystick module for a game I am working on.
I created a pygame.joystick.Joystick object and in the actual loop of the game call its member function get_button to check for user input. The function does everything I need it to do, but the problem is that it also calls print, which slows down the game considerably.
Can I block this call to print?
Python lets you overwrite standard output (stdout) with any file object. This should work cross platform and write to the null device.
import sys, os
# Disable
def blockPrint():
sys.stdout = open(os.devnull, 'w')
# Restore
def enablePrint():
sys.stdout = sys.__stdout__
print 'This will print'
blockPrint()
print "This won't"
enablePrint()
print "This will too"
If you don't want that one function to print, call blockPrint() before it, and enablePrint() when you want it to continue. If you want to disable all printing, start blocking at the top of the file.
Use with
Based on #FakeRainBrigand solution I'm suggesting a safer solution:
import os, sys
class HiddenPrints:
def __enter__(self):
self._original_stdout = sys.stdout
sys.stdout = open(os.devnull, 'w')
def __exit__(self, exc_type, exc_val, exc_tb):
sys.stdout.close()
sys.stdout = self._original_stdout
Then you can use it like this:
with HiddenPrints():
print("This will not be printed")
print("This will be printed as before")
This is much safer because you can not forget to re-enable stdout, which is especially critical when handling exceptions.
Without with — Bad practice
The following example uses enable/disable prints functions that were suggested in previous answer.
Imagine that there is a code that may raise an exception. We had to use finally statement in order to enable prints in any case.
try:
disable_prints()
something_throwing()
enable_prints() # This will not help in case of exception
except ValueError as err:
handle_error(err)
finally:
enable_prints() # That's where it needs to go.
If you forgot the finally clause, none of your print calls would print anything anymore.
It is safer to use the with statement, which makes sure that prints will be reenabled.
Note: It is not safe to use sys.stdout = None, because someone could call methods like sys.stdout.write()
As #Alexander Chzhen suggested, using a context manager would be safer than calling a pair of state-changing functions.
However, you don't need to reimplement the context manager - it's already in the standard library. You can redirect stdout (the file object that print uses) with contextlib.redirect_stdout, and also stderr with contextlib.redirect_stderr.
import os
import contextlib
with open(os.devnull, "w") as f, contextlib.redirect_stdout(f):
print("This won't be printed.")
If you want to block print calls made by a particular function, there is a neater solution using decorators. Define the following decorator:
# decorater used to block function printing to the console
def blockPrinting(func):
def func_wrapper(*args, **kwargs):
# block all printing to the console
sys.stdout = open(os.devnull, 'w')
# call the method in question
value = func(*args, **kwargs)
# enable all printing to the console
sys.stdout = sys.__stdout__
# pass the return value of the method back
return value
return func_wrapper
Then just place #blockPrinting before any function. For example:
# This will print
def helloWorld():
print("Hello World!")
helloWorld()
# This will not print
#blockPrinting
def helloWorld2():
print("Hello World!")
helloWorld2()
If you are using Jupyter Notebook or Colab use this:
from IPython.utils import io
with io.capture_output() as captured:
print("I will not be printed.")
I have had the same problem, and I did not come to another solution but to redirect the output of the program (I don't know exactly whether the spamming happens on stdout or stderr) to /dev/null nirvana.
Indeed, it's open source, but I wasn't passionate enough to dive into the pygame sources - and the build process - to somehow stop the debug spam.
EDIT :
The pygame.joystick module has calls to printf in all functions that return the actual values to Python:
printf("SDL_JoystickGetButton value:%d:\n", value);
Unfortunately you would need to comment these out and recompile the whole thing. Maybe the provided setup.py would make this easier than I thought. You could try this...
A completely different approach would be redirecting at the command line. If you're on Windows, this means a batch script. On Linux, bash.
/full/path/to/my/game/game.py > /dev/null
C:\Full\Path\To\My\Game.exe > nul
Unless you're dealing with multiple processes, this should work. For Windows users this could be the shortcuts you're creating (start menu / desktop).
You can do a simple redirection, this seems a lot safer than messing with stdout, and doesn't pull in any additional libraries.
enable_print = print
disable_print = lambda *x, **y: None
print = disable_print
function_that_has_print_in_it(1) # nothing is printed
print = enable_print
function_that_has_print_in_it(2) # printing works again!
Note: this only works to disable the print() function, and would not disable all output if you're making calls to something else that is producing output. For instance if you were calling a C library that was producing it's own output to stdout, or if you were using intput().
No, there is not, especially that majority of PyGame is written in C.
But if this function calls print, then it's PyGame bug, and you should just report it.
The module I used printed to stderr. So the solution in that case would be:
sys.stdout = open(os.devnull, 'w')
"stop a function from calling print"
# import builtins
# import __builtin__ # python2, not test
printenabled = False
def decorator(func):
def new_func(*args,**kwargs):
if printenabled:
func("print:",*args,**kwargs)
return new_func
print = decorator(print) # current file
# builtins.print = decorator(builtins.print) # all files
# __builtin__.print = decorator(__builtin__.print) # python2
import sys
import xxxxx
def main():
global printenabled
printenabled = True
print("1 True");
printenabled = False
print("2 False");
printenabled = True
print("3 True");
printenabled = False
print("4 False");
if __name__ == '__main__':
sys.exit(main())
#output
print: 1 True
print: 3 True
https://stackoverflow.com/a/27622201
Change value of file object of print() function. By default it's sys.stdout, instead we can write to null device by open(os.devnull, 'w')
import os, sys
mode = 'debug' #'prod'
if mode == 'debug':
fileobj = sys.stdout
else:
fileobj = open(os.devnull,'w')
print('Hello Stackoverflow', file = fileobj)
Based on #Alexander Chzhen solution, I present here the way to apply it on a function with an option to suppress printing or not.
import os, sys
class SuppressPrints:
#different from Alexander`s answer
def __init__(self, suppress=True):
self.suppress = suppress
def __enter__(self):
if self.suppress:
self._original_stdout = sys.stdout
sys.stdout = open(os.devnull, 'w')
def __exit__(self, exc_type, exc_val, exc_tb):
if self.suppress:
sys.stdout.close()
sys.stdout = self._original_stdout
#implementation
def foo(suppress=True):
with SuppressPrints(suppress):
print("It will be printed, or not")
foo(True) #it will not be printed
foo(False) #it will be printed
I hope I can add my solution below answer of Alexander as a comment, but I don`t have enough (50) reputations to do so.
If you want to enable/disable print with a variable, you could call an auxiliary function instead print, something like printe(the name is just for convenience)
def printe(*what_to_print):
if prints_enable:
string = ""
for items in what_to_print:
string += str(items) + " "
print(string)
Define a new Print function where you enable print first. print your output next. And then disable print again.
def Print (*output):
enablePrint()
print (output)
disablePrint()
with one of the above "safe" enable / disable pair of function
I am writing some code that will output a log to either the screen, or a file, but not both.
I thought the easiest way to do this would be to write a class:
class WriteLog:
"write to screen or to file"
def __init__(self, stdout, filename):
self.stdout = stdout
self.logfile = file(filename, 'a')
def write(self, text):
self.stdout.write(text)
self.logfile.write(text)
def close(self):
self.stdout.close()
self.logfile.close()
And then call it something like this:
output = WriteLog(sys.stdout, 'log.txt')
However, I'm not sure how to allow for switching between the two, i.e. there should be an option within the class that will set WriteLog to either use stdout, or filename. Once that option has been set I just use WriteLog without any need for if statements etc.
Any ideas? Most of the solutions I see online are trying to output to both simultaneously.
Thanks.
Maybe something like this? It uses the symbolic name 'stdout' or 'stderr' in the constructor, or a real filename. The usage of if is limited to the constructor. By the way, I think you're trying to prematurely optimize (which is the root of all evil): you're trying to save time on if's while in real life, the program will spend much more time in file I/O; making the potential waste on your if's negligible.
import sys
class WriteLog:
def __init__(self, output):
self.output = output
if output == 'stdout':
self.logfile = sys.stdout
elif output == 'stderr':
self.logfile = sys.stderr
else:
self.logfile = open(output, 'a')
def write(self, text):
self.logfile.write(text)
def close(self):
if self.output != 'stdout' and self.output != 'stderr':
self.logfile.close()
def __del__(self):
self.close()
if __name__ == '__main__':
a = WriteLog('stdout')
a.write('This goes to stdout\n')
b = WriteLog('stderr')
b.write('This goes to stderr\n')
c = WriteLog('/tmp/logfile')
c.write('This goes to /tmp/logfile\n')
I'm not an expert in it, but try to use the logging library, and maybe you can have logger with 2 handlers, one for file and one for stream and then add/remove handlers dynamically.
I like the suggestion about using the logging library. But if you want to hack out something yourself, maybe passing in the file handle is worth considering.
import sys
class WriteLog:
"write to screen or to file"
def __init__(self, output):
self.output = output
def write(self, text):
self.output.write(text)
def close(self):
self.output.close()
logger = WriteLog(file('c:/temp/log.txt','a' ))
logger.write("I write to the log file.\n")
logger.close()
sysout = WriteLog(sys.stdout)
sysout.write("I write to the screen.\n")
You can utilize the logging library to do something similar to this. The following function will set up a logging object at the INFO level.
def setup_logging(file_name, log_to_file=False, log_to_console=False ):
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# Create Console handler
if log_to_file:
console_log = logging.StreamHandler()
console_log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)-8s - %(name)-12s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
# Log file
if log_to_console:
file_log = logging.FileHandler('%s.log' % (file_name), 'a', encoding='UTF-8')
file_log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)-8s - %(name)-12s - %(message)s')
file_log.setFormatter(formatter)
logger.addHandler(file_log)
return logger
You pass it the name of your log, and where you wish to log (either to the file, the console or both). Then you can utilize this function in your code block like this:
logger = setup_logging("mylog.log", log_to_file=True, log_to_console=False)
logger.info('Message')
This example will log to a file named mylog.log (in the current directory) and have output like this:
2014-11-05 17:20:29,933 - INFO - root - Message
This function has areas for improvement (if you wish to add more functionality). Right now it logs to both the console and file at log level INFO on the .setLevel(logging.INFO) lines. This could be set dynamically if you wish.
Additionally, as it is now, you can easily add standard logging lines (logger.debug('Message'), logger.critcal('DANGER!')) without modifying a class. In these examples, the debug messages won't print (because it is set to INFO) and the critical ones will.
I'm currently doing this but I feel like there would be a cleaner way to do this, as I'm editing the code to write to a file everything that it prints. Is there a one-liner or something more concise for the following? Thanks
print 'foo bar'
file.write('foo bar')
You can create a file-like object that writes to a number of other files:
class Tee(object):
"""A file-like that writes to all the file-likes it has."""
def __init__(self, *files):
"""Make a Tee that writes to all the files in `files.`"""
self._files = files
def write(self, data):
"""Write `data` to all the files."""
for f in self._files:
f.write(data)
Then you can create a Tee object:
tee = Tee(sys.stdout, file)
and write to your tee:
tee.write(data)
and the output will go to the file and to stdout.
That's as good as you're going to get1 -- Of course, if you want it in one line, you could wrap it up in a function . . .
def print_and_write(fileobj, data):
print data
fileobj.write(data)
1For general purpose code. For the specific case of logging, you might do a little better using the logging module as others have pointed out in the comments
If your use case is logging in any form, then use a logger with multiple destinations.
In your case, a StreamHandler directed to sys.stdout, and a FileHandler for logging to a file:
import logging
import sys
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(logging.FileHandler('out.log')) # log to file
logger.addHandler(logging.StreamHandler(sys.stdout)) # log to stdout
logger.info('foo bar')
Requires a bit of setup, but after that this solution is as clean as it gets, IMHO. And thread-safe.
Try this
import sys
outfile = open("test.txt",'w')
[print("foo bar",file = i) for i in [outfile,sys.stdout]]
outfile.close()
I have a Python script that makes use of 'Print' for printing to stdout. I've recently added logging via Python Logger and would like to make it so these print statements go to logger if logging is enabled. I do not want to modify or remove these print statements.
I can log by doing 'log.info("some info msg")'. I want to be able to do something like this:
if logging_enabled:
sys.stdout=log.info
print("test")
If logging is enabled, "test" should be logged as if I did log.info("test"). If logging isn't enabled, "test" should just be printed to the screen.
Is this possible? I know I can direct stdout to a file in a similar manner (see: redirect prints to log file)
You have two options:
Open a logfile and replace sys.stdout with it, not a function:
log = open("myprog.log", "a")
sys.stdout = log
>>> print("Hello")
>>> # nothing is printed because it goes to the log file instead.
Replace print with your log function:
# If you're using python 2.x, uncomment the next line
#from __future__ import print_function
print = log.info
>>> print("Hello!")
>>> # nothing is printed because log.info is called instead of print
Of course, you can both print to the standard output and append to a log file, like this:
# Uncomment the line below for python 2.x
#from __future__ import print_function
import logging
logging.basicConfig(level=logging.INFO, format='%(message)s')
logger = logging.getLogger()
logger.addHandler(logging.FileHandler('test.log', 'a'))
print = logger.info
print('yo!')
One more method is to wrap the logger in an object that translates calls to write to the logger's log method.
Ferry Boender does just this, provided under the GPL license in a post on his website. The code below is based on this but solves two issues with the original:
The class doesn't implement the flush method which is called when the program exits.
The class doesn't buffer the writes on newline as io.TextIOWrapper objects are supposed to which results in newlines at odd points.
import logging
import sys
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, log_level=logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
temp_linebuf = self.linebuf + buf
self.linebuf = ''
for line in temp_linebuf.splitlines(True):
# From the io.TextIOWrapper docs:
# On output, if newline is None, any '\n' characters written
# are translated to the system default line separator.
# By default sys.stdout.write() expects '\n' newlines and then
# translates them so this is still cross platform.
if line[-1] == '\n':
self.logger.log(self.log_level, line.rstrip())
else:
self.linebuf += line
def flush(self):
if self.linebuf != '':
self.logger.log(self.log_level, self.linebuf.rstrip())
self.linebuf = ''
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename="out.log",
filemode='a'
)
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
This allows you to easily route all output to a logger of your choice. If needed, you can save sys.stdout and/or sys.stderr as mentioned by others in this thread before replacing it if you need to restore it later.
A much simpler option,
import logging, sys
logging.basicConfig(filename='path/to/logfile', level=logging.DEBUG)
logger = logging.getLogger()
sys.stderr.write = logger.error
sys.stdout.write = logger.info
Once your defined your logger, use this to make print redirect to logger even with mutiple parameters of print.
print = lambda *tup : logger.info(str(" ".join([str(x) for x in tup])))
You really should do that the other way: by adjusting your logging configuration to use print statements or something else, depending on the settings. Do not overwrite print behaviour, as some of the settings that may be introduced in the future (eg. by you or by someone else using your module) may actually output it to the stdout and you will have problems.
There is a handler that is supposed to redirect your log messages to proper stream (file, stdout or anything else file-like). It is called StreamHandler and it is bundled with logging module.
So basically in my opinion you should do, what you stated you don't want to do: replace print statements with actual logging.
Below snipped works perfectly inside my PySpark code. If someone need in case for understanding -->
import os
import sys
import logging
import logging.handlers
log = logging.getLogger(__name_)
handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error
sys.stdout.write = log.info
(will log every error in "spam.log" in the same directory, nothing will be on console/stdout)
(will log every info in "spam.log" in the same directory,nothing will be on console/stdout)
to print output error/info in both file as well as in console remove above two line.
Happy Coding
Cheers!!!