I am writing some code that will output a log to either the screen, or a file, but not both.
I thought the easiest way to do this would be to write a class:
class WriteLog:
"write to screen or to file"
def __init__(self, stdout, filename):
self.stdout = stdout
self.logfile = file(filename, 'a')
def write(self, text):
self.stdout.write(text)
self.logfile.write(text)
def close(self):
self.stdout.close()
self.logfile.close()
And then call it something like this:
output = WriteLog(sys.stdout, 'log.txt')
However, I'm not sure how to allow for switching between the two, i.e. there should be an option within the class that will set WriteLog to either use stdout, or filename. Once that option has been set I just use WriteLog without any need for if statements etc.
Any ideas? Most of the solutions I see online are trying to output to both simultaneously.
Thanks.
Maybe something like this? It uses the symbolic name 'stdout' or 'stderr' in the constructor, or a real filename. The usage of if is limited to the constructor. By the way, I think you're trying to prematurely optimize (which is the root of all evil): you're trying to save time on if's while in real life, the program will spend much more time in file I/O; making the potential waste on your if's negligible.
import sys
class WriteLog:
def __init__(self, output):
self.output = output
if output == 'stdout':
self.logfile = sys.stdout
elif output == 'stderr':
self.logfile = sys.stderr
else:
self.logfile = open(output, 'a')
def write(self, text):
self.logfile.write(text)
def close(self):
if self.output != 'stdout' and self.output != 'stderr':
self.logfile.close()
def __del__(self):
self.close()
if __name__ == '__main__':
a = WriteLog('stdout')
a.write('This goes to stdout\n')
b = WriteLog('stderr')
b.write('This goes to stderr\n')
c = WriteLog('/tmp/logfile')
c.write('This goes to /tmp/logfile\n')
I'm not an expert in it, but try to use the logging library, and maybe you can have logger with 2 handlers, one for file and one for stream and then add/remove handlers dynamically.
I like the suggestion about using the logging library. But if you want to hack out something yourself, maybe passing in the file handle is worth considering.
import sys
class WriteLog:
"write to screen or to file"
def __init__(self, output):
self.output = output
def write(self, text):
self.output.write(text)
def close(self):
self.output.close()
logger = WriteLog(file('c:/temp/log.txt','a' ))
logger.write("I write to the log file.\n")
logger.close()
sysout = WriteLog(sys.stdout)
sysout.write("I write to the screen.\n")
You can utilize the logging library to do something similar to this. The following function will set up a logging object at the INFO level.
def setup_logging(file_name, log_to_file=False, log_to_console=False ):
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# Create Console handler
if log_to_file:
console_log = logging.StreamHandler()
console_log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)-8s - %(name)-12s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
# Log file
if log_to_console:
file_log = logging.FileHandler('%s.log' % (file_name), 'a', encoding='UTF-8')
file_log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)-8s - %(name)-12s - %(message)s')
file_log.setFormatter(formatter)
logger.addHandler(file_log)
return logger
You pass it the name of your log, and where you wish to log (either to the file, the console or both). Then you can utilize this function in your code block like this:
logger = setup_logging("mylog.log", log_to_file=True, log_to_console=False)
logger.info('Message')
This example will log to a file named mylog.log (in the current directory) and have output like this:
2014-11-05 17:20:29,933 - INFO - root - Message
This function has areas for improvement (if you wish to add more functionality). Right now it logs to both the console and file at log level INFO on the .setLevel(logging.INFO) lines. This could be set dynamically if you wish.
Additionally, as it is now, you can easily add standard logging lines (logger.debug('Message'), logger.critcal('DANGER!')) without modifying a class. In these examples, the debug messages won't print (because it is set to INFO) and the critical ones will.
Related
I'm trying to log all my prints to a log file (and add a timestamp before the actual message, only in the log file), but I cannot figure out why sys.stdout.write is called twice.
import sys
from datetime import datetime
class Logger(object):
def __init__(self, filename="Default.log"):
self.stdout = sys.stdout
self.log = open(filename, "w")
def write(self, message):
self.stdout.write(message)
time_stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
self.log.write(f"{time_stamp} - {message}")
self.log.flush()
def flush(self):
pass
sys.stdout = Logger("log_file.txt")
print("Hello world !")
Output from terminal:
Hello World !
Output in log_file.txt:
2021-04-19 18:43:14.800691 - Hello world !2021-04-19 18:43:14.800735 -
What am I missing here? The write method is called again after the self.log.flush() has been called, but here message=''.
If I neglect the time_stamp variable it works like a charm, e.g calling self.log.write(message).
I could of course just check if message is empty, but I would really like to understand my problem here!
My solution
#iBug gave me the answer! I didn't know that print did work this way: one write call for the data and one write call for the end keyword which is \n as default.
My solution was to add a variable self._hidden_end to mthe Logger class that is initiated to 0 and then toggles everytime write is called to check it I should add a time stampe before writing to the log or not.
import sys
from datetime import datetime
class Logger(object):
def __init__(self, filename="Default.log"):
self.stdout = sys.stdout
self.log = open(filename, "w")
self._hidden_end = 0
def write(self, message):
self.stdout.write(message)
if not self._hidden_end:
time_stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
message = f"{time_stamp} - {message}"
self._hidden_end ^= 1
self.log.write(message)
self.log.flush()
def flush(self):
pass
sys.stdout = Logger("log_file.txt")
print("Hello world !")
Any thoughts on my solution above are welcome! :)
Your perceived results indicates that print() calls file.write twice: Once for the data and once for the "end content", which is a newline by default and can be overridden with the end keyword argument. Relevant source code lines confirm this speculation.
However, this is not something you should rely on. It's entirely implementation details, may be different for another implementation (e.g. PyPy) and may change at any time without notice, not to mention that overriding builtins is another bad practice. You should avoid this kind of practice without a compelling reason not to modify other parts of the code to use your custom logging facility.
Should you really need to do monkey-patching, it's safer to override the print() function since it has a defined, reliable interface. You can import the builtins module and assign your new print function to builtins.print, maintaining control over calls to your logging handler.
I am trying to remove some information while logging to console but then keep that information while logging to a file
Here's a basic example:
import logging
import sys
class MyFilter(logging.Filter):
def filter(self, record):
record.msg = record.msg.replace("test", "")
return True
logger = logging.getLogger("mylogger")
logger.setLevel(logging.DEBUG)
console = logging.StreamHandler(sys.stdout)
console.setLevel("INFO")
logger.addHandler(console)
logfile = logging.FileHandler("log.txt", 'w')
logfile.setLevel("ERROR")
logger.addHandler(logfile)
filt = MyFilter()
console.addFilter(filt)
logger.info("test one")
logger.error("test two")
What I'd like to see on console is
one
two
and then in the log file
test two
but it's actually just
two
I'm assuming editing the LogRecord is what's causing this.
Is there a way of achieving what I want or is what I'm trying to do not possible in this way?
I think what you need is a logging.Formatter:
class MyFormatter(logging.Formatter):
def format(self, record):
return record.msg.replace("test", "")
#...
console.setFormatter(MyFormatter())
#...
I have a Python script that makes use of 'Print' for printing to stdout. I've recently added logging via Python Logger and would like to make it so these print statements go to logger if logging is enabled. I do not want to modify or remove these print statements.
I can log by doing 'log.info("some info msg")'. I want to be able to do something like this:
if logging_enabled:
sys.stdout=log.info
print("test")
If logging is enabled, "test" should be logged as if I did log.info("test"). If logging isn't enabled, "test" should just be printed to the screen.
Is this possible? I know I can direct stdout to a file in a similar manner (see: redirect prints to log file)
You have two options:
Open a logfile and replace sys.stdout with it, not a function:
log = open("myprog.log", "a")
sys.stdout = log
>>> print("Hello")
>>> # nothing is printed because it goes to the log file instead.
Replace print with your log function:
# If you're using python 2.x, uncomment the next line
#from __future__ import print_function
print = log.info
>>> print("Hello!")
>>> # nothing is printed because log.info is called instead of print
Of course, you can both print to the standard output and append to a log file, like this:
# Uncomment the line below for python 2.x
#from __future__ import print_function
import logging
logging.basicConfig(level=logging.INFO, format='%(message)s')
logger = logging.getLogger()
logger.addHandler(logging.FileHandler('test.log', 'a'))
print = logger.info
print('yo!')
One more method is to wrap the logger in an object that translates calls to write to the logger's log method.
Ferry Boender does just this, provided under the GPL license in a post on his website. The code below is based on this but solves two issues with the original:
The class doesn't implement the flush method which is called when the program exits.
The class doesn't buffer the writes on newline as io.TextIOWrapper objects are supposed to which results in newlines at odd points.
import logging
import sys
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, log_level=logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
temp_linebuf = self.linebuf + buf
self.linebuf = ''
for line in temp_linebuf.splitlines(True):
# From the io.TextIOWrapper docs:
# On output, if newline is None, any '\n' characters written
# are translated to the system default line separator.
# By default sys.stdout.write() expects '\n' newlines and then
# translates them so this is still cross platform.
if line[-1] == '\n':
self.logger.log(self.log_level, line.rstrip())
else:
self.linebuf += line
def flush(self):
if self.linebuf != '':
self.logger.log(self.log_level, self.linebuf.rstrip())
self.linebuf = ''
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename="out.log",
filemode='a'
)
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
This allows you to easily route all output to a logger of your choice. If needed, you can save sys.stdout and/or sys.stderr as mentioned by others in this thread before replacing it if you need to restore it later.
A much simpler option,
import logging, sys
logging.basicConfig(filename='path/to/logfile', level=logging.DEBUG)
logger = logging.getLogger()
sys.stderr.write = logger.error
sys.stdout.write = logger.info
Once your defined your logger, use this to make print redirect to logger even with mutiple parameters of print.
print = lambda *tup : logger.info(str(" ".join([str(x) for x in tup])))
You really should do that the other way: by adjusting your logging configuration to use print statements or something else, depending on the settings. Do not overwrite print behaviour, as some of the settings that may be introduced in the future (eg. by you or by someone else using your module) may actually output it to the stdout and you will have problems.
There is a handler that is supposed to redirect your log messages to proper stream (file, stdout or anything else file-like). It is called StreamHandler and it is bundled with logging module.
So basically in my opinion you should do, what you stated you don't want to do: replace print statements with actual logging.
Below snipped works perfectly inside my PySpark code. If someone need in case for understanding -->
import os
import sys
import logging
import logging.handlers
log = logging.getLogger(__name_)
handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error
sys.stdout.write = log.info
(will log every error in "spam.log" in the same directory, nothing will be on console/stdout)
(will log every info in "spam.log" in the same directory,nothing will be on console/stdout)
to print output error/info in both file as well as in console remove above two line.
Happy Coding
Cheers!!!
I looked through the tutorials for the python logging class here and didnt see anything that would let me make multiple logs of different levels for the same output. In the end I would like to have three logs:
<timestamp>_DEBUG.log (debug level)
<timestamp>_INFO.log (info Level)
<timestamp>_ERROR.log (error level)
Is there a way to, in one script, generate multiple log files for the same input?
<-------------UPDATE #1-------------------------->
So in implementing #robert's suggestion, I now have a small issue, probably due to not fully understanding what is being done in his code.
Here is my code in scriptRun.py
import os
import logging
logger = logging.getLogger("exceptionsLogger")
debugLogFileHandler = logging.FileHandler("Debug.log")
errorLogFileHandler = logging.FileHandler("Error.Log")
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
errorLogFileHandler.setFormatter(formatter)
debugLogFileHandler.setFormatter(formatter)
logger.addHandler(debugLogFileHandler)
logger.addHandler(errorLogFileHandler)
class LevelFilter(logging.Filter):
def __init__(self, level):
self.level = level
def filter(self, record):
return record.levelno == self.level
debugLogFileHandler.addFilter(LevelFilter(logging.DEBUG))
errorLogFileHandler.addFilter(LevelFilter(logging.ERROR))
directory = []
for dirpath, dirnames, filenames in os.walk("path\to\scripts"):
for filename in [f for f in filenames if f.endswith(".py")]:
directory.append(os.path.join(dirpath, filename))
for entry in directory:
execfile(entry)
for lists in x:
if lists[0] == 2:
logger.error(lists[1]+" "+lists[2])
elif lists[0] == 1:
logger.debug(lists[1]+" "+lists[2])
an example of what this is running is:
import sys
def script2Test2():
print y
def script2Ttest3():
mundo="hungry"
global x
x = []
theTests = (test2, test3)
for test in theTests:
try:
test()
x.append([1,test.__name__," OK"])
except:
error = str(sys.exc_info()[1])
x.append([2,test.__name__,error])
Now to my issue: running scriptRun.py does not throw any errors when i run it, and error.log and debug.log are created, but only error.log is populated with entries.
any idea why?
<------------------------Update #2----------------------->
So I realized that nothing is being logged that is "lower" than warning. even if i remove the filters and debugLogFileHandler.setLevel(logging.DEBUG) it does not seem to matter. If I set the actual log command to logger.warning or higher, it will print to the logs. Of course once I uncomment debugLogFileHandler.addFilter(LevelFilter(logging.DEBUG)) I get no log activity in Debug.log. I;m tempted to just make my own log level, but that seems like a really bad idea, in case anyone/anything else uses this code.
<-------------------------Final UPDATE--------------------->
Well I was stupid and forgot to set the logger itself to log DEBUG level events. Since by default the logging class doesn't log anything below warning, it wasnt logging any of the debug information I send it.
Final thanks and shoutout to #Robert for the filter.
Create multiple Handlers, each for one output file (INFO.log, DEBUG.log etc.).
Add a filter to each handler that only allows the specific level.
For example:
import logging
# Set up loggers and handlers.
# ...
class LevelFilter(logging.Filter):
def __init__(self, level):
self.level = level
def filter(self, record):
return record.levelno == self.level
debugLogFileHandler.addFilter(LevelFilter(logging.DEBUG))
infoLogFileHandler.addFilter(LevelFilter(logging.INFO))
Hi
I would like to extend my logger (taken by logging.getLogger("rrcheck")) with my own methods like:
def warnpfx(...):
How to do it best?
My original wish is to have a root logger writing everything to a file and additionally named logger ("rrcheck") writing to stdout, but the latter should also have some other methods and levels. I need it to prepend some messages with "! PFXWRN" prefix (but only those going to stdout) and to leave other messages unchanged. I would like also to set logging level separately for root and for named logger.
This is my code:
class CheloExtendedLogger(logging.Logger):
"""
Custom logger class with additional levels and methods
"""
WARNPFX = logging.WARNING+1
def __init__(self, name):
logging.Logger.__init__(self, name, logging.DEBUG)
logging.addLevelName(self.WARNPFX, 'WARNING')
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
# create formatter and add it to the handlers
formatter = logging.Formatter("%(asctime)s [%(funcName)s: %(filename)s,%(lineno)d] %(message)s")
console.setFormatter(formatter)
# add the handlers to logger
self.addHandler(console)
return
def warnpfx(self, msg, *args, **kw):
self.log(self.WARNPFX, "! PFXWRN %s" % msg, *args, **kw)
logging.setLoggerClass(CheloExtendedLogger)
rrclogger = logging.getLogger("rrcheck")
rrclogger.setLevel(logging.INFO)
def test():
rrclogger.debug("DEBUG message")
rrclogger.info("INFO message")
rrclogger.warnpfx("warning with prefix")
test()
And this is an output - function and lilne number is wrong: warnpfx instead of test
2011-02-10 14:36:51,482 [test: log4.py,35] INFO message
2011-02-10 14:36:51,497 [warnpfx: log4.py,26] ! PFXWRN warning with prefix
Maybe my own logger approach is not the best one?
Which direction would you propose to go (own logger, own handler, own formatter, etc.)?
How to proceed if I would like to have yet another logger?
Unfortunatelly logging has no possibility to register an own logger, so then getLogger(name) would take a required one...
Regards,
Zbigniew
If you check Python sources, you'll see that the culprit is the Logger.findCaller method that walks through the call stack and searches for a first line that is not in the logging.py file. Because of this, your custom call to self.log in CheloExtendedLogger.warnpfx registers a wrong line.
Unfortunately, the code in logging.py is not very modular, so the fix is rather ugly: you have to redefine the findCaller method yourself in your subclass, so that it takes into account both the logging.py file and the file in which your logger resides (note that there shouldn't be any code other than the logger in your file, or again the results will be inaccurate). This requires a one-line change in the method body:
class CheloExtendedLogger(logging.Logger):
[...]
def findCaller(self):
"""
Find the stack frame of the caller so that we can note the source
file name, line number and function name.
"""
f = logging.currentframe().f_back
rv = "(unknown file)", 0, "(unknown function)"
while hasattr(f, "f_code"):
co = f.f_code
filename = os.path.normcase(co.co_filename)
if filename in (_srcfile, logging._srcfile): # This line is modified.
f = f.f_back
continue
rv = (filename, f.f_lineno, co.co_name)
break
return rv
For this to work, you need to define your own _srcfile variable in your file. Again, logging.py doesn't use a function, but rather puts all the code on the module level, so you have to copy-paste again:
if hasattr(sys, 'frozen'): #support for py2exe
_srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
elif string.lower(__file__[-4:]) in ['.pyc', '.pyo']:
_srcfile = __file__[:-4] + '.py'
else:
_srcfile = __file__
_srcfile = os.path.normcase(_srcfile)
Well, maybe if you don't care for compiled versions, two last lines will suffice.
Now, your code works as expected:
2011-02-10 16:41:48,108 [test: lg.py,16] INFO message
2011-02-10 16:41:48,171 [test: lg.py,17] ! PFXWRN warning with prefix
As for multiple logger classes, if you don't mind the dependency between a logger name and a logger class, you could make a subclass of logging.Logger that would delegate its calls to an appropriate logger class, based on what its name was. There are probably other, more elegant possibilities, but I can't think of any right now.