I'm using python's logging library, but I want the debug logs to have a different format than the warning and error logs. Is this possible?
ETA: I want warnings and errors to appear as:
%(levelname)s: %(message)s
but debug statements to appear as
DEBUG: (only Brian cares about this) : %(message)s
all other questions I've seen have been to change the format, but that changes for EVERYTHING.
First of all, double-check if you really need this. A log output with different record formats is prone to be rather hard to read by both humans and machines.
Maybe what you actually need is different formats for different log destinations (console vs file) which will also have different verbosity (the file will have a debug log with additional information).
Now, the way is to use a custom Formatter:
class MultiformatFormatter(logging.Formatter):
def __init__(self,<args>):
<...>
def format(self,record):
if record.levelno <= logging.DEBUG:
s=<generate string one way>
else:
s=<generate string another way>
return s
<...>
#for each handler that this should apply to
handler.setFormatter(MultiformatFormatter(<args>))
Related
TL;DR:
How can I have the stream handler to print logs with colors, and the file handler log them without the literal code characters?
Any approach is fine with me, as long I'm still using standart python libraries.
--
I have been looking how to add color to the logger when it's streamed.
Also I would like to use the standart python logging library.
I found this solution: https://stackoverflow.com/a/7995762/11037602
It served me well, the issue is that my logger logs into stream and into file. And the file gets the escape/colors characters aswell, like this:
[1;33mWARNING[1;0m - run - log
Instead of:
WARNING - run - log
My code:
def setup_logger():
logger = logging.getLogger()
#Code suggested in SO question
logging.addLevelName( logging.WARNING, "\033[1;31m%s\033[1;0m" % logging.getLevelName(logging.WARNING))
logging.addLevelName( logging.ERROR, "\033[1;41m%s\033[1;0m" % logging.getLevelName(logging.ERROR))
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(funcName)s - %(message)s')
fh = logging.FileHandler('file.log')
fh.setFormatter(formatter)
logger.addHandler(fh)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
logger.addHandler(sh)
return logger
As I said, this works, however it also records the characters into the file log.
If you click in 'show 2 more comments' on the question that I linked, user #lesnik suggested put the answer's code inside if statement, I'm assuming he meant something like this:
if sys.stderr.isatty():
logging.addLevelName( logging.WARNING, "\033[1;31m%s\033[1;0m" % logging.getLevelName(logging.WARNING))
logging.addLevelName( logging.ERROR, "\033[1;41m%s\033[1;0m" % logging.getLevelName(logging.ERROR))
EDIT: Corrected the typo (pointed out by #Brad Solomon) but just adding this if statement didn't solve my problem. The characters are still logged in file.
To sum it up, how can I have the stream handler to print logs with colors, and the file handler log them without the literal code characters? Any approach is fine with me, as long I'm still using standart python libraries.
What about using this module?
https://coloredlogs.readthedocs.io/en/latest/
I'm using Python 3.3.5 and the logging module to log information to a local file (from different threads). There are cases where I'd like to output some additional information, without knowing exactly what that information will be (e.g. it might be one single line of text or a dict).
What I'd like to do is add this additional information to my log file, after the log record has been written. Furthermore, the additional info is only necessary when the log level is error (or higher).
Ideally, it would look something like:
2014-04-08 12:24:01 - INFO - CPU load not exceeded
2014-04-08 12:24:26 - INFO - Service is running
2014-04-08 12:24:34 - ERROR - Could not find any active server processes
Additional information, might be several lines.
Dict structured information would be written as follows:
key1=value1
key2=value2
2014-04-08 12:25:16 - INFO - Database is responding
Short of writing a custom log formatter, I couldn't find much which would fit my requirements. I've read about filters and contexts, but again this doesn't seem like a good match.
Alternatively, I could just write to a file using the standard I/O, but most of the functionality already exists in the Logging module, and moreover it's thread-safe.
Any input would be greatly appreciated. If a custom log formatter is indeed necessary, any pointers on where to start would be fantastic.
Keeping in mind that many people consider a multi-line logging message a bad practice (understandably so, since if you have a log processor like DataDog or Splunk which are very prepared to handle single line logs, multi-line logs will be very hard to parse), you can play with the extra parameter and use a custom formatter to append stuff to the message that is going to be shown (take a look to the usage of 'extra' in the logging package documentation).
import logging
class CustomFilter(logging.Filter):
def filter(self, record):
if hasattr(record, 'dct') and len(record.dct) > 0:
for k, v in record.dct.iteritems():
record.msg = record.msg + '\n\t' + k + ': ' + v
return super(CustomFilter, self).filter(record)
if __name__ == "__main__":
logging.getLogger().setLevel(logging.DEBUG)
extra_logger = logging.getLogger('extra_logger')
extra_logger.setLevel(logging.INFO)
extra_logger.addFilter(CustomFilter())
logging.debug("Nothing special here... Keep walking")
extra_logger.info("This shows extra",
extra={'dct': {"foo": "bar", "baz": "loren"}})
extra_logger.debug("You shouldn't be seeing this in the output")
extra_logger.setLevel(logging.DEBUG)
extra_logger.debug("Now you should be seeing it!")
That code outputs:
DEBUG:root:Nothing special here... Keep walking
INFO:extra_logger:This shows extra
foo: bar
baz: loren
DEBUG:extra_logger:Now you should be seeing it!
I still recommend calling the super's filter function in your custom filter, mainly because that's the function that decides whether showing the message or not (for instance, if your logger's level is set to logging.INFO, and you log something using extra_logger.debug, that message shouldn't be seen, as shown in the example above)
I just add \n symbols to the output text.
i'm using a simple line splitter in my smaller applications:
for line in logmessage.splitlines():
writemessage = logtime + " - " + line + "\n"
logging.info(str(writemessage))
Note that this is not thread-safe and should probably only be used in log-volume logging applications.
However you can output to log almost anything, as it will preserve your formatting. I have used it for example to output JSON API responses formatted using: json.dumps(parsed, indent=4, sort_keys=True)
It seems that I made a small typo when defining my LogFormatter string: by accidentally escaping the newline character, I wrongly assumed that writing multi-line output to a log file was not possible.
Cheers to #Barafu for pointing this out (which is why I assigned him the correct answer).
Here's the sample code:
import logging
lf = logging.Formatter('%(levelname)-8s - %(message)s\n%(detail)s')
lh = logging.FileHandler(filename=r'c:\temp\test.log')
lh.setFormatter(lf)
log = logging.getLogger()
log.setLevel(logging.DEBUG)
log.addHandler(lh)
log.debug('test', extra={'detail': 'This is a multi-line\ncomment to test the formatter'})
The resulting output would look like this:
DEBUG - test
This is a multi-line
comment to test the formatter
Caveat:
If there is no detail information to log, and you pass an empty string, the logger will still output a newline. Thus, the remaining question is: how can we make this conditional?
One approach would be to update the logging formatter before actually logging the information, as described here.
I cannot get the logging module to write to a file for the life of me and I have no idea what is the problem.
I am runnning
form = "%(asctime)s - %(levelname)s - %(message)s"
logging.basicConfig(logfile='/home/gabriel/Developement/cl/cl.log',level=logging.DEBUG, format=form)
logging.debug("oh")
logging.info("oh!")
logging.warning("OH!")
logging.error("OH NO!")
I'm completely unsure of what's going on, the file is not created, nor is it written to. However, python does not raise an exception. I've tried running with python 2.7.4 and ipython. Please let me know what diagnostic steps I can take, I wish I could provide more information but I do not know what is relevant...
Change logfile to filename, like so:
logging.basicConfig(filename='/home/gabriel/Developement/cl/cl.log', level=logging.DEBUG, format=form)
You can see the keyword arguments taken by basicConfig here.
I want to output some strings to a log file and I want the log file to be continuously updated.
I have looked into the logging module pf python and found out that it is
mostly about formatting and concurrent access.
Please let me know if I am missing something or amy other way of doing it
usually i do the following:
# logging
LOG = "/tmp/ccd.log"
logging.basicConfig(filename=LOG, filemode="w", level=logging.DEBUG)
# console handler
console = logging.StreamHandler()
console.setLevel(logging.ERROR)
logging.getLogger("").addHandler(console)
The logging part initialises logging's basic configurations. In the following I set up a console handler that prints out some logging information separately. Usually my console output is set to output only errors (logging.ERROR) and the detailed output in the LOG file.
Your loggings will now printed to file. For instance using:
logger = logging.getLogger(__name__)
logger.debug("hiho debug message")
or even
logging.debug("next line")
should work.
Doug Hellmann has a nice guide.
To add my 10cents with regards to using logging. I've only recently discovered the Logging module and was put off at first. Maybe just because it initially looks like a lot of work, but it's really simple and incredibly handy.
This is the set up that I use. Similar to Mkinds answer, but includes a timestamp.
# Set up logging
log = "bot.log"
logging.basicConfig(filename=log,level=logging.DEBUG,format='%(asctime)s %(message)s', datefmt='%d/%m/%Y %H:%M:%S')
logging.info('Log Entry Here.')
Which will produce something like:
22/09/2015 14:39:34 Log Entry Here.
You can log to a file with the Logging API.
Example: http://docs.python.org/2/howto/logging.html#logging-to-a-file
I use the Python logging framework with default settings.
For some data compare reason:I have to compare the log with other data output.
But the python log begin with a default, something like:
INFO:root:post params in transmitter
Can I set the python log output without INFO:root:, like:
post params in transmitter
with my own log only?
Thx a lot!
Sure thing. You could set the format to watever you like:
format: '%(message)s'
Like this:
logging.basicConfig(format='%(message)s', ...)
See the doc for more info: http://docs.python.org/library/logging.config.html
Those "INFO:..." or "DEBUG:..." appear there because some handler defines that. My guess: the default handler is still there.
You can check it by taking a peek at logger.handlers right after you created it.
logger = logging.getLogger()
logger.handlers = [] # This is the key thing for the question!
# Start defining and assigning your handlers here
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s: %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
Also, you could just override the format for that default handler:
if (len(logger.handlers) > 0): # Check here whatever. Or try/except. You get the idea...
formatter = logging.Formatter("%(asctime)s: %(levelname)s - %(message)s")
logger.handlers[0].setFormatter(formatter)
I am not a Python expert so maybe there is a better way to remove or even not create that default handler but this works pretty well for me.
Note: As stated in the docs, the .basicConfig is useful for simple loggers. If you have multiple streams, with multiple formats, it does not work as far as I know and you need to go with the custom handlers.
endDate = '2015-07-24'
logging.basicConfig(filename="win" + strftime("%Y%m%d", localtime()) + ".txt", level=logging.DEBUG,format='%(message)s')
Use a Formatter.
Your really dont need to get into removing INFO word.. . (it shall really help you when your code would more messy and you would be using more stuff than just info like debugging exception etc)
If you want to compare you data with that data you can do something like skipping first 10 character (INFO:ROOT:) and then do whatever you feel like. Umm something like this:
f = open("my.log","a+")
lines = f.readlines()
for line in lines:
if(line[10:] == my_output):
do_whatever_you_feel_like()