We Keep Coding

How to properly test logger with unittest?

I have a function that sets up logger for my module:
import logging
...
LOGGER = logging.getLogger(__name__)
def setup_logger(level=logging.INFO):
formatter = logging.Formatter(
fmt='%(asctime)s %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
handler = logging.StreamHandler()
handler.setLevel(level)
handler.setFormatter(formatter)
LOGGER.setLevel(level)
LOGGER.addHandler(handler)
LOGGER.debug('Logger is set up...')
Now I want to test my code that for example my logger is indeed fires that debug message. I try the following:
def test_setup_logger(self):
with self.assertLogs(LOGGER, level='DEBUG') as cm:
setup_logger(level=10)
self.assertListEqual(cm.output, [2021-11-21 20:39:23 DEBUG: Logger is set up...])
And then I get this message:
AssertionError: Lists differ: ['DEBUG:app.main:Logger is set up...'] != ['2021-11-21 20:39:23 DEBUG: Logger is set up...']
Which apparently means that my LOGGER formatting was not set up. How do I test this properly? Thanks in advance.

First of all your test will never successful because you set static datetime(as str).
cm.output includes printed messages using format {LOG_LEVEL}:{LOGGER_NAME}:{MESSAGE}. Yes, you can check full format using cm.records + handler.formatter. Something like:
class TestExample(unittest.TestCase):
def test_setup_logger(self):
with self.assertLogs(LOGGER) as cm:
now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
setup_logger(logging.DEBUG)
# Ran 1 test in 0.002s OK...
self.assertEqual(
LOGGER.handlers[1].format(cm.records[0]),
'{} DEBUG: Logger is set up...'.format(now)
)
But... I don't understand what it is for. It looks like you want to write a test for the logging package. Usually you need to check log records(cm.output) after important functionality.
Also you can create a test for the logger configuration. And check only required log records in the other tests. Just an example:
class TestLoggerSetup(unittest.TestCase):
def test_setup_logger(self):
setup_logger(logging.DEBUG)
self.assertTrue(isinstance(LOGGER.handlers[0], logging.StreamHandler))
self.assertEqual(LOGGER.handlers[0].formatter.datefmt, '%Y-%m-%d %H:%M:%S')
self.assertEqual(LOGGER.level, logging.DEBUG)
class TestLogMessages(unittest.TestCase):
def test_my_function(self):
with self.assertLogs(LOGGER) as cm:
LOGGER.info(111)
# or my_function() blablabla... and check only log records:
self.assertListEqual(cm.output, ['INFO:main:111'])

Issue while wrapping logger in custom class

I'm new to python and trying to create wrapper over logging to reuse changes needed to modify formatting etc.
I've written my wrapper class in following way -
import logging
import sys
from datetime import datetime
class CustomLogger:
"""This is custom logger class"""
_format_spec = f"[%(name)-24s | %(asctime)s | %(levelname)s ] (%(filename)-32s : %(lineno)-4d) ==> %(message)s"
_date_format_spec = f"%Y-%m-%d # %I:%M:%S %p"
def __init__(self, name, level=logging.DEBUG, format_spec=None):
""""""
self.name = name
self.level = level
self.format_spec = format_spec if format_spec else CustomLogger._format_spec
# Complete logging configuration.
self.logger = self.get_logger(self.name, self.level)
def get_file_handler(self, name, level):
"""This is a method to get a file handler"""
today = datetime.now().strftime(format="%Y-%m-%d")
file_handler = logging.FileHandler("{}-{}.log".format(name, today))
file_handler.setLevel(level)
file_handler.setFormatter(logging.Formatter(self.format_spec,
datefmt=CustomLogger._date_format_spec))
return file_handler
def get_stream_handler(self, level):
"""This is a method to get a stream handler"""
stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(level)
stream_handler.setFormatter(logging.Formatter(self.format_spec,
datefmt=CustomLogger._date_format_spec))
return stream_handler
def get_logger(self, name, level):
"""This is a method to get a logger"""
logger = logging.getLogger(name)
logger.addHandler(self.get_file_handler(name, level))
# logger.addHandler(self.get_stream_handler(level))
return logger
def info(self, msg):
"""info message logger method"""
self.logger.info(msg)
def error(self, msg):
"""error message logger method"""
self.logger.error(msg)
def debug(self, msg):
"""debug message logger method"""
self.logger.debug(msg)
def warn(self, msg):
"""warning message logger method"""
self.logger.warn(msg)
def critical(self, msg):
"""critical message logger method"""
self.logger.critical(msg)
def exception(self, msg):
"""exception message logger method"""
self.logger.exception(msg)
But when I try to use my CustomLogger, nothing goes into the log file.
def main():
"""This main function"""
logger = CustomLogger(name="debug", level=logging.DEBUG)
logger.info("Called main")
if __name__ == "__main__":
main()
If I do similar thing without class/function wrapper, it works. Not sure where I'm going wrong. Any pointer will help.
Further update on the question
After making this (custom_logger.py) as a package and using in actual application (app.py), I'm noticing it always prints custom_logger.py as filename but not app.py.
How to fix this? I'm ok with rewriting the CustomLogger class if required.

I missed to do setLevel() for the logger. After doing that, problem is resolved. I also added pid for the file handler file-name to avoid any future issue with multi-process env.
Let me know if there's anything I can do better here wrt any other potential issues.

Is it possible to log into multiple log files from a single module in python 3.0

Is it possible to log into multiple log files from a single module in python 3.0, where the logs file are named based on some request parameter while using flask framework.
below code works fine if i run it like a single module and import but when i run using flask, it writes in the first attempt but later falls back to logging to the root logger.
I want a logging factory in flask that can check if the logger is already present and if already present than log in the same file and if not then create a new logger and log to the new file.
Any help is greatly appreciated.
def setup_logger( name, log_file, level=logging.INFO):
my_file = Path(log_file)
print(my_file)
if my_file.is_file():
print("handler details")
print(logging.getLogger(name).hasHandlers())
print(type(logging.getLogger(name).hasHandlers()))
if logging.getLogger(name).hasHandlers():
print("old logger and it has handler")
logger.propagate = False
return logging.getLogger(name)
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
logger.propagate = False
print("old logger that has no handler")
return logger
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
print("new logger with new handler")
logger.propagate = False
return logger

Well I was able to achieve the required functionality with the following method:
def setup_logger( name, log_file, level=logging.DEBUG):
my_file = Path(log_file)
# print("check the if condition for the file")
# print(my_file.is_file())
if my_file.is_file():
#print(logging.getLogger(name).hasHandlers())
# if logging.getLogger(name).hasHandlers():
if len(logging.getLogger(name).handlers)>0:
return logging.getLogger(name)
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
logger.propagate = False
return logger
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
logger.propagate = False
return logger

print statements are not coming on console with logging module

I am using below logging code in unittest module framework
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler(output_dir, "w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
Problem is that when I am having print statements in the code, these are not getting printed anywhere. Not on the console and not in the output file (though I understand why these are not getting printed in file).
Can anybody tell me the reason and solution to print print statements on the console
I am using python 2.7

You aren't setting StreamHandler anywhere, modified your example below.
import logging
output_dir = "/tmp/somefile"
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler(output_dir, "w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# HERE
ch = logging.StreamHandler()
logger.addHandler(ch)
logger.info("test")
logger.info("test")
logger.info("test")
logger.info("test")
This gives me both output on console as well as in file.
EDIT:
Not the best idea, agree, but taken straight from documentation
import logging
import unittest
logger = None
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
logger.info(self)
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
logger.info(self)
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
logger.info(self)
def setup_log():
global logger
output_dir = "/tmp/somefile"
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler(output_dir, "w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
ch = logging.StreamHandler()
logger.addHandler(ch)
logger.info("Logger ready")
if __name__ == '__main__':
setup_log()
unittest.main()
It works, so probably you have some other problem in your code, if you update with more elaborate version, perhaps we can help.

Storing logger messages in a string

I wanted to store all the intermediate log messages (warn, info, error) to a string in Python, and report those log messages to the console at the end of program.
I tried to follow the steps outlined in
http://opensourcehacker.com/2011/02/23/temporarily-capturing-python-logging-output-to-a-string-buffer/
but was unsuccessful .
Could somebody tell me a short, clean way to do this?
This is what I've tried for now:
log = logging.getLogger('basic_logger')
log.setLevel(logging.DEBUG)
report = ""
memory_handler = logging.handlers.MemoryHandler(1024*20, logging.ERROR, report)
memory_handler.setLevel(logging.DEBUG)
log.addHandler(memory_handler)
log.info("hello world")
memory_handler.flush()
print "report:", report

It can be as simple as logging to a StringIO object:
import logging
try:
from cStringIO import StringIO # Python 2
except ImportError:
from io import StringIO
log_stream = StringIO()
logging.basicConfig(stream=log_stream, level=logging.INFO)
logging.info('hello world')
logging.warning('be careful!')
logging.debug("you won't see this")
logging.error('you will see this')
logging.critical('critical is logged too!')
print(log_stream.getvalue())
Output
INFO:root:hello world
WARNING:root:be careful!
ERROR:root:you will see this
CRITICAL:root:critical is logged too!
If you want to log only those messages at levels WARN, INFO and ERROR you can do it with a filter. LevelFilter below checks each log record's level no, allowing only those records of the desired level(s):
import logging
try:
from cStringIO import StringIO # Python 2
except ImportError:
from io import StringIO
class LevelFilter(logging.Filter):
def __init__(self, levels):
self.levels = levels
def filter(self, record):
return record.levelno in self.levels
log_stream = StringIO()
logging.basicConfig(stream=log_stream, level=logging.NOTSET)
logging.getLogger().addFilter(LevelFilter((logging.INFO, logging.WARNING, logging.ERROR)))
logging.info('hello world')
logging.warning('be careful!')
logging.debug("you won't see this")
logging.error('you will see this')
logging.critical('critical is no longer logged!')
print(log_stream.getvalue())
Output
INFO:root:hello world
WARNING:root:be careful!
ERROR:root:you will see this

Note that solutions involving basicConfig set attributes of the root logger which all other loggers inherit from, this can be unwanted because libraries will also log to it. My use case is a website that calls a data processing module, and I only want to capture that module's logs specifically. This also has the advantage of allowing existing handlers that log to file and the terminal to persist:
import io, logging
from django.http import HttpResponse
log_stream = io.StringIO()
log_handler = logging.StreamHandler(log_stream)
logging.getLogger('algorithm.user_output').addHandler(log_handler)
algorithm()
return HttpResponse(f'<pre>{log_stream.getvalue()}</pre>')
In algorithm.py:
logger = logging.getLogger(__name__ + '.user_output') # 'algorithm.user_output'

You can also write your own stream class. As https://docs.python.org/2/library/logging.handlers.html says, only writeand flushare used for the streaming.
Example:
import logging
class LogStream(object):
def __init__(self):
self.logs = ''
def write(self, str):
self.logs += str
def flush(self):
pass
def __str__(self):
return self.logs
log_stream = LogStream()
logging.basicConfig(stream=log_stream, level=logging.DEBUG)
log = logging.getLogger('test')
log.debug('debugging something')
log.info('informing user')
print(log_stream)
Outputs:
DEBUG:test:debugging something
INFO:test:informing user

Quick Recipe to have multiple logger and use the StringIO as storage
Note:
This is an customized version of #mhawke Answer ---> HERE
I needed to have multiple log going each one to do its things, here is a simple script that does that.
from io import StringIO
from datetime import date
# Formatter
LOG_FORMAT = '| %(asctime)s | %(name)s-%(levelname)s: %(message)s '
FORMATTER = logging.Formatter(LOG_FORMAT)
# ------- MAIN LOGGER
main_handler = logging.StreamHandler()
main_handler.setLevel(logging.WARNING)
main_handler.setFormatter(FORMATTER)
# ------- FILE LOGGER
file_handler = logging.FileHandler(f'log_{date.strftime(date.today(), "%Y-%m-%d")}.log')
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(FORMATTER)
# ------- SECONDARY STREAMER (HOLDS ALL THE LOGS FOR RETRIEVE LATER) LOGGER
streamer = StringIO()
stream_handler = logging.StreamHandler(stream=streamer)
stream_handler.setFormatter(FORMATTER)
# Root Logger
logging.basicConfig(level=10, handlers=[main_handler, file_handler, stream_handler]) # Add handlers to Logger
_logger = logging.getLogger(__name__)
_logger.log(10, "DEBUG MESSAGE")
_logger.log(20, "INFO MESSAGE")
_logger.log(30, "WARNING MESSAGE")
_logger.log(40, "ERROR!")
_logger.log(50, "CRITICAL")
print('==='*15)
print('\nGetting All logs from StringIO')
print(streamer.getvalue())
Clearing The Logs from StringIO
In addition, I needed to clear the Data an start from 0 again. The easiest way and faster by performance is just create a new StringIO instance and attach it to the StreamHandler instance.
new_streamer = StringIO() # Creating the new instance
stream_handler.setStream(new_streamer) # here we assign it to the logger
_logger.info("New Message")
_logger.info("New Message")
_logger.info("New Message")
print(new_streamer.getvalue()) # New data
Another way is to 'clear' the Stream, but as per this other **StackOverflow Answer by #Chris Morgan is less performant.
# Python 3
streamer.truncate(0)
streamer.seek(0)
_logger.info("New Message")
_logger.info("New Message")
_logger.info("New Message")
print(streamer.getvalue())
# Python 2
streamer.truncate(0)
_logger.info("New Message")
_logger.info("New Message")
_logger.info("New Message")
print(streamer.getvalue())
Documentation
Logging
StreamHandler

Maybe this example code is enough.
In general, you should post your code so we can see what is going on.
You should also be looking at the actual Python documentation for the logging module while you are following any given tutorial.
https://docs.python.org/2/library/logging.html
The standard Python logging module can log to a file. When you are done logging, you can print the contents of that file to your shell output.
# Do some logging to a file
fname = 'mylog.log'
logging.basicConfig(filename=fname, level=logging.INFO)
logging.info('Started')
logging.info('Finished')
# Print the output
with open(fname, 'r') as f:
print f.read() # You could also store f.read() to a string

We can use StringIO object for both python2 and python3 like this:
Python 3 ::
import logging
from io import StringIO
log_stream = StringIO()
logging.basicConfig(stream=log_stream, level=logging.INFO)
logging.info('this is info')
logging.warning('this is warning!')
logging.debug('this is debug')
logging.error('this is error')
logging.critical('oh ,this is critical!')
print(log_stream.getvalue())
Similarly in Python 2::
import logging
from cStringIO import StringIO
log_stream = StringIO()
logging.basicConfig(stream=log_stream, level=logging.INFO)
logging.info('this is info')
logging.warning('this is warning!')
logging.debug('this is debug')
logging.error('this is error')
logging.critical('oh ,this is critical!')
print(log_stream.getvalue())
Output ::
INFO:root:this is info
WARNING:root:this is warning!
ERROR:root:this is error
CRITICAL:root:oh ,this is critical!