We Keep Coding

Microsecond do not work in Python logger format

For some reason my Python logger does not want to recognize microseconds format.
import logging, io
stream = io.StringIO()
logger = logging.getLogger("TestLogger")
logger.setLevel(logging.INFO)
logger.propagate = False
log_handler = logging.StreamHandler(stream)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s',"%Y-%m-%d %H:%M:%S.%f %Z")
log_handler.setFormatter(log_format)
logger.addHandler(log_handler)
logger.info("This is test info log")
print(stream.getvalue())
It returns:
2023-01-06 18:52:34.%f UTC - TestLogger - INFO - This is test info log
Why are microseconds missing?
Update
I am running
Python 3.10.4
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye

The issue is that the formatTime method uses time.strptime to format the current time.time(), but since struct_time has no information about milliseconds and microseconds the formatter ignores the %f.
Also, note that the LogRecord calculates the milliseconds separately and stores them in another variable named msecs
To get what you're looking for we need a custom version of the Formatter class that uses a different converter than time.localtime and is able to interpret the microseconds:
from datetime import datetime
class MyFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
if not datefmt:
return super().formatTime(record, datefmt=datefmt)
return datetime.fromtimestamp(record.created).astimezone().strftime(datefmt)
...
log_format = MyFormatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s', "%Y-%m-%d %H:%M:%S.%f %Z")
...
Should output:
2023-01-06 17:47:54.828521 EST - TestLogger - INFO - This is test info log

I found that the answer given by Ashwini Chaudhary to not be applicable for me and found a slightly simpler solution in creating a function with the formatting as done with datetime and changing the value of the logging.Formatter.formatTime method to this instead i.e.:
def _formatTime(self, record, datefmt: str = None) -> str:
return datetime.datetime.fromtimestamp(record.created).astimezone().strftime(datefmt)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s',"%Y-%m-%d %H:%M:%S.%f %Z")
logging.Formatter.formatTime = _formatTime
And then generating your logger as normal

Why I can't use different log levels in Python?

I want to use different log levels. For example I need to log actions as INFO if everyting is OK, and WARNING when the required action is NOT proceeded OK (or ERROR, FATAl etc. it doesn't matter right now). How can I reach it? Basically I need just 2 levels, but unfortunately if I am trying to use 2 levels then my log file created as empty. Below is my code but it doesn't work and log file created empty.
class Logger:
#staticmethod
def info_logger(logLevel=log.INFO):
logger_name = inspect.stack()[1][3]
logger = log.getLogger(logger_name)
logger.setLevel(logLevel)
fh = log.FileHandler(log_file)
formatter = log.Formatter("%(asctime)s - %(levelname)s - %(message)s", datefmt='%m/%d/%Y %I: %M: %S %p')
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
#staticmethod
def warning_logger(logLevel=log.WARNING):
logger_name = inspect.stack()[1][3]
logger = log.getLogger(logger_name)
logger.setLevel(logLevel)
fh = log.FileHandler(error_log_file)
formatter = log.Formatter("%(asctime)s - %(levelname)s - %(message)s", datefmt='%m/%d/%Y %I: %M: %S %p')
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger

Logging printing twice only in some parts of the code

I have been using a custom formatter for logging to the terminal in my code. Lately I have been changing stuff in the code an I can't find why now in some parts of the code the log is printed twice.
This is the code for the custom formatter:
import logging
class MyFormatter(logging.Formatter):
debug_format = "[%(levelname)s] (%(module)s::%(funcName)s::%(lineno)d) %(message)s"
normal_format = "[%(levelname)s] %(message)s"
blue = "\x1b[36;21m"
grey = "\x1b[38;21m"
yellow = "\x1b[33;21m"
red = "\x1b[31;21m"
bold_red = "\x1b[31;1m"
reset = "\x1b[0m"
def __init__(self):
super().__init__(fmt="%(levelno)d: %(msg)s", datefmt=None, style="%")
def format(self, record):
# Save the original format configured by the user
# when the logger formatter was instantiated
format_orig = self._style._fmt
# Replace the original format with one customized by logging level
if record.levelno == logging.DEBUG:
self._style._fmt = MyFormatter.debug_format
format = MyFormatter.debug_format
else:
self._style._fmt = MyFormatter.normal_format
format = MyFormatter.normal_format
self.FORMATS = {
logging.DEBUG: MyFormatter.grey + format + MyFormatter.reset,
logging.INFO: MyFormatter.blue + format + MyFormatter.reset,
logging.WARNING: MyFormatter.yellow + format + MyFormatter.reset,
logging.ERROR: MyFormatter.red + format + MyFormatter.reset,
logging.CRITICAL: MyFormatter.bold_red + format + MyFormatter.reset,
}
log_fmt = self.FORMATS.get(record.levelno)
# Restore the original format configured by the user
self._style._fmt = format_orig
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
This is how I create my logger:
from src.logs import set_logger, logging
logger = set_logger(__name__, logging.DEBUG)
This is set_logger function code:
import logging
from .custom_formatter import MyFormatter
def set_logger(module_name: str, level=logging.DEBUG) -> logging.Logger:
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler()
formatter = MyFormatter()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
return logger
Now when I call this logger from main for example or at the top of a module which is imported, there is no problem, and it logs perfectly only once. However when calling the logger from inside a function in the same module it is printed twice.
I have notice by debugging that what is doing is going to the end of the format method in MyFormatter class and then it returns again to this format method, I have no clue what is going on here. Do you have any ideas on what could be happening?
PD: Also if I also call a print when the logger prints twice I only get one print, so that code runs only once for sure.
Thanks for your time!
Andrés

In set_logger(), it calls addHandler() but the logger (or an ancestor logger) will already have a handler, which you're not removing, so you'll have multiple handlers.
Have a look at the docs for Logger.propagate: https://docs.python.org/3/library/logging.html#logging.Logger.propagate

Logging issues with python logging module

I am using python logging module. I am initialising file having following data
def initialize_logger(output_dir):
'''
This initialise the logger
:param output_dir:
:return:
'''
root = logging.getLogger()
root.setLevel(logging.INFO)
format = '%(asctime)s - %(levelname)-8s - %(message)s'
date_format = '%Y-%m-%d %H:%M:%S'
if 'colorlog' in sys.modules and os.isatty(2):
cformat = '%(log_color)s' + format
f = colorlog.ColoredFormatter(cformat, date_format,
log_colors={'DEBUG': 'green', 'INFO': 'green',
'WARNING': 'bold_yellow', 'ERROR': 'bold_red',
'CRITICAL': 'bold_red'})
else:
f = logging.Formatter(format, date_format)
#ch = logging.FileHandler(output_dir, "w")
ch = logging.StreamHandler()
ch.setFormatter(f)
root.addHandler(ch)
As there is only one streamHandler, But I am getting two prints on my console as
INFO:root:clearmessage:%ss1=00
2017-12-21 17:07:20 - INFO - clearmessage:%ss1=00
INFO:root:clearmessage:%ss2=00
2017-12-21 17:07:20 - INFO - clearmessage:%ss2=00
Every message is printed as Root and Info. Any idea Why I am getting two prints. In the above code you can ignore color code.

You have two handlers. Clear the handlers before you add a new one:
root.handlers = [] # clears the list
root.addHandler(ch) # adds a new handler

How to display date format using Python logging module

I am trying to setup a format for logging in python:
import logging,logging.handlers
FORMAT = "%(asctime)-15s %(message)s"
logging.basicConfig(format=FORMAT,level=logging.INFO)
logger = logging.getLogger("twitter")
handler = logging.handlers.RotatingFileHandler('/var/log/twitter_search/message.log', maxBytes=1024000, backupCount=5)
logger.addHandler(handler)
Basically, logging works, but without the date format...

You can add the datefmt parameter to basicConfig:
logging.basicConfig(format=FORMAT,level=logging.INFO,datefmt='%Y-%m-%d %H:%M:%S')
Or, to set the format for the Rotating FileHandler:
fmt = logging.Formatter(FORMAT,datefmt='%Y-%m-%d')
handler.setFormatter(fmt)

Basic example:
import logging
logging.basicConfig(
format='%(asctime)s %(levelname)s %(message)s',
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S'
)
logging.info('Just a random string...')
# 2030-01-01 00:00:00 INFO Just a random string...
If you want to adjust the line spacing between levelname and message change the %(levelname)s like the example:
... %(levelname)-10s ...
# 2030-01-01 00:00:00 INFO Just a random string...