We Keep Coding

Microsecond do not work in Python logger format

For some reason my Python logger does not want to recognize microseconds format.
import logging, io
stream = io.StringIO()
logger = logging.getLogger("TestLogger")
logger.setLevel(logging.INFO)
logger.propagate = False
log_handler = logging.StreamHandler(stream)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s',"%Y-%m-%d %H:%M:%S.%f %Z")
log_handler.setFormatter(log_format)
logger.addHandler(log_handler)
logger.info("This is test info log")
print(stream.getvalue())
It returns:
2023-01-06 18:52:34.%f UTC - TestLogger - INFO - This is test info log
Why are microseconds missing?
Update
I am running
Python 3.10.4
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye

The issue is that the formatTime method uses time.strptime to format the current time.time(), but since struct_time has no information about milliseconds and microseconds the formatter ignores the %f.
Also, note that the LogRecord calculates the milliseconds separately and stores them in another variable named msecs
To get what you're looking for we need a custom version of the Formatter class that uses a different converter than time.localtime and is able to interpret the microseconds:
from datetime import datetime
class MyFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
if not datefmt:
return super().formatTime(record, datefmt=datefmt)
return datetime.fromtimestamp(record.created).astimezone().strftime(datefmt)
...
log_format = MyFormatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s', "%Y-%m-%d %H:%M:%S.%f %Z")
...
Should output:
2023-01-06 17:47:54.828521 EST - TestLogger - INFO - This is test info log

I found that the answer given by Ashwini Chaudhary to not be applicable for me and found a slightly simpler solution in creating a function with the formatting as done with datetime and changing the value of the logging.Formatter.formatTime method to this instead i.e.:
def _formatTime(self, record, datefmt: str = None) -> str:
return datetime.datetime.fromtimestamp(record.created).astimezone().strftime(datefmt)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s',"%Y-%m-%d %H:%M:%S.%f %Z")
logging.Formatter.formatTime = _formatTime
And then generating your logger as normal

Python Logging with timezone in

I have a Python app running on a server that is not in the the local timezone.
Everything works fine but when it runs on the server it's providing the log timestamp with a different timezone.
Ideally I would want it to show in local time which should be GMT.
What I initially had is:
logging.basicConfig(
handlers=[RotatingFileHandler(file_loc + "\\" + logfile_nme, maxBytes=500000, backupCount=1)],
level=logging.INFO,
format='%(asctime)s %(levelname)s %(message)s',
datefmt='%d/%m/%Y %H:%M:%S')
I've tried various online methods of either:
logging.Formatter.converter = time.gmtime
Also I tried this:
def formatTime(self, record, datefmt=None):
ct_local = time.localtime(record.created)
ct_gmt = time.gmtime(record.created)
if datefmt:
s = time.strftime(datefmt, ct_local) + '\t[' + time.strftime(datefmt, ct_gmt) + ']'
else:
t_local = time.strftime(self.default_time_format, ct_local)
t_gmt = time.strftime(self.default_time_format, ct_gmt)
s = self.default_msec_format % (t_local, record.msecs) + '\t[' + self.default_msec_format % (t_gmt, record.msecs) + ']'
return s
But wasn't sure how to apply that to logging as a formatter...
Would appreciate any input.
Thanks,
Shane

Maybe you could benefit of the time-zone in the date/time format
'%d/%m/%Y %H:%M:%S%z'
This will output
... 15:41:10+0100 INFO ....

Logging printing twice only in some parts of the code

I have been using a custom formatter for logging to the terminal in my code. Lately I have been changing stuff in the code an I can't find why now in some parts of the code the log is printed twice.
This is the code for the custom formatter:
import logging
class MyFormatter(logging.Formatter):
debug_format = "[%(levelname)s] (%(module)s::%(funcName)s::%(lineno)d) %(message)s"
normal_format = "[%(levelname)s] %(message)s"
blue = "\x1b[36;21m"
grey = "\x1b[38;21m"
yellow = "\x1b[33;21m"
red = "\x1b[31;21m"
bold_red = "\x1b[31;1m"
reset = "\x1b[0m"
def __init__(self):
super().__init__(fmt="%(levelno)d: %(msg)s", datefmt=None, style="%")
def format(self, record):
# Save the original format configured by the user
# when the logger formatter was instantiated
format_orig = self._style._fmt
# Replace the original format with one customized by logging level
if record.levelno == logging.DEBUG:
self._style._fmt = MyFormatter.debug_format
format = MyFormatter.debug_format
else:
self._style._fmt = MyFormatter.normal_format
format = MyFormatter.normal_format
self.FORMATS = {
logging.DEBUG: MyFormatter.grey + format + MyFormatter.reset,
logging.INFO: MyFormatter.blue + format + MyFormatter.reset,
logging.WARNING: MyFormatter.yellow + format + MyFormatter.reset,
logging.ERROR: MyFormatter.red + format + MyFormatter.reset,
logging.CRITICAL: MyFormatter.bold_red + format + MyFormatter.reset,
}
log_fmt = self.FORMATS.get(record.levelno)
# Restore the original format configured by the user
self._style._fmt = format_orig
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
This is how I create my logger:
from src.logs import set_logger, logging
logger = set_logger(__name__, logging.DEBUG)
This is set_logger function code:
import logging
from .custom_formatter import MyFormatter
def set_logger(module_name: str, level=logging.DEBUG) -> logging.Logger:
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler()
formatter = MyFormatter()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
return logger
Now when I call this logger from main for example or at the top of a module which is imported, there is no problem, and it logs perfectly only once. However when calling the logger from inside a function in the same module it is printed twice.
I have notice by debugging that what is doing is going to the end of the format method in MyFormatter class and then it returns again to this format method, I have no clue what is going on here. Do you have any ideas on what could be happening?
PD: Also if I also call a print when the logger prints twice I only get one print, so that code runs only once for sure.
Thanks for your time!
Andrés

In set_logger(), it calls addHandler() but the logger (or an ancestor logger) will already have a handler, which you're not removing, so you'll have multiple handlers.
Have a look at the docs for Logger.propagate: https://docs.python.org/3/library/logging.html#logging.Logger.propagate

Logging issues with python logging module

I am using python logging module. I am initialising file having following data
def initialize_logger(output_dir):
'''
This initialise the logger
:param output_dir:
:return:
'''
root = logging.getLogger()
root.setLevel(logging.INFO)
format = '%(asctime)s - %(levelname)-8s - %(message)s'
date_format = '%Y-%m-%d %H:%M:%S'
if 'colorlog' in sys.modules and os.isatty(2):
cformat = '%(log_color)s' + format
f = colorlog.ColoredFormatter(cformat, date_format,
log_colors={'DEBUG': 'green', 'INFO': 'green',
'WARNING': 'bold_yellow', 'ERROR': 'bold_red',
'CRITICAL': 'bold_red'})
else:
f = logging.Formatter(format, date_format)
#ch = logging.FileHandler(output_dir, "w")
ch = logging.StreamHandler()
ch.setFormatter(f)
root.addHandler(ch)
As there is only one streamHandler, But I am getting two prints on my console as
INFO:root:clearmessage:%ss1=00
2017-12-21 17:07:20 - INFO - clearmessage:%ss1=00
INFO:root:clearmessage:%ss2=00
2017-12-21 17:07:20 - INFO - clearmessage:%ss2=00
Every message is printed as Root and Info. Any idea Why I am getting two prints. In the above code you can ignore color code.

You have two handlers. Clear the handlers before you add a new one:
root.handlers = [] # clears the list
root.addHandler(ch) # adds a new handler

Python Logging. Messages are with the same date

I get log messages with the same date when I print them to the console (or logfile). But the time-out between messages is two seconds. Here is my code
folder = "logs"
log_name = {}.log
file_name = os.path.join(folder, log_name)
date_format = "%Y-%m-%d_%H:%M:%S"
name_format = "[%(asctime)s] [%(levelname)s] [%(filename)s:%(lineno)s] - %(message)s"
log = logging.getLogger('')
log.setLevel(logging.DEBUG)
format = logging.Formatter(name_format, datetime.now().strftime(date_format))
console_handler = logging.StreamHandler(sys.stderr)
file_handler = handlers.RotatingFileHandler(filename=datetime.now().strftime(file_name.format(date_format)),
maxBytes=(1048576*5),
backupCount=7)
console_handler.setFormatter(format)
file_handler.setFormatter(format)
log.addHandler(console_handler)
log.addHandler(file_handler)
from time import sleep
log.info("1")
sleep(2)
log.info("2")
sleep(2)
log.info("3")
Here is output:
[2017-07-08_17:20:51] [INFO] [logs.py:112] - 1
[2017-07-08_17:20:51] [INFO] [logs.py:114] - 2
[2017-07-08_17:20:51] [INFO] [logs.py:116] - 3

have a look at the documentation of logging.Formatter(fmt=None, datefmt=None, style='%'). the second argument you need to pass is a datefmt ("%Y-%m-%d_%H:%M:%S" in your case). the logger will do the fmt.strftime(...) for you.
you are passing a string that represents datetime.now() in this format. as this is a str (e.g. '2017-07-08_17:20:51') the formatter does not complain but always prints this exact date: '2017-07-08_17:20:51'.strftime(...) will result in '2017-07-08_17:20:51' - there are no format specifiers to fill in.
what you should do is this:
fmt = logging.Formatter(name_format, date_format)
# instead of
# format = logging.Formatter(name_format, datetime.now().strftime(date_format))
(btw: format is a built-in; renamed your formatter to fmt such that the built-in is not overwritten).