We Keep Coding

Python Logging with timezone in

I have a Python app running on a server that is not in the the local timezone.
Everything works fine but when it runs on the server it's providing the log timestamp with a different timezone.
Ideally I would want it to show in local time which should be GMT.
What I initially had is:
logging.basicConfig(
handlers=[RotatingFileHandler(file_loc + "\\" + logfile_nme, maxBytes=500000, backupCount=1)],
level=logging.INFO,
format='%(asctime)s %(levelname)s %(message)s',
datefmt='%d/%m/%Y %H:%M:%S')
I've tried various online methods of either:
logging.Formatter.converter = time.gmtime
Also I tried this:
def formatTime(self, record, datefmt=None):
ct_local = time.localtime(record.created)
ct_gmt = time.gmtime(record.created)
if datefmt:
s = time.strftime(datefmt, ct_local) + '\t[' + time.strftime(datefmt, ct_gmt) + ']'
else:
t_local = time.strftime(self.default_time_format, ct_local)
t_gmt = time.strftime(self.default_time_format, ct_gmt)
s = self.default_msec_format % (t_local, record.msecs) + '\t[' + self.default_msec_format % (t_gmt, record.msecs) + ']'
return s
But wasn't sure how to apply that to logging as a formatter...
Would appreciate any input.
Thanks,
Shane

Maybe you could benefit of the time-zone in the date/time format
'%d/%m/%Y %H:%M:%S%z'
This will output
... 15:41:10+0100 INFO ....

Logging printing twice only in some parts of the code

I have been using a custom formatter for logging to the terminal in my code. Lately I have been changing stuff in the code an I can't find why now in some parts of the code the log is printed twice.
This is the code for the custom formatter:
import logging
class MyFormatter(logging.Formatter):
debug_format = "[%(levelname)s] (%(module)s::%(funcName)s::%(lineno)d) %(message)s"
normal_format = "[%(levelname)s] %(message)s"
blue = "\x1b[36;21m"
grey = "\x1b[38;21m"
yellow = "\x1b[33;21m"
red = "\x1b[31;21m"
bold_red = "\x1b[31;1m"
reset = "\x1b[0m"
def __init__(self):
super().__init__(fmt="%(levelno)d: %(msg)s", datefmt=None, style="%")
def format(self, record):
# Save the original format configured by the user
# when the logger formatter was instantiated
format_orig = self._style._fmt
# Replace the original format with one customized by logging level
if record.levelno == logging.DEBUG:
self._style._fmt = MyFormatter.debug_format
format = MyFormatter.debug_format
else:
self._style._fmt = MyFormatter.normal_format
format = MyFormatter.normal_format
self.FORMATS = {
logging.DEBUG: MyFormatter.grey + format + MyFormatter.reset,
logging.INFO: MyFormatter.blue + format + MyFormatter.reset,
logging.WARNING: MyFormatter.yellow + format + MyFormatter.reset,
logging.ERROR: MyFormatter.red + format + MyFormatter.reset,
logging.CRITICAL: MyFormatter.bold_red + format + MyFormatter.reset,
}
log_fmt = self.FORMATS.get(record.levelno)
# Restore the original format configured by the user
self._style._fmt = format_orig
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
This is how I create my logger:
from src.logs import set_logger, logging
logger = set_logger(__name__, logging.DEBUG)
This is set_logger function code:
import logging
from .custom_formatter import MyFormatter
def set_logger(module_name: str, level=logging.DEBUG) -> logging.Logger:
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler()
formatter = MyFormatter()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
return logger
Now when I call this logger from main for example or at the top of a module which is imported, there is no problem, and it logs perfectly only once. However when calling the logger from inside a function in the same module it is printed twice.
I have notice by debugging that what is doing is going to the end of the format method in MyFormatter class and then it returns again to this format method, I have no clue what is going on here. Do you have any ideas on what could be happening?
PD: Also if I also call a print when the logger prints twice I only get one print, so that code runs only once for sure.
Thanks for your time!
Andrés

In set_logger(), it calls addHandler() but the logger (or an ancestor logger) will already have a handler, which you're not removing, so you'll have multiple handlers.
Have a look at the docs for Logger.propagate: https://docs.python.org/3/library/logging.html#logging.Logger.propagate

Python Logging. Messages are with the same date

I get log messages with the same date when I print them to the console (or logfile). But the time-out between messages is two seconds. Here is my code
folder = "logs"
log_name = {}.log
file_name = os.path.join(folder, log_name)
date_format = "%Y-%m-%d_%H:%M:%S"
name_format = "[%(asctime)s] [%(levelname)s] [%(filename)s:%(lineno)s] - %(message)s"
log = logging.getLogger('')
log.setLevel(logging.DEBUG)
format = logging.Formatter(name_format, datetime.now().strftime(date_format))
console_handler = logging.StreamHandler(sys.stderr)
file_handler = handlers.RotatingFileHandler(filename=datetime.now().strftime(file_name.format(date_format)),
maxBytes=(1048576*5),
backupCount=7)
console_handler.setFormatter(format)
file_handler.setFormatter(format)
log.addHandler(console_handler)
log.addHandler(file_handler)
from time import sleep
log.info("1")
sleep(2)
log.info("2")
sleep(2)
log.info("3")
Here is output:
[2017-07-08_17:20:51] [INFO] [logs.py:112] - 1
[2017-07-08_17:20:51] [INFO] [logs.py:114] - 2
[2017-07-08_17:20:51] [INFO] [logs.py:116] - 3

have a look at the documentation of logging.Formatter(fmt=None, datefmt=None, style='%'). the second argument you need to pass is a datefmt ("%Y-%m-%d_%H:%M:%S" in your case). the logger will do the fmt.strftime(...) for you.
you are passing a string that represents datetime.now() in this format. as this is a str (e.g. '2017-07-08_17:20:51') the formatter does not complain but always prints this exact date: '2017-07-08_17:20:51'.strftime(...) will result in '2017-07-08_17:20:51' - there are no format specifiers to fill in.
what you should do is this:
fmt = logging.Formatter(name_format, date_format)
# instead of
# format = logging.Formatter(name_format, datetime.now().strftime(date_format))
(btw: format is a built-in; renamed your formatter to fmt such that the built-in is not overwritten).

Logging Formatting Right Justify Level Name

I am trying to format the output of the logging to have the levelname on the right side of the terminal always. I currently have a script that looks like:
import logging, os, time
fn = 'FN'
start = time.time()
def getTerminalSize():
import os
env = os.environ
def ioctl_GWINSZ(fd):
try:
import fcntl, termios, struct, os
cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ,
'1234'))
except:
return
return cr
cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
if not cr:
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
cr = ioctl_GWINSZ(fd)
os.close(fd)
except:
pass
if not cr:
cr = (env.get('LINES', 25), env.get('COLUMNS', 80))
return int(cr[1]), int(cr[0])
(width, _) = getTerminalSize()
level_width = 8
message_width = width - level_width - 4
FORMAT = '%(message)-{len1:{width1}d}s [%(levelname){len2:{width2}d}s]'.format(
len1 = message_width,_
len2 = level_width,_
width1 = len(str(message_width)),_
width2 = len(str(level_width)))
logging.basicConfig(format=FORMAT, level="DEBUG")
logging.debug("Debug Message")
logging.info("Info Message")
logging.warning("Warning Message")
logging.error("Error Message")
logging.critical("Critical Message")
logging.info("Starting File: " + os.path.basename(fn) + "\n-----------------------------------------")
logging.info("\tTo read data: %s"%(time.time() - start))
The output looks like:
Debug Message [ DEBUG]
Info Message [ INFO]
Warning Message [ WARNING]
Error Message [ ERROR]
Critical Message [CRITICAL]
Starting File: Channel209.Raw32
----------------------------------------- [ INFO]
To read data: 0.281999826431 [
INFO]
I would like the output to look something like this instead and can't quite figure it out:
Debug Message [ DEBUG]
Info Message [ INFO]
Warning Message [ WARNING]
Error Message [ ERROR]
Critical Message [CRITICAL]
Starting File: Channel209.Raw32
----------------------------------------- [ INFO]
To read data: 0.281999826431 [ INFO]

As #Carpetsmoker said, to do what I truly desired required creating a new formatter class overwriting the default.
The following class worked well for this process:
import logging
import textwrap
import itertools
'''
MyFormatter class
Adapted from: https://stackoverflow.com/questions/6847862/how-to-change-the-format-of-logged-messages-temporarily-in-python
https://stackoverflow.com/questions/3096402/python-string-formatter-for-paragraphs
Authors: Vinay Sajip, unutbu
'''
class MyFormatter(logging.Formatter):
#This function overwrites logging.Formatter.format
#We conver the msg into the overall format we want to see
def format(self,record):
widths=[getTerminalSize()[0] - 12 ,10]
form='{row[0]:<{width[0]}} {row[1]:<{width[1]}}'
#Instead of formatting...rewrite message as desired here
record.msg = self.Create_Columns(form,widths,[record.msg],["[%8s]"%record.levelname])
#Return basic formatter
return super(MyFormatter,self).format(record)
def Create_Columns(self,format_str,widths,*columns):
'''
format_str describes the format of the report.
{row[i]} is replaced by data from the ith element of columns.
widths is expected to be a list of integers.
{width[i]} is replaced by the ith element of the list widths.
All the power of Python's string format spec is available for you to use
in format_str. You can use it to define fill characters, alignment, width, type, etc.
formatter takes an arbitrary number of arguments.
Every argument after format_str and widths should be a list of strings.
Each list contains the data for one column of the report.
formatter returns the report as one big string.
'''
result=[]
for row in zip(*columns):
#Create a indents for each row...
sub = []
#Loop through
for r in row:
#Expand tabs to spaces to make our lives easier
r = r.expandtabs()
#Find the leading spaces and create indend character
if r.find(" ") == 0:
i = 0
for letters in r:
if not letters == " ":
break
i += 1
sub.append(" "*i)
else:
sub.append("")
#Actually wrap and creat the string to return...stolen from internet
lines=[textwrap.wrap(elt, width=num, subsequent_indent=ind) for elt,num,ind in zip(row,widths,sub)]
for line in itertools.izip_longest(*lines,fillvalue=''):
result.append(format_str.format(width=widths,row=line))
return '\n'.join(result)
It relies on getting the terminal size in some function called getTerminalSize. I used Harco Kuppens' Method that I will not repost here.
An example driver program is as follows, where MyFormatter and getTerminalSize are located in Colorer:
import logging
import Colorer
logger = logging.getLogger()
logger_handler = logging.StreamHandler()
logger.addHandler(logger_handler)
logger_handler.setFormatter(Colorer.MyFormatter("%(message)s"))
logger.setLevel("DEBUG")
logging.debug("\t\tTHIS IS A REALY long DEBUG Message that works and wraps around great........")
logging.info(" THIS IS A REALY long INFO Message that works and wraps around great........")
logging.warning("THIS IS A REALY long WARNING Message that works and wraps around great........")
logging.error("\tTHIS IS A REALY long ERROR Message that works and wraps around great........")
logging.critical("THIS IS A REALY long CRITICAL Message that works and wraps around great........")
Where the output is (commented for readability):
# THIS IS A REALY long DEBUG Message that works and [ DEBUG]
# wraps around great........
# THIS IS A REALY long INFO Message that works and wraps around [ INFO]
# great........
# THIS IS A REALY long WARNING Message that works and wraps around [ WARNING]
# great........
# THIS IS A REALY long ERROR Message that works and wraps [ ERROR]
# around great........
# THIS IS A REALY long CRITICAL Message that works and wraps around [CRITICAL]
# great........

I modified the last lines to look like:
logging.info("Starting File: %s" % os.path.basename(fn))
logging.info("%s" % ('-' * 15))
logging.info(" To read data: %s" % (time.time() - start))
Your error was using a newline (\n) and tab (\t) character.
Or, if you must keep the newline (which seems rather odd to me), you could manually add the spaces, like so:
logging.info("Starting File: %s\n%s%s" % (
os.path.basename(fn),
('-' * 15),
' ' * (width - 15 - 12)))
Other notes
You should create a Minimal, Complete, Tested and Readable. Your code wasn't working, I needed to modify a few things just to get the example running. See your messages edit history for what I had to edit.
Since Python 3.3, there's os.get_terminal_size. If that isn't available, then doing subprocess.call(['tput cols'], shell=True) seems a whole lot simpler to me...

Create custom logging function for pythons logging module

I have been trying for some time to figure out how to create a custom function for the python logging module. My goal is, with the usual function such as logging.debug(...) a log message over several channels, such as Telegram or MQTT, to publishing. So my idea is to add extra arguments to the normal log methode. For example logging.debug ("a log", telegram=True, mqtt=False) and maybe other arguments. All I find is the inheritance of the class logging.StreamingHandler and then using the method emit, but this only passes the argument record. So how can I implement my problem in a meaningful way? Do I have a thinking error or the wrong approach?

I solved my problem by creating a interface for the logging module.
A small view on my code:
# ulogging.py
import logging
import telegram
def uloggingConfig(*args, **kwargs):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# general logging config section
fmt = kwargs.pop("fmt", "%(asctime)s %(levelname)s %(message)s")
datefmt = kwargs.pop("datefmt", "%m.%d.%Y %I:%M:%S %p")
streamHandler = logging.StreamHandler()
streamHandler.setLevel(logging.DEBUG)
formater = logging.Formatter(fmt=fmt, datefmt=datefmt)
streamHandler.setFormatter(formater)
logger.addHandler(streamHandler)
# telegram config section
telegramBot = kwargs.pop("teleBot", None)
telegramChatID = kwargs.pop("teleChatID", None)
telegramLevel = kwargs.pop("teleLevel", logging.INFO)
telegramFmt = kwargs.pop("telefmt", "%(message)s")
telegramDatefmt = kwargs.pop("teledatefmt", None)
if telegramBot is not None and telegramChatID is not None:
telegramStream = TelegramStream(telegramBot, telegramChatID)
formater = logging.Formatter(fmt=telegramFmt, datefmt=telegramDatefmt)
telegramStream.setFormatter(formater)
telegramStream.setLevel(telegramLevel)
logger.addHandler(telegramStream)
elif (telegramBot is not None and telegramChatID is None) or (telegramBot is None and telegramChatID is not None):
raise KeyError("teleBot and teleChatID have to be both given")
if kwargs:
keys = ', '.join(kwargs.keys())
raise ValueError('Unrecognised argument(s): %s' % keys)
return logger
def getLogger():
return logging.getLogger(__name__)
class TelegramStream(logging.StreamHandler):
def __init__(self, telegramBot, telegramChatID):
logging.StreamHandler.__init__(self)
self._bot = telegramBot
self._chatID = telegramChatID
def emit(self, record):
if record.levelname == "DEBUG":
self._bot.send_message(self._chatID, record.levelname + ": " + record.msg)
else:
self._bot.send_message(self._chatID, record.msg)
With the uloggingConfig() method I can now pass all settings for the different handlers (at the moment only for telegram, further handlers should follow). The uloggingConfig() method then takes over the configuration and returns a logger with which log messages can be created as usual.
A simple example:
logger = ulogging.uloggingConfig(fmt="format, datefmt="dateformat, teleBot=telegramBot, teleChatID=telegramChatID, teleLevel=logging.DEBUG)
logger.debug("A log message")