We Keep Coding

Why I can't use different log levels in Python?

I want to use different log levels. For example I need to log actions as INFO if everyting is OK, and WARNING when the required action is NOT proceeded OK (or ERROR, FATAl etc. it doesn't matter right now). How can I reach it? Basically I need just 2 levels, but unfortunately if I am trying to use 2 levels then my log file created as empty. Below is my code but it doesn't work and log file created empty.
class Logger:
#staticmethod
def info_logger(logLevel=log.INFO):
logger_name = inspect.stack()[1][3]
logger = log.getLogger(logger_name)
logger.setLevel(logLevel)
fh = log.FileHandler(log_file)
formatter = log.Formatter("%(asctime)s - %(levelname)s - %(message)s", datefmt='%m/%d/%Y %I: %M: %S %p')
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
#staticmethod
def warning_logger(logLevel=log.WARNING):
logger_name = inspect.stack()[1][3]
logger = log.getLogger(logger_name)
logger.setLevel(logLevel)
fh = log.FileHandler(error_log_file)
formatter = log.Formatter("%(asctime)s - %(levelname)s - %(message)s", datefmt='%m/%d/%Y %I: %M: %S %p')
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger

Logging printing twice only in some parts of the code

I have been using a custom formatter for logging to the terminal in my code. Lately I have been changing stuff in the code an I can't find why now in some parts of the code the log is printed twice.
This is the code for the custom formatter:
import logging
class MyFormatter(logging.Formatter):
debug_format = "[%(levelname)s] (%(module)s::%(funcName)s::%(lineno)d) %(message)s"
normal_format = "[%(levelname)s] %(message)s"
blue = "\x1b[36;21m"
grey = "\x1b[38;21m"
yellow = "\x1b[33;21m"
red = "\x1b[31;21m"
bold_red = "\x1b[31;1m"
reset = "\x1b[0m"
def __init__(self):
super().__init__(fmt="%(levelno)d: %(msg)s", datefmt=None, style="%")
def format(self, record):
# Save the original format configured by the user
# when the logger formatter was instantiated
format_orig = self._style._fmt
# Replace the original format with one customized by logging level
if record.levelno == logging.DEBUG:
self._style._fmt = MyFormatter.debug_format
format = MyFormatter.debug_format
else:
self._style._fmt = MyFormatter.normal_format
format = MyFormatter.normal_format
self.FORMATS = {
logging.DEBUG: MyFormatter.grey + format + MyFormatter.reset,
logging.INFO: MyFormatter.blue + format + MyFormatter.reset,
logging.WARNING: MyFormatter.yellow + format + MyFormatter.reset,
logging.ERROR: MyFormatter.red + format + MyFormatter.reset,
logging.CRITICAL: MyFormatter.bold_red + format + MyFormatter.reset,
}
log_fmt = self.FORMATS.get(record.levelno)
# Restore the original format configured by the user
self._style._fmt = format_orig
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
This is how I create my logger:
from src.logs import set_logger, logging
logger = set_logger(__name__, logging.DEBUG)
This is set_logger function code:
import logging
from .custom_formatter import MyFormatter
def set_logger(module_name: str, level=logging.DEBUG) -> logging.Logger:
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler()
formatter = MyFormatter()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
return logger
Now when I call this logger from main for example or at the top of a module which is imported, there is no problem, and it logs perfectly only once. However when calling the logger from inside a function in the same module it is printed twice.
I have notice by debugging that what is doing is going to the end of the format method in MyFormatter class and then it returns again to this format method, I have no clue what is going on here. Do you have any ideas on what could be happening?
PD: Also if I also call a print when the logger prints twice I only get one print, so that code runs only once for sure.
Thanks for your time!
Andrés

In set_logger(), it calls addHandler() but the logger (or an ancestor logger) will already have a handler, which you're not removing, so you'll have multiple handlers.
Have a look at the docs for Logger.propagate: https://docs.python.org/3/library/logging.html#logging.Logger.propagate

python tqdm and logging

I'm having a python script for accessing/manipulating network devices with netmiko. To speed up the processing, I'm using now tqdm and a process pool for multiprocessing.
This is working fine except the logging stuff. The logger works fine for the main, but not for the processes created by tqdm.
The important code snippets are as follows:
def threaded_function(params):
"""the custom function
:param kwargs params: device details packed as keyword arguments
:returns: dictionary containing the results of each step
"""
logging.debug('TEST')
logging.info('TEST2')
logging.error('TEST3')
def main_loop():
logging.getLogger("netmiko").propagate = False
logging.getLogger("netmiko").disabled = True
logging.getLogger("paramiko").propagate = False
logging.getLogger("paramiko").disabled = True
log_time = datetime.now().strftime("%Y%m%d-%H%M%S")
LOG_FILE = os.path.basename(input_file) + '.txt'
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
if args.log2file:
log_format_file = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
rfh = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=1048576, backupCount=10)
rfh.setLevel(args.filelevel)
rfh.setFormatter(log_format_file)
logger.addHandler(rfh)
if args.log2con:
log_format_con = logging.Formatter('%(message)s')
ch = logging.StreamHandler()
ch.setLevel(args.conlevel)
ch.setFormatter(log_format_con)
logger.addHandler(ch)
if not args.log2file and not args.log2con:
logger.disabled = True
logger.propagate = False
logging.info("JOB STARTED: " + str(log_time))
pool = multiprocessing.Pool(max_cpu_count)
with tqdm(total=len(input_list), ascii=True, unit="dev", desc='processing',
disable=(False if args.log2con else True)) as t:
for _ in pool.imap_unordered(threaded_function, input_list):
t.update(1)
results.append(_)
if __name__ == "__main__":
main_loop()
Does anyone has an idea, where I'am wrong? I tried to pass a queue according the 2nd example from this doc (https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes) but no luck..
any help is appreciated
regs nick

Python Logging. Messages are with the same date

I get log messages with the same date when I print them to the console (or logfile). But the time-out between messages is two seconds. Here is my code
folder = "logs"
log_name = {}.log
file_name = os.path.join(folder, log_name)
date_format = "%Y-%m-%d_%H:%M:%S"
name_format = "[%(asctime)s] [%(levelname)s] [%(filename)s:%(lineno)s] - %(message)s"
log = logging.getLogger('')
log.setLevel(logging.DEBUG)
format = logging.Formatter(name_format, datetime.now().strftime(date_format))
console_handler = logging.StreamHandler(sys.stderr)
file_handler = handlers.RotatingFileHandler(filename=datetime.now().strftime(file_name.format(date_format)),
maxBytes=(1048576*5),
backupCount=7)
console_handler.setFormatter(format)
file_handler.setFormatter(format)
log.addHandler(console_handler)
log.addHandler(file_handler)
from time import sleep
log.info("1")
sleep(2)
log.info("2")
sleep(2)
log.info("3")
Here is output:
[2017-07-08_17:20:51] [INFO] [logs.py:112] - 1
[2017-07-08_17:20:51] [INFO] [logs.py:114] - 2
[2017-07-08_17:20:51] [INFO] [logs.py:116] - 3

have a look at the documentation of logging.Formatter(fmt=None, datefmt=None, style='%'). the second argument you need to pass is a datefmt ("%Y-%m-%d_%H:%M:%S" in your case). the logger will do the fmt.strftime(...) for you.
you are passing a string that represents datetime.now() in this format. as this is a str (e.g. '2017-07-08_17:20:51') the formatter does not complain but always prints this exact date: '2017-07-08_17:20:51'.strftime(...) will result in '2017-07-08_17:20:51' - there are no format specifiers to fill in.
what you should do is this:
fmt = logging.Formatter(name_format, date_format)
# instead of
# format = logging.Formatter(name_format, datetime.now().strftime(date_format))
(btw: format is a built-in; renamed your formatter to fmt such that the built-in is not overwritten).

How to display date format using Python logging module

I am trying to setup a format for logging in python:
import logging,logging.handlers
FORMAT = "%(asctime)-15s %(message)s"
logging.basicConfig(format=FORMAT,level=logging.INFO)
logger = logging.getLogger("twitter")
handler = logging.handlers.RotatingFileHandler('/var/log/twitter_search/message.log', maxBytes=1024000, backupCount=5)
logger.addHandler(handler)
Basically, logging works, but without the date format...

You can add the datefmt parameter to basicConfig:
logging.basicConfig(format=FORMAT,level=logging.INFO,datefmt='%Y-%m-%d %H:%M:%S')
Or, to set the format for the Rotating FileHandler:
fmt = logging.Formatter(FORMAT,datefmt='%Y-%m-%d')
handler.setFormatter(fmt)

Basic example:
import logging
logging.basicConfig(
format='%(asctime)s %(levelname)s %(message)s',
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S'
)
logging.info('Just a random string...')
# 2030-01-01 00:00:00 INFO Just a random string...
If you want to adjust the line spacing between levelname and message change the %(levelname)s like the example:
... %(levelname)-10s ...
# 2030-01-01 00:00:00 INFO Just a random string...