We Keep Coding

How to reset root logger config in Python

I originally used logging.basicConfig(filename='logs/example.log') to create a log file. After reading the docs I found that it is not recommended to modify the class attributes of logging. Now that I've changed these attributes how can I change them back/reset the logger module?
Output of the code below creates two log files, app.log and example.log. The latter is an artifact of .basicConfig() being called when I first tried to set up the logger.
UPDATE:
grep -R "example.log" /lib/python3.8/ does not output anything so I'm not sure what was changed in the source code of logging to cause the example.log file to be created every time
import logging
import logging.handlers
LOG_FILENAME = 'logs/app.log'
# https://stackoverflow.com/questions/3630774/logging-remove-inspect-modify-handlers-configured-by-fileconfig
# need to run this every time since I changed class attributes of logger
# still creates example.log file
print(logging.getLogger())
root_log = logging.getLogger()
for handler in root_log.handlers:
root_log.removeHandler(handler)
# OR
# logging.basicConfig(force=True) #one way to reset root logger format
# create logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
# create console and file handler and set level to info
fh = logging.handlers.RotatingFileHandler(LOG_FILENAME, maxBytes=250000, backupCount=5)
fh.setLevel(logging.INFO)
# ch = logging.StreamHandler()
# ch.setLevel(logging.INFO)
# create formatter
ffh = logging.Formatter('%(asctime)s : %(name)-12s : %(levelname)-8s : %(message)s')
# fch = logging.Formatter('%(name)-12s : %(levelname)-8s : %(message)s')
# add formatter handlers
fh.setFormatter(ffh)
# ch.setFormatter(fch)
# add handler to logger
logger.addHandler(fh)
# logger.addHandler(ch)
logger.info('instance logger')
# logging.shutdown()

Python Custom logging handler to create each log file per key in the dictionary

I have a dictionary with some sample data like below
{"screener": "media", "anchor": "media","reader": "media"}
and I wanted to create a log file for each key in the dictionary. I'm planning to use this logging in streaming job which will get reused for every batch in the streaming. here I'm planning to use the rotating file handler per key as well.
here is my snippet
import logging
from logging.handlers import RotatingFileHandler
import time
logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)
dict_store = {"screener": "media", "anchor": "media", "reader": "media"}
dict_log_handler = {}
def createFileHandler(name):
handler = RotatingFileHandler(name, maxBytes=2000000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
return handler
def runBatch():
print("logging batch")
for name in dict_store.keys():
print(name)
if name in dict_log_handler:
print(f"getting logger from dict_handler {name}")
handler = dict_log_handler[name]
else:
handler = createFileHandler(name)
dict_log_handler[name] = handler
logger.addHandler(handler)
logger.info('Hello, world!')
logger.info('Hello, world!')
logger.info('Hello, world!')
logger.info('Hello, world!')
logger.info('Hello, world!')
logger.info('Hello, world!')
logger.removeHandler(handler)
time.sleep(0.1)
for i in range(0, 3):
runBatch()
It is working as expected currently. I'm just thinking of implementing similar stuff inside the overriding or creating a custom handler (like if we pass a name, it should automatically do this stuff) and the overall expectation is it should not affect the performance.
Question is, I wanted to wrap this in a class and using it directly. possible ?

You question is not clear what exactly you want to do, but if the idea is to use multiple loggers as your code shows then you can do something like this:
logging.getLogger(name) this is the method which is used to access the specific logger, in your code you are using the same logger but using addHandler and removeHandler to switch to specific logger.
You can create multiple loggers like this:
import logging
from logging.handlers import RotatingFileHandler
dict_store = {"screener": "media", "anchor": "media", "reader": "media"}
for name in dict_store:
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
handler = RotatingFileHandler(name, maxBytes=2000000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
You can wrap the above code in your own logging class/method and use it as needed. Keep in mind this needs to called only once.
Next time you can access the specific log and use the logging method:
logger = logging.getLogger(<logger_name>)
logger.debug("debug")
logger.info("info")
logger.warning("warning")
logger.error("error")
logger.critical("critical")

Is it possible to log into multiple log files from a single module in python 3.0

Is it possible to log into multiple log files from a single module in python 3.0, where the logs file are named based on some request parameter while using flask framework.
below code works fine if i run it like a single module and import but when i run using flask, it writes in the first attempt but later falls back to logging to the root logger.
I want a logging factory in flask that can check if the logger is already present and if already present than log in the same file and if not then create a new logger and log to the new file.
Any help is greatly appreciated.
def setup_logger( name, log_file, level=logging.INFO):
my_file = Path(log_file)
print(my_file)
if my_file.is_file():
print("handler details")
print(logging.getLogger(name).hasHandlers())
print(type(logging.getLogger(name).hasHandlers()))
if logging.getLogger(name).hasHandlers():
print("old logger and it has handler")
logger.propagate = False
return logging.getLogger(name)
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
logger.propagate = False
print("old logger that has no handler")
return logger
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
print("new logger with new handler")
logger.propagate = False
return logger

Well I was able to achieve the required functionality with the following method:
def setup_logger( name, log_file, level=logging.DEBUG):
my_file = Path(log_file)
# print("check the if condition for the file")
# print(my_file.is_file())
if my_file.is_file():
#print(logging.getLogger(name).hasHandlers())
# if logging.getLogger(name).hasHandlers():
if len(logging.getLogger(name).handlers)>0:
return logging.getLogger(name)
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
logger.propagate = False
return logger
else:
handler = logging.FileHandler(log_file, mode='a')
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
logger.propagate = False
return logger

Location of Python log file should be changed

I am using logger in my python 2.7 project on a legacy code. I want to create logs on specific location but python logging module creates the log files at the default place i.e. from where it is executed.
Is there is any way to change this default location?
Below is the initialization of the logger.
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(message)s')
file_handler = logging.FileHandler('file.log')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)

Below is an example of how you can set the location and other properties of Python logger:
You can define a get_logger function as follows:
import logging
import os
LOG_DIR = 'log_dir'
LOG_FORMATTER = logging.Formatter('[%(asctime)s] %(levelname)s %(name)s: %(message)s')
def get_logger(log_name, log_dir = LOG_DIR):
if not os.path.isdir(log_dir):
os.mkdir(log_dir)
logger = logging.getLogger(log_name)
logging.basicConfig(level = logging.INFO)
log_handler = logging.FileHandler(os.path.join(log_dir, log_name))
logger.addHandler(log_handler)
log_handler.setFormatter(LOG_FORMATTER)
log_handler.setLevel('INFO')
return logger
Then in the file that you want to make logs, you can do as follows:
logger = get_logger('filename')
If you want to make a logging message, you can then do as follows:
logger.info('logging information!')

Python logging.get_Logger(name) with FileHandler does not write to file

If I get the logger with a name and add a FileHandler it does not write to the file.
This works and writes correctly to the file:
log = logging.getLogger()
fh = logging.FileHandler(logfile)
log.addHandler(fh)
fh_fmt = logging.Formatter("%(asctime)s (%(levelname)s)\t: %(message)s")
fh.setFormatter(fh_fmt)
log.setLevel(logging.INFO)
This does not write to the file:
log = logging.getLogger(name)
fh = logging.FileHandler(logfile)
log.addHandler(fh)
fh_fmt = logging.Formatter("%(asctime)s (%(levelname)s)\t: %(message)s")
fh.setFormatter(fh_fmt)
log.setLevel(logging.INFO)
The only difference is that I get a 'named' logger.

This is a rather old question, but I believe I found the underlying problem and solution, at least with a newer version of Python.
The second code example starts with log = logging.getLogger(name), where name is presumed to be a string representing the name of the logger. Since a name is provided, this log will not be the root logger. According to Logger.setLevel(level) docs for Python 3.6+,
When a logger is created, the level is set to NOTSET (which causes all messages to be processed when the logger is the root logger, or delegation to the parent when the logger is a non-root logger).
This tells us that we have to set the level of our logger so that it will actually process the messages instead of passing it to the root logger.
This is a code example I wrote (in Python 3.7) that does not work:
from pathlib import Path
import logging
formatter = logging.Formatter('%(name)s [%(levelname)s] %(message)s')
log_file_dir = Path('./log/')
config_file = 'config_file.txt'
config_file_path = log_file_dir / config_file
logger = logging.getLogger('example_logger')
fh = logging.FileHandler(config_file_path, mode='w')
fh.setLevel(logging.INFO)
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.info('Start Configuration Log')
And this one works by adding one line:
from pathlib import Path
import logging
formatter = logging.Formatter('%(name)s [%(levelname)s] %(message)s')
log_file_dir = Path('./log/')
config_file = 'config_file_2.txt'
config_file_path = log_file_dir / config_file
logger = logging.getLogger('example_logger')
logger.setLevel(logging.INFO) # <------ Or the applicable level for your use-case
fh = logging.FileHandler(config_file_path, mode='w')
fh.setLevel(logging.INFO)
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.info('Start Configuration Log')
Note: The first code example does create the chosen log file, but does not write 'Start Configuration Log' to the file.