I want to create 2 loggers which logs to 2 different outputs in Python. This logger of mine is configured in a single module which will be used by 2 other main modules. My problem is since I configured the root logger to allow my logger to be used by 2 different main modules, I cannot separate logging output.
How can this be done?
Here is how I configure my logging:
# logger.py
import logging
def setup_logging():
# If I give name to my getLogger, it will not be configuring root logger and my changes here cannot cascade to all other child loggers.
logger = logging.getLogger()
streamHandler = logging.StreamHandler()
streamFormat = logging.Formatter('%(name)s - %(message)s')
streamHandler.setFormatter(streamFormat)
logger.addHandler(streamHandler)
# main1.py
import logging
from logger import setup_logging
from submodule import log_me
setup_logging()
logger = logging.getLogger('main1')
logger.log('I am from main1')
log_me()
# main2.py
import logging
from logger import setup_logging
from submodule import log_me
setup_logging()
logger = logging.getLogger('main2')
logger.log('I am from main2')
log_me()
# submodule.py
import logging
logger = logging.getLogger('submodule')
def log_me():
logger.info('I am from submodule')
Result from main1:
main1 - I am from main1
submodule - I am from submodule
Result from main2:
main2 - I am from main2
submodule - I am from submodule
Here is what I am trying to achieve (but beautifully fails of course).
# logger.py
import logging
def setup_logging():
logger = logging.getLogger()
streamHandler = logging.StreamHandler()
streamFormat = logging.Formatter('%(name)s - %(message)s')
streamHandler.setFormatter(streamFormat)
logger.addHandler(streamHandler)
def setup_second_logging():
logger2 = logging.getLogger()
fileHandler = logging.FileHandler('./out.log)
fileFormat = logging.Formatter('%(name)s - %(message)s')
fileHandler.setFormatter(fileFormat)
logger.addHandler(fileHandler)
--same main1.py--
--same main2.py--
# submodule.py
import logging
from logger import setup_second_logging
setup_second_logging()
logger = logging.getLogger('submodule')
def log_me():
logger.info('I am from submodule')
Result from main1:
main1 - I am from main1
# no submodule since it is logged to file
Result from main2:
main2 - I am from main2
# no submodule since it is logged to file
You can try to use hierarchy for the loggers. For example:
logger = logging.getLogger(__name__)
And in this case you can set specific handlers for the submodules in logger.py.
Related
I have a custom logging class, which has the following format:
log_format = "%(asctime)s.%(msecs)d %(levelname)-8s [%(processName)s] [%(threadName)s] %(filename)s:%(lineno)d --- %(message)s"
My project tree looks something like this:
.
├── exceptions.py
├── logger.py
├── settings.py
└── main.py
In main.py I import my custom Logger from logger.py. On several places I perform logging using the following syntax:
Logger.info("Transcribed audio successfully.")
However when looking at the logs, the filename and lineno params are always referring to my Logger class, not the actual function from main.py which invoked the logging:
2023-02-15 10:48:06,241.241 INFO [MainProcess] [MainThread] logger.py:38 --- Transcribed audio successfully.
Is there a way to change this? I would like that the log entry states something like:
2023-02-15 10:48:06,241.241 INFO [MainProcess] [MainThread] main.py:98 --- Transcribed audio successfully.
This is my logger.py file:
import logging
from logging.handlers import RotatingFileHandler
class Logger:
log_format = "%(asctime)s.%(msecs)d %(levelname)-8s [%(processName)s] [%(threadName)s] %(filename)s:%(lineno)d --- %(message)s"
#staticmethod
def setup_single_logger(name, logfile, level):
handler = RotatingFileHandler(logfile, mode='a', maxBytes=1024 * 1024, backupCount=10)
handler.setFormatter(logging.Formatter(Logger.log_format))
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
return logger
#staticmethod
def setup_logging():
Logger.info_logger = Logger.setup_single_logger('INFO', '/path/to/your/logfile.log', logging.INFO)
#staticmethod
def info(msg, *args, **kwargs):
Logger.info_logger.info(msg, *args, **kwargs)
Logger.setup_logging()
And an example main.py is:
from logger import Logger
Logger.info("Transcribed audio successfully.")
The problem is that you setup and create your logger in logging.py. So when you call Logger.info from main, the actual call of info on the logger object is done inside logger.py. You just created an interface for yourself to the logger object, but the underlying call is still the one specifying the attributes of the message.
What you could do is leave only the setup in logger.py, and let main.py have its own logger object.
So log.py (changed name due to clashes) can be:
import logging
from logging.handlers import RotatingFileHandler
def basic_config():
log_format = "%(asctime)s.%(msecs)d %(levelname)-8s [%(processName)s] [%(threadName)s] %(filename)s:%(lineno)d --- %(message)s"
handler = RotatingFileHandler('logfile.log', mode='a', maxBytes=1024 * 1024, backupCount=10)
handler.setFormatter(logging.Formatter(log_format))
logging.basicConfig(handlers=(handler,), level=logging.INFO)
(I felt like the use of the class was not necessary)
And main.py:
import log
import logging
log.basic_config()
logger = logging.getLogger(__file__)
logger.info("test")
Now the log file I got from running once with your code and once with mine was:
2023-02-15 13:14:30,275.275 INFO [MainProcess] [MainThread] log.py:23 --- test
2023-02-15 13:18:51,358.358 INFO [MainProcess] [MainThread] main.py:7 --- test
I am using logger in my python 2.7 project on a legacy code. I want to create logs on specific location but python logging module creates the log files at the default place i.e. from where it is executed.
Is there is any way to change this default location?
Below is the initialization of the logger.
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(message)s')
file_handler = logging.FileHandler('file.log')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
Below is an example of how you can set the location and other properties of Python logger:
You can define a get_logger function as follows:
import logging
import os
LOG_DIR = 'log_dir'
LOG_FORMATTER = logging.Formatter('[%(asctime)s] %(levelname)s %(name)s: %(message)s')
def get_logger(log_name, log_dir = LOG_DIR):
if not os.path.isdir(log_dir):
os.mkdir(log_dir)
logger = logging.getLogger(log_name)
logging.basicConfig(level = logging.INFO)
log_handler = logging.FileHandler(os.path.join(log_dir, log_name))
logger.addHandler(log_handler)
log_handler.setFormatter(LOG_FORMATTER)
log_handler.setLevel('INFO')
return logger
Then in the file that you want to make logs, you can do as follows:
logger = get_logger('filename')
If you want to make a logging message, you can then do as follows:
logger.info('logging information!')
I have multiple python modules that I'd like to use the same logger while preserving the call hierarchy in those logs. I'd also like to do this with a logger whose name is the name of the calling module (or calling module stack). I haven't been able to work out how to get the name of the calling module except with messing with the stack trace, but that doesn't feel very pythonic.
Is this possible?
main.py
import logging
from sub_module import sub_log
logger = logging.getLogger(__name__)
logger.info("main_module")
sub_log()
sub_module.py
import logging
def sub_log():
logger = logging.getLogger(???)
logger.info("sub_module")
Desired Output
TIME main INFO main_module
TIME main.sub_module INFO sub_module
To solve your problem pythonic use the Logger Formatter:
For reference check the
Logging Docs
main.py
import logging
from submodule import sub_log
from submodule2 import sub_log2
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('test.log')
fh.setLevel(logging.DEBUG)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s %(name)s.%(module)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(fh)
sub_log("test")
sub_log2("test")
submodule.py
import logging
import __main__
def sub_log(msg):
logger = logging.getLogger(__main__.__name__)
logger.info(msg)
I've created second submodule. ( same code other name)
My Results:
2018-10-16 20:41:23,860 __main__.submodule - INFO - test
2018-10-16 20:41:23,860 __main__.submodule2 - INFO - test
I hope this will help you :)
I have the following python package structure.
python_logging
python_logging
__init__.py
first_class.py
second_class.py
run.py
Here is the code in __init__.py
init.py
import logging
import logging.config
# Create the Logger
loggers = logging.getLogger(__name__)
loggers.setLevel(logging.DEBUG)
# Create the Handler for logging data to a file
logger_handler = logging.FileHandler(filename='C:\Python\Log\stest.txt')
logger_handler.setLevel(logging.DEBUG)
# Create a Formatter for formatting the log messages
logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
# Add the Formatter to the Handler
logger_handler.setFormatter(logger_formatter)
# Add the Handler to the Logger
loggers.addHandler(logger_handler)
loggers.info('Completed configuring logger()!')
Here is code for first_class.py
import logging
class FirstClass(object):
def __init__(self):
self.current_number = 0
self.logger = logging.getLogger(__name__)
def increment_number(self):
self.current_number += 1
self.logger.warning('Incrementing number!')
self.logger.info('Still incrementing number!!')
Here is code for second_class.py
class SecondClass(object):
def __init__(self):
self.enabled = False
self.logger = logging.getLogger(__name__)
def enable_system(self):
self.enabled = True
self.logger.warning('Enabling system!')
self.logger.info('Still enabling system!!')
Here is code for run.py
from LogModule.first_class import FirstClass
from LogModule.second_class import SecondClass
number = FirstClass()
number.increment_number()
system = SecondClass()
system.enable_system()
This is the output in the log file
LogModule - INFO - Completed configuring logger()!
LogModule.first_class - WARNING - Incrementing number!
LogModule.first_class - INFO - Still incrementing number!!
LogModule.second_class - WARNING - Enabling system!
LogModule.second_class - INFO - Still enabling system!!
Question : How did number.increment_number() and system.enable_system() write to a log file when the file handler was initialized in init.py ? also both the classes have different getloggers . Can anyone explain , it will be helpful.
Every logger has a parent - in Python you can think that all loggers construct into a single "tree". If you add handlers into one logger, its children loggers will also share these handlers. This means you have no need to create handlers for each logger - otherwise setting handlers on every logger will be repetitive and boring.
Back to your sample, your package structure,
python_logging
python_logging
__init__.py
first_class.py
second_class.py
run.py
in __init__.py file,
# Create the Logger
loggers = logging.getLogger(__name__)
the logger name is <python_logging>.
in first_class.py
self.logger = logging.getLogger(__name__)
the logger name is <python_logging>.<first_class>.
So the logger in first_class.py is the child of the logger in __init__.py
I am a python newbie, trying to implement logging into my code. I have two modules
main.py
submodule.py
main.py
import logging
from logging.handlers import RotatingFileHandler
import submodule
import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler('master.log', maxBytes=2000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug('DEBUG LEVEL - MAIN MODULE')
logger.info('INFO LEVEL - MAIN MODULE')
submodule.loggerCall()
submodule.py
import logging
from logging.handlers import RotatingFileHandler
def loggerCall():
logger = logging.getLogger(__name__)
# logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler('master.log', maxBytes=2000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug('SUBMODULE: DEBUG LOGGING MODE : ')
logger.info('Submodule: INFO LOG')
return
I thought as longs as I call the getLogger from my submodule, it should inherit the log level & handler details from root logger. However, in my case, I have to specify log level and handler again in submodule to get them print to same log file.
Also, If I have lots of methods, and classes inside my submodule. How can I go about it without having to define my log level & handler again.
Idea is to have a single log file with main, and sub modules printing in the same log based on the log level set in the main module.
The problem here is that you're not initializing the root logger; you're initializing the logger for your main module.
Try this for main.py:
import logging
from logging.handlers import RotatingFileHandler
import submodule
logger = logging.getLogger() # Gets the root logger
logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler('master.log', maxBytes=2000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug('DEBUG LEVEL - MAIN MODULE')
logger.info('INFO LEVEL - MAIN MODULE')
submodule.loggerCall()
Then try this for submodule.py:
def loggerCall():
logger = logging.getLogger(__name__)
logger.debug('SUBMODULE: DEBUG LOGGING MODE : ')
logger.info('Submodule: INFO LOG')
return
Since you said you wanted to send log messages from all your submodules to the same place, you should initialize the root logger and then simply use the message logging methods (along with setlevel() calls, as appropriate). Because there's no explicit handler for your submodule, logging.getLogger(__name__) will traverse the tree to the root, where it will find the handler you established in main.py.