I have scripts parent.py and child.py (many childs) and I need to have logs for each, so any logging within parent.py should be in parent.log and child.py should be in child.log
I have the below in each script but I get empty logs... why??
#main.py
import child
handler = logging.FileHandler('logs/main.log')
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s [%(filename)s:%(lineno)s - %
(funcName)10s()] %(levelname)s: %(message)s")
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
child.child_func()
logger.info('testing parent...')
#child.py
handler = logging.FileHandler('logs/child.log')
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s [%(filename)s:%(lineno)s - %
(funcName)10s()] %(levelname)s: %(message)s")
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
def child_func():
logger.info('testing child...')
What I need to have is
#parent.log
{format} testing parent...
#child.log
{format} testing child...
The folks above are right about the default level on the loggers. Also, instead of spreading your configuration around everywhere, I find it more manageable to consolidate logging configuration to be early on in the application. See the example below.
Note: I don't expect this to be selected as an answer. I just wanted to point out what I believe is a better way of organizing the code.
main.py
import logging
import child
logger = logging.getLogger(__name__)
def setup_logging():
main_handler = logging.FileHandler('logs/main.log')
child_handler = logging.FileHandler('logs/child.log')
# Note that you can re-use the same formatter for the handlers.
formatter = logging.Formatter("%(asctime)s [%(filename)s:%(lineno)s - %(funcName)10s()] %(levelname)s: %(message)s")
main_handler.setFormatter(formatter)
child_handler.setFormatter(formatter)
# By default, loggers inherit the level from their parent, so we only need
# to set the level on the root logger if you want to have only one knob to
# control the level.
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
main_logger = logging.getLogger(__name__)
child_logger = logging.getLogger('child')
child_logger.propagate = False
main_logger.addHandler(main_handler)
child_logger.addHandler(child_handler)
def main():
setup_logging()
child.child_func()
logger.info('testing parent...')
if __name__ == '__main__':
main()
child.py
import logging
logger = logging.getLogger(__name__)
def child_func():
logger.info('testing child...')
Setting up a root logger and a child logger (no main)
Here's an example of setting up the root logger to log to logs/main.log, and the child logger to go to logs/child.log
def setup_logging():
root_handler = logging.FileHandler('logs/main.log')
child_handler = logging.FileHandler('logs/child.log')
# Note that you can re-use the same formatter for the handlers.
formatter = logging.Formatter("%(asctime)s [%(filename)s:%(lineno)s - %(funcName)10s()] %(levelname)s: %(message)s")
root_handler.setFormatter(formatter)
child_handler.setFormatter(formatter)
# By default, loggers inherit the level from their parent, so we only need
# to set the level on the root logger if you want to have only one knob to
# control the level.
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(root_handler)
child_logger = logging.getLogger('child')
child_logger.propagate = False
child_logger.addHandler(child_handler)
You can set severity-level on both handlers and loggers - I believe the logger is set to logging.WARNING by default, so you would only get warning-logs using your code.
You can read more in this thread: What is the point of setLevel in a python logging handler?
import logging
import child
handler = logging.FileHandler('logs/main.log')
formatter = logging.Formatter("%(asctime)s [%(filename)s:%(lineno)s - %(funcName)10s()] %(levelname)s: %(message)s")
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG) # <-- changed
child.child_func()
logger.info('testing parent...')
logger.warning('testing parent...')
logger.debug('testing parent...')
#child.py
import logging
handler = logging.FileHandler('logs/child.log')
formatter = logging.Formatter("%(asctime)s [%(filename)s:%(lineno)s - %(funcName)10s()] %(levelname)s: %(message)s")
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG) # <-- changed
def child_func():
logger.info('testing child...')
logger.warning('testing child...')
logger.debug('testing child...')
Related
I am using logging module in python. In my main.py file I am using two logger.
Root logger (To get logs from multiple modules in same directory)
Local logger (To log specific information)
I want information of local logger to be separate from root logger. But when I am creating separate logger. Information of local logger is also present in root logger info.
Here is the sample of how I am doing this
# main.py
import logging
def setup_logger(filename, name = ''):
if name == '':
logging.basicConfig(filename=filename,
format='%(asctime)s %(funcName)s %(levelname)s %(message)s',
filemode='a')
logger = logging.getLogger()
else:
"""
handler = logging.FileHandler(filename, mode = 'a')
handler.setFormatter(logging.Formatter('%(asctime)s %(funcName)s %(levelname)s %(message)s'))
logger = logging.getLogger(name)
logger.addHandler(handler)
"""
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler = logging.FileHandler(filename)
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
return logger
logger.setLevel(logging.DEBUG)
return logger
logger = setup_logger('main.log')
local_logger = setup_logger('local_log.log', 'local_log')
# other file under root log
logger = logging.getLogger("__main__." + __name__)
You have to stop propagation if you don't want the local loggers to send their logs to the root loggers handlers:
logger.propagate = False
This part of the documentation explains it well: https://docs.python.org/3/howto/logging.html#logging-flow
I have the following requirements:
To have one global logger which you can configure (setup level, additional handlers,..)
To have per module logger which you can configure (setup level, additional handlers,..)
In other words we need more logs with different configuration
Therefore I did the following
create method to setup logger:
def setup_logger(module_name=None, level=logging.INFO, add_stdout_logger=True):
print("Clear all loggers")
for _handler in logging.root.handlers:
logging.root.removeHandler(_handler)
if add_stdout_logger:
print("Add stdout logger")
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(level)
stdout_handler.setFormatter(logging.Formatter(fmt='%(asctime)-11s [%(levelname)s] [%(name)s] %(message)s'))
logging.root.addHandler(stdout_handler)
print("Set root level log")
logging.root.setLevel(level)
if module_name:
return logging.getLogger(module_name)
else:
return logging.getLogger('global')
Then I create logger as following:
logger_global = setup_logger(level=logging.DEBUG)
logger_module_1 = setup_logger(module_name='module1', level=logging.INFO)
logger_module_2 = setup_logger(module_name='module2', level=logging.DEBUG)
logger_global.debug("This is global log and will be visible because it is setup to DEBUG log")
logger_module_1.debug("This is logger_module_1 log and will NOT be visible because it is setup to INFO log")
logger_module_2.debug("This is logger_module_2 log and will be visible because it is setup to DEBUG log")
Before I will try what works and what not and test it more deeply I want to ask you if this is good practice to do it or do you have any other recommendation how to achieve our requrements?
Thanks for help
Finally I found how to do it:
def setup_logger(module_name=None, level=logging.INFO, add_stdout_logger=True):
custom_logger = logging.getLogger('global')
if module_name:
custom_logger = logging.getLogger(module_name)
print("Clear all handlers in logger") # prevent multiple handler creation
module_logger.handlers.clear()
if add_stdout_logger:
print("Add stdout logger")
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(level)
stdout_handler.setFormatter(logging.Formatter(fmt='%(asctime)-11s [%(levelname)s] [%(name)s] %(message)s'))
module_logger.addHandler(stdout_handler)
# here you can add another handlers ,...
# because we use custom handlers which have the different type of log level,
# then our logger has to have the lowest level of logging
custom_logger.setLevel(logging.DEBUG)
return custom_logger
Then simply call the following
logger_module_1 = setup_logger(module_name='module1', level=logging.INFO)
logger_module_2 = setup_logger(module_name='module2', level=logging.DEBUG)
logger_module_1.debug("This is logger_module_1 log and will NOT be visible because it is setup to INFO log")
logger_module_2.debug("This is logger_module_2 log and will be visible because it is setup to DEBUG log")
I am a python newbie, trying to implement logging into my code. I have two modules
main.py
submodule.py
main.py
import logging
from logging.handlers import RotatingFileHandler
import submodule
import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler('master.log', maxBytes=2000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug('DEBUG LEVEL - MAIN MODULE')
logger.info('INFO LEVEL - MAIN MODULE')
submodule.loggerCall()
submodule.py
import logging
from logging.handlers import RotatingFileHandler
def loggerCall():
logger = logging.getLogger(__name__)
# logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler('master.log', maxBytes=2000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug('SUBMODULE: DEBUG LOGGING MODE : ')
logger.info('Submodule: INFO LOG')
return
I thought as longs as I call the getLogger from my submodule, it should inherit the log level & handler details from root logger. However, in my case, I have to specify log level and handler again in submodule to get them print to same log file.
Also, If I have lots of methods, and classes inside my submodule. How can I go about it without having to define my log level & handler again.
Idea is to have a single log file with main, and sub modules printing in the same log based on the log level set in the main module.
The problem here is that you're not initializing the root logger; you're initializing the logger for your main module.
Try this for main.py:
import logging
from logging.handlers import RotatingFileHandler
import submodule
logger = logging.getLogger() # Gets the root logger
logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler('master.log', maxBytes=2000000, backupCount=10)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug('DEBUG LEVEL - MAIN MODULE')
logger.info('INFO LEVEL - MAIN MODULE')
submodule.loggerCall()
Then try this for submodule.py:
def loggerCall():
logger = logging.getLogger(__name__)
logger.debug('SUBMODULE: DEBUG LOGGING MODE : ')
logger.info('Submodule: INFO LOG')
return
Since you said you wanted to send log messages from all your submodules to the same place, you should initialize the root logger and then simply use the message logging methods (along with setlevel() calls, as appropriate). Because there's no explicit handler for your submodule, logging.getLogger(__name__) will traverse the tree to the root, where it will find the handler you established in main.py.
I am already using a basic logging config where all messages across all modules are stored in a single file. However, I need a more complex solution now:
Two files: the first remains the same.
The second file should have some custom format.
I have been reading the docs for the module, bu they are very complex for me at the moment. Loggers, handlers...
So, in short:
How to log to two files in Python 3, ie:
import logging
# ...
logging.file1.info('Write this to file 1')
logging.file2.info('Write this to file 2')
You can do something like this:
import logging
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
def setup_logger(name, log_file, level=logging.INFO):
"""To setup as many loggers as you want"""
handler = logging.FileHandler(log_file)
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(level)
logger.addHandler(handler)
return logger
# first file logger
logger = setup_logger('first_logger', 'first_logfile.log')
logger.info('This is just info message')
# second file logger
super_logger = setup_logger('second_logger', 'second_logfile.log')
super_logger.error('This is an error message')
def another_method():
# using logger defined above also works here
logger.info('Inside method')
def setup_logger(logger_name, log_file, level=logging.INFO):
l = logging.getLogger(logger_name)
formatter = logging.Formatter('%(message)s')
fileHandler = logging.FileHandler(log_file, mode='w')
fileHandler.setFormatter(formatter)
streamHandler = logging.StreamHandler()
streamHandler.setFormatter(formatter)
l.setLevel(level)
l.addHandler(fileHandler)
l.addHandler(streamHandler)
setup_logger('log1', txtName+"txt")
setup_logger('log2', txtName+"small.txt")
logger_1 = logging.getLogger('log1')
logger_2 = logging.getLogger('log2')
logger_1.info('111messasage 1')
logger_2.info('222ersaror foo')
I'm using the following code to initialize logging in my application:
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# log to a file
directory = '/reserved/DYPE/logfiles'
now = datetime.now().strftime("%Y%m%d_%H%M%S")
filename = os.path.join(directory, 'dype_%s.log' % now)
file_handler = logging.FileHandler(filename)
file_handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s %(filename)s, %(lineno)d, %(funcName)s: %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
# log to the console
console_handler = logging.StreamHandler()
level = logging.INFO
console_handler.setLevel(level)
logger.addHandler(console_handler)
logging.debug('logging initialized')
How can I close the current logging file and restart logging to a new file?
Note: I don't want to use RotatingFileHandler, because I want full control over all the filenames.
You can manually re-assign the handler if you want using the removeHandler and addHandler OR, you can access logger.handlers[index_of_handler_here].stream and replace the stream manually, but I'd recommend the former over the latter.
logger.handlers[0].stream.close()
logger.removeHandler(logger.handlers[0])
file_handler = logging.FileHandler(filename)
file_handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s %(filename)s, %(lineno)d, %(funcName)s: %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
Here is what I do:
def initLogging(filename):
global logger
if logger == None:
logger = logging.getLogger()
else: # wish there was a logger.close()
for handler in logger.handlers[:]: # make a copy of the list
logger.removeHandler(handler)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(fmt='%(asctime)s: %(message)s', datefmt='%I:%M:%S')
fh = logging.FileHandler(filename)
fh.setFormatter(formatter)
logger.addHandler(fh)
sh = logging.StreamHandler(sys.stdout)
sh.setFormatter(formatter)
logger.addHandler(sh)