I have a program with a loop, and I would like to set up a main logger which logs information from the whole program. I would like to set up a different logger inside a loop (I will call this a 'logger_instance' going forward) for each loop instance. The main logger also needs to be able to log some information within a loop.
The issue with my current code is the following: once I initialize a logger_instance inside a loop, the information that I intended to log to the main logger gets logged to the logger_instance instead of the main logger.
Below is the sample code:
class DispatchingFormatter:
"""
This allows to create several formatter objects with desired formats so that logging outputs can take different
formats
"""
def __init__(self, formatters, default_formatter):
self._formatters = formatters
self._default_formatter = default_formatter
def format(self, record):
formatter_obj = self._formatters.get(record.name, self._default_formatter)
return formatter_obj.format(record)
def initiate_logger(log_file_name):
# Set logging to display INFO level or above
logging.basicConfig(level=logging.INFO)
# First empty out list of handlers to remove the default handler created by the running basicConfig above
# logging.getLogger().handlers.clear()
logger = logging.getLogger()
logger.handlers.clear()
# logger = logging.getLogger().handlers.clear()
# Set up formatter objects
formatter_obj = DispatchingFormatter(
# Custom formats - add new desired formats here
{
# This format allows to print the user and the time - use this log format at the start of the execution
'start_log': logging.Formatter('\n%(levelname)s - %(message)s executed the pipeline at %(asctime)s',
datefmt='%Y-%m-%d %H:%M:%S'),
# This format allows to print the time - use this log format at the end of the execution
'end_log': logging.Formatter('\n%(levelname)s - pipeline execution completed at %(asctime)s',
datefmt='%Y-%m-%d %H:%M:%S'),
# This format allows to print the duration - use this log format at the end of the execution
'duration_log': logging.Formatter('\n%(levelname)s - total time elapsed: %(message)s minutes'),
# This is the format of warning messages
'warning_log': logging.Formatter('\n\t\t%(levelname)s - %(message)s'),
# This is the format of error messages (
'error_log': logging.Formatter('\n%(levelname)s! [%(filename)s:line %(lineno)s - %(funcName)20s()] - '
'Pipeline terminating!\n\t%(message)s')
},
# Default format - this default is used to print all ESN outputs
logging.Formatter('%(message)s'),
)
# Log to console (stdout)
c_handler = logging.StreamHandler()
c_handler.setFormatter(formatter_obj)
# logging.getLogger().addHandler(c_handler)
logger.addHandler(c_handler)
# Log to file in runs folder
f_handler = logging.FileHandler(log_file_name, 'w+')
f_handler.setFormatter(formatter_obj)
# logging.getLogger().addHandler(f_handler)
logger.addHandler(f_handler)
# Log user and start time information upon creating the logger
logging.getLogger('start_log').info(f'{getpass.getuser()}')
return logger
if __name__ == '__main__':
# Test logging
# Initial main logger for outside the loop
logger_main = initiate_logger('log_main.txt')
logger_main.info(f'Started the main logging')
for i in range(5):
# Create logger in a loop
this_logger = initiate_logger(f'log_{str(i)}.txt')
this_logger.info(f'Logging in a loop - loop #{str(i)}')
# Log some information to main logger
logger_main.info(f'Something happened in loop #{str(i)}')
# Log more information to main logger after the loop
logger_main.info(f'Loop completed!')
log_main.txt contains
INFO - this_user executed the pipeline at 2019-05-29 19:15:47
Started the main logging
log_0.txt contains
INFO - lqk4061 executed the pipeline at 2019-05-29 19:15:47
Logging in a loop - loop #0
Something happened in loop #0
Desired output for log_main.txt should be
INFO - this_user executed the pipeline at 2019-05-29 19:15:47
Started the main logging
Something happened in loop #0
Something happened in loop #1
Something happened in loop #2
Something happened in loop #3
Something happened in loop #4
Loop completed!
Desired output for log_0.txt should be
INFO - lqk4061 executed the pipeline at 2019-05-29 19:15:47
Logging in a loop - loop #0
Any help will be greatly appreciated!
That happens because your initiate_logger function always returns the root logger since it calls getlogger without a name. See the documentation. What you need to do is give each of them a different name if you want them to be different logger instances. for example logger = logging.getLogger(log_file_name) would work in your code. I would recomment using filters instead though.
Related
I am learning about logging when running multiple process.
Below is how I normally log things when running a single process.
import logging
log_format = '%(asctime)s - %(name)s - %(levelname)s - %(messsage)s'
logger = logging.getLogger(__name__)
logger.setLevel('Debug')
file_handler = logging.FileHandler('C:/my_directory/logs/file_name.log')
formatter = logging.Formatter(log_format)
file_handler.setFormatter(formatter)
# to stop duplication
if not len(logger.handlers):
logger.addHandler(file_handler)
So after my code has run I can go to C:/my_directory/logs/file_name.log & check what I need to.
With multiple processes I understand its not so simple. I have read this great article. I have copied the example code below. What I don't understand is how to output the logged messages to a file like above so that I can read it after the code has finished?
from random import random
from time import sleep
from multiprocessing import current_process
from multiprocessing import Process
from multiprocessing import Queue
from logging.handlers import QueueHandler
import logging
# executed in a process that performs logging
def logger_process(queue):
# create a logger
logger = logging.getLogger('app')
# configure a stream handler
logger.addHandler(logging.StreamHandler())
# log all messages, debug and up
logger.setLevel(logging.DEBUG)
# run forever
while True:
# consume a log message, block until one arrives
message = queue.get()
# check for shutdown
if message is None:
break
# log the message
logger.handle(message)
# task to be executed in child processes
def task(queue):
# create a logger
logger = logging.getLogger('app')
# add a handler that uses the shared queue
logger.addHandler(QueueHandler(queue))
# log all messages, debug and up
logger.setLevel(logging.DEBUG)
# get the current process
process = current_process()
# report initial message
logger.info(f'Child {process.name} starting.')
# simulate doing work
for i in range(5):
# report a message
logger.debug(f'Child {process.name} step {i}.')
# block
sleep(random())
# report final message
logger.info(f'Child {process.name} done.')
# protect the entry point
if __name__ == '__main__':
# create the shared queue
queue = Queue()
# create a logger
logger = logging.getLogger('app')
# add a handler that uses the shared queue
logger.addHandler(QueueHandler(queue))
# log all messages, debug and up
logger.setLevel(logging.DEBUG)
# start the logger process
logger_p = Process(target=logger_process, args=(queue,))
logger_p.start()
# report initial message
logger.info('Main process started.')
# configure child processes
processes = [Process(target=task, args=(queue,)) for i in range(5)]
# start child processes
for process in processes:
process.start()
# wait for child processes to finish
for process in processes:
process.join()
# report final message
logger.info('Main process done.')
# shutdown the queue correctly
queue.put(None)
Update
I added the below code in the logger_process function just before the While True: loop. However when I look in the file, there is nothing there. I'm not seeing any output, not sure what I'm missing?
# add file handler
log_format = '%(asctime)s - %(name)s - %(levelname)s - %(messsage)s'
file_handler = logging.FileHandler('C:/my_directory/logs/file_name.log')
formatter = logging.Formatter(log_format)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
I need your help in solving this problem. So I am using Python's timedrotatingfilehandler lib to append logs for 24 hours in a log file and then rotate the log file. This works as expected but every time I run my script, timedrotatingfilehandler only logs till certain point. For example: If I have 10 log statement, each run only logs till 5 and then nothing gets logged in the log file. Appending happens correctly though. Not sure whats happening. Here is the python code:
#Configuring logger
LOG = logging.getLogger(__name__)
def config_Logs():
# format the log entries
global LOG
formatter = logging.Formatter('[%(asctime)s] : {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s')
handler = TimedRotatingFileHandler('collecting_logs.log',
when='h', interval=24, backupCount=0)
handler.setFormatter(formatter)
LOG.addHandler(handler)
LOG.setLevel(logging.DEBUG)
I am calling config_Logs() at the beginning of the script.
During development I use the python logging module. For example, after an unhandled exception I'd like to re-run the program and freshly initialize the logging.
For some reason it seems that I'm unable to remove all handlers from the log instance. Even so it was never deleted.
import logging
log = logging.getLogger(__name__)
print('Existing handlers:')
print(log.handlers)
#Remove all handlers:
for handler in log.handlers: #get rid of existing old handlers
print('removing handler %s'%handler)
log.removeHandler(handler)
#excpecting "[]" for log.handlers
print('Existing handlers after removal:')
print(log.handlers)
fh1 = logging.StreamHandler()
formatter1 = logging.Formatter('fh1: %(levelname)s - %(message)s')
fh1.setFormatter(formatter1)
fh2 = logging.StreamHandler()
formatter2 = logging.Formatter('fh2: %(levelname)s - %(message)s')
fh2.setFormatter(formatter2)
log.addHandler(fh1)
log.addHandler(fh2)
log.error('Some logging occurs here')
On the first run in a fresh IPython console I get:
fh1: ERROR - Some logging occurs here
fh2: ERROR - Some logging occurs here
Existing handlers:
[]
Existing handlers after removal:
[]
This almost what I expected. The order of appearance bugs me a bit. Why does the log appear before the print output? It really gets strange when starting the program a second time:
fh2: ERROR - Some logging occurs here
fh1: ERROR - Some logging occurs here
fh2: ERROR - Some logging occurs here
Existing handlers:
[<StreamHandler stderr (NOTSET)>, <StreamHandler stderr (NOTSET)>]
removing handler <StreamHandler stderr (NOTSET)>
Existing handlers after removal:
[<StreamHandler stderr (NOTSET)>]
It seems that the for loop, removing the handles, is executed just one time only.
And than consequently I get 3 logging entries, which is not what I want. I expected for the second run:
Existing handlers:
[<StreamHandler stderr (NOTSET)>, <StreamHandler stderr (NOTSET)>]
removing handler <StreamHandler stderr (NOTSET)>
removing handler <StreamHandler stderr (NOTSET)>
Existing handlers after removal:
[]
fh1: ERROR - Some logging occurs here
fh2: ERROR - Some logging occurs here
I seem to have missed some of the concept.
+ Why does the for loop just run one time, although len(log.handlers) returns 2 after the first run and 3 after the second run?
Why is the order of the print and logging commands mixed up?
And most important:
How can all the handlers be removed properly? Or the logging forced to start cleanly?
I'm using python 3.7.1 and logging 0.5.1.2
I think this issue is more related to your system OS and to your computer hardware characteristics. Everything works fine and prints continuously on my side
About 3 logging entries - maybe there is the same issue with flushing text in the logging module
Note: in the part where you remove log handlers, you should make a copy of this list before iterating it. That's why you weren't clearing all logging handlers. Like this:
#Remove all handlers:
for handler in log.handlers[:]: #get rid of existing old handlers
print('removing handler %s'%handler)
log.removeHandler(handler)
This way you remove all handlers:
logger = logging.getLogger()
while logger.hasHandlers():
logger.removeHandler(logger.handlers[0])
I want to log a line describing the logging information for each log file created by the logger.
Currently I'm using a separate logger process which will be running all the time. It receives the information from a queue and writes it to the log. A lot of modules will be passing information to this logging queue.
My current sample code is:
import logging
import time
from logging.handlers import TimedRotatingFileHandler as rotate
def info_log(log_queue):
logger = logging.getLogger("My Log")
logger.setLevel(logging.INFO)
handler = rotate("log/info.log", when="D", interval=30, backupCount=13)
logger.addHandler(handler)
desc_string = "yyyy/mm/dd-HH:MM:SS \t name \t country \n"
logger.info(desc_string)
while True:
result=log_queue.get().split("#")
logger.info(result[0] + "\t" result[1] + "\t" + result[2] + "\n")
Each time the log is rotated, I want desc_string to be written 1st to the log file.How can I do that?
Or in other words, How to know in the program when a log is rotated?
Maybe you can simply override the doRollover method from TimedRotatingFileHandler ?
class CustomFileHandler(TimedRotatingFileHandler):
def doRollover(self):
super().doRollover()
self.stream.write(desc_string)
# to use it
handler = CustomFileHandler("log/info.log", when="D", interval=30, backupCount=13)
logger.addHandler(handler)
I am learning the logging module in Python.
However, if I log like this
logging.basicConfig(filename='mylog.log',format='%(asctime)s - %(levelname)s - %(message)s', level=logging.DEBUG)
while 1:
logging.debug("something")
time.sleep(1)
and interrupt the process with control-C event(or the process is killed), nothing I can got from the log file.
Can I save the most logs whatever happens?
————
EDIT
the question seem become more complex:
I have imported scipy, numpy, pyaudio in my script, and I got:
forrtl: error (200): program aborting due to control-C event
instead of KeyboardInterrupt
I have read this question: Ctrl-C crashes Python after importing scipy.stats
and add these line to my script:
import _thread
import win32api
def handler(dwCtrlType, hook_sigint=_thread.interrupt_main):
if dwCtrlType == 0: # CTRL_C_EVENT
hook_sigint()
return 1 # don't chain to the next handler
return 0 # chain to the next handler
then:
try:
main()
except KeyboardInterrupt:
print("exit manually")
exit()
Now, the script stops without any info if I use ctrl+C. print("exit manually") does not appear. Of course, no logs.
Solved
A stupid mistake!
I run the script when working directory is System32 and want to find log in the script's path.
After I change the route like this, all is well.
logging.basicConfig(filename=os.path.dirname(sys.argv[0])+os.sep+'mylog.log',format='%(asctime)s - %(levelname)s - %(message)s', level=logging.DEBUG)
When you log using logging.debug, logging.info, ..., logging.critical, you're using the root logger. I assume you're not doing anything to configure logging that you haven't shown, so you're running with the out-of-the-box, default configuration. (This is set up for you by the first call to logging.debug, which calls logging.basicConfig()).
The default logging level of the root logger is logging.WARNING (as mentioned in e.g. https://docs.python.org/3/howto/logging.html#logging-basic-tutorial). Thus, nothing you log with logging.debug or logging.info will appear :) If you change logging.debug to logging.warning (or .error or .critical), you will see logging output.
For your code to work as is, set the logging level of the root logger to logging.DEBUG before the loop:
import logging
import time
# logging.getLogger() returns the root logger
logging.getLogger().setLevel(logging.DEBUG)
while 1:
logging.debug("something")
time.sleep(1)
For the CTRL + C event use a try-except to catch the KeyboardInterrupt exception.