I am trying to print logs using logger module in python.
Following is the code I am keeping on the top of file.
if __name__ == '__main__':
LOG_FILENAME = '/home/akash/exdion-pdf-extracter/doc/epod.log'
logging.basicConfig(
filename=LOG_FILENAME,
level=logging.DEBUG,
)
There are different files with function calls from one another. I have used the following line to display a line in the logger.
#staticmethod
def initiate_pdf_processing(ct_doc, pt_doc, feature, startAndEndKeyList):
logging.info("testing logger")
...
There are multiple instances of the similar above logger function. But I can't receive the logger output in the designated file. The code and files are huge. However there are a few error generated by the code which are getting printed in the log file.
Use below code bit out of the main namespace. This way, you are defining a logger and creating a log file as global file, and you can call the logger anywhere in the code. A logger code bit below is how I usually code.
logfile = '<your_file_name>.log'
if(os.path.isfile(logfile)):
os.remove(logfile)
file_handler = logging.FileHandler(logfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(pathname)s [%(process)d]: %(levelname)s:: %(message)s'))
logger = logging.getLogger('wbs-server-log')
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
The issue might be that you have to initialize logging above if __name__ == '__main__' block. That way logging will be initialized when you import this as module.
Suggestion for initializing logging:
import logging
log = logging.getLogger(PACKAGE_NAME)
stream_handler = logging.StreamHandler(stream=open(LOG_FILE_NAME, 'a'))
stream_handler.setLevel(logging.DEBUG)
log.addHandler(stream_handler)
log.debug('your message here')
After this you can tweak log message formatting with logging.Formatter.
Related
I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know why it so but couldn't clearly how I can solve it.
Below is log_config.py which is used by all other modules to get the logger and log.
'''
import logging
import sys
from logging.handlers import TimedRotatingFileHandler
FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")
LOG_FILE = "my_app.log"
def get_file_handler():
file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
file_handler.setFormatter(FORMATTER)
return file_handler
def get_logger(logger_name):
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG) # better to have too much log than not enough
logger.addHandler(get_file_handler())
#with this pattern, it's rarely necessary to propagate the error up to parent
logger.propagate = False
return logger
'''
All other modules call,
'logging = log_config.get_logger(name)'
and use it to log.
I came to know about QueueHandler and QueueListener but not sure how to use them in my code.
How can I use these to serialize logs to single file.?
I had a script with logging capabilities, and it stopped working (the logging, not the script). I wrote a small example to illustrate the problem:
import logging
from os import remove
from os.path import exists
def setup_logger(logger_name, log_file, level=logging.WARNING):
# Erase log if already exists
if exists(log_file):
remove(log_file)
# Configure log file
l = logging.getLogger(logger_name)
formatter = logging.Formatter('%(message)s')
fileHandler = logging.FileHandler(log_file, mode='w')
fileHandler.setFormatter(formatter)
streamHandler = logging.StreamHandler()
streamHandler.setFormatter(formatter)
l.setLevel(level)
l.addHandler(fileHandler)
l.addHandler(streamHandler)
if __name__ == '__main__':
setup_logger('log_pl', '/home/myuser/test.log')
log_pl = logging.getLogger('log_pl')
log_pl.info('TEST')
log_pl.debug('TEST')
At the end of the script, the file test.log is created, but it is empty.
What am I missing?
Your setup_logger function specifies a (default) level of WARNING
def setup_logger(logger_name, log_file, level=logging.WARNING):
...and you later log two events that are at a lower level than WARNING, and are ignored as they should be:
log_pl.info('TEST')
log_pl.debug('TEST')
If you change your code that calls your setup_logger function to:
if __name__ == '__main__':
setup_logger('log_pl', '/home/myuser/test.log', logging.DEBUG)
...I'd expect that it works as you'd like.
See the simple example in the Logging HOWTO page.
I have a project that consists of several modules. There is main module (main.py) that creates a TK GUI and loads the data. It passes this data to process.py which processes the data using functions from checks.py. I am trying to implement logging for all the modules to log to a file. In the main.py log messages are written to the log file but in the other modules they are only written to the console. I assume its to do with the import module line executing part of the code before the code in main.py has set up the logger, but i can't work out how to arrange it to avoid that. It seems like a reasonably common question on Stackoverflow, but i couldn't get the other answers to work for me. I am sure I am not missing much. Simplified code is shown below:
Moving the logging code inside and outside of various functions in the modules. The code I used to start me off is the code from Corey Schaffer's Youtube channel.
Main.py
import logging
from process import process_data
def main():
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')
templogfile = tempfile.gettempdir() + '\\' + 'TST_HA_Debug.log'
file_handler = logging.FileHandler(templogfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stream_handler)
logger.debug('Logging has started') # This gets written to the file
process_data(data_object) # call process_data in process.py
process.py
import logging
def process_data(data):
logger = logging.getLogger(__name__)
logger.debug('This message is logged by process') #This wont get written to the log file but get written to the console
#do some stuff with data here and log some msgs
return
Main.py will write to the log file, process.py will only write to the console.
I've rewritten your scripts a little so that this code can stand alone. If I changed this too much let me know and I can revisit it. These two files are an example of having it log to file. Note my comments:
## main.py
import logging
from process import process_data
import os
def main():
# Give this logger a name
logger = logging.getLogger("Example")
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')
# I changed this to the same directory, for convenience
templogfile = os.path.join(os.getcwd(), 'TST_HA_Debug.log')
file_handler = logging.FileHandler(templogfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stream_handler)
logging.getLogger("Example").debug('Logging has started') # This still gets written to the file
process_data() # call process_data in process.py
if __name__ == '__main__':
main()
## process.py
import logging
def process_data(data=None):
# make sure to grab the correct logger
logger = logging.getLogger("Example")
logger.debug('This message is logged by process') # This does get logged to file now
#do some stuff with data here and log some msgs
return
Why does this work? Because the module-level functions use the default root logger, which is not the one you've configured. For more details on this see these docs. There is a similar question that goes more into depth here.
By getting the configured logger before you start logging, you are able to log to the right configuration. Hope this helps!
Is there any way I can provide the filename for logger from my main module?
I am using following way, however it's not working.all the logs go to xyz.log file rather than main.log
Updated as per suggestion from nosklo
logger.py
formatter = logging.Formatter(fmt='[%(asctime)s] - {%(filename)s:%(lineno)d} %(levelname)s - %(message)s')
def _get_file_handler(file_name="xyz.log"):
file_handler = logging.FileHandler(file_name)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
return file_handler
def get_logger(name):
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
logger.addHandler(_get_file_handler())
return logger
parser.py
log = logger.get_logger(__name__)
def parse():
log.info("is there anyway this could go to main.log and xyz.log")
main.py
log = logger.get_logger(__name__)
if __name__ == '__main__':
for handler in log.handlers:
if isinstance(handler, logging.FileHandler):
log.removeHandler(handler)
log.addHandler(logger._get_file_handler())
log.info("is there anyway this could go to main.log and xyz.log?")
parser.parse()
Is there a way I can set the Log file name from my main.py module and not from logger.py module?
You're calling get_logger() first, so when you set the class attribute in FileName.file_name = "main.log" the get_logger function is already finished, and the logger is already defined to write in xyz.log; Changing the variable later won't change the logger anymore, since it is already defined.
To change the previously selected file, you'd have to retrieve the logger, remove the previous handler and add a new file handler. Another option is to set the variable before calling get_logger() so when you call it, the variable already has the correct value.
Logging instances can have multiple file handlers. Use a function like this to just add another handler with the additional output path you want. Log messages will get sent to both (or all) text logs added to the instance. You can even configure the handlers to have different logging levels so you can filter messages to different logs for critical errors, info message, etc.
import logging
def add_handler(output_log_path, log):
# Set up text logger and add it to logging instance
file_logger = logging.FileHandler(output_log_path)
file_logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s | logger name: %(name)s | module: %(module)s | lineno: %(lineno)d | %(message)s')
file_logger.setFormatter(formatter)
log.addHandler(file_logger)
return log
I have been writing simple scripts and I am trying to use logger to generate log for each functions in the scripts.
1) based on the function name I create a logger filehandler and I try to put logs using that handler. I also delete the previous existing file with the same name.
3) at the end of the function I close the handler.
My problem are:
1)even though I close the handler, the next time I run the same function I get an error that the file I am trying to delete is (as a part of setting the logger file handler) is still being used.
2) Also the logger prints everything to console which I dont want, I just want it to write everything to the file.
Here are the logger functions:
def setLogger(path):
"""
#purpose: Intializes basic logging directory and file
"""
LOG_FILENAME = path + "\\" + "log.txt"
#logging.basicConfig(filename=LOG_FILENAME,
# format='%(levelname)s %(asctime)s %(message)s',level=logging.INFO
# )
logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = logging.FileHandler(LOG_FILENAME)
file_handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s %(asctime)s %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
return logger
def unsetLogger(logger):
"""
#purpose: performs a basic shutdown of logger
"""
logger.handlers[0].close()
logger.removeHandler(logger.handlers[0])
logging.shutdown
The way i use them is:
for eg:
def fun():
os.remove(path)
logger = setLogger(path)
` logging.info("hi") #this writes to file and prints on the console as well
unsetLogger(logger)
if I run the function fun() once, its all good. but if i run it again, I get that can't delete error for the log file.
Thanks in Advance.
learningNinja
After making some slight modifications, I came up with the following test to try to reproduce your error, but I don't get any errors.
import os
import logging
def setLogger(path):
"""
#purpose: Intializes basic logging directory and file
"""
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Simplified log file path (I just use full value passed in, and don't append "\log.txt")
file_handler = logging.FileHandler(path)
file_handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s %(asctime)s %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
return logger
def unsetLogger(logger):
"""
#purpose: performs a basic shutdown of logger
"""
logger.handlers[0].close()
logger.removeHandler(logger.handlers[0])
logging.shutdown
def fun():
try:
# Was getting error trying to remove a file that didn't exist on
# first execution...
os.remove("log.txt")
except:
pass
logger = setLogger("log.txt")
logging.info("hi")
unsetLogger(logger)
fun()
fun()
fun()
See if there is anything I'm doing differently than your actual code and maybe that might help you.