How can I use the logging module in Python to write to a file? Every time I try to use it, it just prints out the message.
An example of using logging.basicConfig rather than logging.fileHandler()
logging.basicConfig(filename=logname,
filemode='a',
format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
datefmt='%H:%M:%S',
level=logging.DEBUG)
logging.info("Running Urban Planning")
logger = logging.getLogger('urbanGUI')
In order, the five parts do the following:
set the output file (filename=logname)
set it to append rather than overwrite (filemode='a')
determine the format of the output message (format=...)
determine the format of the output time (datefmt='%H:%M:%S')
and determine the minimum message level it will accept (level=logging.DEBUG).
Taken from the "logging cookbook":
# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
And you're good to go.
P.S. Make sure to read the logging HOWTO as well.
Here is two examples, one print the logs (stdout) the other write the logs to a file:
import logging
import sys
logger = logging.getLogger()
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s')
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.setFormatter(formatter)
file_handler = logging.FileHandler('logs.log')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stdout_handler)
With this example, all logs will be printed and also be written to a file named logs.log
Use example:
logger.info('This is a log message!')
logger.error('This is an error message.')
List of all built-in logging handlers https://docs.python.org/3/library/logging.handlers.html
I prefer to use a configuration file. It allows me to switch logging levels, locations, etc without changing code when I go from development to release. I simply package a different config file with the same name, and with the same defined loggers.
import logging.config
if __name__ == '__main__':
# Configure the logger
# loggerConfigFileName: The name and path of your configuration file
logging.config.fileConfig(path.normpath(loggerConfigFileName))
# Create the logger
# Admin_Client: The name of a logger defined in the config file
mylogger = logging.getLogger('Admin_Client')
msg='Bite Me'
myLogger.debug(msg)
myLogger.info(msg)
myLogger.warn(msg)
myLogger.error(msg)
myLogger.critical(msg)
# Shut down the logger
logging.shutdown()
Here is my code for the log config file
#These are the loggers that are available from the code
#Each logger requires a handler, but can have more than one
[loggers]
keys=root,Admin_Client
#Each handler requires a single formatter
[handlers]
keys=fileHandler, consoleHandler
[formatters]
keys=logFormatter, consoleFormatter
[logger_root]
level=DEBUG
handlers=fileHandler
[logger_Admin_Client]
level=DEBUG
handlers=fileHandler, consoleHandler
qualname=Admin_Client
#propagate=0 Does not pass messages to ancestor loggers(root)
propagate=0
# Do not use a console logger when running scripts from a bat file without a console
# because it hangs!
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)# The comma is correct, because the parser is looking for args
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=logFormatter
# This causes a new file to be created for each script
# Change time.strftime("%Y%m%d%H%M%S") to time.strftime("%Y%m%d")
# And only one log per day will be created. All messages will be amended to it.
args=("D:\\Logs\\PyLogs\\" + time.strftime("%Y%m%d%H%M%S")+'.log', 'a')
[formatter_logFormatter]
#name is the name of the logger root or Admin_Client
#levelname is the log message level debug, warn, ect
#lineno is the line number from where the call to log is made
#04d is simple formatting to ensure there are four numeric places with leading zeros
#4s would work as well, but would simply pad the string with leading spaces, right justify
#-4s would work as well, but would simply pad the string with trailing spaces, left justify
#filename is the file name from where the call to log is made
#funcName is the method name from where the call to log is made
#format=%(asctime)s | %(lineno)d | %(message)s
#format=%(asctime)s | %(name)s | %(levelname)s | %(message)s
#format=%(asctime)s | %(name)s | %(module)s-%(lineno) | %(levelname)s | %(message)s
#format=%(asctime)s | %(name)s | %(module)s-%(lineno)04d | %(levelname)s | %(message)s
#format=%(asctime)s | %(name)s | %(module)s-%(lineno)4s | %(levelname)-8s | %(message)s
format=%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s
#Use a separate formatter for the console if you want
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(filename)s-%(funcName)s-%(lineno)04d | %(message)s
http://docs.python.org/library/logging.html#logging.basicConfig
logging.basicConfig(filename='/path/to/your/log', level=....)
here's a simpler way to go about it. this solution doesn't use a
config dictionary and uses a rotation file handler, like so:
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(handlers=[RotatingFileHandler(filename=logpath+filename,
mode='w', maxBytes=512000, backupCount=4)], level=debug_level,
format='%(levelname)s %(asctime)s %(message)s',
datefmt='%m/%d/%Y%I:%M:%S %p')
logger = logging.getLogger('my_logger')
or like so:
import logging
from logging.handlers import RotatingFileHandler
handlers = [ RotatingFileHandler(filename=logpath+filename,
mode='w',
maxBytes=512000,
backupCount=4)
]
logging.basicConfig(handlers=handlers,
level=debug_level,
format='%(levelname)s %(asctime)s %(message)s',
datefmt='%m/%d/%Y%I:%M:%S %p')
logger = logging.getLogger('my_logger')
the handlers variable needs to be an iterable. logpath+filename and debug_level are just variables holding the
respective info. of course, the values for the function params are up
to you.
the first time i was using the logging module i made the mistake of writing the following, which generates an OS file lock error (the
above is the solution to that):
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(filename=logpath+filename,
level=debug_level,
format='%(levelname)s %(asctime)s %(message)s',
datefmt='%m/%d/%Y%I:%M:%S %p')
logger = logging.getLogger('my_logger')
logger.addHandler(RotatingFileHandler(
filename=logpath+filename,
mode='w',
maxBytes=512000,
backupCount=4))
http://docs.python.org/library/logging.handlers.html#filehandler
The FileHandler class, located in the core logging package, sends logging output to a disk file.
This example should work fine. I have added streamhandler for console. Console
log and file handler data should be similar.
# MUTHUKUMAR_TIME_DATE.py #>>>>>>>> file name(module)
import sys
import logging
import logging.config
# ================== Logger ================================
def Logger(file_name):
formatter = logging.Formatter(fmt='%(asctime)s %(module)s,line: %(lineno)d %(levelname)8s | %(message)s',
datefmt='%Y/%m/%d %H:%M:%S') # %I:%M:%S %p AM|PM format
logging.basicConfig(filename = '%s.log' %(file_name),format= '%(asctime)s %(module)s,line: %(lineno)d %(levelname)8s | %(message)s',
datefmt='%Y/%m/%d %H:%M:%S', filemode = 'w', level = logging.INFO)
log_obj = logging.getLogger()
log_obj.setLevel(logging.DEBUG)
# log_obj = logging.getLogger().addHandler(logging.StreamHandler())
# console printer
screen_handler = logging.StreamHandler(stream=sys.stdout) #stream=sys.stdout is similar to normal print
screen_handler.setFormatter(formatter)
logging.getLogger().addHandler(screen_handler)
log_obj.info("Logger object created successfully..")
return log_obj
# =======================================================
MUTHUKUMAR_LOGGING_CHECK.py #>>>>>>>>>>> file name
# calling **Logger** function
file_name = 'muthu'
log_obj =Logger(file_name)
log_obj.info("yes hfghghg ghgfh".format())
log_obj.critical("CRIC".format())
log_obj.error("ERR".format())
log_obj.warning("WARN".format())
log_obj.debug("debug".format())
log_obj.info("qwerty".format())
log_obj.info("asdfghjkl".format())
log_obj.info("zxcvbnm".format())
# closing file
log_obj.handlers.clear()
OUTPUT:
2019/07/13 23:54:40 MUTHUKUMAR_TIME_DATE,line: 17 INFO | Logger object created successfully..
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 8 INFO | yes hfghghg ghgfh
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 9 CRITICAL | CRIC
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 10 ERROR | ERR
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 11 WARNING | WARN
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 12 DEBUG | debug
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 13 INFO | qwerty
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 14 INFO | asdfghjkl
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 15 INFO | zxcvbnm
Thanks,
import logging
from datetime import datetime
filename = datetime.now().strftime("%d-%m-%Y %H-%M-%S")#Setting the filename from current date and time
logging.basicConfig(filename=filename, filemode='a',
format="%(asctime)s, %(msecs)d %(name)s %(levelname)s [ %(filename)s-%(module)s-%(lineno)d ] : %(message)s",
datefmt="%H:%M:%S",
level=logging.DEBUG)
asctime
%(asctime)s
Human-readable time when the LogRecord was created. By default this
is of the form ‘2003-07-08 16:49:45,896’ (the numbers after the comma
are millisecond portion of the time).
created
%(created)f
Time when the LogRecord was created (as returned by time.time()).
exc_info
You shouldn’t need to format this yourself.
Exception tuple (à la sys.exc_info) or, if no exception has occurred,
None.
filename
%(filename)s
Filename portion of pathname.
funcName
%(funcName)s
Name of function containing the logging call.
levelname
%(levelname)s
Text logging level for the message ('DEBUG', 'INFO', 'WARNING',
'ERROR', 'CRITICAL').
levelno
%(levelno)s
Numeric logging level for the message (DEBUG, INFO, WARNING, ERROR,
CRITICAL).
lineno
%(lineno)d
Source line number where the logging call was issued (if available).
message
%(message)s
The logged message, computed as msg % args. This is set when
Formatter.format() is invoked.
module
%(module)s
Module (name portion of filename).
msecs
%(msecs)d
Millisecond portion of the time when the LogRecord was created.
msg
You shouldn’t need to format this yourself.
The format string passed in the original logging call. Merged with
args to produce message, or an arbitrary object (see Using arbitrary
objects as messages).
name
%(name)s
Name of the logger used to log the call.
pathname
%(pathname)s
Full pathname of the source file where the logging call was issued
(if available).
process
%(process)d
Process ID (if available).
processName
%(processName)s
Process name (if available).
relativeCreated
%(relativeCreated)d
Time in milliseconds when the LogRecord was created, relative to the
time the logging module was loaded.
stack_info
You shouldn’t need to format this yourself.
Stack frame information (where available) from the bottom of the
stack in the current thread, up to and including the stack frame of
the logging call which resulted in the creation of this record.
thread
%(thread)d
Thread ID (if available).
threadName
%(threadName)s
Thread name (if available).
Head over to official python3 page for more info regarding logging.
Although it is an old question, for people reaching this question these days, you can also use dictConfig. For example, for a file with info level and above :
logging.config.dictConfig({
'version': 1,
'formatters': {
'default': {
'format': '[%(asctime)s] %(message)s',
}
},
'handlers': {
'info': {
'level': logging.INFO,
'class': 'logging.FileHandler',
'filename': 'info.log',
},
},
"root": {
"level": logging.INFO,
"handlers": ["info"]
}
})
Or another example to be more specific, with rotating file and in a specific directory :
today = datetime.date.today()
folder = './log'
Path(folder).mkdir(parents=True, exist_ok=False) # Create folder if not exists
logging.config.dictConfig({
...
'info': {
'level': logging.INFO,
'class': 'logging.handlers.TimedRotatingFileHandler',
'filename': f'{folder}/info-{today.month:02}-{today.year}.log',
# Roll over on the first day of the weekday
'when': 'W0',
# Roll over at midnight
'atTime': datetime.time(hour=0),
# Number of files to keep.
'backupCount': 8
},
...
import sys
import logging
from util import reducer_logfile
logging.basicConfig(filename=reducer_logfile, format='%(message)s',
level=logging.INFO, filemode='w')
best and clean method.
ds = datetime.now().strftime("%Y%m%d_%H%M%S")
try:
# py39 有force参数指定可能强制除去之前的handler，这里使用兼容写法，0708
logging.getLogger().removeHandler(logging.getLogger().handlers[0])
logging.getLogger().removeHandler(logging.getLogger().handlers[0])
except:
pass
logging.basicConfig(
# force=
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler('log_%s_%s_%s.log' % (ds, create_net, os.path.basename(dataset_path))),
logging.StreamHandler(sys.stdout)
]
)
logging.info('start train')
Related
I have problems with logging in Python 3.7 I use Spyder as editor
This code is working normally and creates a file and writes in it.
import logging
LOG_FORMAT="%(Levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="C:\\Users\\MOHAMED\\Desktop\\Learn python\\tst.log",
level=logging.DEBUG)
logger=logging.getLogger()
logger.info("Our first message.")
The problem is when I add the format in my file this code does not write anything in tst file.
import logging
LOG_FORMAT="%(Levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="C:\\Users\\MOHAMED\\Desktop\\Learn python\\tst.log",
level=logging.DEBUG,
format=LOG_FORMAT)
logger=logging.getLogger()
logger.info("Our first message.")
You are specifying logging variable Levelname but you do not use the extra to populate the variable.
try it with
logger.info("Our first message.", extra={"Levelname":"test"})
also, recommend the docs https://docs.python.org/3/library/logging.html
Im working on a project with several modules that should all log to the same file.
Initializing the logger:
parentLogger = logging.getLogger('PARENTLOGGER')
logger = logging.getLogger('PARENTLOGGER.' + __name__)
#set format
fmt = logging.Formatter('%(asctime)s [%(levelname)s] %(name)s: %(message)s')
#set handler for command line
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(fmt)
#set file handler
open('data/logs.log', 'w+').close() #for creating the file in case it doesnt exists. I got exceptions otherwise.
fh = RotatingFileHandler('data/logs.log', maxBytes=5242880, backupCount=1)
fh.setLevel(logging.DEBUG)
fh.setFormatter(fmt)
parentLogger.addHandler(fh)
parentLogger.addHandler(ch)
#
than, on all other modules im calling:
self._logger = logging.getLogger('PARENTLOGGER.' + __name__)
The problem is that the rotating file handler wont write anything to the log file. In any module.
Am I configuring the logger correctly? I've tried several examples from pythons' logging cookbook without success...
Regards and thanks in advance!
You should tell logging when and what to log. Also, the log file will be created by default, no need to create it yourself. Here's a simple example:
$ cat test_log.py
import logging
log = '/home/user/test_log.log'
logging.basicConfig(filename=log,
format='%(asctime)s [%(levelname)s]: %(message)s at line: %(lineno)d (func: "%(funcName)s")')
try:
this_will_fail
except Exception as err:
logging.error('%s' % err)
$ python test_log.py
$ cat /home/user/test_log.log
2018-02-07 11:31:24,127 [ERROR]: name 'this_will_fail' is not defined at line: 10 (func: "<module>")
So, when I copy paste the following x times to the python prompt,
it add the log x times to the end of the designated file.
How can I change the code so that each time I copy paste this to the prompt,
I simply overwrite the existing file (the code seems to not accept the
mode = 'w' option or I do not seem to understand its meaning)
def MinimalLogginf():
import logging
import os
paths = {'work': ''}
logger = logging.getLogger('oneDayFileLoader')
LogHandler = logging.FileHandler(os.path.join(paths["work"] , "oneDayFileLoader.log"), mode='w')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
LogHandler.setFormatter(formatter)
logger.addHandler(LogHandler)
logger.setLevel(logging.DEBUG)
#Let's say this is an error:
if(1 == 1):
logger.error('overwrite')
So I run it once:
MinmalLoggingf()
Now, I want the new log file to overwrite the log file created on the previous run:
MinmalLoggingf()
If I understand correctly, you're running a certain Python process for days at a time, and want to rotate the log every day. I'd recommend you go a different route, using a handler that automatically rotates the log file, e.g. http://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/
But, if you want to control the log using the process in the same method you're comfortable with (Python console, pasting in code.. extremely unpretty and error prone, but sometimes quick-n-dirty is sufficient for the task at hand), well...
Your issue is that you create a new FileHandler each time you paste in the code, and you add it to the Logger object. You end up with a logger that has X FileHandlers attached to it, all of them writing to the same file. Try this:
import logging
paths = {'work': ''}
logger = logging.getLogger('oneDayFileLoader')
if logger.handlers:
logger.handlers[0].close()
logger.handlers = []
logHandler = logging.FileHandler(os.path.join(paths["work"] , "oneDayFileLoader.log"), mode='w')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
logger.setLevel(logging.DEBUG)
logger.error('overwrite')
Based on your request, I've also added an example using TimedRotatingFileHandler. Note I haven't tested it locally, so if you have issues ping back.
import logging
from logging.handlers import TimedRotatingFileHandler
logPath = os.path.join('', "fileLoaderLog")
logger = logging.getLogger('oneDayFileLoader')
logHandler = TimedRotatingFileHandler(logPath,
when="midnight",
interval=1)
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
logger.setLevel(logging.DEBUG)
logger.error('overwrite')
Your log messages are being duplicated because you call addHandler more than once. Each call to addHandler adds an additional log handler.
If you want to make sure the file is created from scratch, add an extra line of code to remove it:
os.remove(os.path.join(paths["work"], "oneDayFileLoader.log"))
The mode is specified as part of logging.basicConfig and is passed through using filemode.
logging.basicConfig(
level = logging.DEBUG,
format = '%(asctime)s %(levelname)s %(message)s',
filename = 'oneDayFileLoader.log,
filemode = 'w'
)
https://docs.python.org/3/library/logging.html#simple-examples
I'd like to have all timestamps in my log file to be UTC timestamp. When specified through code, this is done as follows:
import logging
import time
myHandler = logging.FileHandler('mylogfile.log', 'a')
formatter = logging.Formatter('%(asctime)s %(levelname)-8s %(name)-15s:%(lineno)4s: %(message)-80s')
formatter.converter = time.gmtime
myHandler.setFormatter(formatter)
myLogger = logging.getLogger('MyApp')
myLogger.addHandler(myHandler)
myLogger.setLevel(logging.DEBUG)
myLogger.info('here we are')
I'd like to move away from the above 'in-code' configuration to a config file based mechanism.
Here's the config file section for the formatter:
[handler_MyLogHandler]
args=("mylogfile.log", "a",)
class=FileHandler
level=DEBUG
formatter=simpleFormatter
Now, how do I specify the converter attribute (time.gmtime) in the above section?
Edit: The above config file is loaded thus:
logging.config.fileConfig('myLogConfig.conf')
Sadly, there is no way of doing this using the configuration file, other than having e.g. a
class UTCFormatter(logging.Formatter):
converter = time.gmtime
and then using a UTCFormatter in the configuration.
Here Vinay's solution applied to the logging.basicConfig:
import logging
import time
logging.basicConfig(filename='junk.log', level=logging.DEBUG, format='%(asctime)s: %(levelname)s:%(message)s')
logging.Formatter.converter = time.gmtime
logging.info('A message.')
From the Django documentation, here is an example format for logging:
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(module)s: %(message)s'
}
}
This prints something like:
ERROR 2012-05-22 14:33:07,261 views 42892 4398727168 hello
Is there a list of items you can include in the string formatting? For example, I'd like to be able to see the function and app where the message is being created, for example:
ERROR time myproject.myapp.views.login_function message
From Python logging module documentation:
asctime: %(asctime)s
Human-readable time when the LogRecord was created. By default this is of the form ‘2003-07-08 16:49:45,896’ (the numbers after the comma are millisecond portion of the time).
created: %(created)f
Time when the LogRecord was created (as returned by time.time()).
filename: %(filename)s
Filename portion of pathname.
funcName: %(funcName)s
Name of function containing the logging call.
levelname: %(levelname)s
Text logging level for the message ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL').
levelno: %(levelno)s
Numeric logging level for the message (DEBUG, INFO, WARNING, ERROR, CRITICAL).
lineno: %(lineno)d
Source line number where the logging call was issued (if available).
module: %(module)s
Module (name portion of filename).
msecs: %(msecs)d
Millisecond portion of the time when the LogRecord was created.
message: %(message)s
The logged message, computed as msg % args. This is set when Formatter.format() is invoked.
name: %(name)s
Name of the logger used to log the call.
pathname: %(pathname)s
Full pathname of the source file where the logging call was issued (if available).
process: %(process)d
Process ID (if available).
processName: %(processName)s
Process name (if available).
relativeCreated: %(relativeCreated)d
Time in milliseconds when the LogRecord was created, relative to the time the logging module was loaded.
thread: %(thread)d
Thread ID (if available).
threadName: %(threadName)s
Thread name (if available).
The following arguments are also available to Formatter.format(), although they are not intended to be included in the format string:
args:
The tuple of arguments merged into msg to produce message.
exc_info:
Exception tuple (à la sys.exc_info) or, if no exception has occurred, None.
msg:
The format string passed in the original logging call. Merged with args to produce message, or an arbitrary object (see Using arbitrary objects as messages).
Step1. Edit your settings.py file
$ cd mysite
$ vim mysite/settings.py
'formatters': {
'simple': {
'format': '%(levelname)s %(asctime)s %(name)s.%(funcName)s:%(lineno)s- %(message)s'
},
},
Step2. You should use logger in your coding like this:
import logging
logger = logging.getLogger(__name__)
def fn1() {
logger.info('great!')
logger.info(__name__)
}
Hope that helps you!