We Keep Coding

Python: Reading Logging Level from a Config File and avoiding previous DEBUG Information not getting lost

I have the following python code where logging level of a global logger is set through a Command-line argument:
logging.basicConfig (
level = getattr (logging, clArgs.logLevel),
...
If no logging level has been specified through CL argument, then by default INFO log level is used for global logging:
# Define '--loglevel' argument
clArgsParser.add_argument (
"-l", "--loglevel",
help = "Set the logging level of the program. The default log level is 'INFO'.",
default = 'INFO', # Default log level is 'INFO'
dest = "logLevel", # Store the argument value into the variable 'logLevel'
choices = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
)
Now I would like to give a 2nd option to the user, so that the logging level can be also specified in a Configuration File. Nonetheless, before the Configuration file is read out the script must first read out the CL arguments to figure out if the user has set the log level CL argument. Additionally, the configuration file path can be also specified through a CL argument.
After the script reads out the CL arguments and sets the logging level, it stores some logging information (e.g. log file location, directory where the script is being executed, etc.) as well it stores the DEBUG logging information while reading out the config file in the function readProgramConfig. The config file is read out at the very end (function readProgramConfig) as you can see in the code snippet below:
# Parse the Command-line Arguments
clArgs, progName = parseClArgs ( __program__ )
# Initialize Global Logging with Command-line Arguments
defaultLogFilePath = configLogging ( clArgs )
# Log Program Version used
logging.info ( f"Program version: {__program__.swVersion}")
# Log Script Start Time
logging.info ( f"Program start time: {programStartDate}" )
# Log the program's current directory
logging.debug ( f"Directory where the Program is being executed: {PROGRAM_DIR}" )
# Output the log file location
logging.debug ( f"Log file location: {defaultLogFilePath}")
# Read out the program configuration
programConfig = readProgramConfig ( clArgs.programConfigPath )
This leads to a problem - if no log level is specified through the CL argument and the log level is specified by the user in the config file (e.g. DEBUG), then the following will happen:
No CL-Arg for log level specified -> use INFO log level per default
Do logging (program version, program start time) before reading out the config file -> however, no DEBUG level information is logged as INFO is used by default
Additionally, no DEBUG information is logged while reading out the config file (function readProgramConfig)
Once the config file has been read out, the script will figure out that the config file wants to set the log level to DEBUG, and will then try to change the global logging level from INFO to DEBUG
From now on all DEBUG information will be logged, however the previous DEBUG information is lost, i.e. never logged
So it's sort of like a hen and egg problem.
I do have one solution in mind, but it is rather complicated and I would like to find out if someone of you have a simpler solution.
One possible solution would be:
Start the script with DEBUG log level by default to capture all log messages
Read out the config file:
2.1 If the log level in the config file is set to DEBUG, then continue logging into the log file with the DEBUG log level.
2.2 If the log level in the config file is set to lower log level than DEBUG (e.g. INFO), then delete all DEBUG entries in the log file, and continue logging only using the INFO log level.
You see the solution is rather complicated - it involves editing the log file and writing back and forth ... Not to mention that this approach will not work for logging into the console ...

One solution would be to use logging.config python module. You can read a config file (e.g. in JSON format) and store it as a dictionary. The module provides function logging.config.dictConfig which will configure the logger using the information from the config file, i.e. dictionary. The code for configuring the root logger would look something like this:
import logging # Logging
from logging import config as LogConfig # Logging Configuration
# Create the Log File name
logFileConfigName = "LogConfig.json"
logFileConfigPath = os.path.join ( PROGRAM_DIR, logFileConfigName )
# Open the JSON Config File
jsonLogConfigFile = open ( logFileConfigPath )
# Decode the JSON Config File
jsonLogConfig = json.load ( jsonLogConfigFile )
# Take the logging configuration from a dictionary
LogConfig.dictConfig ( jsonLogConfig )
The logger configuration would be stored in the LogConfig.json file. For both console and file logging there is a seperate handler defined in the config file:
{
"version": 1,
"root":
{
"handlers" : ["console", "file"],
"level": "DEBUG"
},
"handlers":
{
"console":
{
"formatter": "std_out",
"class": "logging.StreamHandler"
},
"file":
{
"filename" : ".\\Program.log",
"formatter" : "std_out",
"class": "logging.FileHandler",
"mode": "w"
}
},
"formatters":
{
"std_out":
{
"format": "%(asctime)s - %(levelname)s: %(message)s",
"datefmt":"%Y-%m-%d - %H:%M:%S"
}
}
}

Logging does not write anything in my file

I have problems with logging in Python 3.7 I use Spyder as editor
This code is working normally and creates a file and writes in it.
import logging
LOG_FORMAT="%(Levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="C:\\Users\\MOHAMED\\Desktop\\Learn python\\tst.log",
level=logging.DEBUG)
logger=logging.getLogger()
logger.info("Our first message.")
The problem is when I add the format in my file this code does not write anything in tst file.
import logging
LOG_FORMAT="%(Levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="C:\\Users\\MOHAMED\\Desktop\\Learn python\\tst.log",
level=logging.DEBUG,
format=LOG_FORMAT)
logger=logging.getLogger()
logger.info("Our first message.")

You are specifying logging variable Levelname but you do not use the extra to populate the variable.
try it with
logger.info("Our first message.", extra={"Levelname":"test"})
also, recommend the docs https://docs.python.org/3/library/logging.html

PermissionError when using python 3.3.4 and RotatingFileHandler

I am trying to get a rotating log file for a GUI application I am writing with python 3.3.4 and PyQt4.
I have the following snippet of code in my main script:
import logging
import resources
logger = logging.getLogger('main.test')
def main():
logger.setLevel(logging.DEBUG)
fh = RotatingFileHandler(resources.LOG_FILE_PATH, maxBytes=500, backupCount=5)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.info('main')
I have the maxBytes low so that I can test the rotating is working correctly, which it is not. I am getting the following error whenever the log should be rotated:
Traceback (most recent call last):
File "C:\Python33\lib\logging\handlers.py", line 73, in emit
self.doRollover()
File "C:\Python33\lib\logging\handlers.py", line 176, in doRollover
self.rotate(self.baseFilename, dfn)
File "C:\Python33\lib\logging\handlers.py", line 116, in rotate
os.rename(source, dest)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\myuser\\.logtest\\test.log.1'
And nothing is logged. Any help is much appreciated.
Thank you

Spent half a day on this as non previous answer resolved my issue.
My working solution is to use https://pypi.org/project/concurrent-log-handler/ instead of RotatingFileHandler. In multiple thread scenarios like Flask app, PermissionError will be raised when we rotate the log file that reaches maximum size.
Install pypiwin32 to get rid of No Module name win32con error.
Thanks go to https://www.programmersought.com/article/43941158027/

Instead of adding handler to the logger object you can directly specify handler in basicConfig(). If you add RotatingFileHandler to the logger object then one object might open the log file and another at the same time might try to rename it which throws the PermissionError.
Below code seems to work pretty well.
import logging
import resources
from logging.handlers import RotatingFileHandler
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[RotatingFileHandler(filename=resources.LOG_FILE_PATH, maxBytes=500, backupCount=5)])
logger = logging.getLogger('main.test')
def main():
logger.setLevel(logging.DEBUG)
logger.info('main')

In my case it happens only in Windows. To solve it I changed the delay parameter to True for my TimedRotatingFileHandler log handler.
Docs -> https://docs.python.org/3/library/logging.handlers.html#logging.handlers.TimedRotatingFileHandler

You cannot specify the same filename in both basicConfig() and RotatingFileHandler(). I had this same issue and I removed the filename parameter from basicConfig() and it now works.

In my case (Windows Server 2016 + IIS + FastCGI + Flask) finally I fix it by turning off files indexing in the folder.
how-to
Source:
https://stackoverflow.com/a/22467917/9199668
Btw, it was working for months correctly... I have no idea why...

Check that the file isn't being kept open by e.g. Windows file indexing, anti-virus or other software. Files that are open can't be renamed.

I changed the application to use dictConfig and created a separate file that holds the dictionary configuration. At the top of my main application I have:
from log.logger import LOGGING
logging.config.dictConfig(LOGGING)
logger = logging.getLogger('testlogging')
Then in log.logger I have:
import logging
import sys
import resources
LOGGING = {
"version":1,
"handlers":{
"fileHandler":{
"class":"logging.handlers.RotatingFileHandler",
"formatter":"myFormatter",
"filename":resources.LOG_FILE_PATH,
"maxBytes":100000,
"backupCount":5
},
"console":{
"class":"logging.StreamHandler",
"formatter":"myFormatter"
}
},
"loggers":{
"aoconnect":{
"handlers":["fileHandler", "console"],
"level":"DEBUG",
}
},
"formatters":{
"myFormatter":{
"format":"%(asctime)s - %(name)s - %(levelname)s - %(message)s"
}
}
}
This all seems to work pretty well.

In My case file size is full, after removing server.log file it worked
LOGS_DIR = os.path.join(BASE_DIR, 'logs')
LOGGING = {
'version': 1,
'handlers': {
'log_file': {
'level': 'INFO',
'class': 'logging.handlers.RotatingFileHandler',
'filename': os.path.join(LOGS_DIR, 'server.log'),
'backupCount': 10, # keep at most 10 log files
'maxBytes': 5*1024*1024 # 5242880 bytes (5MB)
},
},
'loggers': {
'django': {
'handlers':['log_file'],
'propagate': True,
'level':'INFO',
},
},
}

What is the point of setLevel in a python logging handler?

Let's say I have the following code:
import logging
import logging.handlers
a = logging.getLogger('myapp')
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
# The effective log level is still logging.WARN
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
I expect that setting logging.DEBUG on the handler would cause debug-level messages to be written to the log file. However, this prints 30 for the effective level (equal to logging.WARNING, the default), and only logs the warn message to the log file, not the debug message.
It appears that the handler's log level is being dropped on the floor, e.g. it's silently ignored. Which makes me wonder, why have setLevel on the handler at all?

It allows finer control. By default the root logger has WARNING level set, this means that it wont print messages with lower level(no matter how the handlers' levels are set!). But, if you set the root logger's level to DEBUG, indeed the message get sent to the log file:
import logging
import logging.handlers
a = logging.getLogger('myapp')
a.setLevel(logging.DEBUG) # set root's level
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
Now, image that you want to add a new handler that doesn't record debug information.
You can do this by simply setting the handler logging level:
import logging
import logging.handlers
a = logging.getLogger('myapp')
a.setLevel(logging.DEBUG) # set root's level
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
h2 = logging.handlers.RotatingFileHandler('foo2.log')
h2.setLevel(logging.WARNING)
a.addHandler(h2)
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
Now, the log file foo.log will contain both messages, while the file foo2.log will only contain the warning message. You could be interested in having a log file of only error-level messages, then simply add a Handler and set its level to logging.ERROR, everything using the same Logger.
You may think of the Logger logging level as a global restriction on which messages are "interesting" for a given logger and its handlers. The messages that are considered by the logger afterwards get sent to the handlers, which perform their own filtering and logging process.

In Python logging there are two different concepts: the level that the logger logs at and the level that the handler actually activates.
When a call to log is made, what is basically happening is:
if self.level <= loglevel:
for handler in self.handlers:
handler(loglevel, message)
While each of those handlers will then call:
if self.level <= loglevel:
# do something spiffy with the log!
If you'd like a real-world demonstration of this, you can look at Django's config settings. I'll include the relevant code here.
LOGGING = {
#snip
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'logging.NullHandler',
},
'console':{
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'simple'
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
'filters': ['special']
}
},
'loggers': {
#snip
'myproject.custom': {
# notice how there are two handlers here!
'handlers': ['console', 'mail_admins'],
'level': 'INFO',
'filters': ['special']
}
}
}
So, in the configuration above, only logs to getLogger('myproject.custom').info and above will get processed for logging. When that happens, the console will output all of the results (it will output everything because it is set to DEBUG level), while the mail_admins logger will happen for all ERRORs, FATALs and CRITICALs.
I suppose some code which isn't Django might help too:
import logging.handlers as hand
import logging as logging
# to make things easier, we'll name all of the logs by the levels
fatal = logging.getLogger('fatal')
warning = logging.getLogger('warning')
info = logging.getLogger('info')
fatal.setLevel(logging.FATAL)
warning.setLevel(logging.WARNING)
info.setLevel(logging.INFO)
fileHandler = hand.RotatingFileHandler('rotating.log')
# notice all three are re-using the same handler.
fatal.addHandler(fileHandler)
warning.addHandler(fileHandler)
info.addHandler(fileHandler)
# the handler should log everything except logging.NOTSET
fileHandler.setLevel(logging.DEBUG)
for logger in [fatal,warning,info]:
for level in ['debug','info','warning','error','fatal']:
method = getattr(logger,level)
method("Debug " + logger.name + " = " + level)
# now, the handler will only do anything for *fatal* messages...
fileHandler.setLevel(logging.FATAL)
for logger in [fatal,warning,info]:
for level in ['debug','info','warning','error','fatal']:
method = getattr(logger,level)
method("Fatal " + logger.name + " = " + level)
That results in:
Debug fatal = fatal
Debug warning = warning
Debug warning = error
Debug warning = fatal
Debug info = info
Debug info = warning
Debug info = error
Debug info = fatal
Fatal fatal = fatal
Fatal warning = fatal
Fatal info = fatal
Again, notice how info logged something at info, warning, error, and fatal when the log handler was set to DEBUG, but when the handler was set to FATAL all of a sudden only FATAL messages made it to the file.

Handlers represent different audiences for logging events. Levels on handlers are used to control the verbosity of output seen by a particular audience, and act in addition to any levels set on loggers. Levels on loggers are used to control the overall verbosity of logging from different parts of an application or library.
See this diagram for more information about how logging events are handled:

the rule
if and only if
handler.level <= message.level
&&
logger.level <= message.level
then the message prints.
Reminder: lower values are more verbose
Level | Numeric value
---------|--------------
CRITICAL | 50
ERROR | 40
WARNING | 30
INFO | 20
DEBUG | 10
NOTSET | 0
ref: https://docs.python.org/3/library/logging.html#logging-levels
in other words
if the logger is set to WARNING, it won't matter if the handler has a more verbose setting. it'll already be filtered by the time it gets to the handler.
a full example
import logging
handler_info = logging.StreamHandler()
handler_info.setLevel("INFO")
handler_info.setFormatter(logging.Formatter(
f"%(levelname)s message for %(name)s handled by handler_info: %(message)s"))
handler_debug = logging.StreamHandler()
handler_debug.setLevel("DEBUG")
handler_debug.setFormatter(logging.Formatter(
f"%(levelname)s message for %(name)s handled by handler_debug: %(message)s"))
logger_info = logging.getLogger('logger_info')
logger_info.setLevel("INFO")
logger_info.addHandler(handler_info)
logger_info.addHandler(handler_debug)
logger_debug = logging.getLogger('logger_debug')
logger_debug.setLevel("DEBUG")
logger_debug.addHandler(handler_info)
logger_debug.addHandler(handler_debug)
print()
print("output for `logger_info.info('hello')`")
logger_info.info("hello")
print()
print("output for `logger_info.debug('bonjour')`")
logger_info.debug("bonjour")
print()
print("output for `logger_debug.info('hola')`")
logger_debug.info("hola")
print()
print("output for `logger_debug.debug('ciao')`")
logger_debug.debug("ciao")
print()
which gives
output for `logger_info.info('hello')`
INFO message for logger_info handled by handler_info: hello
INFO message for logger_info handled by handler_debug: hello
output for `logger_info.debug('bonjour')`
# nothing, because message.level < logger.level
output for `logger_debug.info('hola')`
INFO message for logger_debug handled by handler_info: hola
INFO message for logger_debug handled by handler_debug: hola
output for `logger_debug.debug('ciao')`
DEBUG message for logger_debug handled by handler_debug: ciao
# nothing from handler_info, because message.level < handler.level

How to write to a file, using the logging Python module?

How can I use the logging module in Python to write to a file? Every time I try to use it, it just prints out the message.

An example of using logging.basicConfig rather than logging.fileHandler()
logging.basicConfig(filename=logname,
filemode='a',
format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
datefmt='%H:%M:%S',
level=logging.DEBUG)
logging.info("Running Urban Planning")
logger = logging.getLogger('urbanGUI')
In order, the five parts do the following:
set the output file (filename=logname)
set it to append rather than overwrite (filemode='a')
determine the format of the output message (format=...)
determine the format of the output time (datefmt='%H:%M:%S')
and determine the minimum message level it will accept (level=logging.DEBUG).

Taken from the "logging cookbook":
# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
And you're good to go.
P.S. Make sure to read the logging HOWTO as well.

Here is two examples, one print the logs (stdout) the other write the logs to a file:
import logging
import sys
logger = logging.getLogger()
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s')
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.setFormatter(formatter)
file_handler = logging.FileHandler('logs.log')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stdout_handler)
With this example, all logs will be printed and also be written to a file named logs.log
Use example:
logger.info('This is a log message!')
logger.error('This is an error message.')
List of all built-in logging handlers https://docs.python.org/3/library/logging.handlers.html

I prefer to use a configuration file. It allows me to switch logging levels, locations, etc without changing code when I go from development to release. I simply package a different config file with the same name, and with the same defined loggers.
import logging.config
if __name__ == '__main__':
# Configure the logger
# loggerConfigFileName: The name and path of your configuration file
logging.config.fileConfig(path.normpath(loggerConfigFileName))
# Create the logger
# Admin_Client: The name of a logger defined in the config file
mylogger = logging.getLogger('Admin_Client')
msg='Bite Me'
myLogger.debug(msg)
myLogger.info(msg)
myLogger.warn(msg)
myLogger.error(msg)
myLogger.critical(msg)
# Shut down the logger
logging.shutdown()
Here is my code for the log config file
#These are the loggers that are available from the code
#Each logger requires a handler, but can have more than one
[loggers]
keys=root,Admin_Client
#Each handler requires a single formatter
[handlers]
keys=fileHandler, consoleHandler
[formatters]
keys=logFormatter, consoleFormatter
[logger_root]
level=DEBUG
handlers=fileHandler
[logger_Admin_Client]
level=DEBUG
handlers=fileHandler, consoleHandler
qualname=Admin_Client
#propagate=0 Does not pass messages to ancestor loggers(root)
propagate=0
# Do not use a console logger when running scripts from a bat file without a console
# because it hangs!
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)# The comma is correct, because the parser is looking for args
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=logFormatter
# This causes a new file to be created for each script
# Change time.strftime("%Y%m%d%H%M%S") to time.strftime("%Y%m%d")
# And only one log per day will be created. All messages will be amended to it.
args=("D:\\Logs\\PyLogs\\" + time.strftime("%Y%m%d%H%M%S")+'.log', 'a')
[formatter_logFormatter]
#name is the name of the logger root or Admin_Client
#levelname is the log message level debug, warn, ect
#lineno is the line number from where the call to log is made
#04d is simple formatting to ensure there are four numeric places with leading zeros
#4s would work as well, but would simply pad the string with leading spaces, right justify
#-4s would work as well, but would simply pad the string with trailing spaces, left justify
#filename is the file name from where the call to log is made
#funcName is the method name from where the call to log is made
#format=%(asctime)s | %(lineno)d | %(message)s
#format=%(asctime)s | %(name)s | %(levelname)s | %(message)s
#format=%(asctime)s | %(name)s | %(module)s-%(lineno) | %(levelname)s | %(message)s
#format=%(asctime)s | %(name)s | %(module)s-%(lineno)04d | %(levelname)s | %(message)s
#format=%(asctime)s | %(name)s | %(module)s-%(lineno)4s | %(levelname)-8s | %(message)s
format=%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s
#Use a separate formatter for the console if you want
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(filename)s-%(funcName)s-%(lineno)04d | %(message)s

http://docs.python.org/library/logging.html#logging.basicConfig
logging.basicConfig(filename='/path/to/your/log', level=....)

here's a simpler way to go about it. this solution doesn't use a
config dictionary and uses a rotation file handler, like so:
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(handlers=[RotatingFileHandler(filename=logpath+filename,
mode='w', maxBytes=512000, backupCount=4)], level=debug_level,
format='%(levelname)s %(asctime)s %(message)s',
datefmt='%m/%d/%Y%I:%M:%S %p')
logger = logging.getLogger('my_logger')
or like so:
import logging
from logging.handlers import RotatingFileHandler
handlers = [ RotatingFileHandler(filename=logpath+filename,
mode='w',
maxBytes=512000,
backupCount=4)
]
logging.basicConfig(handlers=handlers,
level=debug_level,
format='%(levelname)s %(asctime)s %(message)s',
datefmt='%m/%d/%Y%I:%M:%S %p')
logger = logging.getLogger('my_logger')
the handlers variable needs to be an iterable. logpath+filename and debug_level are just variables holding the
respective info. of course, the values for the function params are up
to you.
the first time i was using the logging module i made the mistake of writing the following, which generates an OS file lock error (the
above is the solution to that):
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(filename=logpath+filename,
level=debug_level,
format='%(levelname)s %(asctime)s %(message)s',
datefmt='%m/%d/%Y%I:%M:%S %p')
logger = logging.getLogger('my_logger')
logger.addHandler(RotatingFileHandler(
filename=logpath+filename,
mode='w',
maxBytes=512000,
backupCount=4))

http://docs.python.org/library/logging.handlers.html#filehandler
The FileHandler class, located in the core logging package, sends logging output to a disk file.

This example should work fine. I have added streamhandler for console. Console
log and file handler data should be similar.
# MUTHUKUMAR_TIME_DATE.py #>>>>>>>> file name(module)
import sys
import logging
import logging.config
# ================== Logger ================================
def Logger(file_name):
formatter = logging.Formatter(fmt='%(asctime)s %(module)s,line: %(lineno)d %(levelname)8s | %(message)s',
datefmt='%Y/%m/%d %H:%M:%S') # %I:%M:%S %p AM|PM format
logging.basicConfig(filename = '%s.log' %(file_name),format= '%(asctime)s %(module)s,line: %(lineno)d %(levelname)8s | %(message)s',
datefmt='%Y/%m/%d %H:%M:%S', filemode = 'w', level = logging.INFO)
log_obj = logging.getLogger()
log_obj.setLevel(logging.DEBUG)
# log_obj = logging.getLogger().addHandler(logging.StreamHandler())
# console printer
screen_handler = logging.StreamHandler(stream=sys.stdout) #stream=sys.stdout is similar to normal print
screen_handler.setFormatter(formatter)
logging.getLogger().addHandler(screen_handler)
log_obj.info("Logger object created successfully..")
return log_obj
# =======================================================
MUTHUKUMAR_LOGGING_CHECK.py #>>>>>>>>>>> file name
# calling **Logger** function
file_name = 'muthu'
log_obj =Logger(file_name)
log_obj.info("yes hfghghg ghgfh".format())
log_obj.critical("CRIC".format())
log_obj.error("ERR".format())
log_obj.warning("WARN".format())
log_obj.debug("debug".format())
log_obj.info("qwerty".format())
log_obj.info("asdfghjkl".format())
log_obj.info("zxcvbnm".format())
# closing file
log_obj.handlers.clear()
OUTPUT:
2019/07/13 23:54:40 MUTHUKUMAR_TIME_DATE,line: 17 INFO | Logger object created successfully..
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 8 INFO | yes hfghghg ghgfh
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 9 CRITICAL | CRIC
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 10 ERROR | ERR
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 11 WARNING | WARN
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 12 DEBUG | debug
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 13 INFO | qwerty
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 14 INFO | asdfghjkl
2019/07/13 23:54:40 MUTHUKUMAR_LOGGING_CHECK,line: 15 INFO | zxcvbnm
Thanks,

import logging
from datetime import datetime
filename = datetime.now().strftime("%d-%m-%Y %H-%M-%S")#Setting the filename from current date and time
logging.basicConfig(filename=filename, filemode='a',
format="%(asctime)s, %(msecs)d %(name)s %(levelname)s [ %(filename)s-%(module)s-%(lineno)d ] : %(message)s",
datefmt="%H:%M:%S",
level=logging.DEBUG)
asctime
%(asctime)s
Human-readable time when the LogRecord was created. By default this
is of the form ‘2003-07-08 16:49:45,896’ (the numbers after the comma
are millisecond portion of the time).
created
%(created)f
Time when the LogRecord was created (as returned by time.time()).
exc_info
You shouldn’t need to format this yourself.
Exception tuple (à la sys.exc_info) or, if no exception has occurred,
None.
filename
%(filename)s
Filename portion of pathname.
funcName
%(funcName)s
Name of function containing the logging call.
levelname
%(levelname)s
Text logging level for the message ('DEBUG', 'INFO', 'WARNING',
'ERROR', 'CRITICAL').
levelno
%(levelno)s
Numeric logging level for the message (DEBUG, INFO, WARNING, ERROR,
CRITICAL).
lineno
%(lineno)d
Source line number where the logging call was issued (if available).
message
%(message)s
The logged message, computed as msg % args. This is set when
Formatter.format() is invoked.
module
%(module)s
Module (name portion of filename).
msecs
%(msecs)d
Millisecond portion of the time when the LogRecord was created.
msg
You shouldn’t need to format this yourself.
The format string passed in the original logging call. Merged with
args to produce message, or an arbitrary object (see Using arbitrary
objects as messages).
name
%(name)s
Name of the logger used to log the call.
pathname
%(pathname)s
Full pathname of the source file where the logging call was issued
(if available).
process
%(process)d
Process ID (if available).
processName
%(processName)s
Process name (if available).
relativeCreated
%(relativeCreated)d
Time in milliseconds when the LogRecord was created, relative to the
time the logging module was loaded.
stack_info
You shouldn’t need to format this yourself.
Stack frame information (where available) from the bottom of the
stack in the current thread, up to and including the stack frame of
the logging call which resulted in the creation of this record.
thread
%(thread)d
Thread ID (if available).
threadName
%(threadName)s
Thread name (if available).
Head over to official python3 page for more info regarding logging.

Although it is an old question, for people reaching this question these days, you can also use dictConfig. For example, for a file with info level and above :
logging.config.dictConfig({
'version': 1,
'formatters': {
'default': {
'format': '[%(asctime)s] %(message)s',
}
},
'handlers': {
'info': {
'level': logging.INFO,
'class': 'logging.FileHandler',
'filename': 'info.log',
},
},
"root": {
"level": logging.INFO,
"handlers": ["info"]
}
})
Or another example to be more specific, with rotating file and in a specific directory :
today = datetime.date.today()
folder = './log'
Path(folder).mkdir(parents=True, exist_ok=False) # Create folder if not exists
logging.config.dictConfig({
...
'info': {
'level': logging.INFO,
'class': 'logging.handlers.TimedRotatingFileHandler',
'filename': f'{folder}/info-{today.month:02}-{today.year}.log',
# Roll over on the first day of the weekday
'when': 'W0',
# Roll over at midnight
'atTime': datetime.time(hour=0),
# Number of files to keep.
'backupCount': 8
},
...

import sys
import logging
from util import reducer_logfile
logging.basicConfig(filename=reducer_logfile, format='%(message)s',
level=logging.INFO, filemode='w')

best and clean method.
ds = datetime.now().strftime("%Y%m%d_%H%M%S")
try:
# py39 有force参数指定可能强制除去之前的handler，这里使用兼容写法，0708
logging.getLogger().removeHandler(logging.getLogger().handlers[0])
logging.getLogger().removeHandler(logging.getLogger().handlers[0])
except:
pass
logging.basicConfig(
# force=
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler('log_%s_%s_%s.log' % (ds, create_net, os.path.basename(dataset_path))),
logging.StreamHandler(sys.stdout)
]
)
logging.info('start train')