We Keep Coding

Multi module python logger using the name of the main module

I have multiple python modules that I'd like to use the same logger while preserving the call hierarchy in those logs. I'd also like to do this with a logger whose name is the name of the calling module (or calling module stack). I haven't been able to work out how to get the name of the calling module except with messing with the stack trace, but that doesn't feel very pythonic.
Is this possible?
main.py
import logging
from sub_module import sub_log
logger = logging.getLogger(__name__)
logger.info("main_module")
sub_log()
sub_module.py
import logging
def sub_log():
logger = logging.getLogger(???)
logger.info("sub_module")
Desired Output
TIME main INFO main_module
TIME main.sub_module INFO sub_module

To solve your problem pythonic use the Logger Formatter:
For reference check the
Logging Docs
main.py
import logging
from submodule import sub_log
from submodule2 import sub_log2
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('test.log')
fh.setLevel(logging.DEBUG)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s %(name)s.%(module)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(fh)
sub_log("test")
sub_log2("test")
submodule.py
import logging
import __main__
def sub_log(msg):
logger = logging.getLogger(__main__.__name__)
logger.info(msg)
I've created second submodule. ( same code other name)
My Results:
2018-10-16 20:41:23,860 __main__.submodule - INFO - test
2018-10-16 20:41:23,860 __main__.submodule2 - INFO - test
I hope this will help you :)

How to use Python's RotatingFileHandler

I'm trying to do a test run of the logging module's RotatingFileHandler as follows:
import logging
from logging.handlers import RotatingFileHandler
# logging.basicConfig(filename="example.log", level=logging.DEBUG)
logger = logging.getLogger('my_logger')
handler = RotatingFileHandler("my_log.log", maxBytes=2000, backupCount=10)
logger.addHandler(handler)
for _ in range(10000):
logger.debug("Hello, world!")
However, with logging.basicConfig line commented out, the resulting my_log.log file contains no data:
If I comment in the line with logging.basicConfig(filename="example.log", level=logging.DEBUG), I get the expected my_log.log files with numbered suffixes. However, there is also the example.log which is a (relatively) large file:
How can I set up the logging so that it only generates the my_log.log files, and not the large example.log file?

Python provides 5 logging levels out of the box (in increasing order of severity): DEBUG, INFO, WARNING, ERROR and CRITICAL. The default one is WARNING. The docs says, that
Logging messages which are less severe than lvl will be ignored.
So if you use .debug with the default settings, you won't see anything in your logs.
The easiest fix would be to use logger.warning function rather than logger.debug:
import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger('my_logger')
handler = RotatingFileHandler('my_log.log', maxBytes=2000, backupCount=10)
logger.addHandler(handler)
for _ in range(10000):
logger.warning('Hello, world!')
And if you want to change logger level you can use .setLevel method:
import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)
handler = RotatingFileHandler('my_log.log', maxBytes=2000, backupCount=10)
logger.addHandler(handler)
for _ in range(10000):
logger.debug('Hello, world!')

Going off of Kurt Peek's answer you can also put the rotating file handler in the logging.basicConfig directly
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(
handlers=[RotatingFileHandler('./my_log.log', maxBytes=100000, backupCount=10)],
level=logging.DEBUG,
format="[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
datefmt='%Y-%m-%dT%H:%M:%S')

All previous answers are correct, here another way of doing the same thing except we use logging config file instead.
logging_config.ini
Here is the config file :
[loggers]
keys=root
[handlers]
keys=logfile
[formatters]
keys=logfileformatter
[logger_root]
level=DEBUG
handlers=logfile
[formatter_logfileformatter]
format=%(asctime)s %(name)-12s: %(levelname)s %(message)s
[handler_logfile]
class=handlers.RotatingFileHandler
level=DEBUG
args=('testing.log','a',10,100)
formatter=logfileformatter
myScrypt.py
here is simple logging script that uses the above config file
import logging
from logging.config import fileConfig
fileConfig('logging_config.ini')
logger = logging.getLogger()
logger.debug('the best scripting language is python in the world')
RESULT
here is the result, notice maxBytes is set to 10 but in real life, that's clearly too small.
(args=('testing.log','a',10,100)

I found that to obtain the desired behavior one has to use the same name in the basicConfig and RotatingFileHandler initializations:
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(filename="my_log.log", level=logging.DEBUG)
logger = logging.getLogger('my_logger')
handler = RotatingFileHandler("my_log.log", maxBytes=2000, backupCount=10)
logger.addHandler(handler)
for _ in range(10000):
logger.debug("Hello, world!")
Here, I have chose the same name my_log.log. This results in only the 'size-limited' logs being created:

Logging and inheritance of loggers' configurations in Python

I come from SLF4J and Log4J, so that might be the reason why I don't get how logging works in Python.
I have the following
---- logging.yaml
version: 1
handlers:
console:
class: logging.StreamHandler
level: DEBUG
stream: ext://sys.stderr
formatter: simpleFormatter
file:
class: logging.FileHandler
filename: app.log
mode: w
level: DEBUG
formatter: simpleFormatter
formatters:
simpleFormatter:
#class: !!python/name:logging.Formatter
#class: logging.Formatter
format: '%(name)s %(asctime)s %(levelname)s %(message)s'
datefmt: '%d/%m/%Y %H:%M:%S'
root:
level: INFO
handlers: [console, file]
mod:
level: DEBUG
----- mod.py
import logging
def foo ():
log = logging.getLogger ( __name__ )
log.debug ( 'Hello from the module' )
---- main.py
from logging.config import dictConfig
import yaml
with open ( 'logging.yaml' ) as flog:
dictConfig ( yaml.load ( flog ) )
import logging
from mod import foo
if __name__ == '__main__':
log = logging.getLogger ( __name__ )
log.debug ( 'Hello from main' )
foo ()
With the config above, I would expect to see only the message 'Hello from the module'. Instead, nothing is printed. When I set DEBUG for the root logger, both messages are printed.
So, aren't the messages forwarded to the upper loggers? Isn't the mod logger a child of root? Doesn't the mod logger inherit the handlers configuration? (I've tried to repeat handlers in mod, but nothing changes).
How can I achieve a configuration saying: default level is INFO, the level for this module and sub-modules is DEBUG, everything goes to the handlers defined for root?

You have a fairly simple error: note that, per the docs, configuration for loggers other than root should be under the loggers key as:
a dict in which each key is a logger name and each value is a dict
describing how to configure the corresponding Logger instance
Adding this key and indenting the appropriate lines, to give:
loggers:
mod:
level: DEBUG
works as expected:
$ python main.py
mod 20/07/2016 14:35:32 DEBUG Hello from the module
$ cat app.log
mod 20/07/2016 14:35:32 DEBUG Hello from the module

python logging file is not working when using logging.basicConfig

I have the following lines of code that initialize logging.
I comment one out and leave the other to be used.
The problem that I'm facing is that the one that is meant to log to the file not logging to file. It is instead logging to the console.
Please help.
For logging to Console:
logging.basicConfig(level=logging.INFO,
format='%(asctime)s [%(levelname)s] (%(threadName)-10s) %(message)s',)
for file logging
logging.basicConfig(filename='server-soap.1.log',level=logging.INFO,
format='%(asctime)s [%(levelname)s] (%(threadName)-10s) %(message)s')

I found out what the problem was.
It was in the ordering of the imports and the logging definition.
The effect of the poor ordering was that the libraries that I imported before defining the logging using logging.basicConfig() defined the logging. This therefore took precedence to the logging that I was trying to define later using logging.basicConfig()
Below is how I needed to order it:
import logging
## for file logging
logging.basicConfig(filename='server-soap.1.log',
level=logging.INFO,
format='%(asctime)s %(levelname)s %(threadName)-10s %(message)s',)
from pysimplesoap.server import SoapDispatcher, SOAPHandler
from BaseHTTPServer import HTTPServer
import time,random,datetime,pytz,sys,threading
from datetime import timedelta
#DB
import psycopg2, psycopg2.extras
from psycopg2.pool import ThreadedConnectionPool
#ESB Call
from suds import WebFault
from suds.client import Client
But the faulty ordering that I initially had was:
from pysimplesoap.server import SoapDispatcher, SOAPHandler
from BaseHTTPServer import HTTPServer
import logging
import time,random,datetime,pytz,sys,threading
from datetime import timedelta
#DB
import psycopg2, psycopg2.extras
from psycopg2.pool import ThreadedConnectionPool
#ESB Call
from suds import WebFault
from suds.client import Client
## for file logging
logging.basicConfig(filename='server-soap.1.log',
level=logging.INFO,
format='%(asctime)s %(levelname)s %(threadName)-10s %(message)s',)

"Changed in version 3.8: The force argument was added." I think it's a better choice for new version.
For older Version(< 3.8):
From the source code of logging I found the flows:
This function does nothing if the root logger already has handlers
configured. It is a convenience method intended for use by simple scripts
to do one-shot configuration of the logging package.
So, if some module we import called the basicConfig() method before us, our call will do nothing.
A solution I found can work is that you can reload logging before your own calling to basicConfig(), such as
def init_logger(*, fn=None):
# !!! here
from imp import reload # python 2.x don't need to import reload, use it directly
reload(logging)
logging_params = {
'level': logging.INFO,
'format': '%(asctime)s__[%(levelname)s, %(module)s.%(funcName)s](%(name)s)__[L%(lineno)d] %(message)s',
}
if fn is not None:
logging_params['filename'] = fn
logging.basicConfig(**logging_params)
logging.error('init basic configure of logging success')

In case basicConfig() does not work:
logger = logging.getLogger('Spam Logger')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
fh.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(ch)
logger.addHandler(fh)
# 'application' code
logger.debug('debug Spam message')
logging.debug('debug Spam message')
logger.info('info Ham message')
logger.warning('warn Eggs message')
logger.error('error Spam and Ham message')
logger.critical('critical Ham and Eggs message')
which gives me the following output:
2019-06-20 11:33:48,967 - Spam Logger - DEBUG - debug Spam message
2019-06-20 11:33:48,968 - Spam Logger - INFO - info Ham message
2019-06-20 11:33:48,968 - Spam Logger - WARNING - warn Eggs message
2019-06-20 11:33:48,968 - Spam Logger - ERROR - error Spam and Ham message
2019-06-20 11:33:48,968 - Spam Logger - CRITICAL - critical Ham and Eggs message
For the sake of reference, Python Logging Cookbook is readworthy.

I got the same error, I fixed it by passing the following argument to the basic config.
logging.basicConfig(
level="WARNING",
format="%(asctime)s - %(name)s - [ %(message)s ]",
datefmt='%d-%b-%y %H:%M:%S',
force=True,
handlers=[
logging.FileHandler("debug.log"),
logging.StreamHandler()
])
Here as you can see passing force=True overrides any other BasicConfigs

Another solution that worked for me is instead of tracing down which module might be importing logging or even calling basicConfig before me is to just call setLevel after basicConfig again.
import os
import logging
RUNTIME_DEBUG_LEVEL = os.environ.get('RUNTIME_DEBUG_LEVEL').upper()
LOGGING_KWARGS = {
'level': getattr(logging, RUNTIME_DEBUG_LEVEL)
}
logging.basicConfig(**LOGGING_KWARGS)
logging.setLevel(getattr(logging, RUNTIME_DEBUG_LEVEL))
Sort of crude, seems hacky, fixed my problem, worth a share.

IF YOU JUST WANT TO SET THE LOG LEVEL OF ALL LOGGERS
instead of ordering your imports after logging config:
just set level on the root level:
# Option 1:
logging.root.setLevel(logging.INFO)
# Option 2 - make it configurable:
# env variable + default value INFO
logging.root.setLevel(os.environ.get('LOG_LEVEL', logging.INFO))

Like vacing's answer mentioned, basicConfig has no effect if the root logger already has handlers configured.
I was using pytest which seems to set handlers which means the default logging setup with loglevel WARNING is active -- so it appears my app fails to log, but this only happens when executing unit tests with pytest. In a normal app run logs are produced as expected which is enough for my use case.

Get Output From the logging Module in IPython Notebook

When I running the following inside IPython Notebook I don't see any output:
import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug("test")
Anyone know how to make it so I can see the "test" message inside the notebook?

Try following:
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")
According to logging.basicConfig:
Does basic configuration for the logging system by creating a
StreamHandler with a default Formatter and adding it to the root
logger. The functions debug(), info(), warning(), error() and
critical() will call basicConfig() automatically if no handlers are
defined for the root logger.
This function does nothing if the root logger already has handlers
configured for it.
It seems like ipython notebook call basicConfig (or set handler) somewhere.

If you still want to use basicConfig, reload the logging module like this
from importlib import reload # Not needed in Python 2
import logging
reload(logging)
logging.basicConfig(format='%(asctime)s %(levelname)s:%(message)s', level=logging.DEBUG, datefmt='%I:%M:%S')

My understanding is that the IPython session starts up logging so basicConfig doesn't work. Here is the setup that works for me (I wish this was not so gross looking since I want to use it for almost all my notebooks):
import logging
logger = logging.getLogger()
fhandler = logging.FileHandler(filename='mylog.log', mode='a')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.setLevel(logging.DEBUG)
Now when I run:
logging.error('hello!')
logging.debug('This is a debug message')
logging.info('this is an info message')
logging.warning('tbllalfhldfhd, warning.')
I get a "mylog.log" file in the same directory as my notebook that contains:
2015-01-28 09:49:25,026 - root - ERROR - hello!
2015-01-28 09:49:25,028 - root - DEBUG - This is a debug message
2015-01-28 09:49:25,029 - root - INFO - this is an info message
2015-01-28 09:49:25,032 - root - WARNING - tbllalfhldfhd, warning.
Note that if you rerun this without restarting the IPython session it will write duplicate entries to the file since there would now be two file handlers defined

Bear in mind that stderr is the default stream for the logging module, so in IPython and Jupyter notebooks you might not see anything unless you configure the stream to stdout:
import logging
import sys
logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s',
level=logging.INFO, stream=sys.stdout)
logging.info('Hello world!')

What worked for me now (Jupyter, notebook server is: 5.4.1, IPython 7.0.1)
import logging
logging.basicConfig()
logger = logging.getLogger('Something')
logger.setLevel(logging.DEBUG)
Now I can use logger to print info, otherwise I would see only message from the default level (logging.WARNING) or above.

You can configure logging by running %config Application.log_level="INFO"
For more information, see IPython kernel options

I wanted a simple and straightforward answer to this, with nicely styled output so here's my recommendation
import sys
import logging
logging.basicConfig(
format='%(asctime)s [%(levelname)s] %(name)s - %(message)s',
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S',
stream=sys.stdout,
)
log = logging.getLogger('notebook')
Then you can use log.info() or any of the other logging levels anywhere in your notebook with output that looks like this
2020-10-28 17:07:08 [INFO] notebook - Hello world
2020-10-28 17:12:22 [INFO] notebook - More info here
2020-10-28 17:12:22 [INFO] notebook - And some more

As of logging version 3.8 a force parameter has been added that removes any existing handlers, which allows basicConfig to work. This worked on IPython version 7.29.0 and Jupyter Lab version 3.2.1.
import logging
logging.basicConfig(level=logging.DEBUG,
force = True)
logging.debug("test")

I setup a logger for both file and I wanted it to show up on the notebook. Turns out adding a filehandler clears out the default stream handlder.
logger = logging.getLogger()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# Setup file handler
fhandler = logging.FileHandler('my.log')
fhandler.setLevel(logging.DEBUG)
fhandler.setFormatter(formatter)
# Configure stream handler for the cells
chandler = logging.StreamHandler()
chandler.setLevel(logging.DEBUG)
chandler.setFormatter(formatter)
# Add both handlers
logger.addHandler(fhandler)
logger.addHandler(chandler)
logger.setLevel(logging.DEBUG)
# Show the handlers
logger.handlers
# Log Something
logger.info("Test info")
logger.debug("Test debug")
logger.error("Test error")

setup
import logging
# make a handler
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
# add it to the root logger
logging.getLogger().addHandler(handler)
log from your own logger
# make a logger for this notebook, set verbosity
logger = logging.getLogger(__name__)
logger.setLevel('DEBUG')
# send messages
logger.debug("debug message")
logger.info("so much info")
logger.warning("you've veen warned!")
logger.error("bad news")
logger.critical("really bad news")
2021-09-02 18:18:27,397 - __main__ - DEBUG - debug message
2021-09-02 18:18:27,397 - __main__ - INFO - so much info
2021-09-02 18:18:27,398 - __main__ - WARNING - you've veen warned!
2021-09-02 18:18:27,398 - __main__ - ERROR - bad news
2021-09-02 18:18:27,399 - __main__ - CRITICAL - really bad news
capture logging from other libraries
logging.getLogger('google').setLevel('DEBUG')
from google.cloud import storage
client = storage.Client()
2021-09-02 18:18:27,415 - google.auth._default - DEBUG - Checking None for explicit credentials as part of auth process...
2021-09-02 18:18:27,416 - google.auth._default - DEBUG - Checking Cloud SDK credentials as part of auth process...
2021-09-02 18:18:27,416 - google.auth._default - DEBUG - Cloud SDK credentials not found on disk; not using them
...