We Keep Coding

logging information is being sent to more files than specified by the handler

In an attempt to learn the Python module logging I made the small script below.
However, whenever I use the object my_logger it outputs to the file my_logger.log as specified by the file handler, but it also outputs the same text to the previously specified file in basicConfig log.log. My question is, why is it outputting to both locations instead of just the file specified by the FileHandler?
import logging
logging.basicConfig(level=logging.INFO,
filename='log.log',
filemode="w",
format="%(levelname)s - %(message)s")
logging.debug("debug_message") # Lowest |
logging.info("info_message") # |
logging.warning("warning_message") # |
logging.error("error_message") # |
logging.critical("critical_message") # Highest V
my_logger = logging.getLogger('My_Logger')
my_logger.info("Successfully created my custom logger")
handler = logging.FileHandler("my_logger.log")
formatter = logging.Formatter("%(name)s: %(levelname)s - %(message)s")
handler.setFormatter(formatter)
my_logger.addHandler(handler)
my_logger.info("Successfully created My Logger!")
try:
1/0
except ZeroDivisionError as e:
my_logger.exception("ZeroDivisionError")

You have two loggers in your script. One is the root logger that is setup using basiConfig and accessed using logging keyword itself. Another logger is "My_Logger" which is a child of root logger. Both loggers are active in your script and are instructed to write to specified handlers.
That's why you are getting logs written in both files.
If you want to use a customized handler than don't use basicConfig. Use only one logger in the whole of your module.

How to get current logging formatter?

I'm using the logging module from the python standard library and would like to obtain the current Formatter. The reason is that I'm using multiprocessing module and for each process I'd like to assign its logger another file handler to log to its own log file. When I do this in the following way
logger = logging.getLogger('subprocess')
log_path = 'log.txt'
with open(log_path, 'a') as outfile:
handler = logging.StreamHandler(outfile)
logger.addHandler(handler)
the messages in log.txt have no formatting at all, but I would like the message to be in the same format as my typical logging format. My typical logging setup is shown below
logging.basicConfig(
format='%(asctime)s | %(levelname)s | %(name)s - %(process)d | %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
level=os.environ.get('LOGLEVEL', 'INFO').upper(),
)
logger = logging.getLogger(__name__)
Since it looks like the Formatter object is associated with the Handler object, I tried to obtain handler from the main Logger object which has the correct formatting. So I called
logger.handlers
but I got an empty list [].
So my question is, where do I get the Formatter object which has the same format that my main logger has?
For the record, I'm using python 3.8 on macOS but the code will be deployed to Linux (but still python 3.8).

Judging by the cpython source code on logging.basicConfig, it appears that the formatter object which contains your formatting string is eventually added to a handler that is passed to the root logger (see: here ). So you can obtain the handler (and therefore the formatter) from the root logger object by doing
logging.root.handlers[0].formatter
In particular, to achieve the subprocess logging handler in the question, you can do this instead
handler = logging.StreamHandler(outfile)
handler.setFormatter(logging.root.handlers[0].formatter)
handler.setLevel(logging.root.level)
logger.addHandler(handler)
which will set your handler to the same logging format (and level as well!) as that of your usual logger object.

Can i set logging level of all loaded modules at once?

main module
module A
module B
The main module uses the functions of modules.
The functions of modules include logger.info or logger.warning.. so on to show what i made wrong about the code.
Objective :
- Logging all things in main, A and B.
- A faculty to set logging level of A and B in main jupyter notebook, at the moment. (e.g. When i need more information about a function of A, then increase logging level to DEBUG from INFO)
By the way, the main script has:
import logging, sys
# create logger
logger = logging.getLogger('logger')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
fh = logging.FileHandler('process.log')
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s\n')
# add formatter to ch
fh.setFormatter(formatter)
# add ch to logger
logger.addHandler(fh)
logger.addHandler(ch)
I want to use Logger object and configure this object, instead of logging's baseconfig. But if i can't, other ways is ok.

If you do:
logger = logging.getLogger('logger')
Into Module A and Module B, then they should have access to the same logger as your main file. From there, you can set whatever level at any time you want. E.g.
# ../module_a.py
import logging
logger = logging.getLogger('logger')
logger.setLevel(whatever) # Now all instances of "logger" will be set to that level.
Basically, loggers are globally registered by name and accessible through the logging module directly from any other modules.

Python Logging: Provide log file path from main module

Is there any way I can provide the filename for logger from my main module?
I am using following way, however it's not working.all the logs go to xyz.log file rather than main.log
Updated as per suggestion from nosklo
logger.py
formatter = logging.Formatter(fmt='[%(asctime)s] - {%(filename)s:%(lineno)d} %(levelname)s - %(message)s')
def _get_file_handler(file_name="xyz.log"):
file_handler = logging.FileHandler(file_name)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
return file_handler
def get_logger(name):
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
logger.addHandler(_get_file_handler())
return logger
parser.py
log = logger.get_logger(__name__)
def parse():
log.info("is there anyway this could go to main.log and xyz.log")
main.py
log = logger.get_logger(__name__)
if __name__ == '__main__':
for handler in log.handlers:
if isinstance(handler, logging.FileHandler):
log.removeHandler(handler)
log.addHandler(logger._get_file_handler())
log.info("is there anyway this could go to main.log and xyz.log?")
parser.parse()
Is there a way I can set the Log file name from my main.py module and not from logger.py module?

You're calling get_logger() first, so when you set the class attribute in FileName.file_name = "main.log" the get_logger function is already finished, and the logger is already defined to write in xyz.log; Changing the variable later won't change the logger anymore, since it is already defined.
To change the previously selected file, you'd have to retrieve the logger, remove the previous handler and add a new file handler. Another option is to set the variable before calling get_logger() so when you call it, the variable already has the correct value.

Logging instances can have multiple file handlers. Use a function like this to just add another handler with the additional output path you want. Log messages will get sent to both (or all) text logs added to the instance. You can even configure the handlers to have different logging levels so you can filter messages to different logs for critical errors, info message, etc.
import logging
def add_handler(output_log_path, log):
# Set up text logger and add it to logging instance
file_logger = logging.FileHandler(output_log_path)
file_logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s | logger name: %(name)s | module: %(module)s | lineno: %(lineno)d | %(message)s')
file_logger.setFormatter(formatter)
log.addHandler(file_logger)
return log

Extending the Python Logger

I'm looking for a simple way to extend the logging functionality defined in the standard python library. I just want the ability to choose whether or not my logs are also printed to the screen.
Example: Normally to log a warning you would call:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s', filename='log.log', filemode='w')
logging.warning("WARNING!!!")
This sets the configurations of the log and puts the warning into the log
I would like to have something along the lines of a call like:
logging.warning("WARNING!!!", True)
where the True statement signifys if the log is also printed to stdout.
I've seen some examples of implementations of overriding the logger class
but I am new to the language and don't really follow what is going on, or how to implement this idea. Any help would be greatly appreciated :)

The Python logging module defines these classes:
Loggers that emit log messages.
Handlers that put those messages to a destination.
Formatters that format log messages.
Filters that filter log messages.
A Logger can have Handlers. You add them by invoking the addHandler() method. A Handler can have Filters and Formatters. You similarly add them by invoking the addFilter() and setFormatter() methods, respectively.
It works like this:
import logging
# make a logger
main_logger = logging.getLogger("my logger")
main_logger.setLevel(logging.INFO)
# make some handlers
console_handler = logging.StreamHandler() # by default, sys.stderr
file_handler = logging.FileHandler("my_log_file.txt")
# set logging levels
console_handler.setLevel(logging.WARNING)
file_handler.setLevel(logging.INFO)
# add handlers to logger
main_logger.addHandler(console_handler)
main_logger.addHandler(file_handler)
Now, you can use this object like this:
main_logger.info("logged in the FILE")
main_logger.warning("logged in the FILE and on the CONSOLE")
If you just run python on your machine, you can type the above code into the interactive console and you should see the output. The log file will get crated in your current directory, if you have permissions to create files in it.
I hope this helps!

It is possible to override logging.getLoggerClass() to add new functionality to loggers. I wrote simple class which prints green messages in stdout.
Most important parts of my code:
class ColorLogger(logging.getLoggerClass()):
__GREEN = '\033[0;32m%s\033[0m'
__FORMAT = {
'fmt': '%(asctime)s %(levelname)s: %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S',
}
def __init__(self, format=__FORMAT):
formatter = logging.Formatter(**format)
self.root.setLevel(logging.INFO)
self.root.handlers = []
(...)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
self.root.addHandler(handler)
def info(self, message):
self.root.info(message)
(...)
def info_green(self, message):
self.root.info(self.__GREEN, message)
(...)
if __name__ == '__main__':
logger = ColorLogger()
logger.info("This message has default color.")
logger.info_green("This message is green.")

Handlers send the log records (created by loggers) to the appropriate
destination.
(from the docs: http://docs.python.org/library/logging.html)
Just set up multiple handlers with your logging object, one to write to file, another to write to the screen.
UPDATE
Here is an example function you can call in your classes to get logging set up with a handler.
def set_up_logger(self):
# create logger object
self.log = logging.getLogger("command")
self.log.setLevel(logging.DEBUG)
# create console handler and set min level recorded to debug messages
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# add the handler to the log object
self.log.addHandler(ch)
You would just need to set up another handler for files, ala the StreamHandler code that's already there, and add it to the logging object. The line that says ch.setLevel(logging.DEBUG) means that this particular handler will take logging messages that are DEBUG or higher. You'll likely want to set yours to WARNING or higher, since you only want the more important things to go to the console. So, your logging would work like this:
self.log.info("Hello, World!") -> goes to file
self.log.error("OMG!!") -> goes to file AND console