I have a Django Application whose each API calls are associated with a transaction_id. I want to create separate log files for each transactions_id.
In simple words I want to have multiple files which I will be using for logging.
How can i do this using Django's built in logging system ?
I can have multiple handlers in a single logger. But as per my requirement FileHandlers has to added in run time whose file name will be the transaction_id. This can be done. But the problem is if I have 4 transactions running at a time, 4 handlers will be added to the same logger, and according to documentation logs will be sent to each handlers resulting in 1 transaction log file logging the rest 3 transaction's logs as well.
Following is what I have come with:
class TransactionLogger:
def __init__(self, transaction_id):
self.logger = logging.getLogger('transaction_logger')
logger = self.logger
fileHandler = logging.FileHandler(transaction_id, mode='a')
formatter = logging.Formatter('%(levelname)s %(asctime)s %(filename)s:%(lineno)s - %(funcName)s() ] %(message)s')
fileHandler.setFormatter(formatter)
self.logger.addHandler(fileHandler)
self.logger.propagate = False
At the beginning of each transactions I instantiate the logger as :
logger = TransactionLogger(transaction_id).logger
and log as follows:
logger.debug("Hello World")
How can I maintain n number of log files which will be generated dynamically and log into each file based on transaction_id without interfering with other files.
Any help is appreciated.
I would not say that its a good design to store logs like this. A better approach would be to write custom format to have transaction id in each log via which you will be able to filter your all logs.
Still here are are two ways you can achieve this:
1) By using logging._acquireLock() and logging._releaseLock() , OR you can use locking via LOCK as explained here.
2) Create a new logger every time (by inheriting logging.Manager and adding new logger to self.loggerDict ) and delete it at the end of the execution (so that system doesn't go out of memory).
Related
I am using the python logging library to configure my loggers with an input dict, like this:
logging.config.dictConfig(config)
I have a special function that should use a new function, and then switch back to the original logger at the end of the function. The original logger can vary, so I do not want to hardcode it. Some psuedocode to describe what I want:
def switch_logger_for_this_code():
old_logging_config = logger.get_current_logging_config(). # This is what I want to accomplish
logging.config.dictConfig(new_logging_config)
logging.info('This log goes to the new config!')
logging.config.dictConfig(old_logging_config)
logging.info('This log goes to the old config!')
return
Is this possible to do with the logging library?
I don't believe there is any built in way of retrieving the config of an initialized logger and storing it into a variable. I'm not sure if this would work for your project, but have you considered just creating a temporary logger object for use within the function's scope?
import logging
from sys import stdout
def switch_logger_for_this_code(msg) -> None:
# Create logging.Formatter for output customization
formatter = logging.Formatter(
fmt="[%(asctime)s.%(msecs)d] %(message)s",
datefmt="%Y/%m/%d %H:%M:%S"
)
handler = logging.StreamHandler(stdout) # Create output handler
handler.setLevel(logging.ERROR) # Specify logging level
handler.setFormatter(formatter) # Apply desired format
temp_log = logging.getLogger("temp") # Get new or existing Logger
temp_log.addHandler(handler) # Add output handler
temp_log.error(msg) # Log the message
switch_logger_for_this_code("hello world") # Log 'hello world'
# Which would output the following
# [2020/10/20 15:09:18.548] hello world
The main idea behind this approach is that the original logger object won't be modified, but you can still use the temporary logger to log what you had originally intended in your desired format.
I just looked through the source code for the logging module, and I don't see any way to get at the internal configuration, which starts as the passed in configuration, but can then be modified by other calls to the logging system.
I have a situation where I want to create two separate logger objects in Python, each with their own independent handler. By "separate," I mean that I want to be able to pass a log statement to each object independently, without contaminating the other log.
main.py
import logging
from my_other_logger import init_other_logger
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, handlers=[logging.StreamHandler(sys.stdout)])
other_logger = init_other_logger(__name__)
logger.info('Hello World') # Don't want to see this in the other logger
other_logger.info('Goodbye World') # Don't want to see this in the first logger
my_other_logger.py
import logging
import os, sys
def init_other_logger(namespace):
logger = logging.getLogger(namespace)
logger.setLevel(logging.DEBUG)
fh = logging.FileHandler(LOG_FILE_PATH)
logger.addHandler(fh)
formatter = logging.Formatter('%(levelname)s:%(name)s:%(message)s')
fh.setFormatter(formatter)
#logger.propagate = False
return logger
The only configuration I've been able to identify as useful here is the logger.propagate property. Running the code above as-is pipes all log statements to both the log stream and the log file. When I have logger.propagate = False nothing is piped to the log stream, and both log objects again pipe their output to the log file.
How can I create one log object that sends logs to only one handler, and another log object that sends logs to another handler?
Firstly, let's see what's going on before we can head over to the solution.
logger = logging.getLogger(__name__)
: when you're doing this you're getting or creating a logger with the name 'main'. Since this is the first call, it will create that logger.
other_logger = init_other_logger(__name__) : when you're doing this, again, you're getting or creating a logger with the name 'main'. Since this is the second call, it will fetch the logger created above. So you're not really instantiating a new logger, but you're getting a reference to the same logger created above. You can check this by doing a print after you call init_other_logger of the form: print(logger is other_logger).
What happens next is you add a FileHandler and a Formatter to the 'main' logger (inside the init_other_logger function), and you invoke 2 log calls via the method info(). But you're doing it with the same logger.
So this:
logger.info('Hello World')
other_logger.info('Goodbye World')
is essentially the same thing as this:
logger.info('Hello World')
logger.info('Goodbye World')
Now it's not so surprising anymore that both loggers output to both the file and stream.
Solution
So the obvious thing to do is to call your init_other_logger with another name.
I would recommend against the solution the other answer proposes because that's
NOT how things should be done when you need an independent logger. The documentation has it nicely put that you should NEVER instantiate a logger directly, but always via the function getLogger of the logging module.
As we discovered above when you do a call of logging.getLogger(logger_name) it's either getting or creating a logger with logger_name. So this works perfectly fine when you want a unique logger as well. However remember this function is idemptotent meaning it will only create a logger with a given name the first time you call it and it will return that logger if you call it with the same name no matter how many times you'll call it after.
So, for example:
a first call of the form logging.getLogger('the_rock') - creates your unique logger
a second call of the form logging.getLogger('the_rock') - fetches the above logger
You can see that this is particularly useful if you, for instance:
Have a logger configured with Formatters and Filters somewhere in your project, for instance in project_root/main_package/__init__.py.
Want to use that logger somewhere in a secondary package which sits in project_root/secondary_package/__init__.py.
In secondary_package/__init__.py you could do a simple call of the form: logger = logging.getLogger('main_package') and you'll use that logger with all its bells and whistles.
Attention!
Even if you, at this point, will use your init_other_logger function to create a unique logger it would still output to both the file and the console. Replace this line other_logger = init_other_logger(__name__) with other_logger = init_other_logger('the_rock') to create a unique logger and run the code again. You will still see the output written to both the console and the file.
Why ?
Because it will use both the FileHandler and the StreamHandler.
Why ?
Because the way the logging machinery works. Your logger will emit its message via its handlers, then it will propagate all the way up to the root logger where it will use the StreamHandler which you attached via the basicConfig call. So the propagate property you discovered is actually what you want in your case, because you're creating a custom logger, which you'd want to emit messages only via its manually attached handlers and not emit any further. Uncomment the logger.propagate = False after creating the unique logger and you'll see that everything works as expected.
Both of your handlers are installed on the same logger. This is why they aren't seperate.
logger is other_logger because logging.getLogger(__name__) is logging.getLogger(__name__)
Either create a logger directly for the second log logging.Logger(name) (I know the documentation says never to do this but if you want an entirely independent logger this is how to do it), or use a different name for the second log when calling logging.getLogger().
I am working with the Python logging module but I don't know how get writing in different files for each type of logging level.
For example, I want the debug-level messages (and only debug) saved in the file debug.log.
This is the code I have so far, I have tried with handlers but does not work anymore. I don't know If it is possible to do so I wish. This is the code I have using handlers and different files for each level of logging, but it does not work, since the Debug file save messages of all other levels.
import logging as _logging
self.logger = _logging.getLogger(__name__)
_handler_debug = _logging.FileHandler(self.debug_log_file)
_handler_debug.setLevel(_logging.DEBUG)
self.logger.addHandler(_handler_debug)
_handler_info = _logging.FileHandler(self.info_log_file)
_handler_info.setLevel(_logging.INFO)
self.logger.addHandler(_handler_info)
_handler_warning = _logging.FileHandler(self.warning_log_file)
_handler_warning.setLevel(_logging.WARNING)
self.logger.addHandler(_handler_warning)
_handler_error = _logging.FileHandler(self.error_log_file)
_handler_error.setLevel(_logging.ERROR)
self.logger.addHandler(_handler_error)
_handler_critical = _logging.FileHandler(self.critical_log_file)
_handler_critical.setLevel(_logging.CRITICAL)
self.logger.addHandler(_handler_critical)
self.logger.debug("debug test")
self.logger.info("info test")
self.logger.warning("warning test")
self.logger.error("error test")
self.logger.critical("critical test")
And the debug.log contains this:
debug test
info test
warning test
error test
critical test
I'm working with classes, so I have adapted a bit the code
Possible duplicate to: python logging specific level only
Please see the accepted answer there.
You need to use filters for each handler that restrict the output to the log level supplied.
I'm attempting to create a centralized module to set up my log formatter to be shared across a number of python modules within my lambda function. This function will ultimately be run on AWS Greengrass on a local on-premise device.
For some reason, when I add in my own handler to format the messages the logs are being outputted twice - once at the correct log level and the second time at an incorrect level.
If I use the standard python logger without setting up any handlers it works fine e.g.
main.py:
import logging
logging.debug("test1")
cloudwatch logs:
12:28:42 [DEBUG]-main.py:38,test1
My objective is to have one formatter on my code which will format these log messages into JSON. They will then get ingested into a centralized logging database. However, when I do this I get the log messages twice.
loghelper.py:
def setup_logging(name):
formatter = logging.Formatter("%(name)s, %(asctime)s, %(message)s")
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)
logger = logging.getLogger(name)
if logger.handlers:
for handler in logger.handlers:
logger.removeHandler(handler)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
return logger
main.py:
import logging
logger = loghelper.setup_logging('main.test_function')
def test_function():
logger.debug("test function log statement")
test_function()
When the lambda function is now run I get the debug message twice in the cloud watch logs as follows:
cloudwatch logs:
12:22:53 [DEBUG]-main.py:5, test function log statement
12:22:53 [INFO]-__init__.py:880,main.test_function,2018-06-18 12:22:53,099, test function log statement
Notice that:
The first entry is at the correct level but in the wrong format.
The second entry reports the wrong level, the wrong module but is in the correct format.
I cannot explain this behavior and would appreciate any thoughts on what could be causing it. I also don't know which constructor exists at line 880. This may shed some light on what is happening.
References:
Setting up a global formatter:
How to define a logger in python once for the whole program?
Clearing the default lambda log handlers:
Using python Logging with AWS Lambda
Creating a global logger:
Python: logging module - globally
AWS Lambda also sets up a handler, on the root logger, and anything written to stdout is captured and logged as level INFO. Your log message is thus captured twice:
By the AWS Lambda handler on the root logger (as log messages propagate from nested child loggers to the root), and this logger has its own format configured.
By the AWS Lambda stdout-to-INFO logger.
This is why the messages all start with (asctime) [(levelname)]-(module):(lineno), information; the root logger is configured to output messages with that format and the information written to stdout is just another %(message) part in that output.
Just don't set a handler when you are in the AWS environment, or, disable propagation of the output to the root handler and live with all your messages being recorded as INFO messages by AWS; in the latter case your own formatter could include the levelname level information in the output.
You can disable log propagation with logger.propagate = False, at which point your message is only going to be passed to your handler, not to to the root handler as well.
Another option is to just rely on the AWS root logger configuration. According to this excellent reverse engineering blog post the root logger is configured with:
logging.Formatter.converter = time.gmtime
logger = logging.getLogger()
logger_handler = LambdaLoggerHandler()
logger_handler.setFormatter(logging.Formatter(
'[%(levelname)s]\t%(asctime)s.%(msecs)dZ\t%(aws_request_id)s\t%(message)s\n',
'%Y-%m-%dT%H:%M:%S'
))
logger_handler.addFilter(LambdaLoggerFilter())
logger.addHandler(logger_handler)
This replaces the time.localtime converter on logging.Formatter with time.gmtime (so timestamps use UTC rather than locatime), sets a custom handler that makes sure messages go to the Lambda infrastructure, configures a format, and adds a filter object that only adds aws_request_id attribute to records (so the above formatter can include it) but doesn't actually filter anything.
You could alter the formatter on that handler by updating the attributes on the handler.formatter object:
for handler in logging.getLogger().handlers:
formatter = handler.formatter
if formatter is not None and 'aws_request_id' in formatter._fmt:
# this is the AWS Lambda formatter
# formatter.datefmt => '%Y-%m-%dT%H:%M:%S'
# formatter._style._fmt =>
# '[%(levelname)s]\t%(asctime)s.%(msecs)dZ'
# '\t%(aws_request_id)s\t%(message)s\n'
and then just drop your own logger handler entirely. You do want to be careful with this; AWS Lambda infrastructure could well be counting on a specific format being used. The output you show in your question doesn't include the date component (the %Y-%m-%dT part of the formatter.datefmt string) which probably means that the format has been parsed out and is being presented to you in a web application view of the data.
I'm not sure whether this is the cause of your problem, but by default, Python's loggers propagate their messages up to logging hierarchy. As you probably know, Python loggers are organized in a tree, with the root logger at the top and other loggers below it. In logger names, a dot (.) introduces a new hierarchy level. So if you do
logger = logging.getLogger('some_module.some_function`)
then you actually have 3 loggers:
The root logger (`logging.getLogger()`)
A logger at module level (`logging.getLogger('some_module'))
A logger at function level (`logging.getLogger('some_module.some_function'))
If you emit a log message on a logger and it is not discarded based on the loggers minimum level, then the message is passed on to the logger's handlers and to its parent logger. See this flowchart for more information.
If that parent logger (or any logger higher up in the hierarchy) also has handlers, then they are called, too.
I suspect that in your case, either the root logger or the main logger somehow ends up with some handlers attached, which leads to the duplicate messages. To avoid that, you can set propagate in your logger to False or only attach your handlers to the root logger.
At my current project there are thousand of code lines which looks like this:
logging.info("bla-bla-bla")
I don't want to change all these lines, but I would change log behavior. My idea is changing root logger to other Experimental logger, which is configured by ini-file:
[loggers]
keys = Experimental
[formatter_detailed]
format = %(asctime)s:%(name)s:%(levelname)s %(module)s:%(lineno)d: %(message)s
[handler_logfile]
class = FileHandler
args = ('experimental.log', 'a')
formatter = detailed
[logger_Experimental]
level = DEBUG
qualname = Experimental
handlers = logfile
propagate = 0
Now setting the new root logger is done by this piece of code:
logging.config.fileConfig(path_to_logger_config)
logging.root = logging.getLogger('Experimental')
Is redefining of root logger safe? Maybe there is more convenient way?
I've tried to use google and looked through stackoverflow questions, but I didn't find the answer.
You're advised not to redefine the root logger in the way you describe. In general you should only use the root logger directly for small scripts - for larger applications, best practice is to use
logger = logging.getLogger(__name__)
in each module where you use logging, and then make calls to logger.info() etc.
If all you want to do is to log to a file, why not just add a file handler to the root logger? You can do using e.g.
if __name__ == '__main__':
logging.basicConfig(filename='experimental.log', filemode='w')
main() # or whatever your main entry point is called
or via a configuration file.
Update: When I say "you're advised", I mean by me, here in this answer ;-) While you may not run into any problems in your scenario, it's not good practice to overwrite a module attribute which hasn't been designed to be overwritten. For example, the root logger is an instance of a different class (which is not part of the public API), and there are other references to it in the logging machinery which would still point to the old value. Either of these facts could lead to hard-to-debug problems. Since the logging package allows a number of ways of achieving what you seem to want (seemingly, logging to a file rather than the console), then you should use those mechanisms that have been provided.
logger = logging.getLogger()
Leaving the name empty will return you the root logger.
logger = logging.getLogger('name'）
Gives you another logger.