As the AWS documentation suggests:
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def my_logging_handler(event, context):
logger.info('got event{}'.format(event))
logger.error('something went wrong')
Now I made:
import logging
logging.basicConfig(level = logging.INFO)
logging.info("Hello World!")
The first snippet of code prints in the Cloud Watch console, but the second one no.
I didn't see any difference as the two snippets are using the root logger.
The reason that logging does not seem to work is because the AWS Lambda Python runtime pre-configures a logging handler that, depending on the version of the runtime selected, might modify the format of the message logged, and might also add some metadata to the record if available. What is not preconfigured though is the log-level. This means that no matter the type of log-message you try to send, it will not actually print.
As AWS documents themselves, to correctly use the logging library in the AWS Lambda context, you only need to set the log-level for the root-logger:
import logging
logging.getLogger().setLevel(logging.INFO)
If you want your Python-script to be both executable on AWS Lambda, but also with your local Python interpreter, you can check whether a handler is configured or not, and fall back to basicConfig (which creates the default stderr-handler) otherwise:
if len(logging.getLogger().handlers) > 0:
# The Lambda environment pre-configures a handler logging to stderr. If a handler is already configured,
# `.basicConfig` does not execute. Thus we set the level directly.
logging.getLogger().setLevel(logging.INFO)
else:
logging.basicConfig(level=logging.INFO)
Copied straight from the top answer in the question #StevenBohrer's answer links to (this did the trick for me, replacing the last line with my own config):
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
I've struggled with this exact problem. The solution that works both locally and on AWS CloudWatch is to setup your logging like this:
import logging
# Initialize you log configuration using the base class
logging.basicConfig(level = logging.INFO)
# Retrieve the logger instance
logger = logging.getLogger()
# Log your output to the retrieved logger instance
logger.info("Python for the win!")
I had a similar problem, and I suspect that the lambda container is calling logging.basicConfig to add handlers BEFORE the lambda code is imported. This seems like bad form...
Workaround was to see if root logger handlers were configured and if so, remove them, add my formatter and desired log level (using basicConfig), and restore the handlers.
See this article Python logging before you run logging.basicConfig?
Probably not referencing the same logger, actually.
In the first snippet, log the return of: logging.Logger.manager.loggerDict
It will return a dict of the loggers already initialized.
Also, from the logging documentation, an important note on logging.basicConfig:
Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger. The functions debug(), info(), warning(), error() and critical() will call basicConfig() automatically if no handlers are defined for the root logger.
This function does nothing if the root logger already has handlers configured for it.
Source: https://docs.python.org/2/library/logging.html#logging.basicConfig
It depends upon the aws lambda python version
If python version 3.8 and above
import os
import logging
default_log_args = {
"level": logging.DEBUG if os.environ.get("DEBUG", False) else logging.INFO,
"format": "%(asctime)s [%(levelname)s] %(name)s - %(message)s",
"datefmt": "%d-%b-%y %H:%M",
"force": True,
}
logging.basicConfig(**default_log_args)
log = logging.getLogger("Run-Lambda")
log.info("I m here too)
If python version 3.7 and below
import os
import logging
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
default_log_args = {
"level": logging.DEBUG if os.environ.get("DEBUG", False) else logging.INFO,
"format": "%(asctime)s [%(levelname)s] %(name)s - %(message)s",
"datefmt": "%d-%b-%y %H:%M"
}
logging.basicConfig(**default_log_args)
log = logging.getLogger("Run-Lambda")
log.info("Iam here")
Essentially, the AWS logging monkey patch needs to be handled in a very particular way, where:
The log level is set from the TOP level of the script (e.g., at import time)
The log statements you are interested in are invoked from within the lambda function
Since it's generally considered good form not to run arbitrary code in Python module import, you usually should be able to restructure your code so that the heavy lifting occurs only inside the lambda function.
I would suggest use aws python lambda powertools. The logging doc is here. Code example:
from aws_lambda_powertools import Logger
logger = Logger() # Sets service via env var
# OR logger = Logger(service="example")
It works works both locally and on CloudWatch for me.
I have also solved this issue so that logging would not require change for local and on aws. Below is the sample code:
def set_default_logger():
if "LOG_LEVEL" in os.environ:
# For Lambda
log_level = os.environ["LOG_LEVEL"]
else:
log_level = DEFAULT_LOG_LEVEL # Set default log level for local
root = logging.getLogger()
if len(logging.getLogger().handlers) > 0:
# For Lambda
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(level=log_level,
format='[%(asctime)s.%(msecs)03d] [%(levelname)s] [%(module)s] [%(funcName)s] [L%(lineno)d] [P%(process)d] [T%(thread)d] %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
else:
# For Local
l_name = os.getcwd()+'/'+'count_mac_module.log'
logging.basicConfig(filename=l_name, level=log_level,
format='[%(asctime)s.%(msecs)03d] [%(levelname)s] [%(module)s] [%(funcName)s] [L%(lineno)d] [P%(process)d] [T%(thread)d] %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
logger = logging.getLogger(__name__)
logger.debug(f"************* logging set for Lambda {os.getenv('AWS_LAMBDA_FUNCTION_NAME') } *************")
LOGGER = logging.getLogger()
HANDLER = LOGGER.handlers[0]
HANDLER.setFormatter(
logging.Formatter(“[%(asctime)s] %(levelname)s:%(name)s:%(message)s”, “%Y-%m-%d %H:%M:%S”)
)
import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info('## ENVIRONMENT VARIABLES')
logger.info(os.environ)
logger.info('## EVENT')
logger.info(event)`enter code here`
Related
I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know why it so but couldn't clearly how I can solve it.
Below is log_config.py which is used by all other modules to get the logger and log.
'''
import logging
import sys
from logging.handlers import TimedRotatingFileHandler
FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")
LOG_FILE = "my_app.log"
def get_file_handler():
file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
file_handler.setFormatter(FORMATTER)
return file_handler
def get_logger(logger_name):
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG) # better to have too much log than not enough
logger.addHandler(get_file_handler())
#with this pattern, it's rarely necessary to propagate the error up to parent
logger.propagate = False
return logger
'''
All other modules call,
'logging = log_config.get_logger(name)'
and use it to log.
I came to know about QueueHandler and QueueListener but not sure how to use them in my code.
How can I use these to serialize logs to single file.?
I've got a sagemaker instance running a jupyter notebook. I'd like to use python's logging module to write to a log file, but it doesn't work.
My code is pretty straightforward:
import logging
logger = logging.getLogger()
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(name)s - %(message)s", datefmt="%y/%m/%d %H:%M:%S")
fhandler = logging.FileHandler("taxi_training.log")
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.debug("starting log...")
This should write a line to my file taxi_training.log but it doesn't.
I tried using the reload function from importlib, I also tried setting the output stream to sys.stdout explicitly. Nothing is logging to the file or in cloudwatch.
Do I need to add anything to my Sagemaker instance for this to work properly?
The Python logging module requires a logging level and one or more handlers to process output. By default, the logging level is set to WARNING (30) with a STDOUT handler for that level. If a logging level and/or handler is not explicitly defined, these settings are inherited from the parent root logger settings. These settings can be verified by adding the following lines to the bottom of your code:
# Verify levels and handlers
print("Parent Logger: "+logger.parent.name)
print("Parent Level: "+str(logger.parent.level))
print("Parent Handlers: "+str(logger.parent.handlers))
print("Logger Level: "+str(logger.level))
print("Logger Handlers: "+str(logger.handlers))
The easiest way to instantiate a handler and set a logging level is by running the logging.basicConfig() function (documentation). This will set a logging level and STDOUT handler at the root logger level which will propagate to any child loggers created in the same code. Here is an example using the code provided:
import logging
logger = logging.getLogger('log')
logging.basicConfig(level=logging.INFO) # Set logging level and STDOUT handler
logger.info(5)
Overview
I want to use httpimport as a logging library common to several scripts. This module generates logs of its own which I do not know how to silence.
In other cases such as this one, I would have used
logging.getLogger('httpimport').setLevel(logging.ERROR)
but it did not work.
Details
The following code is a stub of the "common logging code" mentioned above:
# toconsole.py
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s %(message)s')
handler_console = logging.StreamHandler()
level = logging.DEBUG if 'DEV' in os.environ else logging.INFO
handler_console.setLevel(level)
handler_console.setFormatter(formatter)
log.addHandler(handler_console)
# disable httpimport logging except for errors+
logging.getLogger('httpimport').setLevel(logging.ERROR)
A simple usage such as
import httpimport
httpimport.INSECURE = True
with httpimport.remote_repo(['githublogging'], 'http://localhost:8000/') :
from toconsole import log
log.info('yay!')
gives the following output
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
2019-08-25 13:56:48,671 yay!
yay!
The second (bare) yay! must be coming from httpimport, namely from its logging setup.
How can I disable the logging for such a module, or better - raise its level so that only errors+ are logged?
Note: this question was initially asked at the Issues section of the GitHub repository for httpimport but the author did not know either how to fix that.
Author of httpimport here.
I totally forgot I was using the basicConfig logger thing.
It is fixed in master right now (0.7.2) - will be included in next PyPI release:
https://github.com/operatorequals/httpimport/commit/ff2896c8f666c3f16b0f27716c732d68be018ef7
The reason why this is happening is because when you do import httpimport they do the initial configuration for the logging machinery. This happens right here. What this means is that the root logger already has a StreamHandler attached to it. Because of this, and the fact that all loggers inherit from the root logger, when you do log.info('yay') it not only uses your Handler and Formatter, but it also propagates all they way to the root logger, which also emits the message.
Remember that whoever calls basicConfig first when an application starts that sets up the default configuration for the root logger, which in turn, is inherited by all loggers, unless otherwise specified.
If you have a complex logging configuration you need to ensure that you call it before you do any third-party imports which might call basicConfig. basicConfig is idempotent meaning the first call seals the deal, and subsequent calls have no effect.
Solutions
You could do log.propagate = False and you will see that the 2nd yay will not show.
You could attach the Formatter directly to the already existent root Handler by doing something like this (without adding another Handler yourself)
root = logging.getLogger('')
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler = root.handlers[0]
root_handler.setFormatter(formatter)
You could do a basicConfig call when you initialize your application (if you had such a config available, with initial Formatters and Handlers, etc. that will elegantly attach everything to the root logger neatly) and then you would only do something like logger = logging.getLogger(__name__) and logger.info('some message') that would work the way you'd expect because it would propagate all the way to the root logger which already has your configuration.
You could remove the initial Handler that's present on the root logger by doing something like
root = logging.getLogger('')
root.handlers = []
... and many more solutions, but you get the idea.
Also do note that logging.getLogger('httpimport').setLevel(logging.ERROR) this works perfectly fine. No messages below logging.ERROR would be logged by that logger, it's just that the problem wasn't from there.
If you however want to completely disable a logger you can just do logger.disabled = True (also do note, again, that the problem wasn't from the httpimport logger, as aforementioned)
One example demonstrated
Change your toconsole.py with this and you won't see the second yay.
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
root_logger = logging.getLogger('')
root_handler = root_logger.handlers[0]
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler.setFormatter(formatter)
# or you could just keep your old code and just add log.propagate = False
# or any of the above solutions and it would work
logging.getLogger('httpimport').setLevel(logging.ERROR)
I have written following code to enable Cloudwatch support.
import logging
from boto3.session import Session
from watchtower import CloudWatchLogHandler
logging.basicConfig(level=logging.INFO,format='[%(asctime)s.%(msecs).03d] [%(name)s,%(funcName)s:%(lineno)s] [%(levelname)s] %(message)s',datefmt='%d/%b/%Y %H:%M:%S')
log = logging.getLogger('Test')
boto3_session = Session(aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
region_name=REGION_NAME)
cw_handler = CloudWatchLogHandler(log_group=CLOUDWATCH_LOG_GROUP_NAME,stream_name=CLOUDWATCH_LOG_STREAM_NAME,boto3_session=boto3_session)
log.addHandler(cw_handler)
Whenever i try to print any logger statement, i am getting different output on my local system and cloudwatch.
Example:
log.info("Hello world")
Output of above logger statement on my local system (terminal) :
[24/Feb/2019 15:25:06.969] [Test,<module>:1] [INFO] Hello world
Output of above logger statement on cloudwatch (log stream) :
Hello world
Is there something i am missing ?
In the Lambda execution environment, the root logger is already preconfigured. You'll have to work with it or work around it. You could do some of the following:
You can set the formatting directly on the root logger:
root = logging.getLogger()
root.setLevel(logging.INFO)
root.handlers[0].setFormatter(logging.Formatter(fmt='[%(asctime)s.%(msecs).03d] [%(name)s,%(funcName)s:%(lineno)s] [%(levelname)s] %(message)s', datefmt='%d/%b/%Y %H:%M:%S'))
You could add the Watchtower handler to it (disclaimer: I have not tried this approach):
root = logging.getLogger()
root.addHandler(cw_handler)
However I'm wondering if you even need to use Watchtower. In Lambda, every line you print to stdout (so even just using print) get logged to Cloudwatch. So using the standard logging interface might be sufficient.
This worked for me
import logging
import watchtower
watch = watchtower.CloudWatchLogHandler()
watch.setFormatter(fmt = logging.Formatter('%(levelname)s - %(module)s - %(message)s'))
logger = logging.getLogger()
logger.addHandler(watch)
As per comment in https://stackoverflow.com/a/45624044/1021819 (and another answer there),
just add force=True to logging.basicConfig(), so in your case you need
logging.basicConfig(level=logging.INFO, force=True, format='[%(asctime)s.%(msecs).03d] [%(name)s,%(funcName)s:%(lineno)s] [%(levelname)s] %(message)s',datefmt='%d/%b/%Y %H:%M:%S')
log = logging.getLogger('Test')
This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True.
(i.e. AWS case)
Re: force:
If this keyword argument is specified as true, any existing handlers attached to the root logger are removed and closed, before carrying out the configuration as specified by the other arguments.
REF: https://docs.python.org/3/library/logging.html#logging.basicConfig
THANKS:
https://stackoverflow.com/a/72054516/1021819
https://stackoverflow.com/a/45624044/1021819
I'm using logging module, and I've passed in the same parameters that I have on other jobs that are currently working:
import logging
from inst_config import config3
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(message)s',
filename=config3.GET_LOGFILE(config3.REQUESTS_USAGE_LOGFILE))
logging.warning('This should go in the file.')
if __name__ == '__main__':
logging.info('Starting unload.')
Using this method to create the filename:
REQUESTS_USAGE_LOGFILE = r'C:\RunLogs\Requests_Usage\requests_usage_runlog_{}.txt'.format(
CUR_MONTH)
def GET_LOGFILE(logfile):
"""Truncates file and returns."""
with open(logfile, 'w'):
pass
return logfile
When I run it, however, it is creating the file, and then still outputting the logging info to the console. I'm running in Powershell.
Just tried putting it inside the main statement like this:
if __name__ == '__main__':
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(message)s',
filename=config3.GET_LOGFILE(config3.REQUESTS_USAGE_LOGFILE))
logging.warning('This should go in the file.')
Still no luck.
I add the following lines before the logging.basicConfig() and it worked for me.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
You can try running this snippet in your main file.
import logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(message)s',
filename='filename.txt') # pass explicit filename here
logger = logging.get_logger() # get the root logger
logger.warning('This should go in the file.')
print logger.handlers # you should have one FileHandler object
In addition to Forge's answer on using logging.basicConfig(), as of Python 3.8 a parameter to basicConfig() got added. To quote the docs:
"""
This function does nothing if the root logger already has handlers
configured, unless the keyword argument *force* is set to ``True``.
...
force If this keyword is specified as true, any existing handlers
attached to the root logger are removed and closed, before
carrying out the configuration as specified by the other
arguments.
"""
This is why yue dong's answer to remove all handlers has worked for some as well as Alexander's answer to reset logger.handlers to [].
Debugging logger.handlers (as Forge's answer suggests) led me to see a single StreamHandler there (so basicConfig() did nothing for me until I used force=True as parameter).
Hope that helps in addition to all the other answers here too!
If you are using 'root' logger which is by default has name "", than you can do this trick:
logging.getLogger().setLevel(logging.INFO)
logger = logging.getLogger('')
logger.handlers = []
In addition you may want to specify logging level as in code above, this level will persist for all descendant loggers.
If instead, you specified particular logger, than do
logger = logging.getLogger('my_service')
logger.handlers = []
fh = logging.FileHandler(log_path)
fh.setLevel(logging.INFO)
# create console handler
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
logger.addHandler(fh)
logger.addHandler(ch)
logger.info('service started')
The above code will create new logger 'my_service'. In case that logger has been already created it clears all handles. That it adds handles for writing in specified file and console. See official documentation as well.
You can also use hierarchical loggers. It is done directly.
logger = logging.getLogger('my_service.update')
logger.info('updated successfully')