Is there a way to group logs of a python web application which belong to one web request?
Example:
2015-02-11 13:06:32 myapp.middleware.MYAPPMiddleware: INFO Login of user foo was successful
2015-02-11 13:06:32 myapp.middleware.MYAPPMiddleware: INFO Login of user bar failed
2015-02-11 13:06:32 myapp.send_mails: INFO failed to send mail to someone#example.com
The above log lines are unrelated to each other.
How can you solve this the pythonic way?
Log entries in their essence are designed to be independent from each other.
The correct way to connect them together is to include some contextual information into the entries to filter by when looking through the logs later.
Here's a example of a Sharepoint log record with such information:
Timestamp Process TID Area Category EventID Level Message Correlation
02/26/2015 17:49:19.65 w3wp.exe (0x1F40) 0x2358 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (POST:http://reserver2:80/pest/_vti_bin/sitedata.asmx) d1e2b688-e0b2-481e-98ce-497a11acab44
In Python logging docs, Adding contextual information to your logging output recommends either of two methods: using a LoggerAdapter or a Filter.
LoggerAdapter is used like this (examples are based on those in the docs):
class AddConnIdAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return <augment_message(msg,arbitrary_info)>, kwargs
la = AddConnIdAdapter(<logger>,extra=<parameters, saved in self.extra>)
<...>
la.info(<message>)
Filter is used like this:
#Either all messages should have custom fields
# or the Formatter used should support messages
# both with and without custom fields
logging.basicConfig(<...>,format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
class AddClientInfo(logging.Filter):
#override __init__ or set attributes to specify parameters
def filter(self, record):
record.ip = <get_client_ip()>
record.user = <get_client_name()>
return True #do not filter out anything
l=<logger()>
l.addFilter(AddClientInfo()) #can attach to either loggers or handlers
<...>
l.info('message')
As you can see, the difference is LoggerAdapter is non-transparent while Filter is transparent. In the examples, the former modifies the message text while the latter sets custom attributes (and actually writing them requires cooperation of the Formatter used) but in fact, both can do both.
So, the former is more useful if you only need to add the context to some messages while the latter is more fit to augment all, or a large portion of, the messages being logged.
You can assign random UUID to each request in init method, and add it to all log messages.
For example, in Tornado:
class MainRequestHandler(RequestHandler):
def __init__(self, application, request):
super(MainRequestHandler, self).__init__(application, request)
self.uuid = uuid.uuid4()
logging.info("%s | %s %s %s",
self.uuid,
request.method,
request.full_url(),
request.remote_ip)
As result, you will be able to grep log by this UUID to find all messages which belong to separate request.
Related
Is there a way to filter out specific error messages using Django Logging? eg UncompressableFileError
Would like to stop these errors being sent to Sentry.io
You can set a Filter on the Sentry handler which checks for the type of errors you want to filter out, and return False to drop them. Roughly:
def sentry_filter(record):
return 'UncompressableFileError' not in record.getMessage()
and then
sentry_handler.addFilter(sentry_filter)
This might need to be tweaked depending on where the string occurs - e.g. in message or traceback
Found this in the end from another answer on SA which works with Raven:
Is there a way to filter out specific error messages using Django Logging? eg UncompressableFileError
class MyExceptionType(Exception):
def __init__(self, message):
super(MyExceptionType, self).__init__(message)
app = Flask(__name__)
app.config['SENTRY_CONFIG'] = {
'ignore_exceptions': [MyExceptionType],
}
I am using Python logging library to push logs to splunk. This package use HEC method to push logs to splunk.
Issue I am facing is that out of many logger statements in my application, I want selectively only few logger statements to splunk not all.
So i created one method below method which converts string logs in json (key/value) and pushes into splunk. So I am calling this method just after the logger statement I wish to push to splunk. But rest all the logger statements which i dont wish to send to splunk they are also getting pushed to splunk.
Why is this happening?
class Test:
def __init__(self):
self.logger = logging.getLogger('myapp')
def method_test(self,url,data,headers):
response = requests.post(url=url, data=json.dumps(data), headers=abc.headers)
##i dont want to push this below log message to splunk but this is also getting pushed to splunk
self.logger.debug(f"request code:{response.request.url} request body:{response.request.body}")
##I wish to send this below log to splunk
self.logger.debug(f"response code:{response.status_code} response body:{response.text}")
log_dic = {'response_code': response.status_code,'response_body': response.text}
splunklogger = self.logging_override(log_dic, self.splunk_host,
self.index_token, self.port,
self.proto, self.ssl_verify,
self.source)
splunklogger.info(log_dic)
return response
def logging_override(log_dict: dict, splunk_host,index_token,splunk_port,splunk_proto,ssl_ver,source_splnk):
"""
This function help in logging custom fields in JSON key value form by defining fields of our choice in log_dict dictionary
and pushes logs to Splunk Server
"""
splunklogger = logging.getLogger()
splunklogger.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
basic_dict = {"time": "%(asctime)s", "level": "%(levelname)s"}
full_dict = {**basic_dict, **log_dict}
stream_formatter = logging.Formatter(json.dumps(full_dict))
stream_handler.setFormatter(stream_formatter)
if not splunklogger.handlers:
splunklogger.addHandler(stream_handler)
splunklogger.handlers[0] = stream_handler
splunk_handler = SplunkHecHandler(splunk_host,
index_token,
port=splunk_port, proto=splunk_proto, ssl_verify=ssl_ver,
source=source_splnk)
splunklogger.addHandler(splunk_handler)
splunklogger.addHandler(splunk_handler)
return splunklogger
I believe that the problem is with your calls to logging.getLogger, namely when you're configuring your app logger, you're specifying a logger name, but when you're configuring the splunk logger, you're not specifying any and therefore getting, configuring, and attaching the SplunkHandler to the root logger.
As events come in to the lower level loggers by default they propagate their events to higher level loggers (e.g. the root logger) and thus get emitted to Splunk.
I suspect an easy solution would be to look at your logger names... possibly put the Splunk logger at a lower level than your component? or look into the propagation of loggers. The same docs page linked above talks a bit about logger objects and their propagation.
My API of a web server logs like this:
started
started
succeeded
failed
That's two requests received at the same time. It's hard to tell which one succeeded or failed. To separate requests from each other, I created a random number for each and used it as the name of the logger
logger = logging.getLogger(random_number)
The logs became
[111] started
[222] started
[111] succeeded
[222] failed
Looks clear now, but the problem of this approach is that I have to pass the logger to every related class like this:
def __init__(self, logger):
self._logger = logger
So the question is:
Is this the best way to log context of each request?
If so, is it a good idea to pass the logger around? Is there any way to make the code less verbose?
You should not generate a new logger for each request. You should identify a unique attribute of a request (client IP, URL, session ID, some cookie token...) and add it to the log statement. That way you'll be able to link different log entries of a single request in the log output.
Also, logs should not be passed around. Attribute name from logging.getLogger(name) should be used to obtain the same logger from different locations in the code.
Best to read up on logging in more detail, e.g. here https://docs.python.org/3/library/logging.html where you can find maybe a useful example with the client IP:
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
There are several similar questions asked, but none I could find that specifically address this scenario.
In python, I have a class that uses a library that throws exceptions when there are errors or data it cant handle.
My class wraps the library, catches the exceptions and logs them using logger.
I want to email any exception that is raised to myself, but I do not want to get an email for every log entry. That is to say, I only want to get email when something is logged with logger.exception("The Exception"), but not if logged with logger.info("Great things are happening")
As I understand it the SMTP handler would email all log entries to me.
You can create your own logging filter, like so:
import logging
class MessageFilter(logging.Filter):
def filter(self, record):
return 'goodbye' in record.msg
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.addFilter(MessageFilter())
logger.addHandler(handler)
logger.error('hello world')
logger.error('goodbye moon')
which results in
goodbye moon
Just replace the message in filter with the one you want to match and attach it to your SMTPHandler.
i am implementing logging across a python web flask application. The below is an example structure
controller - com.org.python.MainController
has routes say /login, /doSomething, /logout
each of these routes call separate methods in separate modules say,
com.org.python.login.begin()
com.org.python.dosomething.execute()
com.org.python.logout.end()
my maincontroller routes have access to user id who has invoked the transaction and a unique transaction id which is generated. i can log this information easily using loggers.
usecase: i need to log user id and transaction id in the lower level modules as well , without passing the details as parameters
How can i achieve the same?
solution so far
class CustomFilter(logger.Filter):
def __init__(self, username, trid):
self.username = username
self.trid = trid
def filter(record):
record.username = self.username
record.trid = self.trid
i have added the filter to logger as given above
maincontroller.py
def login(username):
trid = ....
log = logger.getLogger("something")
log.addFilter(CustomFilter(username, trid))
#handlers, formatters added as used
log.info("inside login method")
#import login module
login.begin()
login.py
def begin():
log = logger.getLogger("something")
log.info("im inside begin method in login module")
### something something
the above works fine, username and trid is printed everywhere even across modules. however the same trid and username is being used everywhere. even transactions not initiated by the same user. do we have to clear the filters? how can i achieve my usecase m the desired username and diff trid for diff transactions
P.S: for some reason loggerAdaptors are not working. the info in adaptor is not getting transferred across module calls
please help.