We Keep Coding

Issue in sending python logs to Splunk using splunk_hec_handler

I am using Python logging library to push logs to splunk. This package use HEC method to push logs to splunk.
Issue I am facing is that out of many logger statements in my application, I want selectively only few logger statements to splunk not all.
So i created one method below method which converts string logs in json (key/value) and pushes into splunk. So I am calling this method just after the logger statement I wish to push to splunk. But rest all the logger statements which i dont wish to send to splunk they are also getting pushed to splunk.
Why is this happening?
class Test:
def __init__(self):
self.logger = logging.getLogger('myapp')
def method_test(self,url,data,headers):
response = requests.post(url=url, data=json.dumps(data), headers=abc.headers)
##i dont want to push this below log message to splunk but this is also getting pushed to splunk
self.logger.debug(f"request code:{response.request.url} request body:{response.request.body}")
##I wish to send this below log to splunk
self.logger.debug(f"response code:{response.status_code} response body:{response.text}")
log_dic = {'response_code': response.status_code,'response_body': response.text}
splunklogger = self.logging_override(log_dic, self.splunk_host,
self.index_token, self.port,
self.proto, self.ssl_verify,
self.source)
splunklogger.info(log_dic)
return response
def logging_override(log_dict: dict, splunk_host,index_token,splunk_port,splunk_proto,ssl_ver,source_splnk):
"""
This function help in logging custom fields in JSON key value form by defining fields of our choice in log_dict dictionary
and pushes logs to Splunk Server
"""
splunklogger = logging.getLogger()
splunklogger.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
basic_dict = {"time": "%(asctime)s", "level": "%(levelname)s"}
full_dict = {**basic_dict, **log_dict}
stream_formatter = logging.Formatter(json.dumps(full_dict))
stream_handler.setFormatter(stream_formatter)
if not splunklogger.handlers:
splunklogger.addHandler(stream_handler)
splunklogger.handlers[0] = stream_handler
splunk_handler = SplunkHecHandler(splunk_host,
index_token,
port=splunk_port, proto=splunk_proto, ssl_verify=ssl_ver,
source=source_splnk)
splunklogger.addHandler(splunk_handler)
splunklogger.addHandler(splunk_handler)
return splunklogger

I believe that the problem is with your calls to logging.getLogger, namely when you're configuring your app logger, you're specifying a logger name, but when you're configuring the splunk logger, you're not specifying any and therefore getting, configuring, and attaching the SplunkHandler to the root logger.
As events come in to the lower level loggers by default they propagate their events to higher level loggers (e.g. the root logger) and thus get emitted to Splunk.
I suspect an easy solution would be to look at your logger names... possibly put the Splunk logger at a lower level than your component? or look into the propagation of loggers. The same docs page linked above talks a bit about logger objects and their propagation.

Tornado substituting custom logger on server (not on local computer)

I have a tornado application and a custom logger method. My code to build and use the custom logger is the following:
def create_logger():
"""
This function creates the logger functionality to be used throughout the Python application
:return: bool - true if successful
"""
# Configuring the logger
filename = "PythonLogger.log"
# Change the current working directory to the logs folder, so that the logs files is written in it.
os.chdir(os.path.normpath(os.path.normpath(os.path.dirname(os.path.abspath(__file__)) + os.sep + os.pardir + os.sep + os.pardir + os.sep + 'logs')))
# Create the logs file
logging.basicConfig(filename=filename, format='%(asctime)s %(message)s', filemode='w')
# Creating the logger
logger = logging.getLogger()
# Setting the threshold of logger to DEBUG
logger.setLevel(logging.NOTSET)
logger.log(0, 'El logger está inicializado')
return True
def log_info_message(msg):
"""
Utility for message logging with code 20
:param msg:
:return:
"""
return logging.getLogger().log(20, msg)
In the code, I initialize the logger and already write a message to it before the Tornado application initialization:
if __name__ == '__main__':
# Logger initialization
create_logger()
# First log message
log_info_message('Initiating Python application')
# Starting Tornado
tornado.options.parse_command_line()
# Specifying what app exactly is being started
server = tornado.httpserver.HTTPServer(test.app)
server.listen(options.port)
try:
if 'Windows_NT' not in os.environ.values():
server.start(0)
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
Then let's say my method get of HTTP request is as follows (only interesting lines):
class API(tornado.web.RequestHandler):
def get(self):
self.write('Get request ')
logging.getLogger("tornado.access").log(20, 'Hola')
logging.getLogger("tornado.application").log(20, '1')
logging.getLogger("tornado.general").log(20, '2')
log_info_message('Received a GET request at: ' + datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
What I see is a difference between local testing and testing on server.
A) On local, I can see log message at first script running, and log messages of requests (after initializing Tornado app) in my log file and the Tornado logs.
B) On server, I only see the first message, not my log messages when Get requests are accepted and also see Tornado's loggers when there's an error, but even don't see the messages produced by Tornado's loggers. I guess that means that somehow Tornado is re-initializing the logger and making mine and his 3 ones write in some other file (somehow that does not affect when errors happens??).
I am aware that Tornado uses its own 3 logging functions, but somehow I would like to use mine as well, at the same time as keeping the Tornado's ones and writing them all into the same file. Basically reproduce that local behaviour on server but also keeping it when some error happens, of course.
How could I achieve this?
Thanks in advance!
P.S.: if I add a name to the logger, let's say logging.getLogger('Example') and changed log_info_message function to return logging.getLogger('Example').log(20, msg), Tornado's logger would fail and raise error. So that option destroys its own loggers...

It seems the only problem was that, on the server side, tornado was setting the the mininum level for a log message to be written on log file higher (minimum of 40 was required). So that logging.getLogger().log(20, msg) would not write on the logging file but logging.getLogger().log(40, msg) would.
I would like to understand why, so if anybody knows, your knowledge would be more than welcome. For the time being that solution is working though.

tornado.log defines options that can be used to customise logging via command line (check tornado.options) - one of them is logging that defines the log level used. You are likely using this on the server and setting it to error.
When debugging logging I suggest you create a RequestHandler that will log or return the structure of the existing loggers by inspecting the root logger. When you see the structure it is much easier to understand why it works the way it works.

Python: Is it a good idea to pass the logger around?

My API of a web server logs like this:
started
started
succeeded
failed
That's two requests received at the same time. It's hard to tell which one succeeded or failed. To separate requests from each other, I created a random number for each and used it as the name of the logger
logger = logging.getLogger(random_number)
The logs became
[111] started
[222] started
[111] succeeded
[222] failed
Looks clear now, but the problem of this approach is that I have to pass the logger to every related class like this:
def __init__(self, logger):
self._logger = logger
So the question is:
Is this the best way to log context of each request?
If so, is it a good idea to pass the logger around? Is there any way to make the code less verbose?

You should not generate a new logger for each request. You should identify a unique attribute of a request (client IP, URL, session ID, some cookie token...) and add it to the log statement. That way you'll be able to link different log entries of a single request in the log output.
Also, logs should not be passed around. Attribute name from logging.getLogger(name) should be used to obtain the same logger from different locations in the code.
Best to read up on logging in more detail, e.g. here https://docs.python.org/3/library/logging.html where you can find maybe a useful example with the client IP:
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)

Python logging sometimes writing message to file but not format

Having a new problem where the logger is writing about every other line without the format, just the message.
My code:
import logging
from logging.handlers import RotatingFileHandler
# Set up logging
LOG_FILE = argv[0][:-3] + '.log'
logging.basicConfig(
filename=LOG_FILE,
filemode='a',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
handler = RotatingFileHandler(LOG_FILE,maxBytes=1000000)
logger.addHandler(handler)
def main():
target, command, notify_address, wait = get_args(argv)
logger.info('Checking status of %s every %d minutes.' % (target, wait))
logger.info('Running %s and sending output to %s when online.' % (command, notify_address))
The vars returned from get_args() are all strings, even though wait is a number.
Note that I am not receiving any errors in my IDE or when running.
The output I am getting in my log file:
2015-02-12 16:26:27,483 - INFO - Checking status of <ip address> every 30 minutes.
Running <arbitrary bash command string> and sending output to <my email address> when online.
2015-02-12 16:26:27,483 - INFO - Running <arbitrary bash command string> and sending output to <my email address> when online.
What is causing the second logger.info() to print twice, and only once formatted properly?
I have another script that logs perfectly, no idea what I've done here. (Copy/pasted the logging setup section to be safe)

Are you using loggers at different levels of your code? It sounds like the log messages could be propagating upwards. Try adding
logger.propagate = False
after you add the handler. You can check out the python docs for a more detailed explanation here, but the relevant text below sounds exactly like what you're seeing.
Note If you attach a handler to a logger and one or more of its ancestors, it may emit the same record multiple times. In general, you should not need to attach a handler to more than one logger - if you just attach it to the appropriate logger which is highest in the logger hierarchy, then it will see all events logged by all descendant loggers, provided that their propagate setting is left set to True. A common scenario is to attach handlers only to the root logger, and to let propagation take care of the rest.

Python Logging: Group logs which belong to one request

Is there a way to group logs of a python web application which belong to one web request?
Example:
2015-02-11 13:06:32 myapp.middleware.MYAPPMiddleware: INFO Login of user foo was successful
2015-02-11 13:06:32 myapp.middleware.MYAPPMiddleware: INFO Login of user bar failed
2015-02-11 13:06:32 myapp.send_mails: INFO failed to send mail to someone#example.com
The above log lines are unrelated to each other.
How can you solve this the pythonic way?

Log entries in their essence are designed to be independent from each other.
The correct way to connect them together is to include some contextual information into the entries to filter by when looking through the logs later.
Here's a example of a Sharepoint log record with such information:
Timestamp Process TID Area Category EventID Level Message Correlation
02/26/2015 17:49:19.65 w3wp.exe (0x1F40) 0x2358 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (POST:http://reserver2:80/pest/_vti_bin/sitedata.asmx) d1e2b688-e0b2-481e-98ce-497a11acab44
In Python logging docs, Adding contextual information to your logging output recommends either of two methods: using a LoggerAdapter or a Filter.
LoggerAdapter is used like this (examples are based on those in the docs):
class AddConnIdAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return <augment_message(msg,arbitrary_info)>, kwargs
la = AddConnIdAdapter(<logger>,extra=<parameters, saved in self.extra>)
<...>
la.info(<message>)
Filter is used like this:
#Either all messages should have custom fields
# or the Formatter used should support messages
# both with and without custom fields
logging.basicConfig(<...>,format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
class AddClientInfo(logging.Filter):
#override __init__ or set attributes to specify parameters
def filter(self, record):
record.ip = <get_client_ip()>
record.user = <get_client_name()>
return True #do not filter out anything
l=<logger()>
l.addFilter(AddClientInfo()) #can attach to either loggers or handlers
<...>
l.info('message')
As you can see, the difference is LoggerAdapter is non-transparent while Filter is transparent. In the examples, the former modifies the message text while the latter sets custom attributes (and actually writing them requires cooperation of the Formatter used) but in fact, both can do both.
So, the former is more useful if you only need to add the context to some messages while the latter is more fit to augment all, or a large portion of, the messages being logged.

You can assign random UUID to each request in init method, and add it to all log messages.
For example, in Tornado:
class MainRequestHandler(RequestHandler):
def __init__(self, application, request):
super(MainRequestHandler, self).__init__(application, request)
self.uuid = uuid.uuid4()
logging.info("%s | %s %s %s",
self.uuid,
request.method,
request.full_url(),
request.remote_ip)
As result, you will be able to grep log by this UUID to find all messages which belong to separate request.