We Keep Coding

Tornado substituting custom logger on server (not on local computer)

I have a tornado application and a custom logger method. My code to build and use the custom logger is the following:
def create_logger():
"""
This function creates the logger functionality to be used throughout the Python application
:return: bool - true if successful
"""
# Configuring the logger
filename = "PythonLogger.log"
# Change the current working directory to the logs folder, so that the logs files is written in it.
os.chdir(os.path.normpath(os.path.normpath(os.path.dirname(os.path.abspath(__file__)) + os.sep + os.pardir + os.sep + os.pardir + os.sep + 'logs')))
# Create the logs file
logging.basicConfig(filename=filename, format='%(asctime)s %(message)s', filemode='w')
# Creating the logger
logger = logging.getLogger()
# Setting the threshold of logger to DEBUG
logger.setLevel(logging.NOTSET)
logger.log(0, 'El logger está inicializado')
return True
def log_info_message(msg):
"""
Utility for message logging with code 20
:param msg:
:return:
"""
return logging.getLogger().log(20, msg)
In the code, I initialize the logger and already write a message to it before the Tornado application initialization:
if __name__ == '__main__':
# Logger initialization
create_logger()
# First log message
log_info_message('Initiating Python application')
# Starting Tornado
tornado.options.parse_command_line()
# Specifying what app exactly is being started
server = tornado.httpserver.HTTPServer(test.app)
server.listen(options.port)
try:
if 'Windows_NT' not in os.environ.values():
server.start(0)
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
Then let's say my method get of HTTP request is as follows (only interesting lines):
class API(tornado.web.RequestHandler):
def get(self):
self.write('Get request ')
logging.getLogger("tornado.access").log(20, 'Hola')
logging.getLogger("tornado.application").log(20, '1')
logging.getLogger("tornado.general").log(20, '2')
log_info_message('Received a GET request at: ' + datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
What I see is a difference between local testing and testing on server.
A) On local, I can see log message at first script running, and log messages of requests (after initializing Tornado app) in my log file and the Tornado logs.
B) On server, I only see the first message, not my log messages when Get requests are accepted and also see Tornado's loggers when there's an error, but even don't see the messages produced by Tornado's loggers. I guess that means that somehow Tornado is re-initializing the logger and making mine and his 3 ones write in some other file (somehow that does not affect when errors happens??).
I am aware that Tornado uses its own 3 logging functions, but somehow I would like to use mine as well, at the same time as keeping the Tornado's ones and writing them all into the same file. Basically reproduce that local behaviour on server but also keeping it when some error happens, of course.
How could I achieve this?
Thanks in advance!
P.S.: if I add a name to the logger, let's say logging.getLogger('Example') and changed log_info_message function to return logging.getLogger('Example').log(20, msg), Tornado's logger would fail and raise error. So that option destroys its own loggers...

It seems the only problem was that, on the server side, tornado was setting the the mininum level for a log message to be written on log file higher (minimum of 40 was required). So that logging.getLogger().log(20, msg) would not write on the logging file but logging.getLogger().log(40, msg) would.
I would like to understand why, so if anybody knows, your knowledge would be more than welcome. For the time being that solution is working though.

tornado.log defines options that can be used to customise logging via command line (check tornado.options) - one of them is logging that defines the log level used. You are likely using this on the server and setting it to error.
When debugging logging I suggest you create a RequestHandler that will log or return the structure of the existing loggers by inspecting the root logger. When you see the structure it is much easier to understand why it works the way it works.

stream logging over udp

I have problem sending loggging stream over udp to targeted third part machine that collects log streams.
I've got just couple of lines of init to appear on remote console. The encoding over the udp should be UTF-8, too.
## setup_loggin function body
handler=logging.StreamHandler()
handler.setFormatter(formatter)
socketHandler=logging.handlers.SysLogHandler(address(SYSLOG_IP, 514))
socketHandler.setFormatter(formatter)
logger = logging.getLogger(name)
if not logger.handlers:
logger.setLevel(log_level)
logger.addHandler(handler)
logger.addHandler(socketHandler)
return logger
The remote server gets just begining of the logging at end results. Probably because it expects UDP to receive "UTF-8", and than parses it through.
Is there a way to change logging encoding to "UTF-8" using SysLogHandler or any other loggin handler.?

Problem solved. Third party syslog was parsing based on RFC3164
The MSG part has two fields known as the TAG field and the CONTENT field.
So, formatter would be something like
formatter_syslog = logging.Formatter(fmt='foo: %(levelname)s - %(module)s.%(funcName)s - %(message)s')

Python: Is it a good idea to pass the logger around?

My API of a web server logs like this:
started
started
succeeded
failed
That's two requests received at the same time. It's hard to tell which one succeeded or failed. To separate requests from each other, I created a random number for each and used it as the name of the logger
logger = logging.getLogger(random_number)
The logs became
[111] started
[222] started
[111] succeeded
[222] failed
Looks clear now, but the problem of this approach is that I have to pass the logger to every related class like this:
def __init__(self, logger):
self._logger = logger
So the question is:
Is this the best way to log context of each request?
If so, is it a good idea to pass the logger around? Is there any way to make the code less verbose?

You should not generate a new logger for each request. You should identify a unique attribute of a request (client IP, URL, session ID, some cookie token...) and add it to the log statement. That way you'll be able to link different log entries of a single request in the log output.
Also, logs should not be passed around. Attribute name from logging.getLogger(name) should be used to obtain the same logger from different locations in the code.
Best to read up on logging in more detail, e.g. here https://docs.python.org/3/library/logging.html where you can find maybe a useful example with the client IP:
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)

How to send python logs to syslogs?

I am using logging for generating logs on my python (2.7) project. I create one instance of logger class on each python file (LOGGER = logging.getLogger(__name__) like this).
I want to send all the logs to syslog. Ways I have found using SysLogHandler needs to specify the address (/dev/log, /var/run/syslog etc.) of syslogs which can be different on different platforms.
Is there any generic way to do it (send the logs to syslog)?

As you found, you configure logging to use the SysLogHandler. As you also noted, the "address" will be different on different systems and configurations. You will need to make that a configurable part of your application.
If you are configuring the logging programmatically, you could implement some heuristics to search for the various UNIX domain socket paths you expect to find.
Note the syslog can use UDP or TCP sockets to log to a remote syslog server instead of using a UNIX domain socket on the local machine.

import sys, logging
class streamToLogger(object):
def __init__(self, logger, log_level = logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.log_level, line.rstrip())
logging.basicConfig(
level = logging.DEBUG,
format = '%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename = os.path.join('/var/log', 'logfile.log'),
filemode = 'a'
)
sys.stdout = streamToLogger(logging.getLogger('STDOUT'), logging.INFO)
sys.stderr = streamToLogger(logging.getLogger('STDERR'), logging.ERROR)

Python logging sometimes writing message to file but not format

Having a new problem where the logger is writing about every other line without the format, just the message.
My code:
import logging
from logging.handlers import RotatingFileHandler
# Set up logging
LOG_FILE = argv[0][:-3] + '.log'
logging.basicConfig(
filename=LOG_FILE,
filemode='a',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
handler = RotatingFileHandler(LOG_FILE,maxBytes=1000000)
logger.addHandler(handler)
def main():
target, command, notify_address, wait = get_args(argv)
logger.info('Checking status of %s every %d minutes.' % (target, wait))
logger.info('Running %s and sending output to %s when online.' % (command, notify_address))
The vars returned from get_args() are all strings, even though wait is a number.
Note that I am not receiving any errors in my IDE or when running.
The output I am getting in my log file:
2015-02-12 16:26:27,483 - INFO - Checking status of <ip address> every 30 minutes.
Running <arbitrary bash command string> and sending output to <my email address> when online.
2015-02-12 16:26:27,483 - INFO - Running <arbitrary bash command string> and sending output to <my email address> when online.
What is causing the second logger.info() to print twice, and only once formatted properly?
I have another script that logs perfectly, no idea what I've done here. (Copy/pasted the logging setup section to be safe)

Are you using loggers at different levels of your code? It sounds like the log messages could be propagating upwards. Try adding
logger.propagate = False
after you add the handler. You can check out the python docs for a more detailed explanation here, but the relevant text below sounds exactly like what you're seeing.
Note If you attach a handler to a logger and one or more of its ancestors, it may emit the same record multiple times. In general, you should not need to attach a handler to more than one logger - if you just attach it to the appropriate logger which is highest in the logger hierarchy, then it will see all events logged by all descendant loggers, provided that their propagate setting is left set to True. A common scenario is to attach handlers only to the root logger, and to let propagation take care of the rest.