We Keep Coding

Tornado substituting custom logger on server (not on local computer)

I have a tornado application and a custom logger method. My code to build and use the custom logger is the following:
def create_logger():
"""
This function creates the logger functionality to be used throughout the Python application
:return: bool - true if successful
"""
# Configuring the logger
filename = "PythonLogger.log"
# Change the current working directory to the logs folder, so that the logs files is written in it.
os.chdir(os.path.normpath(os.path.normpath(os.path.dirname(os.path.abspath(__file__)) + os.sep + os.pardir + os.sep + os.pardir + os.sep + 'logs')))
# Create the logs file
logging.basicConfig(filename=filename, format='%(asctime)s %(message)s', filemode='w')
# Creating the logger
logger = logging.getLogger()
# Setting the threshold of logger to DEBUG
logger.setLevel(logging.NOTSET)
logger.log(0, 'El logger está inicializado')
return True
def log_info_message(msg):
"""
Utility for message logging with code 20
:param msg:
:return:
"""
return logging.getLogger().log(20, msg)
In the code, I initialize the logger and already write a message to it before the Tornado application initialization:
if __name__ == '__main__':
# Logger initialization
create_logger()
# First log message
log_info_message('Initiating Python application')
# Starting Tornado
tornado.options.parse_command_line()
# Specifying what app exactly is being started
server = tornado.httpserver.HTTPServer(test.app)
server.listen(options.port)
try:
if 'Windows_NT' not in os.environ.values():
server.start(0)
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
Then let's say my method get of HTTP request is as follows (only interesting lines):
class API(tornado.web.RequestHandler):
def get(self):
self.write('Get request ')
logging.getLogger("tornado.access").log(20, 'Hola')
logging.getLogger("tornado.application").log(20, '1')
logging.getLogger("tornado.general").log(20, '2')
log_info_message('Received a GET request at: ' + datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
What I see is a difference between local testing and testing on server.
A) On local, I can see log message at first script running, and log messages of requests (after initializing Tornado app) in my log file and the Tornado logs.
B) On server, I only see the first message, not my log messages when Get requests are accepted and also see Tornado's loggers when there's an error, but even don't see the messages produced by Tornado's loggers. I guess that means that somehow Tornado is re-initializing the logger and making mine and his 3 ones write in some other file (somehow that does not affect when errors happens??).
I am aware that Tornado uses its own 3 logging functions, but somehow I would like to use mine as well, at the same time as keeping the Tornado's ones and writing them all into the same file. Basically reproduce that local behaviour on server but also keeping it when some error happens, of course.
How could I achieve this?
Thanks in advance!
P.S.: if I add a name to the logger, let's say logging.getLogger('Example') and changed log_info_message function to return logging.getLogger('Example').log(20, msg), Tornado's logger would fail and raise error. So that option destroys its own loggers...

It seems the only problem was that, on the server side, tornado was setting the the mininum level for a log message to be written on log file higher (minimum of 40 was required). So that logging.getLogger().log(20, msg) would not write on the logging file but logging.getLogger().log(40, msg) would.
I would like to understand why, so if anybody knows, your knowledge would be more than welcome. For the time being that solution is working though.

tornado.log defines options that can be used to customise logging via command line (check tornado.options) - one of them is logging that defines the log level used. You are likely using this on the server and setting it to error.
When debugging logging I suggest you create a RequestHandler that will log or return the structure of the existing loggers by inspecting the root logger. When you see the structure it is much easier to understand why it works the way it works.

Excessive File Handles (on lsof) When Python logging with Starlette and Uvicorn

I have a minimal starlette app running via uvicorn doing nothing but returning a ping. Strangely, even with a single thread, I have 45 file handles to my log file. The handles increase with hits to the app and plateau at 45. I'm counting file handles using lsof:
lsof | grep kml-bkny.log
Of course, this is a minimal re-production. In actuality, my overall app is producing thousands of file handles. Some (~200) I would expect given threads*controller modules. However, I cant account for where thousands of file handles are being created. In my real program they dont plateau either, they grow forever.
here is my app (api.py):
import logging
from starlette.applications import Starlette
from starlette.responses import JSONResponse
logger = logging.getLogger("kml")
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler('/tmp/kml-bkny.log')
logger.addHandler(handler)
def controller_ping(request):
try:
logger.debug("Received /ping request")
return JSONResponse({}, status_code=200)
except Exception as e:
logger.error(e)
return JSONResponse({}, status_code=500)
app = Starlette()
app.add_route(path='/kml/ping', methods=['GET'], route=controller_ping)
I run this on the command line as such:
uvicorn api:app
I test the app using a simple script:
import time
import requests
while True:
time.sleep(1)
for _ in range(10):
r = requests.get(f'http://localhost:8000/kml/ping', timeout=5)
This is running on Ubuntu 18.04 with starlette==0.13.3 and uvicorn==0.11.3 on Python 3.6
As a comparison, there is no such effect when running w/o starlette+uvicorn. The below control program will produce 2 and only 2 file handles on lsof:
import logging
import time
logger = logging.getLogger("kml")
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler('/tmp/kml-bkny.log')
logger.addHandler(handler)
while True:
time.sleep(1)
for _ in range(10):
logger.debug("Received /ping request")
Why would so many file handles be created that I see on lsof when introducing starlette and uvicorn?

Disable logging for a python module in one context but not another

I have some code that uses the requests module to communicate with a logging API. However, requests itself, through urllib3, does logging. Naturally, I need to disable logging so that requests to the logging API don't cause an infinite loop of logs. So, in the module I do the logging calls in, I do logging.getLogger("requests").setLevel(logging.CRITICAL) to mute routine request logs.
However, this code is intended to load and run arbitrary user code. Since the python logging module apparently uses global state to manage settings for a given logger, I am worried the user's code might turn logging back on and cause problems, for instance if they naively use the requests module in their code without realizing I have disabled logging for it for a reason.
How can I disable logging for the requests module when it is executed from the context of my code, but not affect the state of the logger for the module from the perspective of the user? Some sort of context manager that silences calls to logging for code within the manager would be ideal. Being able to load the requests module with a unique __name__ so the logger uses a different name could also work, though it's a bit convoluted. I can't find a way to do either of these things, though.
Regrettably, the solution will need to handle multiple threads, so procedurally turning off logging, then running the API call, then turning it back on will not work as global state is mutated.

I think I've got a solution for you:
The logging module is built to be thread-safe:
The logging module is intended to be thread-safe without any special
work needing to be done by its clients. It achieves this though using
threading locks; there is one lock to serialize access to the module’s
shared data, and each handler also creates a lock to serialize access
to its underlying I/O.
Fortunately, it exposes the second lock mentioned though a public API: Handler.acquire() lets you acquire a lock for a particular log handler (and Handler.release() releases it again). Acquiring that lock will block all other threads that try to log a record that would be handled by this handler until the lock is released.
This allows you to manipulate the handler's state in a thread-safe way. The caveat is this: Because it's intended as a lock around the I/O operations of the handler, the lock will only be acquired in emit(). So only once a record makes it through filters and log levels and would be emitted by a particular handler will the lock be acquired. That's why I had to subclass a handler and create the SilencableHandler.
So the idea is this:
Get the topmost logger for the requests module and stop propagation for it
Create your custom SilencableHandler and add it to the requests logger
Use the Silenced context manager to selectively silence the SilencableHandler
main.py
from Queue import Queue
from threading import Thread
from usercode import fetch_url
import logging
import requests
import time
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
class SilencableHandler(logging.StreamHandler):
def __init__(self, *args, **kwargs):
self.silenced = False
return super(SilencableHandler, self).__init__(*args, **kwargs)
def emit(self, record):
if not self.silenced:
super(SilencableHandler, self).emit(record)
requests_logger = logging.getLogger('requests')
requests_logger.propagate = False
requests_handler = SilencableHandler()
requests_logger.addHandler(requests_handler)
class Silenced(object):
def __init__(self, handler):
self.handler = handler
def __enter__(self):
log.info("Silencing requests logger...")
self.handler.acquire()
self.handler.silenced = True
return self
def __exit__(self, exc_type, exc_value, traceback):
self.handler.silenced = False
self.handler.release()
log.info("Requests logger unsilenced.")
NUM_THREADS = 2
queue = Queue()
URLS = [
'http://www.stackoverflow.com',
'http://www.stackexchange.com',
'http://www.serverfault.com',
'http://www.superuser.com',
'http://travel.stackexchange.com',
]
for i in range(NUM_THREADS):
worker = Thread(target=fetch_url, args=(i, queue,))
worker.setDaemon(True)
worker.start()
for url in URLS:
queue.put(url)
log.info('Starting long API request...')
with Silenced(requests_handler):
time.sleep(5)
requests.get('http://www.example.org/api')
time.sleep(5)
log.info('Done with long API request.')
queue.join()
usercode.py
import logging
import requests
import time
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
def fetch_url(i, q):
while True:
url = q.get()
response = requests.get(url)
logging.info("{}: {}".format(response.status_code, url))
time.sleep(i + 2)
q.task_done()
Example output:
(Notice how the call to http://www.example.org/api isn't logged, and all threads that try to log requests are blocked for the first 10 seconds).
INFO:__main__:Starting long API request...
INFO:__main__:Silencing requests logger...
INFO:__main__:Requests logger unsilenced.
INFO:__main__:Done with long API request.
Starting new HTTP connection (1): www.stackoverflow.com
Starting new HTTP connection (1): www.stackexchange.com
Starting new HTTP connection (1): stackexchange.com
Starting new HTTP connection (1): stackoverflow.com
INFO:root:200: http://www.stackexchange.com
INFO:root:200: http://www.stackoverflow.com
Starting new HTTP connection (1): www.serverfault.com
Starting new HTTP connection (1): serverfault.com
INFO:root:200: http://www.serverfault.com
Starting new HTTP connection (1): www.superuser.com
Starting new HTTP connection (1): superuser.com
INFO:root:200: http://www.superuser.com
Starting new HTTP connection (1): travel.stackexchange.com
INFO:root:200: http://travel.stackexchange.com
Threading code is based on Doug Hellmann's articles on threading and queues.

How do I write Flask's excellent debug log message to a file in production?

I have a Flask application that works well and produces an occasional error, which is visible when it is running with debug=True:
if __name__ == '__main__':
app.run(debug=True)
I get useful error messages such as:
Traceback (most recent call last):
File "./main.py", line 871, in index_route
KeyError: 'stateIIIII'
I would like to get error messages like these saved to a file when I run the application in production (using Lighttpd + fastcgi).
After looking at various StackOverflow questions (http://flask.pocoo.org/docs/errorhandling/, http://docs.python.org/2/library/logging.html, etc.); the Flask mailing list; and a few blogs, it seems there is no easy way just to send all the great error messages to a file - I need to use the Python logging module to customise things. So I came up with the following code.
At the top of my application file I have various imports followed by:
app = Flask(__name__)
if app.debug is not True:
import logging
from logging.handlers import RotatingFileHandler
file_handler = RotatingFileHandler('python.log', maxBytes=1024 * 1024 * 100, backupCount=20)
file_handler.setLevel(logging.ERROR)
app.logger.setLevel(logging.ERROR)
app.logger.addHandler(file_handler)
I have then put the code for each route in a try/except statement and use traceback to work out which line the error came from and print a nice error message:
def some_route():
try:
# code for route in here (including a return statement)
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
app.logger.error(traceback.print_exception(exc_type, exc_value, exc_traceback, limit=2))
return render_template('error.html')
And then right at the end of the file I remove the debug=True statement. Though I don't think I need to do that as the application is being run by a fastcgi server(?) when it is run in production. The last two lines of my application code look like this:
if __name__ == '__main__':
app.run()
I am struggling to get this working. I think the best I have managed is to get a single error log message to be saved in the file using (app.logger.error('test message') ), but it only prints that one message. An attempt to log another error directly after that one is simply ignored.

I don't know why it's not working but I can tell how am doing this.
First of all, you don't need to set the level of app.logger. So remove this line app.logger.setLevel().
You want to save exception and return error page for every view. It is a lot of work to write this code everywhere. Flask provides a method to do this. Define an errorhandler method like this.
#app.errorhandler(500)
def internal_error(exception):
app.logger.error(exception)
return render_template('500.html'), 500
Whenever a view raises an exception, this method will be called and passed the exception as argument. Python logging provides exception method that is used to save full traceback of the exception.
Since this handles all exception, you don't even need to put code in try/except block. Though, if you want to do something before calling the errorhandler(for e.g. rollback session or transaction) then do this:
try:
#code
except:
#code
raise
If you would like the date and time added for each entry in your log file, the following code can be used (in place of the similar code featured in the question).
if app.debug is not True:
import logging
from logging.handlers import RotatingFileHandler
file_handler = RotatingFileHandler('python.log', maxBytes=1024 * 1024 * 100, backupCount=20)
file_handler.setLevel(logging.ERROR)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
file_handler.setFormatter(formatter)
app.logger.addHandler(file_handler)

For those who read this later on.
I think it is better idea to push more useful info into error messages. URL, client IP, user-agent etc. Flask logs exceptions internally (in app.debug==False mode) with Flask.log_exception function. So, instead of logging things manually in #app.errorhandler I do something like this:
class MoarFlask(Flask):
def log_exception(self, exc_info):
"""...description omitted..."""
self.logger.error(
"""
Request: {method} {path}
IP: {ip}
User: {user}
Agent: {agent_platform} | {agent_browser} {agent_browser_version}
Raw Agent: {agent}
""".format(
method = request.method,
path = request.path,
ip = request.remote_addr,
agent_platform = request.user_agent.platform,
agent_browser = request.user_agent.browser,
agent_browser_version = request.user_agent.version,
agent = request.user_agent.string,
user=user
), exc_info=exc_info
)
Then, at configuration time, bind FileHandler to app.logger and go on.
I don't use StreamHandler cause many servers (e.g. uWSGI) like to pollute it
with their own proprietary-wordy-useless-not-turnable-off messages.
Don't be afraid of extending Flask. You'll be forced to do it sooner or later ;)

This is what I generally do when a service requires logging. I'm not a specialist on the subject, but these observations are something to take in consideration, in my humble opinion:
at the beginning of every function (route), create a timestamp object, in order to registry the exact time when the request was made, independently if it was successful or not
use #app.after_request, for registering every successful request
use #app.errorhandler, for registering general errors + Tracebacks
Here is an example that demonstrates this idea:
#/usr/bin/python3
""" Demonstration of logging feature for a Flask App. """
from logging.handlers import RotatingFileHandler
from flask import Flask, request, jsonify
from time import strftime
__author__ = "#ivanleoncz"
import logging
import traceback
app = Flask(__name__)
#app.route("/")
#app.route("/index")
def get_index():
""" Function for / and /index routes. """
return "Welcome to Flask! "
#app.route("/data")
def get_data():
""" Function for /data route. """
data = {
"Name":"Ivan Leon",
"Occupation":"Software Developer",
"Technologies":"[Python, Flask, JavaScript, Java, SQL]"
}
return jsonify(data)
#app.route("/error")
def get_nothing():
""" Route for intentional error. """
return foobar # intentional non-existent variable
#app.after_request
def after_request(response):
""" Logging after every request. """
# This avoids the duplication of registry in the log,
# since that 500 is already logged via #app.errorhandler.
if response.status_code != 500:
ts = strftime('[%Y-%b-%d %H:%M]')
logger.error('%s %s %s %s %s %s',
ts,
request.remote_addr,
request.method,
request.scheme,
request.full_path,
response.status)
return response
#app.errorhandler(Exception)
def exceptions(e):
""" Logging after every Exception. """
ts = strftime('[%Y-%b-%d %H:%M]')
tb = traceback.format_exc()
logger.error('%s %s %s %s %s 5xx INTERNAL SERVER ERROR\n%s',
ts,
request.remote_addr,
request.method,
request.scheme,
request.full_path,
tb)
return "Internal Server Error", 500
if __name__ == '__main__':
handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=3)
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)
logger.addHandler(handler)
app.run(host="127.0.0.1",port=8000)
For more information regarding logrotate and logs on stdout and file at the same time: this Gist

If you are using gunicorn to run your Flask app, you can log all Flask exceptions to the gunicorn logs by adding the gunicorn error handlers to the Flask logger:
In module/__init__.py:
#app.before_first_request
def setup_logging():
if not app.debug:
import logging
gunicorn_logger = logging.getLogger('gunicorn.error')
for handler in gunicorn_logger.handlers:
app.logger.addHandler(handler)

In Development, make sure to set: app.config['PROPAGATE_EXCEPTIONS'] = False. Default is None: https://flask.palletsprojects.com/en/1.1.x/config/

Tornado websocket logging

I'm trying to implement websockets using Tornado webserver.
My setup looks as follows:
from tornado.options import options, define, parse_command_line
import django.core.handlers.wsgi
import logging
import tornado.httpserver
import tornado.ioloop
import tornado.web
import tornado.wsgi
from pogows.tornado_sockets import GetSocketHandler, UpdateSocketHandler
from mobile.cleaner import start_cleaning
define('port', type=int, default=8080)
tornado.options.options['log_file_prefix'].set('/var/www/pogo_django/logs/tornado_server.log')
tornado.options.parse_command_line()
<snip>
def main():
logger = logging.getLogger(__name__)
wsgi_app = tornado.wsgi.WSGIContainer(
django.core.handlers.wsgi.WSGIHandler())
tornado_app = tornado.web.Application(
[
('/hello-tornado', HelloHandler),
('/socket/get', GetSocketHandler),
('/socket/update', UpdateSocketHandler),
('.*', tornado.web.FallbackHandler, dict(fallback=wsgi_app)),
], debug=True)
logger.info("Tornado POGO server starting...")
server = tornado.httpserver.HTTPServer(tornado_app)
server.listen(options.port)
start_cleaning()
tornado.ioloop.IOLoop.instance().start()
So far everything looks fine, tornado logs, I see the info message.
Now, I'm trying to log some stuff from websocket handler classes.
class GetSocketHandler(tornado.websocket.WebSocketHandler):
def open(self):
print "opening"
def on_closed(self):
print "closing"
def on_message(self, message):
last_update=datetime.datetime.utcnow().replace(tzinfo=utc)
try:
print "getting_user"
...
Tornado is governed by supervisord, with the following configuration:
[program:pogo_tornado] command=/var/www/pogo_django/tornado_server.py
user=www-data stdout_logfile=/var/www/pogo_django/logs/pogo_stdout.log
stderr_logfile=/var/www/pogo_django/logs/pogo_stderr.log
environment=PYTHONPATH="/var/www/pogo_django/",DJANGO_SETTINGS_MODULE="pogo.settings"
I tried a few things.
Just use print statements, as you see from the above snippet, hoping for supervisord to catch it and send to stdout/stderr logs.
Create a separate logging.getLogger() instance inside the websocket class and use that.
None of it produces desired results.
When I run tornado from commandline by hand, I do see the print version printed to console, but logging doesn't work anyway.
Where do I go wrong?

Bah, I got it. I was using getLogger() without setting logging level and just blindly logging to DEBUG.
Explicitly using logger.setLevel(logging.DEBUG) showed me my messages in the logs.
Apparently Tornado sets some other level by defaults.. Stupid me.