I'm building a python application with also a web interface, with Flask web framework.
It runs on Flask internal server in debug/dev mode and in production mode it runs on tornado as wsgi container.
This is how i've set up my logger:
log_formatter = logging.Formatter('%(asctime)s [%(levelname)s] %(message)s')
file_handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=5 * 1024 * 1024, backupCount=10)
file_handler.setFormatter(log_formatter)
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(log_formatter)
log = logging.getLogger('myAppLogger')
log.addHandler(file_handler)
log.addHandler(console_handler)
To add my logger to the Flask app i tried this:
app = Flask('system.web.server')
app.logger_name = 'myAppLogger'
But the log still going to the Flask default log handler, and in addition, I didn't found how to customize the log handlers also for the Tornado web server.
Any help is much appreciated,
thanks in advance
AFAIK, you can't change the default logger in Flask. You can, however, add your handlers to the default logger:
app = Flask('system.web.server')
app.logger.addHandler(file_handler)
app.logger.addHandler(console_handler)
Regarding my comment above - "Why would you want to run Flask in tornado ...", ignore that. If you are not seeing any performance hit, then clearly there's no need to change your setup.
If, however, in future you'd like to migrate to a multithreaded container, you can look into uwsgi or gunicorn.
I managed to do it, multiple handler, each doing their thing, so that ERROR log will not show on the INFO log as well and end up with duplicate info grrr:
app.py
import logging
from logging.handlers import RotatingFileHandler
app = Flask(__name__)
# Set format that both loggers will use:
formatter = logging.Formatter("[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s")
# Set logger A for known errors
log_handler = RotatingFileHandler('errors.log', maxBytes=10000, backupCount=1)
log_handler.setFormatter(formatter)
log_handler.setLevel(logging.INFO)
a = logging.getLogger('errors')
a.addHandler(log_handler)
# Set logger B for bad exceptions
exceptions_handler = RotatingFileHandler('exceptions.log', maxBytes=10000, backupCount=1)
exceptions_handler.setFormatter(formatter)
exceptions_handler.setLevel(logging.ERROR)
b = logging.getLogger('exceptions')
b.addHandler(exceptions_handler)
...
whatever_file_where_you_want_to_log.py
import logging
import traceback
# Will output known error messages to 'errors.log'
logging.getLogger('errors').error("Cannot connect to database, timeout")
# Will output the full stack trace to 'exceptions.log', when trouble hits the fan
logging.getLogger('exceptions').error(traceback.format_exc())
Related
I recently set up logging in my Flask app such that it would log to files and stdout. I was inserting logging statements like
current_app.logger.info('Logging message')
into routes so that I could see logging messages when I ran pytest tests on the routes. This worked (logged to the files and console) for a little while, but at some point it stopped logging to the console while continuing to log to the log files.
I'm not sure what could have caused logging to stdout to stop. The only thing that comes to mind is that I attempted to add a logging statement to the test module at one point. This caused an error, which may have had something to do with the logger not being configured in the app defined by my application factory pytest fixture. However, I've since deleted this logging statement from the test module, and I'm still not getting the console logs that I got previously.
Can anyone see any reason why logging to the console would have stopped?
My logging configuration:
__init__.py:
def create_app(test_config=None):
app = Flask('project_name', instance_relative_config=True)
...
if not app.debug:
if not os.path.exists('logs'):
os.mkdir('logs')
file_handler = RotatingFileHandler('logs/trivia.log', maxBytes=10240, backupCount=10)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'))
file_handler.setLevel(logging.INFO)
app.logger.addHandler(file_handler)
stream_handler = StreamHandler(sys.stdout)
stream_handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'
))
stream_handler.setLevel(logging.INFO)
app.logger.addHandler(stream_handler)
app.logger.setLevel(logging.INFO)
routes.py
#bp.route('/add', methods=['POST'])
def add_question():
current_app.logger.info("Received request to /add")
...
This happened because logging messages only get displayed in the console when a pytest test fails. My tests were all passing for a while, so I wasn't seeing the logging messages.
I'd like to use logging in my Flask app by simply calling logging.getLogger('myapi') in each of the files where I need to log.
There should be a single place to define the handler and format for this "global" application logger. This works, but Flask is also continually logging its own logs in the default format. These logs only exist when I import the library fbprophet. I would like to prevent Flask from logging these extra, unformatted, duplicate logs.
(Flask also has a werkzeug logger, which is fine and can stay.)
Code:
import sys
import logging
import fbprophet
from flask import Flask, jsonify
from werkzeug.serving import run_simple
# Set up my custom global logger
log = logging.getLogger('myapi')
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter('*** %(asctime)s %(levelname)s %(message)s'))
log.addHandler(handler)
log.setLevel(logging.DEBUG)
def create_app(config={}):
''' Application factory to create and configure the app '''
app = Flask('myapi', instance_relative_config=False)
app.config.from_mapping(config)
log.info('TEST INFO')
log.debug('TEST DEBUG')
#app.route('/health')
def health():
log.info('Health OK')
return 'OK'
return app
if __name__ == '__main__':
dev_config = {'SECRET_KEY': 'dev', 'DEBUG': False}
app = create_app(dev_config)
run_simple('localhost', 5000, app)
Output:
I'm expecting to see logs prefixed with ***. The ones starting with LEVEL only appear when I import facebook prophet.
* Serving Flask app "main.py" (lazy loading)
* Environment: development
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
*** 2019-06-05 14:17:56,702 INFO TEST INFO # good log
INFO:myapi:TEST INFO # bad dupe log
*** 2019-06-05 14:17:56,702 DEBUG TEST DEBUG
DEBUG:myapi:TEST DEBUG
*** 2019-06-05 14:17:56,703 INFO TEST INFO
INFO:myapi:TEST INFO
*** 2019-06-05 14:17:56,703 DEBUG TEST DEBUG
DEBUG:myapi:TEST DEBUG
*** 2019-06-05 14:18:10,405 INFO Health OK
INFO:myapi:Health OK
127.0.0.1 - - [05/Jun/2019 14:18:10] "GET /health HTTP/1.1" 200 - # good werkzeug api log
INFO:werkzeug:127.0.0.1 - - [05/Jun/2019 14:18:10] "GET /health HTTP/1.1" 200 - # bad dupe log
More explanation:
I've tried setting the app's logger too, but I don't want to have to call current_app.logger from other modules.
I tried disabling Flask's logger with logging.getLogger('flask.app').handlers.clear() but this also doesn't work.
When importing fbprophet, I get the below console errors (from prophet):
ERROR:fbprophet:Importing matplotlib failed. Plotting will not work.
ERROR:fbprophet:Importing matplotlib failed. Plotting will not work.
*** 2019-06-05 14:29:06,488 INFO TEST INFO
INFO:myapi:TEST INFO
I thought this could be causing the issue, so I fixed the errors following this. But Flask is still logging the extra logs.
import plotly
import matplotlib as mpl
mpl.use('TkAgg')
import fbprophet
Summary:
Looking for formatted global logging in Flask with no duplicate logs. I just need my global logging.getLogger('myapi') and the werkzeug API logs.
I've had the same problem and spent hours to resolve it. Actually it's not even related to Flask (I figured this out after few hours).
Even in a simple script you'll have the duplicated log in your handler.
The only working solution seems to be to add: logger.propagate = False in your own handler. This will prevent the log to be passed to handlers of higher level (ancestor) loggers i.e loggers created by Prophet (even if I dont see where this hierarchy resides exactly).
Found in this answer: https://stackoverflow.com/a/55877763/3615718
You can search through all the registered loggers and change their settings as needed. It sounds like fbprophet is probably setting its own logger instance, so hopefully it will get set to the level you want if you do something like this:
for logger_name in logging.root.manager.loggerDict:
print(f"found a logger:{logger_name}")
logger = logging.getLogger(logger_name)
logger.setLevel(logging.ERROR)
if logger_name == 'myapi':
logger.setLevel(logging.INFO)
I am logging to a log file in a Flask application under gunicorn and nginx using the following setup:
def setup_logging():
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('[%(asctime)s][PID:%(process)d][%(levelname)s][%(name)s.%(funcName)s()] %(message)s')
stream_handler.setFormatter(formatter)
stream_handler.setLevel("DEBUG")
logging.getLogger().addHandler(stream_handler)
file_handler = RotatingFileHandler("log.txt", maxBytes=100000, backupCount=10)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(pathname)s:%(lineno)d]'
))
file_handler.setLevel("DEBUG")
logging.getLogger().addHandler(file_handler)
logging.getLogger().setLevel("DEBUG")
Then inializing logging prior to creating the app:
setup_logging()
def create_app(config_name):
app = Flask(__name__)
Then in modules I am logging using:
import logging
logger = logging.getLogger(__name__)
x = 2
logger.debug('x: {0}' x)
Logging works OK on my local machine - both to stdout and log.txt
However when I run the application on a remote server nothing gets written to log.txt. I have deployed as a user with read and write permission on log.txt on the remote system.
I have tried initializing the app on the remote server with DEBUG = True, still nothing written to log file. The only way I can get view any logs is by viewing /var/log/supervisor/app-stdout---supervisor-nnn.log files but these dont show all logging output
Using the answer from HolgerShurig here Flask logging - Cannot get it to write to a file on the server log file I get only named logger output (ie no output from module level logging)
2017-10-21 00:32:45,125 - file - DEBUG - Debug FILE
running same code lo local machine I get
2017-10-21 08:35:39,046 - file - DEBUG - Debug FILE
2017-10-21 08:35:42,340 - werkzeug - INFO - * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) ie
2017-10-21 08:38:46,236 [MainThread ] [INFO ] 127.0.0.1 - - [21/Oct/2017 08:38:46] "[37mGET /blah/blah HTTP/1.1[0m" 200 -
I then changed the logging config to:
def setup_logging(app):
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('[%(asctime)s][PID:%(process)d][%(levelname)s][%(lineno)s][%(name)s.%(funcName)s()] %(message)s')
stream_handler.setFormatter(formatter)
stream_handler.setLevel(Config.LOG_LEVEL)
app.logger.addHandler(stream_handler)
file_handler = RotatingFileHandler(Config.LOGGING_FILE, maxBytes=Config.LOGGING_MAX_BYTES, backupCount=Config.LOGGING_BACKUP_COUNT)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(pathname)s:%(lineno)d]'
))
file_handler.setLevel(Config.LOG_LEVEL)
loggers = [app.logger]
for logger in loggers:
logger.addHandler(file_handler)
app.logger.setLevel(Config.LOG_LEVEL)
app.logger.debug('this message should be recorded in the log file')
and set to us this just after creating the Flask app:
setup_logging(app)
I each module I am still using
import logging
logger = logging.getLogger(__name__)
#for example
def example():
logger.debug('debug')
logger.info('info')
logger.warn('warn')
When I run the application on ther server with
gunicorn manage:app
The only thing printed in the log.txt file is
2017-10-21 02:48:32,982 DEBUG: this message should be recorded in the log file [in /../../__init__.py:82]
But locally MainThread processeses are shown as well
Any ideas?
If your configuration is working on your local machine and not working on your remote server, then your problem is about permission, regarding the file or the directory where the logfile resides.
Here's something that could help you.
Besides that, here's a Gist which could give another perspective regarding logging configuration for Flask applications.
I am building an applicatrion with Flask and Celery and I am trying to send my application logs to Papertrail. This works fine for my regular (synchronous) application logs. The configuration looks like this:
import logging
from logging.handlers import SysLogHandler
import socket
class ContextFilter(logging.Filter):
hostname = socket.gethostname()
def filter(self, record):
record.hostname = ContextFilter.hostname
return True
f = ContextFilter()
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addFilter(f)
formatter = logging.Formatter(
"%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
syslog = SysLogHandler(address=('<myapp>.papertrailapp.com', <port>))
syslog.setFormatter(formatter)
logger.addHandler(syslog)
I have tried adding this logger to Celery tasks but all I see is output in sdout and nothing in Papertrail. Does Celery do something to get around the normal logging flow?
I realize Celery has a task-specific logger but I cannot find any documentation on how this could be configured with Celery.
If I read this correctly, the secret is to call the function redirect_stdouts_to_logger to send stdout to your SysLogHandler instance. Celery's docs have more.
I have a pretty weird problem with my logging facility i use inside django/python. The logging is not working anymore since i upgraded to django 1.3. It seems to be related to the logging level and the 'debug=' setting in the settings.py file.
1) When i log INFO messages and debug=False, the logging won't happen, my file doesn't get appended.
2) When i log WARNING messages and debug=False, the logging works perfectly like i want it to, the file gets appended
3) When i log INFO messages and debug=True, the logging seems to work, the file get appended.
How could i log INFO messages with debug=False? It worked before django 1.3... is there somewhere a mysterious setting which do the trick? Underneath there is a sample code:
views.py:
import logging
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(levelname)s %(message)s',
filename='/opt/general.log',
filemode='a')
def create_log_file(filename, log_name, level=logging.INFO):
handler = logging.handlers.TimedRotatingFileHandler(filename, 'midnight', 7, backupCount=10)
handler.setLevel(level)
formatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s', '%a, %Y-%m-%d %H:%M:%S')
handler.setFormatter(formatter)
logging.getLogger(log_name).addHandler(handler)
create_log_file('/opt/test.log', 'testlog')
logger_test = logging.getLogger('testlog')
logger_test.info('testing the logging functionality')
With this code the logging does not work in Django 1.3 with debug set to False in the settings.py file. When i should do like this:
logger_test.warning('testing the logging functionality')
This works perfectly when debug is set to False. The levels DEBUG and INFO aint logging but WARNING,ERROR and CRITICAL are doing their job...
Does anyone have an idea?
Since Django 1.3 contains its own logging configuration, you need to ensure that anything you're doing doesn't clash with it. For example, if the root logger has handlers already configured by Django by the time your module first gets imported, your basicConfig() call won't have any effect.
What you're describing is the normal logging situation - WARNINGs and above get handled, while INFO and DEBUG are suppressed by default. It looks as if your basicConfig() is not having any effect; You should consider replacing your basicConfig() call with the appropriate logging configuration in settings.py, or at least investigate the root logging level and what handlers are attached to it, at the time of your basicConfig() call.