We Keep Coding

Flask API Not Writing Out Logs on Elastic Beanstalk

I have a Flask API and I am attempting to implement logging. Everything is working fine in my local machine, but nothing seems to work when deployed on AWS Elastic Beanstalk.
I have a file logger.py that creates and configures the logger that looks like this:
import logging
import os
if not os.path.isdir(os.environ.get('LOG_PATH', 'log')):
os.mkdir(os.environ.get('LOG_PATH', 'log'))
# Configure logger
logger = logging.getLogger(__name__)
formatter = logging.Formatter('[%(asctime)s] %(levelname)s in %(module)s: %(message)s')
debug_file_handler = logging.FileHandler(f'{os.environ.get("LOG_PATH", "log")}/debug.log')
info_file_handler = logging.FileHandler(f'{os.environ.get("LOG_PATH", "log")}/info.log')
error_file_handler = logging.FileHandler(f'{os.environ.get("LOG_PATH", "log")}/error.log')
debug_file_handler.setLevel(logging.DEBUG)
info_file_handler.setLevel(logging.INFO)
error_file_handler.setLevel(logging.ERROR)
debug_file_handler.setFormatter(formatter)
info_file_handler.setFormatter(formatter)
error_file_handler.setFormatter(formatter)
logger.addHandler(debug_file_handler)
logger.addHandler(info_file_handler)
logger.addHandler(error_file_handler)
I import this file int my app factory and add the handlers to my app.
# Configure logger
app.logger.removeHandler(default_handler)
app.logger.addHandler(debug_file_handler)
app.logger.addHandler(info_file_handler)
app.logger.addHandler(error_file_handler)
Then in the app factory still, I have a #app.before_request so that logging should occur on every request (for testing purposes).
#app.before_request
def before_request():
elif request.method == 'GET':
app.logger.debug(f'{request.method} {request.base_url}: parameters {dict(request.args)}')
else:
app.logger.debug(f'{request.method} {request.base_url}: parameters {request.json}')
This all works fine on my local machine without a problem. However when I try to deploy to AWS EB, nothing is written to the log files. The log files appear to be created and present when I pull them down via the cli or gui, but nothing is being written to it. I've followed a few tutorials and have added to my .ebextensions file but am still having no luck. Currently my .ebextensions includes a logging.config that looks like this:
files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/applogs.conf" :
mode: "000777"
owner: root
group: root
content: |
/tmp/*.log
"/opt/elasticbeanstalk/tasks/taillogs.d/applogs.conf" :
mode: "000777"
owner: root
group: root
content: |
/tmp/*.log
The log files appear to be included when I request the logs (both tail and full), but the logs are completely empty even after making multiple request. I've also seen a few answers on here saying I need to include chown wsgi:wsgi in this file, but that has not worked either. I've also tried writing the logs to var/app/current/log and that doesn't work either. The log files are always created, but not written to.
For context, the Elastic Beanstalk platform is Python 3.8 running on Amazon Linux 2.
Any help would be greatly appreciated.

Turns out all I needed to do was set the level of the logger to logging.DEBUG so I could see debug logs.
in __init__.py:
app.logger.setLevel(logging.DEBUG)
And since my app name is src when I configure it, I can grab this logger in other modules with:
import logging
logger = logging.getLogger('src')

Which logger to use in a Python Flask app with Connexion

I'm using both Flask and Connexion for a Python based REST API, and it runs within a Docker container. Here is main.py:
import connexion
import logging
from app.log import handler
# initiate swagger/connexion
application = connexion.App(__name__, specification_dir='./')
application.add_api('swagger.yml')
# logging
application.app.logger.handlers.clear()
application.app.logger.addHandler(handler)
application.app.logger.setLevel(logging.DEBUG)
application.app.logger.debug('application starting...')
# if we're running in standalone mode, run the application
if __name__ == '__main__':
application.run(host='0.0.0.0', port=5000, debug=True)
This works fine, and in my syslog server I can see:
2020-01-14 11:03:14,951 app main:DEBUG application starting...
However, I'm not sure how to log correctly from files outside of main.py. For example, I have a status.py which has a single route for GET /status and the code looks like:
import yaml
from flask import current_app
import logging
def read():
# LOG TESTING
current_app.logger.debug('Test using current_app')
logging.getLogger(__name__).debug('Test using getLogger')
print('Test using print')
with open('./swagger.yml', 'r') as f:
y = yaml.load(f)
return {
# .... some data here
}
In my syslog server, I can see:
Test using print
./status.py:22: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
y = yaml.load(f)
I would like to use the same logging mechanism that main.py is using in all of my separate files, but I can only get it to work from main.py, and the only thing that works outside of main.py is the print function, however, as can be seen above, errors also seem to get picked up (albeit with no timestamp).

Please review the docs here. https://flask.palletsprojects.com/en/1.1.x/logging/ You are changing the logging after calling app.log or is it app.logger (I forget) so the application has already started. You need to override the default. The document covers it but here is a gist.
Before you instantiate the Flask App. do this
from logging.config import dictConfig
dictConfig({
'version': 1,
'formatters': {'default': {
'format': '[%(asctime)s] %(levelname)s in %(module)s: %(message)s',
}},
'handlers': {'wsgi': {
'class': 'logging.StreamHandler',
'stream': 'ext://flask.logging.wsgi_errors_stream',
'formatter': 'default'
}},
'root': {
'level': 'INFO',
'handlers': ['wsgi']
}
})
app = Flask(__name__) # important! logging stuff is set before this.
One thing to note is that error from web request get logged differently than the errors outside of web requests (for e.g. jobs, cli etc). The default behavior is to log to standard error which in your case is syslog

unable to load configuration from uwsgi

I have below setup in my Python application
server.py
from bots.flask_app import app
from bots.flask_app.api import api
from bots.flask_app.public import public
from bots import db
from bots.commons.helpers.flask.json.serializer import make_alternative_encoder
from flask_debugtoolbar import DebugToolbarExtension
import logging
import bots.commons.managers.configuration as ConfigurationManager
logger = logging.getLogger()
#######
# Public functions
#######
def setup_db_and_app():
# Flask application bootstrap
config = ConfigurationManager.get_flask_rest_config()
app.config.update(config)
logger.debug('Flask configuration object: %s', app.config)
# MongoDB connection initialization
db.init_app(app)
# Debug toolbar enabled only if Flask in debug mode
if ConfigurationManager.get_raw_flask_rest_config()['flask']['debug']:
DebugToolbarExtension(app)
# Replace the serializer with the custom one (for ObjectId and DateTime serialization)
app.json_encoder = make_alternative_encoder(app.json_encoder)
# Register the components
app.register_blueprint(api)
app.register_blueprint(public)
def start_server():
setup_db_and_app()
logger.debug('Registered routes: %s', app.url_map)
app.run(host='0.0.0.0')
main.py
import bots.flask_app.server as FlaskApp
import bots.commons.managers.log as LogManager
# Logging initialization
LogManager.init_logging()
# Defined in server.py
FlaskApp.start_server()
I am trying to see whether this applicator can bed served by uwsgi as below
uwsgi --socket 0.0.0.0:8080 --protocol=http -w main
The output is as follows
INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
INFO:werkzeug: * Restarting with stat
unable to load configuration from uwsgi
My questions
1. Where can I find the configurations which are causing this issue?
2. Can main.py be defined as a callable and used as a parameter for -w?
This is an app which is already written by someone and I am trying make this application served through uwsgi.
Any suggestions would be helpful
Thanks

I had the 'unable to load configuration from uwsgi' error too. According to flask uwsgi docs：
Please make sure in advance that any app.run() calls you might have in your application file are inside an if __name__ == '__main__': block or moved to a separate file. Just make sure it’s not called because this will always start a local WSGI server which we do not want if we deploy that application to uWSGI.
I move app.run() to if __name__ == '__main__':, and the problem solved. Maybe you can try to put FlaskApp.start_server() under if __name__ == '__main__':.

I'm a bit late to the party, I've encountered this error when I forgot to remove debug=True from app.run(). It makes sense that you can't run the debug server with uwsgi.

How do I write Flask's excellent debug log message to a file in production?

I have a Flask application that works well and produces an occasional error, which is visible when it is running with debug=True:
if __name__ == '__main__':
app.run(debug=True)
I get useful error messages such as:
Traceback (most recent call last):
File "./main.py", line 871, in index_route
KeyError: 'stateIIIII'
I would like to get error messages like these saved to a file when I run the application in production (using Lighttpd + fastcgi).
After looking at various StackOverflow questions (http://flask.pocoo.org/docs/errorhandling/, http://docs.python.org/2/library/logging.html, etc.); the Flask mailing list; and a few blogs, it seems there is no easy way just to send all the great error messages to a file - I need to use the Python logging module to customise things. So I came up with the following code.
At the top of my application file I have various imports followed by:
app = Flask(__name__)
if app.debug is not True:
import logging
from logging.handlers import RotatingFileHandler
file_handler = RotatingFileHandler('python.log', maxBytes=1024 * 1024 * 100, backupCount=20)
file_handler.setLevel(logging.ERROR)
app.logger.setLevel(logging.ERROR)
app.logger.addHandler(file_handler)
I have then put the code for each route in a try/except statement and use traceback to work out which line the error came from and print a nice error message:
def some_route():
try:
# code for route in here (including a return statement)
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
app.logger.error(traceback.print_exception(exc_type, exc_value, exc_traceback, limit=2))
return render_template('error.html')
And then right at the end of the file I remove the debug=True statement. Though I don't think I need to do that as the application is being run by a fastcgi server(?) when it is run in production. The last two lines of my application code look like this:
if __name__ == '__main__':
app.run()
I am struggling to get this working. I think the best I have managed is to get a single error log message to be saved in the file using (app.logger.error('test message') ), but it only prints that one message. An attempt to log another error directly after that one is simply ignored.

I don't know why it's not working but I can tell how am doing this.
First of all, you don't need to set the level of app.logger. So remove this line app.logger.setLevel().
You want to save exception and return error page for every view. It is a lot of work to write this code everywhere. Flask provides a method to do this. Define an errorhandler method like this.
#app.errorhandler(500)
def internal_error(exception):
app.logger.error(exception)
return render_template('500.html'), 500
Whenever a view raises an exception, this method will be called and passed the exception as argument. Python logging provides exception method that is used to save full traceback of the exception.
Since this handles all exception, you don't even need to put code in try/except block. Though, if you want to do something before calling the errorhandler(for e.g. rollback session or transaction) then do this:
try:
#code
except:
#code
raise
If you would like the date and time added for each entry in your log file, the following code can be used (in place of the similar code featured in the question).
if app.debug is not True:
import logging
from logging.handlers import RotatingFileHandler
file_handler = RotatingFileHandler('python.log', maxBytes=1024 * 1024 * 100, backupCount=20)
file_handler.setLevel(logging.ERROR)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
file_handler.setFormatter(formatter)
app.logger.addHandler(file_handler)

For those who read this later on.
I think it is better idea to push more useful info into error messages. URL, client IP, user-agent etc. Flask logs exceptions internally (in app.debug==False mode) with Flask.log_exception function. So, instead of logging things manually in #app.errorhandler I do something like this:
class MoarFlask(Flask):
def log_exception(self, exc_info):
"""...description omitted..."""
self.logger.error(
"""
Request: {method} {path}
IP: {ip}
User: {user}
Agent: {agent_platform} | {agent_browser} {agent_browser_version}
Raw Agent: {agent}
""".format(
method = request.method,
path = request.path,
ip = request.remote_addr,
agent_platform = request.user_agent.platform,
agent_browser = request.user_agent.browser,
agent_browser_version = request.user_agent.version,
agent = request.user_agent.string,
user=user
), exc_info=exc_info
)
Then, at configuration time, bind FileHandler to app.logger and go on.
I don't use StreamHandler cause many servers (e.g. uWSGI) like to pollute it
with their own proprietary-wordy-useless-not-turnable-off messages.
Don't be afraid of extending Flask. You'll be forced to do it sooner or later ;)

This is what I generally do when a service requires logging. I'm not a specialist on the subject, but these observations are something to take in consideration, in my humble opinion:
at the beginning of every function (route), create a timestamp object, in order to registry the exact time when the request was made, independently if it was successful or not
use #app.after_request, for registering every successful request
use #app.errorhandler, for registering general errors + Tracebacks
Here is an example that demonstrates this idea:
#/usr/bin/python3
""" Demonstration of logging feature for a Flask App. """
from logging.handlers import RotatingFileHandler
from flask import Flask, request, jsonify
from time import strftime
__author__ = "#ivanleoncz"
import logging
import traceback
app = Flask(__name__)
#app.route("/")
#app.route("/index")
def get_index():
""" Function for / and /index routes. """
return "Welcome to Flask! "
#app.route("/data")
def get_data():
""" Function for /data route. """
data = {
"Name":"Ivan Leon",
"Occupation":"Software Developer",
"Technologies":"[Python, Flask, JavaScript, Java, SQL]"
}
return jsonify(data)
#app.route("/error")
def get_nothing():
""" Route for intentional error. """
return foobar # intentional non-existent variable
#app.after_request
def after_request(response):
""" Logging after every request. """
# This avoids the duplication of registry in the log,
# since that 500 is already logged via #app.errorhandler.
if response.status_code != 500:
ts = strftime('[%Y-%b-%d %H:%M]')
logger.error('%s %s %s %s %s %s',
ts,
request.remote_addr,
request.method,
request.scheme,
request.full_path,
response.status)
return response
#app.errorhandler(Exception)
def exceptions(e):
""" Logging after every Exception. """
ts = strftime('[%Y-%b-%d %H:%M]')
tb = traceback.format_exc()
logger.error('%s %s %s %s %s 5xx INTERNAL SERVER ERROR\n%s',
ts,
request.remote_addr,
request.method,
request.scheme,
request.full_path,
tb)
return "Internal Server Error", 500
if __name__ == '__main__':
handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=3)
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)
logger.addHandler(handler)
app.run(host="127.0.0.1",port=8000)
For more information regarding logrotate and logs on stdout and file at the same time: this Gist

If you are using gunicorn to run your Flask app, you can log all Flask exceptions to the gunicorn logs by adding the gunicorn error handlers to the Flask logger:
In module/__init__.py:
#app.before_first_request
def setup_logging():
if not app.debug:
import logging
gunicorn_logger = logging.getLogger('gunicorn.error')
for handler in gunicorn_logger.handlers:
app.logger.addHandler(handler)

In Development, make sure to set: app.config['PROPAGATE_EXCEPTIONS'] = False. Default is None: https://flask.palletsprojects.com/en/1.1.x/config/

Debugging a Flask app running in Gunicorn

I've been working on a new dev platform using nginx/gunicorn and Flask for my application.
Ops-wise, everything works fine - the issue I'm having is with debugging the Flask layer. When there's an error in my code, I just get a straight 500 error returned to the browser and nothing shows up on the console or in my logs.
I've tried many different configs/options.. I guess I must be missing something obvious.
My gunicorn.conf:
import os
bind = '127.0.0.1:8002'
workers = 3
backlog = 2048
worker_class = "sync"
debug = True
proc_name = 'gunicorn.proc'
pidfile = '/tmp/gunicorn.pid'
logfile = '/var/log/gunicorn/debug.log'
loglevel = 'debug'
An example of some Flask code that borks- testserver.py:
from flask import Flask
from flask import render_template_string
from werkzeug.contrib.fixers import ProxyFix
app = Flask(__name__)
#app.route('/')
def index():
n = 1/0
return "DIV/0 worked!"
And finally, the command to run the flask app in gunicorn:
gunicorn -c gunicorn.conf.py testserver:app
Thanks y'all

The accepted solution doesn't work for me.
Gunicorn is a pre-forking environment and apparently the Flask debugger doesn't work in a forking environment.
Attention
Even though the interactive debugger does not work in
forking environments (which makes it nearly impossible to use on
production servers) [...]
Even if you set app.debug = True, you will still only get an empty page with the message Internal Server Error if you run with gunicorn testserver:app. The best you can do with gunicorn is to run it with gunicorn --debug testserver:app. That gives you the trace in addition to the Internal Server Error message. However, this is just the same text trace that you see in the terminal and not the Flask debugger.
Adding the if __name__ ... section to the testserver.py and running python testserver.py to start the server in development gets you the Flask debugger. In other words, don't use gunicorn in development if you want the Flask debugger.
app = Flask(__name__)
app.config['DEBUG'] = True
if __name__ == '__main__':
app.run()
## Tip for Heroku users:
Personally I still like to use `foreman start`, instead of `python testserver.py` since [it sets up all the env variables for me](https://devcenter.heroku.com/articles/config-vars#using-foreman). To get this to work:
Contents of Procfile
web: bin/web
Contents of bin/web, file is relative to project root
#!/bin/sh
if [ "$FLASK_ENV" == "development" ]; then
python app.py
else
gunicorn app:app -w 3
fi
In development, create a .env file relative to project root with the following contents (docs here)
FLASK_ENV=development
DEBUG=True
Also, don't forget to change the app.config['DEBUG']... line in testserver.py to something that won't run Flask in debug mode in production.
app.config['DEBUG'] = os.environ.get('DEBUG', False)

The Flask config is entirely separate from gunicorn's. Following the Flask documentation on config files, a good solution would be change my source to this:
app = Flask(__name__)
app.config.from_pyfile('config.py')
And in config.py:
DEBUG = True

For Heroku users, there is a simpler solution than creating a bin/web script like suggested by Nick.
Instead of foreman start, just use foreman run python app.py if you want to debug your application in development.

I had similiar problem when running flask under gunicorn I didn't see stacktraces in browser (had to look at logs every time). Setting DEBUG, FLASK_DEBUG, or anything mentioned on this page didn't work. Finally I did this:
app = Flask(__name__)
app.config.from_object(settings_map[environment])
if environment == 'development':
from werkzeug.debug import DebuggedApplication
app_runtime = DebuggedApplication(app, evalex=False)
else:
app_runtime = app
Note evalex is disabled because interactive debbugging won't work with forking (gunicorn).

I used this:
gunicorn "swagger_server.__main__:app" -w 4 -b 0.0.0.0:8080

You cannot really run it with gunicorn and for example use the flask reload option upon code changes.
I've used following snippets in my api launchpoint:
app = Flask(__name__)
try:
if os.environ["yourapp_environment"] == "local":
run_as_local = True
# some other local configs e.g. paths
app.logger.info('Running server in local development mode!')
except KeyError as err:
if "yourapp_environment" in err.args:
run_as_local = False
# some other production configs e.g. paths
app.logger.info('No "yourapp_environment env" given so app running server in production mode!')
else:
raise
...
...
...
if __name__ == '__main__':
if run_as_local:
app.run(host='127.0.0.1', port='8058', debug=True)
else:
app.run(host='0.0.0.0')
For above solution you need to give export yourapp_environment = "local" in the console.
now I can run my local as python api.py and prod gunicorn --bind 0.0.0.0:8058 api:app
The else statement app.run() is not actually needed, but I keep it for reminding me about host, port etc.

Try setting the debug flag on the run command like so
gunicorn -c gunicorn.conf.py --debug testserver:app
and keep the DEBUG = True in your Flask application. There must be a reason why your debug option is not being applied from the config file but for now the above note should get you going.