This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 7 hours.
StuffHappens wants to draw more attention to this question.
I have django rest framework application with socket.io. To run it in staging I use gunicorn as WSGI-server and GeventWebSocketWorker as a worker. The thing I want to fix is that there's no logs for web requests like this:
[2023-02-10 10:54:21 -0500] [35885] [DEBUG] GET /users
Here's my gunicorn.config.py:
worker_class = "geventwebsocket.gunicorn.workers.GeventWebSocketWorker"
bind = "0.0.0.0:8000"
workers = 1
log_level = "debug"
accesslog = "-"
errorlog = "-"
access_log_format = "%(h)s %(l)s %(u)s %(t)s '%(r)s' %(s)s %(b)s '%(f)s' '%(a)s'"
Here's the command in docker compose I use deploy app:
command:
- gunicorn
- my_app.wsgi:application
I saw the issue was discussed in gitlab: https://gitlab.com/noppo/gevent-websocket/-/issues/16 but still I have no idea how to fix it.
It looks like you've already configured your gunicorn settings to output debug-level logs, so it's possible that the logs you're looking for are being written to the console output or error stream rather than to a log file.
One thing you could try is redirecting the console output and error stream to log files using the --access-logfile and --error-logfile options, like this:
accesslog = "/path/to/access.log"
errorlog = "/path/to/error.log"
This will redirect the access and error logs to the specified files, rather than writing them to the console output. You can then check the log files to see if the debug-level logs are being written there.
Alternatively, you could try using a logging library like Python's built-in logging module to output more detailed logs. Here's an example configuration that might work for your use case:
import logging
# Create a logger with the name of your application
logger = logging.getLogger('my_app')
# Set the logging level to debug
logger.setLevel(logging.DEBUG)
# Create a stream handler to output logs to the console
handler = logging.StreamHandler()
# Add a formatter to the handler to format the log messages
formatter = logging.Formatter('[%(asctime)s] [%(levelname)s] %(message)s')
handler.setFormatter(formatter)
# Add the handler to the logger
logger.addHandler(handler)
You can place this code in a file called logging_config.py, and then import it in your Django settings file and add it to the LOGGING setting:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
},
'root': {
'handlers': ['console'],
'level': 'DEBUG',
},
'loggers': {
'my_app': {
'handlers': ['console'],
'level': 'DEBUG',
},
},
}
This will configure the logger to output log messages with a timestamp, log level, and message to the console. You can then add logging statements in your Django views or other code to output debug-level logs when certain events occur:
import logging
logger = logging.getLogger('my_app')
def my_view(request):
logger.debug(f'GET {request.path}')
# ... rest of the view logic ...
This will output a log message like [2023-02-19 09:12:43,244] [DEBUG] GET /users when the view is accessed with a GET request.
I hope this helps!
Try adding - --access-logfile=/path/to/access.log parameter to your gunicorn command to write to a file or - --access-logfile=- to write to stdout.
https://docs.gunicorn.org/en/stable/settings.html#accesslog
My celery tasks don't send an email to application admins each time I call logger.critical.
I'm building a Django application. The current configuration of my project, allows the admins of the application to receive an email each time a logger.critical message is created. This was pretty straight forward to set up, I just followed the documentation on both projects (celery and Django). For some reason, which I'm not figuring out, the code that runs inside a celery task, does not have the same behavior, It doesn't send an email to the application admin each time the logger.critical message is created.
Does celery actually even allow this to be done?
Am I missing some configuration?
Does anyone got this problem and was able to solve it?
Using:
Django 1.11
Celery 4.3
Thanks for any help.
As stated in the documentation Celery overrides the current logging configuration to apply its own, it also says that you can set CELERYD_HIJACK_ROOT_LOGGER to False in your Django settings to prevent this behavior, what is not well documented is that this is not really working at the moment.
In my opinion you have 2 options:
1. Prevent Celery to override your configuration (really) using the setup_logging signal
Open your celery.py file and add the following:
from celery.signals import setup_logging
#setup_logging.connect
def config_loggers(*args, **kwags):
pass
After that your file should look more or less like this:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.signals import setup_logging
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#setup_logging.connect
def config_loggers(*args, **kwags):
pass
However I would avoid this option unless you have a really good reason because in this way you will lose the default task logging handled by Celery, which is quite good to have.
2. Use a specific logger
You can define a custom logger in your Django LOGGING configuration and use it in your task, eg:
Django settings:
LOGGING = {
# ... other configs ...
'handlers': {
'my_email_handler': {
# ... handler configuration ...
},
},
'loggers': {
# ... other loggers ...
'my_custom_logger': {
'handlers': ['my_email_handler'],
'level': 'CRITICAL',
'propagate': True,
},
},
}
Tasks:
import logging
logger = logging.getLogger('my_custom_logger')
#shared_task
def log():
logger.critical('Something bad happened!')
I believe this is the best approach for you because, as far as I understand, you need to manually log messages, and this allows you to keep using the Celery logging system.
From Celery docs:
By default any previously configured handlers on the root logger will be removed. If you want to customize your own logging handlers, then you can disable this behavior by setting worker_hijack_root_logger = False.
Celery installs own logger that you can obtain using get_task_logger() call. I assume you wrote your own logger that does the logic you described in the original question. Read more about Celery logging to find out how to disable this behaviour and tweak Celery to your needs.
I need to enable logging if a code has been executed from console application (as manage.py <cmd>) and to disable logging if HTTPRequest is being processed. Probably filter can be very useful here.
LOGGING = {
...
'filters': {
'require_debug_false': {
'()': 'IsFromHTTPRequest'
}
},
...
}
But what is the best way to define if command has been executed or HTTPRequest is being processed? Traceback analysis?
Well, there is no a good way to do. But here is what we do when we need distinguish manage.py jenkins from regular http requests:
Add to settings.py
import sys
JENKINS = "jenkins" in sys.argv
Then you can use that variable whenever you need. In the log filter as well.
I have a Flask server running in standalone mode (using app.run()). But, I don't want any messages in the console, like
127.0.0.1 - - [15/Feb/2013 10:52:22] "GET /index.html HTTP/1.1" 200 -
...
How do I disable verbose mode?
You can set level of the Werkzeug logger to ERROR, in that case only errors are logged:
import logging
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
Here is a fully working example tested on OSX, Python 2.7.5, Flask 0.10.0:
from flask import Flask
app = Flask(__name__)
import logging
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run()
This solution provides you a way to get your own prints and stack traces but without information level logs from flask suck as 127.0.0.1 - - [15/Feb/2013 10:52:22] "GET /index.html HTTP/1.1" 200
from flask import Flask
import logging
app = Flask(__name__)
log = logging.getLogger('werkzeug')
log.disabled = True
None of the other answers worked correctly for me, but I found a solution based off Peter's comment. Flask apparently no longer uses logging for logging, and has switched to the click package. By overriding click.echo and click.secho I eliminated Flask's startup message from app.run().
import logging
import click
from flask import Flask
app = Flask(__name__)
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
def secho(text, file=None, nl=None, err=None, color=None, **styles):
pass
def echo(text, file=None, nl=None, err=None, color=None, **styles):
pass
click.echo = echo
click.secho = secho
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run()
Between setting the logging level to ERROR and overriding the click methods with empty functions, all non-error log output should be prevented.
To suppress Serving Flask app ...:
os.environ['WERKZEUG_RUN_MAIN'] = 'true'
app.run()
In case you are using WSGI server , please set the log to None
gevent_server = gevent.pywsgi.WSGIServer(("0.0.0.0", 8080), app, log=None)
#Drewes solution works most of the time, but in some cases, I still tend to get werkzeug logs. If you really don't want to see any of them, I suggest you disabling it like that.
from flask import Flask
import logging
app = Flask(__name__)
log = logging.getLogger('werkzeug')
log.disabled = True
app.logger.disabled = True
For me it failed when abort(500) was raised.
Late answer but I found a way to suppress EACH AND EVERY CONSOLE MESSAGE (including the ones displayed during an abort(...) error).
import os
import logging
logging.getLogger('werkzeug').disabled = True
os.environ['WERKZEUG_RUN_MAIN'] = 'true'
This is basically a combination of the answers given by Slava V and Tom Wojcik
Another reason you may want to change the logging output is for tests, and redirect the server logs to a log file.
I couldn't get the suggestion above to work either, it looks like loggers are setup as part of the app starting. I was able to get it working by changing the log levels after starting the app:
... (in setUpClass)
server = Thread(target=lambda: app.run(host=hostname, port=port, threaded=True))
server.daemon = True
server.start()
wait_for_boot(hostname, port) # curls a health check endpoint
log_names = ['werkzeug']
app_logs = map(lambda logname: logging.getLogger(logname), log_names)
file_handler = logging.FileHandler('log/app.test.log', 'w')
for app_log in app_logs:
for hdlr in app_log.handlers[:]: # remove all old handlers
app_log.removeHandler(hdlr)
app_log.addHandler(file_handler)
Unfortunately the * Running on localhost:9151 and the first health check is still printed to standard out, but when running lots of tests it cleans up the output a ton.
"So why log_names?", you ask. In my case there were some extra logs I needed to get rid of. I was able to find which loggers to add to log_names via:
from flask import Flask
app = Flask(__name__)
import logging
print(logging.Logger.manager.loggerDict)
Side note: It would be nice if there was a flaskapp.getLogger() or something so this was more robust across versions. Any ideas?
Some more key words: flask test log remove stdout output
thanks to:
http://code.activestate.com/lists/python-list/621740/ and
How to change filehandle with Python logging on the fly with different classes and imports
I spent absolute ages trying to get rid of these response logs with all the different solutions, but as it turns out it wasn't Flask / Werkzeug but Gunicorn access logs dumped on stderr...
The solution was replacing the default access log handler with NullHandler by adding this block in the Gunicorn config file:
logconfig_dict = {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"console": {"class": "logging.StreamHandler", "level": "INFO"},
"null": {"class": "logging.NullHandler"},
},
"loggers": {
"gunicorn.error": {"level": "INFO", "propagate": False, "handlers": ["console"]},
"gunicorn.access": {"level": "INFO", "propagate": False, "handlers": ["null"]},
},
}
somehow none of the above options, including .disabled = True, worked for me.
The following did the trick though:
logging.getLogger('werkzeug').setLevel(logging.CRITICAL)
Using the latest versions as of November 2021 under Python 3.7.3:
pip3 list | grep -E "(connexion|Flask|Werkzeug)"
connexion2 2.10.0
Flask 2.0.2
Werkzeug 2.0.2
Here's the answer to disable all logging, including on werkzeug version 2.1.0 and newer:
import flask.cli
flask.cli.show_server_banner = lambda *args: None
import logging
logging.getLogger("werkzeug").disabled = True
My current workaround to silence Flask as of 2022-10.
It disables the logging and the startup banner:
class WSServer:
...
#staticmethod
def _disabled_server_banner(*args, **kwargs):
"""
Mock for the show_server_banner method to suppress spam to stdout
"""
pass
def _run_server(self):
"""
Winds-up the server.
"""
# disable general logging
log = logging.getLogger('werkzeug')
log.setLevel(logging.CRITICAL)
# disable start-up banner
from flask import cli
if hasattr(cli, "show_server_banner"):
cli.show_server_banner = self._disabled_server_banner
...
A brute force way to do it if you really don't want anything to log into the console beside print() statements is to logging.basicConfig(level=logging.FATAL). This would disable all logs that are of status under fatal. It would not disable printing but yeah, just a thought :/
EDIT:
I realized it would be selfish of me not to put a link to the documentation I used :)
https://docs.python.org/3/howto/logging.html#logging-basic-tutorial
The first point: In according to official Flask documentation, you shouldn't run Flask application using app.run(). The best solution is using uwsgi, so you can disable default flask logs using command "--disable-logging"
For example:
uwsgi --socket 0.0.0.0:8001 --disable-logging --protocol=http -w app:app
I'm testing some Django models with bog-standerd django.test.Testcase. My models.py writes to a debug log, using the following init code:
import logging
logger = logging.getLogger(__name__) # name is myapp.models
and then I write to the log with:
logger.debug("Here is my message")
In my settings.py, I've set up a single FileHandler, and a logger for myapp, using that handler and only that handler. This is great. I see messages to that log. When I'm in the Django shell, I only see messages to that log.
When, however, I run my test suite, my test suite console also sees all those messages. It's using a different formatter that I haven't explicitly defined, and it's writing to stderr. I don't have a log handler defined that writes to stderr.
I don't really want those messages spamming my console. I'll tail my log file if I want to see those messages. Is there a way to make it stop? (Yes, I could redirect stderr, but useful output goes to stderr as well.)
Edit: I've set up two handlers in my settings.py:
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'django.utils.log.NullHandler',
},
'logfile' : {
'level':'DEBUG',
'class':'logging.FileHandler',
'filename':'%s/log/development.log' % PROJECT_DIR,
'formatter': 'simple'
},
},
and tried this:
'loggers': {
'django': {
'level': 'DEBUG',
'handlers': ['null']
},
'myapp': {
'handlers': ['logfile'],
'level':'DEBUG',
},
... but the logging / stderr dumping behavior remains the same. It's like I'm getting another log handler when I'm running tests.
It's not clear from your config snippet which handlers, if any, are configured for the root logger. (I'm also assuming you're using Django 1.3.) Can you investigate and tell us what handlers have been added to the root logger when you're running tests? AFAICT Django doesn't add anything - perhaps some code you're importing does a call to basicConfig without you realising it. Use something like ack-grep to look for any occurrences of fileConfig, dictConfig, basicConfig, and addHandler - all of which could be adding a handler to the root logger.
Another thing to try: set the propagate flag to False for all top-level loggers (like "django", but also those used by your modules - say "myapp"). Does that change things?