We Keep Coding

How to I set logging for gunicorn GeventWebSocketWorker

This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 7 hours.
StuffHappens wants to draw more attention to this question.
I have django rest framework application with socket.io. To run it in staging I use gunicorn as WSGI-server and GeventWebSocketWorker as a worker. The thing I want to fix is that there's no logs for web requests like this:
[2023-02-10 10:54:21 -0500] [35885] [DEBUG] GET /users
Here's my gunicorn.config.py:
worker_class = "geventwebsocket.gunicorn.workers.GeventWebSocketWorker"
bind = "0.0.0.0:8000"
workers = 1
log_level = "debug"
accesslog = "-"
errorlog = "-"
access_log_format = "%(h)s %(l)s %(u)s %(t)s '%(r)s' %(s)s %(b)s '%(f)s' '%(a)s'"
Here's the command in docker compose I use deploy app:
command:
- gunicorn
- my_app.wsgi:application
I saw the issue was discussed in gitlab: https://gitlab.com/noppo/gevent-websocket/-/issues/16 but still I have no idea how to fix it.

It looks like you've already configured your gunicorn settings to output debug-level logs, so it's possible that the logs you're looking for are being written to the console output or error stream rather than to a log file.
One thing you could try is redirecting the console output and error stream to log files using the --access-logfile and --error-logfile options, like this:
accesslog = "/path/to/access.log"
errorlog = "/path/to/error.log"
This will redirect the access and error logs to the specified files, rather than writing them to the console output. You can then check the log files to see if the debug-level logs are being written there.
Alternatively, you could try using a logging library like Python's built-in logging module to output more detailed logs. Here's an example configuration that might work for your use case:
import logging
# Create a logger with the name of your application
logger = logging.getLogger('my_app')
# Set the logging level to debug
logger.setLevel(logging.DEBUG)
# Create a stream handler to output logs to the console
handler = logging.StreamHandler()
# Add a formatter to the handler to format the log messages
formatter = logging.Formatter('[%(asctime)s] [%(levelname)s] %(message)s')
handler.setFormatter(formatter)
# Add the handler to the logger
logger.addHandler(handler)
You can place this code in a file called logging_config.py, and then import it in your Django settings file and add it to the LOGGING setting:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
},
'root': {
'handlers': ['console'],
'level': 'DEBUG',
},
'loggers': {
'my_app': {
'handlers': ['console'],
'level': 'DEBUG',
},
},
}
This will configure the logger to output log messages with a timestamp, log level, and message to the console. You can then add logging statements in your Django views or other code to output debug-level logs when certain events occur:
import logging
logger = logging.getLogger('my_app')
def my_view(request):
logger.debug(f'GET {request.path}')
# ... rest of the view logic ...
This will output a log message like [2023-02-19 09:12:43,244] [DEBUG] GET /users when the view is accessed with a GET request.
I hope this helps!

Try adding - --access-logfile=/path/to/access.log parameter to your gunicorn command to write to a file or - --access-logfile=- to write to stdout.
https://docs.gunicorn.org/en/stable/settings.html#accesslog

Server error (500) Django deployment on heroku

I turned debug to False in my settings.py file (note that before I turned it to false, everything was workign perfectly) and when I ran git push heroku master and went to my website, the home page was working but that wasn't the case for all of my pages: on some, I got an error saying Server Error (500)
Here is my settings.py code :
DEBUG = False
ALLOWED_HOSTS = ["hacka-labs.herokuapp.com"]
What I have noticed is that the urls where I pass an id aren't working
Please help me if you know the answer

I had a similar problem and it's hard to figure out when DEBUG is set to False. When I reconfigured it back to DEBUG = True everything worked fine again. So the problem was
You don't generally see this error in development because when DEBUG = True, ManifestStaticFilesStorage switches to non-hashed urls.
https://stackoverflow.com/a/51060143/7986808
The solution to the problem in your case may differ from mine. But you can find the problem by changing the logging method of your Django project, so you can see more info in the command line when running heroku logs -t -a <heroku-app> even in DEBUG = False mode.
Change your django logging settings in settings.py as follow:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': ('%(asctime)s [%(process)d] [%(levelname)s] '
'pathname=%(pathname)s lineno=%(lineno)s '
'funcname=%(funcName)s %(message)s'),
'datefmt': '%Y-%m-%d %H:%M:%S'
},
'simple': {
'format': '%(levelname)s %(message)s'
}
},
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'logging.NullHandler',
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'verbose'
}
},
'loggers': {
'django': {
'handlers': ['console'],
'level': 'DEBUG',
'propagate': True,
},
'django.request': {
'handlers': ['console'],
'level': 'DEBUG',
'propagate': False,
},
}
}
Then run heroku logs -t -a <heroku-app> and open the url where you previously got 500 Error you will find out the problem in logs. Hope this will help you.
In case there are some static files missing just switch your static root in settings.py as follow STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles'). Running heroku run python manage.py collectstatic -a <heroku-app> creates staticfiles directory.

I had the same error.
I solved issue by removing import django heroku in settings.py

I had the same error while deploying react and django project on heroku.
I had been debugging for 2 days, doing all things like allowed host, staticfiles, json, whitenoise things but that still didn't solve the problem.
If your browswer throws error 500 then set DEBUG=True in settings.py. If it throws a TemplateDoesNotExist error at index.html then changing views.py may help solve that. Don't forget to setDEBUG=False before deploying.
I solved it by changing some code in views.py.
def index(request):
return render(request,'ur_app_name.html')
TO
def index(request):
some_variable_name=TemplateResponse(request,'ur_app_name.html',{})
return some_variable_name
Dont't forgot to import this on top of views.py
from django.template.response import TemplateResponse
I don't know what exact problem I was facing, I guess it's an httpResponse thing because server needs http request and our local machine does not need it.
Some key points points for this error:
Make sure you mention your domain name in ALLOWED_HOST in settings.py file
set STATIC_ROOT=os.path.join(BASE_DIR, 'staticfiles') in settings.py, this maybe needed when heroku throws a
collecticstatic error but I am not sure. If it still throws the
same error then disable collectstatic=1 on command (you can find
the exact command for this in the debug log) which does not create
any problem.
set whitenoise.middleware.WhiteNoiseMiddleware, in MIDDLEWARE in settings.py, this might help.
If your project uses react and django then choose python only on add build pack on heroku. Do not choose node.js.
This is the github repository where I upload my heroku project you can find the config files here which maybe helpful.
If your project uses react and django and above suggestions didn't help this may help:
check .gitignore files, install all requirement.txt files
and copy package.json file in root directory.
set "build": "cd <project folder name>&& react-scripts build", in scripts in package.json
My index.html and manifest.json is in the github repo mentioned above.
The 'staticfiles' directory was created in the root folder using '$ python manage.py collectstatic'. Try copying my index.html file if your project doesn't have one.
That's all.
The herokuapp link :https://sainterface.herokuapp.com/

Writing to Heroku logs in Django

I've created a project in Django and have deployed it to Heroku. Unfortunately, a number of things that were working locally, now don't work on Heroku. To troubleshoot I need to be able to write to the Heroku logs when my program runs so that I can troubleshoot better. So far I have not gotten it to work
My settings/staging.py file contains:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
},
'loggers': {
'django': {
'handlers': ['console'],
'level': os.getenv('DJANGO_LOG_LEVEL', 'DEBUG'),
},
},
}
I have an app called accounts, so my accounts/views.py file contains:
import logging
log = logging.getLogger(__name__)
def auth_profile(request):
log.debug('TESTING THE DEBUGGER')
When auth_profile is accessed I want to see the text 'TESTING THE DEBUGGER' show up in the Heroku logs, but so far I get nothing.
How do I get this to work?

I think if you drop a log.error you would probably see something.
I had the same problem and it turns out heroku's settings tool breaks the pre-existing LOGGING setup. The logger you are using is not registered with django but is making its own way to the console using python's standard logging system.
Try doing:
django_heroku.settings(locals(), logging=False)
Or better yet, don't use it at all anymore since this package is no longer maintained anyway.

Django how to see generated SQL query?

I have a form which takes data and is supposed to insert it into a database. When I am processing that form it gives me a value error, but when I go to the database and try to insert it manually it works fine.
In order to debug this situation, I want to see what query Django is generating which is failing. On the debug webpage I didn't see anything like SQL query.
How can I see the actual query generated by Django?
Please advise.
Thanks.

How about using logging?
you can add this in settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
},
},
'loggers': {
'django': {
'handlers': ['console'],
'level': 'DEBUG',
'propagate': True,
},
},
}
and you can add this in your any views.py
import logging
l = logging.getLogger('django.db.backends')
l.setLevel(logging.DEBUG)
l.addHandler(logging.StreamHandler())
In your console, you can check SQL query.
Another way
go shell
python manage.py shell
>>from yourmodel import Example
>>queryset = Example.objects.all()
>>print(queryset.query)
you can see raw query string.

If you happen to use PyCharm, running your application in the debugger gives you the full context. Set a breakpoint, and browse in your app to the point you are having the error and get a screen like (trivial example):
Running in this way has changed the way I troubleshoot when using Django. I suspect other IDE's may have similar features. Some further video documentation of the process from the vendor at:
https://www.youtube.com/watch?v=QJtWxm12Eo0
As Jayground suggested, logging is probably something you'll turn on eventually anyway; great suggestion.

According to django docs
connection.queries includes all SQL statements – INSERTs, UPDATES, SELECTs, etc. Each time your app hits the database, the query will be recorded.
So you can access these queries by running
from django.db import connection
print(connection.queries)

How to stop logging in Django unittests from printing to stderr?

I'm testing some Django models with bog-standerd django.test.Testcase. My models.py writes to a debug log, using the following init code:
import logging
logger = logging.getLogger(__name__) # name is myapp.models
and then I write to the log with:
logger.debug("Here is my message")
In my settings.py, I've set up a single FileHandler, and a logger for myapp, using that handler and only that handler. This is great. I see messages to that log. When I'm in the Django shell, I only see messages to that log.
When, however, I run my test suite, my test suite console also sees all those messages. It's using a different formatter that I haven't explicitly defined, and it's writing to stderr. I don't have a log handler defined that writes to stderr.
I don't really want those messages spamming my console. I'll tail my log file if I want to see those messages. Is there a way to make it stop? (Yes, I could redirect stderr, but useful output goes to stderr as well.)
Edit: I've set up two handlers in my settings.py:
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'django.utils.log.NullHandler',
},
'logfile' : {
'level':'DEBUG',
'class':'logging.FileHandler',
'filename':'%s/log/development.log' % PROJECT_DIR,
'formatter': 'simple'
},
},
and tried this:
'loggers': {
'django': {
'level': 'DEBUG',
'handlers': ['null']
},
'myapp': {
'handlers': ['logfile'],
'level':'DEBUG',
},
... but the logging / stderr dumping behavior remains the same. It's like I'm getting another log handler when I'm running tests.

It's not clear from your config snippet which handlers, if any, are configured for the root logger. (I'm also assuming you're using Django 1.3.) Can you investigate and tell us what handlers have been added to the root logger when you're running tests? AFAICT Django doesn't add anything - perhaps some code you're importing does a call to basicConfig without you realising it. Use something like ack-grep to look for any occurrences of fileConfig, dictConfig, basicConfig, and addHandler - all of which could be adding a handler to the root logger.
Another thing to try: set the propagate flag to False for all top-level loggers (like "django", but also those used by your modules - say "myapp"). Does that change things?