Looking for a way to add header, body and a user's email address in my error log along with the stack trace of the exception in my views.py
After scouring the web for hours, many suggested to write my own middleware and some suggested to log that sort of information into a separate log. However, knowing where your code went wrong solves one part of the problem, identifying which poor soul it affected and what request data was sent during that exception goes a long a way in rectifying the issue. Having that information in the same log file just makes sense to me.
Currently in my views.py, I have this simple setup:
from django.db.models import Min, Max, Q, F, Count, Sum
from django.db import connection
from django.conf import settings
from django.http import HttpResponse, HttpResponseRedirect
from myapp.models import *
import logging
logging.basicConfig(filename="errors.log",
level=logging.ERROR,
format='%(asctime)s: %(message)s')
def random_view(request):
if request.user.is_authenticated() and request.user.is_active:
# generic view code goes here.
else:
return HttpResponse(status=401)
This setup worked well for a while. Every time there was an exception, it would log out the time, the exception error message and the stack trace.
How can I also add in request.META, request.user.id and request.body along with stack trace?
Any suggestions would help. A worked out answer, even better!
Thank you
I think a complete solution to the logging problem you have is to implement a middleware. The middleware would be able to work with any kind of view implementation you have, irrespective of whether it is a class based view, function based view or APIView from DRF.
You can define a middleware for full logging. Make sure that you place the middleware appropriately after the authentication middleware -
MIDDLEWARE = [
...,
'django.contrib.auth.middleware.AuthenticationMiddleware',
...,
'path.to.your.middleware.LogMiddleware'
]
In the log middleware, you would have access to the request and response. You can store request, the user (if authenticated) and all the META properties coming from the request via a logger, or you can even store it in a database if you want. Although, beware that storing in database comes at a cost. You can learn how to write a middleware by going through the Django middleware documentation.
import traceback
class LogMiddleware():
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
try:
return self.get_response(request)
except:
if request.user.is_authenticated():
# Log the user
path = request.get_full_path() # Get the URL Path
tb = traceback.format_exc() # Get the traceback
meta = request.META # Get request meta information
# Log everything
raise # Raise exception again after catching
You can read about all the meta attributes present from the django documentation of HttpRequest. Let me know if you need any clarification on this.
I would use a decorator here. Cut straight to the code...
import logging
from functools import wraps
from django.http import HttpResponse, HttpRequest
logging.basicConfig(filename="errors.log",
level=logging.ERROR,
format='%(asctime)s: %(message)s')
log = logging.getLogger(__name__)
def log_exceptions(wrapped):
#wraps(wrapped)
def wrapper(*args, **kwargs):
try:
return wrapped(*args, **kwargs)
except:
# log and re-raise
request = args[0] if len(args) > 0 and isinstance(args[0], HttpRequest) else None
msg = ("\nuser.id/email: {}/{}\nMETA: {}...\nbody: {}"
.format(request.user.id,
getattr(request.user, 'email','?'),
str(request.META)[:80],
request.body)
if request
else "not a HttpRequest")
log.exception(msg)
raise
return wrapper
#log_exceptions
def random_view(request):
raise ValueError("simulate a crash")
if request.user.is_authenticated() and request.user.is_active:
return HttpResponse('hi')
# generic view code goes here.
else:
return HttpResponse(status=401)
and errors.log should capture something like
2017-06-27 20:48:09,282:
user.id/email: 1/test#domain.com
META: {'SESSION_MANAGER': 'local/acb:#/tmp/.ICE-unix/4255,unix/acb:/tmp/.ICE-unix/4255...
body: b''
Traceback (most recent call last):
File "/home/rod/pyves/rangetest/rangetest/data/views.py", line 14, in wrapper
return wrapped(*args, **kwargs)
File "/home/rod/pyves/rangetest/rangetest/data/views.py", line 31, in random_view
raise ValueError("simulate a crash")
ValueError: simulate a crash
Note, you'll also probably see the Django crash logging in your errors.log as well. You might split the logs to separate files using Django's well documented, but nonetheless complex logging config
Related
I'm using Django in debug mode in dev environment.
When some views fail, it returns the debug page (not debug_toolbar, just the page with list of installed apps, environment variables, stack trace, ...)
In my middleware, I have some cases (specific URLs, specific users, ...) where I want to remove this data and just return the raw response.
How should I do that?
currently my best idea is to just:
response.data = {}
return response
But I'm not sure if it's the proper way to do that, whether it covers all cases and so on. I just want to use a middleware to control in some cases and avoid the DEBUG mode for them.
You may use exception middleware. See Django docs for more info.
Here is a sample implementation of the exception middleware:
class ExceptionHandlerMiddleware:
def __init__(self, get_response):
# if DEBUG is False we do not need this middleware
if not settings.DEBUG:
raise MiddlewareNotUsed
self.get_response = get_response
def __call__(self, request):
return self.get_response(request)
def process_exception(self, request, exception):
if request.user.is_authenticated: #any custom logic based on request and/or exception
#returning None kicks in default exception handling
#i.e. it will show full debug info page if settings.DEBUG is True
return None
else:
#returning HttpResponse will force applying template response and response
#middleware and the resulting response will be returned to the browser
return HttpResponse('Something went wrong')
Since Django 3.1 you may also use a custom error reporter class by defining the DEFAULT_EXCEPTION_REPORTER setting. The custom error reporter class needs to inherit from django.views.debug.ExceptionReporter and you may override get_traceback_data() to implement custom logic. See Django docs for more info.
During development, I am running Django in Debug mode and I am posting data to my application using a text mode application. Ideally, I need to receive a plain text response when I get an http error code 500 so I don't have to look for the real error inside all that HTML and Javascript.
Is it possible to obtain a Django 500 Internal Server Error as plain text?
If you are looking for a way to get a plain text error page when using curl, you
need to add the HTTP header X-Requested-With with value XMLHttpRequest, e.g.
curl -H 'X-Requested-With: XMLHttpRequest' http://example.com/some/url/
Explanation: this is because Django uses the is_ajax method to determine whether or not to return as plain text or as HTML. is_ajax in turn looks at X-Requested-With.
Update
Since django version 3.1, error reporting ignores the X-Requested-With header. Instead, set the Accept header on the request to any valid value which does not include the text/html mime type.
e.g.
curl -H 'Accept: application/json;charset=utf-8' http://example.comp/some/url
I think to write a middleware, because otherwise the exception isn't available in the 500.html
http://docs.djangoproject.com/en/dev/topics/http/middleware/#process-exception
class ProcessExceptionMiddleware(object):
def process_exception(self, request, exception):
t = Template("500 Error: {{ exception }}")
response_html = t.render(Context({'exception' : exception }))
response = http.HttpResponse(response_html)
response.status_code = 500
return response
There's a setting DEBUG_PROPAGATE_EXCEPTIONS which will force Django not to wrap the exceptions, so you can see them, e.g. in devserver logs.
This is an improvement on Yuji's answer, which provides a stacktrace, more instructions (for us django newbies) and is simpler.
Put this code in a file somewhere in your application, e.g. PROJECT_ROOT/MAIN_APP/middleware/exceptions.py, and make sure you have an empty __init__.py in the same directory.
import traceback
from django.http import HttpResponse
class PlainExceptionsMiddleware(object):
def process_exception(self, request, exception):
return HttpResponse(traceback.format_exc(exception), content_type="text/plain", status=500)
Now edit your settings.py and find MIDDLEWARE_CLASSES = (. Add another entry so it is like this:
MIDDLEWARE_CLASSES = (
# (all the previous entries)
# Plain text exception pages.
'MAIN_APP.middleware.exceptions.PlainExceptionsMiddleware',
)
Restart django and you are good to go!
User-agent aware formatting.
If you're like me and developing an app and a website both backed by django, you probably want to show plain text error pages to the app, and the nice formatted ones to the browser. A simple way to to that is to check the user agent:
import traceback
from django.http import HttpResponse
class PlainExceptionsMiddleware(object):
def process_exception(self, request, exception):
if "HTTP_USER_AGENT" in request.META and "chrome" in request.META["HTTP_USER_AGENT"].lower():
return
return HttpResponse(traceback.format_exc(exception), content_type="text/plain", status=500)
Building off of Timmmm's answer, I had to make several modifications for it to work in Django 3.1:
Create a file somewhere in your application, such as YOUR_APP_NAME/middleware/exceptions.py and paste the following code:
import traceback
from django.http import HttpResponse, HttpRequest
class PlainExceptionsMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
return self.get_response(request)
def process_exception(self, request: HttpRequest, exception: Exception):
if "HTTP_USER_AGENT" in request.META and "chrome" in request.META["HTTP_USER_AGENT"].lower():
return
print(traceback.format_exc())
return HttpResponse(repr(exception), content_type="text/plain", status=500)
It is not necessary to create an __init__.py file in the middleware folder.
In settings.py, add the following item to the end of the MIDDLEWARE variable, so that it looks like:
MIDDLEWARE = [
# ...
'YOUR_APP_NAME.middleware.exceptions.PlainExceptionsMiddleware'
]
Now, if "HTTP_USER_AGENT" and "chrome" are in the request header, this middleware doesn't do anything, so Django returns an HTML response as usual. Otherwise, it returns a plain-text representation of the error as a response (e.g., ValueError("Field 'id' expected a number but got 'undefined'.")) and prints out the traceback to the Django console, as Django normally would. Of course, you can instead return the full traceback as your response.
I'm trying to make middleware which alters some fields for the user based on subdomain, etc...
The only problem is the request.user always comes in as AnonymousUser within the middleware, but is then the correct user within the views. I've left the default authentication and session middleware django uses within the settings.
There is a similar question here: Django, request.user is always Anonymous User
But doesn't overly answer the total question because I'm not using different authentication methods, and djangos authentication is running before I invoke my own middleware.
Is there a way, while using DRF, to get the request.user within the middleware? I'll show some sample code here:
class SampleMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
#This will be AnonymousUser. I need it to be the actual user making the request.
print (request.user)
def process_response(self, request, response):
return response
with process_request:
class SampleMiddleware(object):
def process_request(self, request):
#This will be AnonymousUser. I need it to be the actual user making the request.
print (request.user)
def process_response(self, request, response):
return response
I've solved this problem by getting DRF token from the requests and loading request.user to the user associated to that model.
I had the default django authentication and session middleware, but it seems DRF was using it's token auth after middleware to resolve the user (All requests were CORS requests, this might have been why). Here's my updated middleware class:
from re import sub
from rest_framework.authtoken.models import Token
from core.models import OrganizationRole, Organization, User
class OrganizationMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
header_token = request.META.get('HTTP_AUTHORIZATION', None)
if header_token is not None:
try:
token = sub('Token ', '', header_token)
token_obj = Token.objects.get(key = token)
request.user = token_obj.user
except Token.DoesNotExist:
pass
#This is now the correct user
print (request.user)
This can be used on process_view or process_request as well.
Hopefully this can help someone out in the future.
Came across this today while having the same problem.
TL;DR;
Skip below for code example
Explanation
Thing is DRF have their own flow of things, right in the middle of the django request life-cycle.
So if the normal middleware flow is :
request_middleware (before starting to work on the request)
view_middleware (before calling the view)
template_middleware (before render)
response_middleware (before final response)
DRF code, overrides the default django view code, and executes their own code.
In the above link, you can see that they wrap the original request with their own methods, where one of those methods is DRF authentication.
So back to your question, this is the reason using request.user in a middleware is premature, as it only gets it's value after view_middleware** executes.
The solution I went with, is having my middleware set a LazyObject.
This helps, because my code (the actual DRF ApiVIew) executes when the actual user is already set by DRF's authentication.
This solution was proposed here together with a discussion.
Might have been better if DRF had a better way to extend their functionality, but as things are, this seems better than the provided solution (both performance and readability wise).
Code Example
from django.utils.functional import SimpleLazyObject
def get_actual_value(request):
if request.user is None:
return None
return request.user #here should have value, so any code using request.user will work
class MyCustomMiddleware(object):
def process_request(self, request):
request.custom_prop = SimpleLazyObject(lambda: get_actual_value(request))
The accepted answer only takes TokenAuthentication in consideration - in my case, there are more authentication methods configured. I thus went with initializing the DRF's Request directly which invokes DRF's authentication machinery and loops through all configured authentication methods.
Unfortunately, it still introduces an additional load on the database since the Token object must be queried (the accepted answer has this problem as well). The trick with SimpleLazyObject in this answer is a much better solution, but it didn't work for my use case because I need the user info in the middleware directly - I'm extending the metrics in django_prometheus and it processes the request before get_response is called.
from rest_framework.request import Request as RestFrameworkRequest
from rest_framework.views import APIView
class MyMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
drf_request: RestFrameworkRequest = APIView().initialize_request(request)
user = drf_request.user
...
return self.get_response(request)
Based on Daniel Dubovski's very elegant solution above, here's an example of middleware for Django 1.11:
from django.utils.functional import SimpleLazyObject
from organization.models import OrganizationMember
from django.core.exceptions import ObjectDoesNotExist
def get_active_member(request):
try:
active_member = OrganizationMember.objects.get(user=request.user)
except (ObjectDoesNotExist, TypeError):
active_member = None
return active_member
class OrganizationMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
request.active_member = SimpleLazyObject(lambda: get_active_member(request))
response = self.get_response(request)
# Code to be executed for each request/response after
# the view is called.
return response
Daniel Dubovski's solution is probably the best in most cases.
The problem with the lazy object approach is if you need to rely on the side effects. In my case, I need something to happen for each request, no matter what.
If I'd use a special value like request.custom_prop, it has to be evaluated for each request for the side effects to happen. I noticed that other people are setting request.user, but it doesn't work for me since some middleware or authentication class overwrites this property.
What if DRF supported its own middleware? Where could I plug it in? The easiest way in my case (I don't need to access the request object, only the authenticated user) seems to be to hook into the authentication class itself:
from rest_framework.authentication import TokenAuthentication
class TokenAuthenticationWithSideffects(TokenAuthentication):
def authenticate(self, request):
user_auth_tuple = super().authenticate(request)
if user_auth_tuple is None:
return
(user, token) = user_auth_tuple
# Do stuff with the user here!
return (user, token)
Then I could just replace this line in my settings:
REST_FRAMEWORK = {
"DEFAULT_AUTHENTICATION_CLASSES": (
#"rest_framework.authentication.TokenAuthentication",
"my_project.authentication.TokenAuthenticationWithSideffects",
),
# ...
}
I'm not promoting this solution, but maybe it will help someone else.
Pros:
It to solves this specific problem
There's no double authentication
Easy to maintain
Cons:
Not tested in production
Things happen in an unexpected place
Side effects...
I know it's not exactly answering the 'can we access that from the middleware' question, but I think it's a more elegant solution VS doing the same work in the middleware VS what DRJ does in its base view class. At least for what I needed, it made more sense to add here.
Basically, I'm just overriding the method 'perform_authentication()' from DRF's code, since I needed to add more things related to the current user in the request. The method just originally call 'request.user'.
class MyGenericViewset(viewsets.GenericViewSet):
def perform_authentication(self, request):
request.user
if request.user and request.user.is_authenticated():
request.my_param1 = 'whatever'
After that in your own views, instead of settings APIView from DRF as a parent class, simply set that class as a parent.
I wasn't quite happy with the solutions out there. Here's a solution that uses some DRF internals to make sure that the correct authentication is applied in the middleware, even if the view has specific permissions classes. It uses the middleware hook process_view which gives us access to the view we're about to hit:
class CustomTenantMiddleware():
def process_view(self, request, view_func, view_args, view_kwargs):
# DRF saves the class of the view function as the .cls property
view_class = view_func.cls
try:
# We need to instantiate the class
view = view_class()
# And give it an action_map. It's not relevant for us, but otherwise it errors.
view.action_map = {}
# Here's our fully formed and authenticated (or not, depending on credentials) request
request = view.initialize_request(request)
except (AttributeError, TypeError):
# Can't initialize the request from this view. Fallback to using default permission classes
request = APIView().initialize_request(request)
# Here the request is fully formed, with the correct permissions depending on the view.
Note that this doesn't avoid having to authenticate twice. DRF will still happily authenticate right afterwards.
I had the same issue and decided to change my design. Instead of using a Middleware I simply monkey-patch rest_framework.views.APIView.
In my case I needed to patch check_permissions but you can patch whatever fits your problem. Have a look at the the source code.
settings.py
INSTALLED_APPS = [
..
'myapp',
]
myapp/patching.py
import sys
from rest_framework.views import APIView as OriginalAPIView
class PatchedAPIView(OriginalAPIView):
def check_permissions(self, request):
print(f"We should do something with user {request.user}"
return OriginalAPIView.check_permissions(self, request)
# We replace the Django REST view with our patched one
sys.modules['rest_framework'].views.APIView = PatchedAPIView
myapp/__init__.py
from .patching import *
I have like this:
try:
#bunch of code
except:
return HttpResponse....
I want to send an error to client and to print stack trace on console. How can i do that?
You can do something like this
import traceback
from django.http import HttpReponse
def view(request):
try:
#throw exception
except:
tb = traceback.format_exc()
return HttpResponse(tb)
# normal flow
You can create a custom middleware class where you can catch all exceptions:
class ErrorMiddleware(object):
def process_exception(self, request, exception):
# send error
return HttpResponse(...) # or None
and put it on first place to MIDDLEWARE_CLASSES tuple in settings.py.
From middleware process_exception docs:
Django calls process_exception() when a view raises an exception.
process_exception() should return either None or an HttpResponse
object. If it returns an HttpResponse object, the template response
and response middleware will be applied, and the resulting response
returned to the browser. Otherwise, default exception handling kicks
in.
You might want to enable logging of all exceptions How do you log server errors on django sites
and make a custom 500 django error page https://docs.djangoproject.com/en/dev/topics/http/views/#the-500-server-error-view
and you should also make a 500 error page at web server level (apache / nginx) just in case the framework doesn't work.
That way you can "catch" all errors and show a nice error message to the client.
I am trying to use Google App Engine queues API, I am having problems on testing this. It seems that in some part of the process the CSRF it's not working.
as I understand the api executes the task calling the url and making and http request in background.
The complete url is the API is calling is → http://localhost.localdomain:8000/admin/cooking/recipe/36/chefworker/
When it raises this exception:
Traceback (most recent call last):
File "/home/mariocesar/Proyectos/Cooking/cooking/django/core/handlers/base.py", line 100, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/mariocesar/Proyectos/Cooking/cooking/django/views/decorators/csrf.py", line 24, in wrapped_view
resp.csrf_exempt = True
AttributeError: 'NoneType' object has no attribute 'csrf_exempt'
So, the csrf middleware, the cookie, some data or the response itself is missing from the request that the GAE api makes to execute the task in the background.
How to solve this without disabling CSRF on Django? however, it's posible with djangoappengine at all?
Down are the models.py and admin.py files I am using.
models.py
from django.db import models
class Recipe(models.Model):
name = models.CharField(max_length=140)
description = models.TextField()
cooking_time = models.PositiveIntegerField()
status = models.CharField(max_length=40)
def __unicode__(self):
return self.name
def cookthis(self):
import time
self.status = 'The chef is cooking this recipe'
self.save()
time.sleep(obj.cooking_time)
self.status = 'It\'s done ! the recipe is ready to serve'
self.save()
admin.py
import logging
from django.contrib import admin, messages
from django.http import HttpResponse
from django.utils.functional import update_wrapper
from django.contrib.admin.util import unquote
from django.shortcuts import get_object_or_404, render_to_response
from django import template
from django.core.urlresolvers import reverse
from google.appengine.api import taskqueue
from google.appengine.api.taskqueue import TaskAlreadyExistsError
from cooking.models import Recipe
from django.views.decorators.csrf import csrf_exempt
class AdminRecipe(admin.ModelAdmin):
def get_urls(self):
from django.conf.urls.defaults import patterns, url
def wrap(view):
def wrapper(*args, **kwargs):
return self.admin_site.admin_view(view)(*args, **kwargs)
return update_wrapper(wrapper, view)
info = self.model._meta.app_label, self.model._meta.module_name
urlpatterns = super(AdminRecipe, self).get_urls()
myurls = patterns('',
url(r'^(.+)/cook/$',
wrap(self.cook_view),
name='%s_%s_chefworker' % info),
url(r'^(.+)/chefworker/$',
wrap(self.chefworker_worker),
name='%s_%s_chefworker' % info),
)
return myurls + urlpatterns
def cook_view(self, request, object_id, extra_context=None):
obj = get_object_or_404(Recipe, pk=unquote(object_id))
if request.POST:
try:
taskqueue.add(
name="recipie-%s" % obj.id,
url=reverse('admin:cooking_recipe_chefworker', args=(obj.id,))
)
messages.add_message(request, messages.INFO, 'Chef is cooking the recipe.')
except TaskAlreadyExistsError:
messages.add_message(request, messages.ERROR, 'chef is already cooking that recipe.')
context_instance = template.RequestContext(request, current_app=self.admin_site.name)
return render_to_response("admin/cooking/recipe/cook_view.html", {'object': obj}, context_instance=context_instance)
#TODO: Add csrf token on form
#csrf_exempt
def chefworker_worker(self, request, object_id, extra_context=None):
import time
if request.POST:
obj = get_object_or_404(Recipe, pk=unquote(object_id))
obj.cookthis()
return HttpResponse('done')
admin.site.register(Recipe, AdminRecipe)
IMPORTANT NOTE:
Was hard to debug this error, cause the dev_appserver logger was just raising 403 errors, no other info; so, I have to patch the file google/appengine/api/taskqueue/taskqueue_stub.py line 574 and add "logging.info('response --- \n%s' % result)" to get the output.
If you have the CsrfViewMiddleware enabled, Django will require a csrf_token in all POSTs to your views.
Django provides a decorator, #csrf_exempt, that you should place on your task queue views. This turns off the middleware just for those views.
Alternatively, you can avoid using CsrfViewMiddleware altogether and instead use the #csrf_protect decorator where you need it. I don't recommend doing this -- it's probably safer to protect everywhere and carve out a small number of exemptions for your task queue views.
(One last note: both answers above -- that something is wrong with your view, or that you should just use GET for the task queue -- strike me wrong. There's nothing wrong with your view, and POST is the right verb to use for task queue tasks.)
Looking at the source of csrf.py, it looks like this would only occur if your view function is returning None (or not explicitly returning, in which case Python would return None implicitly). Looking at your code, I don't see how that could occur, though - are you sure this is your exact deployed code?
Also, you probably don't want to use get_object_or_404 inside a task queue task - if it can't find the object, it'll throw a 404, which will cause the task to error and retry indefinitely.
You also shouldn't need CSRF protection (per your TODO); instead, make sure the task queue URL is marked admin-only, and it will only ever be called by the task queue service.
I'm not an expert, but you may try using GET instead of POST. See http://groups.google.com/group/django-non-relational/browse_thread/thread/e6baed5291aed957/d6c42150c8e246e1?lnk=gst&q=queue#d6c42150c8e246e1 (the last entry)