I'm using Django in debug mode in dev environment.
When some views fail, it returns the debug page (not debug_toolbar, just the page with list of installed apps, environment variables, stack trace, ...)
In my middleware, I have some cases (specific URLs, specific users, ...) where I want to remove this data and just return the raw response.
How should I do that?
currently my best idea is to just:
response.data = {}
return response
But I'm not sure if it's the proper way to do that, whether it covers all cases and so on. I just want to use a middleware to control in some cases and avoid the DEBUG mode for them.
You may use exception middleware. See Django docs for more info.
Here is a sample implementation of the exception middleware:
class ExceptionHandlerMiddleware:
def __init__(self, get_response):
# if DEBUG is False we do not need this middleware
if not settings.DEBUG:
raise MiddlewareNotUsed
self.get_response = get_response
def __call__(self, request):
return self.get_response(request)
def process_exception(self, request, exception):
if request.user.is_authenticated: #any custom logic based on request and/or exception
#returning None kicks in default exception handling
#i.e. it will show full debug info page if settings.DEBUG is True
return None
else:
#returning HttpResponse will force applying template response and response
#middleware and the resulting response will be returned to the browser
return HttpResponse('Something went wrong')
Since Django 3.1 you may also use a custom error reporter class by defining the DEFAULT_EXCEPTION_REPORTER setting. The custom error reporter class needs to inherit from django.views.debug.ExceptionReporter and you may override get_traceback_data() to implement custom logic. See Django docs for more info.
Related
Looking for a way to add header, body and a user's email address in my error log along with the stack trace of the exception in my views.py
After scouring the web for hours, many suggested to write my own middleware and some suggested to log that sort of information into a separate log. However, knowing where your code went wrong solves one part of the problem, identifying which poor soul it affected and what request data was sent during that exception goes a long a way in rectifying the issue. Having that information in the same log file just makes sense to me.
Currently in my views.py, I have this simple setup:
from django.db.models import Min, Max, Q, F, Count, Sum
from django.db import connection
from django.conf import settings
from django.http import HttpResponse, HttpResponseRedirect
from myapp.models import *
import logging
logging.basicConfig(filename="errors.log",
level=logging.ERROR,
format='%(asctime)s: %(message)s')
def random_view(request):
if request.user.is_authenticated() and request.user.is_active:
# generic view code goes here.
else:
return HttpResponse(status=401)
This setup worked well for a while. Every time there was an exception, it would log out the time, the exception error message and the stack trace.
How can I also add in request.META, request.user.id and request.body along with stack trace?
Any suggestions would help. A worked out answer, even better!
Thank you
I think a complete solution to the logging problem you have is to implement a middleware. The middleware would be able to work with any kind of view implementation you have, irrespective of whether it is a class based view, function based view or APIView from DRF.
You can define a middleware for full logging. Make sure that you place the middleware appropriately after the authentication middleware -
MIDDLEWARE = [
...,
'django.contrib.auth.middleware.AuthenticationMiddleware',
...,
'path.to.your.middleware.LogMiddleware'
]
In the log middleware, you would have access to the request and response. You can store request, the user (if authenticated) and all the META properties coming from the request via a logger, or you can even store it in a database if you want. Although, beware that storing in database comes at a cost. You can learn how to write a middleware by going through the Django middleware documentation.
import traceback
class LogMiddleware():
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
try:
return self.get_response(request)
except:
if request.user.is_authenticated():
# Log the user
path = request.get_full_path() # Get the URL Path
tb = traceback.format_exc() # Get the traceback
meta = request.META # Get request meta information
# Log everything
raise # Raise exception again after catching
You can read about all the meta attributes present from the django documentation of HttpRequest. Let me know if you need any clarification on this.
I would use a decorator here. Cut straight to the code...
import logging
from functools import wraps
from django.http import HttpResponse, HttpRequest
logging.basicConfig(filename="errors.log",
level=logging.ERROR,
format='%(asctime)s: %(message)s')
log = logging.getLogger(__name__)
def log_exceptions(wrapped):
#wraps(wrapped)
def wrapper(*args, **kwargs):
try:
return wrapped(*args, **kwargs)
except:
# log and re-raise
request = args[0] if len(args) > 0 and isinstance(args[0], HttpRequest) else None
msg = ("\nuser.id/email: {}/{}\nMETA: {}...\nbody: {}"
.format(request.user.id,
getattr(request.user, 'email','?'),
str(request.META)[:80],
request.body)
if request
else "not a HttpRequest")
log.exception(msg)
raise
return wrapper
#log_exceptions
def random_view(request):
raise ValueError("simulate a crash")
if request.user.is_authenticated() and request.user.is_active:
return HttpResponse('hi')
# generic view code goes here.
else:
return HttpResponse(status=401)
and errors.log should capture something like
2017-06-27 20:48:09,282:
user.id/email: 1/test#domain.com
META: {'SESSION_MANAGER': 'local/acb:#/tmp/.ICE-unix/4255,unix/acb:/tmp/.ICE-unix/4255...
body: b''
Traceback (most recent call last):
File "/home/rod/pyves/rangetest/rangetest/data/views.py", line 14, in wrapper
return wrapped(*args, **kwargs)
File "/home/rod/pyves/rangetest/rangetest/data/views.py", line 31, in random_view
raise ValueError("simulate a crash")
ValueError: simulate a crash
Note, you'll also probably see the Django crash logging in your errors.log as well. You might split the logs to separate files using Django's well documented, but nonetheless complex logging config
I would like to use Custom Django Middleware (Django 1.9) to check whether or not an anonymous user has accepted a site's terms & conditions - I'll show the user a dialog if they've not yet clicked 'Agree'.
I have no need to store this in the database and would prefer to simply use a Cookie. I have the following in settings.py:
MIDDLEWARE_CLASSES = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
...
'myapp.middleware.app_custom_middleware.TermsMiddleware',]
SESSION_ENGINE = 'django.contrib.sessions.backends.signed_cookies'
I'm then trying something very rudimentary in TermsMiddleware - here I'm deliberately trying to modify the request and the response just to try to get it to work:
class TermsMiddleware(object):
def process_request(self, request):
request.session['myCookieKey'] = 'myCookieValue'
request.session.save()
return
def process_response(self, request, response):
request.session['myCookieKey'] = 'myCookieValue'
request.session.save()
return response
If I inspect the response in the browser, I see that myCookieKey isn't being set, so I think I've misunderstood how this should be written.
Can I manipulate the session within middleware like this, having it take effect in the cookie sent in the response Django was going to send anyway? The documentation makes it sound like I should be able to:
It should return either None or an HttpResponse object. If it returns
None, Django will continue processing this request, executing any
other process_request() middleware, then, process_view() middleware,
and finally, the appropriate view. If it returns an HttpResponse
object, Django won’t bother calling any other request, view or
exception middleware, or the appropriate view; it’ll apply response
middleware to that HttpResponse, and return the result.
I've also seen the note in the documentation to use SessionStore() when outside of a view function. In theory I don't think I should need to do that here because I've got access to request (and response). I tried it anyway, and I still couldn't get it to work.
Help much appreciated!
The thing you've misunderstood is the relationship between the session and the cookies.
You can't set arbitrary cookies by modifying the session. The session is a collection of data, usually stored in the db or a file, and the only cookie is the one that contains the session key for the current user.
You certainly can modify the session in middleware, but wherever you do so, you shouldn't expect to see a cookie being set.
(Separately, you should never call double underscore methods like __setitem__ directly. The session is accessed via a dict-like interface: request.session['foo'] = 'bar'.)
During development, I am running Django in Debug mode and I am posting data to my application using a text mode application. Ideally, I need to receive a plain text response when I get an http error code 500 so I don't have to look for the real error inside all that HTML and Javascript.
Is it possible to obtain a Django 500 Internal Server Error as plain text?
If you are looking for a way to get a plain text error page when using curl, you
need to add the HTTP header X-Requested-With with value XMLHttpRequest, e.g.
curl -H 'X-Requested-With: XMLHttpRequest' http://example.com/some/url/
Explanation: this is because Django uses the is_ajax method to determine whether or not to return as plain text or as HTML. is_ajax in turn looks at X-Requested-With.
Update
Since django version 3.1, error reporting ignores the X-Requested-With header. Instead, set the Accept header on the request to any valid value which does not include the text/html mime type.
e.g.
curl -H 'Accept: application/json;charset=utf-8' http://example.comp/some/url
I think to write a middleware, because otherwise the exception isn't available in the 500.html
http://docs.djangoproject.com/en/dev/topics/http/middleware/#process-exception
class ProcessExceptionMiddleware(object):
def process_exception(self, request, exception):
t = Template("500 Error: {{ exception }}")
response_html = t.render(Context({'exception' : exception }))
response = http.HttpResponse(response_html)
response.status_code = 500
return response
There's a setting DEBUG_PROPAGATE_EXCEPTIONS which will force Django not to wrap the exceptions, so you can see them, e.g. in devserver logs.
This is an improvement on Yuji's answer, which provides a stacktrace, more instructions (for us django newbies) and is simpler.
Put this code in a file somewhere in your application, e.g. PROJECT_ROOT/MAIN_APP/middleware/exceptions.py, and make sure you have an empty __init__.py in the same directory.
import traceback
from django.http import HttpResponse
class PlainExceptionsMiddleware(object):
def process_exception(self, request, exception):
return HttpResponse(traceback.format_exc(exception), content_type="text/plain", status=500)
Now edit your settings.py and find MIDDLEWARE_CLASSES = (. Add another entry so it is like this:
MIDDLEWARE_CLASSES = (
# (all the previous entries)
# Plain text exception pages.
'MAIN_APP.middleware.exceptions.PlainExceptionsMiddleware',
)
Restart django and you are good to go!
User-agent aware formatting.
If you're like me and developing an app and a website both backed by django, you probably want to show plain text error pages to the app, and the nice formatted ones to the browser. A simple way to to that is to check the user agent:
import traceback
from django.http import HttpResponse
class PlainExceptionsMiddleware(object):
def process_exception(self, request, exception):
if "HTTP_USER_AGENT" in request.META and "chrome" in request.META["HTTP_USER_AGENT"].lower():
return
return HttpResponse(traceback.format_exc(exception), content_type="text/plain", status=500)
Building off of Timmmm's answer, I had to make several modifications for it to work in Django 3.1:
Create a file somewhere in your application, such as YOUR_APP_NAME/middleware/exceptions.py and paste the following code:
import traceback
from django.http import HttpResponse, HttpRequest
class PlainExceptionsMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
return self.get_response(request)
def process_exception(self, request: HttpRequest, exception: Exception):
if "HTTP_USER_AGENT" in request.META and "chrome" in request.META["HTTP_USER_AGENT"].lower():
return
print(traceback.format_exc())
return HttpResponse(repr(exception), content_type="text/plain", status=500)
It is not necessary to create an __init__.py file in the middleware folder.
In settings.py, add the following item to the end of the MIDDLEWARE variable, so that it looks like:
MIDDLEWARE = [
# ...
'YOUR_APP_NAME.middleware.exceptions.PlainExceptionsMiddleware'
]
Now, if "HTTP_USER_AGENT" and "chrome" are in the request header, this middleware doesn't do anything, so Django returns an HTML response as usual. Otherwise, it returns a plain-text representation of the error as a response (e.g., ValueError("Field 'id' expected a number but got 'undefined'.")) and prints out the traceback to the Django console, as Django normally would. Of course, you can instead return the full traceback as your response.
I'm trying to make middleware which alters some fields for the user based on subdomain, etc...
The only problem is the request.user always comes in as AnonymousUser within the middleware, but is then the correct user within the views. I've left the default authentication and session middleware django uses within the settings.
There is a similar question here: Django, request.user is always Anonymous User
But doesn't overly answer the total question because I'm not using different authentication methods, and djangos authentication is running before I invoke my own middleware.
Is there a way, while using DRF, to get the request.user within the middleware? I'll show some sample code here:
class SampleMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
#This will be AnonymousUser. I need it to be the actual user making the request.
print (request.user)
def process_response(self, request, response):
return response
with process_request:
class SampleMiddleware(object):
def process_request(self, request):
#This will be AnonymousUser. I need it to be the actual user making the request.
print (request.user)
def process_response(self, request, response):
return response
I've solved this problem by getting DRF token from the requests and loading request.user to the user associated to that model.
I had the default django authentication and session middleware, but it seems DRF was using it's token auth after middleware to resolve the user (All requests were CORS requests, this might have been why). Here's my updated middleware class:
from re import sub
from rest_framework.authtoken.models import Token
from core.models import OrganizationRole, Organization, User
class OrganizationMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
header_token = request.META.get('HTTP_AUTHORIZATION', None)
if header_token is not None:
try:
token = sub('Token ', '', header_token)
token_obj = Token.objects.get(key = token)
request.user = token_obj.user
except Token.DoesNotExist:
pass
#This is now the correct user
print (request.user)
This can be used on process_view or process_request as well.
Hopefully this can help someone out in the future.
Came across this today while having the same problem.
TL;DR;
Skip below for code example
Explanation
Thing is DRF have their own flow of things, right in the middle of the django request life-cycle.
So if the normal middleware flow is :
request_middleware (before starting to work on the request)
view_middleware (before calling the view)
template_middleware (before render)
response_middleware (before final response)
DRF code, overrides the default django view code, and executes their own code.
In the above link, you can see that they wrap the original request with their own methods, where one of those methods is DRF authentication.
So back to your question, this is the reason using request.user in a middleware is premature, as it only gets it's value after view_middleware** executes.
The solution I went with, is having my middleware set a LazyObject.
This helps, because my code (the actual DRF ApiVIew) executes when the actual user is already set by DRF's authentication.
This solution was proposed here together with a discussion.
Might have been better if DRF had a better way to extend their functionality, but as things are, this seems better than the provided solution (both performance and readability wise).
Code Example
from django.utils.functional import SimpleLazyObject
def get_actual_value(request):
if request.user is None:
return None
return request.user #here should have value, so any code using request.user will work
class MyCustomMiddleware(object):
def process_request(self, request):
request.custom_prop = SimpleLazyObject(lambda: get_actual_value(request))
The accepted answer only takes TokenAuthentication in consideration - in my case, there are more authentication methods configured. I thus went with initializing the DRF's Request directly which invokes DRF's authentication machinery and loops through all configured authentication methods.
Unfortunately, it still introduces an additional load on the database since the Token object must be queried (the accepted answer has this problem as well). The trick with SimpleLazyObject in this answer is a much better solution, but it didn't work for my use case because I need the user info in the middleware directly - I'm extending the metrics in django_prometheus and it processes the request before get_response is called.
from rest_framework.request import Request as RestFrameworkRequest
from rest_framework.views import APIView
class MyMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
drf_request: RestFrameworkRequest = APIView().initialize_request(request)
user = drf_request.user
...
return self.get_response(request)
Based on Daniel Dubovski's very elegant solution above, here's an example of middleware for Django 1.11:
from django.utils.functional import SimpleLazyObject
from organization.models import OrganizationMember
from django.core.exceptions import ObjectDoesNotExist
def get_active_member(request):
try:
active_member = OrganizationMember.objects.get(user=request.user)
except (ObjectDoesNotExist, TypeError):
active_member = None
return active_member
class OrganizationMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
request.active_member = SimpleLazyObject(lambda: get_active_member(request))
response = self.get_response(request)
# Code to be executed for each request/response after
# the view is called.
return response
Daniel Dubovski's solution is probably the best in most cases.
The problem with the lazy object approach is if you need to rely on the side effects. In my case, I need something to happen for each request, no matter what.
If I'd use a special value like request.custom_prop, it has to be evaluated for each request for the side effects to happen. I noticed that other people are setting request.user, but it doesn't work for me since some middleware or authentication class overwrites this property.
What if DRF supported its own middleware? Where could I plug it in? The easiest way in my case (I don't need to access the request object, only the authenticated user) seems to be to hook into the authentication class itself:
from rest_framework.authentication import TokenAuthentication
class TokenAuthenticationWithSideffects(TokenAuthentication):
def authenticate(self, request):
user_auth_tuple = super().authenticate(request)
if user_auth_tuple is None:
return
(user, token) = user_auth_tuple
# Do stuff with the user here!
return (user, token)
Then I could just replace this line in my settings:
REST_FRAMEWORK = {
"DEFAULT_AUTHENTICATION_CLASSES": (
#"rest_framework.authentication.TokenAuthentication",
"my_project.authentication.TokenAuthenticationWithSideffects",
),
# ...
}
I'm not promoting this solution, but maybe it will help someone else.
Pros:
It to solves this specific problem
There's no double authentication
Easy to maintain
Cons:
Not tested in production
Things happen in an unexpected place
Side effects...
I know it's not exactly answering the 'can we access that from the middleware' question, but I think it's a more elegant solution VS doing the same work in the middleware VS what DRJ does in its base view class. At least for what I needed, it made more sense to add here.
Basically, I'm just overriding the method 'perform_authentication()' from DRF's code, since I needed to add more things related to the current user in the request. The method just originally call 'request.user'.
class MyGenericViewset(viewsets.GenericViewSet):
def perform_authentication(self, request):
request.user
if request.user and request.user.is_authenticated():
request.my_param1 = 'whatever'
After that in your own views, instead of settings APIView from DRF as a parent class, simply set that class as a parent.
I wasn't quite happy with the solutions out there. Here's a solution that uses some DRF internals to make sure that the correct authentication is applied in the middleware, even if the view has specific permissions classes. It uses the middleware hook process_view which gives us access to the view we're about to hit:
class CustomTenantMiddleware():
def process_view(self, request, view_func, view_args, view_kwargs):
# DRF saves the class of the view function as the .cls property
view_class = view_func.cls
try:
# We need to instantiate the class
view = view_class()
# And give it an action_map. It's not relevant for us, but otherwise it errors.
view.action_map = {}
# Here's our fully formed and authenticated (or not, depending on credentials) request
request = view.initialize_request(request)
except (AttributeError, TypeError):
# Can't initialize the request from this view. Fallback to using default permission classes
request = APIView().initialize_request(request)
# Here the request is fully formed, with the correct permissions depending on the view.
Note that this doesn't avoid having to authenticate twice. DRF will still happily authenticate right afterwards.
I had the same issue and decided to change my design. Instead of using a Middleware I simply monkey-patch rest_framework.views.APIView.
In my case I needed to patch check_permissions but you can patch whatever fits your problem. Have a look at the the source code.
settings.py
INSTALLED_APPS = [
..
'myapp',
]
myapp/patching.py
import sys
from rest_framework.views import APIView as OriginalAPIView
class PatchedAPIView(OriginalAPIView):
def check_permissions(self, request):
print(f"We should do something with user {request.user}"
return OriginalAPIView.check_permissions(self, request)
# We replace the Django REST view with our patched one
sys.modules['rest_framework'].views.APIView = PatchedAPIView
myapp/__init__.py
from .patching import *
I'm calling Django from within another application.
While debugging, if a Django exception occurs, the (html) response body is being picked up and then wrapped in an exception generated in the calling app.
This makes it a real pain to determine what the problem was as I need to wade through CSS, Html, Js, etc.
Is there any way to get Django to only output an exception message and a stack trace as plain text?
You can write your own exception-handling middleware class. See the documentation for the process_exception middleware method:
process_exception(self, request, exception)
request is an HttpRequest object. exception is an Exception object raised by the view function.
Django calls process_exception() when a view raises an exception. process_exception() should return either None or an HttpResponse object. If it returns an HttpResponse object, the response will be returned to the browser. Otherwise, default exception handling kicks in.
So I think you need to write something like this:
import traceback
from django.http import HttpResponse
class PlainTextExceptionMiddleware(object):
def process_exception(self, request, exception):
return HttpResponse(traceback.format_exc(), "text/plain")
and then add the class to your MIDDLEWARE_CLASSES setting.
Just set the DEBUG_PROPAGATE_EXCEPTIONS in your settings:
If set to True, Django’s normal exception handling of view functions
will be suppressed, and exceptions will propagate upwards. This
can be useful for some test setups, and should never be used on a live site.
See the Django docs