We Keep Coding

Assertion error parsing response to POST request from DRF viewset with DRF APIClient in tests

I've run into a strange issue with Django Rest Framework testing engine. The weird thing is that everything used to work fine with Django 3 and this issue turned up after I migrated to Django 4. Apart from testing, everything works well, and responds to queries as expected.
The problem
I'm using DRF APIClient to make queries for unit tests. While GET requests perform predictably, I fail to make POST requests work.
Here is some minimalistic example code I created to figure out the issue. The versions I'm using:
Python 3.9
Django==4.0.3
djangorestframework==3.13.1
from django.db import models
from django.urls import include, path
from django.utils import timezone
from rest_framework import routers, serializers, viewsets
router = routers.DefaultRouter()
# models.py
class SomeThing(models.Model):
created_at = models.DateTimeField(default=timezone.now)
title = models.CharField(max_length=100, null=True, blank=True)
# serializers.py
class SomeThingSerializer(serializers.ModelSerializer):
class Meta:
fields = "__all__"
model = SomeThing
# views.py
class SomeThingViewSet(viewsets.ModelViewSet):
queryset = SomeThing.objects.all().order_by('id')
serializer_class = SomeThingSerializer
# urls.py
router.register("some-things", SomeThingViewSet, basename="some_thing")
app_name = 'question'
urlpatterns = (
path('', include(router.urls)),
)
Here is my test case:
import json
from django.contrib.auth import get_user_model
from rest_framework import status
from rest_framework.test import APITestCase, APIClient
class TestUserView(APITestCase):
self.some_user = get_user_model().objects.create(login="some_user#test.ru")
#staticmethod
def get_client(user):
client = APIClient()
client.force_authenticate(user=user)
return client
def test_do_something(self):
client = self.get_client(self.compliance_chief)
url = reverse('question:some_things-list')
resp = client.post(
path=url,
data=json.dumps({"title": "Created Something"}),
content_type="application/json",
)
assert resp.status_code == status.HTTP_201_OK
(Yes, I have to use some authentication to get access to the data, but I don't think it is relevant to the problem.) To which I receive a lengthy traceback, ending with an assertion error:
File "/****/****/****/venv/lib/python3.9/site-packages/django/test/client.py", line 82, in read
assert (
AssertionError: Cannot read more than the available bytes from the HTTP incoming data.
As it is really fairly long, I'll leave it just in case in a gist without posting it here.
Steps to fix
The problem clearly happens after the correct response is returned by the viewset. To make sure the response is correct I made a slight customisation in the create method to print out the response before it is returned, like so:
class SomeThingViewSet(viewsets.ModelViewSet):
queryset = SomeThing.objects.all().order_by('id')
serializer_class = SomeThingSerializer
def create(self, request, *args, **kwargs):
response = super().create(request, *args, **kwargs)
print("THIS IS THE RESPONSE FROM THE VIEWSET", response)
return response
And, sure enough, the result is correct:
THIS IS THE RESPONSE FROM THE VIEWSET <Response status_code=201, "text/html; charset=utf-8">
Which makes me think something goes wrong at the parsing stage (actually, the traceback implies the same). I tried to tweak the way I build the query, namely:
using format instead of content type like so: resp = client.post(path=url, data={"title": "Created Something"}, format="json")
using the .generic method instead of .post like so: resp = client.generic(method="POST", path=url, data=json.dumps({"title": "Created Something"}), content_type="application/json")
The result is the same.
From googling I found out that this error indeed has occasionally occurred in connection with DRF APIClient and Django, but really long ago (like this discussion, which claims that the issue was fixed in the later versions of Django).
I'm sure the reason for this behaviour is rather obvious (some stupid mistake most likely) and the solution must be very simple, but so far I've failed to find it. I would be very grateful if somebody shared their experience, if there is any, of dealing with such an issue, or their considerations as to where to move from this deadlock.

Alright, the mystery's been resolved and I'm going to share it here in case somebody runs into something similar, although it would take quite a coincidence, so it is unlikely.
Long story short: I messed up the source code of my Django4.0.3. installed in this project.
Now, how it happened. While I was testing some stuff, I ran into an error, which I failed to locate, so I went along the whole chain of events checking if the output was what I expected it to be. Soon enough I found myself checking the output from functions in the libraries installed under my virtual environment. I realise it's a malpractice to directly modify their code, but as I was working in my local environment with an option to reinstall everything at any moment, I decided it was fine to play with them. As it resulted in nothing, I removed all the code I had added (or so I thought).
After a while I realised what caused the initial error (an overlooked condition in my testing setup), fixed it and tried to run the test. That's when the problem in question showed up.
Later I found out that the same very test performs correctly in an identical environment. Then I suspected that I broke something in my local library code. Next I simply compared the code I had dealt with in my local environment with the code from the official source and soon enough I established the offending line. It happened to be in django/test/client.py, in the definition of the RequestFactory.generic method. Something like this:
...
if not r.get("QUERY_STRING"):
# WSGI requires latin-1 encoded strings. See get_path_info().
query_string = parsed[4].encode().decode("iso-8859-1")
r["QUERY_STRING"] = query_string
req = self.request(**r)
return self.request(**r)
...
The offending line (which I added and forgot to remove) was req = self.request(**r). After I deleted it, everything returned back to normal.

How to get Twilio's RequestValidator to work

I'm not sure why this doesn't work. I have been looking at Twilio's documentation here, and I tried just making it a function call, but that didn't work, so I put it directly in my view, and it still didn't work. It always returns 403.
I have verified that my auth token is the same as what is on Twilio as well.
from braces.views import CsrfExemptMixin
from django.http import HttpResponse, HttpResponseForbidden
from twilio.util import RequestValidator
from secretphone.settings import TWILIO_AUTH_TOKEN
class SMSWebhook(CsrfExemptMixin, View):
def post(self, request):
validator = RequestValidator(TWILIO_AUTH_TOKEN)
request_valid = validator.validate(
request.build_absolute_uri(),
request.POST,
request.META.get('HTTP_X_TWILIO_SIGNATURE', '')
)
if not request_valid:
return HttpResponseForbidden()

So that nobody has the same problem I did, and no answer, apparently ngrok's https URL does not get passed through to django. My problem was that my webhook was passed using ngrok's https link, but when it got to my app, it changed to an http link. My guess is that request.build_absolute_uri() gets the protocol from django(webserver) itself, which uses http as the protocol.
So anyone having this problem while using ngrok, make sure you are not using the https link.

django require_http_methods. default page

I use require_http_methods for restricting access to views. Here example my code:
# myapp/views.py
#require_http_methods(["POST"])
def my_view(request):
return HttpResponse('my_view')
But when I go to url /myapp/my_view I see white page. So, my question is: How I can set default view if method not match rules? For example I want to show 404, 403 or something else. It is possible? Can you provide me a small example? Thank you!

In your case 405 HTTP error is returned.
You could try to create middleware for custom errors:
from django.http import HttpResponseNotAllowed
class CustomHTTPErrorsMiddleware(object):
def process_response(self, request, response):
if isinstance(response, HttpResponseNotAllowed):
# Custom response for `HttpResponseNotAllowed`.
pass
return response
Also, Django's error handling may be useful.

django.test.client got 404 error on existing url

I just started learning unittests and stuck with this problem.
I got project structure like this (it’s Django 1.6.2 now):
./manage.py
./myproject
./myproject/urls.py
./myproject/myapp/
./myproject/myapp/urls.py
./myproject/myapp/views.py
./tests/
./test/test_example.py
In the ./myproject/urls.py I have:
from django.conf.urls import patterns, include, url
urlpatterns = patterns('',
url(r'^myapp/', include('myproject.myapp.urls')),
)
In the ./myproject/myapp/urls.py I have:
from django.conf.urls import patterns, url
urlpatterns = patterns('myproject.myapp.views',
url(r'^example1/$', 'itemlist'),
url(r'^example1/(?P<item_id>\w+)/$', 'item'),
)
I wrote basic test and put it into ./test/test_example.py
import unittest
from django.test import Client
class PagesTestCase(unittest.TestCase):
def setUp(self):
self.client = Client()
def test_itemlist(self):
response = self.client.get('/myapp/example1/')
self.assertEqual(response.status_code, 200)
def test_item(self):
response = self.client.get('/myapp/example1/100100/')
self.assertEqual(response.status_code, 200)
I run this tests from shell like this:
cd ./tests
python manage.py test
First test runs OK, but he second always fails with ‘404 not found’ status code.
Both urls are working OK in the browser.
Also, I tried this:
cd ./
python manage.py shell
>>> from django.test.client import Client
>>> c = Client()
>>> r = c.get('/myapp/example1/100100/')
>>> r.status_code
200
I just can’t figure out how to run those tests properly. It seems no pattern that is passed into views as parameter ever works for me. But all fixed urls are found correctly by django.test.client.
Thank you!
EDIT: I just found that 404 fires in my myproject/myapp/views.py
There is a code:
def item(request, item_id):
try:
item = Item.objects.get(pk = int(item_id))
except (ValueError, Item.DoesNotExist):
raise Http404
And here goes the Item.DoesNotExist exception. I have no any idea, why that item not found?

Use the reverse() function instead to build URLs, that is:
In ./myproject/myapp/urls.py file give each URL pattern a name parameter that is for example:
from django.conf.urls import patterns, url
urlpatterns = patterns('myproject.myapp.views',
url(r'^example1/$', 'itemlist', name='example-one'),
url(r'^example1/(?P<item_id>\w+)/$', 'item', name='example-two'),
)
We will use the value given to the name parameter to build URLs.
Then in ./test/test_example.py:
from django.core.urlresolvers import reverse
class PagesTestCase(unittest.TestCase):
...
def test_itemlist(self):
url = reverse('example-one')
response = self.client.get(url)
self.assertEqual(response.status_code, 200)
def test_item(self):
url = reverse('example-two')
response = self.client.get(url)
self.assertEqual(response.status_code, 200)
That should do the trick.

In addition to using reverse, you can get a 404 if the expected model isn't available in the test database (as mentioned in one of the comments). You should also use Django's TestCase instead of python's unittest since the former inherits from the latter, but makes interacting with the database much easier (among other things).
An example that sets up test data:
from django.test import TestCase
from django.urls import reverse
# Or whatever your object is.
from .models import Item
class ItemViewsTestCase(TestCase):
"""Tests for Item views."""
#classmethod
def setUpTestData(cls):
"""Set up test data available for all tests in this class."""
cls.item = Item.objects.create(name='Testing')
def test_item_list_view(self):
# Note, self.client we get for free from Django's TestCase.
response = self.client.get(reverse('itemlist'))
self.assertEqual(response.status_code, 200)
def test_item_detail_view(self):
# This url pattern expects an Item.id so use the item we set up.
url = reverse('item', args=[self.item.id])
response = self.client.get(url)
self.assertEqual(response.status_code, 200)

Switch on django error page only for admins when debug = False

It is a copy of:
is there a way to show the Django debug stacktrace page to admin users even if DEBUG=False in settings?
but there is no answer
How to show django error page with stacktrace when debug=False only for admin users.
I don't want to use sentry.

I find an answer
In urls.py add
handler404 = 'site_utils.handler404'
Create site_utils.py in the same folder where urls.py and add there
from django.http import HttpResponseRedirect, HttpResponsePermanentRedirect
import sys
from django.views.debug import technical_404_response, technical_500_response
def handler404(request):
if (request.user.is_active and request.user.is_staff) or request.user.is_superuser:
exc_type, exc_value, tb = sys.exc_info()
return technical_404_response(request, exc_value)
else:
return HttpResponsePermanentRedirect("/")

What you are asking to do is incredibly insecure, which is likely why Django provides no default way to do this. Running DEBUG on a production app can expose your SETTINGS file (including API tokens) to the world.
If you really want to do this, take a look at django.views.debug.technical_500_response and write a custom exception handler that returns the value from there.

Here is a version for Django 3.2.
Save the following file in, for example, mysite/debug.py:
from django.utils.deprecation import MiddlewareMixin
from django.views.debug import technical_500_response
import sys
class UserBasedExceptionMiddleware(MiddlewareMixin):
def process_exception(self, request, exception):
if request.user.is_superuser:
return technical_500_response(request, *sys.exc_info())
Update your mysite/settings.py as follows:
MIDDLEWARE_CLASSES = (
... whatever ...
'mysite.debug.UserBasedExceptionMiddleware',
)