We Keep Coding

Assertion error parsing response to POST request from DRF viewset with DRF APIClient in tests

I've run into a strange issue with Django Rest Framework testing engine. The weird thing is that everything used to work fine with Django 3 and this issue turned up after I migrated to Django 4. Apart from testing, everything works well, and responds to queries as expected.
The problem
I'm using DRF APIClient to make queries for unit tests. While GET requests perform predictably, I fail to make POST requests work.
Here is some minimalistic example code I created to figure out the issue. The versions I'm using:
Python 3.9
Django==4.0.3
djangorestframework==3.13.1
from django.db import models
from django.urls import include, path
from django.utils import timezone
from rest_framework import routers, serializers, viewsets
router = routers.DefaultRouter()
# models.py
class SomeThing(models.Model):
created_at = models.DateTimeField(default=timezone.now)
title = models.CharField(max_length=100, null=True, blank=True)
# serializers.py
class SomeThingSerializer(serializers.ModelSerializer):
class Meta:
fields = "__all__"
model = SomeThing
# views.py
class SomeThingViewSet(viewsets.ModelViewSet):
queryset = SomeThing.objects.all().order_by('id')
serializer_class = SomeThingSerializer
# urls.py
router.register("some-things", SomeThingViewSet, basename="some_thing")
app_name = 'question'
urlpatterns = (
path('', include(router.urls)),
)
Here is my test case:
import json
from django.contrib.auth import get_user_model
from rest_framework import status
from rest_framework.test import APITestCase, APIClient
class TestUserView(APITestCase):
self.some_user = get_user_model().objects.create(login="some_user#test.ru")
#staticmethod
def get_client(user):
client = APIClient()
client.force_authenticate(user=user)
return client
def test_do_something(self):
client = self.get_client(self.compliance_chief)
url = reverse('question:some_things-list')
resp = client.post(
path=url,
data=json.dumps({"title": "Created Something"}),
content_type="application/json",
)
assert resp.status_code == status.HTTP_201_OK
(Yes, I have to use some authentication to get access to the data, but I don't think it is relevant to the problem.) To which I receive a lengthy traceback, ending with an assertion error:
File "/****/****/****/venv/lib/python3.9/site-packages/django/test/client.py", line 82, in read
assert (
AssertionError: Cannot read more than the available bytes from the HTTP incoming data.
As it is really fairly long, I'll leave it just in case in a gist without posting it here.
Steps to fix
The problem clearly happens after the correct response is returned by the viewset. To make sure the response is correct I made a slight customisation in the create method to print out the response before it is returned, like so:
class SomeThingViewSet(viewsets.ModelViewSet):
queryset = SomeThing.objects.all().order_by('id')
serializer_class = SomeThingSerializer
def create(self, request, *args, **kwargs):
response = super().create(request, *args, **kwargs)
print("THIS IS THE RESPONSE FROM THE VIEWSET", response)
return response
And, sure enough, the result is correct:
THIS IS THE RESPONSE FROM THE VIEWSET <Response status_code=201, "text/html; charset=utf-8">
Which makes me think something goes wrong at the parsing stage (actually, the traceback implies the same). I tried to tweak the way I build the query, namely:
using format instead of content type like so: resp = client.post(path=url, data={"title": "Created Something"}, format="json")
using the .generic method instead of .post like so: resp = client.generic(method="POST", path=url, data=json.dumps({"title": "Created Something"}), content_type="application/json")
The result is the same.
From googling I found out that this error indeed has occasionally occurred in connection with DRF APIClient and Django, but really long ago (like this discussion, which claims that the issue was fixed in the later versions of Django).
I'm sure the reason for this behaviour is rather obvious (some stupid mistake most likely) and the solution must be very simple, but so far I've failed to find it. I would be very grateful if somebody shared their experience, if there is any, of dealing with such an issue, or their considerations as to where to move from this deadlock.

Alright, the mystery's been resolved and I'm going to share it here in case somebody runs into something similar, although it would take quite a coincidence, so it is unlikely.
Long story short: I messed up the source code of my Django4.0.3. installed in this project.
Now, how it happened. While I was testing some stuff, I ran into an error, which I failed to locate, so I went along the whole chain of events checking if the output was what I expected it to be. Soon enough I found myself checking the output from functions in the libraries installed under my virtual environment. I realise it's a malpractice to directly modify their code, but as I was working in my local environment with an option to reinstall everything at any moment, I decided it was fine to play with them. As it resulted in nothing, I removed all the code I had added (or so I thought).
After a while I realised what caused the initial error (an overlooked condition in my testing setup), fixed it and tried to run the test. That's when the problem in question showed up.
Later I found out that the same very test performs correctly in an identical environment. Then I suspected that I broke something in my local library code. Next I simply compared the code I had dealt with in my local environment with the code from the official source and soon enough I established the offending line. It happened to be in django/test/client.py, in the definition of the RequestFactory.generic method. Something like this:
...
if not r.get("QUERY_STRING"):
# WSGI requires latin-1 encoded strings. See get_path_info().
query_string = parsed[4].encode().decode("iso-8859-1")
r["QUERY_STRING"] = query_string
req = self.request(**r)
return self.request(**r)
...
The offending line (which I added and forgot to remove) was req = self.request(**r). After I deleted it, everything returned back to normal.

Generating OpenAPI Schema in Django Rest Framework: URL Keyword in Update ViewSet

I'm trying to do something that should be pretty straight forward but I'm running into a problem that I can't make heads or tails of. I'm hoping someone can help me sort it out.
I've created some API endpoints using DRF. Naturally, I have Create, Update, Retrieve, and Delete endpoints that I've constructed with the django_rest_framework.generics. I need to send some extra data into the Serializer from the ViewSet on both the CreateAPIView and UpdateAPIView. To accomplish this, I've extended the get_serializer_context() method on those ViewSets.
I'm now trying to create OpenAPI documentation for those said endpoints. However, when I run generateschema in the manage.py it returns an error. This error occurs in each of the UpdateAPIViews where I've extended the get_serializer_context() method. An example of this error is shown below.
AssertionError: Expected view UpdateStaffViewSet to be called with a URL keyword argument named "staff_id". Fix your URL conf, or set the `.lookup_field` attribute on the view correctly.
Naturally, we'd think that I goofed the URL keyword or the lookup_field, just as the error states. Unless I'm looking over something obvious though, I just cannot see what the issue is. These endpoints function perfectly when tested using curl or postman. I do know that if I remove the extended get_serializer_context() method, the generateschema command works fine. (But then an test using postman/curl will fail.) Why in the heck would that extended method matter!?
urls.py
url_patterns = [
path('v1/staff/retrieve/<int:staff_id>/', RetrieveStaffViewSet.as_view()),
path('staff/update/<int:staff_id>/', UpdateStaffViewSet.as_view()),
...
]
viewsets
class UpdateRetrieveBase:
serializer_class = StaffSerializer
queryset = Staff.objects.select_related('user')
lookup_url_kwarg = 'staff_id'
lookup_field = 'pk'
class UpdateStaffViewSet(UpdateRetrieveBase, UpdateAPIView):
def get_serializer_context(self):
context = super().get_serializer_context()
context['validation_refs'] = get_validation_refs(staff=self.get_object())
return context
class RetrieveStaffViewSet(UpdateRetrieveBase, RetrieveAPIView):
pass

Alright well after all that, I found my own answer. I'll post it here in case someone else also gets stumped.
Since the generateschema command runs a test but then doesn't include URL kwargs, we cannot call get_object() in the viewset. The test must account for that but doesn't consider that we may be calling that method anywhere else. To fix this in my particular case I edited the code as thus.
viewset
class UpdateAEAContractViewSet(UpdateRetrieveBase, UpdateAPIView):
def get_serializer_context(self):
context = super().get_serializer_context()
contract_id = self.kwargs.get(self.lookup_url_kwarg)
context['validation_refs'] = get_validation_refs(contract_id=contract_id)
return context
validation ref subjob
def get_validation_refs(staff_id=None):
staff = Staff.objects.get(pk=staff_id) if staff_id else None
...

django RequestFactory loses url kwargs

I am trying to switch from using Django Test Client to RequestFactory to speed up my tests. However, requests generated by RequestFactory do not supply proper kwargs to views.
Example: this is my view
class SomeView(View):
def get(self, request, *args, **kwargs):
return JsonResponse({'your kwargs': str(kwargs)})
with urlconf
url(r'^some_view/(?P<some_kwarg>[\-0-9a-fA-F]+)/$',
views.SomeView.as_view(),
name='some_view'),
and two tests:
def test_different_kwargs():
c = Client()
response = c.get(
reverse('bots:some_view',
kwargs={'some_kwarg': '12345'}),
)
print('\n\nResponse for TestClient: ', response.content.decode())
rf = RequestFactory()
request = rf.get(
reverse('bots:some_view',
kwargs={'some_kwarg': '12345'}),
)
response = SomeView.as_view()(request)
print('\n\nResponse for RequestFactory: ', response.content.decode())
What they produce is:
Response for TestClient: {"your kwargs": "{'some_kwarg': '12345'}"}
Response for RequestFactory: {"your kwargs": "{}"}
So, what's the point of RequestFactory if it loses url kwargs? Or is there a way to put them into the view somehow?

EDIT 2020-03-07: As gain I more experience in testing, I updated my answer to remove confusion around functional testing and I added some advises.
There are two aspects to your answer.
Quick answer: how to put the kwargs in the view? You need to change your code to this:
def test_different_kwargs():
kwargs={'some_kwarg': '12345'}
url = reverse('bots:some_view', kwargs=kwargs)
c = Client()
response = c.get(url)
print('\n\nResponse for TestClient: ', response.content.decode())
rf = RequestFactory()
request = rf.get(url)
response = SomeView.as_view()(request, **kwargs)
print('\n\nResponse for RequestFactory: ', response.content.decode())
Long answer:
Then the difference between RequestFactory and Client:
It has been developed a bit here: Django test RequestFactory vs Client
but I would like to complete it a bit.
In terms of functionality Client handle the whole stack used to process the response including middlewares and url resolutions (url matching and parameters extraction). On the other side RequestFactory, just build a request object, leaving the responsibility to the user to add the proper attributes and to call the appropriate view functions or view methods.
Hence the call to ClassView.as_view()(request, *args, **kwargs) in the second case.
In terms of testing, Client is focused on integration testing (you will test that all the different parts fit together: middlewares, class-based/function view, template, templatetags), it's the end-to-end mechanism that you are testing here.
Client -> { UrlResolver -> Middleware -> View -> Middlewares -> TemplateResponse } -> Tests
Using RequestFactory you can focus on testing smaller parts, a class-based view method, a function view, a middleware method etc. In that sense RequestFactory is more related to unit testing.
See Request Factory Reference
If you are interest in unit tests and mocking methods, there is not much literature on this but you can look at this article (Testing Django Views in Isolation)[https://matthewdaly.co.uk/blog/2015/08/02/testing-django-views-in-isolation/].
In the end it all depends on how much you time can focus on testing. Integration/Functional/Unit testings have different advantages and drawbacks.
Advises (may be biased): If you develop a website, I advise the following:
focus on integration testing by testing routes and their expected behaviors ;
add unit/integration tests for your business logic (models in Django) ;
Unit testing parts using RequestFactory will take you more time and won't bring much benefits over using the Client API.
Using Client API you will be closer to how your website will be used and how it will behave.

What's the most elegant way to convert requests' response to DRF response in Django?

Consider the following flow:
public client ----> DRF API on Service A ------> DRF API on Service B
Some of the DRF API on Service A merely proxying to Service B, so in the particular API on Service A looks like this:
class SomeServiceAPI(APIView):
def get(request):
resp = requests.get('http://service-b.com/api/...')
return Response(resp.json())
While this works on normal status, but it has a few issues:
It doesn't proxy the actual status code from service b.
Unnecessary round-trip of json serialization within Response()
If service b returns a non-json error, service does not return actual error from service b.
The question is, is there a better way to do it? I had a look at Django Rest Framework Proxy project, but I am not entirely sure if it actually suits my use case here.

You can solve the status code part by modifying your Response:
return Response(resp.json(), status=resp.status_code)
For the second part though, this is the essence of Proxying... (True, sometimes you want to manipulate the request and/or the response in the middleman of the proxy, but what you do is the essence).
Notes:
The DRF Proxy that you are suggesting seems to do the job just
fine, without the need for you to write a specific view just for the
roundtrip.
There exist another tool, DRF Reverse Proxy which is a DRF port of Django Revproxy and you may want to consider.
The general idea of both of the above is that you create a URL path specifically to Proxy the path to another API:
DRF Proxy:
Add your proxy to settings.py:
REST_PROXY = {
'HOST': 'http://service-b.com/api/'
}
In urls.py:
url(
r'^somewere_in_a/$',
ProxyView.as_view(source='somewere_in_b/'),
name='a_name'
)
DRF Reverse Proxy:
Pretty much similar with the above, without the settings part:
url(
r'^(?P<path>.*)$',
ProxyView.as_view(upstream='http://service-b.com/api/somewere_in_b/'),
name='a_name'
)
Opinion: the DRF Proxy seems more solid...

I had a look at both existing packages mentioned in John's answer but they don't seem to perfectly suit in my use case, so I have created a simple wrapper to proxy the requests' response to DRF response.
# encoding: utf-8
from __future__ import unicode_literals
from __future__ import absolute_import
from rest_framework.response import Response
from requests.models import Response as RResponse
class InCompatibleError(Exception):
pass
class DRFResponseWrapper(Response):
"""
Wraps the requests' response
"""
def __init__(self, data, *args, **kwargs):
if not isinstance(data, RResponse):
raise InCompatibleError
status = data.status_code
content_type = data.headers.get('content_type')
try:
content = data.json()
except:
content = data.content
super(DRFResponseWrapper, self).__init__(content, status=status, content_type=content_type)
And use as below:
resp = requests.get(
'{}://{}/api/v5/business/'.format(settings.SEARCH_HOST_SCHEMA, settings.SEARCH_HOST),
params=request.query_params
)
return DRFResponseWrapper(resp)

Django: Serving a Download in a Generic View

So I want to serve a couple of mp3s from a folder in /home/username/music. I didn't think this would be such a big deal but I am a bit confused on how to do it using generic views and my own url.
urls.py
url(r'^song/(?P<song_id>\d+)/download/$', song_download, name='song_download'),
The example I am following is found in the generic view section of the Django documentations:
http://docs.djangoproject.com/en/dev/topics/generic-views/ (It's all the way at the bottom)
I am not 100% sure on how to tailor this to my needs. Here is my views.py
def song_download(request, song_id):
song = Song.objects.get(id=song_id)
response = object_detail(
request,
object_id = song_id,
mimetype = "audio/mpeg",
)
response['Content-Disposition'= "attachment; filename=%s - %s.mp3" % (song.artist, song.title)
return response
I am actually at a loss of how to convey that I want it to spit out my mp3 instead of what it does now which is to output a .mp3 with all of the current pages html contained. Should my template be my mp3? Do I need to setup apache to serve the files or is Django able to retrieve the mp3 from the filesystem(proper permissions of course) and serve that? If it do need to configure Apache how do I tell Django that?
Thanks in advance. These files are all on the HD so I don't need to "generate" anything on the spot and I'd like to prevent revealing the location of these files if at all possible. A simple /song/1234/download would be fantastic.

Why do you want to do this with a generic view? It's very easy to do this without generic views:
from django.http import HttpResponse
def song_download(request, song_id):
song = Song.objects.get(id=song_id)
fsock = open('/path/to/file.mp3', 'rb')
response = HttpResponse(fsock, content_type='audio/mpeg')
response['Content-Disposition'] = "attachment; filename=%s - %s.mp3" % \
(song.artist, song.title)
return response
I'm not sure if it's possible to make this work somehow with a generic view. But either way, using one is redundant here. With no template to render, the context that is automatically provided by the generic view is useless.

To wrap my comment to Tomasz Zielinski into a real answer:
For several reasons it is indeed better to let apache/nginx/etc do the work of sending files.
Most servers have mechanisms to help in that usecase: Apache and lighttpd have xsendfile, nginx has X-Accel-Redirect.
The idea is that you can use all the features of django like nice urls, authentification methods, etc, but let the server do the work of serving files. What your django view has to do, is to return a response with a special header. The server will then replace the response with the actual file.
Example for apache:
def song_download(request):
path = '/path/to/file.mp3'
response = HttpResponse()
response['X-Sendfile'] = smart_str(path)
response['Content-Type'] = "audio/mpeg"
response['Content-Length'] = os.stat(path).st_size
return response
install mode_xsendfile
add XSendFileOn on and (depending on the version) XSendFileAllowAbove on or XSendFilePath the/path/to/serve/from to your apache configuration.
This way you don't reveale the file location, and keep all the url management in django.

Serving static files with Django is a bad idea, use Apache, nginx etc.
https://docs.djangoproject.com/en/dev/howto/static-files/deployment/

To answer the original question how to use a generic view, you could do the following:
from django.views.generic import DetailView
from django.http.response import FileResponse
class DownloadSong(DetailView):
model = Song
def get(self, request, *args, **kwargs):
super().get(request, *args, **kwargs)
song = self.object
return FileResponse(open(song, 'rb'),
as_attachment=True,
filename=f'{song.artist} - {song.title}.mp3')
Docs:
Detailview: https://docs.djangoproject.com/en/3.2/ref/class-based-views/generic-display/#detailview
FileResponse: https://docs.djangoproject.com/en/3.2/ref/request-response/#fileresponse-objects
If your Django version does not have the FileResponse object, use the HttpResponse as shown in the other answers.