We Keep Coding

Getting "Exception: You cannot access body after reading from request's data stream" while working with DRF and django-revproxy

I'm using django-revproxy and Django REST Framework in my project. And I'm exposing an API where users get analytics about their chatbots, and it works as the following:
The user send requests the analytics from the Django project
Django project, checks if the user is authenticated and owns that chatbot
if True it contacts another external service.
My urls.py:
# urls.py
urlpatterns = (
url(r'^analytics/(?P<path>.*)$', api.AnalyticsFunctionsProxyView.as_view()),
)
And in my view:
# views.py
from rest_framework.authentication import TokenAuthentication
from rest_framework_jwt.authentication import JSONWebTokenAuthentication
from rest_framework.permissions import IsAuthenticated
from revproxy.views import ProxyView
from .permissions import HasChatBotPermission
...
class AnalyticsFunctionsProxyView(ProxyView):
upstream = settings.ANALYTICS_FAAS_URL
def parse_body(self, request):
if isinstance(request, rest_framework.request.Request):
return request.data
return super(AnalyticsFunctionsProxyView, self).parse_body(request)
#classmethod
def as_view(cls, *args, **kwargs):
view = super(AnalyticsFunctionsProxyView, cls).as_view(*args, **kwargs)
view = permission_classes((IsAuthenticated, HasChatBotPermission,))(view)
view = authentication_classes([TokenAuthentication, JSONWebTokenAuthentication])(view)
view = api_view(['GET', 'POST'])(view)
return view
And my HasChatBotPermission permissions
#permissions.py
class HasChatBotPermission(permissions.BasePermission):
def has_permission(self, request, view):
try:
bot_name = request.data.get('name')
user = request.user
self.message = 'Permission denied for {}'.format(name)
return ChatBot.objects.filter(user=user, project_name=project_id).exists()
except Exception:
self.message = 'Permission denied, no project_id was defined!'
return False
When the view is called it raises this exception:
Traceback (most recent call last):
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/rest_framework/request.py", line 379, in __getattribute__
return super(Request, self).__getattribute__(attr)
AttributeError: 'Request' object has no attribute 'body'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/core/handlers/exception.py", line 42, in inner
response = get_response(request)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/fcmam5/anaconda3/lib/python3.6/contextlib.py", line 52, in inner
return func(*args, **kwds)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/rest_framework/views.py", line 477, in dispatch
response = self.handle_exception(exc)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/rest_framework/views.py", line 437, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/rest_framework/views.py", line 474, in dispatch
response = handler(request, *args, **kwargs)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/rest_framework/decorators.py", line 52, in handler
return func(*args, **kwargs)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/revproxy/views.py", line 204, in dispatch
proxy_response = self._created_proxy_response(request, path)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/revproxy/views.py", line 139, in _created_proxy_response
request_payload = request.body
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/rest_framework/request.py", line 383, in __getattribute__
return getattr(self._request, attr)
File "/home/fcmam5/dela3a/env/lib/python3.6/site-packages/django/http/request.py", line 264, in body
raise RawPostDataException("You cannot access body after reading from request's data stream")
django.http.request.RawPostDataException: You cannot access body after reading from request's data stream
The issue is caused by this line bot_name = request.data.get('name') in my permissions.py, when I pass a string directly it passes without any problems.
My question is:
How can I access the request body without having this error? Why I'm having this error?
Is there a better solution for checking the user permission with Django revproxy.
This is my first question in Stackoverflow, sorry if my question is not clea, and for my poor English :)

You're having this error because django-revproxy attempts to read the raw request body so it can create a proxy request to the upstream server.
However, with Django's (and WSGI's, and buffering) semantics, this isn't possible once you've accessed the request body as anything but a raw stream, which you do when you request.data.get('name'). This parses the request body as JSON, HTTP multipart, whatever, depending on DRF's request negotiation configuration, and consumes the stream.
There are two ways you could get around this, as far as I see:
pass bot_name somewhere else than the body; a query string parameter, an HTTP header, part of the URL, for instance, so you don't need to access the body, or
make the backend request yourself using requests instead of reverse-proxying (which is basically the same thing, but with added magic to try and copy the request through as-is).

How to test get request in Django with a data?

I am now writing tests for my web application in Django. I have an URL 127.0.0.1/home/device/(?P<item>[^/]+). I was testing an invalid URL path.
Here item is a device name in the database. Consider an invalid device and while testing I have given the following code:
response=self.client.get("/health/errorfiles/",{'item':'testdevice_R'})
This give me a 404 response. The same thing I have tried with:
response=self.client.get("/health/errorfiles/testdevice_R/")
But this time, the test runner executes my view function and gives a TypeError since it is not an invalid device.
In both of the methods, which one is correct usage for a get request?
views.py
def showfiles(request,item):
if request.user.is_anonymous(): ## check if the user is valid
return HttpResponseRedirect("/")
db = MySQLdb.connect(host='localhost',user=root,passwd='123mcvis',db='test_db')
s = db.cursor()
username=request.user
s.execute("Select tzone,type from device where user='%s' and device='%s'"%(username,item)
tz=s.fetchone()
timezone.activate(tz[0])
s.close()
return render(request,"Home/showdevices.html")
This is my view function and since the device is invalid, it shows typ error.
class showfile_test(TestCase):
def test_invalid(self):
response=self.client.get("/health/errorfiles/testdevice_R/")
self.assertEqual(response.status_code,404)
Traceback
ERROR: test_invalid (HealthApp.tests2.showfile_test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vishnumc/vishnu/project/django/official/version6.2.3/HealthApp/tests2.py", line 162, in test_showfiles_with_invalid_deviceid
response=self.client.get("/health/errorfiles/invaliddevice/")
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/test/client.py", line 529, in get
**extra)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/test/client.py", line 333, in get
return self.generic('GET', path, secure=secure, **r)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/test/client.py", line 409, in generic
return self.request(**r)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/test/client.py", line 494, in request
six.reraise(*exc_info)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
response = self._get_response(request)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/vishnumc/vishnu/project/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/vishnumc/vishnu/project/django/official/version6.2.3/HealthApp/views.py", line 118, in showfiles
timezone.activate(tz[0]) ##setting current timezone to user's time zone for display timestamps
TypeError: 'NoneType' object has no attribute '__getitem__'

The first example is a GET request to /health/errorfiles/?item=testdevice_R. In the view, you would then find item in request.GET.
The second example is a GET request to /health/errorfiles/testdevice_R/. There is no data in request.GET and Django will pass item to your view since you have a named group (?P<item>[^/]+) in your URL pattern.
You should use the second version, because you want to test the URL pattern r'/home/device/(?P<item>[^/]+)'.
The second version of your test has uncovered problems in your view. You need to fix the view so that it doesn't raise TypeError.
Ideally, you shouldn't be writing raw SQL like that. Django allows you to do something like the following:
from .models import Device
from django.shortcuts import get_object_or_404, render
def showhome(request, item):
device = get_object_or_404(Device, user=request.user, item=item)
if device is not None:
timezone.activate(device.tzone)
return render(request,"Home/showdevices.html", {'device': device})
If you must use raw SQL then don't use string substitution %(username,item). Your current code exposes you to an SQL injection account. You should change it to:
s.execute("Select tzone,type from device where user=%s" and device=%s, (username, item))
Your code then has handle the case where tz is None, to avoid the TypeError.
tz = s.fetchone()
if tz is not None:
timezone.activate(tz[0])
else:
# Decide what to do if no items returned

django test RequestFactory cannot get route parameter to work

I have a problem that I don't know how to make it work.
urls.py:
urlpatterns = [
url(r'athletes/search$', SearchAthletes.as_view()),
url(r'athletes/([0-9]+)$', ViewAthlete.as_view())
]
views.py:
class ViewAthlete(APIView):
def get(self, request, id, format=None):
athlete = Athlete.objects.get(id=id)
serializer = AthleteSerializer(athlete)
return Response(serializer.data)
test.py:
def test_view_athlete(self):
tmp = Athlete.objects.order_by('?')[0]
request = self.factory.get('/_api/v1/athletes/' + str(tmp.id))
request.user = AnonymousUser()
response = ViewAthlete.as_view()(request)
self.assertEquals(response.data.id, tmp.id)
I keep getting the following error:
Traceback (most recent call last):
File "/tests.py", line 44, in test_view_athlete
response = ViewAthlete.as_view()(request)
File "/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/venv/lib/python3.5/site-packages/rest_framework/views.py", line 474, in dispatch
response = self.handle_exception(exc)
File "/venv/lib/python3.5/site-packages/rest_framework/views.py", line 471, in dispatch
response = handler(request, *args, **kwargs)
TypeError: get() missing 1 required positional argument: 'id'
To my understanding, the problem is that, there is no id parameter passed to get function of ViewAthelete view class. What is the reason for this? In development environment (not testing), it displays the data but testing environment doesn't recognize the arguments from the route.

AFAIK urlpatterns are considered when testing through the full django request stack, e.g.: through django.test.Client, using it's get/post methods
When testing your view directly (MyView.as_view()(request)) the whole url resolver logic is bypassed, and then the args/kwargs need to be supplied by the caller (e.g.: MyView.as_view()(request, 'arg1', 'arg2', id='34'))

As zsepi says, your URLs aren't used here. To avoid repeating the arguments, rather than calling the view directly you could use the test client to "call" the URL: another advantage of doing this is that the middleware runs, so you don't need to assign the user attribute separately.
response = self.client.get('/_api/v1/athletes/' + str(tmp.id))

Patch persisting across multiple Django unit tests?

I'm at a bit of a loss as to why two of my tests are conflicting. The patch in one test seems to be persisting to the other test. If I comment the patching test out, or just run the other test by itself there is no problem. My tests are:
class TestUrlRouting(TestCase):
#patch('myapp.views.Home')
def test_home_url_routes_to_home_view(self, mock_home_view):
url = reverse('home')
match = resolve(url)
assert match.func == mock_home_view.as_view()
class TestHomeView(TestCase):
def test_front_page_renders_home_template(self):
response = self.client.get('/')
self.assertTemplateUsed(response, 'home.html')
With the Home view being a TemplateView. The error I end up with is:
Traceback (most recent call last):
File "/path/to/code/source/myapp/tests/test_views.py", line 11, in test_front_page_renders_home_template
response = self.client.get('/')
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/test/client.py", line 503, in get
**extra)
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/test/client.py", line 304, in get
return self.generic('GET', path, secure=secure, **r)
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/test/client.py", line 380, in generic
return self.request(**r)
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/test/client.py", line 467, in request
six.reraise(*exc_info)
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
raise value
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/core/handlers/base.py", line 242, in get_response
response = self.apply_response_fixes(request, response)
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/core/handlers/base.py", line 305, in apply_response_fixes
response = func(request, response)
File "/path/to/code/virtualenv/lib/python3.5/site-packages/django/http/utils.py", line 17, in conditional_content_removal
if 100 <= response.status_code < 200 or response.status_code in (204, 304):
TypeError: unorderable types: int() <= MagicMock()
Because I'm not doing anything special with the patch decorator, my guess is that it has something to do with how Django is setting things up. In particular, I found that patching myapp.urls.Home does not work. Instead I need to patch myapp.views.Home, despite the fact that the line under test is in urls.py as url(r'^$', Home.as_view(), name='home'),. Any thoughts as to why this patch is persisting and how I can fix it? Thank you much!
Update:
The problem persists even when mocking myapp.views.Home.as_view instead of just myapp.views.Home (once again, note views is the option that correctly mocks and not urls for some reason).

The call to .as_view() is made only once at import time (first time Django needs to visit "urls.py"), that's why it's "persisted".
IMO your first test case is just testing the Django framework and not anything in your own code except that the home view has a url name of "home". It isn't worth it to be that picky. You can easily cover that reverse('home') gets you to the right view in a another test case where you don't need to mock Django globals.

Test written for django URL Dispatcher not working

I'm working on a simple webservice in django. This is my first web app in django/python so I wouldn't be surprised if I'm missing something obvious here, but...
I'm currently trying to test my logic for filtering a url.
# Works as expected
response = self.client.post("/mysite/goodurl/")
self.assertEqual(response.status_code, 200)
# Has an exception rather than a 404
response = self.client.post("/mysite/badurl/")
self.assertEqual(response.status_code, 404)
So, the badurl case isn't simply being not found and throwing a 404, instead I'm getting this error:
Traceback (most recent call last):
File "/home/user/me/mysite/tests.py", line 55, in test_add_tracker
response = self.client.post("/mysite/badurl/")
File "/home/path/to/some/bin/dir/freeware/Python/lib/python2.7/site-packages/django/test/client.py", line 449, in post
response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
File "/home/path/to/some/bin/dir/freeware/Python/lib/python2.7/site-packages/django/test/client.py", line 262, in post
return self.request(**r)
File "/home/path/to/some/bin/dir/freeware/Python/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/path/to/some/bin/dir/freeware/Python/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
return view_func(*args, **kwargs)
TypeError: EtimeFetcher() got an unexpected keyword argument 'alias'
I've tried googling for the EtimeFetcher message, but no luck. Any ideas?

Uou may probably have some view to catch all Http404 errors. It seems that Django however had found a view to execute for the /badurl. Search in your code for statements containing the alias named parameter, like: "alias=yyyy". Or probably some url pattern, contaning 'alias' as extra parameter in urls.py?