Using Google App Engine Queues service with Django - python

I am trying to use Google App Engine queues API, I am having problems on testing this. It seems that in some part of the process the CSRF it's not working.
as I understand the api executes the task calling the url and making and http request in background.
The complete url is the API is calling is → http://localhost.localdomain:8000/admin/cooking/recipe/36/chefworker/
When it raises this exception:
Traceback (most recent call last):
File "/home/mariocesar/Proyectos/Cooking/cooking/django/core/handlers/base.py", line 100, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/mariocesar/Proyectos/Cooking/cooking/django/views/decorators/csrf.py", line 24, in wrapped_view
resp.csrf_exempt = True
AttributeError: 'NoneType' object has no attribute 'csrf_exempt'
So, the csrf middleware, the cookie, some data or the response itself is missing from the request that the GAE api makes to execute the task in the background.
How to solve this without disabling CSRF on Django? however, it's posible with djangoappengine at all?
Down are the models.py and admin.py files I am using.
models.py
from django.db import models
class Recipe(models.Model):
name = models.CharField(max_length=140)
description = models.TextField()
cooking_time = models.PositiveIntegerField()
status = models.CharField(max_length=40)
def __unicode__(self):
return self.name
def cookthis(self):
import time
self.status = 'The chef is cooking this recipe'
self.save()
time.sleep(obj.cooking_time)
self.status = 'It\'s done ! the recipe is ready to serve'
self.save()
admin.py
import logging
from django.contrib import admin, messages
from django.http import HttpResponse
from django.utils.functional import update_wrapper
from django.contrib.admin.util import unquote
from django.shortcuts import get_object_or_404, render_to_response
from django import template
from django.core.urlresolvers import reverse
from google.appengine.api import taskqueue
from google.appengine.api.taskqueue import TaskAlreadyExistsError
from cooking.models import Recipe
from django.views.decorators.csrf import csrf_exempt
class AdminRecipe(admin.ModelAdmin):
def get_urls(self):
from django.conf.urls.defaults import patterns, url
def wrap(view):
def wrapper(*args, **kwargs):
return self.admin_site.admin_view(view)(*args, **kwargs)
return update_wrapper(wrapper, view)
info = self.model._meta.app_label, self.model._meta.module_name
urlpatterns = super(AdminRecipe, self).get_urls()
myurls = patterns('',
url(r'^(.+)/cook/$',
wrap(self.cook_view),
name='%s_%s_chefworker' % info),
url(r'^(.+)/chefworker/$',
wrap(self.chefworker_worker),
name='%s_%s_chefworker' % info),
)
return myurls + urlpatterns
def cook_view(self, request, object_id, extra_context=None):
obj = get_object_or_404(Recipe, pk=unquote(object_id))
if request.POST:
try:
taskqueue.add(
name="recipie-%s" % obj.id,
url=reverse('admin:cooking_recipe_chefworker', args=(obj.id,))
)
messages.add_message(request, messages.INFO, 'Chef is cooking the recipe.')
except TaskAlreadyExistsError:
messages.add_message(request, messages.ERROR, 'chef is already cooking that recipe.')
context_instance = template.RequestContext(request, current_app=self.admin_site.name)
return render_to_response("admin/cooking/recipe/cook_view.html", {'object': obj}, context_instance=context_instance)
#TODO: Add csrf token on form
#csrf_exempt
def chefworker_worker(self, request, object_id, extra_context=None):
import time
if request.POST:
obj = get_object_or_404(Recipe, pk=unquote(object_id))
obj.cookthis()
return HttpResponse('done')
admin.site.register(Recipe, AdminRecipe)
IMPORTANT NOTE:
Was hard to debug this error, cause the dev_appserver logger was just raising 403 errors, no other info; so, I have to patch the file google/appengine/api/taskqueue/taskqueue_stub.py line 574 and add "logging.info('response --- \n%s' % result)" to get the output.

If you have the CsrfViewMiddleware enabled, Django will require a csrf_token in all POSTs to your views.
Django provides a decorator, #csrf_exempt, that you should place on your task queue views. This turns off the middleware just for those views.
Alternatively, you can avoid using CsrfViewMiddleware altogether and instead use the #csrf_protect decorator where you need it. I don't recommend doing this -- it's probably safer to protect everywhere and carve out a small number of exemptions for your task queue views.
(One last note: both answers above -- that something is wrong with your view, or that you should just use GET for the task queue -- strike me wrong. There's nothing wrong with your view, and POST is the right verb to use for task queue tasks.)

Looking at the source of csrf.py, it looks like this would only occur if your view function is returning None (or not explicitly returning, in which case Python would return None implicitly). Looking at your code, I don't see how that could occur, though - are you sure this is your exact deployed code?
Also, you probably don't want to use get_object_or_404 inside a task queue task - if it can't find the object, it'll throw a 404, which will cause the task to error and retry indefinitely.
You also shouldn't need CSRF protection (per your TODO); instead, make sure the task queue URL is marked admin-only, and it will only ever be called by the task queue service.

I'm not an expert, but you may try using GET instead of POST. See http://groups.google.com/group/django-non-relational/browse_thread/thread/e6baed5291aed957/d6c42150c8e246e1?lnk=gst&q=queue#d6c42150c8e246e1 (the last entry)

Related

Apply django authentication for all views

I am trying to implement Django basic authentication for all of the views in my views.py file. Although I can add the authentication code snippet in every view, but it will not be easy to apply this to upcoming views. Is there any way that every view in my views.py will automatically check for the authentication?
views.py
def mgmt_home(request):
##############################################################
# This code is repetitive
##############################################################
if request.user.is_anonymous:
return redirect("/login")
##############################################################
test_name = Test.objects.all()[0].test_name
metadata = {
"test_name": test_name,
}
return render(request, "mgmt_home.html", metadata)
Is there any way where I can avoid this repetitive code in all of my views?
you can use 'login_required()' decorator or 'LoginRequiredMixin' class from django authentication.
https://docs.djangoproject.com/en/3.1/topics/auth/default/
How to specify the login_required redirect url in django?
You have 2 options:
from django.contrib.auth.decorators import login_required
You can add this #login_required() decorator to your every view and it will automatically redirect a user to the login page (or whatever page you want to send the user to) any time your user is not logged in.
This option, in your case, I would not recommend, as this might be an overkill and not required for your simple problem. The solution is to create a custom Middleware and add your code to it, and then, of course, add the Middleware to the Settings.py file. This way, each time your views run, your Middlewares will run prior to that. In fact, that's the purpose of Middlewares. They are designed to reduce redundancies and problems exactly such as yours.
Create a middleware.py file anywhere on your python path. Add the below codes to your created middleware.py file
from django.http import HttpResponseRedirect
from django.urls import reverse_lazy
def redirect_to_login():
return HttpResponseRedirect(reverse_lazy('users:login'))
class AuthPageProtectionMiddleware:
def __init__(self, get_response):
self.get_response = get_response
# One-time configuration and initialization.
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
if request.user.is_authenticated:
if not request.user.is_admin:
return redirect_to_login()
else:
return redirect_to_login()
response = self.get_response(request)
# Code to be executed for each request/response after
# the view is called.
return response
NOTE
You can replace the redirection URL with your application-specific one.

Send the response to user from other file instead of views.py form DRF

I am working on DRF in which I have created multiple pythons and importing that file into views.py but when I am trying to send the response to the user from another file it is not working I understand the behind the reason but is there any way to send the response to the user from other than view.py.
view.py Structure
#restframework view and all
from concurrent.futures import ThreadPoolExecutor
import other as o
call FileView(APIView):
#parseclass
def post(self,request):
o.func(d)
Other.py
from rest_framework.response import Response
from rest_framework.status import status
from django.conf import setting
def func(d):
return Response({"A":"OK"})
but Response is not coming to the user end.
is there any way to send a response from other.py to the user instead of getting value and then send the response from the view.py?
I am also implementing multithreading in my DRF.
Just add return and it should work:
class FileView(APIView):
#parseclass
def post(self,request):
return o.func(d)
If you want to skip this step and call view in others.py directly. Just make that an APIView and register it in urls.py. It should work

Django Error Logging: Adding request header, body and user information

Looking for a way to add header, body and a user's email address in my error log along with the stack trace of the exception in my views.py
After scouring the web for hours, many suggested to write my own middleware and some suggested to log that sort of information into a separate log. However, knowing where your code went wrong solves one part of the problem, identifying which poor soul it affected and what request data was sent during that exception goes a long a way in rectifying the issue. Having that information in the same log file just makes sense to me.
Currently in my views.py, I have this simple setup:
from django.db.models import Min, Max, Q, F, Count, Sum
from django.db import connection
from django.conf import settings
from django.http import HttpResponse, HttpResponseRedirect
from myapp.models import *
import logging
logging.basicConfig(filename="errors.log",
level=logging.ERROR,
format='%(asctime)s: %(message)s')
def random_view(request):
if request.user.is_authenticated() and request.user.is_active:
# generic view code goes here.
else:
return HttpResponse(status=401)
This setup worked well for a while. Every time there was an exception, it would log out the time, the exception error message and the stack trace.
How can I also add in request.META, request.user.id and request.body along with stack trace?
Any suggestions would help. A worked out answer, even better!
Thank you
I think a complete solution to the logging problem you have is to implement a middleware. The middleware would be able to work with any kind of view implementation you have, irrespective of whether it is a class based view, function based view or APIView from DRF.
You can define a middleware for full logging. Make sure that you place the middleware appropriately after the authentication middleware -
MIDDLEWARE = [
...,
'django.contrib.auth.middleware.AuthenticationMiddleware',
...,
'path.to.your.middleware.LogMiddleware'
]
In the log middleware, you would have access to the request and response. You can store request, the user (if authenticated) and all the META properties coming from the request via a logger, or you can even store it in a database if you want. Although, beware that storing in database comes at a cost. You can learn how to write a middleware by going through the Django middleware documentation.
import traceback
class LogMiddleware():
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
try:
return self.get_response(request)
except:
if request.user.is_authenticated():
# Log the user
path = request.get_full_path() # Get the URL Path
tb = traceback.format_exc() # Get the traceback
meta = request.META # Get request meta information
# Log everything
raise # Raise exception again after catching
You can read about all the meta attributes present from the django documentation of HttpRequest. Let me know if you need any clarification on this.
I would use a decorator here. Cut straight to the code...
import logging
from functools import wraps
from django.http import HttpResponse, HttpRequest
logging.basicConfig(filename="errors.log",
level=logging.ERROR,
format='%(asctime)s: %(message)s')
log = logging.getLogger(__name__)
def log_exceptions(wrapped):
#wraps(wrapped)
def wrapper(*args, **kwargs):
try:
return wrapped(*args, **kwargs)
except:
# log and re-raise
request = args[0] if len(args) > 0 and isinstance(args[0], HttpRequest) else None
msg = ("\nuser.id/email: {}/{}\nMETA: {}...\nbody: {}"
.format(request.user.id,
getattr(request.user, 'email','?'),
str(request.META)[:80],
request.body)
if request
else "not a HttpRequest")
log.exception(msg)
raise
return wrapper
#log_exceptions
def random_view(request):
raise ValueError("simulate a crash")
if request.user.is_authenticated() and request.user.is_active:
return HttpResponse('hi')
# generic view code goes here.
else:
return HttpResponse(status=401)
and errors.log should capture something like
2017-06-27 20:48:09,282:
user.id/email: 1/test#domain.com
META: {'SESSION_MANAGER': 'local/acb:#/tmp/.ICE-unix/4255,unix/acb:/tmp/.ICE-unix/4255...
body: b''
Traceback (most recent call last):
File "/home/rod/pyves/rangetest/rangetest/data/views.py", line 14, in wrapper
return wrapped(*args, **kwargs)
File "/home/rod/pyves/rangetest/rangetest/data/views.py", line 31, in random_view
raise ValueError("simulate a crash")
ValueError: simulate a crash
Note, you'll also probably see the Django crash logging in your errors.log as well. You might split the logs to separate files using Django's well documented, but nonetheless complex logging config

Wagtail: Serializing page model

I am using wagtail as a REST backend for a website. The website is built using react and fetches data via wagtails API v2.
The SPA website needs to be able to show previews of pages in wagtail. My thought was to override serve_preview on the page model and simply seralize the new page as JSON and write it to a cache which could be accessed by my frontend. But im having trouble serializing my page to json. All attempts made feel very "hackish"
I've made several attempts using extentions of wagtails built in serializers but without success:
Atempt 1:
def serve_preview(self, request, mode_name):
from wagtail.api.v2.endpoints import PagesAPIEndpoint
endpoint = PagesAPIEndpoint()
setattr(request, 'wagtailapi_router',
WagtailAPIRouter('wagtailapi_v2'))
endpoint.request = request
endpoint.action = None
endpoint.kwargs = {'slug': self.slug, 'pk': self.pk}
endpoint.lookup_field = 'pk'
serializer = endpoint.get_serializer(self)
Feels very ugly to use router here and set a bunch of attrs
Attempt 2:
def serve_preview(self, request, mode_name):
from wagtail.api.v2.endpoints import PagesAPIEndpoint
fields = PagesAPIEndpoint.get_available_fields(self)
if hasattr(self, 'api_fields'):
fields.extend(self.api_fields)
serializer_class = get_serializer_class(
type(self), fields, meta_fields=[PagesAPIEndpoint.meta_fields], base=PageSerializer)
serializer = serializer_class(self)
Better but i get context issues:
Traceback (most recent call last):
...
File "/usr/local/lib/python3.5/site-packages/wagtail/api/v2/serializers.py", line 92, in to_representation
self.context['view'].seen_types[name] = page.specific_class
KeyError: 'view'
Any toughts?
Solved it by diving through the source code.
First define an empty dummy view:
class DummyView(GenericViewSet):
def __init__(self, *args, **kwargs):
super(DummyView, self).__init__(*args, **kwargs)
# seen_types is a mapping of type name strings (format: "app_label.ModelName")
# to model classes. When an object is serialised in the API, its model
# is added to this mapping. This is used by the Admin API which appends a
# summary of the used types to the response.
self.seen_types = OrderedDict()
Then use this view and set the context of your serializer manually. Im also using the same router as in my api in my context. It has methods which are called by the PageSerializer to resolve some fields. Kinda strange it is so tightly coupled with the wagtail api but at least this works:
def serve_preview(self, request, mode_name):
import starrepublic.api as StarApi
fields = StarApi.PagesAPIEndpoint.get_available_fields(self)
if hasattr(self, 'api_fields'):
fields.extend(self.api_fields)
serializer_class = get_serializer_class(
type(self), fields, meta_fields=[StarApi.PagesAPIEndpoint.meta_fields], base=PageSerializer)
serializer = serializer_class(
self, context={'request': request, 'view': DummyView(), 'router': StarApi.api_router})
Dont forget to import:
from wagtail.api.v2.serializers import get_serializer_class
from rest_framework.viewsets import GenericViewSet
from rest_framework import status
from rest_framework.response import Response
from django.http import JsonResponse
from django.http import HttpResponse
Possibly a non-answer answer, but I too have had challenges in the area of DRF, Wagtail's layering on top of DRF, and the need to cache json results (DRF has no built-in caching as far as I can tell, so that's an additional challenge). In a recent project, I ended up just building a list of dictionaries in a view and sending them back out with HttpResponse(), bypassing DRF and Wagtail API altogether. The code ended up simple, readable, and was easy to cache:
import json
from django.http import HttpResponse
from django.core.cache import cache
data = cache.get('mydata')
if not data:
datalist = []
for foo in bar:
somedata = {}
# Populate somedata, "serializing" fields manually...
datalist.append(somedata)
# Cache for a week.
data = datalist
cache.set('mydata', datalist, 60 * 60 * 24 * 7)
return HttpResponse(json.dumps(data), content_type='application/json')
Not as elegant as using the pre-built REST framework, but sometimes the simpler approach is just more productive...

Django and Middleware which uses request.user is always Anonymous

I'm trying to make middleware which alters some fields for the user based on subdomain, etc...
The only problem is the request.user always comes in as AnonymousUser within the middleware, but is then the correct user within the views. I've left the default authentication and session middleware django uses within the settings.
There is a similar question here: Django, request.user is always Anonymous User
But doesn't overly answer the total question because I'm not using different authentication methods, and djangos authentication is running before I invoke my own middleware.
Is there a way, while using DRF, to get the request.user within the middleware? I'll show some sample code here:
class SampleMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
#This will be AnonymousUser. I need it to be the actual user making the request.
print (request.user)
def process_response(self, request, response):
return response
with process_request:
class SampleMiddleware(object):
def process_request(self, request):
#This will be AnonymousUser. I need it to be the actual user making the request.
print (request.user)
def process_response(self, request, response):
return response
I've solved this problem by getting DRF token from the requests and loading request.user to the user associated to that model.
I had the default django authentication and session middleware, but it seems DRF was using it's token auth after middleware to resolve the user (All requests were CORS requests, this might have been why). Here's my updated middleware class:
from re import sub
from rest_framework.authtoken.models import Token
from core.models import OrganizationRole, Organization, User
class OrganizationMiddleware(object):
def process_view(self, request, view_func, view_args, view_kwargs):
header_token = request.META.get('HTTP_AUTHORIZATION', None)
if header_token is not None:
try:
token = sub('Token ', '', header_token)
token_obj = Token.objects.get(key = token)
request.user = token_obj.user
except Token.DoesNotExist:
pass
#This is now the correct user
print (request.user)
This can be used on process_view or process_request as well.
Hopefully this can help someone out in the future.
Came across this today while having the same problem.
TL;DR;
Skip below for code example
Explanation
Thing is DRF have their own flow of things, right in the middle of the django request life-cycle.
So if the normal middleware flow is :
request_middleware (before starting to work on the request)
view_middleware (before calling the view)
template_middleware (before render)
response_middleware (before final response)
DRF code, overrides the default django view code, and executes their own code.
In the above link, you can see that they wrap the original request with their own methods, where one of those methods is DRF authentication.
So back to your question, this is the reason using request.user in a middleware is premature, as it only gets it's value after view_middleware** executes.
The solution I went with, is having my middleware set a LazyObject.
This helps, because my code (the actual DRF ApiVIew) executes when the actual user is already set by DRF's authentication.
This solution was proposed here together with a discussion.
Might have been better if DRF had a better way to extend their functionality, but as things are, this seems better than the provided solution (both performance and readability wise).
Code Example
from django.utils.functional import SimpleLazyObject
def get_actual_value(request):
if request.user is None:
return None
return request.user #here should have value, so any code using request.user will work
class MyCustomMiddleware(object):
def process_request(self, request):
request.custom_prop = SimpleLazyObject(lambda: get_actual_value(request))
The accepted answer only takes TokenAuthentication in consideration - in my case, there are more authentication methods configured. I thus went with initializing the DRF's Request directly which invokes DRF's authentication machinery and loops through all configured authentication methods.
Unfortunately, it still introduces an additional load on the database since the Token object must be queried (the accepted answer has this problem as well). The trick with SimpleLazyObject in this answer is a much better solution, but it didn't work for my use case because I need the user info in the middleware directly - I'm extending the metrics in django_prometheus and it processes the request before get_response is called.
from rest_framework.request import Request as RestFrameworkRequest
from rest_framework.views import APIView
class MyMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
drf_request: RestFrameworkRequest = APIView().initialize_request(request)
user = drf_request.user
...
return self.get_response(request)
Based on Daniel Dubovski's very elegant solution above, here's an example of middleware for Django 1.11:
from django.utils.functional import SimpleLazyObject
from organization.models import OrganizationMember
from django.core.exceptions import ObjectDoesNotExist
def get_active_member(request):
try:
active_member = OrganizationMember.objects.get(user=request.user)
except (ObjectDoesNotExist, TypeError):
active_member = None
return active_member
class OrganizationMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
request.active_member = SimpleLazyObject(lambda: get_active_member(request))
response = self.get_response(request)
# Code to be executed for each request/response after
# the view is called.
return response
Daniel Dubovski's solution is probably the best in most cases.
The problem with the lazy object approach is if you need to rely on the side effects. In my case, I need something to happen for each request, no matter what.
If I'd use a special value like request.custom_prop, it has to be evaluated for each request for the side effects to happen. I noticed that other people are setting request.user, but it doesn't work for me since some middleware or authentication class overwrites this property.
What if DRF supported its own middleware? Where could I plug it in? The easiest way in my case (I don't need to access the request object, only the authenticated user) seems to be to hook into the authentication class itself:
from rest_framework.authentication import TokenAuthentication
class TokenAuthenticationWithSideffects(TokenAuthentication):
def authenticate(self, request):
user_auth_tuple = super().authenticate(request)
if user_auth_tuple is None:
return
(user, token) = user_auth_tuple
# Do stuff with the user here!
return (user, token)
Then I could just replace this line in my settings:
REST_FRAMEWORK = {
"DEFAULT_AUTHENTICATION_CLASSES": (
#"rest_framework.authentication.TokenAuthentication",
"my_project.authentication.TokenAuthenticationWithSideffects",
),
# ...
}
I'm not promoting this solution, but maybe it will help someone else.
Pros:
It to solves this specific problem
There's no double authentication
Easy to maintain
Cons:
Not tested in production
Things happen in an unexpected place
Side effects...
I know it's not exactly answering the 'can we access that from the middleware' question, but I think it's a more elegant solution VS doing the same work in the middleware VS what DRJ does in its base view class. At least for what I needed, it made more sense to add here.
Basically, I'm just overriding the method 'perform_authentication()' from DRF's code, since I needed to add more things related to the current user in the request. The method just originally call 'request.user'.
class MyGenericViewset(viewsets.GenericViewSet):
def perform_authentication(self, request):
request.user
if request.user and request.user.is_authenticated():
request.my_param1 = 'whatever'
After that in your own views, instead of settings APIView from DRF as a parent class, simply set that class as a parent.
I wasn't quite happy with the solutions out there. Here's a solution that uses some DRF internals to make sure that the correct authentication is applied in the middleware, even if the view has specific permissions classes. It uses the middleware hook process_view which gives us access to the view we're about to hit:
class CustomTenantMiddleware():
def process_view(self, request, view_func, view_args, view_kwargs):
# DRF saves the class of the view function as the .cls property
view_class = view_func.cls
try:
# We need to instantiate the class
view = view_class()
# And give it an action_map. It's not relevant for us, but otherwise it errors.
view.action_map = {}
# Here's our fully formed and authenticated (or not, depending on credentials) request
request = view.initialize_request(request)
except (AttributeError, TypeError):
# Can't initialize the request from this view. Fallback to using default permission classes
request = APIView().initialize_request(request)
# Here the request is fully formed, with the correct permissions depending on the view.
Note that this doesn't avoid having to authenticate twice. DRF will still happily authenticate right afterwards.
I had the same issue and decided to change my design. Instead of using a Middleware I simply monkey-patch rest_framework.views.APIView.
In my case I needed to patch check_permissions but you can patch whatever fits your problem. Have a look at the the source code.
settings.py
INSTALLED_APPS = [
..
'myapp',
]
myapp/patching.py
import sys
from rest_framework.views import APIView as OriginalAPIView
class PatchedAPIView(OriginalAPIView):
def check_permissions(self, request):
print(f"We should do something with user {request.user}"
return OriginalAPIView.check_permissions(self, request)
# We replace the Django REST view with our patched one
sys.modules['rest_framework'].views.APIView = PatchedAPIView
myapp/__init__.py
from .patching import *

Categories