We Keep Coding

Add a virtual field to a django query

I want to add an extra field to a query set in Django.
The field does not exist in the model but I want to add it to the query set.
Basically I want to add an extra field called "upcoming" which should return "True"
I already tried adding a #property method to my model class. This does not work because apparently django queries access the DB directly.
models.py
class upcomingActivity(models.Model):
title = models.CharField (max_length=150)
address = models.CharField (max_length=150)
Views.py
def get(self, request):
query = upcomingActivity.objects.all()
feature_collection = serialize('geojson', query ,
geometry_field='location',
fields= ( 'upcoming','title','address','pk' )
)

This answer is for the case that you do not want to add a virtual property to the model (the model remains as is).
To add an additional field to the queryset result objects:
from django.db.models import BooleanField, Value
upcomingActivity.objects.annotate(upcoming=Value(True, output_field=BooleanField())).all()
Each object in the resulting queryset will have the attribute upcoming with the boolean value True.
(Performance should be nice because this is easy work for the DB, and Django/Python does not need to do much additional work.)
EDIT after comment by Juan Carlos:
The Django serializer is for serializing model objects and thus will not serialize any non-model fields by default (because basically, the serializer in this case is for loading/dumping DB data).
See https://docs.djangoproject.com/en/2.2/topics/serialization/
Django’s serialization framework provides a mechanism for “translating” Django models into other formats.
See also: Django - How can you include annotated results in a serialized QuerySet?
From my own experience:
In most cases, we are using json.dumps for serialization in views and this works like a charm. You can prepare the data very flexibly for whatever needs arize, either by annotations or by processing the data (list comprehension etc.) before dumping it via json.
Another possibility in your situation could be to customize the serializer or the input to the serializer after fetching the data from the DB.

You can use a class function to return the upcoming value like this:
def upcoming(self):
is_upcoming = # some logic query or just basically set it to true.
return is_upcoming
then call it normally in your serializer the way you did it.
fields= ( 'upcoming','title','address','pk' )

Python Django Rest Framework UnorderedObjectListWarning

I upgraded from Django 1.10.4 to 1.11.1 and all of a sudden I'm getting a ton of these messages when I run my tests:
lib/python3.5/site-packages/rest_framework/pagination.py:208:
UnorderedObjectListWarning:
Pagination may yield inconsistent results with an unordered object_list:
<QuerySet [<Group: Requester>]>
paginator = self.django_paginator_class(queryset, page_size)
I've traced that back to the Django Pagination module:
https://github.com/django/django/blob/master/django/core/paginator.py#L100
It seems to be related to my queryset code:
return get_user_model().objects.filter(id=self.request.user.id)
How can I find more details on this warning? It seems to be that I need to add a order_by(id) on the end of every filter, but I can't seem to find which code needs the order_by added (because the warning doesn't return a stack trace and so it happens randomly during my test run).
Thanks!
Edit:
So by using #KlausD. verbosity tip, I looked at a test causing this error:
response = self.client.get('/api/orders/')
This goes to OrderViewSet but none of the things in get_queryset cause it and nothing in serializer class causes it. I have other tests that use the same code to get /api/orders and those don't cause it.... What does DRF do after get_queryset?
https://github.com/encode/django-rest-framework/blob/master/rest_framework/pagination.py#L166
If I put a traceback into pagination then I get a whole bunch of stuff related to django rest framework but nothing that points back to which of my queries is triggering the order warning.

So in order to fix this I had to find all of the all, offset, filter, and limit clauses and add a order_by clause to them. Some I fixed by adding a default ordering:
class Meta:
ordering = ['-id']
In the ViewSets for Django Rest Framework (app/apiviews.py) I had to update all of the get_queryset methods as adding a default ordering didn't seem to work.

I was getting this warning when i used objects.all() in my view.py
profile_list = Profile.objects.all()
paginator = Paginator(profile_list, 25)
to fix this i changed my code to :
profile_list = Profile.objects.get_queryset().order_by('id')
paginator = Paginator(profile_list, 25)

In my case, I had to add order_by('id') instead of ordering.
class IntakeCaseViewSet(viewsets.ModelViewSet):
schema = None
queryset = IntakeCase.objects.all().order_by('id')
Ordering needs to be in the model using Class Meta (not View).

Let me give an answer updated to new developments...
https://code.djangoproject.com/ticket/6089
The default ordering of the User model has been removed in Django. If you found yourself at this page because of an upgrade, it's very likely connected to this change.
There are 2 versions of this problem you might be dealing with.
your own model does not have a default ordering in its Meta (see accepted answer)
you are using a model from an app you are using as a dependency which does not have a default ordering
Since literally the Django User model itself does not adhere to ordering, it's very clear that the second scenario cannot be resolved by asking the maintainers of those dependencies to put in a default ordering. Okay, so now you either have to override the model being used for whatever your doing (sometimes a good idea, but not good for addressing such a minor issue).
So you're left with addressing it on the view level. You also want to do something that will play nicely with any ordering filter class you have applied. For that, set the view's ordering parameter.
class Reviewers(ListView):
model = User
paginate_by = 50
ordering = ['username']
Also see Is there Django List View model sort?

Another option is to add OrderingFilter
http://www.django-rest-framework.org/api-guide/filtering/#orderingfilter

Including this didn't work for me.
class Meta:
ordering = ['-id']
But changing get_queryset(self) and sorting the list with .order_by('id') did. Maybe worked because I'm using filters, I don't know
class MyView(viewsets.ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MySerializerSerializer
def get_queryset(self):
user = self.request.user
return MyModel.objects.filter(owner=user).order_by('id')

Update the model meta class instead.
class UsefulModel(models.Model):
class Meta:
ordering='-created' # for example
you can still override the ordering from the view attribute 'ordering' As advised by AlanSE previously.
class UsefulView(ListView):
ordering = ['-created']

In my case, it expected a tuple and that tuple has to contain a comma even when you are parsing just an item in it.
class Meta:
ordering = ('-name',)

How to add a static User field to django query

I need to perform some special serialization / deserialization with DRF, but one of the required parameters for those is the user that generated the query.
AFAIK a serializers.Field instance does not have access to ViewSet and thus no access to ViewSet.request.user.
So I thought I'd just add the user as a static field into the queryset so that each record would have access to it.
qry = qry.annotate(user=Value(user, models.ForeignKey(settings.AUTH_USER_MODEL)))
However, this gives me
ValueError: Related model 'auth.User' cannot be resolved
I also tried
q.annotate(user=Value(user, models.ForeignKey(user.__class__)))
but that also excepts.
Exactly what do I have to include so that this will resolve as needed?

Have a look at the CurrentUserDefault:
A default class that can be used to represent the current user. In order to use this, the 'request' must have been provided as part of the context dictionary when instantiating the serializer.
owner = serializers.HiddenField(
default=serializers.CurrentUserDefault()
)
To use this, you need to pass the request in the context as following:
serializer = AccountSerializer(account, context={'request': request})

Override serializer.data in Django REST Framework

I've been trying to alter the value of a form field the Django REST Framework's admin panel and for some reason the change never takes place. I have the serializer below
class SomeView(ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MyModelSerializer
# I Want to override this and change the POST data
def perform_create(self, serializer):
user = self.request.user.id
# this was a form field where I manually entered the user ID
# I now want it to default to the logged in user
serializer.data['user'] = user
# This returns the original user that was entered into the form field
print serializer.data
I checked out serializer.data with dir() and it's just a python dictionary so I can't figure out why I can't modify the value. As a test, I tried to add extra values but that doesn't work either
# this doesnt work
serializer.data['some_new_field'] = 'test'
EDIT
On another note, I can copy the data and edit it
fake_data = serializer.data.copy()
fake_data['old_value'] = 'new value'
However, it always fails to validate
serializer = MyModelSerializer(data=fake_data)
serializer.is_valid() # returns false
EDIT EDIT:
Ok, so the validation error was caused by Django returning a SimpleLazyObject. Everything works now when I perform a copy of the data, but I'm really curious as to why I can't edit serializer.data directly without copying it. The problem is solved now, but if anyone can provide insight on the issue just for curiosity, that would be awesome.

I checked out serializer.data with dir() and it's just a python dictionary so I can't figure out why I can't modify the value.
While the value returned from Serializer.data is indeed a dictionary, Serializer.data is not a simple instance variable.
If you look at rest_framework/serializers.py:
class Serializer(BaseSerializer, metaclass=SerializerMetaclass):
# [...]
#property
def data(self):
ret = super().data
return ReturnDict(ret, serializer=self)
ReturnDict inherits from OrderedDict, but you still get a new dictionary every time you access Serializer.data.
The real data is in _data, however as noted by the underscore you might not want to modify that either as it is not intended to be public. The values are filled by Serializer.to_representation() which you could override on the viewset.
As for the second part: ModelViewSet defines get_serializer() that is called with the request POST data to create the serializer you want to modify. I'd suggest try to change the input data before the serializer is created, instead.

Tastypie Dehydrate reverse relation count

I have a simple model which includes a product and category table. The Product model has a foreign key Category.
When I make a tastypie API call that returns a list of categories /api/vi/categories/
I would like to add a field that determines the "product count" / the number of products that have a giving category. The result would be something like:
category_objects[
{
id: 53
name: Laptops
product_count: 7
},
...
]
The following code is working but the hit on my DB is heavy
def dehydrate(self, bundle):
category = Category.objects.get(pk=bundle.obj.id)
products = Product.objects.filter(category=category)
bundle.data['product_count'] = products.count()
return bundle
Is there a more efficient way to build this query? Perhaps with annotate ?

You can use prefetch_related method of QuerSet to reverse select_related.
Asper documentation,
prefetch_related(*lookups)
Returns a QuerySet that will automatically
retrieve, in a single batch, related objects for each of the specified
lookups.
This has a similar purpose to select_related, in that both are
designed to stop the deluge of database queries that is caused by
accessing related objects, but the strategy is quite different.
If you change your dehydrate function to following then database will be hit single time.
def dehydrate(self, bundle):
category = Category.objects.prefetch_related("product_set").get(pk=bundle.obj.id)
bundle.data['product_count'] = category.product_set.count()
return bundle
UPDATE 1
You should not initialize queryset inside dehydrate function. queryset should be always set in Meta class only. Please have a look at following example from django-tastypie documentation.
class MyResource(ModelResource):
class Meta:
queryset = User.objects.all()
excludes = ['email', 'password', 'is_staff', 'is_superuser']
def dehydrate(self, bundle):
# If they're requesting their own record, add in their email address.
if bundle.request.user.pk == bundle.obj.pk:
# Note that there isn't an ``email`` field on the ``Resource``.
# By this time, it doesn't matter, as the built data will no
# longer be checked against the fields on the ``Resource``.
bundle.data['email'] = bundle.obj.email
return bundle
As per official django-tastypie documentation on dehydrate() function,
dehydrate
The dehydrate method takes a now fully-populated bundle.data & make
any last alterations to it. This is useful for when a piece of data
might depend on more than one field, if you want to shove in extra
data that isn’t worth having its own field or if you want to
dynamically remove things from the data to be returned.
dehydrate() is only meant to make any last alterations to bundle.data.

Your code does additional count query for each category. You're right about annotate being helpfull in this kind of a problem.
Django will include all queryset's fields in GROUP BY statement. Notice .values() and empty .group_by() serve limiting field set to required fields.
cat_to_prod_count = dict(Product.objects
.values('category_id')
.order_by()
.annotate(product_count=Count('id'))
.values_list('category_id', 'product_count'))
The above dict object is a map [category_id -> product_count].
It can be used in dehydrate method:
bundle.data['product_count'] = cat_to_prod_count[bundle.obj.id]
If that doesn't help, try to keep similar counter on category records and use singals to keep it up to date.
Note categories are usually a tree-like beings and you probably want to keep count of all subcategories as well.
In that case look at the package django-mptt.