I upgraded from Django 1.10.4 to 1.11.1 and all of a sudden I'm getting a ton of these messages when I run my tests:
lib/python3.5/site-packages/rest_framework/pagination.py:208:
UnorderedObjectListWarning:
Pagination may yield inconsistent results with an unordered object_list:
<QuerySet [<Group: Requester>]>
paginator = self.django_paginator_class(queryset, page_size)
I've traced that back to the Django Pagination module:
https://github.com/django/django/blob/master/django/core/paginator.py#L100
It seems to be related to my queryset code:
return get_user_model().objects.filter(id=self.request.user.id)
How can I find more details on this warning? It seems to be that I need to add a order_by(id) on the end of every filter, but I can't seem to find which code needs the order_by added (because the warning doesn't return a stack trace and so it happens randomly during my test run).
Thanks!
Edit:
So by using #KlausD. verbosity tip, I looked at a test causing this error:
response = self.client.get('/api/orders/')
This goes to OrderViewSet but none of the things in get_queryset cause it and nothing in serializer class causes it. I have other tests that use the same code to get /api/orders and those don't cause it.... What does DRF do after get_queryset?
https://github.com/encode/django-rest-framework/blob/master/rest_framework/pagination.py#L166
If I put a traceback into pagination then I get a whole bunch of stuff related to django rest framework but nothing that points back to which of my queries is triggering the order warning.
So in order to fix this I had to find all of the all, offset, filter, and limit clauses and add a order_by clause to them. Some I fixed by adding a default ordering:
class Meta:
ordering = ['-id']
In the ViewSets for Django Rest Framework (app/apiviews.py) I had to update all of the get_queryset methods as adding a default ordering didn't seem to work.
I was getting this warning when i used objects.all() in my view.py
profile_list = Profile.objects.all()
paginator = Paginator(profile_list, 25)
to fix this i changed my code to :
profile_list = Profile.objects.get_queryset().order_by('id')
paginator = Paginator(profile_list, 25)
In my case, I had to add order_by('id') instead of ordering.
class IntakeCaseViewSet(viewsets.ModelViewSet):
schema = None
queryset = IntakeCase.objects.all().order_by('id')
Ordering needs to be in the model using Class Meta (not View).
Let me give an answer updated to new developments...
https://code.djangoproject.com/ticket/6089
The default ordering of the User model has been removed in Django. If you found yourself at this page because of an upgrade, it's very likely connected to this change.
There are 2 versions of this problem you might be dealing with.
your own model does not have a default ordering in its Meta (see accepted answer)
you are using a model from an app you are using as a dependency which does not have a default ordering
Since literally the Django User model itself does not adhere to ordering, it's very clear that the second scenario cannot be resolved by asking the maintainers of those dependencies to put in a default ordering. Okay, so now you either have to override the model being used for whatever your doing (sometimes a good idea, but not good for addressing such a minor issue).
So you're left with addressing it on the view level. You also want to do something that will play nicely with any ordering filter class you have applied. For that, set the view's ordering parameter.
class Reviewers(ListView):
model = User
paginate_by = 50
ordering = ['username']
Also see Is there Django List View model sort?
Another option is to add OrderingFilter
http://www.django-rest-framework.org/api-guide/filtering/#orderingfilter
Including this didn't work for me.
class Meta:
ordering = ['-id']
But changing get_queryset(self) and sorting the list with .order_by('id') did. Maybe worked because I'm using filters, I don't know
class MyView(viewsets.ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MySerializerSerializer
def get_queryset(self):
user = self.request.user
return MyModel.objects.filter(owner=user).order_by('id')
Update the model meta class instead.
class UsefulModel(models.Model):
class Meta:
ordering='-created' # for example
you can still override the ordering from the view attribute 'ordering' As advised by AlanSE previously.
class UsefulView(ListView):
ordering = ['-created']
In my case, it expected a tuple and that tuple has to contain a comma even when you are parsing just an item in it.
class Meta:
ordering = ('-name',)
Related
I want to add an extra field to a query set in Django.
The field does not exist in the model but I want to add it to the query set.
Basically I want to add an extra field called "upcoming" which should return "True"
I already tried adding a #property method to my model class. This does not work because apparently django queries access the DB directly.
models.py
class upcomingActivity(models.Model):
title = models.CharField (max_length=150)
address = models.CharField (max_length=150)
Views.py
def get(self, request):
query = upcomingActivity.objects.all()
feature_collection = serialize('geojson', query ,
geometry_field='location',
fields= ( 'upcoming','title','address','pk' )
)
This answer is for the case that you do not want to add a virtual property to the model (the model remains as is).
To add an additional field to the queryset result objects:
from django.db.models import BooleanField, Value
upcomingActivity.objects.annotate(upcoming=Value(True, output_field=BooleanField())).all()
Each object in the resulting queryset will have the attribute upcoming with the boolean value True.
(Performance should be nice because this is easy work for the DB, and Django/Python does not need to do much additional work.)
EDIT after comment by Juan Carlos:
The Django serializer is for serializing model objects and thus will not serialize any non-model fields by default (because basically, the serializer in this case is for loading/dumping DB data).
See https://docs.djangoproject.com/en/2.2/topics/serialization/
Django’s serialization framework provides a mechanism for “translating” Django models into other formats.
See also: Django - How can you include annotated results in a serialized QuerySet?
From my own experience:
In most cases, we are using json.dumps for serialization in views and this works like a charm. You can prepare the data very flexibly for whatever needs arize, either by annotations or by processing the data (list comprehension etc.) before dumping it via json.
Another possibility in your situation could be to customize the serializer or the input to the serializer after fetching the data from the DB.
You can use a class function to return the upcoming value like this:
def upcoming(self):
is_upcoming = # some logic query or just basically set it to true.
return is_upcoming
then call it normally in your serializer the way you did it.
fields= ( 'upcoming','title','address','pk' )
Using Django 11 with PostgreSQL db. I have the models as shown below. I'm trying to prefetch a related queryset, using the Prefetch object and prefetch_related without assigning it to an attribute.
class Person(Model):
name = Charfield()
#property
def latest_photo(self):
return self.photos.order_by('created_at')[-1]
class Photo(Model):
person = ForeignKey(Person, related_name='photos')
created_at = models.DateTimeField(auto_now_add=True)
first_person = Person.objects.prefetch_related(Prefetch('photos', queryset=Photo.objects.order_by('created_at'))).first()
first_person.photos.order_by('created_at') # still hits the database
first_person.latest_photo # still hits the database
In the ideal case, calling person.latest_photo will not hit the database again. This will allow me to use that property safely in a list display.
However, as noted in the comments in the code, the prefetched queryset is not being used when I try to get the latest photo. Why is that?
Note: I've tried using the to_attr argument of Prefetch and that seems to work, however, it's not ideal since it means I would have to edit latest_photo to try to use the prefetched attribute.
The problem is with slicing, it creates a different query.
You can work around it like this:
...
#property
def latest_photo(self):
first_use_the_prefetch = list(self.photos.order_by('created_at'))
then_slice = first_use_the_prefetch[-1]
return then_slice
And in case you want to try, it is not possible to use slicing inside the Prefetch(query=...no slicing here...) (there is a wontfix feature request for this somewhere in Django tracker).
I'm creating filters for my models in Django Rest Framework. This is my filters.py.
class EmployeeListView(generics.ListAPIView):
queryset = Employee.objects.all()
serializer = EmployeeSerializer
filter_backends = (filters.DjangoFilterBackend,)
filter_fields = ('id', 'employee_type', 'rfid_tag')
search_fields = ('event_type', 'employee_status')
ordering_fields = ('id', 'employee_type')
In the DRF documentation they have examples that look like this:
I've checked my syntax in the documentation examples and everything looks right, but mine looks like this:
what do I need to do get the search and ordering options I've set to display like in the examples in the docs? Any help is greatly appreciated. :)
NOE (New Operator Error)
I made a mistake in the place of my filter settings first of all. Moved them to proper file (views.py). I also realized I was combining to different kinds of filter settings. Using DjangoFilterBackends - by setting the filter_fields - automatically creates the types of filters.
I had combined the SearchFilters setting and the OrderingFiltering setting.
Now I have this:
Problem solved (and lesson learned).
I have a simple model which includes a product and category table. The Product model has a foreign key Category.
When I make a tastypie API call that returns a list of categories /api/vi/categories/
I would like to add a field that determines the "product count" / the number of products that have a giving category. The result would be something like:
category_objects[
{
id: 53
name: Laptops
product_count: 7
},
...
]
The following code is working but the hit on my DB is heavy
def dehydrate(self, bundle):
category = Category.objects.get(pk=bundle.obj.id)
products = Product.objects.filter(category=category)
bundle.data['product_count'] = products.count()
return bundle
Is there a more efficient way to build this query? Perhaps with annotate ?
You can use prefetch_related method of QuerSet to reverse select_related.
Asper documentation,
prefetch_related(*lookups)
Returns a QuerySet that will automatically
retrieve, in a single batch, related objects for each of the specified
lookups.
This has a similar purpose to select_related, in that both are
designed to stop the deluge of database queries that is caused by
accessing related objects, but the strategy is quite different.
If you change your dehydrate function to following then database will be hit single time.
def dehydrate(self, bundle):
category = Category.objects.prefetch_related("product_set").get(pk=bundle.obj.id)
bundle.data['product_count'] = category.product_set.count()
return bundle
UPDATE 1
You should not initialize queryset inside dehydrate function. queryset should be always set in Meta class only. Please have a look at following example from django-tastypie documentation.
class MyResource(ModelResource):
class Meta:
queryset = User.objects.all()
excludes = ['email', 'password', 'is_staff', 'is_superuser']
def dehydrate(self, bundle):
# If they're requesting their own record, add in their email address.
if bundle.request.user.pk == bundle.obj.pk:
# Note that there isn't an ``email`` field on the ``Resource``.
# By this time, it doesn't matter, as the built data will no
# longer be checked against the fields on the ``Resource``.
bundle.data['email'] = bundle.obj.email
return bundle
As per official django-tastypie documentation on dehydrate() function,
dehydrate
The dehydrate method takes a now fully-populated bundle.data & make
any last alterations to it. This is useful for when a piece of data
might depend on more than one field, if you want to shove in extra
data that isn’t worth having its own field or if you want to
dynamically remove things from the data to be returned.
dehydrate() is only meant to make any last alterations to bundle.data.
Your code does additional count query for each category. You're right about annotate being helpfull in this kind of a problem.
Django will include all queryset's fields in GROUP BY statement. Notice .values() and empty .group_by() serve limiting field set to required fields.
cat_to_prod_count = dict(Product.objects
.values('category_id')
.order_by()
.annotate(product_count=Count('id'))
.values_list('category_id', 'product_count'))
The above dict object is a map [category_id -> product_count].
It can be used in dehydrate method:
bundle.data['product_count'] = cat_to_prod_count[bundle.obj.id]
If that doesn't help, try to keep similar counter on category records and use singals to keep it up to date.
Note categories are usually a tree-like beings and you probably want to keep count of all subcategories as well.
In that case look at the package django-mptt.
I've looked through Tastypie's documentation and searched for a while, but can't seem to find an answer to this.
Let's say that we've got two models: Student and Assignment, with a one-to-many relationship between them. The Assignment model includes an assignment_date field. Basically, I'd like to build an API using Tastypie that returns Student objects sorted by most recent assignment date. Whether the sorting is done on the server or in the client side doesn't matter - but wherever the sorting is done, the assignment_date is needed to sort by.
Idea #1: just return the assignments along with the students.
class StudentResource(ModelResource):
assignments = fields.OneToManyField(
AssignmentResource, 'assignments', full=True)
class Meta:
queryset = models.Student.objects.all()
resource_name = 'student'
Unfortunately, each student may have tens or hundreds of assignments, so this is bloated and unnecessary.
Idea #2: augment the data during the dehydrate cycle.
class StudentResource(ModelResource):
class Meta:
queryset = models.Student.objects.all()
resource_name = 'student'
def dehydrate(self, bundle):
bundle.data['last_assignment_date'] = (models.Assignment
.filter(student=bundle.data['id'])
.order_by('assignment_date')[0].assignment_date)
This is not ideal, since it'll be performing a separate database roundtrip for each student record. It's also not very declarative, nor elegant.
So, is there a good way to get this kind of functionality with Tastypie? Or is there a better way to do what I'm trying to achieve?
You can sort a ModelResource by a field name. Check out this part of the documentation http://django-tastypie.readthedocs.org/en/latest/resources.html#ordering
You could also set this ordering by default in the Model: https://docs.djangoproject.com/en/dev/ref/models/options/#ordering