We Keep Coding

Django filter relation field set

I have two models, here is simplified example of it:
class Application(TimestampedModel):
...
forms = models.ManyToManyField(Form, related_name='applications', through='ApplicationForm', blank=True)
class ApplicationForm(models.Model):
application = models.ForeignKey(Application, on_delete=models.CASCADE)
form = models.ForeignKey(Form, on_delete=models.CASCADE)
created_at = models.DateTimeField()
updated_at = models.DateTimeField(auto_now=True)
class Meta:
ordering = ['-created_at']
I want to filter forms field on Application model. I try to do this:
queryset = Application.objects.get_active().filter(is_public=True, pk=self.kwargs['pk'])
for application in queryset:
forms = application.forms.filter(form_sections__form_fields__pk__in=application.public_form_fields.all())
application.forms.set(forms)
But I get an error:
AttributeError at /api/applications/public/79
Cannot set values on a ManyToManyField which specifies an intermediary model. Use applications.ApplicationForm's Manager instead.
So my question is it possible, and if possible how can I do this?

You're using an intermediary model, you should read this. You can only use set() if you provide defaults for the extra fields (in your case created_at). Otherwise, you have to create the ApplicationForm objects yourself.
So this would probably work in Django 2.2:
application.forms.set(forms, through_defaults={'created_at': timezone.now()})
In earlier versions of Django you have to go through the intermediate model:
application.forms.clear() # remove all existing relations
for form in forms:
ApplicationForm.objects.create(application=application, form=form, created_at=timezone.now())

Badly affecting performance in populating ManyToMany field values in rest api (using django rest framework)

As I'm using django rest framework for building my product api.
Here is my model in models.py
class Tag(models.Model):
tag = models.CharField(max_length=10, unique=True)
class Product(models.Model):
product_name = models.CharField(max_length=100)
tag = models.ManyToManyField(Tag, blank=True, default=None, related_name='product_tag')
serializers.py :
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
fields = '__all__'
class ProductSerializer(serializers.HyperlinkedModelSerializer):
tag = TagSerializer(many=True, read_only=True)
class Meta:
model = Product
fields = '__all__'
views.py :
class ProductViewSet(viewsets.ModelViewSet):
queryset = Product.objects.all()
serializer_class = ProductSerializer
I have given the url for ProductViewset, so when I hit the api it gives me the results as well but it takes too much time to load, it takes around 2 minutes to give me the response.
I'm having 2000 product objects in database which needs to be populated.
When I exclude the 'tag' field in "ProductSerializer", response comes very fast with all 2000 records.
Please suggest where is the loophole, why its affecting performance so much especially when I add this ManyToMany field.

I always use django-debug-toolbar to debug my queryset to find bottleneck/duplicate query in my project. Django orm always using lazy load to retrieve related fields from database.
You can change this default behavior of your queryset by eager load your many to many field using prefetch_related.
class ProductViewSet(viewsets.ModelViewSet):
queryset = Product.objects.prefetch_related('tag').all()
serializer_class = ProductSerializer
Reference: prefetch_related

How do I specify database model in a serpy serializer?

I have two models,
class Publication(models.Model):
title = models.CharField(max_length=30)
class Article(models.Model):
headline = models.CharField(max_length=100)
publications = models.ManyToManyField(Publication)
I am trying to learn serializing using serpy.
I wrote two serializers, but I am not sure how to mention the model. I wrote a django rest framework serializer, as follows,
class PublicationSerializer(serializers.ModelSerializer):
class Meta:
model = Publication
fields = 'title',
class ArticleSerializer(serializers.ModelSerializer):
publications = PublicationSerializer(many=True)
class Meta:
model = Article
fields = '__all__'
This is the serializers that I wrote for using with Serpy.
class PublicationSerializer(serpy.Serializer):
title = serpy.Field()
class ArticleSerializer(serpy.Serializer):
headline = serpy.Field()
publications = PublicationSerializer()
I dont know where should I mention the model,
I would like to be able to serialize a queryset, say
Article.objects.all()
what changes must be made to use it with Django Rest Framework?

You apparently don't need to specify an associated model for the serpy serializer. Passing your Django objects to the appropriate serpy serializer class should suffice. Or not?
articles = Article.objects.all()
articles_serialized = ArticleSerializer(articles, many=True).data

Optimizing database queries in Django REST framework

I have the following models:
class User(models.Model):
name = models.Charfield()
email = models.EmailField()
class Friendship(models.Model):
from_friend = models.ForeignKey(User)
to_friend = models.ForeignKey(User)
And those models are used in the following view and serializer:
class GetAllUsers(generics.ListAPIView):
authentication_classes = (SessionAuthentication, TokenAuthentication)
permission_classes = (permissions.IsAuthenticated,)
serializer_class = GetAllUsersSerializer
model = User
def get_queryset(self):
return User.objects.all()
class GetAllUsersSerializer(serializers.ModelSerializer):
is_friend_already = serializers.SerializerMethodField('get_is_friend_already')
class Meta:
model = User
fields = ('id', 'name', 'email', 'is_friend_already',)
def get_is_friend_already(self, obj):
request = self.context.get('request', None)
if request.user != obj and Friendship.objects.filter(from_friend = user):
return True
else:
return False
So basically, for each user returned by the GetAllUsers view, I want to print out whether the user is a friend with the requester (actually I should check both from_ and to_friend, but does not matter for the question in point)
What I see is that for N users in database, there is 1 query for getting all N users, and then 1xN queries in the serializer's get_is_friend_already
Is there a way to avoid this in the rest-framework way? Maybe something like passing a select_related included query to the serializer that has the relevant Friendship rows?

Django REST Framework cannot automatically optimize queries for you, in the same way that Django itself won't. There are places you can look at for tips, including the Django documentation. It has been mentioned that Django REST Framework should automatically, though there are some challenges associated with that.
This question is very specific to your case, where you are using a custom SerializerMethodField that makes a request for each object that is returned. Because you are making a new request (using the Friends.objects manager), it is very difficult to optimize the query.
You can make the problem better though, by not creating a new queryset and instead getting the friend count from other places. This will require a backwards relation to be created on the Friendship model, most likely through the related_name parameter on the field, so you can prefetch all of the Friendship objects. But this is only useful if you need the full objects, and not just a count of the objects.
This would result in a view and serializer similar to the following:
class Friendship(models.Model):
from_friend = models.ForeignKey(User, related_name="friends")
to_friend = models.ForeignKey(User)
class GetAllUsers(generics.ListAPIView):
...
def get_queryset(self):
return User.objects.all().prefetch_related("friends")
class GetAllUsersSerializer(serializers.ModelSerializer):
...
def get_is_friend_already(self, obj):
request = self.context.get('request', None)
friends = set(friend.from_friend_id for friend in obj.friends)
if request.user != obj and request.user.id in friends:
return True
else:
return False
If you just need a count of the objects (similar to using queryset.count() or queryset.exists()), you can include annotate the rows in the queryset with the counts of reverse relationships. This would be done in your get_queryset method, by adding .annotate(friends_count=Count("friends")) to the end (if the related_name was friends), which will set the friends_count attribute on each object to the number of friends.
This would result in a view and serializer similar to the following:
class Friendship(models.Model):
from_friend = models.ForeignKey(User, related_name="friends")
to_friend = models.ForeignKey(User)
class GetAllUsers(generics.ListAPIView):
...
def get_queryset(self):
from django.db.models import Count
return User.objects.all().annotate(friends_count=Count("friends"))
class GetAllUsersSerializer(serializers.ModelSerializer):
...
def get_is_friend_already(self, obj):
request = self.context.get('request', None)
if request.user != obj and obj.friends_count > 0:
return True
else:
return False
Both of these solutions will avoid N+1 queries, but the one you pick depends on what you are trying to achieve.

Described N+1 problem is a number one issue during Django REST Framework performance optimization, so from various opinions, it requires more solid approach then direct prefetch_related() or select_related() in get_queryset() view method.
Based on collected information, here's a robust solution that eliminates N+1 (using OP's code as an example). It's based on decorators and slightly less coupled for larger applications.
Serializer:
class GetAllUsersSerializer(serializers.ModelSerializer):
friends = FriendSerializer(read_only=True, many=True)
# ...
#staticmethod
def setup_eager_loading(queryset):
queryset = queryset.prefetch_related("friends")
return queryset
Here we use static class method to build the specific queryset.
Decorator:
def setup_eager_loading(get_queryset):
def decorator(self):
queryset = get_queryset(self)
queryset = self.get_serializer_class().setup_eager_loading(queryset)
return queryset
return decorator
This function modifies returned queryset in order to fetch related records for a model as defined in setup_eager_loading serializer method.
View:
class GetAllUsers(generics.ListAPIView):
serializer_class = GetAllUsersSerializer
#setup_eager_loading
def get_queryset(self):
return User.objects.all()
This pattern may look like an overkill, but it's certainly more DRY and has advantage over direct queryset modification inside views, as it allows more control over related entities and eliminates unnecessary nesting of related objects.

Using this metaclass DRF optimize ModelViewSet MetaClass
from django.utils import six
#six.add_metaclass(OptimizeRelatedModelViewSetMetaclass)
class MyModelViewSet(viewsets.ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MyModelSerializer

You can split the view into two query.
First, only get the Users list (without is_friend_already field). This only require one query.
Second, get the friends list of request.user.
Third, modify the results depending on if the user is in the request.user's friend list.
class GetAllUsersSerializer(serializers.ModelSerializer):
...
class UserListView(ListView):
def get(self, request):
friends = request.user.friends
data = []
for user in self.get_queryset():
user_data = GetAllUsersSerializer(user).data
if user in friends:
user_data['is_friend_already'] = True
else:
user_data['is_friend_already'] = False
data.append(user_data)
return Response(status=200, data=data)

how to add annotate data in django-rest-framework queryset responses?

I am generating aggregates for each item in a QuerySet:
def get_queryset(self):
from django.db.models import Count
queryset = Book.objects.annotate(Count('authors'))
return queryset
But I am not getting the count in the JSON response.
thank you in advance.

The accepted solution will hit the database as many times as results are returned. For each result, a count query to the database will be made.
The question is about adding annotations to the serializer, which is way more effective than doing a count query for each item in the response.
A solution for that:
models.py
class Author(models.Model):
name = models.CharField(...)
other_stuff = models...
...
class Book(models.Model):
author = models.ForeignKey(Author)
title = models.CharField(...)
publication_year = models...
...
serializers.py
class BookSerializer(serializers.ModelSerializer):
authors = serializers.IntegerField()
class Meta:
model = Book
fields = ('id', 'title', 'authors')
views.py
class BookViewSet(viewsets.ModelViewSet):
queryset = Book.objects.annotate(authors=Count('author'))
serializer_class = BookSerializer
...
That will make the counting at database level, avoiding to hit database to retrieve authors count for each one of the returned Book items.

The queryset returned from get_queryset provides the list of things that will go through the serializer, which controls how the objects will be represented. Try adding an additional field in your Book serializer, like:
author_count = serializers.IntegerField(
source='author_set.count',
read_only=True
)
Edit: As others have stated, this is not the most efficient way to add counts for cases where many results are returned, as it will hit the database for each instance. See the answer by #José for a more efficient solution.

Fiver's solution will hit the db for every instance in the queryset so if you have a large queryset, his solution will create a lot of queries.
I would override the to_representation of your Book serializer, it reuses the result from the annotation. It will look something like:
class BookSerializer(serializers.ModelSerializer):
def to_representation(self, instance):
return {'id': instance.pk, 'num_authors': instance.authors__count}
class Meta:
model = Book

So, if you make an annotation like
Model.objects.annotate(
some_new_col=Case(
When(some_field=some_value, then=Value(something)),
# etc...
default=Value(something_default),
output_field=SomeTypeOfField(),
)
).filter()#etccc
and the interpreter throws in an error that something is not a model field for the related serializer, there is a workaround. It's not nice but if you add a method some_new_col, it recognizes the value from the query above.
The following will do just fine.
def some_new_col(self):
pass;