We Keep Coding

Django recursive serializing

Let's say I have models like this:
class A(models.Model):
...some fields here
class B(models.Model):
...some fields here
a = models.ForeignKey(A, on_delete=CASCADE)
class C(models.Model):
...some fields here
b = models.ForeignKey(B, on_delete=CASCADE)
...
And I want my API endpoint to return something like this
{
...some fields here
b: [
{
...some field here
c: [{...}, {...} ...]
},
{
...some field here
c: [{...}, {...} ...]
}
...
]
}
I know I can do something like this:
class Bserializer(serializers.ModelSerializer):
c = Cserializer(source="c_set", many=True, read_only=True,)
class Meta:
model = B
fields = [...some fields, "c"]
class Aserializer(serializers.ModelSerializer):
b = Bserializer(source="b_set", many=True, read_only=True,)
class Meta:
model = A
fields = [...some fields, "b"]
But if this goes deeper or/and models have more foreign keys it starts to become really complicated. Is there a way to add recursively all instances referencing the model.

You can specify a depth value in the serializer Meta class, however, this won't' work if you want to customize any part of the nested data, as this will automatically create model serializers (with all fields) for nested relations. Check docs here
class Aserializer(serializers.ModelSerializer):
class Meta:
model = A
fields = [...some fields, "b"]
depth = 2
The bottom line, if you don't do any customization of the nested serializers, use it, otherwise, stick with custom serializers for each of the relations. But I'll suggest keeping the nested data at a minimum as it will impact performance

Django rest framework - Optimizng serialization of nested related field

Given the following models:
class Model_A:
...
class Model_B:
...
class Model_C:
model_a = ForeignKey(Model_A, related_name='c_items')
model_b = ForeignKey(Model_B)
...
And the following model serializers setup:
class Model_A_Serializer:
class Model_C_Serializer:
class Meta:
model = Model_C
fields = ( 'model_b', ... )
c_items = Model_C_Serializer(many=True)
class Meta:
model = Model_A
fields = ( 'c_items', ... )
And a basic vieweset:
class Model_A_Viewset:
model = Model_A
queryset = model.objects.all()
serializer_class = Model_A_Serializer
...
When the user POST's the following JSON payload to create an instance of Model_A along with instances of Model_C:
{
'c_items': [
{
'model_b': 1
},
{
'model_b': 2
},
{
'model_b': 3
}
]
}
Note: Model_B instances with IDs 1, 2, and 3 already exist, but no Model_A and no Model_C instances exist in the above example.
Then I noticed that django seems to execute the following queries when serializing the incoming data:
SELECT ... FROM Model_B WHERE id = 1;
SELECT ... FROM Model_B WHERE id = 2;
SELECT ... FROM Model_B WHERE id = 3;
This seems unnecessary to me as a single SELECT ... FROM Model_B WHERE id IN (1,2,3) would do the job.
How do I go about optimizng this?
I have tried to modify the queryset in the viewset above like so:
queryset = model.objects.prefetch_related('c_items__model_b').all()
But this did not make a difference in the number of queries being executed.

been a while since I django'd but I'm pretty sure you need to prefetch the relations and the relations relations, and I'm pretty sure prefetch is for many to many or reverse foreign keys, while select is for direct foreign keys, which means you can save one query by using select_related instead:
model.objects.prefetch_related('c_items').select_related('c_items__model_b').all()

Django Serializer Nested Creation: How to avoid N+1 queries on relations

There are dozens of posts about n+1 queries in nested relations in Django, but I can't seem to find the answer to my question. Here's the context:
The Models
class Book(models.Model):
title = models.CharField(max_length=255)
class Tag(models.Model):
book = models.ForeignKey('app.Book', on_delete=models.CASCADE, related_name='tags')
category = models.ForeignKey('app.TagCategory', on_delete=models.PROTECT)
page = models.PositiveIntegerField()
class TagCategory(models.Model):
title = models.CharField(max_length=255)
key = models.CharField(max_length=255)
A book has many tags, each tag belongs to a tag category.
The Serializers
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
exclude = ['id', 'book']
class BookSerializer(serializers.ModelSerializer):
tags = TagSerializer(many=True, required=False)
class Meta:
model = Book
fields = ['title', 'tags']
def create(self, validated_data):
with transaction.atomic():
tags = validated_data.pop('tags')
book = Book.objects.create(**validated_data)
Tag.objects.bulk_create([Tag(book=book, **tag) for tag in tags])
return book
The Problem
I am trying to POST to the BookViewSet with the following example data:
{
"title": "The Jungle Book"
"tags": [
{ "page": 1, "category": 36 }, // plot intro
{ "page": 2, "category": 37 }, // character intro
{ "page": 4, "category": 37 }, // character intro
// ... up to 1000 tags
]
}
This all works, however, during the post, the serializer proceeds to make a call for each tag to check if the category_id is a valid one:
With up to 1000 nested tags in a call, I can't afford this.
How do I "prefetch" for the validation?
If this is impossible, how do I turn off the validation that checks if a foreign_key id is in the database?
EDIT: Additional Info
Here is the view:
class BookViewSet(views.APIView):
queryset = Book.objects.all().select_related('tags', 'tags__category')
permission_classes = [IsAdminUser]
def post(self, request, format=None):
serializer = BookSerializer(data=request.data)
if serializer.is_valid():
serializer.save()
return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

The DRF serializer is not the place (in my own opinion) to optimize a DB query. Serializer has 2 jobs:
Serialize and check the validity of input data.
Serialize output data.
Therefore the correct place to optimize your query is the corresponding view.
We will use the select_related method that:
Returns a QuerySet that will “follow” foreign-key relationships, selecting additional related-object data when it executes its query. This is a performance booster which results in a single more complex query but means later use of foreign-key relationships won’t require database queries.
to avoid the N+1 database queries.
You will need to modify the part of your view code that creates the corresponding queryset, in order to include a select_related call.
You will also need to add a related_name to the Tag.category field definition.
Example:
# In your Tag model:
category = models.ForeignKey(
'app.TagCategory', on_delete=models.PROTECT, related_name='categories'
)
# In your queryset defining part of your View:
class BookViewSet(views.APIView):
queryset = Book.objects.all().select_related(
'tags', 'tags__categories'
) # We are using the related_name of the ForeignKey relationships.
If you want to test something different that uses also the serializer to cut down the number of queries, you can check this article.

I think the issue here is that the Tag constructor is automatically converting the category id that you pass in as category into a TagCategory instance by looking it up from the database. The way to avoid that is by doing something like the following if you know that all of the category ids are valid:
def create(self, validated_data):
with transaction.atomic():
tags = validated_data.pop('tags')
book = Book.objects.create(**validated_data)
tag_instances = [ Tag(book_id=book.id, page=x['page'], category_id=x['category']) for x in tags ]
Tag.objects.bulk_create(tag_instances)
return book

I've come up with an answer that gets things working (but that I'm not thrilled about): Modify the Tag Serializer like this:
class TagSerializer(serializers.ModelSerializer):
category_id = serializers.IntegerField()
class Meta:
model = Tag
exclude = ['id', 'book', 'category']
This allows me to read/write a category_id without having the overhead of validations. Adding category to exclude does mean that the serializer will ignore category if it's set on the instance.

Problem is that you don't set created tags to the book instance so serializer try to get this while returning.
You need to set it to the book as a list:
def create(self, validated_data):
with transaction.atomic():
book = Book.objects.create(**validated_data)
# Add None as a default and check that tags are provided
# If you don't do that, serializer will raise error if request don't have 'tags'
tags = validated_data.pop('tags', None)
tags_to_create = []
if tags:
tags_to_create = [Tag(book=book, **tag) for tag in tags]
Tag.objects.bulk_create(tags_to_create)
# Here I set tags to the book instance
setattr(book, 'tags', tags_to_create)
return book
Provide Meta.fields tuple for TagSerializer (it's weird that this serializer don't raise error saying that fields tuple is required)
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
fields = ('category', 'page',)
Prefetching tag.category should be NOT necessary in this case because it's just id.
You will need prefetching Book.tags for GET method. The simplest solution is to create static method for serializer and use it in viewset get_queryset method like this:
class BookSerializer(serializers.ModelSerializer):
...
#staticmethod
def setup_eager_loading(queryset): # It can be named any name you like
queryset = queryset.prefetch_related('tags')
return queryset
class BookViewSet(views.APIView):
...
def get_queryset(self):
self.queryset = BookSerializer.setup_eager_loading(self.queryset)
# Every GET request will prefetch 'tags' for every book by default
return super(BookViewSet, self).get_queryset()

select_related function will check ForeignKey in the first time.
Actually,this is a ForeignKey check in the relational database and you can use SET FOREIGN_KEY_CHECKS=0; in database to close inspection.

How to nest two serializers with same model

I have two serializers with same model. I want to nest them.
Unfortunately this approach does not work:
class BookSerializer(serializers.ModelSerializer):
class Meta:
model = Book
fields = ['name', 'word_count']
class BetterBookSerializer(serializers.ModelSerializer):
book = BookSerializer(many=False)
class Meta:
model = Book
fields = ('id', 'book')
Expected result:
{
"id": 123,
"book": {
"name": "book_name",
"word_count": 123
}
}

Use source=* instead of many=True as
class BetterBookSerializer(serializers.ModelSerializer):
book = BookSerializer(source='*')
class Meta:
model = Book
fields = ('id', 'book')
From the doc,
The value source='*' has a special meaning, and is used to indicate that the entire object should be passed through to the field. This can be useful for creating nested representations, or for fields which require access to the complete object in order to determine the output representation.

You can achieve the desired output like this:
class BookSerializer(serializers.ModelSerializer):
class Meta:
model = Book
fields = ['name', 'word_count']
class BetterBookSerializer(serializers.ModelSerializer):
book = serializers.SerializerMethodField(read_only=True)
class Meta:
model = Book
fields = ('id', 'book')
def get_book(self, obj):
return BookSerializer(obj).data
Small Update:
Although my approach to solve your problem works just fine, the answer from #JPG mentioning source='*' option is a good way to go. In that way you can easily use the nested serializer when creating new object.

django output with array or list of object

I am doing my first Django project and I am stuck here. I need two arrays in result output
Pending reservations
Approved reservations
Reservation contain type of services user opted for. There can be multiple reservations.
status=0 - pending
status=1 - approved
Required output
result: {
"upcoming_pending":
[{reservation_id,location_id,location_name,arrival_date,arrival_time,
departure_date,departure_time,
services:[{service_id,service_title,service_description},{}],
add-ons:[{service_id,service_title,service_description},{}],comments},
{},etc]},
"upcoming_approved":
[{reservation_id,location_id,location_name,arrival_date,arrival_time,
departure_date,departure_time,
services:[{service_id,service_title,service_description},{}],
add-ons:[{service_id,service_title,service_description},{}],comments},
{},etc]}
My code I am trying something like this below
data={}
data["upcoming_pending"]=[]
data["upcoming_approved"]=[]
queryset1 = reservations.objects.filter(user_id=user_id,arrival__gte=datetime.now(),status=0)
for a in queryset1:
data["upcoming_pending"][a.id]=reservations.objects.filter(id=a.id)
data["upcoming_pending"][a.id]["services"]=services.objects.filter(service_id__in=ser,type=1)
data["upcoming_pending"][a.id]["add-ons"]=services.objects.filter(service_id__in=ser,type=2)
queryset2 = reservations.objects.filter(user_id=user_id,arrival__gte=datetime.now(),status=1)
for b in queryset2:
data["upcoming_approved"][b.id]=reservations.objects.filter(id=a.id)
data["upcoming_approved"][b.id]["services"]=services.objects.filter(service_id__in=ser,type=1)
data["upcoming_approved"][b.id]["add-ons"]=services.objects.filter(service_id__in=ser,type=2)
return Response({"result":data}, status=status.HTTP_200_OK)

Let's build it with Serializers, bottom-up:
first, a serializer for services;
one serializer for reservations with two extra fields (services and add-ons);
one serializer containing the two reservations groups (pending and approved).
I assume your models are named Service and Reservations.
1. Services
Create a serializers.py file in your Django app, and in there, create a ModelSerializer for the Service model instances as follows.
from rest_framework import serializers
class ServiceSerializer(serializers.ModelSerializer):
class Meta:
model = Service
2. Reservations
For the Reservations model instances, since we want to add two custom related fields, we'll create a custom serializer containing all the fields we want to show, as follows.
class ReservationSerializer(serializers.Serializer):
# By using 'source' you can rename a field
reservation_id = serializers.IntegerField(source='pk')
location_id = serializers.IntegerField()
location_name = serializers.CharField()
arrival_date = serializers.DateField()
arrival_time = serializers.TimeField()
departure_date = serializers.DateField()
departure_time = serializers.TimeField()
# ... other model fields you may need
# These are nested serializers, 'many' means that they are lists
services = ServiceSerializer(many=True)
add_ons = ServiceSerializer(many=True)
Please note: serialization will fail if the serializer fields are not found in the objects passed to the serializer. E.g. if departure_date is not a field of Reservation, the above code won't work.
3. Pending and approved
The last serializer is the one we are returning as response: it simply takes two ReservationSerializers as its fields.
class PendingApprovedReservationsSerializer(serializers.Serializer):
upcoming_pending = ReservationSerializer(many=True)
upcoming_approved = ReservationSerializer(many=True)
Putting it all together
Ok, now we have all the serializers in place. In our view we can do something similar:
pending = reservations.objects.filter(user_id=user_id,
arrival__gte=datetime.now(), status=0)
for a in pending:
a.services = services.objects.filter(service_id__in=ser, type=1)
a.add_ons = services.objects.filter(service_id__in=ser, type=2)
approved = reservations.objects.filter(user_id=user_id,
arrival__gte=datetime.now(), status=1)
for b in approved:
b.services = services.objects.filter(service_id__in=ser, type=1)
b.add_ons = services.objects.filter(service_id__in=ser, type=2)
data = dict(upcoming_pending=pending,upcoming_approved=approved)
serializer = PendingApprovedReservationsSerializer(data)
# HTTP 200 status is set by default
return Response(serializer.data)
Disclaimer: this is untested code, it only aims to be a suggestion.
Reference: DRF nested serializers