I have Char objects with ManyToMany relationships to Source objects. Because a Char can appear in many Sources and many Sources can contain multiple Chars. The MtM relationship goes through a through table which also contains page number. In my API response, which I built using the Django REST framework I want to avoid resolving the full Source title, author etc. for every Char. Rather, in order to reduce the size of the JSON response, I want to refer it by id and include a sources section so the client can look it up.
I.e. a client visiting /api/char/26 should get the following response:
"chars": [
{
"id": 26,
"name": "龜",
"locations": [
{
"page": 136,
"source": 1
},
{
"page": 162,
"source": 1
}
]
}
],
"sources": [
{
"id": 1,
"title": "Bruksanvisning Foamglass",
"author": "Bluppfisk"
}
]
Here's the API view:
class CharAPIView(generics.RetrieveAPIView):
queryset = Char.objects.all()
serializer_class = CharSerializer
and the Serializers:
class CharSerializer(serializers.ModelSerializer):
locations = serializers.SerializerMethodField()
class Meta:
model = Char
fields = ('id', 'name', 'locations',)
depth = 1
def get_locations(self, obj):
qset = CharInSource.objects.filter(char=obj)
return [CharInSourceSerializer(m).data for m in qset]
class CharInSourceSerializer(serializers.ModelSerializer):
class Meta:
model = CharInSource
fields = ('page', 'source',)
The problem is I do not know how to hook into the generics.RetrieveAPIView class so it will include a list of relevant sources. I've been digging through the source, but I cannot figure out how to even get the pk value.
In the end, I ended up solving it as follows, by overwriting the retrieve method of my view.
class CharAPIView(generics.RetrieveAPIView):
queryset = Char.objects.all()
def retrieve(self, *args, **kwargs):
instance = self.get_object()
char = CharSerializer(instance).data
qset = Source.objects.all()
sources = [SourceSerializer(m).data for m in [i for i in instance.location.all()]]
return Response({
'char': char,
'sources': sources,
})
This could be accomplished with another SerializerMethodField on your CharSerializer and creating a SourceSerializer; no extension of the base methods to GenericAPIView or RetrieveModelMixin.
def SourceSerializer(ModelSerializer):
class Meta:
model = Source
fields = ('id', 'title', 'author') # assuming author is not also a
# ForeignKey, otherwise returns an id
def CharSerializer(...):
....
sources = SerializerMethodField()
def get_sources(self, obj):
return SourceSerializer(
Source.objects.filter(chars__in=[obj.id]).distinct(),
many=True).data
class Meta:
fields = (...,'sources')
Assuming the attribute to the MTM model related_name is chars you can use chars__in and pass a list of Char ids; which in this case is the single char we are referencing. This would however contain the sources within each char object instead of outside as you had indicated by the question. However, I imagine you would want to know which sources have which char, as my solution would provide.
Without seeing the exact structure of your models, I cannot be certain of exactly how you should retrieve the Source objects. I feel like you could also replace the queryset with obj.sources.all() instead of the clunky __in query in the SourceSerializer.
Related
I am using graphene-python, django-filters and relay in my GraphQL API. Let's imagine I have a type FrameType which has an integer field time_offset and I would like to be able to use a range on it - ask only for frames which have the time_offset within the given range. I prepared my schema.py according to the graphene-python docs with a custom FilterSet:
import django_filters
from graphene import ObjectType, relay
from graphene_django import DjangoObjectType, filter
from my_app.blabla import models
class FrameFilter(django_filters.FilterSet):
time_offset = django_filters.RangeFilter()
class Meta:
model = models.Frame
fields = ("time_offset",)
class FrameType(DjangoObjectType):
class Meta:
model = models.Frame
filterset_class = FrameFilter
interfaces = (relay.Node,)
class Query(ObjectType):
frames = filter.DjangoFilterConnectionField(FrameType)
class Meta:
abstract = True
However, I have no idea how to query the timeOffset field now. I found no examples online for the django_filters.RangeFilter field. This is a query I tried:
query Frame {
frames(first: 20, timeOffset: "{\"gt\":\"4350\", \"lt\":\"5000\"}") {
edges {
node {
timeOffset
}
}
}
... also with these alternatives:
timeOffset: "{\"gt\":4350, \"lt\":5000}"
timeOffset: "{\"start\":\"4350\", \"end\":\"5000\"}"
timeOffset: "{\"min\":\"4350\", \"max\": \"4500\"}"
timeOffset: "[\"4350\", \"5000\"]"
timeOffset: "[4350, 5000]"
timeOffset: "[4350]"
timeOffset: "4350,5000"
These queries don't raise any error, but they don't filter either (all results are returned). I am lost, I'm not sure if I still haven't found the proper syntax, or maybe there's some mistake in my backend code. How should I use and query the django_filters.RangeFilter on a field?
Sadly, this isn't possible. But, there is a workaround for it
Adjust your filter class as
def custom_range_filter_method(queryset, field_name, value):
if value:
queryset = queryset.filter(**{f'{field_name}__range': value.split(',')})
return queryset
class FrameFilter(django_filters.FilterSet):
time_offset = filters.Filter(method=custom_range_filter_method)
class Meta:
model = models.Frame
fields = ("time_offset",)
Now query the schema with
query Frame {
frames(first: 20, timeOffset: "4350,5000") {
edges {
node {
timeOffset
}
}
}
Reference
Customize filter result with Filter.method--(django-filter doc)
You can handle the range option at Django's queryset level without disturbing the existing relay query.
In your case,
Pass start_time_offset and end_time_offset arguments to DjangoConnectionField
Override resolve_frames
filter on django queryset if start_time_offset or end_time_offset is provided by user else return objects.all()
class Query(ObjectType):
frames = filter.DjangoFilterConnectionField(FrameType, start_time_offset=graphene.Int(), end_time_offset=graphene.Int())
def resolve_frames(self, info, start_time_offset=None, end_time_offset=None, **kwargs):
if start_time_offset and end_time_offset:
return Frame.objects.filter(time_offset__range=(start_time_offset, end_time_offset))
elif start_time_offset:
return Frame.objects.filter(time_offset__gte=start_time_offset)
elif end_time_offset:
return Frame.objects.filter(time_offset__lte=end_time_offset)
return Frame.objects.all()
Now you can query on it with your regular filters provided by relay:
query Frame {
frames(last: 5, startTimeOffset: 4350, endTimeOffset:5000) {
edges {
node {
timeOffset
}
}
}
It's a bit old but since it might help others, you can check this thread, it used django filterset for DateRangeFiled and I think you can use a similar approach for integers using filterset RangeFilter. Also check this for more info about filterset in graphene.
All you need to do is update your FrameType class as follows:
class FrameType(DjangoObjectType):
class Meta:
model = models.Frame
filterset_fields = {
'time_offset': ['range']
}
interfaces = (relay.Node,)
You do not need a custom filterset for this.
Then you can query as follows:
query Frame {
frames(first: 20, timeOffset_Range: ["4350", "5000"]) {
edges {
node {
timeOffset
}
}
}
I have this serializer that represents content abstract where I would like to add an hyperlink field that is not in the model, but calculated by the framework linked to the ContentsSerializer.
class ContentsAbstractSerializer(serializers.HyperlinkedModelSerializer):
content_url = ???
class Meta:
model = Contents
fields = ('content_url','content_id','content_title', 'content_abstract','start_date','stop_date','last_date','content_status','version')
class ContentsSerializer(serializers.HyperlinkedModelSerializer):
categories = CategoriesContentsSerializer(read_only=True, many=True)
class Meta:
model = Contents
fields = ('content_id','content_title', 'content_abstract', 'content_body','start_date','stop_date','last_date','content_status','version','sections', 'images','attaches','categories')
I would like to have a result like this:
{
"content_url":"http://mysite/Content/125",
"content_id": 125,
"content_title": "this is the title",
"content_abstract": "This is the abstract",
"start_date": "2005-01-12",
"stop_date": "3000-01-12",
"last_date": "2019-02-27T09:40:38Z",
"content_status": "PUBLISHED",
"version": 0
},
I think that instead of define your own custom field you should use the manually specify the view_name for each nested resource as part of extra_kwargs.
I think that you simply could do something like that:
class ContentsAbstractSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Contents
fields = ('content_url','content_id','content_title', 'content_abstract','start_date','stop_date','last_date','content_status','version')
extra_kwargs = {
'content_url': {'view_name': 'name_of_your_detail_view'},
}
Output:
{
"content_url":"http://mysite/Content/125",
"content_id": 125,
....
},
The Resource: 'HyperlinkedModelSerializer' with custom nested
'view_name's does not work in combination with
'depth'
Oficial Resource: How hyperlinked views are determined
There are dozens of posts about n+1 queries in nested relations in Django, but I can't seem to find the answer to my question. Here's the context:
The Models
class Book(models.Model):
title = models.CharField(max_length=255)
class Tag(models.Model):
book = models.ForeignKey('app.Book', on_delete=models.CASCADE, related_name='tags')
category = models.ForeignKey('app.TagCategory', on_delete=models.PROTECT)
page = models.PositiveIntegerField()
class TagCategory(models.Model):
title = models.CharField(max_length=255)
key = models.CharField(max_length=255)
A book has many tags, each tag belongs to a tag category.
The Serializers
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
exclude = ['id', 'book']
class BookSerializer(serializers.ModelSerializer):
tags = TagSerializer(many=True, required=False)
class Meta:
model = Book
fields = ['title', 'tags']
def create(self, validated_data):
with transaction.atomic():
tags = validated_data.pop('tags')
book = Book.objects.create(**validated_data)
Tag.objects.bulk_create([Tag(book=book, **tag) for tag in tags])
return book
The Problem
I am trying to POST to the BookViewSet with the following example data:
{
"title": "The Jungle Book"
"tags": [
{ "page": 1, "category": 36 }, // plot intro
{ "page": 2, "category": 37 }, // character intro
{ "page": 4, "category": 37 }, // character intro
// ... up to 1000 tags
]
}
This all works, however, during the post, the serializer proceeds to make a call for each tag to check if the category_id is a valid one:
With up to 1000 nested tags in a call, I can't afford this.
How do I "prefetch" for the validation?
If this is impossible, how do I turn off the validation that checks if a foreign_key id is in the database?
EDIT: Additional Info
Here is the view:
class BookViewSet(views.APIView):
queryset = Book.objects.all().select_related('tags', 'tags__category')
permission_classes = [IsAdminUser]
def post(self, request, format=None):
serializer = BookSerializer(data=request.data)
if serializer.is_valid():
serializer.save()
return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
The DRF serializer is not the place (in my own opinion) to optimize a DB query. Serializer has 2 jobs:
Serialize and check the validity of input data.
Serialize output data.
Therefore the correct place to optimize your query is the corresponding view.
We will use the select_related method that:
Returns a QuerySet that will “follow” foreign-key relationships, selecting additional related-object data when it executes its query. This is a performance booster which results in a single more complex query but means later use of foreign-key relationships won’t require database queries.
to avoid the N+1 database queries.
You will need to modify the part of your view code that creates the corresponding queryset, in order to include a select_related call.
You will also need to add a related_name to the Tag.category field definition.
Example:
# In your Tag model:
category = models.ForeignKey(
'app.TagCategory', on_delete=models.PROTECT, related_name='categories'
)
# In your queryset defining part of your View:
class BookViewSet(views.APIView):
queryset = Book.objects.all().select_related(
'tags', 'tags__categories'
) # We are using the related_name of the ForeignKey relationships.
If you want to test something different that uses also the serializer to cut down the number of queries, you can check this article.
I think the issue here is that the Tag constructor is automatically converting the category id that you pass in as category into a TagCategory instance by looking it up from the database. The way to avoid that is by doing something like the following if you know that all of the category ids are valid:
def create(self, validated_data):
with transaction.atomic():
tags = validated_data.pop('tags')
book = Book.objects.create(**validated_data)
tag_instances = [ Tag(book_id=book.id, page=x['page'], category_id=x['category']) for x in tags ]
Tag.objects.bulk_create(tag_instances)
return book
I've come up with an answer that gets things working (but that I'm not thrilled about): Modify the Tag Serializer like this:
class TagSerializer(serializers.ModelSerializer):
category_id = serializers.IntegerField()
class Meta:
model = Tag
exclude = ['id', 'book', 'category']
This allows me to read/write a category_id without having the overhead of validations. Adding category to exclude does mean that the serializer will ignore category if it's set on the instance.
Problem is that you don't set created tags to the book instance so serializer try to get this while returning.
You need to set it to the book as a list:
def create(self, validated_data):
with transaction.atomic():
book = Book.objects.create(**validated_data)
# Add None as a default and check that tags are provided
# If you don't do that, serializer will raise error if request don't have 'tags'
tags = validated_data.pop('tags', None)
tags_to_create = []
if tags:
tags_to_create = [Tag(book=book, **tag) for tag in tags]
Tag.objects.bulk_create(tags_to_create)
# Here I set tags to the book instance
setattr(book, 'tags', tags_to_create)
return book
Provide Meta.fields tuple for TagSerializer (it's weird that this serializer don't raise error saying that fields tuple is required)
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
fields = ('category', 'page',)
Prefetching tag.category should be NOT necessary in this case because it's just id.
You will need prefetching Book.tags for GET method. The simplest solution is to create static method for serializer and use it in viewset get_queryset method like this:
class BookSerializer(serializers.ModelSerializer):
...
#staticmethod
def setup_eager_loading(queryset): # It can be named any name you like
queryset = queryset.prefetch_related('tags')
return queryset
class BookViewSet(views.APIView):
...
def get_queryset(self):
self.queryset = BookSerializer.setup_eager_loading(self.queryset)
# Every GET request will prefetch 'tags' for every book by default
return super(BookViewSet, self).get_queryset()
select_related function will check ForeignKey in the first time.
Actually,this is a ForeignKey check in the relational database and you can use SET FOREIGN_KEY_CHECKS=0; in database to close inspection.
I'm trying to display a 'nested' model in my API response and having trouble shaping the data.
I have the model the API is called from:
something like
class Rules(Model):
conditions = models.ManyToManyField(RulesPoliciesConditions)
...
...
class RulesPoliciesConditions(Model):
rules = models.ForeignKey(Rules, ...)
policies = models.ForeignKey(Policy, ...)
Rules and Policies are their own models with a few TextFields (name, nickname, timestamp, etc)
My problem is that when I use the Rules model to call a field called conditions, only the rules and policies PK display. I want to reach the other attributes like name, timestamp, nickname, etc.
I tried making my fields (in my Serializer) try to call specifically like "conditions__rules__name" but it's invalid, I also tried "conditions.rules.name" which is also invalid. Maybe I'm using the wrong field in my serializer, I'm trying out conditions = serializers.SlugRelatedField(many=True, queryset=q, slug_field="id")
My intention is to display something like:
conditions: [
{
rules: {id: rulesId, name: rulesName, ...},
policies: {id: policiesId, name: policiesName, ...}
}, ...
]
or just even:
conditions: [
{
rules: rulesName,
policies: policiesName
}, ...
]
since right now it just returns the rulesId and policiesId and it doesn't "know" about the other fields
EDIT: I found a relevant question on SO but couldn't get a relevant answer
Django REST Framework: Add field from related object to ModelSerializer
This can be achieved by using nested serializers. The level of nesting can be controlled/customized by various methods
class RulesPoliciesConditionsSerializer(serializers.ModelSerializer):
class Meta:
fields = '__all__'
model = RulesPoliciesConditions
depth = 1
class RulesSerializer(serializers.ModelSerializer):
conditions = RulesPoliciesConditionsSerializer(many=True)
class Meta:
fields = '__all__'
model = Rules
Pass your Rules queryset to the RulesSerializer serializer to
get the desired output
Example
rules_qs = Rules.objects.all()
rules_serializer = RulesSerializer(rules_qs, many=True)
data = rules_serializer.data
References
1. serializer depth
2. Nested serializer
You can use nested serializers for the purpose.
class RuleSerializer(serializers.ModelSerializer):
...
class Meta:
model = Rules(rulesId, rulesName)
fields = ('id', 'email', 'country')
class RulesPoliciesConditionsSerializer(serializers.ModelSerializer):
rules = RuleSerializer()
policies = PolicySerializer()
...
I'm trying to get data as json format. I've one ManyToMany field which is returning just id. But I need that contents too. Here is my models.py
class Pricing(models.Model):
name = models.CharField(max_length = 100)
price = models.CharField(max_length = 100)
def __str__(self):
return self.name+' and '+self.price
class Service(models.Model):
name = models.CharField(max_length=100)
price = models.ManyToManyField(Pricing, blank=True)
def __str__(self):
return self.name
And also the views.py which is returning json format data
def all_service_json(request, name):
data = serializers.serialize("json", Service.objects.filter(name__icontains=name))
return HttpResponse(data)
Now Getting the output like below
[
{
"model": "myapp.service",
"pk": 2,
"fields":
{
"name": "Service name",
"price": [1, 2]
}
}
]
But want like below
[
{
"model": "myapp.service",
"pk": 2,
"fields":
{
"name": "Service name",
"price":
{
1: "Price 1",
2: "Price 2"
}
}
}
]
Creating ModelSerializer objects from within Django Rest Framework will let you display nested object data:
http://www.django-rest-framework.org/api-guide/serializers/#dealing-with-nested-objects
# myapp/serializers.py
...
from rest_framework import serializers
class PricingSerializer(serializers.ModelSerializer):
class Meta:
fields = '__all__'
model = Pricing
class ServiceSerializer(serializers.ModelSerializer):
price = PricingSerializer(read_only=True, many=True)
class Meta:
fields = '__all__'
model = Service
# myapp/views.py
def all_service_json(request, name):
services = Service.objects.filter(name__icontains=name)
data = ServiceSerializer(services, many=True).data
return HttpResponse(data)
As #robert mentioned using nested serializers will fix your issue.
But note that by default nested serializers are read-only. So If you
want to support write operations to a nested serializer field you'll
need to add create() and/or update() methods in order to explicitly
specify how the child relationships should be saved.
Writable Service Serializer
class ServiceSerializer(serializers.ModelSerializer):
price = PricingSerializer(many=True)
class Meta:
fields = '__all__'
model = Service
# sample create
def create(self, validated_data):
prices_data = validated_data.pop('price')
service = Service.objects.create(**validated_data)
for price_data in prices_data:
Price.objects.create(service=service, **price_data)
return service
# add update here
myapp/views.py
def all_service_json(request, name):
services = Service.objects.filter(name__icontains=name)
serializer = ServiceSerializer(services)
return HttpResponse(serializer.data)
So in your case all you have to do is to add depth = 1 and you will get nested representations.
Docs
The default ModelSerializer uses primary keys for relationships, but
you can also easily generate nested representations using the depth
option:
class AccountSerializer(serializers.ModelSerializer):
class Meta:
model = Account
fields = ['id', 'account_name', 'users', 'created']
depth = 1
I just start learning Django from last 8 hours and I stuck into this situation where many to many relationship is returning id instead of child data. I wrote some custom code. For me it solve my problem, hope this helps someone.
from django.core.serializers import serialize
import json
def modelToDict(model):
jsn = serialize("json", model) # convert to json
mydict = json.loads(jsn) # again convert to dictionary
return mydict
def all_service_json(request, name):
data = Service.objects.filter(name__icontains=name)
dictdata = modelToDict(data)
for i in range(len(dictdata)):
price = modelToDict(data[i].price.all())
dictdata[i]['fields']['price'] = price
return HttpResponse(json.dumps(`dictdata`), content_type="application/json")