We Keep Coding

Django + Django Rest Framework: get correct related objects on intermediate model

I have an intermediate model with the following fields:
class UserSkill(models.Model):
user = models.ForeignKey(User, on_delete=models.CASCADE)
skill = models.ForeignKey(Skill, on_delete=models.CASCADE, related_name='user_skills')
disabled = models.BooleanField(default=False)
As you can see, it has two foreign keys, one to the auth user and one to another table called skill.
I am trying to get all Skills assigned to an specific user, so I do the following get_queryset in my ViewSet:
class AssignedSkillViewSet(viewsets.ModelViewSet):
queryset = Skill.objects.all()
serializer_class = AssignedSkillSerializer
permission_classes = [permissions.IsAuthenticated]
def get_queryset(self):
user = self.request.user
return Skill.objects.filter(user_skills__user=user, user_skills_user__disabled=False))
Now, I also need to include the intermediate model information in the API, which I can access trough users_skills related name in DRF's Serializer, as follows:
class AssignedSkillSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Skill
fields = [
'id',
'url',
'title',
'description',
'user_skills',
]
But when I try to get that information it returns ALL user_skills related to the assigned skill, no matter if they are assigned to other users. I need the related model information only for that user and that skill.
For Example:
If I have a skill named Math, and a user named Maria
related_skills = Skill.objects.filter(user_skills__user=user, user_skills_user__disabled=False)).user_skills.all()
The above code will return:
[
<UserSkill: Math+Jenniffer>,
<UserSkill: Math+Gabriel>,
<UserSkill: Math+John>,
<UserSkill: Math+Maria>,
]
I only need to get the item <UserSkill: Math+Maria>. The list is not ordered in any way so getting the last item on the list does not work in all cases.
I know there is something I'm probably missing. I appreciate any help or clues you people can give me.

In this case you need to use a Prefetch..[Django-doc] object with a custom queryset, that uses the same filters as your main queryset like this:
from django.db.models import Prefetch
def get_queryset(self):
user = self.request.user
return Skill.objects.filter(
user_skills__user=user,
user_skills__user__disabled=False,
).prefetch_related(
"user_skills",
queryset=UserSkill.objects.filter(
user=user,
user__disabled=False,
)
)

I think that when you do the filter:
Skill.objects.filter(
user_skills__user=user, #condition_1
user_skills_user__disabled=False, #condition_2
).user_skills.all()
You already did a query related to the UserSkill model. Because the filter is done in the Skill model and the #condition_1 (user_skills__user=user) uses the information from the UserSkill model to filter by users. But when you do .user_skills.all() at the end of the query you are overring the filter with all the data from the UserSkill model.
To get a list of UserSkill instances from the filter you could try:
UserSkill.objects.filter(
user="Maria",
skill="Math",
)

Maybe this will help
serializers.py
class SkillSerializer(serializers.ModelSerializer):
class Meta:
model = Skill
fields = ['id', ...]
class UserSkillSerializer(serializers.ModelSerializer):
skill_detail = SkillSerializer(many=True)
class Meta:
model = UserSkill
fields = ['id', 'user', 'skill_detail']
views.py
class AssignedSkillViewSet(viewsets.ModelViewSet):
queryset = UserSkill.objects.all()
serializer_class = UserSkillSerializer
permission_classes = [permissions.IsAuthenticated]
def get_queryset(self):
user = self.request.user
return UserSkill.objects.filter(user=user, disabled=False))

How to display only one record from child model for each header item in Django ListView using distinct()

I am using two related models in my Django application. The objects so created in the models are being displayed using the listview class. In the child model I can create multiple rows based on some key date. When I try to display values from both the models, all the child objects for the respective FK fields are displayed (wherever more than one records are there).
Let me make the situation clearer as below:
models.py
class MatPriceDoc(models.Model):
mat_price_doc_number = models.IntegerField(null=True, blank=True.....)
mat_num = models.ForeignKey(Material, on_delete=.......)
mat_group = models.ForeignKey(MatGrp, on_delete=.......)
doc_type = models.CharField(max_length=2, null=True, default='RF'....)
create_date = models.DateField(default=timezone.now,...)
class MatPriceItems(models.Model):
price_doc_header = models.ForeignKey(MatPriceDoc, on_delete=...)
price_item_num = models.CharField(max_length=3, default=10,...)
price_valid_from_date = models.DateField(null=True, verbose_name='From date')
price_valid_to_date = models.DateField(null=True, verbose_name='To date')
mat_price = models.DecimalField(max_digits=7, decimal_places=2, null=True, blank=True,...)
views.py
class MatPriceListView(ListView):
template_name = "mat_price_list.html"
context_object_name = 'matprice'
model = MatPriceDoc
def get_context_data(self, **kwargs):
context = super(MatPriceListView, self).get_context_data(**kwargs)
context.update({
'mat_price_item_list': MatPriceItems.objects.distinct().order_by('price_doc_header'), #This is where I have tried **distinct()**
})
return context
def get_queryset(self):
return MatPriceDoc.objects.order_by('mat_num')
Now the material price changes periodically and there would always be more than one price item for each material for each key date (price_valid_from_date). My objective is to display the latest price of all those materials for which price documents exist. My predicament is how to pick up only one of the many price items for the same material / document combination. My code line 'mat_price_item_list': MatPriceItems.objects.distinct().order_by('price_doc_header'), is of course not yielding any result (all price items are being displayed in successive columns).
Is there a way to show only one price item in the listview?
Edit
In the following image the prices maintained for various dates for materials are shown. What I was trying to get was only the price for the latest (valid) date is displayed for a particular material. So in the instant case (as displayed), prices for only 4th March 2020 should be displayed for each item MS and HSD.
Edit 2
This is how the child model data for an instance of header doc number (no. 32 here) looks like (image grab from the child table using data browser):
The columns are: Child obj. item no. / Price / Valid From / Valid To / Hdr Doc no. / Child Obj Row no.
My thinking was: Can I not pick up only the first object (a subset) from a collection of doc number? In the instant case (ref the image), doc no. 32 has three child items (bearing number 148, 149 and 156). Is it not possible to only pick up item number 156 and discard the rest?
I tried:
MatPriceItems.objects.order_by('-price_valid_to_date').first()
but raises error "MatPriceItems" object is not iterable.
What would enable me to get the only item for a header item and display it?

I believe you need the following:
class MatPriceListView(ListView):
template_name = "mat_price_list.html"
context_object_name = 'matprice'
model = MatPriceDoc
def get_context_data(self, **kwargs):
context = super(MatPriceListView, self).get_context_data(**kwargs)
context.update({
'mat_price_item_list': MatPriceItems.objects.all().('-price_doc_header').first(),
})
return context
def get_queryset(self):
return MatPriceDoc.objects.order_by('mat_num')
The key line change being:
MatPriceItems.objects.all().order_by('-price_doc_header').first()
Such that you order by the price_doc_header descending (hence the minus infront of that value). Then take the .first() in the returned <QuerySet> object.
Likewise, the following would also be valid:
MatPriceItems.objects.all().order_by('price_doc_header').last()
You could also make use of Django's built in aggregation in the ORM:
from django.db.models import Max
MatPriceItems.objects.all().aggregate(Max('price_doc_header'))
Putting this all together, I would say the best solution for readabiltiy of the code would be using Django's built-in Max function:
from django.db.models import Max
class MatPriceListView(ListView):
template_name = "mat_price_list.html"
context_object_name = 'matprice'
model = MatPriceDoc
def get_context_data(self, **kwargs):
context = super(MatPriceListView, self).get_context_data(**kwargs)
context.update({
'mat_price_item_list': MatPriceItems.objects.all().aggregate(Max('price_doc_header'))
})
return context
def get_queryset(self):
return MatPriceDoc.objects.order_by('mat_num')
Because it tells any colleague exactly what you're doing: taking the maximum value.
Update: On a second glance of your question - I believe you may also need to do some sort of .filter()...then make an .annotate() for the Max...

I'm also adding what I now believe is the correct answer based on a re-reading of your question.
I feel like the query needs to be ammended to:
MatPriceItems.objects.all().select_related(
'mat_price_docs'
).order_by(
'price_doc_header__mat_num'
)
i.e.,
class MatPriceListView(ListView):
template_name = "mat_price_list.html"
context_object_name = 'matprice'
model = MatPriceDoc
def get_context_data(self, **kwargs):
context = super(MatPriceListView, self).get_context_data(**kwargs)
context.update({
'mat_price_item_list': MatPriceItems.objects.all().select_related('mat_price_docs').order_by('price_doc_header__mat_num'),
})
return context
def get_queryset(self):
return MatPriceDoc.objects.order_by('mat_num')
To achieve this, you also add the related_name argument to the price_doc_header ForeignKey field.
class MatPriceItems(models.Model):
price_doc_header = models.ForeignKey(MatPriceDoc, related_name='mat_price_docs', on_delete=...)
price_item_num = models.CharField(max_length=3, default=10,...)
price_valid_from_date = models.DateField(null=True, verbose_name='From date')
price_valid_to_date = models.DateField(null=True, verbose_name='To date')
mat_price = models.DecimalField(max_digits=7, decimal_places=2, null=True, blank=True,...)
As always, any changes to your models - please re-run the makemigarations and migrate management commands.
Edit: You may even be able to define a new queryset using an annotation to link to the ForeignKey:
def get_queryset(self):
return MatPriceItems.objects.all().order_by(
'price_doc_header__mat_num'
).annotate(
mat_price_doc_number = F('price_doc_header__mat_price_doc_number')
...
)
N.B. You can import the F() expression as from django.db.models import F

How can I display the values from a ManyToMany Field in Django instead of their Id?

I have two models
Influencer
class Influencer(models.Model):
full_name = models.CharField('Full Name',max_length=255)
username = models.CharField('Username',max_length=255,unique=True)
photo = models.URLField(blank=True,max_length = 500)
email_id = models.EmailField('Email Id',blank=True,max_length=500)
categories = models.ManyToManyField(Category,blank=True,max_length=400)
and
2. Categories
class Category(models.Model):
name = models.CharField(max_length=400)
def __str__(self):
return self.name
Influencer has a many to many field categories.
My views functions to display all the influencers is:
def index(request):
influencers = Influencer.objects.all().order_by('followers')
paginator = Paginator(influencers,16)
page = request.GET.get('page')
paged_listings = paginator.get_page(page)
user_list = UserList.objects.all().filter(user_id = request.user.id)
queryset = list(chain(paged_listings,user_list))
ser_query = serializers.serialize('json', queryset)
return HttpResponse(ser_query,content_type='application/json')
The HttpResponse contains category id's instead of category names, something like this:
where categories is an array which contains category id's.
I want to display the name of categories instead of their id's.
I think this can be achived using Django Rest Framework nested serializer, but at this moment I am not using DRF.

there is an natural_key method in django to convert id to string
add natural_key method in your models.py file
class Category(models.Model):
name = models.CharField(max_length=400,)
def __str__(self):
return self.name
def natural_key(self):
return self.name
and then in your serializers.serializer you need to pass use_natural_foreign_keys=True,
ser_query = serializers.serialize('json', influencers,indent=2,use_natural_foreign_keys=True,)

convert your array element --integer value -- into string then access from a dict or a tuple which contains categories with their ids
id_to_category=["0":"whatever your category is","1":"whatever your category is","2":"whatever your category is","3":"whatever your category is"]
id=str(ur_array[id])
category_name=id_to_category[id]

Django Serializer Nested Creation: How to avoid N+1 queries on relations

There are dozens of posts about n+1 queries in nested relations in Django, but I can't seem to find the answer to my question. Here's the context:
The Models
class Book(models.Model):
title = models.CharField(max_length=255)
class Tag(models.Model):
book = models.ForeignKey('app.Book', on_delete=models.CASCADE, related_name='tags')
category = models.ForeignKey('app.TagCategory', on_delete=models.PROTECT)
page = models.PositiveIntegerField()
class TagCategory(models.Model):
title = models.CharField(max_length=255)
key = models.CharField(max_length=255)
A book has many tags, each tag belongs to a tag category.
The Serializers
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
exclude = ['id', 'book']
class BookSerializer(serializers.ModelSerializer):
tags = TagSerializer(many=True, required=False)
class Meta:
model = Book
fields = ['title', 'tags']
def create(self, validated_data):
with transaction.atomic():
tags = validated_data.pop('tags')
book = Book.objects.create(**validated_data)
Tag.objects.bulk_create([Tag(book=book, **tag) for tag in tags])
return book
The Problem
I am trying to POST to the BookViewSet with the following example data:
{
"title": "The Jungle Book"
"tags": [
{ "page": 1, "category": 36 }, // plot intro
{ "page": 2, "category": 37 }, // character intro
{ "page": 4, "category": 37 }, // character intro
// ... up to 1000 tags
]
}
This all works, however, during the post, the serializer proceeds to make a call for each tag to check if the category_id is a valid one:
With up to 1000 nested tags in a call, I can't afford this.
How do I "prefetch" for the validation?
If this is impossible, how do I turn off the validation that checks if a foreign_key id is in the database?
EDIT: Additional Info
Here is the view:
class BookViewSet(views.APIView):
queryset = Book.objects.all().select_related('tags', 'tags__category')
permission_classes = [IsAdminUser]
def post(self, request, format=None):
serializer = BookSerializer(data=request.data)
if serializer.is_valid():
serializer.save()
return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

The DRF serializer is not the place (in my own opinion) to optimize a DB query. Serializer has 2 jobs:
Serialize and check the validity of input data.
Serialize output data.
Therefore the correct place to optimize your query is the corresponding view.
We will use the select_related method that:
Returns a QuerySet that will “follow” foreign-key relationships, selecting additional related-object data when it executes its query. This is a performance booster which results in a single more complex query but means later use of foreign-key relationships won’t require database queries.
to avoid the N+1 database queries.
You will need to modify the part of your view code that creates the corresponding queryset, in order to include a select_related call.
You will also need to add a related_name to the Tag.category field definition.
Example:
# In your Tag model:
category = models.ForeignKey(
'app.TagCategory', on_delete=models.PROTECT, related_name='categories'
)
# In your queryset defining part of your View:
class BookViewSet(views.APIView):
queryset = Book.objects.all().select_related(
'tags', 'tags__categories'
) # We are using the related_name of the ForeignKey relationships.
If you want to test something different that uses also the serializer to cut down the number of queries, you can check this article.

I think the issue here is that the Tag constructor is automatically converting the category id that you pass in as category into a TagCategory instance by looking it up from the database. The way to avoid that is by doing something like the following if you know that all of the category ids are valid:
def create(self, validated_data):
with transaction.atomic():
tags = validated_data.pop('tags')
book = Book.objects.create(**validated_data)
tag_instances = [ Tag(book_id=book.id, page=x['page'], category_id=x['category']) for x in tags ]
Tag.objects.bulk_create(tag_instances)
return book

I've come up with an answer that gets things working (but that I'm not thrilled about): Modify the Tag Serializer like this:
class TagSerializer(serializers.ModelSerializer):
category_id = serializers.IntegerField()
class Meta:
model = Tag
exclude = ['id', 'book', 'category']
This allows me to read/write a category_id without having the overhead of validations. Adding category to exclude does mean that the serializer will ignore category if it's set on the instance.

Problem is that you don't set created tags to the book instance so serializer try to get this while returning.
You need to set it to the book as a list:
def create(self, validated_data):
with transaction.atomic():
book = Book.objects.create(**validated_data)
# Add None as a default and check that tags are provided
# If you don't do that, serializer will raise error if request don't have 'tags'
tags = validated_data.pop('tags', None)
tags_to_create = []
if tags:
tags_to_create = [Tag(book=book, **tag) for tag in tags]
Tag.objects.bulk_create(tags_to_create)
# Here I set tags to the book instance
setattr(book, 'tags', tags_to_create)
return book
Provide Meta.fields tuple for TagSerializer (it's weird that this serializer don't raise error saying that fields tuple is required)
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
fields = ('category', 'page',)
Prefetching tag.category should be NOT necessary in this case because it's just id.
You will need prefetching Book.tags for GET method. The simplest solution is to create static method for serializer and use it in viewset get_queryset method like this:
class BookSerializer(serializers.ModelSerializer):
...
#staticmethod
def setup_eager_loading(queryset): # It can be named any name you like
queryset = queryset.prefetch_related('tags')
return queryset
class BookViewSet(views.APIView):
...
def get_queryset(self):
self.queryset = BookSerializer.setup_eager_loading(self.queryset)
# Every GET request will prefetch 'tags' for every book by default
return super(BookViewSet, self).get_queryset()

select_related function will check ForeignKey in the first time.
Actually,this is a ForeignKey check in the relational database and you can use SET FOREIGN_KEY_CHECKS=0; in database to close inspection.

How to display instances from two different django models on the same template, inside the same table?

In a django app I've created different models and everything looks okay until I try using data from two different models inside the same table.
To sum it up: in the homepage, I need to create a table that contains data from both the models, ordered by date.
The two models I need to display are the following.
models.py
class Document(models.Model):
number = models.CharField(max_length=10)
description = models.CharField(max_length=50)
assigned = models.BooleanField
validity_date = models.DateField
is_issued = models.BooleanField
class Program(models.Model):
name = models.CharField(max_length=25)
description = models.CharField(max_length=100)
validity_date = models.DateField
Then, I tried to create a view that would allow me to work with different models.
This is my view.py:
class BaseView(generic.ListView):
template_name = 'base/base_list.html'
context_object_name = 'base_list'
def get_queryset(self):
queryset = Document.objects.order_by('due_date')
return queryset
def get_context_data(self, **kwargs):
context = super(BaseView, self).get_context_data(**kwargs)
context['Programs'] = Program.objects.all()
context['Employees'] = Employee.objects.all()
return context
Now how can I create inside the template a table that shows both the models at once, ordering each entry by validity date (no matter if the entry belongs to Program or to Document)?
Thank you in advance!

You need to first query both the models, chain them together (Concatenate them) and then order by their shared attribute, that is the validity_date. You may do something like:
from itertools import chain
documents = Documents.objects.all()
programs = Program.objects.all()
final_combined_list = sorted(chain(documents,programs),key=lambda instance: instance.validity_date)
You can now simply pass final_combine_list to the template and render it to display it in the manner you want.

def get_context_data(self, **kwargs):
context = super(BaseView, self).get_context_data(**kwargs)
context['object_list'] = sorted(
itertools.chain(Document.objects.all(), Program.objects.all()),
key=lambda x: x.validity_date
)
return context