We Keep Coding

How to do SELECT COUNT(*) GROUP BY of ForeignKey field in Django?

Let's say we have three models:
class Category(models.Model):
name = models.CharField(max_length=255)
class Platform(models.Model):
name = models.CharField(max_length=255)
class Product(models.Model):
name = models.CharField(max_length=255)
category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='products')
platform = models.ForeignKey(Platform, on_delete=models.CASCADE, related_name='products')
class SellingItem(models.Model):
name = models.CharField(max_length=255)
product = models.ForeignKey(Product, on_delete=models.CASCADE, related_name='selling_items')
price = models.DecimalField(max_digits=5, decimal_places=2)
The idea here is to get a total of the different categories and platforms based on a SellingItem's queryset.
My initial approach is:
categories = queryset.values('product__category__id', 'product__category__name', 'product__category__label').annotate(total=Count('product__category__id')) # noqa: E501
platforms = queryset.exclude(product__platform__isnull=True).values('product__platform__id', 'product__platform__name', 'product__platform__label').annotate(total=Count('product__platform__id')) # noqa: E501
The problem is that the result is not grouped...
The code is working on based on a Product's queryset if removing all product__ in the previous code.

Your approach towards the problem is absolutely correct. Although you are missing a small thing at the end which is '.order_by()'. If you add it, you should get your desired result. What this does is that it will tell django to not apply any type of ordering. Thus, you will achieve your result.
This issue occurred because django tried to apply default ordering it has thus your queryset didn't output the desired results.
categories = queryset.values('product__category__id', 'product__category__name', 'product__category__label').annotate(total=Count('product__category__id')).order_by() # order_by at the end.
platforms = queryset.exclude(product__platform__isnull=True).values('product__platform__id', 'product__platform__name', 'product__platform__label').annotate(total=Count('product__platform__id')).order_by() # order_by at the end.

Django rest framework simple model serializer list view takes forever to show 10 records

I have this model -
class News(BaseEntityBasicAbstract, HitCountMixin):
"""
News added from the dashboard with content
"""
NEWS_STATUS = (
('draft', _('Draft')),
('pending', _('Pending')),
('review', _('Review')),
('public', _('Public')),
('private', _('Private'))
)
backup = models.BooleanField(default=False)
prev_id = models.BigIntegerField(null=True, blank=True)
language = models.CharField(max_length=10, choices=LANGUAGES, default='bn', db_index=True)
heading = models.CharField(max_length=255, null=True, blank=True,
verbose_name=_('News Heading'),
help_text=_('Provide a news heading/caption.'))
sub_caption = models.TextField(max_length=255, null=True, blank=True,
verbose_name=_('Summary'),
help_text=_('Provide summary of the news.'))
url = models.CharField(max_length=255, unique=True, verbose_name=_('URL/Slug/Link'),
help_text=_('Unique url for the news without whitspace.'))
content = HTMLField(null=True, blank=True, verbose_name=_('Content'),
help_text=_('HTML content with texts, links & images.'))
featured_image = models.FileField(upload_to=FilePrefix('news/'), null=True, blank=True,
verbose_name=_('Featured Image'),
help_text=_('Upload a featured image for news.'))
image_caption = models.TextField(max_length=255, null=True, blank=True,
verbose_name=_('Image Caption'),
help_text=_('Provide a image caption.'))
status = models.CharField(max_length=20, choices=NEWS_STATUS, default='pending',
verbose_name=_('News Status'), db_index=True,
help_text=_('Only public news can be seen on front end.'))
source = models.ForeignKey(NewsSource, on_delete=models.SET_NULL, null=True, blank=True,
verbose_name=_('News Source'),
help_text=_('Select a news source.'))
category = models.ForeignKey(Category, on_delete=models.SET_NULL, null=True, blank=True,
verbose_name=_('Category'),
help_text=_('Select a news category.'))
tags = tagulous.models.TagField(
blank=True,
to=Tags,
verbose_name=_('News Tags'),
help_text=_('Provide news tags separated with commas.')
)
published_at = models.DateTimeField(null=True, blank=True, db_index=True,
verbose_name=_('Published At'))
menu_items = GenericRelation(MenuItems, object_id_field='id',
related_query_name='news_as_menu')
hit_count_generic = GenericRelation(HitCount, object_id_field='object_pk',
related_query_name='news_hit_count')
created_by = models.ForeignKey(User, related_name='news_created_by',
on_delete=models.SET_NULL, null=True, blank=True,
verbose_name=_('Created By'))
updated_by = models.ForeignKey(User, related_name='news_updated_by',
on_delete=models.SET_NULL, null=True, blank=True,
verbose_name=_('Last Updated By'))
published_by = models.ForeignKey(User, related_name='news_published_by',
on_delete=models.SET_NULL, null=True, blank=True,
verbose_name=_('Published By'))
deleted_by = models.ForeignKey(User, related_name='news_deleted_by',
on_delete=models.SET_NULL, null=True, blank=True,
verbose_name=_('Deleted By'))
Below is the serializer -
class NewsSerializer(serializers.ModelSerializer):
class Meta:
model = News
fields = ['id', 'heading', 'sub_caption', 'url', 'content', 'featured_image',
'image_caption', 'category', 'source', 'tags', 'published_at']
This is the view -
class NewsViewSets(viewsets.ModelViewSet):
queryset = News.objects.filter(
is_active=True,
status='public'
)
serializer_class = NewsSerializer
def get_queryset(self):
queryset = self.queryset.filter(
language=self.request.LANGUAGE_CODE
).order_by('-id')
return queryset
Pagination is set to only 10 in the settings.py, when I hit the news api url it takes 9/10 seconds to load only 10 records. Here's a screenshot showing django-debug-toolbar reports -
I have around 400k records on the database table, it may be an issue, but I think this is too much loading time. Please help me find the problem here! Thanks in advance.

Filtering can often be slow. It looks like you have database indexes on the relevant fields, but take note that if you have multiple filters, only one of the indexes will be used.
I'm guessing based on your columns but it seems like the most common query will always be looking for is_active=1 and status='public'. If this isn't the case you might have to make some tweaks.
Firstly, get rid of the db_index=True on the status, is_active, and language fields, otherwise your database writes will be slowed unnecessarily.
Then you can formulate an index such as this:
class Meta:
indexes = [
models.Index(
fields=["is_active", "status", "language"],
name="idx_filtering",
)
]
This will help the database when you filter on all three columns at once. If you're ever filtering on only one of these columns however, you may want to keep the original db_index=True.
If you were using PostgreSQL, you could do one better:
class Meta:
indexes = [
models.Index(
fields=["is_active", "status", "language"],
name="idx_filtering",
condition=Q(is_active=True, status="public"),
)
This would reduce the size of the index to only those matching the Q(), making traversing it faster.
One other thing to note is that pagination using OFFSET is very slow once you get to higher offsets. If at all possible you should be using DRF's cursor pagination instead.

Django: How to conditionally define fields in a model mixin?

In my back-end (API) site, there are certain fields that are common to most models, but not all. I have been using mixins for those fields so that I can include them for the models they apply to, and omit them from the ones they don't. For example:
class AddressPhoneModelMixin(models.Model):
address = models.TextField(
verbose_name=_('Address'),
blank=True,
null=True,
)
country = models.ForeignKey(
Country,
on_delete=models.SET_NULL,
verbose_name=_('Country'),
blank=True,
null=True,
)
phone_number = PhoneNumberField(
verbose_name=_('Phone'),
blank=True,
null=True,
)
mobile_number = PhoneNumberField(
verbose_name=_('Mobile Phone'),
blank=True,
null=True,
)
fax_number = PhoneNumberField(
verbose_name=_('Fax'),
blank=True,
null=True,
)
class Meta:
abstract = True
But I have other such mixins, and when a model needs to include all fields, the model definition gets to be quite long:
class Client(AddressPhoneModelMixin, DateFieldsModelMixin, models.Model):
And I now have other "common" fields I want to add, so it's only going to get worse. I want to keep all these common fields in one place for DRY, but also in case anything changes about a field, I only have one place to make the changes.
My idea is to have one mixin called CommonFieldsModelMixin, so that I will have only one mixin to include in the model definition. But for those models that don't need certain fields, the field definitions would all be wrapped in conditionals. Taking the above mixin as an example, and adding a conditional "Email Address" field, this is what I want to do:
class CommonFieldsModelMixin(models.Model):
address = models.TextField(
verbose_name=_('Address'),
blank=True,
null=True,
)
country = models.ForeignKey(
Country,
on_delete=models.SET_NULL,
verbose_name=_('Country'),
blank=True,
null=True,
)
if include_email: # <--this is what I want to add
email = models.EmailField(
verbose_name=_('Email Address'),
blank=True,
)
phone_number = PhoneNumberField(
verbose_name=_('Phone'),
blank=True,
null=True,
)
mobile_number = PhoneNumberField(
verbose_name=_('Mobile Phone'),
blank=True,
null=True,
)
fax_number = PhoneNumberField(
verbose_name=_('Fax'),
blank=True,
null=True,
)
class Meta:
abstract = True
Then when using the mixin on a model, it would be something like this:
class Client(CommonFieldsModelMixin, models.Model):
include_email = True
name = models.CharField(
verbose_name=_('Client'),
max_length=100,
)
status = models.CharField(
verbose_name=_('Status'),
max_length=25,
)
Notice the include_email = True property. In reality, all fields would be wrapped in conditionals, but this is intended as a simple example.
My question is, how can I access the include_email property of the parent from within the mixin? There isn't a self to use. I've also tried using super(), but that didn't work either. Is there any way to accomplish this?
And I'll also need to do the same (or similar) thing for the serializers. So if that would work differently, any suggestions there would be appreciated.

How can I query a ManyToMany relation table for a custom field

As can be seen in the code below, I have an intermediate model that defines a custom date field. How can I reference the date_assigned field in the relation model. I am trying find the number of tenders that were assigned to the "user" (i.e. CompanyProfile) on a particular date as per the date_assigned field in the relation model. The CompanyProfile (i.e. user) model and the Tender model share one thing in common, they both have a relationship with the Keywords model through a ManyToMany relationship. This is how I am able to find Tenders allocated to the CompanyProfile through the Keywords model.
This is the final result I am aiming for.
class CompanyProfile(models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE, primary_key=True)
accountNumber = models.CharField(max_length=25, default=1, blank=False, null=False)
companyName = models.CharField(max_length=200, blank=False)
companyRegNum = models.CharField(max_length=30, blank=True)
contactNumber = models.CharField(max_length=20, blank=False)
address = models.CharField(max_length=300, blank=True)
areaCode = models.CharField(max_length=10, blank=False)
deliveryEmails = models.TextField(blank=True) #this is the list of all the people chosen to recieve daily notification.
tenderCategory = models.ManyToManyField(Category, blank=False) #links the user to the chosen category.
provinces = models.ManyToManyField(Province, blank=False) #links the user to the chosen Provinces.
package = models.ForeignKey(Packages, default=1, blank=False) #links the user to the chosen package.
pymntMethod = models.IntegerField(blank=True, default=3) #this is the chosen payment method (e.g credit card=1, debit order=2 or direct debit=3)
keywords = models.ManyToManyField(Keywords) #links the user to the chosen keywords.
extraKeywords = models.TextField(default='', blank=True) #this field acts as a container of extra keywords from the user. These are keywords that we do not have in our database.
contractDuration = models.IntegerField(blank=False, default=12)
termsAndConditions = models.BooleanField(blank=False, default=1) #this is the T&C's field that must be agreed to by the client.
commencementDate = models.DateTimeField(default=timezone.now, blank=True)
class Keywords(models.Model):
keyword = models.CharField(max_length=150)
class Meta:
verbose_name_plural = ('Keywords')
ordering = ['keyword', ]
def __str__(self):
return self.keyword
#This is the model that stores the tender.
class Tender(models.Model):
tenderCategory = models.ManyToManyField(category, blank=False) #this field holds the tender category, e.g. construction, engineering, human resources etc.
tenderProvince = models.ForeignKey(Province, default=1, blank=False) #this is the province the tender was advertised from.
keywordTags = models.TextField(blank=False) #this field holds keywords for the tender as per the tender title or as determined by the tender capturer.
buyersName = models.CharField(max_length=100) #this is the name of the Buyer e.g. Dept. of Transport, Transnet, Dept of Agriculture etc.
summary = models.TextField(blank=False) #this is the tender title as per the Buyer.
refNum = models.CharField(max_length=100) #tender ref number as per the Buyer.
issueDate = models.DateTimeField(blank=True, null=True) #date the tender was published
closingDate = models.DateTimeField(blank=True, null=True) #tender closing date
siteInspection = models.TextField(blank=True, null=True) #site inspection date, if any
enquiries = models.TextField(blank=True, null=True) #this field stores details of the contact person, for the tender.
description = models.TextField(blank=True, null=True) #this is the body of the tender. the tender details are captured here.
assigned_keywords = models.ManyToManyField(Keywords, blank=True, through='tenderKeywords')
matched = models.BooleanField(default=0, blank=False)
capture_date = models.DateField(default=timezone.now, blank=False, null=False)
class TendersKeywords(models.Model):
tender = models.ForeignKey(tender, related_name='tender_keywords')
keyword = models.ForeignKey(Keywords, related_name='tender_keywords')
date_assigned = models.DateField(default=timezone.now, blank=False, null=False)
I am able to find the associated tenders for the CompanyProfile but I have a problem aggregating the results base on the tender issued_date which is defined in the relation model TendersKeywords.

The expected result you show is a model aggregation query. In particular, computing a count of matches grouped by a particular field, is a good use of the Count aggregation, applied to the queryset with an annotate operation.
import django.db
Tender.objects.values('issued').annotate(
tenders=django.db.models.Count('issued'))
What that does:
Use the Tender model manager (the Tender.objects attribute);
Use the values method to make a queryset containing just the issued field value;
Annotate that queryset with a field named tenders, that:
Has the value computed by the Count aggregate function for each group.
That all assumes a coherent model schema; I had to make some guesses (your code as presented doesn't work, please review the guidelines on creating a minimal, complete, verifiable example. In brief: Strip the example down to the minimum needed, and then actually run it to make sure it'll work for us too).
If you have an existing query and want to annotate that, then just apply the same technique:
existing_queryset.values('tender_issued').annotate(
tenders=django.db.models.Count('tender_issued'))
where tender_issued is whatever field in your queryset contains the value you want to group and count.

First, I want to modify your models
class tenderKeywords(models.Model):
tender = models.ForeignKey(tender, related_name='tenderkeywords')
keyword = models.ForeignKey(Keywords, related_name='tenderkeywords')
date_assigned = models.DateField(default=timezone.now, blank=False, null=False)
then:
count = tender.objects.filter(tenderKeywords__keyword__Keywords="", tenderKeywords__date_assigned= datetime)
Python developers have some standard in writing code. for example first letter of model's name should be capital. thanks for this.

Django models - related objects validation

I'm wondering how to have validations for related objects. To my surprise I haven't found much relevant information on this.
For example:
class Listing(models.Model):
categories = models.ManyToManyField('Category')
price_sale = models.DecimalField(max_digits=8, decimal_places=0, null=True)
price_rent = models.DecimalField(max_digits=8, decimal_places=0, null=True)
price_vacation = models.DecimalField(max_digits=8, decimal_places=0, null=True)
class Category(models.Model):
value = models.CharField(max_length=32)
class Image(models.Model):
listing = models.ForeignKey('Listing')
image = models.ImageField(upload_to=get_file_path)
How can I make sure that at least one category is set, there are no
duplicates for the listing?
How can I make sure that if one of the categories is 'sale', price_sale must be set or else set to null?
How can I make sure that at least one image is inserted but not more
than say 10 images?
I'm thinking this should be done in the model in case I choose to input data aside from forms (something like parsing a feed), would this be correct? I tried dealing with clean() but it requires a PK before letting me deal with m2m relationships, etc.
Bonus question: Why would I choose to limit a field using choices rather than limiting by FK?

Try explicitly creating your mapping table, and have your ManyToMany relationship go through this model. Since it is a normal Django model you should be able to define most of your validation logic within its clean method.
class Listing(models.Model):
categories = models.ManyToManyField('Category', through='CategoryListing')
price_sale = models.DecimalField(max_digits=8, decimal_places=0, null=True)
price_rent = models.DecimalField(max_digits=8, decimal_places=0, null=True)
price_vacation = models.DecimalField(max_digits=8, decimal_places=0, null=True)
class Category(models.Model):
value = models.CharField(max_length=32)
class CategoryListing(models.Model):
category = models.ForeignKey(Category)
listing = models.ForeignKey(Listing)
def clean(self):
# validation logic
https://docs.djangoproject.com/en/1.3/topics/db/models/#intermediary-manytomany