We Keep Coding

Annotation inside an annotation in Django Subquery?

I've got a few models and am trying to speed up the page where I list out users.
The issue is that I was leveraging model methods to display some of the data - but when I listed the Users out it was hitting the DB multiple times per User which ended up with hundreds of extra queries (thousands when there were thousands of User objects in the list) so it was a serious performance hit.
I've since began using annotate and prefetch_related which has cut the queries down significantly. I've just got one bit I can't figure out how to annotate.
I have a model method (on Summation model) I use to get a summary of Evaluation data for a user like this:
def evaluations_summary(self):
evaluations_summary = (
self.evaluation_set.all()
.values("evaluation_type__name")
.annotate(Count("evaluation_type"))
)
return evaluations_summary
I'm trying to figure out how to annotate that particular query on a User object.
So the relationship looks like this User has multiple Summations, but only one is ever 'active', which is the one we display in the User list. Each Summation has multiple Evaluations - the summary of which we're trying to show as well.
Here is a summary of the relevant parts of code (including the Summation model method which gives an example of what is currently 'working' to display the data as needed) - I have also made a pastebin example for easier viewing.
# MODELS
class User(AbstractUser):
employee_no = models.IntegerField(default=1)
...all the other usual attributes...
class Summation(CreateUpdateMixin, CreateUpdateUserMixin):
# CreateUpdateMixin adds 'created_at' & 'updated_at
# CreateUpdateUserMixin adds 'created_by' & 'updated_by'
employee = models.ForeignKey(
User, on_delete=models.PROTECT, related_name="%(class)s_employee"
)
report_url = models.CharField(max_length=350, blank=True)
...other unimportant attributes...
def evaluations_summary(self):
evaluations_summary = (
self.evaluation_set.all()
.values("evaluation_type__name")
.annotate(Count("evaluation_type"))
)
return evaluations_summary
class Evaluation(CreateUpdateMixin, CreateUpdateUserMixin):
summation = models.ForeignKey(Summation, on_delete=models.PROTECT)
evaluation_type = models.ForeignKey(
EvaluationType, on_delete=models.PROTECT
)
evaluation_level = models.ForeignKey(
EvaluationLevel, on_delete=models.PROTECT
)
evaluation_date = models.DateField(
auto_now=False, auto_now_add=False, null=True, blank=True
)
published = models.BooleanField(default=False)
class EvaluationLevel(CreateUpdateMixin):
name = models.CharField(max_length=50)
description = models.CharField(max_length=50)
class EvaluationType(CreateUpdateMixin):
name = models.CharField(max_length=50)
description = models.CharField(max_length=50)
evaluation_levels = models.ManyToManyField(EvaluationLevel)
# SERIALIZERS
class UserSerializer(serializers.HyperlinkedModelSerializer):
multiple_locations = serializers.BooleanField()
multiple_jobs = serializers.BooleanField()
summation_status_due_date = serializers.DateField()
summation_employee = SummationSerializer(many=True, read_only=True)
evaluations_summary = serializers.SerializerMethodField()
class Meta:
model = User
fields = [
"url",
"id",
"username",
"first_name",
"last_name",
"full_name",
"email",
"is_staff",
"multiple_locations",
"multiple_jobs",
"summation_status_due_date",
"summation_employee",
"evaluations_summary",
]
def get_evaluations_summary(self, obj):
return (
obj.summation_employee__evaluation_set.all()
.values("evaluation_type__name")
.annotate(Count("evaluation_type"))
)
# CURRENT ANNOTATIONS
# Subqueries for evaluation_summary
active_summations = (
Summation.objects.filter(employee=OuterRef("pk"), locked=False)
)
evaluations_set = (
Evaluation.objects.filter(summation__in=active_summations)
.order_by()
.values("evaluation_type__name")
)
summary_set = evaluations_set.annotate(Count("evaluation_type"))
# the 'summation_employee__evaluation_set' prefetch does not seem
# to make an impact on queries needed
user_list = (
User.objects.prefetch_related("summation_employee")
.prefetch_related("summation_employee__evaluation_set")
.filter(id__in=all_user_ids)
# Get the total locations and if > 1, set multiple_locations to True
.annotate(total_locations=Subquery(total_locations))
.annotate(
multiple_locations=Case(
When(total_locations__gt=1, then=Value(True)),
default=Value(False),
output_field=BooleanField(),
)
)
# Get the total jobs and if > 1 set mutiple_jobs to True
.annotate(total_jobs=Subquery(total_jobs))
.annotate(
multiple_jobs=Case(
When(total_jobs__gt=1, then=Value(True)),
default=Value(False),
output_field=BooleanField(),
)
)
# Get the due_date of the summation from the SummationStatus object
.annotate(
summation_status_due_date=Subquery(
summation_status.values("summation_due")
)
)
# I need to add the annotation here for the 'evaluations_summary' to avoid
# having the database hit for every user (which could possibly range into the
# thousands in certain cases)
# I have tried a number of ways to obtain what I'm looking for
.annotate(
evaluations_summary=Subquery(
evaluations_set.order_by()
.values("evaluation_type__name")
.annotate(Count("evaluation_type"))
)
)
# this annotation gives the error: Only one expression can be specified in the
# select list when the subquery is not introduced with EXISTS.
Is it even possible to transition that model method annotation?? Am I close?

Is it possible to reuse annotations from one model in another one in Django?

I'm using Django 2.2.16 and I was trying to get rid of replicated logic across querysets. I didn't manage to find a way (without impacting performance, like subquery in select clause) to reuse annotations from one model in annotations of another one.
Here is my current implementation:
# models.py
from django.contrib.gis.db import models
from querysets import PostManager, TagManager
class Tag(models.Model):
objects = TagManager()
name = CICharField(db_index=True, max_length=100)
class Post(models.Model):
objects = PostManager()
STATUS_ACCEPTED = "ACCEPTED"
STATUS_DECLINED = "DECLINED"
STATUS_CHOICES = (
(STATUS_ACCEPTED, "Accepted"),
(STATUS_DECLINED, "Declined"),
)
status = models.CharField(
default=STATUS_DECLINED,
choices=STATUS_CHOICES,
max_length=63,
verbose_name="Źródło ogłoszenia",
)
expiration_date = models.DateField(blank=True, null=True)
tags = models.ManyToManyField(Tag, blank=True, db_index=True)
# querysets.py
from django.db.models import Case, Count, Q, QuerySet, When, Manager
class PostQuerySet(QuerySet):
def annotate_statuses(self):
today = date.today()
return self.annotate(
live=Case(
When(
Q(status="ACCEPTED")
& (Q(expiration_date__isnull=True) | Q(expiration_date__gte=today)),
then=True,
),
default=False,
output_field=models.BooleanField(),
),
expired=Case(
When(
Q(status="ACCEPTED")
& (Q(expiration_date__isnull=False) & Q(expiration_date__lt=today)),
then=True,
),
default=False,
output_field=models.BooleanField(),
),
)
PostManager = Manager.from_queryset(PostQuerySet)
class TagQuerySet(QuerySet):
def annotate_live_listings():
today = date.today()
return queryset.annotate(
posts_count=Count("post"),
listings_count=Sum(
Case(
When(
Q(post__status="ACCEPTED")
& (Q(post__expiration_date__isnull=True) | Q(post__expiration_date__gte=today)),
then=1,
),
output_field=IntegerField(),
default=0,
)
),
)
TagManager = Manager.from_queryset(TagQuerySet)
My question is if it's possible to reimplement method annotate_live_listings in TagQuerySet to reuse somehow already implemented annotate_statuses from PostQuerySet. Currently, because of the code being more or less copied over, I've got logic of live annotation in two places.
Thanks in advance!

Add quantity for every the same Client and Product in Django

I have 3 models, CustomerPurchaseOrderDetail, Customer and Product model, if Customer1 buy a product for example, Meat. it will save in CustomerPurchaseOrderDetail and if that Customer1 add another Meat Product, instead of adding another record to the database it will simply just add quantity.
this is my views.py
def batchaddtocart(request):
userID = request.POST.get("userID")
client = Customer(id=userID)
vegetables_id = request.POST.get("id")
v = Product(id=vegetables_id)
price = request.POST.get("price")
discount = request.POST.get("discount_price")
insert, create = CustomerPurchaseOrderDetail.objects.get_or_create(
profile=client,
products=v,
unitprice=price,
quantity=1,
discounted_amount=discount,
discounted_unitprice=discount,
)
order_qs = CustomerPurchaseOrderDetail.objects.filter\
(
profile=client,
products=v,
unitprice=price,
quantity=1,
discounted_amount=discount,
discounted_unitprice=discount
)
for order in order_qs:
if order.profile == client and order.products == v:
insert.quantity += 1
print(insert.quantity)
insert.save()
insert.save()
this is my models.py
class CustomerPurchaseOrderDetail(models.Model):
profile = models.ForeignKey(Customer,
on_delete=models.SET_NULL, null=True, blank=True,
verbose_name="Client Account")
products = models.ForeignKey(Product,
on_delete=models.SET_NULL, null=True, blank=True,
verbose_name="Product")
quantity = models.IntegerField(max_length=500, null=True, blank=True, default=1)
class Product(models.Model):
product = models.CharField(max_length=500)
class Customer(models.Model):
user = models.OneToOneField(User, related_name="profile", on_delete=models.CASCADE)
firstname = models.CharField(max_length=500, blank=True)
lastname = models.CharField(max_length=500, blank=True)
contactNumber = models.CharField(max_length=500, blank=True)
email = models.CharField(max_length=500, blank=True)
I did not encounter an error but the functionality I wanted did not work. it does not add additional quantity even if the same product is added to the list purchased by Customer1.

the problem is in following lines client = Customer(id=userID) and v = Product(id=vegetables_id) every time your function is called you are creating new customer and product objects instead of using existing objects is your database. replace them with client,created = Customer.objects.get_or_create(id=userID) and same for product v,v_created = Product.objects.get_or_create(id=vegetables_id)

When you use get_or_create method, it will create a new entry whenever at least one parameter is different enough to did not match any registered value. So, if you pass the quantity parameter equals 1, it always will create a new entry when quantity is 2+, for instance.
You should filter first with only the "fixed" parameters and create a new entry if you get nothing. Otherwise, just increment quantity.
Something like this:
order = None
order_qs = CustomerPurchaseOrderDetail.objects.filter\
(
profile=client,
products=v,
unitprice=price,
discounted_amount=discount,
discounted_unitprice=discount
)
if not order_qs:
order = CustomerPurchaseOrderDetail.objects.create(
profile=client,
products=v,
unitprice=price,
quantity=1,
discounted_amount=discount,
discounted_unitprice=discount,
)
else:
for order in order_qs:
if order.profile == client and order.products == v:
# NOTE: Please, note if you need to check your other parameters too (unityprice, discounted....)
order.quantity += 1
print(order.quantity)
order.save()

Django Models - How do you add subtype choice when user selects choice?

I'm working on a project where they have various job types that I've tackled with CHOICES, however, I want to add conditionals for WHEN job type 1 is chosen, SUBTYPES x-y become choices. I am having trouble with the syntax of how you would do that. I've included my pseudocode below... I appreciate any help!
from django.db import models
class User(models.Model):
name = models.CharField(max_length=255)
def __str__(self):
return self.name
class Job(models.Model):
name = models.CharField(max_length=255)
user = models.ForeignKey(User, on_delete=models.CASCADE, related_name='jobs')
JOB_CHOICES = (
('carpentry', 'Carpentry'),
('cleaning', 'Cleaning'),
('electrician', 'Electrician'),
('handyman', 'Handyman'),
('hvac', 'HVAC'),
('painting', 'Painting'),
('pest', 'Pest'),
('plumbing', 'Plumbing'),
('roofing', 'Roofing'),
('property', 'Property'),
)
jobType = models.CharField(max_length=30, choices=JOB_CHOICES, default='handyman')
# If JobType = Carpentry:
# CARPENTRY_CHOICES = (
# ('trim', 'trim')
# ('sheetrock', 'Sheetrock')
# ('windows', 'Windows')
# ('doors', 'Doors')
# ('cabinets and shelving', 'Cabinets and Shelving')
# ('other', 'Other')
# )
# jobType = models.CharField(max_length=30, choices=CARPENTRY_CHOICES, default='other')
def __str__(self):
return self.name
Django Models
Django Serializer
/api editor

I would probably go with a job_type model, which has a name and a 'subtype' field.
class JobType(models.Model):
SubTypeChoices = (...)
name = models.CharField()
subtype = models.CharField(choices=SubTypeChoices, ...)
class Job(models.Model):
....
job_type = models.ForeignKey(JobType, ...)
....
This way you can associate your 'subtypes' with one job_type. And if for some reason you can have several job_types for a Job, use a ManyToMany field.

Django filter only on aggregate/annotate

I'm trying to construct a fairly complicated Django query and I'm not making much progress. I was hoping some wizard here could help me out?
I have the following models:
class Person(models.Model):
MALE = "M"
FEMALE = "F"
OTHER = "O"
UNKNOWN = "U"
GENDER_CHOICES = (
(MALE, "Male"),
(FEMALE, "Female"),
(UNKNOWN, "Other"),
)
firstName = models.CharField(max_length=200, null=True, db_column="firstname")
lastName = models.CharField(max_length=200, null=True, db_column="lastname")
gender = models.CharField(max_length=1, choices=GENDER_CHOICES, default=UNKNOWN, null=True)
dateOfBirth = models.DateField(null=True, db_column="dateofbirth")
dateInService = models.DateField(null=True, db_column="dateinservice")
photo = models.ImageField(upload_to='person_photos', null=True)
class SuccessionTerm(models.Model):
originalName = models.CharField(max_length=200, null=True, db_column="originalname")
description = models.CharField(max_length=200, blank=True, null=True)
score = models.IntegerField()
class Succession(model.Model):
position = models.ForeignKey(Position, to_field='positionId', db_column="position_id")
employee = models.ForeignKey(Employee, to_field='employeeId', db_column="employee_id")
term = models.ForeignKey(SuccessionTerm)
class Position(models.Model):
positionId = models.CharField(max_length=200, unique=True, db_column="positionid")
title = models.CharField(max_length=200, null=True)
# There cannot be a DB constraint, as that would make it impossible to add the first position.
dottedLine = models.ForeignKey("Position", to_field='positionId', related_name="Dotted Line",
null=True, db_constraint=False, db_column="dottedline_id")
solidLine = models.ForeignKey("Position", to_field='positionId', related_name="SolidLine",
null=True, db_constraint=False, db_column="solidline_id")
grade = models.ForeignKey(Grade)
businessUnit = models.ForeignKey(BusinessUnit, null=True, db_column="businessunit_id")
functionalArea = models.ForeignKey(FunctionalArea, db_column="functionalarea_id")
location = models.ForeignKey(Location, db_column="location_id")
class Employee(models.Model):
person = models.OneToOneField(Person, db_column="person_id")
fte = models.IntegerField(default=100)
dataSource = models.ForeignKey(DataSource, db_column="datasource_id")
talentStatus = models.ForeignKey(TalentStatus, db_column="talentstatus_id")
retentionRisk = models.ForeignKey(RetentionRisk, db_column="retentionrisk_id")
retentionRiskReason = models.ForeignKey(RetentionRiskReason, db_column="retentionriskreason_id")
performanceStatus = models.ForeignKey(PerformanceStatus, db_column="performancestatus_id")
potential = models.ForeignKey(Potential, db_column="potential_id")
mobility = models.ForeignKey(Mobility, db_column="mobility_id")
currency = models.ForeignKey(Currency, null=True, db_column="currency_id")
grade = models.ForeignKey(Grade, db_column="grade_id")
position = models.OneToOneField(Position, to_field='positionId', null=True,
blank=True, db_column="position_id")
employeeId = models.CharField(max_length=200, unique=True, db_column="employeeid")
dateInPosition = models.DateField(null=True, db_column="dateinposition")
Now, what I want is for each employee to get the position title, the person's name, and for each succession term (of which there are three) how many times the position of that employee is in the succession table, and the number of times each of these employees occurs in the successors table. Above all, I want to do all of this in a singe query (or more specifically, a single Django ORM statement), as I'm doing this in a paginated way, but I want to be able to order the result on any of these columns!
So far, I have this:
emps = Employee.objects.all()
.annotate(ls_st=Count('succession__term'))
.filter(succession__term__description="ShortTerm")
.order_by(ls_st)
.prefetch_related('person', 'position')[lower_limit:upper_limit]
This is only one of the succession terms, and I would like to extend it to all terms by adding more annotate calls.
My problem is that the filter call works on the entire query. I would like to only filter on the Count call.
I've tried doing something like Count(succession__term__description'="ShortTerm") but that doesn't work. Is there any other way to do this?
Thank you very much in advance,
Regards,
Linus

So what you want is a count of each different type of succession__term? That is pretty complex, and I don't think you can do this with the built in django orm right now. (unless you did a .extra() query)
In django 1.8, I believe you will be able to do it with the new Query Expressions (https://docs.djangoproject.com/en/dev/releases/1.8/#query-expressions). But of course 1.8 isn't released yet, so that doesn't help you.
In the meantime, you can use the very handy django-aggregate-if package. (https://github.com/henriquebastos/django-aggregate-if/, https://pypi.python.org/pypi/django-aggregate-if)
With django-aggregate-if, your query might look like this:
emps = Employee.objects.annotate(
ls_st=Count('succession__term', only=Q(succession__term__description="ShortTerm")),
ls_lt=Count('succession__term', only=Q(succession__term__description="LongTerm")), # whatever your other term descriptions are.
ls_ot=Count('succession__term', only=Q(succession__term__description="OtherTerm"))
)
.order_by('ls_st')
.prefetch_related('person', 'position')[lower_limit:upper_limit]
Disclaimer: I have never used django-aggregate-if, so I'm not entirely sure if this will work, but according the the README, it seems like it should.