We Keep Coding

Django aggregate calculating incorrectly because of many-to-many join in queryset

We have a model with a many-to-many relationship with another model, and doing a Sum aggregation on it seems to multiply the result by the number of many-to-many relationships. Example models below:
class Attendee(models.Model):
name = models.TextField()
category = models.TextField()
class Event(models.Model):
start_date = models.DateTimeField()
end_date = models.DateTimeField()
attendees = models.ManyToManyField(Attendee, related_name="event")
class CalendarEvent(models.Model):
calendar = models.TextField()
event = models.ForeignKey(Event, related_name="calendar_events")
We filter out some records based on Attendee, and then aggregate the duration of the events.
queryset = queryset.filter(event__attendee__category="Accountant")
queryset = queryset.annotate(duration=models.ExpressionWrapper(
models.F('event__end_date') - models.F('event__start_date'),
output_field=models.fields.DurationField()
))
Whenever we run the aggregation at the end, all duration values would be multiplied by the number of Attendees associated with the Event/CalendarEvent. For example, if the event is 15 minutes long and has 4 attendees, we would expect the duration to be 900 seconds, but instead it's 3600 seconds. We noticed in the raw PSQL query that there is a LEFT OUTER JOIN against Attendees, so we tried removing the Attendee filter and it gave us the proper data.
Is there any way to aggregate across a relationship like this without the joined results getting mixed in?

LEFT JOIN with other param in ON Django ORM

I have the following models:
class Customer(models.Model):
name = models.CharField(max_length=255)
email = models.EmailField(max_length = 255, default='example#example.com')
authorized_credit = models.IntegerField(default=0)
balance = models.IntegerField(default=0)
class Transaction(models.Model):
customer = models.ForeignKey(Customer, on_delete=models.CASCADE)
payment_amount = models.IntegerField(default=0) #can be 0 or have value
exit_amount = models.IntegerField(default=0) #can be 0 or have value
transaction_date = models.DateField()
I want to query for get all customer information and date of last payment.
I have this query in postgres that is correct, is just that i need:
select e.*, max(l.transaction_date) as last_date_payment
from app_customer as e
left join app_transaction as l
on e.id = l.customer_id and l.payment_amount != 0
group by e.id
order by e.id
But i need this query in django for an serializer. I try with that but return other query
In Python:
print(Customer.objects.filter(transaction__isnull=True).order_by('id').query)
>>> SELECT app_customer.id, app_customer.name, app_customer.email, app_customer.balance FROM app_customer
LEFT OUTER JOIN app_transaction
ON (app_customer.id = app_transaction.customer_id)
WHERE app_transaction.id IS NULL
ORDER BY app_customer.id ASC
But that i need is this rows
example

Whether you are working with a serializer or not you can reuse the same view/function for both the tasks.
First to get the transaction detail for the current customer object you have you have to be aware of related_name.related_name have default values but you can mention something unique so that you remember.
Change your model:
class Transaction(models.Model):
customer = models.ForeignKey(Customer, related_name="transac_set",on_delete=models.CASCADE)
related_names are a way in django to create reverse relationship from Customer to Transaction this way you will be able to do Customer cus.transac_set.all() and it will fetch all the transaction of cus object.
Since you might have multiple customers to get transaction details for you can use select_related() when querying this will hit the database least number of times and get all the data for you.
Create a function definition to get the data of all transaction of Customers:
def get_cus_transac(cus_id_list):
#here cus_id_list is the list of ids you need to fetch
cus_transac_list = Transaction.objects.select_related(customer).filter(id__in = cus_id_list)
return cus_transac_list
For your purpose you need to use another way that is the reason you needed related_name, prefetch_related().
Create a function definition to get the data of latest transaction of Customers: ***Warning: I was typing this answer before sleeping so there is no way the latest value of transaction is being fetched here.I will add it later but you can work on similar terms and get it done this way.
def get_latest_transac(cus_id_list):
#here cus_id_list is the list of ids you need to fetch
latest_transac_list = Customer.objects.filter(id__in = cus_id_list).prefetch_related('transac_set')
return latest_transac_list
Now coming to serializer,you need to have 3 serializers (Actually you need 2 only but third one can serialize Customer data + latest transaction that you need) ...one for Transaction and another for customer then the 3rd Serializer to combine them.
There might be some mistakes in code or i might have missed some details.As i have not checked it.I am assuming you know how to make serializers and views for the same.

One approach is to use subqueries:
transaction_subquery = Transaction.objects.filter(
customer=OuterRef('pk'), payment_amount__gt=0,
).order_by('-transaction_date')
Customer.objects.annotate(
last_date_payment=Subquery(
transaction_subquery.values('transaction_date')[:1]
)
)
This will get all customer data, and annotate with their last transaction date that has payment_amount as non-zero, in one query.

To solve your problem:
I want to query for get all customer information and date of last payment.
You can try use order by combine with distinct:
Customer.objects.prefetch_related('transaction_set').values('id', 'name', 'email', 'authorized_credit', 'balance', 'transaction__transaction_date').order_by('-transaction__transaction_date').distinct('transaction__transaction_date')
Note:
It only applies to PostgreSQL when distinct followed by parameters.
Usage of distinct: https://docs.djangoproject.com/en/3.2/ref/models/querysets/#distinct

Defining a Django query set with a many-to-one relationship with calculation

I have three models: Assets, AssetTypes and Services. Assets need to get serviced every n months, and have a many-to-one relation with services.
class AssetType(models.Model):
id = models.AutoField(primary_key=True)
name = models.CharField(verbose_name="Asset Type", max_length=100)
service_period = models.PositiveSmallIntegerField(verbose_name="Service Period (in months)", null=True, blank=True, default=12)
class Asset(models.Model):
id = models.AutoField(primary_key=True)
type = models.ForeignKey(AssetType, on_delete=models.PROTECT)
def service_period(self):
return AssetType.objects.get(pk=self.type.id).service_period
def service_history(self):
return self.service_set.all().order_by('-date')
def service_due_date(self):
if self.service_period()==None:
return None
elif self.service_history().count()==0:
return datetime.strptime('2017-01-01', '%Y-%m-%d').date()
else:
last_service_date = self.service_history().latest('date').date
return last_service_date + timedelta(self.service_period()*30)
def service_overdue(self):
return ((self.service_due_date()!=None) and self.service_due_date() < date.today())
class Service(models.Model):
id = models.AutoField(primary_key=True)
date = models.DateField()
asset = models.ForeignKey(Asset, on_delete=models.CASCADE)
I'm trying to work out how to make a query set that would return a list of assets that are overdue for their service. I feel like using a model method is a red herring, and that I would be better off defining a query set filter?
I need the list of overdue assets to be a query set so I can use further query set filters on it.
Any assistance would be greatly appreciated!

So this is a bit tricky.
To put the query in words, you are looking for all Assets whose latest Service date is earlier than today minus the Type's service period multiplied by 30.
To be honest, I would be tempted to denormalize this; you could add a next_service_due field on Asset which is updated when you add a new Service. Then the query is simply all assets with that field less than today.

How to filter queryset not equal to in Django?

I have models called Stores and SaleItems which look some what like this.
class Stores(models.Model):
name = models.CharField(unique=True, max_length=20)
class SaleItems(models.Model):
sale_by = models.ForeignKey(Stores)
start_date = models.DateTimeField()
end_date = models.DateTimeField()
So I need to retrieve sale items based on the following conditions.
If its not my store, then leave out items having start_date greater than today and end_date lesser than today.
If its my store, then get items irrespective of start_date and end_date.
So in my views.py, this is how far I have come up.
class SaleItemsView(viewsets.ModelViewSet):
querys = SaleItems.objects.all()
def get_queryset(self):
#get my store id
store_id = self.request.query_params.get('store_id')
querys = SaleItems.objects\
.exclude(store__ne=store_id, end_date__lt=timezone.now())\
.exclude(store__ne=store_id, start_date__gt=timezone.now())
return querys
But it seems django doesn't have not equal operator any more. Is there any other way I can achieve this?

I think you would combine the records for each of your conditions using Qobjects, then do distinct() to remove duplicates:
now = timezone.now()
items_within_date_range = Q(start_date__lte=today, end_date__gte=today)
curr_store_items = Q(store=store_id)
result = SaleItems.objects.filter(items_within_date_range|curr_store_items).distinct()
The first query would get all items within time range for both your store and other stores, second query would get all items for your store, then combine then would get everything.

Django sorting/order_by a list of model objects with a computed value

I have two models a Slot and an Appointment.
Slot objects are basically appointment requests by Users and they are defined for week_starting DateField, and I need to match two slots to create an Appointment object for that week_starting. I am iterating slots and creating appointments as follows:
while Slot.objects.filter(week_starting=next_weekday(datetime.datetime.today(), 0), is_matched=False).count() > 1:
slot = Slot.objects.filter(week_starting=next_weekday(datetime.datetime.today(), 0), is_matched=False)[:1].get()
users = User.objects.filter(~Q(id=slot.user.id))
min_count = Appointment.objects.filter(users_id=slot.user.id).filter(users_id=users[0].id).count()
for user in users:
if Appointment.objects.filter(users_id=slot.user.id).filter(users_id=user.id).count() < min_count and Slot.objects.filter(Q(user=user) & Q(is_matched=False) & Q(week_starting=next_weekday(datetime.datetime.today()))).count() > 0:
min_count = Appointment.objects.filter(users_id=slot.user.id).filter(users_id=user.id).count()
temp_slot = Slot.objects.filter(Q(user=user) & Q(is_matched=False) & Q(week_starting=next_weekday(datetime.datetime.today())))[:1].get()
slot_to_match = temp_slot
appointment = Appointment(week_starting=slot.week_starting)
appointment.users.add(slot.user)
appointment.users.add(slot_to_match.user)
appointment.save()
slot.is_matched = True
slot.matched_on = datetime.datetime.now()
slot.save()
slot_to_match.is_matched = True
slot_to_match.matched_on = datetime.datetime.now()
slot_to_match.save()
next_weekday is a method where I find the next monday's date.
A new requirement has just popped up, and apparently I need to match the slots according to their users' previous appointments count together, i.e. For any slot, I need to find another one, of which the User has fewer appointment count than with anybody else.
Slot model:
class Slot(models.Model):
user = models.ForeignKey(User)
week_starting = models.DateField()
is_matched = models.BooleanField(default=False, blank=False, null=False)
matched_on = models.DateTimeField(blank=True, null=True)
Appointment model:
class Appointment(models.Model):
week_starting = models.DateField()
start_date = models.DateTimeField(null=True, blank=True)
users = models.ManyToManyField(User, related_name='appointment_users')
supervisor = models.OneToOneField(User, related_name='appointment_supervisor', null=True, blank=True, unique=False)
I need to match two Slot objects that is not is_matched to create an Appointment object. I need to do it for every single Slot object. When doing that, I need to match two users who have the least number of Appointments together.
I don't know any other solutions but looping. Any thoughts?

You may need to tweak this for related names and the like but something like the below should help. If I understand you correctly you need yourSlot objects to be aware of their User's Appointment counts.
# define as a method of your Slot class
def user_appointment_count(self):
return [(x, x.appointment_set.count()) for x in self.user_set.all()]
That will return a list of 2-tuples, where the first item is the user and the second is their appointment count. Taking the first two items in such a list will give you the two related users with the fewest appointments for any given slot. Remember to handle cases where there are fewer than 2 people in the list, or where there are ties in the appointment count.
I might add another model method for just that along the lines of:
def relevant_appointment_count(self):
return sum(x[1] for x in self.user_appointment_count()[:2])
You would combine this with something similar to the suggestion in my first comment as a queryset (either as extra context or as the main queryset depending on your preference) if you want to sort slots by how few appointments their two users from the above have.
# use this perhaps to return a queryset or extra context in a view
return sorted(
Slot.objects.filter(is_matched=False),
key=lambda s: s.relevant_appointment_count(),
reverse=True
)
That should (haven't tested so again, may need a slight tweak) return a list of slots in the order I've described. Again you'd need to handle the same cases I described before.