We Keep Coding

Django: Understanding QuerySet API

I'm a newbie in Django and I have some questions about making queries by QuerySet API.
For instance, I have User, his Orders, and its Statuses
class User(models.Model):
first_name = models.CharField(max_length=100)
last_name = models.CharField(max_length=100)
is_active = models.BooleanField()
class OrderStatus(models.Model):
name = models.CharField(max_length=100)
class Order(models.Model):
number = models.CharField(max_length=10)
amount = models.DecimalField(max_digits=19, decimal_places=2)
user = models.ForeignKey(User, on_delete=models.PROTECT, related_name="orders")
order_status = models.ForeignKey(OrderStatus, on_delete=models.PROTECT)
creation_datetime = models.DateTimeField(auto_now_add=True)
# Some filtering field
filtering_field = models.IntegerField()
I combined all of my questions to this one query:
Get active users with some additional data for each user:
'Amount' of the Orders filtered by 'filtering_field' and aggregated by Min and Max
'Number' and 'Amount' of the first Order filtered by 'filtering_field'
Count of the Orders filtered by 'filtering_field', aggregated by Count and grouped by 'Order Status'. This grouping means that data from query #1 and #2 can be duplicated and it's ok.
I could make this query in T-SQL by 3 separated subquery with own grouping, filtering, ordering:
SELECT
u.id,
u.first_name,
u.last_name,
ts.min_amount,
ts.max_amount,
first_order.number as first_order_number,
first_order.amount as first_order_amount,
cnt.order_status_id,
cnt.cnt
FROM
[User] u
-- 1. 'Amount' of the Orders filtered by 'filtering_field' and aggregated by Min and Max
LEFT OUTER JOIN (
SELECT
[user_id],
MIN(amount) min_amount,
MAX(amount) max_amount
FROM
[Order]
WHERE
filtering_field = 1
GROUP BY
[user_id]
) ts ON u.id = ts.[user_id]
-- 2. 'Number' and 'Amount' of the first Order filtered by 'filtering_field'
OUTER APPLY (
SELECT TOP 1
o.number,
o.amount
FROM
[Order] o
WHERE
u.id = o.[user_id] AND
o.filtering_field = 2
ORDER BY
o.creation_datetime
) first_order
-- 3. Count of the Orders filtered by 'filtering_field', aggregated by Count and grouped by 'Order Status'.
LEFT OUTER JOIN (
SELECT
[user_id],
order_status_id,
COUNT(*) cnt
FROM
[Order]
WHERE
filtering_field = 3
GROUP BY
[user_id],
order_status_id
) cnt ON u.id = cnt.[user_id]
WHERE
u.is_active = 1
How I can do the same by QuerySet API?
Query #1 I can do Min and Max in Annotate.
data = User.objects.filter(
Q(is_active=True)
).values(
'id',
'first_name',
'last_name',
).annotate(
min_amount=Min(
'orders__amount',
filter=Q(orders__filtering_field=1)
),
max_amount=Max(
'orders__amount',
filter=Q(orders__filtering_field=1)
)
)
But what about query #2 & #3?
I've considered Subquery(), but It supports the only one output value.
I mean if you wanna get 5 fields from 1 queryset, sql server runs 5 queries. I think it's not good for performance.
How I can join the first order once to use its fields and How can I use Count() with grouping by filtered rows of child model?
I'd like to use .prefetch_related() as a substitution of Subquery in T-SQL for each query like this:
Prefetch(
'orders',
queryset=Order.objects.filter(filtering_field=1)..., #staff with .values(), annotate(Min(), Max()) and etc.
to_attr='pf_query_1'
)
And then use 'pf_query_1' like 'orders__pf_query_1__amount' in User.objects...values()...annotate().
But I can't use .values() in Prefetch as well as 'pf_query_1' as a model field.
So what is the best practice to make this one query by QuerySet API?
I'd like to see the whole QuerySet API query just like T-SQL query

Have you considered the Django Subquery as described in the docs?
Regarding your 3rd question the only approach coming to my mind is dynamically creating the annotations.
Here a (tested) code sample using your models:
def test_query(self):
# 1st question
min_order = Order.objects.filter(user=OuterRef('pk'), filtering_field=1)\
.order_by().values('user').annotate(min=Min('amount')).values('min')
max_order = Order.objects.filter(user=OuterRef('pk'), filtering_field=1)\
.order_by().values('user').annotate(max=Max('amount')).values('max')
# 2nd question
first_number = Order.objects.filter(user=OuterRef('pk'), filtering_field=1)\
.order_by().values('user').annotate(fnumber=F('number')).values('fnumber')
first_amount = Order.objects.filter(user=OuterRef('pk'), filtering_field=1)\
.order_by().values('user').annotate(fnumber=F('amount')).values('amount')
kwargs = {
'min': Subquery(min_order, output_field=DecimalField()),
'max': Subquery(max_order, output_field=DecimalField()),
'first_n': Subquery(first_number, output_field=CharField()),
'first_a': Subquery(first_amount, output_field=DecimalField())
}
# 3rd question
for o in OrderStatus.objects.all():
kwargs['%s_count' % o.name] = \
Subquery(Order.objects.filter(user=OuterRef('pk'), filtering_field=1, order_status=o)\
.order_by().values('user').annotate(c=Count('pk')).values('c'), output_field=IntegerField())
# Putting it all together
qs2 = User.objects.annotate(**kwargs)
# Testing the results
for user in qs2:
v = Order.objects.filter(user=user, filtering_field=1).aggregate(Min('amount'), Max('amount'))
self.assertEqual(v['amount__min'], user.min)
self.assertEqual(v['amount__max'], user.max)
v = Order.objects.filter(user=user, filtering_field=1).first()
self.assertEqual(v.number, user.first_n)
self.assertEqual(v.amount, user.first_a)
for o in OrderStatus.objects.all():
v = Order.objects.filter(user=user, filtering_field=1, order_status=o).count()
if v == 0:
v = None
k = '%s_count' % o.name
v1 = getattr(user, k)
self.assertEqual(v, v1)
# The sql
print(qs2.query)
Please note:
The code is part of a TestCase where I put it to check if it worked
as expected
I know some parts of the query can be generated without
Subquery using the filter attribute of the aggregation functions. As
this filter attribute was only introduced in Django 2.0 and not
supported in the LTS version 1.11 I did not use it.
EDIT: Here is another approach I came up with starting with a "base queryset" and annotating that one:
def test_query2(self):
qs = Order.objects.filter(filtering_field=1).values('user', 'order_status').distinct()
# 1st question
min_order = Order.objects.filter(user=OuterRef('user'), filtering_field=1)\
.order_by().values('user').annotate(min=Min('amount')).values('min')
max_order = Order.objects.filter(user=OuterRef('user'), filtering_field=1)\
.order_by().values('user').annotate(max=Max('amount')).values('max')
# 2nd question
first_number = Order.objects.filter(user=OuterRef('user'), filtering_field=1)\
.order_by().values('user').annotate(fnumber=F('number')).values('fnumber')
first_amount = Order.objects.filter(user=OuterRef('user'), filtering_field=1)\
.order_by().values('user').annotate(fnumber=F('amount')).values('amount')
# 3rd question
total_count = Order.objects.filter(user=OuterRef('user'), filtering_field=1, order_status=OuterRef('order_status'))\
.order_by().values('user').annotate(c=Count('pk')).values('c')
qs2 = qs.annotate(
min = Subquery(min_order, output_field=DecimalField()),
max = Subquery(max_order, output_field=DecimalField()),
first_n = Subquery(first_number, output_field=CharField()),
first_a = Subquery(first_amount, output_field=CharField()),
c = Subquery(total_count, output_field=IntegerField())
)
# Testing the results
for d in qs2:
v = Order.objects.filter(user=d['user'], filtering_field=1).aggregate(Min('amount'), Max('amount'))
self.assertEqual(v['amount__min'], d['min'])
self.assertEqual(v['amount__max'], d['max'])
v = Order.objects.filter(user=d['user'], filtering_field=1).first()
self.assertEqual(v.number, d['first_n'])
self.assertEqual(v.amount, d['first_a'])
v = Order.objects.filter(user=d['user'], filtering_field=1, order_status=d['order_status']).count()
self.assertEqual(v, d['c'])
print(qs2.query)

Django: Move calculations query set, away from Python code

In the following code snippet, my goal is to get outstanding_event_total_gross.
To get that I first lookup for each ticket that belongs to the event the amount of sold_tickets. Out of that, I can calculate tickets_left. For each ticket I then calculate the outstanding_ticket_total_gross which I add up to outstanding_event_total_gross.
A lot of the business logic happens in Python, but I wonder now if there is a more efficient query set to achieve what I am doing while calling the data from the database?
tickets = Ticket.objects.filter(event=3)
outstanding_event_total_gross = 0
for ticket in tickets:
sold_tickets = ticket.attendees.filter(
canceled=False,
order__status__in=(
OrderStatus.PAID,
OrderStatus.PENDING,
OrderStatus.PARTIALLY_REFUNDED,
OrderStatus.FREE,
),
).count()
tickets_left = ticket.quantity - sold_tickets
outstanding_ticket_total_gross = tickets_left * ticket.price_gross
outstanding_event_total_gross += outstanding_ticket_total_gross
print(outstanding_event_total_gross)
Here a part of the models. I simplified them for better readability.
class Ticket(TimeStampedModel):
event = models.ForeignKey()
price_gross = models.PositiveIntegerField()
quantity = models.PositiveIntegerField()
class Order(AbstractTransaction, LogMixin):
event = models.ForeignKey()
status = models.CharField(
max_length=18, choices=OrderStatus.CHOICES, verbose_name=_("Status")
)
total_gross = models.PositiveIntegerField()

Maybe you can try like this with help of conditional aggregation:
from django.db.models import Q, Count, Sum
tickets = Ticket.objects.filter(event=3).annotate(
sold_tickets=Count(
'attendees',
filter=Q(
attendees__canceled=False,
attendees__order__status__in=(
OrderStatus.PAID,
OrderStatus.PENDING,
OrderStatus.PARTIALLY_REFUNDED,
OrderStatus.FREE,
)
),
distinct=True
)
).annotate(
tickets_left=F('quantity')-F('sold_tickets')
).annotate(
outstanding_gross=F('tickets_left') * F('price_gross')
)
outstanding_event_total_gross = tickets.aggregate(total=Sum('outstanding_gross'))['total']

Django ORM conditional filter LIKE CASE WHEN THEN

I'm using Django 1.11, Postgresql 9.2, python 3.4
I want to select data based on table's column named event_type if event type is single then compare date that should be of same date (today's) date else select all dates of given (today's) date that type would be of recurring.
But can't we manage this using single query? Like we do CASE and WHEN, THEN in Aggregation? I tried using Q object but no luck.
I want to check when value is 'single' then add condition, else another condition.
I could not find any good solution, currently I've achieved using this
today = datetime.date.today().strftime('%Y-%m-%d')
single_events = crm_models.EventsMeta.objects.filter(
event_type == "single",
repeat_start=today
)
recurring_events = crm_models.EventsMeta.objects.filter(
event_type == "recurring"
repeat_start__lte=today
)
all_events = single_events | recurring_events
For more information my model is:
class EventsMeta(models.Model):
event_type = models.CharField(max_length=50, choices=(("single","Single"),("recurring","Recurring")),
null=False, blank=False,default='single',verbose_name="Event Type")
repeat_start = models.DateTimeField()
repeat_end = models.DateTimeField()

You can combine many Q objects with () signs. I your case I suppose this will work:
single_events = crm_models.EventsMeta.objects.filter(
(Q(event_type="single") & Q(repeat_start=today)) |
(Q(event_type="recurring") & Q(repeat_start__lte=today))
)

use this:-
today = datetime.date.today().strftime('%Y-%m-%d')
single_events = crm_models.EventsMeta.objects.filter(
event_type__in = ["single", "recurring"]
repeat_start=today
)

Django can't lookup datetime field. int() argument must be a string, a bytes-like object or a number, not datetime.datetime

I'm trying to group objects 'Event' by their 'due' field, and finally return a dict of day names with a list of events on that day. {'Monday': [SomeEvent, SomeOther]} - that's the idea. However while looking up event's due__day I get: int() argument must be a string, a bytes-like object or a number, not datetime.datetime.
Here's manager's code:
# models.py
class EventManager(models.Manager):
def get_week(self, group_object):
if datetime.datetime.now().time() > datetime.time(16, 0, 0):
day = datetime.date.today() + datetime.timedelta(1)
else:
day = datetime.date.today()
friday = day + datetime.timedelta((4 - day.weekday()) % 7)
events = {}
while day != friday + datetime.timedelta(1):
events[str(day.strftime("%A"))] = self.get_queryset().filter(group=group_object, due__day=day)
# ^^^ That's where the error happens, from what I understood it tries to convert this datetime to int() to be displayed by template
day += datetime.timedelta(1)
return events
Here is the model:
# models.py
class Event(models.Model):
title = models.CharField(max_length=30)
slug = models.SlugField(blank=True)
description = models.TextField()
subject = models.CharField(max_length=20, choices=SUBJECTS)
event_type = models.CharField(max_length=8, choices=EVENT_TYPES)
due = models.DateTimeField()
author = models.ForeignKey(User, blank=True)
group = models.ForeignKey(Group, related_name='events')
objects = EventManager()
I created a template filter to call this method:
#register.filter
def get_week(group_object):
return Event.objects.get_week(group_object=group_object)
And I called it in event_list.html (Using an index because it's a list view, also this is just temporary to see if the returned dict will be correct. Looping over it is to be implemented.)
{{ event_list.0.group|get_week }}
My guess is: I've broken something with this weird lookup. .filter(due__day=day) However I cannot find the solution.
I also tried to look for due__lte=(day - datetime.timedelta(hours=12)), due__gte=(day + datetime.timedelta(hours=12)) something like this but that doesn't work. Any solution is pretty much fine and appreciated.

Right, so the solution for my project is to check for due__day=day(the one from the loop).day <- datetimes property to compare day agains day not day against the datetime. That worked for me, however it does not necessarily answer the question since I don't exactly what caused the error itself. Feel free to answer for the future SO help seekers or just my information :)

This works for me:
Event.objects.filter(due__gte=datetime.date(2016, 8, 29), due__lt=datetime.date(2016, 8, 30))
Note that it uses date instead of datetime to avoid too much extraneous code dealing with the times (which don't seem to matter here).

How do i compare the sum of two fields to another field on the same model in Django

Here is my model
class Wallet(models.Model):
"""
Keep track of the monetary values of a company's wallet
"""
serializer_class = WalletSocketSerializer
company = models.OneToOneField(Company, verbose_name=_('company'))
packaged_credits = models.BigIntegerField(_('packaged credits'), default=0)
purchased_credits = models.BigIntegerField(_('purchased credits'), default=0)
low_credits_threshold = models.BigIntegerField(default=0)
Now i would like to send an alert if the total credits are less than the threshold, this would be the equivalent of getting all low wallets in this SQL
select * from wallets_wallet where (packaged_credits + purchased_credits) < low_credits_threshold;
I want to know how to execute that in django, right now i have tried the following, it works, but i think it should be done in a more Django way:
low_wallets = []
for wallet in Wallet.objects.all():
if wallet.packaged_credits + wallet.purchased_credits < wallet.low_credits_threshold:
low_wallets.append(wallet)

from django.db.models import F
low_wallets = Wallet.objects.filter(
low_credits_threshold__gt=F('packaged_credits')+F('purchased_credits')
)

Wallet.objects.extra(where=["packaged_credits + purchased_credits < low_credits_threshold"])