Context
I am quite new to Django and I am trying to write a complex query that I think would be easily writable in raw SQL, but for which I am struggling using the ORM.
Models
I have several models named SignalValue, SignalCategory, SignalSubcategory, SignalType, SignalSubtype that have the same structure like the following model:
class MyModel(models.Model):
id = models.BigAutoField(primary_key=True)
name = models.CharField()
fullname = models.CharField()
I also have explicit models that represent the relationships between the model SignalValue and the other models SignalCategory, SignalSubcategory, SignalType, SignalSubtype. Each of these relationships are named SignalValueCategory, SignalValueSubcategory, SignalValueType, SignalValueSubtype respectively. Below is the SignalValueCategory model as an example:
class SignalValueCategory(models.Model):
signal_value = models.OneToOneField(SignalValue)
signal_category = models.ForeignKey(SignalCategory)
Finally, I also have the two following models. ResultSignal stores all the signals related to the model Result:
class Result(models.Model):
pass
class ResultSignal(models.Model):
id = models.BigAutoField(primary_key=True)
result = models.ForeignKey(
Result
)
signal_value = models.ForeignKey(
SignalValue
)
Query
What I am trying to achieve is the following.
For a given Result, I want to retrieve all the ResultSignals that belong to it, filter them to keep the ones of my interest, and annotate them with two fields that we will call filter_group_id and filter_group_name. The values of two fields are determined by the SignalValue of the given ResultSignal.
From my perspective, the easiest way to achieve this would be first to annotate the SignalValues with their corresponding filter_group_name and filter_group_id, and then to join the resulting QuerySet with the ResultSignals. However, I think that it is not possible to join two QuerySets together in Django. Consequently, I thought that we could maybe use Prefetch objects to achieve what I am trying to do, but it seems that I am unable to make it work properly.
Code
I will now describe the current state of my queries.
First, annotating the SignalValues with their corresponding filter_group_name and filter_group_id. Note that filter_aggregator in the following code is just a complex filter that allows me to select the wanted SignalValues only. group_filter is the same filter but as a list of subfilters. Additionally, filter_name_case is a conditional expression (Case() construct):
# Attribute a group_filter_id and group_filter_name for each signal
signal_filters = SignalValue.objects.filter(
filter_aggregator
).annotate(
filter_group_id=Window(
expression=DenseRank(),
order_by=group_filters
),
filter_group_name=filter_name_case
)
Then, trying to join/annotate the SignalResults:
prefetch_object = Prefetch(
lookup="signal_value",
queryset=signal_filters,
to_attr="test"
)
result_signals: QuerySet = (
last_interview_result
.resultsignal_set
.filter(signal_value__in=signal_values_of_interest)
.select_related(
'signal_value__signalvaluecategory__signal_category',
'signal_value__signalvaluesubcategory__signal_subcategory',
'signal_value__signalvaluetype__signal_type',
'signal_value__signalvaluesubtype__signal_subtype',
)
.prefetch_related(
prefetch_object
)
.values(
"signal_value",
"test",
category=F('signal_value__signalvaluecategory__signal_category__name'),
subcategory=F('signal_value__signalvaluesubcategory__signal_subcategory__name'),
type=F('signal_value__signalvaluetype__signal_type__name'),
subtype=F('signal_value__signalvaluesubtype__signal_subtype__name'),
)
)
Normally, from my understanding, the resulting QuerySet should have a field "test" that is now available, that would contain the fields of signal_filter, the first QuerySet. However, Django complains that "test" is not found when calling .values(...) in the last part of my code: Cannot resolve keyword 'test' into field. Choices are: [...]. It is like the to_attr parameter of the Prefetch object was not taken into account at all.
Questions
Did I missunderstand the functioning of annotate() and prefetch_related() functions? If not, what am I doing wrong in my code for the specified parameter to_attr to not exist in my resulting QuerySet?
Is there a better way to join two QuerySets in Django or am I better off using RawSQL? An alternative way would be to switch to Pandas to make the join in-memory, but it is very often more efficient to do such transformations on the SQL side with well-designed queries.
You're on the right path, but just missing what prefetch does.
Your annotations are correct, but the "test" prefetch isn't really an attribute. You batch up the SELECT * FROM signal_value queries so you don't have to execute the select per row. Just drop the "test" annotation and you should be fine. https://docs.djangoproject.com/en/3.2/ref/models/querysets/#prefetch-related
Please don't use pandas, it's definitely not necessary and is a ton of overhead. As you say yourself, it's more efficient to do the transforms on the sql side
From the docs on prefetch_related:
Remember that, as always with QuerySets, any subsequent chained methods which imply a different database query will ignore previously cached results, and retrieve data using a fresh database query.
It's not obvious but the values() call is part of these chained methods that imply a different query, and will actually cancel prefetch_related. This should work if you remove it.
I have three models
class A(Model):
...
class B(Model):
id = IntegerField()
a = ForeignKey(A)
class C(Model):
id = IntegerField()
a = ForeignKey(A)
I want get the pairs of (B.id, C.id), for which B.a==C.a. How do I make that join using the django orm?
Django allows you to reverse the lookup in much the same way that you can use do a forward lookup using __:
It works backwards, too. To refer to a “reverse” relationship, just use the lowercase name of the model.
This example retrieves all Blog objects which have at least one Entry whose headline contains 'Lennon':
Blog.objects.filter(entry__headline__contains='Lennon')
I think you can do something like this, with #Daniel Roseman's caveat about the type of result set that you will get back.
ids = B.objects.prefetch_related('a', 'a__c').values_list('id', 'a__c__id')
The prefetch related will help with performance in older versions of django if memory serves.
Suppose I have following models, which store questions and options for those questions.
P.S: I am just writing the basic code just to give you idea.
class Question(models.Model):
text = models.CharField()
class Option(models.Model):
text = models.CharField()
class QuestionOption(models.Model):
question = models.ForeignKey(Question)
option = models.ForeignKey(Option)
And then I have models which store user Feedback and Options selected for each Question in the Feedback survey.
class Feedback(models.Model):
full_name = models.CharField()
cell_phone = models.CharField()
created_at = models.DateTime()
class FeedbackOption(models.Model):
feedback = models.ForeignKey(Feedback, related_name='feedback_options')
option = models.ForeignKey(QuestionOption)
Every feedback will have lots of feedback option objects. Now I want to filter all the feedback whose feedback options contain specific QuestionOption object and then perform aggregate query where I am checking if that feedback's FeedbackOptions option text is 'boo' add count. Can i do this in one step, Something like this
# lets say i want to filter all feedback with QuestionOption id 1
stats = Feedback.objects.filter(feedback_options__option=1).aggregate(
boo=Count(Case(When(feedback_options__option__option__text='boo', then=1))),
hoo=Count(Case(When(feedback_options__option__option__text='hoo', then=1))))
It looks like it's applying aggregate on the only feedback option where option id is 1 not the rest of the feedback options for each feedback.
.filter() works as a where clause. When you add another method like .aggregate()to the resulting queryset, it is bounded by the same where clause.
In Django 1.11 you can use Subquery
from django.db.models import OuterRef, Subquery, Count
Feedback.objects.filter(feedback_options__option=o1).annotate(
boo=Subquery(
FeedbackOption.objects.filter(option__option__text="boo", feedback=OuterRef('pk')).values('feedback').annotate(count=Count('pk')).values('count'),
output_field=models.IntegerField())
)
Similarly you can add another annotation for hoo.
Doc reference
I have a couple models
class Order(models.Model):
user = models.ForeignKey(User)
class Lot(models.Model):
order = models.ForeignKey(Order)
buyer = models.ForeignKey(User)
What I'm trying to do is to annotate Lot objects with a number of buys made by a given user to the same seller. (it's not a mistake, Order.user is really a seller). Like “you’ve bought 4 items from this user recently”.
The closest I get was
recent_sold_lots = Lot.objects.filter(
order__user_id=OuterRef('order__user_id'),
status=Lot.STATUS_SOLD,
buyer_id=self.user_id,
date_sold__gte=now() - timedelta(hours=24),
)
qs = Lot.objects.filter(
status=Lot.STATUS_READY,
date_ready__lte=now() - timedelta(seconds=self.lag)
).annotate(same_user_recent_buys=Count(Subquery(recent_sold_lots.values('id'))))
But it fails when recent_sold_lots count is more than one: more than one row returned by a subquery used as an expression.
.annotate(same_user_recent_buys=Subquery(recent_sold_lots.aggregate(Count('id'))) doesn't seem to work also: This queryset contains a reference to an outer query and may only be used in a subquery.
.annotate(same_user_recent_buys=Subquery(recent_sold_lots.annotate(c=Count('id')).values('c')) is giving me Expression contains mixed types. You must set output_field.. If I add output_field=models.IntegerField() to the subquery call, it throws more than one row returned by a subquery used as an expression.
I'm stuck with this one. I feel I'm close to the solution, but what am I missing here?
The models you defined in the question do not correctly reflect the query you are making. In any case i'll use the model as a reference to my query.
from django.db.models import Count
user_id = 123 # my user id and also the buyer
buyer = User.objects.get(pk=user_id)
Lot.objects.filter(buyer=buyer).values('order__user').annotate(unique_seller_order_count=Count('id'))
What the query does is:
Filters the lot objects to the ones you have bought
Groups the Returned lots into the user who created the order
Annotates/Counts the responses for each group
I'd like to find how to select all objects whose ManyToMany field contains another object. I have the following models (stripped down)
class Category(models.Model):
pass
class Picture(models.Model):
categories = models.ManyToManyField(Category)
visible = models.BooleanField()
I need a function to select all the Pictures in one or more Categories:
def pics_in_cats(cat_ids=()):
pass
BUT it needs to return a QuerySet if possible so that I can do something like:
pics_in_cats((1,2,3)).filter(visible=True)
It could be done by loading all the relevant Category objects and merging their picture_set attributes, but that seems inefficient. I'd also like to avoid falling back to raw SQL if possible.
Thanks in advance
Why write a custom function and not use something like this? (untested)
pics = Picture.objects.filter(categories__in = [1,2,3]).filter(visible=True)