My application is heavily reliant on APIs that unpredictably make changes to the way they return data. For this reason, I've chosen to use PSQL and JSONFields with Django.
I've seen plenty of examples/docs on how to filter by values in a JSONField, but I haven't seen any that allow me to SELECT on these values.
What I know works;
queryset.filter(jsonfield__key_name = 'value')
What I want to know how to do;
queryset.values('jsonfield__key_name')
Thanks in advance!
The answer is a RawSQL expression;
queryset.annotate(value = RawSQL("(jsonfield->%s)", ('key_name',)))
queryset.values('value')
The first argument to RawSQL is like a template string, the second argument will fill in the first's %s
UPDATE: apparently Django 2.1+ now supports my original expected behavior;
queryset.values('jsonfield__key_name')
Since Django 2.1, transforms are supported in order_by(), values() and values_list(), so you can now use this:
queryset.values('jsonfield__key_name')
Here's the relevant ticket and pull request.
Related
I have a problem with Django and would like to ask for some advice:
One of my models contains specific indicators with variable values. This is why I am using an Arrayfield for those.
For example one Indicator has 3 values, another only 2 and some only have 1 value.
Now I want to compare values from within the Arrayfield with each other.
So I know there are those fancy F Expressions and I tried to use them, but here I am stuck, because it seems like F doesn't allow an positional-lookup from inside ArrayField.
Does anyone know if i can use F for positional lookup of arrayfield? Or do I have to make a direct SQL-Call from Django or change my db somehow to make one value per field?
Here is the Model:
class Indicators(models.Model):
datetime = models.DateTimeField(default=datetime.now)
name = models.TextField(default="none")
values = ArrayField(models.FloatField(default=0))
This is what I want to achieve:
indicator_compareable = Indicators.objects.filter(name='compareable',
values__0=F('values__1')).values_list('values')
And it raises an Exception..
operator does not exist: double precision = double precision[]
LINE 1: ...areable' AND "indicators"."values"[1] = ("...
As Mikhail suggested (in first comment to my post), here is the translated SQL from my Django filter:
SELECT "indicators"."values"[1] FROM "indicators"
WHERE ("indicators"."name" = 'compareable'
AND "indicators"."values"[1] = ("indicators"."values"))
From the Exception it seems like I am not allowed to give ArrayField position to F Expression (or that Django ignores the position..). And the translated sql-query shows, that it seems like F doesn't handle ArrayField-position...
In postgres a correct query for my needs would be:
SELECT values FROM indicators WHERE name='compareable' and
values[1]=values[2];
During my search I found those links here, but they didn't help..:
Django F field iteration
https://code.djangoproject.com/ticket/24709
So actually after investing the translated postgres query it's clear that django's F Expression is (still) not offering positional lookup (and might not be implemented).
I'll leave this as the answer, maybe it helps someone out there who is stuck at the same point.
I decided to use Django's SQL-RAW for my specific needs. But in general the rest of my code uses Django ORM whenever possible.
Here there has been a Django-issue to this topic, but by now it's 1 and a half years old: https://code.djangoproject.com/ticket/24709
by the way, my django version: Django 2.1 (the mentioned django-issue was with v1.8)
Thanks again to Mikhail for his suggestions.
I have the following query:
sales_for_date_for_provider.exclude(sales_or_return='R').values_list('royalty_price', 'conversion_to_usd'))
Is it possible to multiple the royalty_price * conversion_to_usd in the query? Or do I need to do a list comprehension or dive into raw SQL?
The Django documentation warns us that the extra() method will soon be deprecated. Since I came upon this answer when searching for a similar question, I thought I'd share an alternative approach using Django's built-in expressions, specifically the F() object.
In response to the original question, you could achieve the desired outcome with the following:
sales_for_date_for_provider.annotate(
result=F('royalty_price') * F('conversion_to_usd'))
From the documentation:
F() can be used to create dynamic fields on your models by combining different fields with arithmetic:
And here's another example taken from the documentation that further illustrates its usage:
company = Company.objects.annotate(
chairs_needed=F('num_employees') - F('num_chairs'))
You can use extra() and specify select argument:
sales_for_date_for_provider.extra(select={'result': 'royalty_price * conversion_to_usd'})
The result would contain a QuerySet where each object would have an attribute result containing the multiplication of royalty_price and conversion_to_usd fields.
Consider this query:
query = Novel.objects.< ...some filtering... >.annotate(
latest_chapter_id=Max("volume__chapter__id")
)
Actually what I need is to annotate each Novel with its latest Chapter object, so after this query, I have to execute another query to select actual objects by annotated IDs. IMO this is ugly. Is there a way to combine them into a single query?
Yes, it's possible.
To get a queryset containing all Chapters which are the last in their Novels, simply do:
from django.db.models.expressions import F
from django.db.models.aggregates import Max
Chapters.objects.annotate(last_chapter_pk=Max('novel__chapter__pk')
).filter(pk=F('last_chapter_pk'))
Tested on Django 1.7.
Possible with Django 3.2+
Make use of django.db.models.functions.JSONObject (added in Django 3.2) to combine multiple fields (in this example, I'm fetching the latest object, however it is possible to fetch any arbitrary object provided that you can get LIMIT 1) to yield your object):
MainModel.objects.annotate(
last_object=RelatedModel.objects.filter(mainmodel=OuterRef("pk"))
.order_by("-date_created")
.values(
data=JSONObject(
id="id", body="body", date_created="date_created"
)
)[:1]
)
Yes, using Subqueries, docs: https://docs.djangoproject.com/en/3.0/ref/models/expressions/#subquery-expressions
latest_chapters = Chapter.objects.filter(novel = OuterRef("pk"))\
.order_by("chapter_order")
novels_with_chapter = Novel.objects.annotate(
latest_chapter = Subquery(latest_chapters.values("chapter")[:1]))
Tested on Django 3.0
The subquery creates a select statement inside the select statement for the novels, then adds this as an annotation. This means you only hit the database once.
I also prefer this to Rune's answer as it actually annotates a Novel object.
Hope this helps, anyone who came looking like much later like I did.
No, it's not possible to combine them into a single query.
You can read the following blog post to find two workarounds.
I want to run a django update through the ORM that looks something like this:
MyModel.objects.filter(**kwargs).update(my_field=F('my_other_field')+'a string')
This causes MySQL to throw an exception. Is there anyway to do this without writing raw SQL?
I had a similar issue; basically I wanted to concatenate two fields to the get the full name of a user. I got it solved this way(but must say that I was using Postgres):
from django.db.models.functions import Concat
from django.db.models import F, Value, CharField
AnyModel.objects.filter(**kwargs).annotate(full_name=Concat(F('model__user_first_name'), Value(' '), F('model__user_last_name'), output_field=CharField()))
where, F('...') evaluates its argument as a query, so you can query a field of the model itself, or span across models as you would do in filter/get, while Value('...') evaluates its argument literally(in my case I needed a space to be placed in between first_name and last_name), and output_field=... specifies the Type of the annotated field(I wanted to be a CharField).
For more info, you can read Django docs about Concat
What's happening is that Django is passing the '+' through to SQL - but SQL doesn't allow the use of '+' for concatenation, so it tries to add numerically. If you use an integer in place of 'a string', it does work in the sense that it adds the integer value of my_other_field to your variable.
It's debatable whether this is a bug. The documentation for F() objects in lookup queries states:
Django supports the use of addition, subtraction, multiplication, division and modulo arithmetic with F() objects
so it could be argued that you shouldn't be trying to use it to update with strings. But that's certainly not documented, and the error message 'Incorrect DOUBLE value' is not very helpful. I'll open a ticket.
You can use Concat function https://docs.djangoproject.com/en/1.9/ref/models/database-functions/#concat
from django.db.models.functions import Concat
from django.db.models import Value
MyModel.objects.filter(**kwargs).update(my_field=Concat('my_other_field', Value('a string'))
Given a class:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=20)
Is it possible, and if so how, to have a QuerySet that filters based on dynamic arguments? For example:
# Instead of:
Person.objects.filter(name__startswith='B')
# ... and:
Person.objects.filter(name__endswith='B')
# ... is there some way, given:
filter_by = '{0}__{1}'.format('name', 'startswith')
filter_value = 'B'
# ... that you can run the equivalent of this?
Person.objects.filter(filter_by=filter_value)
# ... which will throw an exception, since `filter_by` is not
# an attribute of `Person`.
Python's argument expansion may be used to solve this problem:
kwargs = {
'{0}__{1}'.format('name', 'startswith'): 'A',
'{0}__{1}'.format('name', 'endswith'): 'Z'
}
Person.objects.filter(**kwargs)
This is a very common and useful Python idiom.
A simplified example:
In a Django survey app, I wanted an HTML select list showing registered users. But because we have 5000 registered users, I needed a way to filter that list based on query criteria (such as just people who completed a certain workshop). In order for the survey element to be re-usable, I needed for the person creating the survey question to be able to attach those criteria to that question (don't want to hard-code the query into the app).
The solution I came up with isn't 100% user friendly (requires help from a tech person to create the query) but it does solve the problem. When creating the question, the editor can enter a dictionary into a custom field, e.g.:
{'is_staff':True,'last_name__startswith':'A',}
That string is stored in the database. In the view code, it comes back in as self.question.custom_query . The value of that is a string that looks like a dictionary. We turn it back into a real dictionary with eval() and then stuff it into the queryset with **kwargs:
kwargs = eval(self.question.custom_query)
user_list = User.objects.filter(**kwargs).order_by("last_name")
Additionally to extend on previous answer that made some requests for further code elements I am adding some working code that I am using
in my code with Q. Let's say that I in my request it is possible to have or not filter on fields like:
publisher_id
date_from
date_until
Those fields can appear in query but they may also be missed.
This is how I am building filters based on those fields on an aggregated query that cannot be further filtered after the initial queryset execution:
# prepare filters to apply to queryset
filters = {}
if publisher_id:
filters['publisher_id'] = publisher_id
if date_from:
filters['metric_date__gte'] = date_from
if date_until:
filters['metric_date__lte'] = date_until
filter_q = Q(**filters)
queryset = Something.objects.filter(filter_q)...
Hope this helps since I've spent quite some time to dig this up.
Edit:
As an additional benefit, you can use lists too. For previous example, if instead of publisher_id you have a list called publisher_ids, than you could use this piece of code:
if publisher_ids:
filters['publisher_id__in'] = publisher_ids
Django.db.models.Q is exactly what you want in a Django way.
This looks much more understandable to me:
kwargs = {
'name__startswith': 'A',
'name__endswith': 'Z',
***(Add more filters here)***
}
Person.objects.filter(**kwargs)
A really complex search forms usually indicates that a simpler model is trying to dig it's way out.
How, exactly, do you expect to get the values for the column name and operation?
Where do you get the values of 'name' an 'startswith'?
filter_by = '%s__%s' % ('name', 'startswith')
A "search" form? You're going to -- what? -- pick the name from a list of names? Pick the operation from a list of operations? While open-ended, most people find this confusing and hard-to-use.
How many columns have such filters? 6? 12? 18?
A few? A complex pick-list doesn't make sense. A few fields and a few if-statements make sense.
A large number? Your model doesn't sound right. It sounds like the "field" is actually a key to a row in another table, not a column.
Specific filter buttons. Wait... That's the way the Django admin works. Specific filters are turned into buttons. And the same analysis as above applies. A few filters make sense. A large number of filters usually means a kind of first normal form violation.
A lot of similar fields often means there should have been more rows and fewer fields.