We Keep Coding

How do I use web2py smart_query for a GET request?

So I'm trying to use smart_query in web2py to find specific values in a db, but the only explanation I can find is in the web2py book and it's not very clear. The example GET request from the book is formatted like this:
def GET(search):
try:
rows = db.smart_query([db.person, db.pet], search).select()
return dict(result=rows)
except:
...
I'm confused as to what values I would put in place of db.person and db.pet. Here is what the book says on it:
The method db.smart_query takes two arguments:
a list of field or table that should be allowed in the query
a string containing the query expressed in natural language
I'm thinking the first value would be the database I'm searching, but then I don't know what the second value would be. The book makes it sound like it should be the string I'm searching for, but I think that that's what the variable search is for.
Could someone please help me understand what exactly each argument is supposed to do?

The first argument to smart_query is a list of DAL Table and/or Field objects (a Table object in the list will simply be expanded to include all of the table's fields). This list determines which fields can be included in the query.
The second argument is the query itself, which can include field names and comparison operators (and their natural language counterparts) as well as "and" and "or" to expression conjunctions and disjunctions. For an idea of what is allowed, you can examine the relevant code here.
The SQLFORM.grid advanced search widget generates queries that are ultimately parsed by smart_query, so to get a better idea of how to generate such queries, try creating a test SQLFORM.grid and play with the search widget in the UI to see the queries it generates.

How to do an accent-insensitive TrigramSimilarity search in django?

How can I add accent-insensitive search to following snippet from the django docs:
>>> from django.contrib.postgres.search import TrigramSimilarity
>>> Author.objects.create(name='Katy Stevens')
>>> Author.objects.create(name='Stephen Keats')
>>> test = 'Katie Stephens'
>>> Author.objects.annotate(
... similarity=TrigramSimilarity('name', test),
... ).filter(similarity__gt=0.3).order_by('-similarity')
[<Author: Katy Stevens>, <Author: Stephen Keats>]
How could this match test = 'Kâtié Stéphèns'?

There exist the unaccent lookup:
The unaccent lookup allows you to perform accent-insensitive lookups using a dedicated PostgreSQL extension.
Also if you take a look at the aggregation part of django docs, you can read the following:
When specifying the field to be aggregated in an aggregate function,
Django will allow you to use the same double underscore notation that
is used when referring to related fields in filters. Django will then
handle any table joins that are required to retrieve and aggregate the
related value.
Derived from the above:
You can use the trigram_similar lookup, combined with unaccent, then annotate on the result:
Author.objects.filter(
name__unaccent__trigram_similar=test
).annotate(
similarity=TrigramSimilarity('name__unaccent', test),
).filter(similarity__gt=0.3).order_by('-similarity')
OR
if you want to keep it as close as possible to the original sample (and omit one potentially slow filtering followed by another):
Author.objects.annotate(
similarity=TrigramSimilarity('name__unaccent', test),
).filter(similarity__gt=0.3).order_by('-similarity')
Those will only work in Django version >= 1.10
EDIT:
Although the above should work, #Private reports this error occurred:
Cannot resolve keyword 'unaccent' into a field. Join on 'unaccented' not permitted.
This may be a bug, or unaccent is not intended to work that way. The following code works without the error:
Author.objects.filter(
name__unaccent__trigram_similar=test
).annotate(
similarity=TrigramSimilarity('name', test),
).filter(similarity__gt=0.3).order_by('-similarity')

SELECT on JSONField with Django

My application is heavily reliant on APIs that unpredictably make changes to the way they return data. For this reason, I've chosen to use PSQL and JSONFields with Django.
I've seen plenty of examples/docs on how to filter by values in a JSONField, but I haven't seen any that allow me to SELECT on these values.
What I know works;
queryset.filter(jsonfield__key_name = 'value')
What I want to know how to do;
queryset.values('jsonfield__key_name')
Thanks in advance!

The answer is a RawSQL expression;
queryset.annotate(value = RawSQL("(jsonfield->%s)", ('key_name',)))
queryset.values('value')
The first argument to RawSQL is like a template string, the second argument will fill in the first's %s
UPDATE: apparently Django 2.1+ now supports my original expected behavior;
queryset.values('jsonfield__key_name')

Since Django 2.1, transforms are supported in order_by(), values() and values_list(), so you can now use this:
queryset.values('jsonfield__key_name')
Here's the relevant ticket and pull request.

In django, is there a way to directly annotate a query with a related object in single query?

Consider this query:
query = Novel.objects.< ...some filtering... >.annotate(
latest_chapter_id=Max("volume__chapter__id")
)
Actually what I need is to annotate each Novel with its latest Chapter object, so after this query, I have to execute another query to select actual objects by annotated IDs. IMO this is ugly. Is there a way to combine them into a single query?

Yes, it's possible.
To get a queryset containing all Chapters which are the last in their Novels, simply do:
from django.db.models.expressions import F
from django.db.models.aggregates import Max
Chapters.objects.annotate(last_chapter_pk=Max('novel__chapter__pk')
).filter(pk=F('last_chapter_pk'))
Tested on Django 1.7.

Possible with Django 3.2+
Make use of django.db.models.functions.JSONObject (added in Django 3.2) to combine multiple fields (in this example, I'm fetching the latest object, however it is possible to fetch any arbitrary object provided that you can get LIMIT 1) to yield your object):
MainModel.objects.annotate(
last_object=RelatedModel.objects.filter(mainmodel=OuterRef("pk"))
.order_by("-date_created")
.values(
data=JSONObject(
id="id", body="body", date_created="date_created"
)
)[:1]
)

Yes, using Subqueries, docs: https://docs.djangoproject.com/en/3.0/ref/models/expressions/#subquery-expressions
latest_chapters = Chapter.objects.filter(novel = OuterRef("pk"))\
.order_by("chapter_order")
novels_with_chapter = Novel.objects.annotate(
latest_chapter = Subquery(latest_chapters.values("chapter")[:1]))
Tested on Django 3.0
The subquery creates a select statement inside the select statement for the novels, then adds this as an annotation. This means you only hit the database once.
I also prefer this to Rune's answer as it actually annotates a Novel object.
Hope this helps, anyone who came looking like much later like I did.

No, it's not possible to combine them into a single query.
You can read the following blog post to find two workarounds.

In Django, how does one filter a QuerySet with dynamic field lookups?

Given a class:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=20)
Is it possible, and if so how, to have a QuerySet that filters based on dynamic arguments? For example:
# Instead of:
Person.objects.filter(name__startswith='B')
# ... and:
Person.objects.filter(name__endswith='B')
# ... is there some way, given:
filter_by = '{0}__{1}'.format('name', 'startswith')
filter_value = 'B'
# ... that you can run the equivalent of this?
Person.objects.filter(filter_by=filter_value)
# ... which will throw an exception, since `filter_by` is not
# an attribute of `Person`.

Python's argument expansion may be used to solve this problem:
kwargs = {
'{0}__{1}'.format('name', 'startswith'): 'A',
'{0}__{1}'.format('name', 'endswith'): 'Z'
}
Person.objects.filter(**kwargs)
This is a very common and useful Python idiom.

A simplified example:
In a Django survey app, I wanted an HTML select list showing registered users. But because we have 5000 registered users, I needed a way to filter that list based on query criteria (such as just people who completed a certain workshop). In order for the survey element to be re-usable, I needed for the person creating the survey question to be able to attach those criteria to that question (don't want to hard-code the query into the app).
The solution I came up with isn't 100% user friendly (requires help from a tech person to create the query) but it does solve the problem. When creating the question, the editor can enter a dictionary into a custom field, e.g.:
{'is_staff':True,'last_name__startswith':'A',}
That string is stored in the database. In the view code, it comes back in as self.question.custom_query . The value of that is a string that looks like a dictionary. We turn it back into a real dictionary with eval() and then stuff it into the queryset with **kwargs:
kwargs = eval(self.question.custom_query)
user_list = User.objects.filter(**kwargs).order_by("last_name")

Additionally to extend on previous answer that made some requests for further code elements I am adding some working code that I am using
in my code with Q. Let's say that I in my request it is possible to have or not filter on fields like:
publisher_id
date_from
date_until
Those fields can appear in query but they may also be missed.
This is how I am building filters based on those fields on an aggregated query that cannot be further filtered after the initial queryset execution:
# prepare filters to apply to queryset
filters = {}
if publisher_id:
filters['publisher_id'] = publisher_id
if date_from:
filters['metric_date__gte'] = date_from
if date_until:
filters['metric_date__lte'] = date_until
filter_q = Q(**filters)
queryset = Something.objects.filter(filter_q)...
Hope this helps since I've spent quite some time to dig this up.
Edit:
As an additional benefit, you can use lists too. For previous example, if instead of publisher_id you have a list called publisher_ids, than you could use this piece of code:
if publisher_ids:
filters['publisher_id__in'] = publisher_ids

Django.db.models.Q is exactly what you want in a Django way.

This looks much more understandable to me:
kwargs = {
'name__startswith': 'A',
'name__endswith': 'Z',
***(Add more filters here)***
}
Person.objects.filter(**kwargs)

A really complex search forms usually indicates that a simpler model is trying to dig it's way out.
How, exactly, do you expect to get the values for the column name and operation?
Where do you get the values of 'name' an 'startswith'?
filter_by = '%s__%s' % ('name', 'startswith')
A "search" form? You're going to -- what? -- pick the name from a list of names? Pick the operation from a list of operations? While open-ended, most people find this confusing and hard-to-use.
How many columns have such filters? 6? 12? 18?
A few? A complex pick-list doesn't make sense. A few fields and a few if-statements make sense.
A large number? Your model doesn't sound right. It sounds like the "field" is actually a key to a row in another table, not a column.
Specific filter buttons. Wait... That's the way the Django admin works. Specific filters are turned into buttons. And the same analysis as above applies. A few filters make sense. A large number of filters usually means a kind of first normal form violation.
A lot of similar fields often means there should have been more rows and fewer fields.