We Keep Coding

Django: Admin - order results based on regular expression

I have a problem where I want to order model rows in Django admin based on a number within a string and ignore the letters - like XX0345XX, X0346XXX, XXX0347XX. I'm thinking that I should probably use a regular expression for that. I have an SQL query (PostgreSQL) that returns what I want:
select * from mytable order by substring(my_field, '\d+')::int DESC
But I'm having trouble applying it to get the same result in the Django admin get_queryset().
I've tried doing something like:
def get_queryset():
return Model.objects.raw("select * from mytable order by substring(my_field, '\d+')::int DESC")
but the problem is that I'm not returning the right type this way. Model.objects.raw('...') returns RawQuerySet, but get_queryset() should return QuerySet instance not RawQuerySet one, so I can't do it this way.
Any suggestions on how to solve this problem? Thanks!

You can use the .extra() method of to convert from a rawqueryset to a queryset, see here
This example is taken from, here
class CustomManager(manager.Manager):
def get_queryset():
qs = self.get_queryset()
sql = "myapp_model_b.id IN (SELECT UNNEST(myapp_model_a.pk_values) FROM myapp_model_a WHERE myapp_model_a.id='%s')" % index_id
return qs.extra(where=[sql])

As pointed out by #Azy_Crw4282, you can use QuerySet.extra() to run raw SQL queries and still have the results returned as a QuerySet instance.
Here's how I managed to do a regex based "ORDER BY" on objects in admin view, based on a number within a string. I.e. order fields like XX0345XX, X0346XXX, XXX0347XX, based on the number they contain - basically get a substring that matches '\d+' regular expression.
Just override the get_queryset() function in your admin.ModelAdmin class.
Solution:
def get_queryset(self, request):
sql_query = "CAST(substring(booking_reference, '\d+') as int)"
bookings = Booking.objects.extra(select={'book_ref_digits': sql_query}).order_by('-book_ref_digits')
return bookings
As far as I understand, it adds a new temporary field, e.g. book_ref_digits in the QuerySet object and then you can do .order_by() on that.
Note: Using older version of Django 1.10.5

Django F expression and ArrayField position

I have a problem with Django and would like to ask for some advice:
One of my models contains specific indicators with variable values. This is why I am using an Arrayfield for those.
For example one Indicator has 3 values, another only 2 and some only have 1 value.
Now I want to compare values from within the Arrayfield with each other.
So I know there are those fancy F Expressions and I tried to use them, but here I am stuck, because it seems like F doesn't allow an positional-lookup from inside ArrayField.
Does anyone know if i can use F for positional lookup of arrayfield? Or do I have to make a direct SQL-Call from Django or change my db somehow to make one value per field?
Here is the Model:
class Indicators(models.Model):
datetime = models.DateTimeField(default=datetime.now)
name = models.TextField(default="none")
values = ArrayField(models.FloatField(default=0))
This is what I want to achieve:
indicator_compareable = Indicators.objects.filter(name='compareable',
values__0=F('values__1')).values_list('values')
And it raises an Exception..
operator does not exist: double precision = double precision[]
LINE 1: ...areable' AND "indicators"."values"[1] = ("...
As Mikhail suggested (in first comment to my post), here is the translated SQL from my Django filter:
SELECT "indicators"."values"[1] FROM "indicators"
WHERE ("indicators"."name" = 'compareable'
AND "indicators"."values"[1] = ("indicators"."values"))
From the Exception it seems like I am not allowed to give ArrayField position to F Expression (or that Django ignores the position..). And the translated sql-query shows, that it seems like F doesn't handle ArrayField-position...
In postgres a correct query for my needs would be:
SELECT values FROM indicators WHERE name='compareable' and
values[1]=values[2];
During my search I found those links here, but they didn't help..:
Django F field iteration
https://code.djangoproject.com/ticket/24709

So actually after investing the translated postgres query it's clear that django's F Expression is (still) not offering positional lookup (and might not be implemented).
I'll leave this as the answer, maybe it helps someone out there who is stuck at the same point.
I decided to use Django's SQL-RAW for my specific needs. But in general the rest of my code uses Django ORM whenever possible.
Here there has been a Django-issue to this topic, but by now it's 1 and a half years old: https://code.djangoproject.com/ticket/24709
by the way, my django version: Django 2.1 (the mentioned django-issue was with v1.8)
Thanks again to Mikhail for his suggestions.

Build search with filter to search over at least two fields

class Objects(models.Model):
name = Charfield
term = Charfield
I have search field where I type like name/term of the object in.
Is there an easy way to filter for both fields and concat the queryset to present it then as a result.
Or I have to work with checkboxes? To realize this simple
Any idea?

If I'm correctly understanding your question, you need an OR condition, that can be achieved using Q objects and | operator between them.
from django.db.models import Q
Objects.objects.filter(Q(name=search_field_value) | Q(term=search_field_value))
where search_field_value is a value of the search field.
Also see documentation.

Order Query results by startswith match

We are doing a site search where we search a field with an icontains. It's working great, but the results that come up dont have the most relevant result at the top.
An example is searching for "Game of Thrones". If the search is "Game of", the first result could be "Crazy Game of..." and the second is "Game of Thrones"
Essentially, I'd like to search with icontains, but order by startswith. Any ideas?

Django added new features since this question was asked, and now handles this use case natively.
You can order results by an arbitrary comparison using the following features in Django's QuerySet API:
Q() Object: encapsulates a reusable query expression. Our expression compares the object's field against our "Game of" search value.
ExressionWrapper(): wraps a query expression, in order to control the ModelField type with output_field. In this case, the type is a Boolean.
annotate(): dynamically add a ModelField to each QuerySet object, based on the result of a query expression. We want to add a True/False field to sort on.
order_by(): order the QuerySet objects by specified fields. We want to order by the annotated field value, in reverse, so that True values are displayed first.
from django.db.model import Q, ExpressionWrapper, BooleanField
....
# Your original setup.
search_term = 'Game of'
data = MyModel.objects.filter(name__icontains=search_term)
# Encapsulate the comparison expression.
expression = Q(name__startswith=search_term)
# Wrap the expression to specify the field type.
is_match = ExpressionWrapper(expression, output_field=BooleanField())
# Annotate each object with the comparison.
data = data.annotate(my_field=is_match)
# Order by the annotated field in reverse, so `True` is first (0 < 1).
data = data.order_by('-my_field')
....

The ordering you're describing is subjective and there's no data like that that comes out of the database (that I'm aware of). If a feature like this is important, you might want to look into a search engine like Solr or Sphinx where you can configure how relevancy scores are determined.

In Django, how does one filter a QuerySet with dynamic field lookups?

Given a class:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=20)
Is it possible, and if so how, to have a QuerySet that filters based on dynamic arguments? For example:
# Instead of:
Person.objects.filter(name__startswith='B')
# ... and:
Person.objects.filter(name__endswith='B')
# ... is there some way, given:
filter_by = '{0}__{1}'.format('name', 'startswith')
filter_value = 'B'
# ... that you can run the equivalent of this?
Person.objects.filter(filter_by=filter_value)
# ... which will throw an exception, since `filter_by` is not
# an attribute of `Person`.

Python's argument expansion may be used to solve this problem:
kwargs = {
'{0}__{1}'.format('name', 'startswith'): 'A',
'{0}__{1}'.format('name', 'endswith'): 'Z'
}
Person.objects.filter(**kwargs)
This is a very common and useful Python idiom.

A simplified example:
In a Django survey app, I wanted an HTML select list showing registered users. But because we have 5000 registered users, I needed a way to filter that list based on query criteria (such as just people who completed a certain workshop). In order for the survey element to be re-usable, I needed for the person creating the survey question to be able to attach those criteria to that question (don't want to hard-code the query into the app).
The solution I came up with isn't 100% user friendly (requires help from a tech person to create the query) but it does solve the problem. When creating the question, the editor can enter a dictionary into a custom field, e.g.:
{'is_staff':True,'last_name__startswith':'A',}
That string is stored in the database. In the view code, it comes back in as self.question.custom_query . The value of that is a string that looks like a dictionary. We turn it back into a real dictionary with eval() and then stuff it into the queryset with **kwargs:
kwargs = eval(self.question.custom_query)
user_list = User.objects.filter(**kwargs).order_by("last_name")

Additionally to extend on previous answer that made some requests for further code elements I am adding some working code that I am using
in my code with Q. Let's say that I in my request it is possible to have or not filter on fields like:
publisher_id
date_from
date_until
Those fields can appear in query but they may also be missed.
This is how I am building filters based on those fields on an aggregated query that cannot be further filtered after the initial queryset execution:
# prepare filters to apply to queryset
filters = {}
if publisher_id:
filters['publisher_id'] = publisher_id
if date_from:
filters['metric_date__gte'] = date_from
if date_until:
filters['metric_date__lte'] = date_until
filter_q = Q(**filters)
queryset = Something.objects.filter(filter_q)...
Hope this helps since I've spent quite some time to dig this up.
Edit:
As an additional benefit, you can use lists too. For previous example, if instead of publisher_id you have a list called publisher_ids, than you could use this piece of code:
if publisher_ids:
filters['publisher_id__in'] = publisher_ids

Django.db.models.Q is exactly what you want in a Django way.

This looks much more understandable to me:
kwargs = {
'name__startswith': 'A',
'name__endswith': 'Z',
***(Add more filters here)***
}
Person.objects.filter(**kwargs)

A really complex search forms usually indicates that a simpler model is trying to dig it's way out.
How, exactly, do you expect to get the values for the column name and operation?
Where do you get the values of 'name' an 'startswith'?
filter_by = '%s__%s' % ('name', 'startswith')
A "search" form? You're going to -- what? -- pick the name from a list of names? Pick the operation from a list of operations? While open-ended, most people find this confusing and hard-to-use.
How many columns have such filters? 6? 12? 18?
A few? A complex pick-list doesn't make sense. A few fields and a few if-statements make sense.
A large number? Your model doesn't sound right. It sounds like the "field" is actually a key to a row in another table, not a column.
Specific filter buttons. Wait... That's the way the Django admin works. Specific filters are turned into buttons. And the same analysis as above applies. A few filters make sense. A large number of filters usually means a kind of first normal form violation.
A lot of similar fields often means there should have been more rows and fewer fields.