We Keep Coding

How do I write a Django query that uses a complex "on" clause in its inner join?

I'm using Django, Python 3.7, and PostgreSQL 9.5. I have these models:
class Article(models.Model):
...
label = models.TextField(default='', null=True)
class Label(models.Model):
name = models.CharField(max_length=200)
I want to write a Django query that retrieves all the articles whose label contains a name from the Labels table. In PostGres, I can structure my query like so:
select a.* from myapp_article a join myapp_label l on a.label ilike '%' || l.name || '%';
but I have no idea how to pull this off in Django on account of the "on" clause and "ilike". How do I pull this off?

If you've to do a case insensitive search on Article's label for matching names, then you can use regex and pass it a flat list of all the label names like so:
Article.objects.filter(label__iregex=r'(' + '|'.join(Label.objects.all().values_list('name', flat=True)) + ')')
What the above query does is, it makes a flat list of labels:
['label1' , 'label2', 'label3']
and then the string is joined like this:
'(label1|label2|label3)'
and a similar SQL query is used:
SELECT * from FROM "app_article" WHERE "app_article"."label" ~* (label1|label2|label3)
Otherwise, for case sensitive approach, you can use this:
names_list = Label.objects.all().values_list('name', flat=True)
Article.objects.filter(label__in=names_list)

This wouldn't translate into same SQL query, but would yield the same results, using an inner query.
inner_query = Label.objects.annotate(article_label=OuterRef('label')).filter(article_label__icontains=F('name'))
articles = Article.objects.annotate(labels=Subquery(inner_query.values('name')[:1])).filter(labels__isnull=False)
This should roughly should translate to this:
select a.* from myapp_article a where exists (select l.* from myapp_label l where a.label ilike '%' || l.name || '%')
But due to a current issue in Django regarding using OuterRef's in annotations, this approach doesn't work. We need to use a workaround suggested here until the issue is fixed to make this query work, like this:
Define a custom expression first
class RawCol(Expression):
def __init__(self, model, field_name):
field = model._meta.get_field(field_name)
self.table = model._meta.db_table
self.column = field.column
super().__init__(output_field=CharField())
def as_sql(self, compiler, connection):
sql = f'"{self.table}"."{self.column}"'
return sql, []
Then build your query using this expression
articles = Article.objects.all().annotate(
labels=Subquery(
Label.objects.all().annotate(
article_label=RawCol(Article, 'label')
).filter(article_label__icontains=F('name')).values('name')[:1]
)
).filter(labels__isnull=False)
This should return instances of Article model whose label field contain a value from the name field of Label model

In your class Article you will have to declare label as foreignkey to class Label
class Article(models.Model):
...
label = models.ForeignKey(Label, default='', on_delete=models.CASCADE)
And then you can access it.

Find parent data using Django ORM

I have 3 tables
AllTests
ReportFormat
Parameter
AllTests has reportFormat as ForeignKey with one to one relation.
class AllTests(models.Model):
id=models.AutoField(primary_key=True)
name=models.CharField(max_length=200)
reportFormat=models.ForeignKey(ReportFormat)
.....
.....
class Meta:
db_table = 'AllTests'
Parameter table has reportFormat as ForeignKey with one too many relations. Means one report format has many parameters.
class Parameter(models.Model):
id=models.AutoField(primary_key=True)
name=models.CharField(max_length=200)
reportFormat=models.ForeignKey(ReportFormat)
.....
.....
class Meta:
db_table = 'AllTests
ReportFormat Table:-
class ReportFormat(models.Model):
id=models.AutoField(primary_key=True)
.....
.....
class Meta:
db_table = 'ReportFormat'
I want to make a query on the parameter model that return parameter data with the related test data. Please suggest me a better way for the same.
my current query is like this.
from django.db.models import (
Sum, Value, Count, OuterRef, Subquery
)
data = Parameter.objects.filter(isDisable=0).values('id', 'name', 'reportFormat_id').annotate(
test=Subquery(Alltsets.objects.filter(reportFormat_id=OuterRef('reportFormat_id').values('id', 'name')))
)

Since, report format column of both the tables refer to the same column of reportformat table, maybe you can directly relate them with something like below.
select * from parameter inner join alltests on parameter.reportformat=alltests.reportformat
Maybe in ORM something like this?
Parameter.objects.filter().annotate(alltests_id=F('reportformat__alltests__id'),alltests_name=F('reportformat__alltests__name'),....other fields)

Fetch all Parameter and AllTests objects. Find the parameter's alltests object using reportFormat field. Add the corresponding data to the parameter dict.
Solution:
parameter_qs = Parameter.objects.all().select_related('reportFormat')
alltests_qs = AllTests.objects.all().select_related('reportFormat')
data = []
for parameter in parameter_qs:
item = {
'id': parameter.id,
'name': parameter.name,
'alltests': {}
}
# fetch related AllTests object using `reportFormat` field
alltests_obj = None
for alltests in alltests_qs:
if parameter.reportFormat == alltests.reportFormat:
alltests_obj = alltests
break
if alltests_obj is not None:
item['alltests'] = {
'id': alltests_obj.id,
'name': alltests_obj.name
}
data.append(item)

Convert django RawQuerySet to Queryset

I have 2 Django models, ModelA with an ArrayField that is used to store a large list of primary key values (possibly 50k+ list)
class ModelA(models.Model):
pk_values = ArrayField(models.IntegerField())
class CustomManager(manager.Manager):
def get_for_index(self, index_id):
qs = self.get_queryset()
obj = ModelA.objects.get(pk=index_id)
return qs.filter(id__in=obj.pk_values)
class ModelB(models.Model):
# [...] some fields
objects = CustomManager()
This works:
qs = ModelB.objects.get_for_index(index_id=1)
However, this would be super slow where "pk_values" is a large list.
So I tried doing raw SQL queries:
class CustomManager(manager.Manager):
def get_for_index(self, index_id):
qs = self.get_queryset()
sql = "SELECT * FROM myapp_model_b JOIN myapp_model_a ON myapp_model_b.id = ANY(myapp_model_a.pk_values) WHERE myapp_model_a.id = '%s'" % index_id
return qs.raw(sql)
But this returns a django.db.models.query.RawQuerySet instance.
But with this, I cant do things like queryset.values() afterwards.
How can I convert this to a normal Django queryset?
Is there a better way of doing this?
Docs:
ArrayField https://docs.djangoproject.com/en/2.0/ref/contrib/postgres/fields/#arrayfield
Custom Manager https://docs.djangoproject.com/en/2.0/topics/db/managers/#custom-managers-and-model-inheritance
Raw queries https://docs.djangoproject.com/en/2.0/topics/db/sql/#performing-raw-sql-queries

You can use a RawSQL expression:
ModelB.objects.filter(id__in=RawSQL(
'SELECT unnest(a.pk_values) FROM app_modela a WHERE a.id = %s',
[index_id]
))
Alternatively you can reproduce the exact query you have in your question with extra():
ModelB.objects.extra(
tables=['foo_modela'],
where=[
'"app_modelb"."id" = ANY("app_modela"."pk_values")',
'"app_modela"."id" = %s',
],
params=[index_id],
)

Update: I got something working using .extra()
class CustomManager(manager.Manager):
def get_for_index(self, index_id):
qs = self.get_queryset()
sql = "myapp_model_b.id IN (SELECT UNNEST(myapp_model_a.pk_values) FROM myapp_model_a WHERE myapp_model_a.id='%s')" % index_id
return qs.extra(where=[sql])
Docs: https://docs.djangoproject.com/en/2.0/ref/models/querysets/#django.db.models.query.QuerySet.extra

Join Django model from other database

The situation is as follows in the Django implementation:
Two different database;
Two models with different information
Order number is relation between these models.
I would like to join these models together to get related information. Example
class CurtainOrder(models.Model):
order = models.ForeignKey('curtainconfig.Order', related_name='curtains')
order_number = models.IntegerField(unique=True)
order_type = models.CharField(max_length=50)
class ProductionOrder(models.Model):
id = models.IntegerField(primary_key=True, db_column=u'lOrder_id')
order_number = models.IntegerField(db_column=u'nOrderNumber')
status = models.TextField(db_column=u'cExternalDescription')
class Meta:
app_label = u'backend'
db_table = u'ORDER'
database_name = 'production'
To get the result by an query would be as follow:
SELECT
co.order_number, co.order_type, po.status
FROM
CurtainOrder co
INNER JOIN
[production].dbo.ProductionOrder po on po.order_number = co.order_number
But how can I get the same result in Django code?
I have tried to use SQL query in Django, https://docs.djangoproject.com/en/1.8/topics/db/sql/, like below:
from django.db import connections
cursor = connections['default'].cursor()
sql = """SELECT
co.order_number, co.order_type, po.status
FROM
CurtainOrder co
INNER JOIN
[production].dbo.ProductionOrder po on po.order_number = co.order_number """
cursor.execute(sql)
row = cursor.fetchone()
This is returning the error: return Database.Cursor.execute(self, query)
OperationalError: near ".": syntax error
It's going wrong when the 'production' db is joined, but the SQL query is executed good by doing it directly in the database management app.

Sql query in django. How to translate?

I have this SQL query:
"INSERT INTO $TABLENAME (address, count) VALUES(%s, %s) \
ON DUPLICATE KEY UPDATE count = VALUES(count) + count"
How to translate it into django ?

I think you're looking for get_or_create https://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create
Fore example lets assume $TABLENAME is represented by a django model called MyModel:
models.py:
class MyModel(models.Model):
address = models.CharField(max_length=300, unique=True)
count = models.IntegerField(default=0)
views.py:
new_model, created = MyModel.objects.get_or_create(address="123 1st st")
new_model.count += 1
new_model.save()
get_or_create will either create a new object with the given properties or if it already exists it will return the existing object which you can then update.