We Keep Coding

Specify order of columns in SELECT with UNION using Django ORM

How could I specify the order of columns in SELECT query in Django ORM?
I am trying to union elements from two tables, but apparently elements in union are matched by the order of columns in SELECT, instead of the names of the columns (even if name of the columns are the same).
Consider following Models:
class Person(models.Model):
first_name = models.CharField(max_length=256)
last_name = models.CharField(max_length=256)
age = models.IntegerField()
class Car(models.Model):
number = models.IntegerField()
brand = models.CharField(max_length=256)
name = models.CharField(max_length=256)
and following piece of code:
Person.objects.create(first_name="John", last_name="Smith", age=25)
Car.objects.create(number=42, name="Cybertruck", brand="Tesla")
q1 = Person.objects.all().annotate(name=F('first_name'), group=F('last_name'), number=F('age')).values(
'name', 'group', 'number')
q2 = Car.objects.all().annotate(group=F('brand')).values('name', 'group', 'number')
data = q1.union(q2)
print(data.query)
assert list(data) == [
{'name': 'John', 'group': 'Smith', 'number': 25},
{'name': 'Cybertruck', 'group': 'Tesla', 'number': 42},
])
As you can see I put correct order in .values().
What one could expect is that columns in union would be matched in the order passed to values (or by column names), but this is what happens:
SELECT "testunion_person"."first_name" AS "name", "testunion_person"."last_name" AS "group", "testunion_person"."age" AS "number" FROM "testunion_person" UNION SELECT "testunion_car"."name", "testunion_car"."number", "testunion_car"."brand" AS "group" FROM "testunion_car"
In the queries "testunion_car"."number" is before "testunion_car"."brand", which makes the Car in UNION have a values:
{'name': 'Cybertruck', 'group': '42', 'number': 'Tesla'}
EDIT: I am using 2.2 (LTS) version of Django

Instead of specifying the alias under annotate(), you can also specify them straight under values():
q1 = Person.objects.all().values(
name=F('first_name'), group=F('last_name'), xnumber=F('age'))
q2 = Car.objects.all().values(
'name', group=F('brand'), xnumber=F('number'))
I noticed that even then, it wasn't ordering the fields properly. I renamed the number field to xnumber to avoid conflicts with the model field of the same name and everything is grouped properly.

You can set the order of the fields using .values_list.
qs1 = Person.objects.values_list('name', 'group', 'number')
qs2 = Car.objects.values_list('brand', 'name', 'number')
qs1.union(qs2)
Check the docs for more detailed explanation.

Not a Django bug. Although query columns not sorted as values, the queryset display the right order:
In [13]: print(data)
<QuerySet [{'name': 'Cybertruck', 'group': 42, 'number': 'Tesla'}, {'name': 'John', 'group': 'Smith', 'number': 25}]>
It is because the data will be sorted after fetch from database. Source code snippet of QuerySet:
class QuerySet:
def __iter__(self):
"""
The queryset iterator protocol uses three nested iterators in the
default case:
1. sql.compiler.execute_sql()
- Returns 100 rows at time (constants.GET_ITERATOR_CHUNK_SIZE)
using cursor.fetchmany(). This part is responsible for
doing some column masking, and returning the rows in chunks.
2. sql.compiler.results_iter()
- Returns one row at time. At this point the rows are still just
tuples. In some cases the return values are converted to
Python values at this location.
3. self.iterator()
- Responsible for turning the rows into model objects.
"""
self._fetch_all()
return iter(self._result_cache)

Django: queryset "loosing" a value

I have 2 models, one with a list of clients and the other with a list of sales.
My intention is to add sales rank value to the clients queryset.
all_clients = Contactos.objects.values("id", "Vendedor", "codigo", 'Nombre', "NombrePcia", "Localidad", "FechaUltVenta")
sales = Ventas.objects.all()
Once loaded I aggregate all the sales per client summing the subtotal values of their sales and then order the result by their total sales.
sales_client = sales.values('cliente').annotate(
fact_total=Sum('subtotal'))
client_rank = sales_client .order_by('-fact_total')
Then I set the rank of those clients and store the value in a the "Rank" values in the same client_rank queryset.
a = 0
for rank in client_rank:
a = a + 1
rank['Rank'] = a
Everything fine up to now. When I print the results in the template I get the expected values in the "client_rank" queryset: "client name" + "total sales per client" + "Rank".
{'cliente': '684 DROGUERIA SUR', 'fact_total': Decimal('846470'), 'Rank': 1}
{'cliente': '699 KINE ESTETIC', 'fact_total': Decimal('418160'), 'Rank': 2}
etc....
The problem starts here
First we should take into account that not all the clients in the "all_clients" queryset have actual sales in the "sales" queryset. So I must find which ones do have sales, assign them the "Rank" value and a assign a standard value for the ones who don´t.
for subject in all_clients:
subject_code = str(client["codigo"])
try:
selected_subject = ranking_clientes.get(cliente__icontains=subject_code)
subject ['rank'] = selected_subject['Rank']
except:
subject ['rank'] = "Some value"
The Try always fails because "selected_subject" doesn´t seems to hace the "Rank" value. If I print the "selected_subject" I get the following:
{'cliente': '904 BAHIA BLANCA BASKET', 'fact_total': Decimal('33890')}
Any clues on why I´, lossing the "Rank" value? The original "client_rank" queryset still has that value included.
Thanks!

I presume that ranking_clientes is the same as client_rank.
The problem is that .get will always do a new query against the database. This means that any modifications you made to the dictionaries returned in the original query will not have been applied to the result of the get call.
You would need to iterate through your query to find the one you need:
selected_subject = next(client for client in ranking_clientes if subject_code in client.cliente)
Note, this is pretty inefficient if you have a lot of clients. I would rethink your model structure. Alternatively, you could look into using a database function to return the rank directly as part of the original query.

Django Postgresql JsonField query related dictionary keys

A part of the model that I have, which uses Django Model field, is like the following:
class SalesModel(models.Model):
some_data = models.PositiveIntegerField(db_index=True)
some_other_data = models.CharField(max_length=50)
json_data = JSONField(blank=True, null=True)
Now following is the format of the JsonData field:
[{"id": val, "contribution": "some_val", }, {"id": some_val, "contribution": "some_other_val",}, {"id": other_val, "contribution": "some_another_val"}]
i.e., the format is:
[{'id':XX, 'contribution':XX},{'id':YY, 'contribution':YY},{'id':ZZ, 'contribution':ZZ}]
Currently I can filter the Django table with the val of ID. I would now, like to know the contribution of that particular ID.
For eg, if val = 1, I would like to filter the model SalesModel which has JsonField with id = 1, and I want to show the related contribution. So, that would mean, out of the 3 possible dictionaries (as per the field construction), I would only show one dictionary (filtered by the 'ID' key of that dictionary). That would mean, if the 2nd dictionary has a matching ID, show only the 2nd contribution, if the 1st ID is matching show only the 1st contribution, and similarly for the 3rd dictionary.
Is there a way that can be done?

You could restructure your JSONField differently, by giving it a dict where the key, value pairs are id: contribution directly. This way you could use the has_key filter and KeyTransform will work, as I'm not sure it works on an array of dicts. So assuming your json_data looks like this:
{1: 'xx', 3: 'yy', 9: 'zz'}
you could query this way, based on #vanojx1 contribution:
SalesModel.filter(json_data__has_key=id)\
.annotate(contrib=KeyTransform(id, 'json_data')\
.values('contrib')
Alternatively, using raw jsonb in postgresql:
SalesModel.filter(json_data__has_key=id)\
.extra(select={'contrib': "json_data->{0}".format(id)})\
.values('contrib')

This should work DOC
SalesModel.objects.filter(json_data__id=1).values('id', 'json_data__contribution')

Yes, I guess. If I have understood it right, you will have an id to be matched or a list of ID's. So if your ID is 2:
my_id = 2
dict1 = [{"id":1, "contribution":10},{"id":2, "contribution":20},{"id":3, "contribution":30}]
for i in dict1:
if i["id"] == my_id:
print(i["contribution"])

getting a list after grouping in sqlalchemy

I have a table Food. It has fields: cust_name, phone_number , order_date
I am trying to build a dictionary where a key of pair (cust_name, phone_number) gives a list of order_date. For that I need to query appropriately in sqlalchemy. I'm using Postgres.
So far I have:
db.session.query(Food.cust_name, Food.phone_number).group_by(Food.cust_name, Food.phone_number).all()
What do I need to change so that I get a corresponding list of order_date

Use the array_agg() aggregate function to produce a list of order dates:
res = db.session.query(Food.cust_name,
Food.phone_number,
db.func.array_agg(Food.order_date).label('order_dates')).\
group_by(Food.cust_name, Food.phone_number).\
all()
the_dict = {(r.cust_name, r.phone_number): r.order_dates for r in res}

Django - Query models where id equals attribute on python list

I need to query all models where the ID matches the 'id' attribute of JSON array, something like that:
I have 3 saved model objects with respective ID's:
ID 1
ID 3
ID 4
I have a JSON array like that:
[{'id' : 1}, {'id' : 2}, {'id' : 5}]
I want to filter in that way:
model.objects.filter('Objects by ID that is not listed in the JSON array')
The result of the filter should be a list with models objects that the ID is not in the JSON:
result = [model_pk=3, model_pk=4]
Any ideas?

You can use exclude method to achieve that:
ids = [i['id'] for i in json_array]
qs = model.objects.exclude(id__in=ids)