We Keep Coding

Postgres: values query on json key with django

I need to do a values/values_list query on nested key on a postgres backed jsonfield in django 1.10
eg.
class AbcModel(models.model):
context = fields.JSONField()
If it has values like:
{
'lev1': {
'lev': 2
}
}
I want to run a queries like
AbcModel.objects.values('context__lev1__lev2').distinct()
AbcModel.objects.values_list('context__lev1__lev2', flat=True).distinct()
EDIT: The JSON fields are the official django JSONField from django.contrib.postgres.fields

So I found a solution, this works with django 1.10 and above.
I used the KeyTransform to annotate and extract the nexted key and did a values_list on that.
from django.contrib.postgres.fields.jsonb import KeyTransform
extracted_query = AbcModel.objects.annotate(lev1=KeyTransform('lev1', 'context')).annotate(lev2=KeyTransform('lev', 'lev1'))
This query allows me to use lev1 and lev2 as normal fields in the model, so I can do a values, values_list or any other valid query on the fields.
Django 1.11 allows to nest the the two Transforms in one annotate, not sure about 1.10 about the nesting as I have upgraded to 1.11

It's not ideal, but I was able to get this working by adding the json field as an extra field and then calling values on that extra field:
AbcModel.objects.extra(select={
"extra_field": "context->'lev1'->'lev2'"
}).values('extra_field').distinct()
AbcModel.objects.extra(select={
"extra_field": "context->'lev1'->'lev2'"
}).values_list('extra_field', flat=True).distinct()

django substitute dictionary with a faster JSON serializer

I'm using Django raw queryset to select data from database.
I will need a translation (by using ugettext) on a field before I return this json serialized data to django rest_framework as an API
However I'm having optimization issue as this I found out it takes quite a while to manually append dictionary to a list especially if I have a lot of database rows.
After some searching i found a library ujson that claims can serialize JSON faster. However I'm struggling to use this as I need this raw query to return translated name of a field (fruits)
Anyone have any idea how to replace this dictionary method with other faster method to serialize JSON data?
all_fruits = []
activate ("en")
raw_query = MyObject.objects.raw(" select id, fruits from my_table ")
for each_name in raw_query:
json_obj = dict( id = each_name.id,
fruits= ugettext(each_name.fruits)
)
all_fruits.append(json_obj)

It's better to avoid raw SQL only when you really have no other Solution, Django QuerySet provide very great and full-featured API for database query, the following solution could fit your needs:
all_fruits = []
activate ("en")
my_object_list = MyObject.objects.all()
for obj in my_object_list.values():
all_fruits.append({"id":obj.id, "fruits" : ugettext(obj.fruits)})

Does django core pagination retrieve all data first?

I am using django 1.5. I need to split pages to data. I read docs here. I am not sure about whether it retrieves all data first or not. Since I have a large table, it should be better to using something like 'limit'. Thanks.
EDIT
I am using queryset in ModelManager.
example:
class KeywordManager(models.Manager):
def currentkeyword(self, kw, bd, ed):
wholeres = super(KeywordManager, self).get_query_set() \
.values("sc", "begindate", "enddate") \
.filter(keyword=kw, begindate__gte=bd, enddate__lte=ed) \
.order_by('enddate')
return wholeres

First, a queryset is a lazy object, and django will retrieve the data as soon you request it, but if you dont, django won't hit the DB. If you use over a queryset any list methods as len(), you will evaluate all the queryset and forcing django to retrieve all the data.
If you pass a queryset to the Paginator, it would not retrieve all the data, because, as docs says, if you pass a queryset, it will use .count() methods avoiding converting the queryset into a list and the use of len() method.

If your data is not coming from the database, then yes - Paginator will have to load all the information first in order to determine how to "split" it.
If you're not and you're simply interacting with the database with Django's auto-generated SQL, then the Paginator performs a query to determine the number of items in the database (i.e. an SQL COUNT()) and uses the value you supplied to determine how many pages to generate. Example: count() returns 43, and you want pages of 10 results - the number of pages generated is equivalent to: 43 % 10 + 1 = 5

Need Optimization & Performance related to Django ORM Query

I am writing an API in django 1.4.5 which return JSON data to third party application.
This is my current method to retrieve data but it takes more time because i also need related data to be available in JSON.
def get_speakers(request)
speakers = Person.objects.filter(profile__person_type__name='Speaker').select_related('series')
for speaker in speakers:
data['first_name'] = speaker.first_name
data['last_name'] = speaker.last_name
data['series_name'] = speaker.series.name
data['series_id'] = speaker.series.id
return JSONResponse(data_dict)
To achieve optimization i tried as following.
def get_speakers(request)
speakers = Person.objects.filter(profile__person_type__name='Speaker').select_related('series')
data_dict = serializers.serialize("python", speakers)
return JSONResponse(data_dict)
But it returns foreign key for related data in JSON which is useless because i can't get its related data.
Also it try for raw sql then data return in tuple but we need in dictionary format.
Need help to achieve this.
Thanks in advance.

If I were you, I would use the following.
def get_speakers(request)
speakers = Person.objects.filter(profile__person_type__name='Speaker').values('first_name', 'last_name', 'series__name', 'series__id')
return JSONResponse(speakers)
In django ORM, while naming fields only one underscore _ is used. Since two underscore has another usage. In your example series is a foreign key, and if you would like to reach its fields while filtering and getting values, you could use like 'series__name' to get name field of related record.
Besides values method there is values_list method. You could use which is best for you.

django serialize foreign key objects

Serialize django model with foreign key models
Serializing Foreign Key objects in Django
get foreign key objects in a single query - Django
There are couple of question asking for the same thing already. But they are from 2010, and it didn't help me so much. So I figure it maybe been some update to this front since 2010?
On google I found this link, which explain usage of natural keys. However my problem concerns of getting foreign objects from django.contrib.auth.models.User so it doesn't help.
My problem is as following. I want to serialize the QuerySet so I get the foreign key objects also, because I want to pass it as JSON to the client. The serializer from django.core doesn't do that. So in my case to simply the problem I had added another field to the model to contain the value I need from the foreign object. But it however introduce redundant data.
My example model it contains the username which I would like if possible remove, and instead get it by the foreign key.
user = models.ForeignKey(User)
username = models.CharField(max_length=100, null=False)

One potential way around this is to construct your own dictionary object based on the returns of a queryset. You'd do something like this:
queryset = Model.objects.all()
list = [] #create list
for row in queryset: #populate list
list.append({'title':row.title, 'body': row.body, 'name': row.user.username})
recipe_list_json = json.dumps(list) #dump list as JSON
return HttpResponse(recipe_list_json, 'application/javascript')
You need to import json for this to work.
import json

You could use Django REST frameworks' serializers.