I'm using json field on my django model:
class JsonTable(models.Model):
data = JSONField()
type = models.IntegerField()
I tried next query, which works for normal sql fields:
JsonTable.objects.filter(type=1).values('type').annotate(Avg('data__superkey'))
But this throws next error:
FieldError: Cannot resolve keyword 'superkey' into field. Join on 'data' not permitted.
Is there way to make group by on json key, using Django ORM or some python lib, without use of raw sql?
Versions: Django 1.9b, PostgreSQL 9.4
UPDATE
Example 2:
JsonTable.objects.filter(type=1).values('data__happykey').annotate(Avg('data_superkey'))
throws same error on happykey
After some researching I found next solution:
from django.db.models import Count
from django.contrib.postgres.fields.jsonb import KeyTextTransform
superkey = KeyTextTransform('superkey', 'data')
table_items = JsonTable.objects.annotate(superkey = superkey).values('superkey').annotate(Count('id')).order_by()
I did not sure about order_by(), but documentation says that is needed.
For another aggregation function type casting needed:
from django.db.models import IntegerField
from django.db.models.functions import Cast
superkey = Cast(KeyTextTransform('superkey', 'data'), IntegerField())
I test with another model, hope that write this code without misprints. PostgreSQL 9.6, Django 2.07
If you are using this package https://github.com/bradjasper/django-jsonfield,
there is nothing in the code for managing such simulated related queries (data__some_json_key)
As Json data is text, you will have to go to raw sql or better : use queryset extra() method, but parsing Json in sql seems to be difficult.
Related
I need to do a values/values_list query on nested key on a postgres backed jsonfield in django 1.10
eg.
class AbcModel(models.model):
context = fields.JSONField()
If it has values like:
{
'lev1': {
'lev': 2
}
}
I want to run a queries like
AbcModel.objects.values('context__lev1__lev2').distinct()
AbcModel.objects.values_list('context__lev1__lev2', flat=True).distinct()
EDIT: The JSON fields are the official django JSONField from django.contrib.postgres.fields
So I found a solution, this works with django 1.10 and above.
I used the KeyTransform to annotate and extract the nexted key and did a values_list on that.
from django.contrib.postgres.fields.jsonb import KeyTransform
extracted_query = AbcModel.objects.annotate(lev1=KeyTransform('lev1', 'context')).annotate(lev2=KeyTransform('lev', 'lev1'))
This query allows me to use lev1 and lev2 as normal fields in the model, so I can do a values, values_list or any other valid query on the fields.
Django 1.11 allows to nest the the two Transforms in one annotate, not sure about 1.10 about the nesting as I have upgraded to 1.11
It's not ideal, but I was able to get this working by adding the json field as an extra field and then calling values on that extra field:
AbcModel.objects.extra(select={
"extra_field": "context->'lev1'->'lev2'"
}).values('extra_field').distinct()
AbcModel.objects.extra(select={
"extra_field": "context->'lev1'->'lev2'"
}).values_list('extra_field', flat=True).distinct()
I'm using Django 1.9 and Postgres 9.5. I have a model (MyModel) with a JSONField (django.contrib.postgres.fields). The JSON value has the same structure in all the objects in the model.
{
"key1": int_val_1,
"key2": int_val_2
}
I want to order my queryset by a value of a specified key in my JSONField. Something like
MyModel.objects.annotate(val=F('jsonfield__key1')).order_by('val')
This does not work - i get the following error
Cannot resolve keyword 'key1' into field. Join on 'jsonfield' not permitted.
Is there anyway i can achieve this?
for Django >= 1.11, you can use:
from django.contrib.postgres.fields.jsonb import KeyTextTransform
MyModel.objects.annotate(val=KeyTextTransform('key1', 'jsonfield')).order_by('val')
An updated answer for this question:
At current time of writing (Django 3.1), you can now order exactly as you had hoped without needing any helper functions:
MyModel.objects.order_by('jsonfield__key1')
JSONExtract is use with mysql
Syntax ('json-field-name', 'json-field-key')
~Q is used for "not equal"
from django_mysql.models.functions import JSONExtract
Listing.objects.annotate(weight=JSONExtract('package_dimensions','$.p>ackage_weight')).filter(~Q(weight=0))
I have a problem inserting to a field using ArrayField with JSONField inside.
models.py
locations = ArrayField(JSONField(null = True,blank = True), blank=True, null = True)
Insert
location_arr = [{"locations" : "loc1","amount":Decimal(100.00)},{"locations" : "loc2","amount":Decimal(200.25)}]
instance.locations = location_arr
instance.save()
When I do this, I got
column "locations" is of type jsonb[] but expression is of type text[]
LINE 1: ...d" = 2517, "locations" = ARRAY['{"loc...
Hint: You will need to rewrite or cast the expression.
So I tried to dump it using:
import json
location_arr = [{"locations" : "loc1","amount":Decimal(100.00)},{"locations" : "loc2","amount":Decimal(200.25)}]
instance.locations = json.dumps(location_arr)
instance.save()
then I got this
LINE 1: ...d" = 2517, "locations" = '[{"loc":...
DETAIL: "[" must introduce explicitly-specified array dimensions.
I am using:
Django 1.9
Python 2.7
Postgres 9.4.10
psycopg2 2.6.2
Arrays
First of all, let's take a close look at this important text from the Postgresql Arrays document.
Tip: Arrays are not sets; searching for specific array elements can be
a sign of database misdesign. Consider using a separate table with a
row for each item that would be an array element. This will be easier
to search, and is likely to scale better for a large number of
elements.
Most of the time, you should not be using arrays.
JSONB
JSONB is available in Django as the JSONField type. This field is more scalable and flexible than array fields and can be searched more efficiently. However if you find yourself searching inside JSONB fields all the time the above statement about Arrays is equally valid for JSONB.
Now what do you have in your system? A an array that holds JSONB field. This is a disaster waiting to happen. Please normalize your data.
Recap
so when to use ArrayField?
On the rare occasion when you don't need to search in that column and you don't need to use that column for a join.
I have encountered the same scenario. Here is the way how I solved it
models.py
from django.contrib.postgres.fields.jsonb import JSONField as JSONBField
location = JSONBField(default=list,null=True,blank=True)
insert
model_object.location = [{"locations" : "loc1","amount":Decimal(100.00)},{"locations" : "loc2","amount":Decimal(200.25)}]
update
model_object.location.append({"locations" : "loc1","amount":Decimal(100.00)})
model_object.save()
This worked for me in
Django - 2.0.2
Postgres - 9.5
psycopg2 - 2.7.4
python - 3.4.3
You can sidestep this issue by using the JSONField as the column field type with a list as the root element.
from django.contrib.postgres.fields import JSONField
class MyDBArray(models.Model):
array_data = models.JSONField(default=list)
my_db_array = MyDBArray(array_data=[1, 2, 3])
my_db_array.save()
You would need to validate in the save method that the array_data field is actually list-like.
This was fixed in the latest unreleased version of Django 2.2a1
pip install Django==2.2a1
PS I believe that it will work with versions >= 2.2a1
I guess the easiest way is to turn the field from an array of jsonfield into a jsonfield. Just add a key to the location_arr.
Related thread: ArrayField with JSONField as base_field in Django
I'm using Django raw queryset to select data from database.
I will need a translation (by using ugettext) on a field before I return this json serialized data to django rest_framework as an API
However I'm having optimization issue as this I found out it takes quite a while to manually append dictionary to a list especially if I have a lot of database rows.
After some searching i found a library ujson that claims can serialize JSON faster. However I'm struggling to use this as I need this raw query to return translated name of a field (fruits)
Anyone have any idea how to replace this dictionary method with other faster method to serialize JSON data?
all_fruits = []
activate ("en")
raw_query = MyObject.objects.raw(" select id, fruits from my_table ")
for each_name in raw_query:
json_obj = dict( id = each_name.id,
fruits= ugettext(each_name.fruits)
)
all_fruits.append(json_obj)
It's better to avoid raw SQL only when you really have no other Solution, Django QuerySet provide very great and full-featured API for database query, the following solution could fit your needs:
all_fruits = []
activate ("en")
my_object_list = MyObject.objects.all()
for obj in my_object_list.values():
all_fruits.append({"id":obj.id, "fruits" : ugettext(obj.fruits)})
I have a model with a text field on it. I want to do a lookup that will return all the items that have a string with a length of 7 or more in that field. Possible?
How about a lookup for all objects in which that field isn't ''?
I think regex lookup can help you:
ModelWithTextField.objects.filter(text_field__iregex=r'^.{7,}$')
or you can always perform raw SQL queries on Django model:
ModelWithTextField.objects.raw('SELECT * FROM model_with_text_field WHERE LEN_FUNC_NAME(text_field) > 7')
where len_func_name is the name of "string length" function for your DBMS. For example in mysql it's named "length".
Since django 1.8, you can use the Length database function:
from django.db.models.functions import Length
qs = ModelWithTextField.objects \
.annotate(text_len=Length('text_field_name')) \
.filter(text_len__gte=7)
Since Django 1.9 the Length database function may be registered as a queryset lookup:
from django.db.models import CharField
from django.db.models.functions import Length
CharField.register_lookup(Length, 'length')
So that then you are able use it in a following manner:
# Get authors whose name is longer than 7 characters
authors = Author.objects.filter(name__length__gt=7)
See: Database Functions - Length() in official Django 3.2 Docs