We Keep Coding

How do you incrementally add lexeme/s to an existing Django SearchVectorField document value through the ORM?

You can add to an existing Postgresql tsvector value using ||, for example:
UPDATE acme_table
SET my_tsvector = my_tsvector ||
to_tsvector('english', 'some new words to add to existing ones')
WHERE id = 1234;
Is there any way to access this functionality via the Django ORM? I.e. incrementally add to an existing SearchVectorField value rather than reconstruct from scratch?
The issue I'm having is the SearchVectorField property returns the tsvector as a string. So when I use the || operator as +, eg:
from django.contrib.postgres.search import SearchVector
instance.my_tsvector_prop += SearchVector(
["new", "fields"],
weight="A",
config='english'
)
I get the error:
TypeError: SearchVector can only be combined with other SearchVector instances, got str.
Because:
type(instance.my_tsvector_prop) == str
A fix to this open Django bug whereby a SearchVectorField property returns a SearchVector instance would probably enable this, if possible. (Although less efficient than combining in the database. In our case the update will run asynchronously so performance is not too important.)
MyModel.objects
.filter(pk=1234)
.update(my_tsvector_prop=
F("my_tsvector_prop") +
SearchVector(
["new_field_name"],
weight="A",
config='english')
)
)
Returns:
FieldError: Cannot resolve expression type, unknown output_field
Another solution would be to run a raw SQL UPDATE, although I'd rather do it through the Django ORM if possible as our tsvector fields often reference values many joins away, so it'd be nice to find a sustainable solution.

is it possible to return a single value from Q in Django?

I am using the Django ORM and I would like to do something like:
self.queryset.annotate(cupcake_name=Q(baked_goods__frosted_goods__cupcakes__cupcake_id='xxx'))
but return the individual cupcake name field value somehow so I can serve it as a data attribute.

Assuming you have the cupcake id and you want to do this in a single query you can use a Subquery:
from django.db.models import Subquery
self.queryset.annotate(
cupcake_name=Subquery(Cupcake.objects.filter(id='xxx').values('name')),
)
See the Subquery docs here if you need to link the subquery to the queryset: https://docs.djangoproject.com/en/3.1/ref/models/expressions/#subquery-expressions
If you don't mind making two queries it's a bit more clear to use the literal Value expression:
from django.db.models import Value
cupcake_name = Cupcake.objects.get(id='xxx').name
self.queryset.annotate(
cupcake_name=Value(cupcake_name),
)

Django: Admin - order results based on regular expression

I have a problem where I want to order model rows in Django admin based on a number within a string and ignore the letters - like XX0345XX, X0346XXX, XXX0347XX. I'm thinking that I should probably use a regular expression for that. I have an SQL query (PostgreSQL) that returns what I want:
select * from mytable order by substring(my_field, '\d+')::int DESC
But I'm having trouble applying it to get the same result in the Django admin get_queryset().
I've tried doing something like:
def get_queryset():
return Model.objects.raw("select * from mytable order by substring(my_field, '\d+')::int DESC")
but the problem is that I'm not returning the right type this way. Model.objects.raw('...') returns RawQuerySet, but get_queryset() should return QuerySet instance not RawQuerySet one, so I can't do it this way.
Any suggestions on how to solve this problem? Thanks!

You can use the .extra() method of to convert from a rawqueryset to a queryset, see here
This example is taken from, here
class CustomManager(manager.Manager):
def get_queryset():
qs = self.get_queryset()
sql = "myapp_model_b.id IN (SELECT UNNEST(myapp_model_a.pk_values) FROM myapp_model_a WHERE myapp_model_a.id='%s')" % index_id
return qs.extra(where=[sql])

As pointed out by #Azy_Crw4282, you can use QuerySet.extra() to run raw SQL queries and still have the results returned as a QuerySet instance.
Here's how I managed to do a regex based "ORDER BY" on objects in admin view, based on a number within a string. I.e. order fields like XX0345XX, X0346XXX, XXX0347XX, based on the number they contain - basically get a substring that matches '\d+' regular expression.
Just override the get_queryset() function in your admin.ModelAdmin class.
Solution:
def get_queryset(self, request):
sql_query = "CAST(substring(booking_reference, '\d+') as int)"
bookings = Booking.objects.extra(select={'book_ref_digits': sql_query}).order_by('-book_ref_digits')
return bookings
As far as I understand, it adds a new temporary field, e.g. book_ref_digits in the QuerySet object and then you can do .order_by() on that.
Note: Using older version of Django 1.10.5

Django JSONField inside ArrayField

I have a problem inserting to a field using ArrayField with JSONField inside.
models.py
locations = ArrayField(JSONField(null = True,blank = True), blank=True, null = True)
Insert
location_arr = [{"locations" : "loc1","amount":Decimal(100.00)},{"locations" : "loc2","amount":Decimal(200.25)}]
instance.locations = location_arr
instance.save()
When I do this, I got
column "locations" is of type jsonb[] but expression is of type text[]
LINE 1: ...d" = 2517, "locations" = ARRAY['{"loc...
Hint: You will need to rewrite or cast the expression.
So I tried to dump it using:
import json
location_arr = [{"locations" : "loc1","amount":Decimal(100.00)},{"locations" : "loc2","amount":Decimal(200.25)}]
instance.locations = json.dumps(location_arr)
instance.save()
then I got this
LINE 1: ...d" = 2517, "locations" = '[{"loc":...
DETAIL: "[" must introduce explicitly-specified array dimensions.
I am using:
Django 1.9
Python 2.7
Postgres 9.4.10
psycopg2 2.6.2

Arrays
First of all, let's take a close look at this important text from the Postgresql Arrays document.
Tip: Arrays are not sets; searching for specific array elements can be
a sign of database misdesign. Consider using a separate table with a
row for each item that would be an array element. This will be easier
to search, and is likely to scale better for a large number of
elements.
Most of the time, you should not be using arrays.
JSONB
JSONB is available in Django as the JSONField type. This field is more scalable and flexible than array fields and can be searched more efficiently. However if you find yourself searching inside JSONB fields all the time the above statement about Arrays is equally valid for JSONB.
Now what do you have in your system? A an array that holds JSONB field. This is a disaster waiting to happen. Please normalize your data.
Recap
so when to use ArrayField?
On the rare occasion when you don't need to search in that column and you don't need to use that column for a join.

I have encountered the same scenario. Here is the way how I solved it
models.py
from django.contrib.postgres.fields.jsonb import JSONField as JSONBField
location = JSONBField(default=list,null=True,blank=True)
insert
model_object.location = [{"locations" : "loc1","amount":Decimal(100.00)},{"locations" : "loc2","amount":Decimal(200.25)}]
update
model_object.location.append({"locations" : "loc1","amount":Decimal(100.00)})
model_object.save()
This worked for me in
Django - 2.0.2
Postgres - 9.5
psycopg2 - 2.7.4
python - 3.4.3

You can sidestep this issue by using the JSONField as the column field type with a list as the root element.
from django.contrib.postgres.fields import JSONField
class MyDBArray(models.Model):
array_data = models.JSONField(default=list)
my_db_array = MyDBArray(array_data=[1, 2, 3])
my_db_array.save()
You would need to validate in the save method that the array_data field is actually list-like.

This was fixed in the latest unreleased version of Django 2.2a1
pip install Django==2.2a1
PS I believe that it will work with versions >= 2.2a1

I guess the easiest way is to turn the field from an array of jsonfield into a jsonfield. Just add a key to the location_arr.
Related thread: ArrayField with JSONField as base_field in Django

django orm group by json key in json field

I'm using json field on my django model:
class JsonTable(models.Model):
data = JSONField()
type = models.IntegerField()
I tried next query, which works for normal sql fields:
JsonTable.objects.filter(type=1).values('type').annotate(Avg('data__superkey'))
But this throws next error:
FieldError: Cannot resolve keyword 'superkey' into field. Join on 'data' not permitted.
Is there way to make group by on json key, using Django ORM or some python lib, without use of raw sql?
Versions: Django 1.9b, PostgreSQL 9.4
UPDATE
Example 2:
JsonTable.objects.filter(type=1).values('data__happykey').annotate(Avg('data_superkey'))
throws same error on happykey

After some researching I found next solution:
from django.db.models import Count
from django.contrib.postgres.fields.jsonb import KeyTextTransform
superkey = KeyTextTransform('superkey', 'data')
table_items = JsonTable.objects.annotate(superkey = superkey).values('superkey').annotate(Count('id')).order_by()
I did not sure about order_by(), but documentation says that is needed.
For another aggregation function type casting needed:
from django.db.models import IntegerField
from django.db.models.functions import Cast
superkey = Cast(KeyTextTransform('superkey', 'data'), IntegerField())
I test with another model, hope that write this code without misprints. PostgreSQL 9.6, Django 2.07

If you are using this package https://github.com/bradjasper/django-jsonfield,
there is nothing in the code for managing such simulated related queries (data__some_json_key)
As Json data is text, you will have to go to raw sql or better : use queryset extra() method, but parsing Json in sql seems to be difficult.