I have a django model that has the following four fields:
class File:
id = models.PrimaryKey()
name = models.CharField()
is_active = models.BooleanField()
data = models.JSONField()
The data field is massive, perhaps 5MB per entry. Is there a way I can hide that field when I do an ORM query without having to specify all the fields I want to view each time? Something like:
File.objects.all() # exclude data field
File.objects.values('id', 'data') # include the data field
In some complex data-modeling situations, your models might contain a lot of fields, some of which could contain a lot of data (for example, text fields), or require expensive processing to convert them to Python objects. If you are using the results of a queryset in some situation where you don’t know if you need those particular fields when you initially fetch the data, you can tell Django not to retrieve them from the database.
This is done by passing the names of the fields to not load to defer():
Entry.objects.defer("headline", "body")
Also mention that, Whenever you call only() it replaces the set of fields to load immediately. The method’s name is mnemonic: only those fields are loaded immediately; the remainder are deferred.
You can use only() to specify the fields you want
File.objects.only('id', 'data')
Related
My data set in the table is like this, lets call table name as plans_tracker (1st screen shot) I am trying to retrieve query set like (2nd screen shot). can some one please help me on this, I could not modify the table structure. I am trying to do this in Django templates
It is possible when using PostgreSQL. I'm not sure how to achieve it with other DB backends though. Note that your way of storing data is not ideal and a better solution can be used. There are 2 examples of better solutions below, one of them is database independent.
Let's consider you have a model defined as follows (I've replaced irrelevant fields from your example with a single char field):
from django.core.validators import validate_comma_separated_integer_list
from django.db import models
class Plan(models.Model):
group = models.CharField(max_length=256)
student_course_records = models.TextField(validators=[validate_comma_separated_integer_list])
Given that, you can convert students list to a PostgreSQL array using string_to_array through Django's Func:
Plan.objects.annotate(
students=Func(
F('student_course_records'),
Value(','),
function='string_to_array',
output_field=ArrayField(models.IntegerField())
)
)
That will add a "virtual" field to your object, named students. It will represent all student IDs in an array that is easier to process later.
As you now have an array of IDs, you can explode the values into separate records, using unnest function:
Plan.objects.annotate(
student=Func(
Func(
F('student_course_records'),
Value(','),
function='string_to_array',
output_field=ArrayField(models.IntegerField())
),
function='unnest'
)
)
Now, instead of students field, you will have separate records, each one with a single student value. Note that student_course_records is still available in every record, but you can change it using values, values_list, only or exclude.
To simplify writing those queries, you can subclass Func
class StringToArray(models.Func):
function = 'string_to_array'
def __init__(self, *args, output_field, **kwargs):
super().__init__(*args, output_field=ArrayField(output_field), **kwargs)
class Unnest(models.Func):
function = 'unnest'
arity = 1
And then your call will look like:
Plan.objects.annotate(
student=Unnest(
StringToArray(
F('student_course_records'),
Value(','),
output_field=models.IntegerField(),
)
)
)
As I've said before, this data model is not ideal, as it requires parsing the comma separated list on each data access. Not to mention the troubles with querying the data over this field. There are 2 better approaches: use the ArrayField instead of a comma separated text field or use a nested model. First one will still require a PostgreSQL database, a 2nd approach can be used with any database officially supported by Django, but may seem noisy and redundant in such simple use case. For the first approach, you would define your model as:
class Plan(models.Model):
group = models.CharField(max_length=256)
student_course_records = ArrayField(models.IntegerField())
For this model, your query will be simplified to:
Plan.objects.annotate(
student=Unnest(F('student_course_records'))
)
And the student_course_records will be represented as an array in all cases, without the need to convert it to anything else.
2nd approach would look like:
class Plan(models.Model):
group = models.CharField(max_length=256)
class Student(models.Model):
plan = models.ForeignKey(Plan, related_name='student_course_records', on_delete=models.CASCADE
For this model, your query will be simplified to the:
Student.objects.all()
With an optional select_related or values calls to extract the data from Plan model at the same time.
I have a model with some fields, one of them is a FK. I would like to use
MyModel.objects.all().defer("pk").values()
and to add a field of my FK (myfk__name) without writing all model fields + 1 in the values.
I also would like to avoid doing another query after this one to add it manually.
Is it possible?
use annotate:
MyModel.objects.all().defer('pk').values().annotate(name=F('myfk__name'))
I have a model where I needed historical data for a couple specific fields, so I put those fields into a separate model with a foreign key relationship.
Something sort of like this:
class DataThing(models.Model):
# a bunch of fields here...
class DataThingHistory(models.Model):
datathing_id = models.ForeignKey('DataThing', on_delete=models.CASCADE)
text_with_history = models.CharField(max_length=500, null=True, blank=True)
# other similar fields...
timestamp = models.DateTimeField()
Now I'm trying to filter the former model using a text field in the latest corresponding entry in the latter.
Basically if these were not separate models I'd just try this:
search_results = DataThing.objects.filter(text_with_history__icontains=searchterm)
But I haven't figured out a good way to do this across this one-to-many relationship and using only the entry with the latest timestamp in the latter model, at least by using the Django ORM.
I have an idea of how to do the query I want using raw SQL, but I'd really like to avoid using raw if at all possible.
This solution makes use of distinct(*fields) which is currently only supported by Postgres:
latest_things = DataThingHistory.objects.
order_by('datathing_id_id', '-timestamp').
distinct('datathing_id_id')
lt_with_searchterm = DataThingHistory.objects.
filter(id__in=latest_things, text_with_history__icontains=searchterm)
search_results = DataThing.objects.filter(datathinghistory__in=lt_with_searchterm)
This should result in single db query. I have split the query for readability, but you can nest it into a single statement. Btw, as you might see here, foo_id is not a good name for a ForeignKey field.
You would do the same by querying DataThing while referring to DataThingHistory:
search_results = DataThing.objects.filter(datathinghistory__text_with_history__icontains=searchterm)
Check django doc on how to query on reverse relationship.
Edit:
My previous answer is incomplete. In order to search on latest history for each DataThing, you need to annotate on timestamp using Max:
from django.db.models import Max
search_results = search_results.values('field1', 'field2',...).annotate(latest_history=Max('datathinghistory__timestemp'))
This wouldn't give you complete DataThing objects, but you could add as many fields to values as you want.
I was recently adding large amounts of data from a JSON file to my Django "member" model. For this purpose I had to make most of my variables able to take in null or blank in order to handle cases when the JSON file did not have information on a specific variable etc etc.
event = models.CharField(max_length=200, blank=True, null=True)
tool = models.CharField(max_length=200, blank=True, null=True)
other = models.CharField(max_length=200)
protocol = models.CharField(max_length=200)
However, I accidentally forgot to change 2 variables for this purpose. Surprisingly my objects were generated without a problem even when the JSON data was not adding anything to this variables.
So here is my question(s):
How was the member object created when some variables that are not
allowed to be null, were actually null or at the very least had no
data.
I thought this restriction was at the Database level but it seems
here like it is only for user submitted data since when I created
the object internally it actually allowed me to have several
null/blank variables.
Is Django taking care of something behind the scenes for me?
I know this is a lot but I hope it makes sense.... thanks
First, Avoid using null on string-based fields such as CharField and TextField unless you have an excellent reason (https://docs.djangoproject.com/en/dev/ref/models/fields/#null)
The following schema would be generated for your fields in DB
event character varying(200),
tool character varying(200),
other character varying(200) NOT NULL,
protocol character varying(200) NOT NULL,
This would mean that you can enter "NULL" values in database for 'event' and 'tool' field but not for 'other' and 'protocol' fields.
Yet, you can enter a blank text for all the fields irrespective of weather they are "blank=True" or not. Reason being that "Blank=True" is only used by Django for its own validations and its not reflected in Database schema.
I have quite unique problem with django.
Im providing website users interface for editing large data. Each row on this data represents a row in database. Or one object of certain Type.
Users click on cells in the table and form opens where they can edit this fields/column value.
In essence it works like this:
1) based on where user clicks, query is sent to server containting object id and the field that he is editing.
2) based on this information form is created on the fly:
class FieldEditorForm(forms.ModelForm):
class Meta:
model = MyObject
fields = ['id', field ]
Notice the field there is Variable not name of the field.
3) this field passes its own modelform validation and all is fine. in save method Model.save() is enough to update the value.
But now to the problem. Sometimes empty value is sent to server in this form. Empy value such as u'' or almost emtpty like u' '. I want to repace this with None so NULL would be saved to database.
There are two places where i could do that. In field validation modifying the cleaned_data or in form save method.
Both approaches raise unique problem as i dont know how to create variable function names.
def clean_%(field)s():
or in case of form save method
r.%(field)s = None
is what i need, but those methods dont work. So how can i create method name which is variable or set objects variable parameter to something. Is it even possible or do i have to rethink my approach there?
Alan
In the latter case, setattr(r, field + 's', None).