We Keep Coding

Django ORM : model with a list of items from another model

In my Django models, I have two models : one called Site and the other SiteFeature.
Object-wise, it is very clear how this should work : every instance of the Site class should have as property a list containing instances of the SiteFeature class, simply because the SiteFeature objects should only exist in relation to a Site object.
Database-wise, it is also very clear how it should work : the SiteFeature table should contain a not-nullable column referencing the primary key id column of the Site table, with a foreign key.
But in terms of Django ORM, I don't know how to code this.
Based on this question, and this other example, it seems the classical way to proceed works the other way round :
The Site model class contains no ORM model field referencing the SiteFeature list.
Instead, the SiteFeature ORM model class has a ForeignKey field referencing the Site class.
I see there is a way to code this out : by adding a function to the Site model class that searches all the related SiteFeature, and make this function a property (decorator #property):
#property
def site_features(self):
return SiteFeature.objects.filter(site_id=site_id)
But that leaves me doubts :
The proper logic for me would also be that when I save, update or create an instance of the Site class, it would also automatically save / update / create the instances of SiteFeature that are related to it. (same thing for deleting the object, but that can be covered by the on_delete=models.CASCADE parameter of the ForeignKey field).
I could add my own save_with_features / update_with_features / create_with_features methods that cascade all but I am not sure what would happen in case of calls made automatically by Django to the standard save / update / create such as in bulk operations.
This problem seems to basic that I suppose there is already a proper way to do it. How would that be ?

Eventually, I solved the problem with the sitefeature_set Manager.
Reference: https://docs.djangoproject.com/en/3.0/topics/db/queries/#following-relationships-backward

Where is the declaration of objects of django model?

In django documentation, it say that we can retrieve data entry as below
entry = Entry.objects.get(pk=1)
Entry is a model class in models.py. I tried to find the declaration of objects, but I can't find its declaration in manager.py, just know it is a instance of Manager. So, where is the declaration of objects? Does it represent a set of Entry instances?

When you define model, you extend Model class from django.db.models module.
It will provide default model manager in objects property.
If you want to define custom model manager, you can do it by subclassing django.db.models.Manager class.
Look at the docs how to do that: https://docs.djangoproject.com/en/1.11/topics/db/managers/
Add methods to custom model managers to if you want to manipulate with the collection of data. If you manipulate with single row, add method to your model.

It's defined in ModelBase, which is the metaclass for model classes. See https://github.com/django/django/blob/master/django/db/models/base.py#L360

'makemigrations' asks for a default with none-abstract model inheritance

I'm trying a simple model inheritance:
class Product(models.Model):
creation_date = models.DateTimeField(auto_now_add=True)
class Application(Product):
name = models.CharField(max_length=200)
'makemigrations' asks for a default:
You are trying to add a non-nullable field 'product_ptr' to
application without a default; we can't do that (the database needs
something to populate existing rows).
I saw here that I could Product an abstract model with a Meta class, but I can't do that since I'm refereing it specifically in other models as an actual model:
class Comment(models.Model):
product = models.ForeignKey('Product', related_name="comments")
Running 'makemigrations' when the database is removed also leads to the same issue.
Anything I can do?
Django 1.9

You haven't explained what exactly is the change that you have made, it seems that you have changed your Application model to inherit from Product where it previously inherited from models.Model. That causes django to create a 1 to 1 mapping behind the scenes. The addition of the product_ptr which you didn't add to the model yourself enters the picture
ref: https://docs.djangoproject.com/en/1.9/topics/db/models/#multi-table-inheritance
The inheritance relationship introduces links between the child model
and each of its parents (via an automatically-created OneToOneField).
It's a bit tricky to add this field to a table with data during a migration because this field needs to be unique. If you are merely creatig a new model named Application, a value of 1 would be ok.

Using dynamic models in Django framework

I am currently using Django framework including its Models mechanism to abstract the database schema declaration and general db access, which is working fine for most scenarios.
However, my application also requires tables to be created and accessed dynamically during runtime, which as far as I can see, is not supported by Django out of the box.
These tables usually have an identical structure, and can basically be abstracted by the same Model class, but Django doesn't let you change the underlying db_table of a certain model query, as it is declared on the Model class and not on the Manager.
My solution for this is to do this process whenever I need a new table to be created, populated and accessed:
Create and populate the table using raw sql
Add indexes to the table using raw sql
When I need to access the table (using django queryset api), I declare a new type dynamically and return it as the model for the query, by using this code:
table_name = # name of the table created by sql
model_name = '%d_%s' % (connection.tenant.id, table_name)
try:
model = apps.get_registered_model('myapp', model_name)
return model
except LookupError:
pass
logger.debug("no model exists for model %s, creating one" % model_name)
class Meta:
db_table = table_name
managed = False
attrs = {
'field1' : models.CharField(max_length=200),
'field2' : models.CharField(max_length=200),
'field3' : models.CharField(max_length=200)
'__module__': 'myapp.models',
'Meta':Meta
}
model = type(str(model_name), (models.Model,), attrs)
return model
Note that I do check if the model is already registered in django and I'm using an existing model in case it does. The model name is always unique for each table. Since I'm using multi tenants, the tenant name is also part of the model name to avoid conflict with similar tables declared on different schemas.
In case it's not clear: the tables created dynamically will and should be persisted permanently for future sessions.
This solution works fine for me so far.
However, the application will need to support a large number of these tables. i.e. 10,000 - 100,000 such tables(and corresponding model classes), with up to a million rows per table.
Assuming the underlying db is fine with this load, my questions are:
Do you see any problem with this solution, with and without regards to the expected scale ?
Anybody has a better solution for this scenario ?
Thanks.

There is a wiki page on creating models dynamically, although it has been a while since it was last updated:
DynamicModels Django
There are also a few apps that are designed for this use case, but I don't think any of them is being actively maintained:
Django Packages: Dynamic models
I understand that if you are already committed to Django this isn't very helpful, but this a use case for which Django isn't really good. It might be more costly to fight against the abstractions provided by Django's model layer, than to just use psycopg2 or whatever other adapter is appropriate for your data.
Depending on what sort of operations you are going to perform on your data, it may be also more reasonable to use a single model with an indexed field that allows you to distinguish in which table that row would be and then sharding the data by that column.
If you still need to do this, the general idea would be:
Create a metaclass that extends Django's ModelBase. This metaclass you would use as a factory for your actual models.
Consider stuff mentioned on that wiki page, like circumventing the app_label issue.
Generate and execute the sql for the creation of the model as also shown on the wiki page.

Any Python ORM make full use of NoSQL's schema-less feature?

I tried Django-nonrel(which is a NoSQL fork of Django project) with MongoDB. In it's ORM, class can only inherit from abstract class, so even two class are inherit from the same base class, they are saved into different collections in MongoDB.
This design added a pre-defined schema to the schema-less database, every collection has its unique schema now.
What I want is, if ClassA and ClassB are inherited from BaseClass, every instance of ClassA and ClassB should be put in the same collection ( or something similar in NoSQL other than MongoDB )

The goal of Django-nonrel is to maintain the same behaviors as Django's ORM has, but to allow use of non-relational storage backends. Since Django doesn't permit multiple different models to be stored in the same (relational) table, I'm not surprised to learn that Django-nonrel doesn't either.
If you want the behavior you have described, you might check out Mongoengine, an ODM (object-document mapper) for MongoDB, which is inspired by and has a similar feel to the Django ORM, but which allows more flexible schemas and behaviors, like the one you've described.
In particular, when using Mongoengine, you can create multiple classes whose records are stored in the same collection like:
class BlogPost(mongoengine.Document):
create_date = mongoengine.DateTimeField()
mod_date = mongoengine.DateTimeField()
title = mongoengine.StringField()
meta = {
'abstract': True,
}
def save(self):
if not self.pk:
self.create_date = datetime.utcnow()
self.mod_date = datetime.utcnow()
super(BlogPost, self).save()
class TextBlogPost(BlogPost):
body = mongoengine.StringField()
class ImageBlogPost(BlogPost):
image = mongoengine.FileField()
caption = mongoengine.StringField()
Since TextBlogPost and ImageBlogPost both inherit from BlogPost, they will both be saved in the same collection (by default, named "blog_post").