We Keep Coding

Getting data from related tables using Django ORM

Using the Django ORM, how does one access data from related tables without effectively making a separate call for each record (or redundantly denormalizing data to make it more easily accessible)?
Say I have 3 Models:
class Tournament(models.Model):
name = models.CharField(max_length=250)
active = models.BooleanField(null=True,default=1)
class Team(models.Model):
name = models.CharField(max_length=250)
coach_name = models.CharField(max_length=250)
active = models.BooleanField(null=True,default=1)
class Player(models.Model):
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.DO_NOTHING
)
number = models.PositiveIntegerField()
age = models.PositiveIntegerField()
active = models.BooleanField(null=True,default=1)
Note that this Player model is important in the application as it's a major connection to most of the models - from registration to teams to stats to results to prizes. But this Player model doesn't actually contain the person's name as the model contains a user field which is the foreign key to a custom AUTH_USER_MODEL ('user') model which contains first/last name information. This allows the player to log in to the application and perform certain actions.
In addition to these base models, say that since a player can play on different teams in different tournaments, I also have a connecting ManyToMany model:
class PlayerToTeam(models.Model):
player = models.ForeignKey(
Player,
on_delete=models.DO_NOTHING
)
team = models.ForeignKey(
Team,
on_delete=models.DO_NOTHING
)
tournament = models.ForeignKey(
Tournament,
on_delete=models.DO_NOTHING
)
As an example of one of the challenges I'm encountering, let's say I'm trying to create a form that allows coaches to select their starting lineup. So I need my form to list the names of the Players on a particular Team at a particular Tournament.
Given the tournament and team IDs, I can easily pull back the necessary QuerySet to describe the initial records I'm interested in.
playersOnTeam = PlayerToTeam.objects.filter(tournament=[Tournament_id]).filter(team=[Team_id])
This returns the QuerySet of the IDs (but only the IDs) of the team, the tournament, and the players. However, the name data is two models away:
PlayerToTeam->[player_id]->Player->[user_id]->User->[first_name] [last_name]
Now, if I pull back only a single record, I could simply do
onlyPlayerOnTeam = PlayerToTeam.objects.filter(tournament=[Tournament_id]).filter(team=[Team_id]).filter(player=[Player_id]).get()
onlyPlayerOnTeam.player.user.first_name
So if I was only needing to display the names, I believe I could pass the QuerySet in the view return and loop through it in the template and display what I need. But I can't figure out if you can do something similar when I need the names to be displayed as part of a form.
To populate the form, I believe I could loop through the initial QuerySet and build a new datastructure:
playersOnTeam = PlayerToTeam.objects.filter(tournament=[Tournament_id]).filter(team=[Team_id])
allPlayersData= []
for nextPlayer in playersOnTeam:
playerDetails= {
"player_id": nextPlayer.player.id,
"first_name": nextPlayer.player.user.first_name,
"last_name": nextPlayer.player.user.last_name,
}
allPlayersData.append(playerDetails)
form = StartingLineupForm(allPlayersData)
However, I fear that would result in a separate database call for every player/user!
And while that may be tolerable for 6-10 players, for larger datasets, that seems less than ideal. Looping through performing a query for every user seems completely wrong.
Furthermore, what's frustrating is that this would be simple enough with a straight SQL query:
SELECT User.first_name, User.last_name
FROM PlayerToTeam
INNER JOIN Player ON PlayerToTeam.player_id = Player.id
INNER JOIN User ON Player.user_id = User.id
WHERE PlayerToTeam.tournament_id=[tourney_id] AND PlayerToTeam.team_id=[team_id]
But I'm trying to stick to the Django ORM best practices as much as I can and avoid just dropping to SQL queries when I can't immediately figure something out, and I'm all but certain that this isn't so complicated of a situation that I can't accomplish this without resorting to direct SQL queries.
I'm starting to look at select_related and prefetch_related, but I'm having trouble wrapping my head around how those work for relations more than a single table connection away. Like it seems like I could access the Player.age data using the prefetch, but I don't know how to get to User.first_name from that.
Any help would be appreciated.

I would suggest two approaches:
A) select related (one DB query):
objects = PlayerToTeam.objects.filter(
...
).select_related(
'player__user',
).only('player__user__name')
name = objects.first().user.name
B) annotate (one DB query):
objects = PlayerToTeam.objects.filter(
...
).annotate(
player_name=F('player__user__name'),
)
name = objects.first().player_name
To be sure you have only one object for specific player, team and tournament, I would suggest adding unique_together:
class PlayerToTeam(models.Model):
...
class Meta:
unique_together = ('player', 'team', 'tournament', )

Django still stuck (Many to Many)

the idea is that students can send a ping to their teacher if they do not understand something, but first I am trying to get the student, teacher and whatever class they are in relationship
so in my models after a few attempts I have come up with this
Models
from django.db import models
class Student(models.Model):
name = models.CharField(max_length=50)
class Teacher(models.Model):
name = models.CharField(max_length = 70)
class Lecture(models.Model):
name = models.CharField(max_length = 70)
members = models.ManyToManyField(
Student,
through = 'part_of_class',
through_fields = ('lecture', 'student'),
)
lecturers = models.ManyToManyField(
Teacher,
through = 'Teacher_of_class',
through_fields = ('lecture', 'teacher'),
)
class Teacher_of_class(models.Model):
lecture = models.ForeignKey(Lecture)
teacher = models.ForeignKey(Teacher)
class part_of_class(models.Model):
lecture = models.ForeignKey(Lecture)
student = models.ForeignKey(Student)
Question is adding a student field messes up, and Im not even sure this is the right way, it sounds right in my head but I am sure I am missing something, any clues?

I'm not entirely sure that you need explicit intermediary relationships (Teacher_of_class and part_of_class). Instead, you should try utilizing fields within the other models to do what you want. For example, will a lecture generally have more than one teacher, or just one? You could model either of those with a field in Lecture.
Edit: Apologies, I misread your code initially with regards to the fields. My suggestion is to not use an additional relationship class unless you know why you need it.

If you just want to have a lecture that can have multiple students and multiple lecturers, you do not need to have the intermediate models, Django will handle this for you if you don't specify any. I would only use the intermediate tables if you wanted to store something with that relationship.
Also when trying to add your student and teacher objects to the lecture model, be sure to save them.
Your save code should look something like this:
aStudent = Student(name="Taco")
aTeacher = Teacher(name="Burrito")
aStudent.save()
aTeacher.save()
aLecture = Lecture(name="Cooking With Python!")
aLecture.members.add(aStudent)
aLecture.lecturers.add(aTeacher)
aLecture.save()
Hope this helps!

How can I store history of ManyToManyField using django-simple-history.

How can I store history of ManyToManyField using django-simple-history. I used HistoricalRecords with attribute m2m_filds but it is throwing error: unexpected keyword argument 'm2m_fields'

I'm macro1 on GitHub, and I guess de facto maintainer of django-simple-history.
From your question it seems that you're just asking about general ManyToManyField support compared with other fields. The short answer is that we do not currently support it.
ManyToManyFields actually create an in-between model that represents the relationship between the two models you're working with.
If you want tracking on that relationship I would suggest making a 'through' model representing the relationship and passing that into the ManyToManyField constructor. You could then register that through model to have its history tracked. If you get errors like "unexpected keyword argument 'm2m_fields'" with that set up please open an issue in our tracker.

Even though django-simple-history does not allow to have history tables for many to many relations there is actually a way to achieve this.
What you can do is that you manually create the many to many table and instead of using djangos add and remove you simply create and delete the relations. If you look at it with an example we would have:
class Class(models.Model):
name = models.CharField(max_length=255)
surname = models.CharField(max_length=255)
history = HistoricalRecords()
class Student(models.Model):
name = models.CharField(max_length=255)
surname = models.CharField(max_length=255)
classes = models.ManyToManyField(Class)
history = HistoricalRecords()
you can manually create the many to many table with:
class Class(models.Model):
name = models.CharField(max_length=255)
surname = models.CharField(max_length=255)
history = HistoricalRecords()
class Student(models.Model):
name = models.CharField(max_length=255)
surname = models.CharField(max_length=255)
history = HistoricalRecords()
class StudentClasses(models.Model):
student = models.ForeignKey(Student)
class = models.ForeignKey(Class)
history = HistoricalRecords()
if you now use:
StudentClasses.objects.create(student=student, class=class) instead of student.classes.add(class) and delete() instead of student.classes.remove(class) you will have everything tracked in a history table and the same many to many table.

As the author of django-simple-history says this isn't possible to detect change in only specific fields because
As you already know simple-history doesn't look at the values being
saved at all. It blindly saves a new historical version on every save
or delete signal.
He also says it may be possible Field Tracker do this job.

What are the benefits of having two models instead of one?

I've a Django model
class Person(models.Model):
name = models.CharField(max_length=50)
team = models.ForeignKey(Team)
And a team model
class Team(models.Model):
name = models.CharField(max_length=50)
Then, I would like to add a 'coach' property which is a one to one relationship to person. If I am not wrong, I have two ways of doing it.
The first approach would be adding the field to Team:
class Team(models.Model):
name = models.CharField(max_length=50)
coach = models.OneToOneField(Person, related_name='master')
The second one would be creating a new model:
class TeamCoach(models.Model):
team = models.OneToOneField(Team)
coach = models.OneToOneField(Person)
Is this right ? is there a big difference for practical purposes ? which are the pro and cons of each approach ?

I will say NEITHER, as every Person has a Team and if every Team has a Coach, it's rather redundant circulation and somewhat unnecessary.
Better to add a field in Person called type directly is more clean and direct, something like:
class Person(models.Model):
# use _ if you care about i18n
TYPES = ('member', 'member',
'coach', 'coach',)
name = models.CharField(max_length=50)
team = models.ForeignKey(Team)
type = models.CharField(max_length=20, choices=TYPES)
Although I would seriously consider refactoring Person to be more generic and get Team to have a ManyToMany to Person... in that case, you can re-use Person in other areas, like Cheerleaders.
class Person(models.Model):
# use _ if you care about i18n
TYPES = ('member', 'member',
'coach', 'coach',)
name = models.CharField(max_length=50)
type = models.CharField(max_length=20, choices=TYPES)
class Team(models.Model):
name = models.CharField(max_length=50)
member = models.ManyToManyField(Person, related_name='master')
Make your models more generic and DRY, they should be easily manageable and not tightly coupled to certain fields (unless absolutely necessary), then the models are more future proof and will not fall under migration nightmare that easily.
Hope this helps.

I can't agree so easy with #Anzel, and since the name of the question is
What are the benefits of having two models instead of one?
I'll try to give my two cents. But before i start i want to place some quotes from the docs.
It doesn’t matter which model has the ManyToManyField, but you should
only put it in one of the models – not both.
Generally, ManyToManyField instances should go in the object that’s
going to be edited on a form. In the above example, toppings is in
Pizza (rather than Topping having a pizzas ManyToManyField ) because
it’s more natural to think about a pizza having toppings than a
topping being on multiple pizzas. The way it’s set up above, the Pizza
form would let users select the toppings.
Basically that's the first thing you should have in mind when creating a M2M relation (your TeamCoach model is that, but more on that in a second) which one is the object holding the relation. What would be more suitable for your problem - choosing a coach for a team when you create it, or choosing a team for a person when you create it? IF you ask me i would prefer the second variant and keep the teams inside of the Person class.
Now lets go to the next section of the docs
Extra fields on many-to-many relationships
When you’re only dealing with simple many-to-many relationships such
as mixing and matching pizzas and toppings, a standard ManyToManyField
is all you need. However, sometimes you may need to associate data
with the relationship between two models.
For example, consider the case of an application tracking the musical
groups which musicians belong to. There is a many-to-many relationship
between a person and the groups of which they are a member, so you
could use a ManyToManyField to represent this relationship. However,
there is a lot of detail about the membership that you might want to
collect, such as the date at which the person joined the group.
For these situations, Django allows you to specify the model that will
be used to govern the many-to-many relationship. You can then put
extra fields on the intermediate model. The intermediate model is
associated with the ManyToManyField using the through argument to
point to the model that will act as an intermediary.
That's actually the answer of your question, having an intermediate model give you the ability to store additional data about the collection. Consider the situation where a coach moves to another team next season, if you just update the M2M relation, you will loose the track of his past teams where he was coaching. Or you will never be able to answer the question who was the coach of that team at year XXX. So if you need more data, go with intermediate model. This is also were #Anzel going wrong, the type field is an additional data of that intermediate model, it's place must be inside it.
Now here is how i would probably create the relations:
class Person(models.Model):
name = models.CharField(max_length=50)
teams = models.ManyToManyField('Team', through='TeamRole')
class Team(models.Model):
name = models.CharField(max_length=50)
class TeamRole(models.Model):
COACH = 1
PLAYER = 2
CHEERLEADER = 3
ROLES = (
(COACH, 'Coach'),
(PLAYER, 'Player'),
(CHEERLEADER, 'Cheerleader'),
)
team = models.ForeignKey(Team)
person = models.ForeignKey(Person)
role = models.IntegerField(choices=ROLES)
date_joined = models.DateField()
date_left = models.DateField(blank=True, null=True, default=None)
How will I query this? Well, I can use the role to get what type of persons I'm looking for, and I can also use the date_left field to get the current persons participating in that team right now. Here are a few example methods:
class Person(models.Model):
#...
def get_current_team(self):
return self.teams.filter(teamrole__date_left__isnull=True).get()
class Team(models.Model):
#...
def _get_persons_by_role(self, role, only_active):
persons = self.person_set.filter(teamrole__role=role)
if only_active:
return persons.filter(teamrole__date_left__isnull=True)
return persons
def get_coaches(self, only_active=True):
return self._get_persons_by_role(TeamRole.COACH, only_active)
def get_players(self, only_active=True):
return self._get_persons_by_role(TeamRole.PLAYER, only_active)

django: need to design models/forms for a 'multiple level nested' structures

Assume some Company with Employees. There are Name and Contact information bound to each Employee. Each Contact contains Street and Phones fields.
What I want is a page which lists employees within a company. But everything must be listed as forms. Because I want to be able to modify the particular Employee information and the most important - I want to be able to add new Employees (clicking a button "Add new employee" must add a new empty "Employee form"). As well as it must allow to add a new phone number to the existing Employee's Contact information any time.
The data model looks like:
--Company
----Employee1
------Name
------Contact
--------Street
--------Phones
----------Phone1
----------Phone2
----Employee2
------Name
------Contact
--------Street
--------Phones
----------Phone1
----------Phone2
----------Phone3
...
Could someone please help to design Models and Forms for such a task? Your help is very much appreciated. Many thanks!
P.S. Forgot to mention that I want all the data "collected" in the Company object at the end of the day. I mean when I serialize c = Comapany.objects.all()[0] on the back end the entire employee information must be visible, like c.employees[0].contact.phones[0] must be the first employee's first phone number. Thanks.
P.P.S.
That is not the case that I'm just forwarding my project. This is just an hypothetical example I'd created to present the problem. I'm a django newbie and trying to understand how the framework gets things rolling.
I've spent lot of time on this. I've found several ways to go, but no one got me to the end. For instance, a wonderful blog about nested formsets http://yergler.net/blog/2013/09/03/nested-formsets-redux/ helped with forms and rendering. But, it solved only the half of the problem. The data like I mentioned above is not "being collected" into an object. At the end of the day I want to serialize a Company object and save it in yaml format using pyyaml (see my previous post django: want to have a form for dynamically changed sequence data).
Django is perfect with "static" models and forms, ModelForms are awesome. But what if your model needs to be changed dynamically? No standard way to go. Either no appropriate documentation nor I could find a one. Thus, I'd like to hear how experts imagine the solution for such a problem.

Try this:
from django.db import models
class _Contact(object):
pass
class Company(models.Model):
name = models.CharField(max_length=255)
created_at = models.DateTimeField(auto_now_add=True)
#property
def employees(self):
return self.employee_set.prefetch_related('phones').order_by('-created_at')
class Phone(models.Model):
number = models.CharField(max_length=255)
created_at = models.DateTimeField(auto_now_add=True)
class Employee(models.Model):
name = models.CharField(max_length=255)
street = models.CharField(max_length=255)
phones = models.ManyToManyField('Phone', through='EmployeePhone', blank=True)
created_at = models.DateTimeField(auto_now_add=True)
company = models.ForeignKey(Company)
#property
def contact(self):
_contact = _Contact()
_contact.street = self.street
_contact.phones = self.phones.order_by('-employeephone__created_at')
return _contact
class EmployeePhone(models.Model):
employee = models.ForeignKey(Employee)
phone = models.ForeignKey(Phone)
created_at = models.DateTimeField(auto_now_add=True)
However, you should just use employee.street and employee.phones. employee.contact is redundant.