Django newbie here!
I am coming from .NET background I am frustrated as to how to do the following simple thing:
My simplified models are as follows
class Circle(BaseClass):
name = models.CharField("Name", max_length=2048, blank=False, null=False)
active = models.BooleanField(default=False)
...
class CircleParticipant(BaseClass):
circle = models.ForeignKey(Circle, on_delete=models.CASCADE, null=True, blank=True)
user = models.ForeignKey(User, on_delete=models.SET_NULL, null=True, blank=True)
status = models.CharField("Status", max_length=256, blank=False, null=False)
...
class User(AbstractBaseUser, PermissionsMixin):
email = models.EmailField(verbose_name="Email", unique=True, max_length=255, validators=[email_validator])
first_name = models.CharField(verbose_name="First name", max_length=30, default="first")
last_name = models.CharField(verbose_name="Last name", max_length=30, default="last")
...
My goal is to get a single circle with participants that include the users as well. With the extra requirement to do all that in a single DB trip.
in SQL terms I want to accomplish this:
SELECT circle.name, circle.active, circle_participant.status, user.email. user.first_name. user.last_name
FROM circle
JOIN circle_participant on circle.id = circle_participant.id
JOIN user on user.id = circle_participant.id
WHERE circle.id = 43
I've tried the following:
Circle.objects.filter(id=43) \
.prefetch_related(Prefetch('circleparticipant_set', queryset=CircleParticipant.objects.prefetch_related('user')))
This is supposed to be working but when I check the query property on that statement it returns
SELECT "circle"."id", "circle"."created", "circle"."updated", "circle"."name", "circle"."active", FROM "circle" WHERE "circle"."id" = 43
(additional fields omitted for brevity.)
Am I missing something or is the query property incorrect?
More importantly how can I achieve fetching all that data with a single DB trip.
For reference here's how to do it in .NET Entity Framework
dbContext.Circle
.Filter(x => x.id == 43)
.Include(x => x.CircleParticipants) // This will exist in the entity/model
.ThenInclude(x => x.User)
.prefetch_related will use a second query to reduce the bandwidth, otherwise it will repeat data for the same Circle and CircleParticipants multiple times. Your CircleParticipant however acts as a junction table, so you can use:
Circle.objects.filter(id=43).prefetch_related(
Prefetch('circleparticipant_set', queryset=CircleParticipant.objects.select_related('user')
)
)
Am I missing something or is the query property incorrect?
There are two ways that Django gives you to solve the SELECT N+1 problem. The first is prefetch_related(), which creates two queries, and joins the result in memory. The second is select_related(), which creates a join, but has a few more restrictions. (You also haven't set related_name on any of your foriegn keys. IIRC that is required before using select_related().)
More importantly how can I achieve fetching all that data with a single DB trip.
I would suggest that you not worry too much about doing it all in one query. One of the downsides of doing this in one query as you suggest is that lots of the data that comes back will be redundant. For example, the circle.name column will be the same for every row in the table which is returned.
You should absolutely care about how many queries you do - but only to the extent that you avoid a SELECT N+1 problem. If you're doing one query for each model class involved, that's pretty good.
If you care strongly about SQL performance, I also recommend the tool Django Debug Toolbar, which can show you the number of queries, the exact SQL, and the time taken by each.
in SQL terms I want to accomplish this:
There are a few ways you could accomplish that.
Use many-to-many
Django has a field which can be used to create a many-to-many relationship. It's called ManyToManyField. It will implicitly create a many-to-many table to represent the relationship, and some helper methods to allow you to easily query for all circles a user is in, or all users that a circle has.
You're also attaching some metadata to each user/circle relationship. That means you'll need to define an explicit table using ManyToManyField.through.
There are examples in the docs here.
Use a related model query
If I specifically wanted a join, and not a subquery, I would query the users like this:
Users.objects.filter(circleparticipant_set__circle_id=43)
Use a subquery
This also creates only one query, but it uses a subquery instead.
Users.objects.filter(circleparticipant_set=CircleParticipant.objects.filter(circle_id=43))
Related
i am using Django along with DRF, and I am using many tables and complex relationships,
using soft delete (that is only marked deleted_at and not actually deleting anything) on my models from the begging didn't hurt much, but now we are making some new changes and deleted instances are growing.
so unique db constraints started giving errors, I know and implemented two possible solutions, one is unique_together in metaclass, with deleted at and every other unique value,
the other is:
class Meta:
constraints = [UniqueConstraint(fields=["name"], condition=Q(deleted_at != True))]
yet what I want is quite different, I want to avoid repeating all this and create a flag of uniqueness like this:
something_id = models.CharField(null=False, blank=False,
max_length=256, unique_undeleted=True)
notice the unique_undeleted parameter,
i know doing this directly requires changes in the library which is obviously bad,
but i have a parent model that i use in all others,
i thought of making a customized Constructor that will check every unique value, and make the unique on condition addition, or i could use this also
class MySpecialModel(parentModel):
somethingboolean= models.BooleanField(default=True, null=False)
something_id = models.CharField(null=False, blank=False,
max_length=256)
something_unique_too = models.CharField(null=False, blank=False,
max_length=256)
unique_undeleted_fields = ["something_id", "something_unique_too""]
and iterate in the parent model and create the UniqueConstraint for each field,
yet that does not feel right!
any guidance will be appreciated.
Using the Django ORM, how does one access data from related tables without effectively making a separate call for each record (or redundantly denormalizing data to make it more easily accessible)?
Say I have 3 Models:
class Tournament(models.Model):
name = models.CharField(max_length=250)
active = models.BooleanField(null=True,default=1)
class Team(models.Model):
name = models.CharField(max_length=250)
coach_name = models.CharField(max_length=250)
active = models.BooleanField(null=True,default=1)
class Player(models.Model):
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.DO_NOTHING
)
number = models.PositiveIntegerField()
age = models.PositiveIntegerField()
active = models.BooleanField(null=True,default=1)
Note that this Player model is important in the application as it's a major connection to most of the models - from registration to teams to stats to results to prizes. But this Player model doesn't actually contain the person's name as the model contains a user field which is the foreign key to a custom AUTH_USER_MODEL ('user') model which contains first/last name information. This allows the player to log in to the application and perform certain actions.
In addition to these base models, say that since a player can play on different teams in different tournaments, I also have a connecting ManyToMany model:
class PlayerToTeam(models.Model):
player = models.ForeignKey(
Player,
on_delete=models.DO_NOTHING
)
team = models.ForeignKey(
Team,
on_delete=models.DO_NOTHING
)
tournament = models.ForeignKey(
Tournament,
on_delete=models.DO_NOTHING
)
As an example of one of the challenges I'm encountering, let's say I'm trying to create a form that allows coaches to select their starting lineup. So I need my form to list the names of the Players on a particular Team at a particular Tournament.
Given the tournament and team IDs, I can easily pull back the necessary QuerySet to describe the initial records I'm interested in.
playersOnTeam = PlayerToTeam.objects.filter(tournament=[Tournament_id]).filter(team=[Team_id])
This returns the QuerySet of the IDs (but only the IDs) of the team, the tournament, and the players. However, the name data is two models away:
PlayerToTeam->[player_id]->Player->[user_id]->User->[first_name] [last_name]
Now, if I pull back only a single record, I could simply do
onlyPlayerOnTeam = PlayerToTeam.objects.filter(tournament=[Tournament_id]).filter(team=[Team_id]).filter(player=[Player_id]).get()
onlyPlayerOnTeam.player.user.first_name
So if I was only needing to display the names, I believe I could pass the QuerySet in the view return and loop through it in the template and display what I need. But I can't figure out if you can do something similar when I need the names to be displayed as part of a form.
To populate the form, I believe I could loop through the initial QuerySet and build a new datastructure:
playersOnTeam = PlayerToTeam.objects.filter(tournament=[Tournament_id]).filter(team=[Team_id])
allPlayersData= []
for nextPlayer in playersOnTeam:
playerDetails= {
"player_id": nextPlayer.player.id,
"first_name": nextPlayer.player.user.first_name,
"last_name": nextPlayer.player.user.last_name,
}
allPlayersData.append(playerDetails)
form = StartingLineupForm(allPlayersData)
However, I fear that would result in a separate database call for every player/user!
And while that may be tolerable for 6-10 players, for larger datasets, that seems less than ideal. Looping through performing a query for every user seems completely wrong.
Furthermore, what's frustrating is that this would be simple enough with a straight SQL query:
SELECT User.first_name, User.last_name
FROM PlayerToTeam
INNER JOIN Player ON PlayerToTeam.player_id = Player.id
INNER JOIN User ON Player.user_id = User.id
WHERE PlayerToTeam.tournament_id=[tourney_id] AND PlayerToTeam.team_id=[team_id]
But I'm trying to stick to the Django ORM best practices as much as I can and avoid just dropping to SQL queries when I can't immediately figure something out, and I'm all but certain that this isn't so complicated of a situation that I can't accomplish this without resorting to direct SQL queries.
I'm starting to look at select_related and prefetch_related, but I'm having trouble wrapping my head around how those work for relations more than a single table connection away. Like it seems like I could access the Player.age data using the prefetch, but I don't know how to get to User.first_name from that.
Any help would be appreciated.
I would suggest two approaches:
A) select related (one DB query):
objects = PlayerToTeam.objects.filter(
...
).select_related(
'player__user',
).only('player__user__name')
name = objects.first().user.name
B) annotate (one DB query):
objects = PlayerToTeam.objects.filter(
...
).annotate(
player_name=F('player__user__name'),
)
name = objects.first().player_name
To be sure you have only one object for specific player, team and tournament, I would suggest adding unique_together:
class PlayerToTeam(models.Model):
...
class Meta:
unique_together = ('player', 'team', 'tournament', )
So I've got the following models:
SuggestedArticle:
linked_article = models.ForeignKey('Article', blank=True, null=True, db_constraint=False)
... more fields which aren't relevant to this question ...
Article:
title=models.CharField(max_length=60)
draft_state=models.BooleanField(default=True)
... More fields which aren't relevant to this questions ...
In the article model there is a field, draft_state, which allows me to change the state of an article to published, draft, blocked (1, 2, 3).
I'm fetching all the SuggestedArticle's and am wanting to filter the Foreignkey article's to make sure there draft_state === 1. Here is what I have at the moment with some pseudo code inside to what I think the bit of code to achieve this may look like:
suggested_article = SuggestedArticle.objects\
.filter(
is_active=1,
Article.draft_state=1, # Something like this?! <~~~~~
state__in=[state, 'ALL'],
)\
Am I able to do this with django?
Django offers a powerful and intuitive way to “follow” relationships in lookups, taking care of the SQL JOINs for you automatically, behind the scenes. To span a relationship, just use the field name of related fields across models, separated by double underscores, until you get to the field you want:
suggested_article = SuggestedArticle.objects.filter(
is_active=1,
state__in=[state, 'ALL'],
linked_article__draft_state=True
)
I have two models:
class Account(models.Model):
...
class Transaction(models.Model):
....
account = models.ForeignKey(Account)
source_account = models.ForeignKey(Account, null=True)
I need to display the number of transactions for each of a user's accounts. Django's annotate seemed like the proper tool for this task. I did:
queryset = models.Account.objects.filter(user=self.request.user)
queryset.annotate(transactions_count=Count('transaction'))
This gives the correct number for transactions with account field set to the predicate account but leaves out transactions where source_account is set to the predicate account.
Using the Django shell I am able to do something like:
accounts_count = user_transactions.filter(Q(account=account)|Q(source_account=account)).count()
This gives the correct answer. Is there something I am doing wrong? Can someone point me in the correct direction. Any assistance is highly appreciated.
I would set related_name to your ForeignKey fields. Then it's a bit easier to work with them. So for example in your models let's set:
class Transaction(models.Model):
...
account = models.ForeignKey(Account, related_name='transactions')
source_account = models.ForeignKey(Account, null=True, related_name='source_transactions')
then can do something like:
queryset = models.Account.objects.filter(user=self.request.user).annotate(transactions_count=(Count('transactions')+Count('source_transactions'))
it would work without the naming too, it's just more readable and easier. The main point is adding the two Count as one field in annotate.
The best approach for these types of problems is to imagine them in raw SQL and then try to mimic it in Django ORM.
(in raw sql you would also simply just add two columns like SELECT (a.col + a.col2) AS count
The problem is that your transaction has to ForgeinKeys to Account. I would suggest trying something like this
class Transaction(models.Model):
....
account = models.ForeignKey(Account, related_name="transaction_account")
source_account = models.ForeignKey(Account, null=True, related_name="transaction_source_account")
Then in your query:
queryset.annotate(transactions_count=((Count('transaction_account') + Count('transaction_source_account'))
I am creating a web application to manage robotics teams for our area. In the application I have a django model that looks like this:
class TeamFormNote(models.Model):
team = models.ForeignKey(Team, blank=True, null=True)
member = models.ForeignKey(TeamMember, blank=True, null=True)
notes = models.TextField()
def __unicode__(self):
if self.team:
return "Team Form Record: " + unicode(self.team)
if self.member:
return "Member Form Record: " + unicode(self.member)
Essentially, I want it to have a relationship with team or a relationship with member, but not both. Is there a way to enforce this?
I can only see two viable solutions. First is actually the same as #mariodev suggested in the comment which is to use Genetic foreign key. That will look something like:
# make sure to change the app name
ALLOWED_RELATIONSHIPS = models.Q(app_label = 'app_name', model = 'team') | models.Q(app_label = 'app_name', model = 'teammember')
class TeamFormNote(models.Model):
content_type = models.ForeignKey(ContentType, limit_choices_to=ALLOWED_RELATIONSHIPS)
relation_id = models.PositiveIntegerField()
relation = generic.GenericForeignKey('content_type', 'relation_id')
What that does is it sets up a generic foreign key which will allow you to link to any other model within your project. Since it can link to any other model, to restrict it to only the models you need, I use the limit_choices_to parameter of the ForeignKey. This will solve your problem since there is only one generic foreign key hence there is no way multiple relationships will be created. The disadvantage is that you cannot easily apply joins to generic foreign keys so you will not be able to do things like:
Team.objects.filter(teamformnote_set__notes__contains='foo')
The second approach is to leave the model as it and manually go into the database backend and add a db constaint. For example in postgres:
ALTER TABLE foo ADD CONSTRAINT bar CHECK ...;
This will work however it will not be transparent to your code.
This sounds like a malformed object model under the hood...
How about an abstract class which defines all common elements and two dreived classes, one for team and one for member?
If you are running into trouble with this because you want to have both referenced in the same table, you can use Generic Relations.