We Keep Coding

What is the "instance" being passed to the to_representation function of my ListSerializer?

The goal of this project is to create an API that refreshes hourly with the most up to date betting odds for a list of games that I'll be scraping hourly from the internet. The goal structure for the JSON returned will be each game as the parent object and the nested children will be the top 1 record for each of linesmakers being scraped by updated date. My understanding is that the best way to accomplish this is to modify the to_representation function within the ListSerializer to return the appropriate queryset.
Because I need the game_id of the parent element to grab the children of the appropriate game, I've attempted to pull the game_id out of the data that gets passed. The issue is that this line looks to be populated correctly when I see what it contains through an exception, but when I let the full code run, I get a list index is out of range exception.
For ex.
class OddsMakerListSerializer(serializers.ListSerializer):
def to_representation(self, data):
game = data.all()[0].game_id
#if I put this here it evaluates to 1 which should run the raw sql below correctly
raise Exception(game)
data = OddsMaker.objects.filter(odds_id__in = RawSQL(''' SELECT o.odds_id
FROM gamesbackend_oddsmaker o
INNER JOIN (
SELECT game_id
, oddsmaker
, max(updated_datetime) as last_updated
FROM gamesbackend_oddsmaker
WHERE game_id = %s
GROUP BY game_id
, oddsmaker
) l on o.game_id = l.game_id
and o.oddsmaker = l.oddsmaker
and o.updated_datetime = l.last_updated
''', [game]))
#if I put this here the data appears to be populated correctly and contain the right data
raise Exception(data)
data = [game for game in data]
return data
Now, if I remove these raise Exceptions, I get the list index is out of range. My initial thought was that there's something else that depends on "data" being returned as a list, so I created the list comprehension snippet, but that doesn't resolve the issue.
So, my question is 1) Is there an easier way to accomplish what I'm going for? I'm not using a postgres backend so distinct on isn't available to me. and 2) If not, its not clear to me what instance is that's being passed in or what is expected to be returned. I've consulted the documentation and it looks as though it expects a dictionary and that might be part of the issue, but again the error message references a list. https://www.django-rest-framework.org/api-guide/serializers/#overriding-serialization-and-deserialization-behavior
I appreciate any help in understanding what is going on here in advance.
Edit:
The rest of the serializers:
class OddsMakerSerializer(serializers.ModelSerializer):
class Meta:
list_serializer_class = OddsMakerListSerializer
model = OddsMaker
fields = ('odds_id','game_id','oddsmaker','home_ml',
'away_ml','home_spread','home_spread_odds',
'away_spread_odds','total','total_over_odds',
'total_under_odds','updated_datetime')
class GameSerializer(serializers.ModelSerializer):
oddsmaker_set = OddsMakerSerializer(many=True, read_only=True)
class Meta:
model = Game
fields = ('game_id','date','sport', 'home_team',
'away_team','home_score', 'away_score',
'home_win','away_win', 'game_completed',
'oddsmaker_set')
models.py:
class Game(models.Model):
game_id = models.AutoField(primary_key=True)
date = models.DateTimeField(null=True)
sport=models.CharField(max_length=256, null=True)
home_team = models.CharField(max_length=256, null=True)
away_team = models.CharField(max_length=256, null=True)
home_score = models.IntegerField(default=0, null=True)
away_score = models.IntegerField(default=0, null=True)
home_win = models.BooleanField(default=0, null=True)
away_win = models.BooleanField(default=0, null=True)
game_completed = models.BooleanField(default=0, null=True)
class OddsMaker(models.Model):
odds_id = models.AutoField(primary_key=True)
game = models.ForeignKey('Game', on_delete = models.CASCADE)
oddsmaker = models.CharField(max_length=256)
home_ml = models.IntegerField(default=999999)
away_ml = models.IntegerField(default=999999)
home_spread = models.FloatField(default=999)
home_spread_odds = models.IntegerField(default=9999)
away_spread_odds = models.IntegerField(default=9999)
total = models.FloatField(default=999)
total_over_odds = models.IntegerField(default=999)
total_under_odds = models.IntegerField(default=999)
updated_datetime = models.DateTimeField(auto_now=True)
views.py:
class GameView(viewsets.ModelViewSet):
queryset = Game.objects.all()
serializer_class = GameSerializer
Thanks

To answer the question in the title:
The instance being passed to the Serializer.to_representation() is the instance you pass when initializing the serializer
queryset = MyModel.objects.all()
Serializer(queryset, many=True)
instance = MyModel.objects.all().first()
Serializer(data)
Usually you don't have to inherit from ListSerializer per se. You can inherit from BaseSerializer and whenever you pass many=True during initialization, it will automatically 'becomeaListSerializer`. You can see this in action here
To answer your problem
from django.db.models import Max
class OddsMakerListSerializer(serializers.ListSerializer):
def to_representation(self, data): # data passed is a queryset of oddsmaker
# Do your filtering here
latest_date = data.aggregate(
latest_date=Max('updated_datetime')
).get('latest_date').date()
latest_records = data.filter(
updated_date_time__year=latest_date.year,
updated_date_time__month=latest_date.month,
updated_date_time__day=latest_date.day
)
return super().to_representation(latest_records)

Django ForeignKey accept two models

I'm working on this big project with Django and I have to update the database. I have to add another table which will replace another later.
So I want to add in a model the possibility to have a field where I can have either the old model OR the new one.
Here is the code of the old model:
class Harvests(models.Model):
ident_culture = models.IntegerField(primary_key=True)
intitule_culture = models.CharField(max_length=50, blank=True)
nom_fertiweb = models.CharField(max_length=50, blank=True, null = True)
affichage_quintaux_tonne = models.CharField(max_length=1,
choices=RENDEMENT_CHOICES, default = 'T')
type_culture = models.ForeignKey("TypeCulture", null=True)
slug = models.SlugField(null=True, blank=True)
image = models.ImageField(upload_to = 'images_doc_culture/',
null=True, blank = True)
affichage = models.BooleanField(default = True)
class Meta:
verbose_name = "Liste - Culture"
verbose_name_plural = "Liste - Cultures"
ordering = ['intitule_culture']
def __str__(self):
return self.intitule_culture
def label(self):
return self.intitule_culture or ''
#classmethod
def get_choices(cls):
choices = [('', corp.EMPTY_CHOICE_LBL)]
c_category_lbl, c_category = '', []
for item in cls.objects.all():
choices.append((item.pk, item.intitule_culture))
return choices
And there is the code od the new one I created:
class Crops(models.Model):
intitule_culture = models.CharField(max_length=75, blank=True)
affichage_quintaux_tonne = models.CharField(max_length=2,
choices=RENDEMENT_CHOICES, default = 'T')
type_culture = models.ForeignKey("TypeCulture", null=True)
ident_culture = models.IntegerField(primary_key=True)
affichage = models.BooleanField(default = True)
id_marle = models.IntegerField(null=True)
class Meta:
verbose_name = "Liste - Culture 2019"
verbose_name_plural = "Liste - Cultures 2019"
ordering = ['intitule_culture']
def __str__(self):
return self.intitule_culture
def label(self):
return self.intitule_culture or ''
#classmethod
def get_choices(cls):
choices = [('', corp.EMPTY_CHOICE_LBL)]
c_category_lbl, c_category = '', []
for item in cls.objects.all():
choices.append((item.pk, item.intitule_culture))
return choices
I want to accept both models in the field culture in this model:
class CompanyHarvest(models.Model):
company = models.ForeignKey('corp.Company', verbose_name='Exploitation',
related_name ='cultures')
culture = models.ForeignKey(Harvests, verbose_name ='Culture')
precision = models.CharField(max_length=255, blank=True)
saison_culture = models.CharField(max_length=1, choices=SAISON_CHOICES,
default = 'P')
class Meta:
verbose_name = "Expl. - Culture"
verbose_name_plural = "Expl. - Cultures"
unique_together = ('company', 'culture', 'precision', 'saison_culture')
def __str__(self):
return str(self.culture) + ' ' + self.precision + \
' ' + str(self.get_saison_culture_display() )
#property
def slug(self):
return "_".join([slugify(str(self.culture or '')),
slugify(str(self.precision or ''))]
)
I'm new to Django, can anyone help me with this please ? (^-^)

This is not possible - at least not this way. And this is not a Django limitation but a SQL one, a foreign key cannot reference either one table or another.
A possible and simple obvious solution here would be to have two foreign keys in CompanyHarvest - one for each of the old and new model -, each with blank=True et default=None, but it can quickly make a mess of all the client code (all code using CompanyHarvest).
Much better solutions would be to either only keep the existing model (adding any new field/feature to it and eventually hiding obsolete ones) or migrate all old model records to the new model (this can be combined with the naive "two foreign keys" solution so you can keep the old table and records as archives if necessary).
Also - totally unrelated but -, this:
#classmethod
def get_choices(cls):
choices = [('', corp.EMPTY_CHOICE_LBL)]
c_category_lbl, c_category = '', []
for item in cls.objects.all():
choices.append((item.pk, item.intitule_culture))
return choices
1/ should be defined on the manager (cf https://docs.djangoproject.com/en/2.1/topics/db/managers/#adding-extra-manager-methods)
2/ should be written using .values() queryset (which will save on both the db query and building full-blown instances for no good reason):
for item in cls.objects.values("pk", "intitule_culture"):
choices.append(item)
3/ and could very possibly (i'd have to see how it's used) replaced by a ModelChoiceField in the calling code.
Oh and yes: if you allow blanks for text fields, you very probably want to force the empty string as default so you don't two possible (and incompatible) cases (sql NULL and the empty string) when no value is given.

Reference multiple foreign keys in Django Model

I'm making a program that helps log missions in a game. In each of these missions I would like to be able to select a number of astronauts that will go along with it out of the astronauts table. This is fine when I only need one, but how could I approach multiple foreign keys in a field?
I currently use a 'binary' string that specifies which astronauts are to be associated with the mission (1 refers to Jeb, but not Bill, Bob, or Val and 0001 means only Val), with the first digit specifying the astronaut with id 1 and so forth. This works, but it feels quite clunky.
Here's the model.py for the two tables in question.
class astronauts(models.Model):
name = models.CharField(max_length=200)
adddate = models.IntegerField(default=0)
experience = models.IntegerField(default=0)
career = models.CharField(max_length=9, blank=True, null=True)
alive = models.BooleanField(default=True)
def __str__(self):
return self.name
class Meta:
verbose_name_plural = "Kerbals"
class missions(models.Model):
# mission details
programid = models.ForeignKey(programs, on_delete=models.SET("Unknown"))
missionid = models.IntegerField(default=0)
status = models.ForeignKey(
missionstatuses, on_delete=models.SET("Unknown"))
plan = models.CharField(max_length=1000)
# launch
launchdate = models.IntegerField(default=0)
crewmembers = models.IntegerField(default=0)
# recovery
summary = models.CharField(max_length=1000, blank=True)
recdate = models.IntegerField(default=0)
def __str__(self):
return str(self.programid) + '-' + str(self.missionid)
class Meta:
verbose_name_plural = "Missions"
I saw a post about an 'intermediate linking table' to store the crew list but that also isn't ideal.
Thanks!

This is the use case for Django's ManyToManyField. Change the appropriate field on the missions:
class missions(models.Model):
crewmembers = models.ManyToManyField('astronauts')
You can access this from the Astronaut model side like so:
jeb = astronaut.objects.get(name='Jebediah Kerman')
crewed_missions = jeb.missions_set.all()
Or from the mission side like so:
mission = missions.objects.order_by('?')[0]
crew = mission.crewmembers.all()
This creates another table in the database, in case that is somehow a problem for you.

django - inner join queryset not working

The SQL I want to accomplish is this -
SELECT jobmst_id, jobmst_name, jobdtl_cmd, jobdtl_params FROM jobmst
INNER JOIN jobdtl ON jobmst.jobdtl_id = jobdtl.jobdtl_id
WHERE jobmst_id = 3296
I've only had success once with an inner join in django off of a annote and order_by but I can't seem to get it to work doing either prefetch_related() or select_related()
My models are as so -
class Jobdtl(models.Model):
jobdtl_id = models.IntegerField(primary_key=True)
jobdtl_cmd = models.TextField(blank=True)
jobdtl_fromdt = models.DateTimeField(blank=True, null=True)
jobdtl_untildt = models.DateTimeField(blank=True, null=True)
jobdtl_fromtm = models.DateTimeField(blank=True, null=True)
jobdtl_untiltm = models.DateTimeField(blank=True, null=True)
jobdtl_priority = models.SmallIntegerField(blank=True, null=True)
jobdtl_params = models.TextField(blank=True) # This field type is a guess.
class Meta:
managed = False
db_table = 'jobdtl'
class Jobmst(MPTTModel):
jobmst_id = models.IntegerField(primary_key=True)
jobmst_type = models.SmallIntegerField()
jobmst_prntid = TreeForeignKey('self', null=True, blank=True, related_name='children', db_column='jobmst_prntid')
jobmst_name = models.TextField(db_column='jobmst_name', blank=True)
# jobmst_owner = models.IntegerField(blank=True, null=True)
jobmst_owner = models.ForeignKey('Owner', db_column='jobmst_owner', related_name = 'Jobmst_Jobmst_owner', blank=True, null=True)
jobmst_crttm = models.DateTimeField()
jobdtl_id = models.ForeignKey('Jobdtl', db_column='jobdtl_id', blank=True, null=True)
jobmst_prntname = models.TextField(blank=True)
class MPTTMeta:
order_insertion_by = ['jobmst_id']
class Meta:
managed = True
db_table = 'jobmst'
I have a really simple view like so -
# Test Query with Join
def test_queryjoin(request):
queryset = Jobmst.objects.filter(jobmst_id=3296).order_by('jobdtl_id')
queryresults = serializers.serialize("python", queryset, fields=('jobmst_prntid', 'jobmst_id', 'jobmst_prntname', 'jobmst_name', 'jobmst_owner', 'jobdtl_cmd', 'jobdtl_params'))
t = get_template('test_queryjoin.html')
html = t.render(Context({'query_output': queryresults}))
return HttpResponse(html)
I've tried doing a bunch of things -
queryset = Jobmst.objects.all().prefetch_related()
queryset = Jobmst.objects.all().select_related()
queryset = jobmst.objects.filter(jobmst_id=3296).order_by('jobdtl_id')
a few others as well I forget.
Each time the json I'm getting is only from the jobmst table with no mention of the jobdtl results which I want. If I go the other way and do Jobdtl.objects.xxxxxxxxx same thing it's not giving me the results from the other model.
To recap I want to display fields from both tables where a certain clause is met.
What gives?

Seems that I was constantly looking in the wrong place. Coming from SQL I kept thinking in terms of inner joining tables which is not how this works. I'm joining the results from models.
Hence, rethinking my search I came across itertools and the chain function.
I now have 2 queries under a def in my views.py
from itertools import chain
jobmstquery = Jobmst.objects.filter(jobmst_id=3296)
jobdtlquery = Jobdtl.objects.filter(jobdtl_id=3296)
queryset = chain(jobmstquery, jobdtlquery)
queryresults = serializers.serialize("python", queryset)
That shows me the results from each table "joined" like I would want in SQL. Now I can focus on filtering down the results to give me what I want.
Remember folks, the information you need is almost always there, it's just a matter of knowing how to look for it :)

What you are looking for might be this
queryset = Jobmst.objects.filter(id=3296).values_list(
'id', 'name', 'jobmst_owner__cmd', 'jobmst_owner__params')
You would get your results with only one query and you should be able to use sort with this.
P.S. Coming from SQL you might find some great insights playing with queryset.query (the SQL generated by django) in a django shell.

Django query with .order_by('title') doesn't give alphabetical order, sorts by date changed

With the following model in my Django app:
class Story(models.Model):
#1:1 Fields
title = models.CharField(max_length=100, unique=True)
tagline = models.TextField(blank=True, null=True)
contents = models.TextField()
updated = models.DateTimeField(auto_now_add=True)
slug = models.SlugField(unique=True)
full_length_book = models.BooleanField(default=False)
publish_now = models.BooleanField(default=False)
date_added = models.DateTimeField('date added to archive')
def __unicode__(self):
return self.title
class Meta:
verbose_name_plural = "Stories"
ordering = ['title']
I have several views that query the data from this model and use .order_by('title'), for example:
newlist = Story.objects.filter(tag=tag, publish_now=True).order_by('title')
The Problem: in the query results, as well as when I use the 'sort column' arrows in the django admin for the story group, it lists stories in order of last changed, and alphabetized within that ordering. I need the stories ordered firstly and solely by title, alphabetically. Why does it put recently changed entries at the end of the list?
I may be misunderstanding something about SQLite- even when I go into the database manager and call SELECT * FROM table_name ORDER BY LOWER(title) , the order is still by update first and then alphabetically within that.
edit The SQL Django is calling is as follows:
'SELECT "app_story"."id", "app_story"."title", "app_story"."tagline", "app_story"."contents", "app_story"."updated", "app_story"."slug", "app_story"."full_length_book", "app_story"."publish_now", "app_story","date_added" FROM "app_story" ORDER BY "app_story"."title" ASC'
This looks like it should order by title, so I still don't know why the order is coming up with newly edited items on the end of the list.