Django __unicode__ and FK is very slow - python

if I write something like
class Chip(models.Model):
name = models.CharField(max_length=16)
shortname = models.CharField(primary_key=True, unique=True, max_length = 16)
def __unicode__(self):
return self.shortname
class ChipStepping(models.Model):
stepping = models.CharField (max_length=16)
ChipShortname = models.ForeignKey('Chip', db_column="ChipShortname")
def __unicode__(self):
return "%s:%s" % (self.ChipShortname, self.stepping)
class ComponentType(models.Model):
name = models.CharField (max_length=32)
ChipStepping = models.ForeignKey('ChipStepping', db_column="ChipStepping")
def __unicode__(self):
return "%s(%s)" % (self.name, self.ChipStepping);
class ComponentVendor(models.Model):
name = models.CharField (unique=True, max_length=16)
products = models.ManyToManyField('ComponentType', through='ComponentVendorProduct', related_name='vendors')
def __unicode__(self):
return "%s" % (self.name)
class ComponentVendorProduct(models.Model):
ComponentVendor = models.ForeignKey('ComponentVendor', db_column="ComponentVendor")
ComponentType = models.ForeignKey('ComponentType' , db_column="ComponentType")
And try to create an admin page for ComponentVendor
class ProductInline(admin.TabularInline):
model = ComponentVendor.products.through
extra = 0
class ComponentVendorAdmin(admin.ModelAdmin):
inlines = [ProductInline]
list_filter = ['products__name']
exclude = ['products']
admin.site.register(ComponentVendor, ComponentVendorAdmin)
The resulting page can take upwards of 30 sec. to load
From some debugging I've done, I found that it repeatedly makes redundant singular queries for ChipStepping and then Chip, with the same argument in the where clause instead of intelligently building a query that can lookup all the data.
This problem is reduced if I remove the foreign key references from the unicode functions of ChipStepping and ComponentType
If there are enough entries in ComponentVendorProducts for a vendor I click on in the admin page, the page can take several minutes!
Is there a way I can reduce the number of database hits on the admin page?

Your problem comes from the fact that Django is doing a DB call everytime you call __unicode__ on a ComponentType instance.
You have two solutions to your issue:
You override your ProductInline's queryset method to include select_related('ChipStepping') (Django 1.3 and superior).
Alternatively, if you want to fix the issue elsewhere too, you might want to change your ComponentType's default manager (objects) get_query_set method to have it include the select_related call.

you might also want to checkout the suggestion given here:
http://blog.ionelmc.ro/2012/01/19/tweaks-for-making-django-admin-faster/
It seems that the choices are evaluated for every row, so using formfield_for_dbfield you can cache the choices as suggested in the link. This saves going to the db for rendering every single dropdown/select box for foreign_keys

Related

Django 2.2 cannot serialize default values once migration has been done

I have a model which is refered to as a foreignkey with on_delete set to SET_DEFAULT. Because of that, I need to provide this model with a default item. I created a static method which does just that.
class ScheduleChoice(models.Model):
"""
This model allows users to define crontab schedules.
"""
label = models.CharField(max_length=256, verbose_name="Label", unique=True)
job_schedule = models.CharField(
max_length=256,
default="0 0 * * *", verbose_name="Crontab"
)
#staticmethod
def get_default_schedule_choice():
"""
Because some models rely on ScheduleChoice and need a default value,
we need to give them a default ScheduleChoice.
"""
try:
choice = ScheduleChoice.objects.get_or_create(
label="Every minute",
job_schedule="* * * * *"
)
except ProgrammingError:
choice = None
return choice
#classmethod
def get_choice_count(cls):
"""
Returns how many schedule choices have been defined.
"""
return len(cls.objects.all())
class Meta:
verbose_name = "schedule choice"
verbose_name_plural = "schedule choices"
def __str__(self):
return self.label
class MyOtherModel(models.Model):
"""
Model using ScheduleChoices.
"""
job_schedule = models.ForeignKey(
"ScheduleChoice",
on_delete=models.SET_DEFAULT,
default=ScheduleChoice.get_default_schedule_choice(),
verbose_name="Schedule"
)
activated = models.BooleanField(default=False, verbose_name="activated")
I am able to run makemigrations and migrate without issue.
My problem starts when I modifiy my models and try to use makemigrations again in order to update the migrations files. I get the error :
ValueError: Cannot serialize: <ScheduleChoice: Every minute>
There are some values Django cannot serialize into migration files.
For more, see https://docs.djangoproject.com/en/2.2/topics/migrations/#migration-serializing
I tried to apply this answer, but it didn't help. Why does Django need to serialize my default value ? Why does it only do so after the first migration has successfully ended ?
I can always use reset_db and do my migration, but it is not acceptable in my production environment.
How can I fix this ?
Django needs to serialize your models to make migration files for them. Hence it also needs to serialize most of the attributes you set on the model fields (default included). Currently you define a method and directly call that instead of providing the method as the default, also your method returns a tuple with ScheduleChoice and a boolean.
Django can serialize booleans but not the model instance (for the migration) hence you get an error, not to mention the tuple would have caused an error anyway. You should not call the method and instead just pass the method as the default, also instead of returning the instance return only the pk, also ideally this method should simply be a function:
class ScheduleChoice(models.Model):
"""
This model allows users to define crontab schedules.
"""
label = models.CharField(max_length=256, verbose_name="Label", unique=True)
job_schedule = models.CharField(
max_length=256,
default="0 0 * * *", verbose_name="Crontab"
)
#classmethod
def get_choice_count(cls):
"""
Returns how many schedule choices have been defined.
"""
return len(cls.objects.all())
class Meta:
verbose_name = "schedule choice"
verbose_name_plural = "schedule choices"
def __str__(self):
return self.label
def get_default_schedule_choice():
choice, _created = ScheduleChoice.objects.get_or_create(
label="Every minute",
job_schedule="* * * * *"
)
# Why would programming error occur?
return choice.pk # Return the pk
class MyOtherModel(models.Model):
"""
Model using ScheduleChoices.
"""
job_schedule = models.ForeignKey(
"ScheduleChoice",
on_delete=models.SET_DEFAULT,
default=get_default_schedule_choice, # Don't call the function, only pass it
verbose_name="Schedule"
)
activated = models.BooleanField(default=False, verbose_name="activated")

How to change field in ModelForm generated html form?

I'm making one of my first django apps with sqlite database. I have some models like for example:
class Connection(models.Model):
routeID = models.ForeignKey(Route, on_delete=models.CASCADE)
activityStatus = models.BooleanField()
car = models.ForeignKey(Car, on_delete=models.CASCADE)
class Route(models.Model):
name = models.CharField(max_length=20)
and forms
class RouteForm(ModelForm):
class Meta:
model = Route
fields = ['name']
class ConnectionForm(ModelForm):
class Meta:
model = Connection
fields = ['routeID', 'activityStatus', 'car']
And in my website, in the url for adding new Connection, I have cascade list containing RouteIDs. And I'd like it to contain RouteName, not ID, so it would be easier to choose. How should I change my ConnectionForm, so I could still use foreign key to Route table, but see RouteName instead of RouteID?
For now it's looking like this, but I'd love to have list of RouteNames, while still adding to Connection table good foreign key, RouteID
Update the Route Model's __str__ method:
class Route(models.Model):
name = models.CharField(max_length=20)
def __str__(self):
return self.name
Because the __str__() method is called whenever you call str() on an object. Django uses str(obj) in a number of places like in Modelform. By default it returns id or pk that is why you were seeing ids in model form. So by overriding it with name, you will see the names appear in choice field. Please see the documentation for more details on this.

DJANGO - Queryset and model design

So I've been stuck with a design problem for the last couple of days and have sunk countless hours into it, to no avail.
My problem is that I wish return all the active Articles. I have made a method within the model, however am unable to use .filter(is_active=True)which would be the worlds best solution.
So now I have made the method into one long filter in the ArticleManager, the problem being that I cannot seem to figure out a way to count the current clicks in a way that can be useful to me. (The current_clicks method in the Article model is what I am aiming for).
Models.py
class ArticleManager(models.Manager):
def get_queryset(self):
return super(ArticleManager, self).get_queryset().filter(article_finish_date=None).filter(article_publish_date__lte=timezone.now())
#this is where i need something to the effect of .filter(article_max_clicks__gt=click_set.count())
class Article(models.Model):
article_name_text = models.CharField(max_length=200)
article_max_clicks = models.IntegerField(default=0)
article_creation_date = models.DateTimeField('date created')
article_publish_date = models.DateTimeField('date published', null=True, blank=True)
article_finish_date = models.DateTimeField('date finished', null=True, blank=True)
def __str__(self):
return self.article_name_text
def is_active(self):
if self.article_finish_date==None:
if self.article_publish_date <= timezone.now():
return self.current_clicks() < self.article_max_clicks
else:
return False
else:
return False
def current_clicks(self):
return self.click_set.count()
is_active.boolean = True
actives = ArticleManager()
class Click(models.Model):
click_article = models.ForeignKey(Article, on_delete=models.CASCADE)
click_user = models.ForeignKey(User, on_delete=models.CASCADE)
click_date = models.DateTimeField('date clicked')
def __str__(self):
return str(self.id) + " " + str(self.click_date)
This is how the clicks are created in views.py if this helps
article.click_set.create(click_article=article, click_user=user, click_date=timezone.now())
If anyone has any sort of idea of how abouts I should do this it would be greatly appreciated!
Many thanks in advance, just let me know if you need anymore information!
Django's annotate functionality is great for adding properties at the time of querying. From the docs -
Per-object summaries can be generated using the annotate() clause. When an annotate() clause is specified, each object in the QuerySet will be annotated with the specified values.
In order to keep your querying performance-minded, you can use this in your manager and not make the (possibly very slow) call of related objects for each of your Articles. Once you have an annotated property, you can use it in your query. Since Django only executes your query when objects are called, you can use this annotation instead of counting the click_set, which would call a separate query per related item. The current_clicks method may still be useful to you, but if calling it for multiple articles your queries will add up quickly and cause a big performance hit.
Please note - I added a related_name of clicks keyword arg to your click_article field in order to use it in place of 'click_set'.
In addition, you'll see the use of Q objects in the query below. What this allows us to do is chain together multiple filters together. These can be nested while using AND (,) / OR(|) operands. So, a reading of the Q objects below would be:
Find all articles where article publish date is before now AND (article has no finish date OR article finish date is after now)
from django.db.models.query import Q,Count
class ArticleManager(models.Manager):
def get_queryset(self):
return super(ArticleManager, self).get_queryset().filter(
Q(article_publish_date__lte=timezone.now()),
( Q(article_finish_date__isnull=True)|
Q(article_finish_date__gte=timezone.now())
).annotate(
click_count=Count('clicks')
).filter(
article_max_clicks__gt=click_count
)
class Article(models.Model):
actives = ArticleManager()
def current_clicks(self):
return self.clicks.count()
# Now you can call Article.actives.all() to get all active articles
class Click(models.Model):
click_article = models.ForeignKey(Article, on_delete=models.CASCADE, related_name='clicks') # added a related_name for more explicit calling of prefetch_related

Django QuerySet with Models

I'm new to Django and trying to understand how to use querysets with models.
Model
class Channel(models.Model):
name = models.CharField(max_length=200)
accountid = models.CharField(max_length=34)
def get_channel_list(self):
return self.get_queryset().name()
What I want to do is return the entire name column as an array if account id matches. I'd like to use a function in the models.py but I haven't found an online sample that caters to what I'm looking for.
The above isn't returning any data even without a filter.
Any point in the right direction would be amazing.
Use objects.filter and classmethod:
class Channel(models.Model):
name = models.CharField(max_length=200)
accountid = models.CharField(max_length=34)
#classmethod
def get_channel_list(cls, acc):
return cls.objects.filter(accountid=acc).values_list('name', flat=True)
There is another technique to do such things in django - define custom manager to model. (for example, you have several Channel models inherited from one base proxy model and you want to put same get_channel_list functions to some models - custom Manager is the way to go):
class ChannelManager(models.Manager):
def get_channel_list(self, acc):
return self.filter(accountid=acc).values_list('name', flat=True)
class Channel(models.Model):
name = models.CharField(max_length=200)
accountid = models.CharField(max_length=34)
objects = ChannelManager()
You have failed to understand the difference between managers and models. It's the manager that it's responsible for creating queries, and which has the get_queryset method. From a model, you need to access the manager, which is usually named objects. Note, you cannot do that from an instance, so this needs to be a classmethod.
#classmethod
def get_channel_list(cls, accountid):
return cls.objects.filter(accountid=accountid).values_list('name')

Django model manager didn't work with related object when I do aggregated query

I'm having trouble doing an aggregation query on a many-to-many related field.
Here are my models:
class SortedTagManager(models.Manager):
use_for_related_fields = True
def get_query_set(self):
orig_query_set = super(SortedTagManager, self).get_query_set()
# FIXME `used` is wrongly counted
return orig_query_set.distinct().annotate(
used=models.Count('users')).order_by('-used')
class Tag(models.Model):
content = models.CharField(max_length=32, unique=True)
creator = models.ForeignKey(User, related_name='tags_i_created')
users = models.ManyToManyField(User, through='TaggedNote',
related_name='tags_i_used')
objects_sorted_by_used = SortedTagManager()
class TaggedNote(models.Model):
"""Association table of both (Tag , Note) and (Tag, User)"""
note = models.ForeignKey(Note) # Note is what's tagged in my app
tag = models.ForeignKey(Tag)
tagged_by = models.ForeignKey(User)
class Meta:
unique_together = (('note', 'tag'),)
However, the value of the aggregated field used is only correct when the model is queried directly:
for t in Tag.objects.all(): print t.used # this works correctly
for t in user.tags_i_used.all(): print t.used #prints n^2 when it should give n
Would you please tell me what's wrong with it? Thanks in advance.
I have figured out what's wrong and how to fix it now :)
As stated in the Django doc:
Django interprets the first Manager defined in a class as the "default" Manager, and several parts of Django will use that Manager exclusively for that model.
In my case, I should make sure that SortedTagManager is the first Manager defined.
2.I should have count notes instead of users:
Count('notes', distinct=True)

Categories