We Keep Coding

How to write Django query to grab all objects in descending order according to two number fields?

I'm using Django. i'm trying to write query according to the top rated products. i have product table. as you can see below.
class Product(models.Model):
user = models.ForeignKey(User, verbose_name=_("Owner"), on_delete=models.CASCADE)
name = models.CharField(_("Name"), max_length=150,null=True)
average_rating =models.DecimalField(_("average rating"), max_digits=10, decimal_places=2,null=True,blank=True)
total_reviews = models.IntegerField(_("total reviews "),default=0,null=True,blank=True)
is_remove = models.BooleanField(_("Remove"), default=False)
create_time = models.DateTimeField(_("Create time"), default=timezone.now)
Now i want to get all objects which have highest average rating and total count.
I have tried many things below. but none of them worked.
1 -
def get_all_top_rated_products(self):
query = self.filter(is_remove=False).order_by("total_reviews","average_rating")
print(query)
return query
2
def get_all_top_rated_products(self):
query = self.filter(is_remove=False).aggregate(Max('average_rating'),Max('total_reviews'))
print(query)
return query

You should order in descending order, you can do this by prefixing the fieldname with a minus (-):
def get_all_top_rated_products(self):
return self.filter(is_remove=False).order_by(
'-average_rating', '-total_reviews'
)

Django order_by query runs incredibly slow in Python, but fast in DB

I have the following models:
class Shelf(models.Model):
user = models.ForeignKey(User, on_delete=models.CASCADE)
name = models.CharField(max_length=200, db_index=True)
slug = models.SlugField(max_length=200, editable=False)
games = models.ManyToManyField(Game, blank=True, through='SortedShelfGames')
objects = ShelfManager()
description = models.TextField(blank=True, null=True)
class SortedShelfGames(models.Model):
game = models.ForeignKey(Game, on_delete=models.CASCADE)
shelf = models.ForeignKey(Shelf, on_delete=models.CASCADE)
date_added = models.DateTimeField()
order = models.IntegerField(blank=True, null=True)
releases = models.ManyToManyField(Release)
objects = SortedShelfGamesManager.as_manager()
class Game(models.Model):
name = models.CharField(max_length=300, db_index=True)
sort_name = models.CharField(max_length=300, db_index=True)
...
I have a view where I want to get all of a user's SortedShelfGames, distinct on the Game relationship. I then want to be able to sort that list of SortedShelfGames on a few different fields. So right now, I'm doing the following inside of the SortedShelfGamesManager (which inherits from models.QuerySet) to get the list:
games = self.filter(
pk__in=Subquery(
self.filter(shelf__user=user).distinct('game').order_by('game', 'date_added').values('pk') # The order_by statement in here is to get the earliest date_added field for display
)
)
That works the way it's supposed to. However, whenever I try and do an order_by('game__sort_name'), the query takes forever in my python. When I'm actually trying to use it on my site, it just times out. If I take the generated SQL and just run it on my database, it returns all of my results in a fraction of a second. I can't figure out what I'm doing wrong here. The SortedShelfGames table has millions of records in it if that matters.
This is the generated SQL:
SELECT
"collection_sortedshelfgames"."id", "collection_sortedshelfgames"."game_id", "collection_sortedshelfgames"."shelf_id", "collection_sortedshelfgames"."date_added", "collection_sortedshelfgames"."order",
(SELECT U0."rating" FROM "reviews_review" U0 WHERE (U0."game_id" = "collection_sortedshelfgames"."game_id" AND U0."user_id" = 1 AND U0."main") LIMIT 1) AS "score",
"games_game"."id", "games_game"."created", "games_game"."last_updated", "games_game"."exact", "games_game"."date", "games_game"."year", "games_game"."quarter", "games_game"."month", "games_game"."name", "games_game"."sort_name", "games_game"."rating_id", "games_game"."box_art", "games_game"."description", "games_game"."slug", "games_game"."giantbomb_id", "games_game"."ignore_giantbomb", "games_game"."ignore_front_page", "games_game"."approved", "games_game"."user_id", "games_game"."last_edited_by_id", "games_game"."dlc", "games_game"."parent_game_id"
FROM
"collection_sortedshelfgames"
INNER JOIN
"games_game"
ON
("collection_sortedshelfgames"."game_id" = "games_game"."id")
WHERE
"collection_sortedshelfgames"."id"
IN (
SELECT
DISTINCT ON (U0."game_id") U0."id"
FROM
"collection_sortedshelfgames" U0
INNER JOIN
"collection_shelf" U1 ON (U0."shelf_id" = U1."id")
WHERE
U1."user_id" = 1
ORDER
BY U0."game_id" ASC, U0."date_added" ASC
)
ORDER BY
"games_game"."sort_name" ASC

I think you don't need a Subquery for this.

Here's what I ended up doing to solve this. Instead of using a Subquery, I created a list of primary keys by evaluating what I was using as the Subquery in, then feeding that into my query. It looks like this:
pks = list(self.filter(shelf__user=user).distinct('game').values_list('pk', flat=True))
games = self.filter(
pk__in=pks)
)
games = games.order_by('game__sort_name')
This ended up being pretty fast. This is essentially the same thing as the Subquery method, but whatever was going on underneath the hood in python/Django was slowing this way down.

how to select data between 2 dates using sql queries in django?

models.py
My models.py
class Custom_user_model(User):
daily_target = models.IntegerField()
monthly_target = models.IntegerField()
yearly_target = models.IntegerField()
weekly_target = models.IntegerField()
call_target = models.IntegerField()
email_target = models.IntegerField()
meeting_target = models.IntegerField()
added_under = models.IntegerField()
profile_pic = models.TextField()
doj = models.DateTimeField(default='')
location_id = models.IntegerField()
locked = models.BooleanField()
default_currency = models.IntegerField()
date_change_permission = models.BooleanField()
deal_back_log = models.BooleanField()
created_date=models.DateTimeField(auto_now_add=True)
role_id=models.IntegerField()
profile_pic = models.FileField(upload_to='.')
objects = UserManager()
//This custom_user model is the extension of django's default user model.
class Deal(models.Model):
a_choices = ((0,'yes'),(1,'no'))
approved = models.IntegerField(choices=a_choices,default=1)
user_id = models.IntegerField()
company_id = models.IntegerField()
contact_id = models.IntegerField()
deal_title=models.CharField(max_length=200)
deal_value = models.CharField(max_length=20)
currency_id = models.IntegerField()
process_id = models.IntegerField()
expected_close_date = models.DateField(default='')
closed_date = models.DateField()
deal_milestone=models.IntegerField()
created=models.DateTimeField(auto_now_add=True)
last_modified=models.DateTimeField(auto_now_add=True)
s_choices = ((0,'active'),(1,'won'),(2,'junk'),(3,'lost'))
status = models.IntegerField(choices=a_choices,default=0)
type = models.CharField(max_length=50, default='deal')
source = models.CharField(max_length=50,default='O')
class user_Roles(models.Model):
code = models.CharField(max_length=20)
description = models.CharField(max_length=30)
permitted_menus = models.CharField(max_length=200)
created = models.DateTimeField(auto_now_add=True)
Using user_roles model, I have assigned permission for accessing data to the newly created user based on his/her role. I want to get the created deals which are added by the users having role_id = 2 and deals created date between the specified dates .
### views.py
st_date, end_date = week_magic(date.today())
cur = connection.cursor()
cur.execute("select *, CONCAT(au.first_name,' ',au.last_name) as full_name from myapp_custom_user_model mu left join auth_user au on mu.user_ptr_id = au.id INNER JOIN myapp_user_roles ml on ml.id= 2 and ml.id = mu.role_id LEFT JOIN (SELECT user_id,SUM( deal_value ) AS cnt FROM myapp_deal where status = 1 and DATE_FORMAT(closed_date,'%Y-%m-%d') BETWEEN " '%s' " and " '%s' " GROUP BY user_id)d ON mu.user_ptr_id = d.user_id where mu.locked !=1 and mu.role_id = 2 order by COALESCE( d.cnt, 0 ) DESC",(st_date,end_date))
users = dictfetchall(cur)
cur.close()
While executing the query it shows unsupported format error. So I used one more % symbol in the same query as follows:
cur.execute("select *, CONCAT(au.first_name,' ',au.last_name) as full_name from myapp_custom_user_model mu left join auth_user au on mu.user_ptr_id = au.id INNER JOIN myapp_user_roles ml on ml.id= 2 and ml.id = mu.role_id LEFT JOIN (SELECT user_id,SUM( deal_value ) AS cnt FROM myapp_deal where status = 1 and DATE_FORMAT(closed_date,'%%Y-%%m-%%d') BETWEEN " '%s' " and " '%s' " GROUP BY user_id)d ON mu.user_ptr_id = d.user_id where mu.locked !=1 and mu.role_id = 2 order by COALESCE( d.cnt, 0 ) DESC" %(st_date,end_date))
It doesn't give any error but the result is empty even though there is data because of this syntax: DATE_FORMAT(closed_date,'%%Y-%%m-%%d'). How to solve this?

First of all you should use ForeignKey fields for role_id in Custom_user_model and user_id in Deal. The same is probably true for some of the other _id fields in your models.
class Custom_user_model(User):
...
role = models.ForeignKey('Role')
...
class Deal(models.Model):
...
user = models.ForeignKey('Custom_user_model')
...
After that you can do your query like this:
# get deals from users with role_id=2
query = Deal.objects.filter(user__role_id=2)
# add filter for deals created by that user created between
start_date, end_date = week_magic(date.today())
query = query.filter(created__between=(start_date, end_date))

Group by in Django?

I have the next structure in my django app:
class Telefono(models.Model):
tipo = models.CharField(max_length=200)
lada = models.IntegerField()
numero = models.CharField(max_length=12)
def __unicode__(self):
return (self.tipo +" - (" +str(self.lada)+")"+str(self.numero))
class Persona(models.Model):
nombre = models.CharField(max_length=200)
apellidoPaterno = models.CharField(max_length=200)
apellidoMaterno = models.CharField(max_length=200)
rfc = models.CharField(max_length=10)
ubicacion = models.OneToOneField(Ubicacion)
telefonos = models.ManyToManyField(Telefono)
The problem is: I need a single "telefono" to show in my datagrid.. so I built the next query in MYSQL:
select p.nombre,tf.numero from persona as p join persona_telefonos AS t ON t.persona_id=p.id join telefono As tf ON t.telefono_id=tf.id group by p.id
And actually gives this result:
"nombre" "numero"
"MiKchorro123213" "5345234523"
But in django I have this query:
Cliente.objects.filter().values("id","nombre","telefonos__numero", "ubicacion__direccion")
And I get two results.. one for each ("telefono") object in the many to many relationship..
How can I do a group by function in django?

Please refer documentation : Raw Query, Filterings, Group_by
example:
query = Cliente.objects.all().query
query.group_by = ['field_name']
results = QuerySet(query=query, model=Model_name)

MySQL INSERT ... ON DUPLICATE KEY UPDATE with django 1.4 for bulk insert

I am having issues figuring out MySQL INSERT ... ON DUPLICATE KEY UPDATE with django 1.4.
The table that I am trying to insert records has a 2 column(composite) unique key. Records that I am receiving is from a 3rd party source and values will change over time except for those fields that makes the unique key set. I am receiving 1 ~ 5k records at a time, and would need to
Currently I am using Model.objects.bulk_create to bulk insert, performance is really amazing as it issues generally one query no matter how big the record set is. However, as my records can change over time on the 3rd party end, I need to perform the MySQL INSERT ... ON DUPLICATE KEY UPDATE query on the recordset.
I am planning to write raw SQL statements and execute using something like here:
sql = "MySQL INSERT ... ON DUPLICATE KEY UPDATE"
raw_insert(sql)
def raw_insert(sql):
from django.db import connection, transaction
cursor = connection.cursor()
# Data modifying operation - commit required
cursor.execute(sql)
transaction.commit_unless_managed()
return 1
Wondering if there is a better solution to my problem. Also how would I sanitize the field values for raw insert?

So I created a custom manager. Here is the manager:
class BulkInsertManager(models.Manager):
def _bulk_insert_or_update(self, create_fields, update_fields, values):
from django.db import connection, transaction
cursor = connection.cursor()
db_table = self.model._meta.db_table
values_sql = []
values_data =[]
for value_lists in values:
values_sql.append( "(%s)" % (','.join([ "%s" for i in range(len(value_lists))]),) )
values_data.extend(value_lists)
base_sql = "INSERT INTO %s (%s) VALUES " % (db_table, ",".join(create_fields))
on_duplicates = []
for field in update_fields:
on_duplicates.append(field + "=VALUES(" + field +")")
sql = "%s %s ON DUPLICATE KEY UPDATE %s" % (base_sql, ", ".join(values_sql), ",".join(on_duplicates))
cursor.executemany(sql, [values_data])
transaction.commit_unless_managed()
And a sample model:
class User_Friend(models.Model):
objects = BulkInsertManager() # assign a custom manager to handle bulk insert
id = models.CharField(max_length=255)
user = models.ForeignKey(User, null=False, blank=False)
first_name = models.CharField(max_length=30)
last_name = models.CharField(max_length=30)
city = models.CharField(max_length=50, null=True, blank=True)
province = models.CharField(max_length=50, null=True, blank=True)
country = models.CharField(max_length=30, null=True, blank=True)
And sample implementation:
def save_user_friends(user, friends):
user_friends = []
for friend in friends:
create_fields = ['id', 'user_id', 'first_name', 'last_name', 'city', 'province', 'country']
update_fields = ['first_name', 'last_name', 'city', 'province', 'country']
user_friends.append(
[
str(user.id),
str(friend['id']),
friend['first_name'],
friend['last_name'],
friend['city'],
friend['province'],
friend['country'],
]
)
User_Friend.objects._bulk_insert_or_update(create_fields, update_fields, user_friends)
Here is the gist.

You could sanitize with a ModelForm:
from django.forms.models import modelform_factory
form_class = modelform_factory(MyModel)
for obj in my_data:
form = form_class(obj)
if not form.is_valid():
raise Hell()
As to the raw SQL, I say go for it. It doesn't appear that Django's ORM supports ON DUPLICATE KEY UPDATE, so don't let it get in your way. The Django docs talk about doing it without any reservation.
It may be worth using Manager.raw, though.