We Keep Coding

Django model set lookup very slow

I'm getting a very slow lookup in my Django models.
I have two tables:
class Scan(models.Model):
scan_name = models.CharField(max_length=32, unique=True, validators=[alphanumeric_plus_validator])
class ScanProcessingInfo(models.Model):
scan_name = models.CharField(max_length=32)
processing_name = models.CharField(max_length=64)
in_progress = models.BooleanField(default=False)
When I perform the following operation to get a list of all Scan objects which have a ScanProcessingInfo for a specific processing_name:
scans = models.Scan.objects.all()
scan_set = []
for scan in scans:
if self.set_type_definition.test_scan(scan, self.arg1, self.arg2):
scan_set.append(scan)
(test_scan routes to)
def get_proc_info_been_done(scan, spd_name):
try:
proc_info = models.ScanProcessingInfo.objects.get(scan_name = scan.scan_name)
except models.ScanProcessingInfo.DoesNotExist:
proc_info = None
if proc_info == None:
return False
return not proc_info.in_progress
the request takes about 10 seconds. There are 300 Scans in total and 10 ScanProcessingInfos. The db backend is an RDS MySQL db. I also expect someone will tell me off for using strings for the cross-table identifiers, but I doubt that's the cause here.
I'm sure I'm doing something obvious wrong, but would appreciate a pointer, thank you.

I think what you're asking is how to get all Scans for which a matching ScanProcessingInfo exists.
The first thing to do is to declare the actual relationship. You don't need to change your database (you should, but you don't have to); you can use your existing underlying field, but just tell Django to treat it as a foreign key.
class ScanProcessingInfo(models.Model):
scan = models.ForeignKey('Scan', to_field='scan_name', db_field='scan_name', on_delete=models.DO_NOTHING)
Now you can use this relationship to get all the scans in one go:
scan_set = Scan.objects.exclude(scanprocessinginfo=None)
Edit
To get all matching objects with a specific attribute, use the double-underscore syntax:
scan_set = Scan.objects.filter(scanprocessinginfo__processing_name=spd_name)

Use Many-to-one relationship.
scan_name = ForeignKey(Scan, related_name='processing_infos',on_delete=models.CASCADE)

Django prefetch_select and reverse join issue

I am working on multiple join with Django.
questions = Question.objects.select_related(
'publishedquestionmapping__library_question'
).prefetch_related(
'publishedquestionmapping__library_question__contributor_set__user',
).filter(
hide=False).order_by('-pub_date')[:count]
for question in questions:
contributers = question.publishedquestionmapping.library_question.contributor_set.all()
for contributer in contributers:
print contributer.user
The Error is RelatedObjectDoesNotExist: Contributor has no user.
The problem is that I can use prefetch_related
...contributor_set__user, but I can't use contributer.user?
btw, print contributer.user_id is ok.
This Model Contributer
class Contributor(models.Model):
user = models.ForeignKey(User)
question = models.ForeignKey(Question)
I want to do like contributer.user.username. How should I do that?!
Thanks!

How to optimize a DB call existing under a FOR loop (platform: Django, Python, Postgres) [duplicate]

This question already has an answer here:
Rewriting this database access query to circumvent 'DISTINCT ON' error, in Django
(1 answer)
Closed 7 years ago.
I'm optimizing a Django website I maintain, for performance. Among other things, I've made the mistake of including non-trivial db calls under a FOR loop. Much better practice would be to make the DB call once, and then loop over the data as much as I want. How can I accomplish that in the following piece of code? Need a protip here!
link_ids = [link.id for link in context["object_list"]]
seen_replies = Publicreply.objects.filter(answer_to_id__in=link_ids,publicreply_seen_related__seen_user = user)
for link in context["object_list"]:
try:
latest_reply = link.publicreply_set.latest('submitted_on')
if latest_reply in seen_replies:
#do something
except:
pass
Essentially, profiling tells me that the line latest_reply = link.publicreply_set.latest('submitted_on') is adding significant overhead because it's doing a ton of DB queries (being under the FOR loop and all).
I can't seem to figure out a nice, clean way to move the call to outside the loop, and then process it's constituents within it. Anyone got any ideas?
Note: link.publicreply_set.latest('submitted_on') may yield DoesNotExist. I am on Postgres in production, but SQLite locally.
Models are:
class Link(models.Model):
description = models.TextField(validators=[MaxLengthValidator(500)])
submitter = models.ForeignKey(User)
submitted_on = models.DateTimeField(auto_now_add=True)
class Publicreply(models.Model):
submitted_by = models.ForeignKey(User)
answer_to = models.ForeignKey(Link)
submitted_on = models.DateTimeField(auto_now_add=True)
description = models.TextField(validators=[MaxLengthValidator(250)])
class Seen(models.Model):
seen_status = models.BooleanField(default=False)
seen_user = models.ForeignKey(User)
seen_at = models.DateTimeField(auto_now_add=True)
which_reply = models.ForeignKey(Publicreply, related_name="publicreply_seen_related")

You can take advantage of postgresql window function first_value. This is an approach how you can improve performance:
link_ids = [link.id for link in context["object_list"]]
seen_replies = ( Publicreply
.objects
.filter(answer_to_id__in=link_ids,
publicreply_seen_related__seen_user = user)
)
latest_replies = dict ( seen_replies
.extra(
select={
"id_latest_reply":
"""first_value(id) over
(partition by answer_to_id
order by submitted_on desc)
""" } )
.order_by()
.values_list( 'answer_to_id', 'id' )
.distinct()
)
for link in context["object_list"]:
try:
if link.id in latest_replies:
last_replie= (Publicreply
.objects
.get(id=latest_reply[link.id])
)
#do something
except:
pass

Django. Python. How to get current user's activity log?

I need to pass the current user's entire activity log to an html page, but it seems I cannot find any helpful solution regarding the same.
Is it possible? If yes, please direct me in the right way?
Thanks in advance!
Update:
I found a solution making use of a get() call to django's LogEntry model, but I am clueless as to what shall be the appropriate parameters for doing the same.
Yet another UPDATE:
I am looking for a way to access the activity log of a particular user from the django's log entries WITHOUT saving it to any database

Take a look below listed.....Hope it will help::
lets example::
Create Two Field in Models:
last_activity_ip = models.IPAddressField()
last_activity_date = models.DateTimeField(default = datetime(1960, 1, 1))
user = models.OneToOneField(User, primary_key=True)
Since the User and UserActivity models are now related one-to-one we can now type:
Run the Query Like this:
a = User.objects.get(username__exact='mpcabd')
print a.useractivity.last_activity_ip
b = UserActivity.objects.get(user=a)
print b.user.username
** To track the activity use this **
activity = None
try:
activity = request.user.useractivity
except:
activity = UserActivity()
activity.user = request.user
activity.last_activity_date = datetime.now()
activity.last_activity_ip = request.META['REMOTE_ADDR']
activity.save()
return
activity.last_activity_date = datetime.now()
activity.last_activity_ip = request.META['REMOTE_ADDR']
activity.save()

This question don't has a short answer, you can use sentry project by side of main django project. below link can helping you:
https://sentry.readthedocs.org/en/latest/

Django: Proper Way to Update Models

Suppose I have the following function, which retrieves data from a server in the form of user-defined objects. For example, let's define the objects as PersonalEntry.
def retrieve_people()
// returns a list of all latest people objects, with fields equal to those in PersonEntry
def retrieve_books()
// returns a list of all latest book objects regardless of author, with fields equal to those in BookEntry, contains attribute,
Both user-defined classes has an .as_dict() method which returns all its attributes in a dictionary.
I would like to update the model whenever this function is called (ie. update the fields if the instance of that model already exists, else, define a new instance of the model). This is my current setup.
class PersonEntry(models.Model):
name = models.CharField(max_length = 50)
age = models.IntegerField()
biography = models.CharField(max_length = 200)
def update_persons():
try:
temp = retrieve_person()
for person in temp:
match = PersonEntry.objects.filter(name = person.name(), age = person.age())
match.update(**person.as_dict())
except DoesNotExist:
PersonEntry.create(**person.as_dict())
class BookEntry(models.Model):
author = models.ForeignKey(PersonEntry)
author_name = models.CharField(max_length = 50) //books return redundant info
author_age = models.IntegerField() //books return redundant info
title = models.CharField(max_length = 50)
summary = models.CharField(max_length = 200)
def update_books():
try:
temp = retrieve_books()
for book in temp:
match = BookEntry.objects.filter(title = temp.title())
match.update(**book.as_dict(), associate_person(book.author_age(), book.author_name()))
except DoesNotExist:
BookEntry.create(**book.as_dict(), associate_person(book.author_age(), book.author_name())
def associate_person(age, name):
return PersonEntry.get(name = name, age = age)
I suppose a more general question is, how do I update models with relationships if I have a function which returns data? Do I have a method in the model itself, or do I have it one level up (ie. move update_books to the Person model) I'm new to Django, so not really sure how the organization should be.

I confess I haven't completely grokked your question, but I'll take a punt that you should look into
Managers
Generally, in django, everything is done as lazily as possible - meaning nothing gets updated until you actually try to use it - so you don't update models/relationships as you go, rather you just declare what they are (perhaps with a manager) then it works it out the current value only when asked.

Methods returning a collection (or queryset) of some kind of a model, should be a part of the Managers. So in your case update_books should be in a custom manager for BookEntry and update_persons should be in custom manager for PersonEntry.
Also do not call the function retrieve_* from inside the model or manager. Call it in your application logic and then pass the result to the manager method unless that method itself is part of the manager/model.
You do not need a separate filter and update method. Depending on how you have retrieved the data and it has a pk you can directly do a save. See How Does Django Know when to create or update