We Keep Coding

How to avoid ordering by in django queryset, order_by() not working

I have a "big" db whith over 60M records, and I'm trying to paginate by 50.
I have another db whith ~8M records and it works perfectly, but with the 60M amount it just never loads and overflows the db.
I found that the problem was the order_by(id) made by django so I tried using a mysql view already ordered by id, but then django tries to order it again. To avoid this, I used order_by(), which is supposed to avoid any ordering, but it still does it.
def get_queryset(self, request):
qs = super(CropAdmin, self).get_queryset(request)
qs1 = qs.only('id', 'grain__id', 'scan__id', 'scan__acquisition__id',
'validated', 'area', 'crop_date', 'matched_label', 'grain__grain_number', 'filename').order_by()
if request.user.is_superuser:
return qs1
The query made is still using order_by:
SELECT `crops_ordered`.`crop_id`,
`crops_ordered`.`crop_date`,
`crops_ordered`.`area`,
`crops_ordered`.`matched_label`,
`crops_ordered`.`validated`,
`crops_ordered`.`scan_id`,
`crops_ordered`.`grain_id`,
`crops_ordered`.`filename`,
`scans`.`scan_id`,
`scans`.`acquisition_id`,
`acquisitions`.`acquisition_id`,
`grains`.`grain_id`,
`grains`.`grain_number`
FROM `crops_ordered`
INNER JOIN `scans`
ON (`crops_ordered`.`scan_id` = `scans`.`scan_id`)
INNER JOIN `acquisitions`
ON (`scans`.`acquisition_id` = `acquisitions`.`acquisition_id`)
INNER JOIN `grains`
ON (`crops_ordered`.`grain_id` = `grains`.`grain_id`)
**ORDER BY `crops_ordered`.`crop_id` DESC**
LIMIT 50
Any idea on how to fix this? Or a better way to work with a db of this size?

I don't believe order_by() will work as there will most likely be a default parameter when Django implemented this function. Having said that, I believe this thread has the answer that you want.
Edit
The link in that thread might provide too much information at once, although there aren't many details on this either. If you don't like Github, there's also an official documentation page on this method but you'll have to manually look for clear_ordering by using CTRL + f or any equivalence.

How to use order_by on booleanfield using mongoengine

I want to order the result of my mongoengine call on two different fields.
Open. This has status True or False. I want this because I want to
display the open questions first
Opendate. Because I want to newest questions to show on top.
Combined this should create a list where I can see on top the open questions (ordered by creation date) and than the already closed questions, also ordered by creationdate.
The code I started with is:
To call the API:
questions = Questions.questions_of_user
To handle the call:
#queryset_manager
def questions_of_user(doc_cls, queryset):
return queryset.filter(questioner=current_user.id).order_by('-openDate')
My first suggestion was to just add 'status' to the order_by (maybe with or without + or - ) would do it. But so far no luck.
Than I tried to only order by the open field, because I thought I was just making an mistake combining the two. So I got this:
#queryset_manager
def questions_of_user(doc_cls, queryset):
return queryset.filter(questioner=current_user.id).order_by('-open')
That however did not work as well. I hope someone can help me out. Thanks in advance

More than one key can be passed in the order_by method of the queryset.
queryset.filter(questioner=current_user.id).order_by('-open', '-openDate')

django.views / django.models, can't delete entities with foreign key

Actually I have two things that don't seem to work. I'll list couple of models and their dependencies (shortened). The StudentGroup has students (which may be active/inactive), and messages, which are listed as chat.
In views.py, when I call delete_group(), I want to make all students inactive and delete all of the messages relevant to the group.
class StudentsGroup(models.Model):
students = models.ManyToManyField(User,limit_choices_to={'is_staff': False}, related_name="user_groups",blank=True)
finished = models.ManyToManyField(User,limit_choices_to={'is_staff': False}, related_name="finished_user_groups",blank=True)
class Message(models.Model):
group=models.ForeignKey(StudentsGroup)
def delete_group(request,group):
Message.objects.filter(group=group).delete()
groupl=StudentsGroup.objects.get(id=group)
for s in group1.students.all():
groupl.finished.add(s)
group1.save()
Nothing changes. I've tried similar things in console and it seemed to be ok. Tried bunch of similar code.
Tried to add makemigrations to the server restarting but still no result.
Kinda noob in django and webdev overall, any help would be appreciated.

In your line
Message.objects.filter(group=group).delete()
it looks like you haven't yet converted group from the raw ID value to an actual StudentGroup instance. If group is supplied to the function as a StudentGroup instance, then this should work. If, as I suspect, group is just a raw ID value, then your filter queries should be:
Message.objects.filter(group_id=group).delete()
It's nicer however to work with objects rather than IDs. Try instead doing this:
group = StudentGroup.objects.get(id=group)
Message.objects.filter(group=group).delete()
Then you can do other things like group = get_object_or_404(StudentGroup, id=group).
As for the second part, moving the Students to the group's finished list, your code looks reasonable, though you don't need the .save() part. I suspect the errors may be due to the object not being correctly instantiated yet, as with above.
However you should be able to do all this in a single line in Django 1.9+, see https://docs.djangoproject.com/en/1.10/ref/models/relations/#django.db.models.fields.related.RelatedManager.add
group1.finished.add(*list(group1.students.all()))

Duplicate/Archive entry in Django to another model

I have been at this for Two days and I'm now hoping someone can point me in the right direction. All I am trying to do is duplicate an entry in a table/model into another model with mirrored fields, essentially creating an archived version. I want this to happen when the user calls the update view.
What I have tried so far is setting pk to None and then trying to find a way to move the previous version to the mirrored/archive model. After a couple of hours of research I gave up on this path. Next I thought the answer would lie with the pre_save receiver but I can't find a way to access the model instance to then save that to the archive model.
#receiver(pre_save, sender=InstrumentAnnual)
def archive_calc_instance(sender, instance, **kwargs):
stored_id = getattr(instance, 'id', None)
e = InstrumentAnnual.objects.filter(id = stored_id)
archive = InstrumentAnnualArchive(e.field_name, e.another_field_name...)
archive.save()
As far as I can tell this should work however e only contains the First field from the model.
Is there something that can be done with this code to achieve my goal or is there a more 'Django' way to solve this? I.e. some sort of official archive feature?
Thanks in advance.

With the help of #Igor's comment I amended my solution to this:
def archive_calc(self, object_id):
annual = InstrumentAnnual.objects.get(id = object_id)
annual_archive = InstrumentAnnualArchive()
for field in annual._meta.fields:
setattr(annual_archive, field.name, getattr(annual, field.name))
annual_archive.pk = None
annual_archive.save()
It occured to me that using pre_save wouldn't work as it is listening/linked to a model, not a view as I originally thought. So I placed the above code in my Update View and called it passing the id in object_id.
Thanks again for the help.

You should be using named arguments in your constructor, otherwise the first argument will be interpreted as the id, so try:
# omitted code
e = InstrumentAnnual.objects.filter(id=stored_id)
archive = InstrumentalAnnualArchive(field_name=e.field_name, another_name=e.another_field_name, …)
archive.save()
But you could also use Django's create function, so:
# omitted code
e = InstrumentAnnual.objects.filter(id=stored_id)
archive = InstrumentalAnnualArchive.objects.create(field_name=e.field_name, another_name=e.another_field_name, …)
This way handles the save for you, so you don't need to explicitly save your object.

simulating django settings to retrieve from database, values getting cached

am trying to simulate django's settings file. Where I've built a model to host some of the settings where it can be changed by admin. The concepts works fine, but its acting strange when the value is changed the code is not picking up the new changes.
Here is my core_settings
class CoreSettings(object):
def __getattr__(self, item):
try:
return Configuration.objects.get(key=item).value
except Configuration.DoesNotExist :
return getattr(settings, item)
core_settings = CoreSettings()
am basically using the above as follows
SF_PUBLICATION_PAGINATION_PP = int(getattr(core_settings, 'SF_PUBLICATION_PAGINATION_PP'))
SF_PUBLICATION_PAGINATION_PP is getting the correct value from the Configuration model, but when I update the field value its not reflected. Only when I alter the file causing it to recompile where am getting the changes..
Any ideas?
update:
it seems like am seeing the changes only when the runserver is refreshed.

Yes, the value of your setting is not being refreshed, because its value is set when your settings.py is 'loaded', which happen when e.g you do 'runserver' in a dev enviroment.
So, you can deal with, doing something like this:
def get_live_setting(key):
try:
return Configuration.objects.get(key=key).value
except Configuration.DoesNotExist :
return getattr(settings, key)
# get SF_PUBLICATION_PAGINATION_PP's value
get_live_setting('SF_PUBLICATION_PAGINATION_PP')
with this you can solve your problem, but the correct way is using lazy evaluation, here some samples and here a funny post about 'lazy-python'.

What about using a package designed for this purpose?
Have a look at: django-livesettings
Even if you decide not to use it, you can always have a look at how it's done there!
Regarding your specific issue. how do you update the field value? Are you sure the value is retrieved from the database and not from your except clause?