Is there a way to speed this up?
section_list_active = Section.objects.all().filter(course=this_course,Active=True,filter(teacher__in=[this_teacher]).order_by('pk')
if section_list_active.count() > 0:
active_section_id = section_list_active.first().pk
else:
section_list = Section.objects.all().filter(course=this_course).filter(teacher__in=[this_teacher]).order_by('pk')
active_section_id = section_list.first().pk
I know that the querysets aren't evaluated until they're used (the .first().pk bit?), so is there then no faster way to do this? If they weren't lazy, I could hit the DB for the whole list of sections, then just filter that set for ones with the property Active, but I'm not seeing how I can do this without hitting it twice (assuming that I have to go to the else).
Given you are only intersted in the Section (or the corresponding pk), Yes. You can order by theActive field first:
active_section_id = Section.objects.filter(
course=this_course,
teacher=this_teacher
).order_by('-Active', 'pk').first().pk
or even a bit more elegant:
active_section_id = Section.objects.filter(
course=this_course,
teacher=this_teacher
).earliest('-Active', 'pk').pk
We thus will first order Active in descending order such that if there is a row with Active=True, then this one will be sorted at the top. Next we thus take the .first() object, and optionally take the pk.
In case there is no Section that satsifies the filter conditions, then .first() or .earliest() will return None, and this will thus result in an AttributeError when fetching the pk attribute.
Note: typically fields are written in lowercase, so active instead of Active.
Related
I have Order objects and OrderOperation objects that represent an action on a Order (creation, modification, cancellation).
Conceptually, an order has 1 to many order operations. Each time there is an operation on the order, the total is computed in this operation. Which means when I need to find an attribute of an order, I just get the last order operation attribute instead, using a Subquery.
The simplified code
class OrderOperation(models.Model):
order = models.ForeignKey(Order)
total = DecimalField(max_digits=9, decimal_places=2)
class Order(models.Model)
# ...
class OrderQuerySet(query.Queryset):
#staticmethod
def _last_oo(field):
return Subquery(OrderOperation.objects
.filter(order_id=OuterRef("pk"))
.order_by('-id')
.values(field)
[:1])
def annotated_total(self):
return self.annotate(oo_total=self._last_oo('total'))
This way, I can run my_order_total = Order.objects.annotated_total()[0].oo_total. It works great.
The issue
Computing total is easy as it's a simple value. However, when there is a M2M or OneToMany field, this method does not work. For example, using the example above, let's add this field:
class OrderOperation(models.Model):
order = models.ForeignKey(Order)
total = DecimalField(max_digits=9, decimal_places=2)
ordered_articles = models.ManyToManyField(Article,through='orders.OrderedArticle')
Writing something like the following does NOT work as it returns only 1 foreign key (not a list of all the FKs):
def annotated_ordered_articles(self):
return self.annotate(oo_ordered_articles=self._last_oo('ordered_articles'))
The purpose
The whole purpose is to allow a user to search among all orders, providing a list or articles in input. For example: "Please find all orders containing at least article 42 or article 43", or "Please find all orders containing exactly article 42 and 43", etc.
If I could get something like:
>>> Order.objects.annotated_ordered_articles()[0].oo_ordered_articles
<ArticleQuerySet [<Article: Article42>, <Article: Article43>]>
or even:
>>> Order.objects.annotated_ordered_articles()[0].oo_ordered_articles
[42,43]
That would solve my issue.
My current idea
Maybe something like ArrayAgg (I'm using pgSQL) could do the trick, but I'm not sure to understand how to use it in my case.
Maybe this has to do with values() method that seems to not be intended to handle M2M and 1TM relations as stated in the doc:
values() and values_list() are both intended as optimizations for a
specific use case: retrieving a subset of data without the overhead of
creating a model instance. This metaphor falls apart when dealing with
many-to-many and other multivalued relations (such as the one-to-many
relation of a reverse foreign key) because the “one row, one object”
assumption doesn’t hold.
ArrayAgg will be great if you want to fetch only one variable (ie. name) from all articles. If you need more, there is a better option for that:
prefetch_related
Instead, you can prefetch for each Order, latest OrderOperation as a whole object. This adds the ability to easily get any field from OrderOperation without extra magic.
The only caveat with that is that you will always get a list with one operation or an empty list when there are no operations for selected order.
To do that, you should use prefetch_related queryset model together with Prefetch object and custom query for OrderOperation. Example:
from django.db.models import Max, F, Prefetch
last_order_operation_qs = OrderOperation.objects.annotate(
lop_pk=Max('order__orderoperation__pk')
).filter(pk=F('lop_pk'))
orders = Order.objects.prefetch_related(
Prefetch('orderoperation_set', queryset=last_order_operation_qs, to_attr='last_operation')
)
Then you can just use order.last_operation[0].ordered_articles to get all ordered articles for particular order. You can add prefetch_related('ordered_articles') to first queryset to have improved performance and less queries on database.
To my surprise, your idea with ArrayAgg is right on the money. I didn't know there was a way to annotate with an array (and I believe there still isn't for backends other than Postgres).
from django.contrib.postgres.aggregates.general import ArrayAgg
qs = Order.objects.annotate(oo_articles=ArrayAgg(
'order_operation__ordered_articles__id',
'DISTINCT'))
You can then filter the resulting queryset using the ArrayField lookups:
# Articles that contain the specified array
qs.filter(oo_articles__contains=[42,43])
# Articles that are identical to the specified array
qs.filter(oo_articles=[42,43,44])
# Articles that are contained in the specified array
qs.filter(oo_articles__contained_by=[41,42,43,44,45])
# Articles that have at least one element in common
# with the specified array
qs.filter(oo_articles__overlap=[41,42])
'DISTINCT' is needed only if the operation may contain duplicate articles.
You may need to tweak the exact name of the field passed to the ArrayAgg function. For subsequent filtering to work, you may also need to cast id fields in the ArrayAgg to int as otherwise Django casts the id array to ::serial[], and my Postgres complained about type "serial[]" does not exist:
from django.db.models import IntegerField
from django.contrib.postgres.fields.array import ArrayField
from django.db.models.functions import Cast
ArrayAgg(Cast('order_operation__ordered_articles__id', IntegerField()))
# OR
Cast(ArrayAgg('order_operation__ordered_articles__id'), ArrayField(IntegerField()))
Looking at your posted code more closely, you'll also have to filter on the one OrderOperation you are interested in; the query above looks at all operations for the relevant order.
The question related to python - django framework, and probably to experienced django developers. Googled it for some time, also seeked in django queryset itself, but have no answer. Is it possible to know if queryset has been filtered and if so, get key value of filtered parameters?
I'm developing web system with huge filter set, and I must predefine some user-background behavior if some filters had been affected.
Yes, but since to the best of my knowledge this is not documented, you probably should not use it. Furthermore it looks to me like bad design if you need to obtain this from a QuerySet.
For a QuerySet, for example qs, you can obtain the .query attribute, and then query for the .where attribute. The truthiness of that attribute checks if that node (this attribute is a WhereNode, which is a node in the syntax of the query) has children (these children are then individual WHERE conditions, or groups of such conditions), hence has done some filtering.
So for example:
qs = Model.objects.all()
bool(qs.query.where) # --> False
qs = Model.objects.filter(foo='bar')
bool(qs.query.where) # --> True
If you inspect the WhereNode, you can see the elements out of which it is composed, for example:
>>> qs.query.where
<WhereNode: (AND: <django.db.models.lookups.Exact object at 0x7f2c55615160>)>
and by looking to the children, we even can obtain details:
>>> qs.query.where.children[0]
>>> c1.lhs
Col(app_model, app.Model.foo)
>>> c1.lookup_name
'exact'
>>> c1.rhs
'bar'
But the notation is rather cryptic. Furthermore the WhereNode is not per se a conjunctive one (the AND), it can also be an disjunctive one (the OR), and it is not said that any filtering will be done (since the tests can trivially be true, like 1 > 0). We thus only query if there will be a non-empty WHERE in the SQL query. Not whether this query will restrict the queryset in any way (although you can of course inspect the WhereNode, and look if that holds).
Note that some constraints are not part of the WHERE, for example if you make a JOIN, you will perform an ON, but this is not a WHERE clause.
Since however the above is - to the best of my knowledge - not extenstively documented, it is probably not a good idea to depend on this, since that means that it can easily change, and thus no longer work.
You can use the query attribute (i.e. queryset.query) to get the data used in the SQL query (the output isn't exactly valid SQL).
You can also use queryset.query.__dict__ to get that data in a dictionary format.
I agree with Willem Van Onsen, in that accessing the internals of the query object isn't guaranteed to work in the future. It's correct for now, but might change.
But going half-way down that path, you could use the following:
is_filtered_query = bool(' WHERE ' in str(queryset.query))
which will pretty much do the job!
I need to get multiple random objects from a Django model.
I know I can get one random object from the model Person by typing:
person = Person.objects.order_by('?')[0]
Then, I saw suggestions in How to get two random records with Django saying I could simply do this by:
people = Person.objects.order_by('?')[0:n]
However, as soon as I add that [0:n], instead of returning the objects, Django returns a QuerySet object. This results in the unfortunate consequences that if I then ask for
print(people[0].first_name, people[0].last_name)
I get the first_name and last_name for 2 different people as QuerySets are evaluated as they are called (right?). How do I get the actual list of people that were returned from the first query?
I am using Python 3.4.0 and Django 1.7.1
Simeon Popov's answer solves the problem, but let me explain where it comes from.
As you probably know querysets are lazy and won't be evaluated until it's necessary. They also have an internal cache that gets filled once the entire queryset is evaluated. If only a single object is taken from a queryset (or a slice with a step specified, i.e. [0:n:2]), Django evaluates it, but the results won't get cached.
Take these two examples:
Example 1
>>> people = Person.objects.order_by('?')[0:n]
>>> print(people[0].first_name, people[0].last_name)
# first and last name of different people
Example 2
>>> people = Person.objects.order_by('?')[0:n]
>>> for person in people:
>>> print(person.first_name, person.last_name)
# first and last name are properly matched
In example 1, the queryset is not yet evaluated when you access the first item. It won't get cached, so when you access the first item again it runs another query on the database.
In the second example, the entire queryset is evaluated when you loop over it. Thus, the cache is filled and there won't be any additional database queries that would change the order of the returned items. In that case the names are properly aligned to each other.
Methods for evaluating an entire queryset are a.o. iteration, list(), bool() and len(). There are some subtle differences between these methods. If all you want to do is make sure the queryset is cached, I'd suggest using bool(), i.e.:
>>> people = Person.objects.order_by('?')[0:n]
>>> bool(people)
True
>>> print(people[0].first_name, people[0].last_name)
# matching names
Try this ...
people = []
for person in Person.objects.order_by('?')[0:n]:
people.append(person)
So QuerySets are "lazy" and only run in certain instances (repr,list,etc.). I have created a custom QuerySet for doing many queries, but these could end up having millions of items!! That is way more than I want to return.
When returning the evaluated QuerySet, it should not have more than 25 results! Now I know I could do the following:
first = example.objects.filter(...)
last = first.filter(...)
result = last[:25]
#do stuff with result
but I will be doing so many queries with example objects that I feel it unnecessary to have the line result = last[:25]. Is there a way to specify how a QuerySet is returned?
If there is, how can I change it so that whenever the QuerySet would be evaluated it only returns the first x items in the QuerySet where, in this case, x = 25
Important note:
slicing must be on evaluation because that way I can chain queries without limited results, but when I return a result upon evaluation, it would have a max of x
I'm not sure I understand what your issue with slicing the queryset is. If it's the extra line of code or the hardcoded number that's bothering you, you can run
example.objects.filter(**filters)[:x]
and pass x into whatever method you're using.
You can write a custom Manager:
class LimitedNumberOfResultsManager(models.Manager):
def get_queryset(self):
return super(LimitedNumberOfResultsManager, self).get_queryset()[:25]
Note: You may think that adding a slice here will immediately evaluate the queryset. It won't. Instead information about the query limit will be saved to an underlying Query object and used later, during the final evaluation - as long as it is not overwritten by an another slice in the meantime.
Then add the manager to your model:
class YourModel(models.Model):
# ...
objects = LimitedNumberOfResultsManager()
After setting this up YourModel.objects.all() and other operations on your queryset will always return only up to 25 results. You can still overwrite this any time using slicing. For example:
This will return up to 25 resuls:
YourModel.objects.filter(lorem='ipsum')
but this will return up to 100 resuls:
YourModel.objects.filter(lorem='ipsum')[:100]
One more point. Overwriting the default manager may be confusing to other people reading your code. So I think it would be better to leave the default manager alone and use the custom one as an optional alternative:
class YourModel(models.Model):
# ...
objects = models.Manager()
limited = LimitedNumberOfResultsManager()
With this set up this will return all results:
YourModel.objects.all()
and this will return only up to 25 results:
YourModel.limited.all()
Depending on your exact use case you man also want to look at pagination in Django.
I'm having a attribute on my model to allow the user to order the objects. I have to update the element's order depending on a list, that contains the object's ids in the new order; right now I'm iterating over the whole queryset and set one objects after the other. What would be the easiest/fastest way to do the same with the whole queryset?
def update_ordering(model, order):
""" order is in the form [id,id,id,id] for example: [8,4,5,1,3] """
id_to_order = dict((order[i], i) for i in range(len(order)))
for x in model.objects.all():
x.order = id_to_order[x.id]
x.save()
This cannot be done in one single queryset operation. As far as I know it can't even be done in one query with raw SQL. So you will always need the update call for each object that has to be updated. So both your and Collin Anderson's solutions seem quite optimal for your description.
However, what's your use case? Is really the whole list going to change every time? In most situations this seems very unlikely. I can see some different approaches.
You save the order field like you say, but you generate a diff for the order list:def
update_ordering(model, order):
""" order is in the form [id,id,id,id] for example: [8,4,5,1,3] """
original_order = model.objects.value_list('id', flat=True).order_by('order')
order = filter( lambda x: x[1]!=x[2], zip(xrange(len(order)), order, original_order))
for i in order:
model.objects.filter(id=i[1]).update(order=i[0])
Another approach, depending on what you're making is to do a partial update (e.g. using AJAX) if possible, instead of updating the whole re-ordered set, just update every update separately. This will often increase the total load, but will spread it more over time. Take for example moving the 5th element step-by-step to place 2, this will introduce 3 swaps: (5,4); (4,3); (3,2). Resulting in 6 updates, while with the all-in-one-time approach only 4 will be needed. But the small operations will be spread over time.
I don't know if it is possible to do this using one query, but this is more efficient in any case:
def update_ordering(model, order):
""" order is in the form [id,id,id,id] for example: [8,4,5,1,3] """
for id in model.objects.values_list('id', flat=True):
model.objects.filter(id=id).update(order=order.index(id))