This should be easy... I'm taking a course and the way this is structured the view.py file can stand alone for the purpose of this quiz - it's more like a challenge. Ok, so the request is this:
"Now, create a view named article_list that selects all Article instances and returns an HttpResponse like "There are 5 articles." Be sure to use the len() of the Article queryset to get the number of articles."
As a point of reference, the first part works. Anyway, the view.py file looks like this (The first part is where I import Article... guess I don't need the render import.):
from django.http import HttpResponse
from django.shortcuts import render
from .models import Article
# Write your views here
def article_list(request):
articles = Article.objects.all()
article_count = articles.len()
output = "There are {} articles.".format(str(article_count))
return HttpResponse(output)
I'm not sure if the len function and/or the str function should be chained like this or if they should wrap the objects. I tried putting articles.len().str() and that didn't seem to work.
Thanks in advance for any help,
Bruce
You can try this:
article_count = len(articles)
output = "There are {} articles.".format(article_count)
len is a function that you call on a object and the format method already calls the str method for you.
Using .all() just to count the number of records introduces an unnecessary overhead:
If you only need to determine the number of records in the set (and
don’t need the actual objects), it’s much more efficient to handle a
count at the database level using SQL’s SELECT COUNT(*). Django
provides a count() method for precisely this reason.
Directly get the count from the database using .count():
articles_count = Article.objects.count()
output = "There are {} articles.".format(article_count)
This is a generally accepted way to count the records unless the record instances are actually needed, or, if there is a specific requirement, as in your case:
Be sure to use the len() of the Article queryset to get the number of articles.
Taking into account how the problem was stated, #Andoni's answer should be accepted.
Note: Don't use len() on QuerySets if all you want to do is determine the number of records in the set. It's much more efficient to handle a count at the database level, using SQL's SELECT COUNT(*), and Django provides a count() method for precisely this reason. See count() below.
Source
Related
I have Order objects and OrderOperation objects that represent an action on a Order (creation, modification, cancellation).
Conceptually, an order has 1 to many order operations. Each time there is an operation on the order, the total is computed in this operation. Which means when I need to find an attribute of an order, I just get the last order operation attribute instead, using a Subquery.
The simplified code
class OrderOperation(models.Model):
order = models.ForeignKey(Order)
total = DecimalField(max_digits=9, decimal_places=2)
class Order(models.Model)
# ...
class OrderQuerySet(query.Queryset):
#staticmethod
def _last_oo(field):
return Subquery(OrderOperation.objects
.filter(order_id=OuterRef("pk"))
.order_by('-id')
.values(field)
[:1])
def annotated_total(self):
return self.annotate(oo_total=self._last_oo('total'))
This way, I can run my_order_total = Order.objects.annotated_total()[0].oo_total. It works great.
The issue
Computing total is easy as it's a simple value. However, when there is a M2M or OneToMany field, this method does not work. For example, using the example above, let's add this field:
class OrderOperation(models.Model):
order = models.ForeignKey(Order)
total = DecimalField(max_digits=9, decimal_places=2)
ordered_articles = models.ManyToManyField(Article,through='orders.OrderedArticle')
Writing something like the following does NOT work as it returns only 1 foreign key (not a list of all the FKs):
def annotated_ordered_articles(self):
return self.annotate(oo_ordered_articles=self._last_oo('ordered_articles'))
The purpose
The whole purpose is to allow a user to search among all orders, providing a list or articles in input. For example: "Please find all orders containing at least article 42 or article 43", or "Please find all orders containing exactly article 42 and 43", etc.
If I could get something like:
>>> Order.objects.annotated_ordered_articles()[0].oo_ordered_articles
<ArticleQuerySet [<Article: Article42>, <Article: Article43>]>
or even:
>>> Order.objects.annotated_ordered_articles()[0].oo_ordered_articles
[42,43]
That would solve my issue.
My current idea
Maybe something like ArrayAgg (I'm using pgSQL) could do the trick, but I'm not sure to understand how to use it in my case.
Maybe this has to do with values() method that seems to not be intended to handle M2M and 1TM relations as stated in the doc:
values() and values_list() are both intended as optimizations for a
specific use case: retrieving a subset of data without the overhead of
creating a model instance. This metaphor falls apart when dealing with
many-to-many and other multivalued relations (such as the one-to-many
relation of a reverse foreign key) because the “one row, one object”
assumption doesn’t hold.
ArrayAgg will be great if you want to fetch only one variable (ie. name) from all articles. If you need more, there is a better option for that:
prefetch_related
Instead, you can prefetch for each Order, latest OrderOperation as a whole object. This adds the ability to easily get any field from OrderOperation without extra magic.
The only caveat with that is that you will always get a list with one operation or an empty list when there are no operations for selected order.
To do that, you should use prefetch_related queryset model together with Prefetch object and custom query for OrderOperation. Example:
from django.db.models import Max, F, Prefetch
last_order_operation_qs = OrderOperation.objects.annotate(
lop_pk=Max('order__orderoperation__pk')
).filter(pk=F('lop_pk'))
orders = Order.objects.prefetch_related(
Prefetch('orderoperation_set', queryset=last_order_operation_qs, to_attr='last_operation')
)
Then you can just use order.last_operation[0].ordered_articles to get all ordered articles for particular order. You can add prefetch_related('ordered_articles') to first queryset to have improved performance and less queries on database.
To my surprise, your idea with ArrayAgg is right on the money. I didn't know there was a way to annotate with an array (and I believe there still isn't for backends other than Postgres).
from django.contrib.postgres.aggregates.general import ArrayAgg
qs = Order.objects.annotate(oo_articles=ArrayAgg(
'order_operation__ordered_articles__id',
'DISTINCT'))
You can then filter the resulting queryset using the ArrayField lookups:
# Articles that contain the specified array
qs.filter(oo_articles__contains=[42,43])
# Articles that are identical to the specified array
qs.filter(oo_articles=[42,43,44])
# Articles that are contained in the specified array
qs.filter(oo_articles__contained_by=[41,42,43,44,45])
# Articles that have at least one element in common
# with the specified array
qs.filter(oo_articles__overlap=[41,42])
'DISTINCT' is needed only if the operation may contain duplicate articles.
You may need to tweak the exact name of the field passed to the ArrayAgg function. For subsequent filtering to work, you may also need to cast id fields in the ArrayAgg to int as otherwise Django casts the id array to ::serial[], and my Postgres complained about type "serial[]" does not exist:
from django.db.models import IntegerField
from django.contrib.postgres.fields.array import ArrayField
from django.db.models.functions import Cast
ArrayAgg(Cast('order_operation__ordered_articles__id', IntegerField()))
# OR
Cast(ArrayAgg('order_operation__ordered_articles__id'), ArrayField(IntegerField()))
Looking at your posted code more closely, you'll also have to filter on the one OrderOperation you are interested in; the query above looks at all operations for the relevant order.
A have piece of code, which fetches some QuerySet from DB and then appends new calculated field to every object in the Query Set. It's not an option to add this field via annotation (because it's legacy and because this calculation based on another already pre-fetched data).
Like this:
from django.db import models
class Human(models.Model):
name = models.CharField()
surname = models.CharField()
def calculate_new_field(s):
return len(s.name)*42
people = Human.objects.filter(id__in=[1,2,3,4,5])
for s in people:
s.new_column = calculate_new_field(s)
# people.somehow_reorder(new_order_by=new_column)
So now all people in QuerySet have a new column. And I want order these objects by new_column field. order_by() will not work obviously, since it is a database option. I understand thatI can pass them as a sorted list, but there is a lot of templates and other logic, which expect from this object QuerySet-like inteface with it's methods and so on.
So question is: is there some not very bad and dirty way to reorder existing QuerySet by dinamically added field or create new QuerySet-like object with this data? I believe I'm not the only one who faced this problem and it's already solved with django. But I can't find anything (except for adding third-party libs, and this is not an option too).
Conceptually, the QuerySet is not a list of results, but the "instructions to get those results". It's lazily evaluated and also cached. The internal attribute of the QuerySet that keeps the cached results is qs._result_cache
So, the for s in people sentence is forcing the evaluation of the query and caching the results.
You could, after that, sort the results by doing:
people._result_cache.sort(key=attrgetter('new_column'))
But, after evaluating a QuerySet, it makes little sense (in my opinion) to keep the QuerySet interface, as many of the operations will cause a reevaluation of the query. From this point on you should be dealing with a list of Models
Can you try it functions.Length:
from django.db.models.functions import Length
qs = Human.objects.filter(id__in=[1,2,3,4,5])
qs.annotate(reorder=Length('name') * 42).order_by('reorder')
I commonly find myself writing the same criteria in my Django application(s) more than once. I'll usually encapsulate it in a function that returns a Django Q() object, so that I can maintain the criteria in just one place.
I will do something like this in my code:
def CurrentAgentAgreementCriteria(useraccountid):
'''Returns Q that finds agent agreements that gives the useraccountid account current delegated permissions.'''
AgentAccountMatch = Q(agent__account__id=useraccountid)
StartBeforeNow = Q(start__lte=timezone.now())
EndAfterNow = Q(end__gte=timezone.now())
NoEnd = Q(end=None)
# Now put the criteria together
AgentAgreementCriteria = AgentAccountMatch & StartBeforeNow & (NoEnd | EndAfterNow)
return AgentAgreementCriteria
This makes it so that I don't have to think through the DB model more than once, and I can combine the return values from these functions to build more complex criterion. That works well so far, and has saved me time already when the DB model changes.
Something I have realized as I start to combine the criterion from these functions that is that a Q() object is inherently tied to the type of object .filter() is being called on. That is what I would expect.
I occasionally find myself wanting to use a Q() object from one of my functions to construct another Q object that is designed to filter a different, but related, model's instances.
Let's use a simple/contrived example to show what I mean. (It's simple enough that normally this would not be worth the overhead, but remember that I'm using a simple example here to illustrate what is more complicated in my app.)
Say I have a function that returns a Q() object that finds all Django users, whose username starts with an 'a':
def UsernameStartsWithAaccount():
return Q(username__startswith='a')
Say that I have a related model that is a user profile with settings including whether they want emails from us:
class UserProfile(models.Model):
account = models.OneToOneField(User, unique=True, related_name='azendalesappprofile')
emailMe = models.BooleanField(default=False)
Say I want to find all UserProfiles which have a username starting with 'a' AND want use to send them some email newsletter. I can easily write a Q() object for the latter:
wantsEmails = Q(emailMe=True)
but find myself wanting to something to do something like this for the former:
startsWithA = Q(account=UsernameStartsWithAaccount())
# And then
UserProfile.objects.filter(startsWithA & wantsEmails)
Unfortunately, that doesn't work (it generates invalid PSQL syntax when I tried it).
To put it another way, I'm looking for a syntax along the lines of Q(account=Q(id=9)) that would return the same results as Q(account__id=9).
So, a few questions arise from this:
Is there a syntax with Django Q() objects that allows you to add "context" to them to allow them to cross relational boundaries from the model you are running .filter() on?
If not, is this logically possible? (Since I can write Q(account__id=9) when I want to do something like Q(account=Q(id=9)) it seems like it would).
Maybe someone suggests something better, but I ended up passing the context manually to such functions. I don't think there is an easy solution, as you might need to call a whole chain of related tables to get to your field, like table1__table2__table3__profile__user__username, how would you guess that? User table could be linked to table2 too, but you don't need it in this case, so I think you can't avoid setting the path manually.
Also you can pass a dictionary to Q() and a list or a dictionary to filter() functions which is much easier to work with than using keyword parameters and applying &.
def UsernameStartsWithAaccount(context=''):
field = 'username__startswith'
if context:
field = context + '__' + field
return Q(**{field: 'a'})
Then if you simply need to AND your conditions you can combine them into a list and pass to filter:
UserProfile.objects.filter(*[startsWithA, wantsEmails])
With a method to get all related connections, is it better to simply return a queryset, or iterate through the queryset to return a list of IDs?
example code:
class Foo(models.Model):
bar = models.ForeignKey(Bar, related_name="foos")
class Bar(models.Model):
is_active = models.BooleanField(default=True)
def get_foos(self):
#this is the method to focus on for the question
to return a queryset all that would need to be in get_foos is the following:
return self.foos.all()
but to get a list of ids, it would have to look as such:
foos = self.foos.all()
ids = []
for foo in foos:
ids.append(foo.pk)
return ids
I was looking through an API, and they used the latter, but I don't really understand why you would ever actually do that if you could just do a one-liner like the former! Can someone please explain the benefits of these methods and if there are specific cases in which one is better than the other?
You can get Foo ids in one line:
ids = self.foos.values_list('pk', flat=True)
Using id list is more efficient if used in other expressions, e.g.:
my_query.filter(foo__pk__in=ids)
If you really saw something like your second snippet in some published app, then you can send them a patch (and probably more than one, since chances are you'll find a lot of bad code in this app). Point is : there's a lot of very poorly written Django apps around. As Niekas posted, you can get the same result with whay better performances in a no-brainer one-liner.
Now wrt/ why return a list of pks instead of a full queryset, well, sometimes that's really just what you need, and sometimes that's just silly - depends on the context, really.
I don't think this is a better vs worse scenario. The ID list can also be generated with a one-liner:
list(self.foos.values_list('pk', flat=True))
It's sometimes good practice for a function to return the minimal or least powerful representation of the data requested. Maybe you don't want callers to have access to the queryset object.
Maybe those IDs are going to the template rendering engine, and you'd rather not have templates with write access to querysets, for example.
I am trying to test my Django views. This view passes a QuerySet to the template:
def merchant_home(request, slug):
merchant = Merchant.objects.get(slug=slug)
product_list = merchant.products.all()
return render_to_response('merchant_home.html',
{'merchant': merchant,
'product_list': product_list},
context_instance=RequestContext(request))
and test:
def test(self):
"Merchant home view should send merchant and merchant products to the template"
merchant = Merchant.objects.create(name='test merchant')
product = Product.objects.create(name='test product', price=100.00)
merchant.products.add(product)
test_client = Client()
response = test_client.get('/' + merchant.slug)
# self.assertListEqual(response.context['product_list'], merchant.products.all())
self.assertQuerysetEqual(response.context['product_list'], merchant.products.all())
EDIT
I am using self.assertQuerysetEqual instead of self.assertListEqual. Unfortunately this still doesn't work, and the terminal displays this:
['<Product: Product object>'] != [<Product: Product object>]
assertListEqual raises: 'QuerySet' object has no attribute 'difference' and
assertEqual does not work either, although self.assertSetEqual(response.context['product_list'][0], merchant.products.all()[0]) does pass.
I assume this is because the QuerySets are different objects even though they contain the same model instances.
How do I test that two QuerySets contain the same data? I am even testing this correctly? This is my 4th day learning Django so I would like to know best practices, if possible. Thanks.
By default assertQuerysetEqual uses repr() on the first argument. This is why you were having issues with the strings in the queryset comparison.
To work around this you can override the transform argument with a lambda function that doesn't use repr():
self.assertQuerysetEqual(queryset_1, queryset_2, transform=lambda x: x)
Use assertQuerysetEqual, which is built to compare the two querysets for you. You will need to subclass Django's django.test.TestCase for it to be available in your tests.
I just had the same problem. The second argument of assertQuerysetEqual needs to be a list of the expected repr()s as strings. Here is an example from the Django test suite:
self.assertQuerysetEqual(c1.tags.all(), ["<Tag: t1>", "<Tag: t2>"], ordered=False)
I ended up solving this issue using map to repr() each entry in the queryset inside the self.assertQuerysetEqual call, e.g.
self.assertQuerysetEqual(queryset_1, map(repr, queryset_2))
An alternative, but not necessarily better, method might look like this (testing context in a view, for example) when using pytest:
all_the_things = Things.objects.all()
assert set(response.context_data['all_the_things']) == set(all_the_things)
This converts it to a set, which is directly comparable with another set. Be careful with the behaviour of set though, it might not be exactly what you want since it will remove duplicates and ignore the order of objects.
I found that using self.assertCountEqual(queryset1, queryset2) also solves the issue.