Applying an arbitrary sort to Django ORM querysets - python

I have a database table containing sets of items representing yearly recurring events. The record sets are stored by month and day. I often need to retrieve the event records corresponding to a range of calendar dates. I'm using the Django ORM to work with the records, so at present I convert the dates to corresponding Q objects (e.g. Q(month=month, day=day) and OR them together in the call to MyModel.objects.filter().
The problem comes if my date range intersects the new year. If I want the events from Dec. 31, 2013 to Jan 1, 2014, I do something like:
MyModel.objects.filter(Q(month=12, day=31) | Q(month=1, day=1))
But I get my results in the order:
month = 1, day = 1
month = 12, day = 31
Instead, I would like to get my results in the order:
month = 12, day = 31
month = 1, day = 1
For reasons that would unnecessarily complicate the question, I can't simply partition the query into two queries, one for each year. I would like to make one query and get the results in the desired order. I can reformulate the query, if necessary.
I know that extra should be useful, but I don't quite see how to use it.
Update:
To head a little closer to the intended solution, to impose an absolute ordering I could somehow slip the Julian Day into the results as a "calculated" field, and order by that field. But how to do that?

I have tricky solution Using extra and SQL CASE statement:
start_month = 12
start_day = 31
end_month = 1
end_day = 1
query = (models.MyModel.objects.filter(Q(month=start_month, day=start_day) |
Q(month=end_month, day=end_day))
.extra(select={'order_me': '''CASE WHEN month*31+day < %s*31+%s
THEN (12+month)*31+day
ELSE (month)*31+day
END''' % (start_month, start_day)})
.extra(order_by=['order_me']))
As soon as I have added this order_me field (which is not nice) I think It should be used in predicate instead of Q(...)|Q(...) for date range
query = (models.MyModel.objects.all()
.extra(select={'order_me': """CASE WHEN month*31+day < %s*31+%s
THEN (12+month)*31+day
ELSE (month)*31+day
END""" % (start_month, start_day) })
.extra(order_by=['order_me'])
.extra(where=['order_me < (12 + %s) * 31 + %s' % (end_month,
end_day)]))

Related

Filter Django query by week of month?

In a Django query, how would you filter by a timestamp's week within a month?
There's a built-in week accessor, but that refers to week-of-the-year, e.g. 1-52. As far as I can tell, there's no other built-in option.
The only way I see to do this is to calculate the start and end date range for the week, and then filter on that using the conventional means.
So I'm using a function like:
def week_of_month_date(year, month, week):
"""
Returns the date of the first day in the week of the given date's month,
where Monday is the first day of the week.
e.g. week_of_month_date(year=2022, month=8, week=2) -> date(2022, 8, 7)
"""
assert 1 <= week <= 5
assert 1 <= month <= 12
for i in range(1, 32):
dt = date(year, month, i)
_week = week_of_month(dt)
if _week == week:
return dt
and then to calculate for, say, the 3rd week of July, 2022, I'd do:
start_date = week_of_month_date(2022, 7, 3)
end_date = week_of_month_date(2022, 7, 3) + timedelta(days=7)
qs = MyModel.objects.filter(created__gte=start_date, created__lte=end_date)
Is there an easier or more efficient way to do this with the Django ORM or SQL?
The easiest way to do this using datetime objects is to quite simply subtract the current date weekly year value, with the yearly week value for the 1st day (or 1st week) of the month.
You can use the .isocalendar() function to achieve this:
dt.isocalendar[1] - dt.replace(day=1).isocalendar()[1] + 1
Basically if the week is 46 and that means the first week is week 44 then the resulting output should be 2.
UPDATE
I misunderstood the question, the answer is clear below. However, you may want to consider revising your function based on my above comments.
Come to think of it, if you have a datetime object, you can get the isocalendar week and filter using that like so:
MyModel.objects.filter(created__week=dt.isocalendar()[1])
dt.isocalendar() returns essentially a tuple of 3 integers, [0], is the year, [1], is the iso week (1-52 or 53) and [2], the day of the week (1-7).
As per the docs here:
https://docs.djangoproject.com/en/4.1/ref/models/querysets/#week
There is a built-in filter for isoweek out of the box :)
However, filtering by "week of month" is not possible within the realms of "out of the box".
You might consider writing your own query expression object which accepts an isocalendar object and converts that? But I think you would be better off converting a datetime object and use the isoweek filter.
There's a neat little blog post here to get you started if you really want to do that:
https://dev.to/idrisrampurawala/writing-custom-django-database-functions-4dmb

Get max of column data for entire day 7 days ago

Using pyodbc for python and a postgreSQL database, I am looking to gather the max data of a specific day 7 days ago from the script's run date. The table has the following columns which may prove useful, Timestamp (yyyy-MM-dd hh:mm:ss.ffff), year, month, day.
I've tried the following few
mon = currentDate - dt.timedelta(days=7)
monPeak = cursor.execute("SELECT MAX(total) FROM {} WHERE timestamp = {};".format(tableName, mon)).fetchval()
Error 42883: operator does not exist: timestamp with timezone = integer
monPeak = cursor.execute("SELECT MAX(total) FROM {} WHERE timestamp = NOW() - INTERVAL '7 DAY'".format(tableName)).fetchval()
No error, but value is returned as 'None' (didn't think this was a viable solution anyways because I want the max of the entire day, not that specific time)
I've tried a few different ways of incorporating year, date, and time columns from db table but no good. The goal is to gather the max of data for every day of the prior week. Any ideas?
You have to cast the timestamp to a date if you want to do date comparisons.
WHERE timestamp::date = (NOW() - INTERVAL '7 DAY')::date
Note that timestamptz to date conversions are not immutable, they depend on your current timezone setting.

SQLAlchemy filter by upcoming birthday / annual anniversary

Env: python 3.8, flask-sqlalchemy, postgres
class User(db.Model):
name = db.Column(db.Text)
birthday = db.Column(db.DateTime)
#classmethod
def upcoming_birthdays(cls):
return (cls.query
.filter("??")
.all()
)
I'd like to create a sqlalchemy query that filters users with an upcoming birthday within X number of days. I thought about using the extract function, to extract the month and day from the birthday, but that doesn't work for days at the end of the month or year. I also thought about trying to convert the birthday to a julian date for comparison, but I don't know how to go about that.
For example if today was August 30, 2020 it would return users with birthdays
September 1 1995
August 31 2010 .... etc
Thanks for your help
You can do achieve your goal of having a simple query as below:
q = (
db.session.query(User)
.filter(has_birthday_next_days(User.birthday, 7))
)
This is not a #classmethod on User, but you can transform the solution to one if so you desire.
What is left to do is to actually implement the has_birthday_next_days(...), which is listed below and is mostly the documentation of the principle:
def has_birthday_next_days(sa_col, next_days: int = 0):
"""
sqlalchemy expression to indicate that an sa_col (such as`User.birthday`)
has anniversary within next `next_days` days.
It is implemented by simply checking if the 'age' of the person (in years)
has changed between today and the `next_days` date.
"""
return age_years_at(sa_col, next_days) > age_years_at(sa_col)
There can be multiple implementations of the age_years_at and below is just one possibility, speficic to postgresql (including the required imports):
import datetime
import sqlalchemy as sa
def age_years_at(sa_col, next_days: int = 0):
"""
Generates a postgresql specific statement to return 'age' (in years)'
from an provided field either today (next_days == 0) or with the `next_days` offset.
"""
stmt = func.age(
(sa_col - sa.func.cast(datetime.timedelta(next_days), sa.Interval))
if next_days != 0
else sa_col
)
stmt = func.date_part("year", stmt)
return stmt
Finally, the desired query q = db.session.query(User).filter(has_birthday_next_days(User.birthday, 30)) generates:
SELECT "user".id,
"user".name,
"user".birthday
FROM "user"
WHERE date_part(%(date_part_1)s, age("user".birthday - CAST(%(param_1)s AS INTERVAL)))
> date_part(%(date_part_2)s, age("user".birthday))
{'date_part_1': 'year', 'param_1': datetime.timedelta(days=30), 'date_part_2': 'year'}
Bonus: Having implemented this using generic functions, it can be used not only on the User.birthday column but any other type compatible value. Also the functions can be used separately in both the select and where parts of the statement. For example:
q = (
db.session.query(
User,
age_years_at(User.birthday).label("age_today"),
age_years_at(User.birthday, 7).label("age_in_a_week"),
has_birthday_next_days(User.birthday, 7).label("has_bday_7-days"),
has_birthday_next_days(User.birthday, 30).label("has_bday_30-days"),
)
.filter(has_birthday_next_days(User.birthday, 30))
)
Your original idea about extracting day and month is good.
If you start with e.g. a birthday September 1 1995, you can move that day and month to the current year (i.e. September 1 2020) and afterwards check if that date is within your specified range (between today and today+X days).
As for the days at the end of the year - you can solve that by moving the birthday day and month not only to the current year, but also to the next one.
For example, let's say that today is December 30 2020 and you have a birthday January 2 1995. Then you would check whether either one of the dates January 2 2020 or January 2 2021 are within your specified range.
from datetime import date, timedelta
from sqlalchemy import func,or_
#classmethod
def upcoming_birthdays(cls):
dateFrom = date.today()
dateTo = date.today() + timedelta(days=5)
thisYear = dateFrom.year
nextYear = dateFrom.year + 1
return (cls.query
.filter(
or_(
func.to_date(func.concat(func.to_char(cls.birthday, "DDMM"), thisYear), "DDMMYYYY").between(dateFrom,dateTo),
func.to_date(func.concat(func.to_char(cls.birthday, "DDMM"), nextYear), "DDMMYYYY").between(dateFrom,dateTo)
)
)
.all()
)

Django filter using two conditions

I am developing a web app using django. Among with other tables, I have a table called GeneralContract, which has issueDate and Expiration Date as date fields.
I want to find out the profit of an insurance agent in my case would get from these contracts between a period. For example, if the date range is 15 January 2015 - 25 February 2015 I would like to filter all GeneralContract objects who are issued ANY year between this period.
(i.e. issueDate__month = Date1.month AND IssueDate.day_gte= Date1.day) AND (IssueDate__month = Date2.month AND IssueDate.day_lte = Date2.day) ??
I tried the following but it is not giving me the results I wanted and I am not sure if I am writing this in the correct syntax or if my logic is wrong.
criterion1 = Q(issuedate__month=date1.month)
criterion2 = Q(issuedate__day__gte=date1.day)
criterion3 = Q(issuedate__month=date2.month)
criterion4 = Q(issuedate__day__lte=date2.day)
criterionA = criterion1 & criterion2
criterionB = criterion3 & criterion4
criterionC = criterionA & criterionB
currentGenProfits = GeneralContract.objects.filter(criterionC, cancelled=False)
Is this the right way of doing this filtering logic?
You can't do that because if date1.day = 5 and date2.day = 4 you will have no result, you must check the month and the date together, syntax is right but logic is wrong.
I may suggest to start by taking the biggest set and then apply filtering on it
Start with filtering between the two month and then remove from the queryset objects which are on the first month but before the min day and remove from the queryset objects which are on the last month but after the max day, i think that can do the job.

Django Group By Weekday?

I'm using Django 1.5.1, Python 3.3.x, and can't use raw queries for this.
Is there a way to get a QuerySet grouped by weekday, for a QuerySet that uses a date __range filter? I'm trying to group results by weekday, for a query that ranges between any two dates (could be as much as a year apart). I know how to get rows that match a weekday, but that would require pounding the DB with 7 queries just to find out the data for each weekday.
I've been trying to figure this out for a couple hours by trying different tweaks with the __week_day filter, but nothing's working. Even Googling doesn't help, which makes me wonder if this is even possible. Any Django guru's here know how, if it is possible to do?
Since extra is deprecated, here is a new way of grouping on the day of the week using ExtractDayOfWeek.
from django.db.models.functions import ExtractWeekDay
YourObjects.objects
.annotate(weekday=ExtractWeekDay('timestamp'))
.values('weekday')
.annotate(count=Count('id'))
.values('weekday', 'count')
This will return a result like:
[{'weekday': 1, 'count': 534}, {'weekday': 2, 'count': 574},.......}
It is also important to note that 1 = Sunday and Saturday = 7
Well man I did an algorithm this one brings you all the records since the beginning of the week (Monday) until today
for example if you have a model like this in your app:
from django.db import models
class x(models.Model):
date = models.DateField()
from datetime import datetime
from myapp.models import x
start_date = datetime.date(datetime.now())
week = start_date.isocalendar()[1]
day_week =start_date.isoweekday()
days_quited = 0
less_days = day_week
while less_days != 1:
days_quited += 1
less_days -= 1
week_begin = datetime.date(datetime(start_date.year,start_date.month,start_date.day-days_quited))
records = x.objects.filter(date__range=(week_begin, datetime.date(datetime.now())))
And if you add some records in the admin with a range between June 17 (Monday) and June 22 (today) you will see all those records, and if you add more records with the date of tomorrow for example or with the date of the next Monday you will not see those records.
If you want the records of other week unntil now you only have to put this:
start_date = datetime.date(datetime(year, month, day))
records = x.objects.filter(date__range=(week_begin, datetime.date(datetime.now())))
Hope this helps! :D
You need to add an extra weekday field to the selection, then group by that in the sum or average aggregation. Note that this becomes a database specific query, because the 'extra' notation becomes passed through to the DB select statement.
Given the model:
class x(models.Model):
date = models.DateField()
value = models.FloatField()
Then, for mysql, with a mapping of the ODBC weekday to the python datetime weekday:
x.objects.extra(select={'weekday':"MOD(dayofweek(date)+5,7)"}).values('weekday').annotate(weekday_value=Avg('value'), weekday_value_std=StdDev('value'))
Note that if you do not need to convert the MySql ODBC weekday (1 = Sunday, 2 = Monday...) to python weekday (Monday is 0 and Sunday is 6), then you do not need to do the modulo.
For model like this:
class A(models.Model):
date = models.DateField()
value = models.FloatField()
You can use query:
weekday = {"w": """strftime('%%w', date)"""}
qs = A.objects.extra(select=weekday).values('w').annotate(stat = Sum("value")).order_by()

Categories