We Keep Coding

pandas read_sql. How to query with where clause of date field

I have a field month-year which is in datetime64[ns] format.
How do i use this field in where clause to get rolling 12 months data(past 12 months data).
Below query does not work, but I would like something that filters data for 12 months.
select * from ABCD.DEFG_TABLE where monthyear > '2019-01-01'
FYI - It is an oracle database. If i can avoid hard coding the value 2019-01-01 that would be great!!

You need to use the datetime and set the date format as below.
Just get your relative date and if you follow datetime format as YYYYMMDD, use strftime from date time with regex string as ("%Y%m%d")
import datetime
import pandas
from dateutil.relativedelta import relativedelta
query = "SELECT * FROM ng_scott.Emp"
between_first = datetime.date.today()
between_second = between_first - relativedelta(years=1)
# GET THE DATASET
dataset = pd.read_sql(query , con=engine)
# PARSE THE DATASET
filtered_dataset = dataset[(dataset['DOJ'] > between_first ) & (dataset['DOJ'] > between_second )]
print(filtered_dataset)

You can do this with pure SQL.
The following expression dynamically computes the beginning of the months 1 year ago:
add_months(trunc(sysdate, 'month'), -12)
This phrases as: take the date at the first day of the current month, and withdraw 12 months from it.
You can just use it as a filter condition:
select * from ABCD.DEFG_TABLE where monthyear > add_months(trunc(sysdate, 'month'), -12)
NB: this assumes that monthyear is of datatype date.

SQLAlchemy - Querying with DateTime columns to filter by month/day/year

I'm building a Flask website that involves keeping track of payments, and I've run into an issue where I can't really seem to filter one of my db models by date.
For instance, if this is what my table looks like:
payment_to, amount, due_date (a DateTime object)
company A, 3000, 7-20-2018
comapny B, 3000, 7-21-2018
company C, 3000, 8-20-2018
and I want to filter it so that I get all rows that's after July 20th, or all rows that are in August, etc.
I can think of a crude, brute-force way to filter all payments and THEN iterate through the list to filter by month/year, but I'd rather stay away from those methods.
This is my payment db model:
class Payment(db.Model, UserMixin):
id = db.Column(db.Integer, unique = True, primary_key = True)
payment_to = db.Column(db.String, nullable = False)
amount = db.Column(db.Float, nullable = False)
due_date = db.Column(db.DateTime, nullable = False, default = datetime.strftime(datetime.today(), "%b %d %Y"))
week_of = db.Column(db.String, nullable = False)
And this is me attempting to filter Payment by date:
Payment.query.filter(Payment.due_date.month == today.month, Payment.due_date.year == today.year, Payment.due_date.day >= today.day).all()
where today is simply datetime.today().
I assumed the due_date column would have all DateTime attributes when I call it (e.g. .month), but it seems I was wrong.
What is the best way to filter the columns of Payment by date? Thank you for your help.

SQLAlchemy effectively translates your query expressed in Python into SQL. But it does that at a relatively superficial level, based on the data type that you assign to the Column when defining your model.
This means that it won't necessarily replicate Python's datetime.datetime API on its DateTime construct - after all, those two classes are meant to do very different things! (datetime.datetime provides datetime functionality to Python, while SQLAlchemy's DateTime tells its SQL-translation logic that it's dealing with a SQL DATETIME or TIMESTAMP column).
But don't worry! There are quite a few different ways for you to do achieve what you're trying to do, and some of them are super easy. The three easiest I think are:
Construct your filter using a complete datetime instance, rather than its component pieces (day, month, year).
Using SQLAlchemy's extract construct in your filter.
Define three hybrid properties in your model that return the payment month, day, and year which you can then filter against.
Filtering on a datetime Object
This is the simplest of the three (easy) ways to achieve what you're trying, and it should also perform the fastest. Basically, instead of trying to filter on each component (day, month, year) separately in your query, just use a single datetime value.
Basically, the following should be equivalent to what you're trying to do in your query above:
from datetime import datetime
todays_datetime = datetime(datetime.today().year, datetime.today().month, datetime.today().day)
payments = Payment.query.filter(Payment.due_date >= todays_datetime).all()
Now, payments should be all payments whose due date occurs after the start (time 00:00:00) of your system's current date.
If you want to get more complicated, like filter payments that were made in the last 30 days. You could do that with the following code:
from datetime import datetime, timedelta
filter_after = datetime.today() - timedelta(days = 30)
payments = Payment.query.filter(Payment.due_date >= filter_after).all()
You can combine multiple filter targets using and_ and or_. For example to return payments that were due within the last 30 days AND were due more than 15 ago, you can use:
from datetime import datetime, timedelta
from sqlalchemy import and_
thirty_days_ago = datetime.today() - timedelta(days = 30)
fifteen_days_ago = datetime.today() - timedelta(days = 15)
# Using and_ IMPLICITLY:
payments = Payment.query.filter(Payment.due_date >= thirty_days_ago,
Payment.due_date <= fifteen_days_ago).all()
# Using and_ explicitly:
payments = Payment.query.filter(and_(Payment.due_date >= thirty_days_ago,
Payment.due_date <= fifteen_days_ago)).all()
The trick here - from your perspective - is to construct your filter target datetime instances correctly before executing your query.
Using the extract Construct
SQLAlchemy's extract expression (documented here) is used to execute a SQL EXTRACT statement, which is how in SQL you can extract a month, day, or year from a DATETIME/TIMESTAMP value.
Using this approach, SQLAlchemy tells your SQL database "first, pull the month, day, and year out of my DATETIME column and then filter on that extracted value". Be aware that this approach will be slower than filtering on a datetime value as described above. But here's how this works:
from sqlalchemy import extract
payments = Payment.query.filter(extract('month', Payment.due_date) >= datetime.today().month,
extract('year', Payment.due_date) >= datetime.today().year,
extract('day', Payment.due_date) >= datetime.today().day).all()
Using Hybrid Attributes
SQLAlchemy Hybrid Attributes are wonderful things. They allow you to transparently apply Python functionality without modifying your database. I suspect for this specific use case they might be overkill, but they are a third way to achieve what you want.
Basically, you can think of hybrid attributes as "virtual columns" that don't actually exist in your database, but which SQLAlchemy can calculate on-the-fly from your database columns when it needs to.
In your specific question, we would define three hybrid properties: due_date_day, due_date_month, due_date_year in your Payment model. Here's how that would work:
... your existing import statements
from sqlalchemy import extract
from sqlalchemy.ext.hybrid import hybrid_property
class Payment(db.Model, UserMixin):
id = db.Column(db.Integer, unique = True, primary_key = True)
payment_to = db.Column(db.String, nullable = False)
amount = db.Column(db.Float, nullable = False)
due_date = db.Column(db.DateTime, nullable = False, default = datetime.strftime(datetime.today(), "%b %d %Y"))
week_of = db.Column(db.String, nullable = False)
#hybrid_property
def due_date_year(self):
return self.due_date.year
#due_date_year.expression
def due_date_year(cls):
return extract('year', cls.due_date)
#hybrid_property
def due_date_month(self):
return self.due_date.month
#due_date_month.expression
def due_date_month(cls):
return extract('month', cls.due_date)
#hybrid_property
def due_date_day(self):
return self.due_date.day
#due_date_day.expression
def due_date_day(cls):
return extract('day', cls.due_date)
payments = Payment.query.filter(Payment.due_date_year >= datetime.today().year,
Payment.due_date_month >= datetime.today().month,
Payment.due_date_day >= datetime.today().day).all()
Here's what the above is doing:
You're defining your Payment model as you already do.
But then you're adding some read-only instance attributes called due_date_year, due_date_month, and due_date_day. Using due_date_year as an example, this is an instance attribute which operates on instances of your Payment class. This means that when you execute one_of_my_payments.due_date_year the property will extract the due_date value from the Python instance. Because this is all happening within Python (i.e. not touching your database) it will operate on the already-translated datetime.datetime object that SQLAlchemy has stored in your instance. And it will return back the result of due_date.year.
Then you're adding a class attribute. This is the bit that is decorated with #due_date_year.expression. This decorator tells SQLAlchemy that when it is translating references to due_date_year into SQL expressions, it should do so as defined in in this method. So the example above tells SQLAlchemy "if you need to use due_date_year in a SQL expression, then extract('year', Payment.due_date) is how due_date_year should be expressed.
(note: The example above assumes due_date_year, due_date_month, and due_date_day are all read-only properties. You can of course define custom setters as well using #due_date_year.setter which accepts arguments (self, value) as well)
In Conclusion
Of these three approaches, I think the first approach (filtering on datetime) is both the easiest to understand, the easiest to implement, and will perform the fastest. It's probably the best way to go. But the principles of these three approaches are very important and I think will help you get the most value out of SQLAlchemy. I hope this proves helpful!

Annotate Django queryset with previous Sunday based on row's date field

I have a table with a set of Orders that my customers made (purchased, to say so).
The customers can choose the delivery date. That value is stored in each Order in a field called Order.delivery_date (not too much inventive there)
class Order(BaseModel):
customer = ForeignKey(Customer, on_delete=CASCADE, related_name='orders')
delivery_date = DateField(null=True, blank=True)
I would like to annotate a queryset that fetches Orders with the previous Sunday for that delivery_date (mostly to create a weekly report, "bucketized" per week)
I thought "Oh! I know! I'll get the date index in the week and I'll subtract a datetime.timedelta with the number of days of that week index, and I'll use that to get the Sunday (like Python's .weekday() function)":
from server.models import *
import datetime
from django.db.models import F, DateField, ExpressionWrapper
from django.db.models.functions import ExtractWeekDay
Order.objects.filter(
delivery_date__isnull=False
).annotate(
sunday=ExpressionWrapper(
F('delivery_date') - datetime.timedelta(days=ExtractWeekDay(F('delivery_date')) + 1),
output_field=DateField()
)
).last().sunday
But if I do that, I get a TypeError: unsupported type for timedelta days component: CombinedExpression when trying to "construct": the timedelta expression.
Not using the F function in the Extract doesn't make a difference either: I get the same error regardless of whether I use Extract(F('delivery_date')) or Extract('delivery_date')
This is a Python 3.4, with Django 2.0.3 over MySQL 5.7.21
I know that I can always fetch the Order object and do this in Python (I even have a little helper function that would do this) but it'd be nice to fetch the objects with that annotation from the DB (and also for learning purposes)
Thank you in advance.

Oh, I had forgotten about extra
It looks like this should do (at least for MySQL)
orders_q = Order.objects.filter(
delivery_date__isnull=False
).extra(
select={
'sunday': "DATE_ADD(`delivery_date`, interval(1 - DAYOFWEEK(`delivery_date`)) DAY)"
},
).order_by('-id')
It seems to work:
for record in orders_q.values('sunday', 'delivery_date'):
print("date: {delivery_date}, sunday: {sunday} is_sunday?: {is_sunday}".format(
is_sunday=record['sunday'].weekday() == 6, **record)
)
date: 2018-06-04, sunday: 2018-06-03 is_sunday?: True
date: 2018-05-30, sunday: 2018-05-27 is_sunday?: True
date: 2018-05-21, sunday: 2018-05-20 is_sunday?: True
date: 2018-06-04, sunday: 2018-06-03 is_sunday?: True
EDIT: Apparently, extra is on its way to being deprecated/unsupported. At the very least, is not very... erm... fondly received by Django developers. Maybe it'd be better using RawSQL instead. Actually, I was having issues trying to do further filter in the sunday annotation using extra which I'm not getting with the RawSQL method..
This seems to work better:
orders_q = orders_q.annotate(
sunday=RawSQL("DATE_ADD(`delivery_date`, interval(1 - DAYOFWEEK(`delivery_date`)) DAY)", ())
)
Which allows me to further annotate...
orders_q.annotate(sunday_count=Count('sunday'))
I'm not sure why, but when I was using extra, I'd get Cannot resolve keyword 'sunday' into field

Querying a DateField with a range of 2 years

This is a pretty straight-forward question:
I have two models, each with a DateField. I want to query Model-A based on the date in Model-B. I want a query that returns all the objects of Model-A that have a date within 2 years, plus or minus, of the date in one object of Model-B. How can this be done?

Assuming you have a date value from model B, calculate two dates: one - 2 years in the past and another - 2 years in the future by the help of python-dateutil module (taken partially from here). Then, use __range notation to filter out A records by date range:
from dateutil.relativedelta import relativedelta
def yearsago(from_date, years):
return from_date - relativedelta(years=years)
b_date = b.my_date
date_min, date_max = yearsago(b_date, 2), yearsago(b_date, -2)
data = A.objects.filter(my_date__range=(date_min, date_max))
where b is a B model instance.
Also see: Django database query: How to filter objects by date range?
Hope that helps.

Django Group By Weekday?

I'm using Django 1.5.1, Python 3.3.x, and can't use raw queries for this.
Is there a way to get a QuerySet grouped by weekday, for a QuerySet that uses a date __range filter? I'm trying to group results by weekday, for a query that ranges between any two dates (could be as much as a year apart). I know how to get rows that match a weekday, but that would require pounding the DB with 7 queries just to find out the data for each weekday.
I've been trying to figure this out for a couple hours by trying different tweaks with the __week_day filter, but nothing's working. Even Googling doesn't help, which makes me wonder if this is even possible. Any Django guru's here know how, if it is possible to do?

Since extra is deprecated, here is a new way of grouping on the day of the week using ExtractDayOfWeek.
from django.db.models.functions import ExtractWeekDay
YourObjects.objects
.annotate(weekday=ExtractWeekDay('timestamp'))
.values('weekday')
.annotate(count=Count('id'))
.values('weekday', 'count')
This will return a result like:
[{'weekday': 1, 'count': 534}, {'weekday': 2, 'count': 574},.......}
It is also important to note that 1 = Sunday and Saturday = 7

Well man I did an algorithm this one brings you all the records since the beginning of the week (Monday) until today
for example if you have a model like this in your app:
from django.db import models
class x(models.Model):
date = models.DateField()
from datetime import datetime
from myapp.models import x
start_date = datetime.date(datetime.now())
week = start_date.isocalendar()[1]
day_week =start_date.isoweekday()
days_quited = 0
less_days = day_week
while less_days != 1:
days_quited += 1
less_days -= 1
week_begin = datetime.date(datetime(start_date.year,start_date.month,start_date.day-days_quited))
records = x.objects.filter(date__range=(week_begin, datetime.date(datetime.now())))
And if you add some records in the admin with a range between June 17 (Monday) and June 22 (today) you will see all those records, and if you add more records with the date of tomorrow for example or with the date of the next Monday you will not see those records.
If you want the records of other week unntil now you only have to put this:
start_date = datetime.date(datetime(year, month, day))
records = x.objects.filter(date__range=(week_begin, datetime.date(datetime.now())))
Hope this helps! :D

You need to add an extra weekday field to the selection, then group by that in the sum or average aggregation. Note that this becomes a database specific query, because the 'extra' notation becomes passed through to the DB select statement.
Given the model:
class x(models.Model):
date = models.DateField()
value = models.FloatField()
Then, for mysql, with a mapping of the ODBC weekday to the python datetime weekday:
x.objects.extra(select={'weekday':"MOD(dayofweek(date)+5,7)"}).values('weekday').annotate(weekday_value=Avg('value'), weekday_value_std=StdDev('value'))
Note that if you do not need to convert the MySql ODBC weekday (1 = Sunday, 2 = Monday...) to python weekday (Monday is 0 and Sunday is 6), then you do not need to do the modulo.

For model like this:
class A(models.Model):
date = models.DateField()
value = models.FloatField()
You can use query:
weekday = {"w": """strftime('%%w', date)"""}
qs = A.objects.extra(select=weekday).values('w').annotate(stat = Sum("value")).order_by()