Get max of column data for entire day 7 days ago - python

Using pyodbc for python and a postgreSQL database, I am looking to gather the max data of a specific day 7 days ago from the script's run date. The table has the following columns which may prove useful, Timestamp (yyyy-MM-dd hh:mm:ss.ffff), year, month, day.
I've tried the following few
mon = currentDate - dt.timedelta(days=7)
monPeak = cursor.execute("SELECT MAX(total) FROM {} WHERE timestamp = {};".format(tableName, mon)).fetchval()
Error 42883: operator does not exist: timestamp with timezone = integer
monPeak = cursor.execute("SELECT MAX(total) FROM {} WHERE timestamp = NOW() - INTERVAL '7 DAY'".format(tableName)).fetchval()
No error, but value is returned as 'None' (didn't think this was a viable solution anyways because I want the max of the entire day, not that specific time)
I've tried a few different ways of incorporating year, date, and time columns from db table but no good. The goal is to gather the max of data for every day of the prior week. Any ideas?


You have to cast the timestamp to a date if you want to do date comparisons.
WHERE timestamp::date = (NOW() - INTERVAL '7 DAY')::date
Note that timestamptz to date conversions are not immutable, they depend on your current timezone setting.

Related

Python/SQLite: Error inserting datetime.time variable into column of type Time

I am having trouble passing a datetime.time variable into a SQLite database, I have some very basic code here to show what exactly the variable is.
import datetime as dt
time = dt.datetime.now().time()
time = time.strftime('%H:%M')
time = dt.datetime.strptime(time, '%H:%M').time()
print(time)
print(type(time))
time = dt.datetime.now().time() gets the current time in type datetime.time.
Output:
17:34:48.286215
<class 'datetime.time'>
time = time.strftime('%H:%M') is then retrieving just the hour and minute but is of type str
Output:
17:35
<class 'str'>
I then convert it back to a datetime.time with time = dt.datetime.strptime(time, '%H:%M').time() which gives the the output:
17:32:00
<class 'datetime.time'>
The column of type Time accepts the format of HH:SS as shown in the documentation (SQLite3 DateTime Documentation), so I am not sure why I am getting this error:
sqlite3.InterfaceError: Error binding parameter 11 - probably unsupported type.
From this INSERT statement:
cursor.execute("INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price, booking_date, booking_time, ))
EDIT: As requested, here is a snippet of code to recreate the table with the broken columns:
import datetime as dt
import sqlite3
connection = sqlite3.connect("your_database.db")
cursor = connection.cursor()
# Get the current time
time = dt.datetime.now().time()
# Format the time as a string using the '%H:%M' format
time_str = time.strftime('%H:%M')
# Parse the string back to a time object using the '%H:%M' format
time = dt.datetime.strptime(time_str, '%H:%M').time()
# Create the table
cursor.execute("CREATE TABLE test (example_time Time)")
# Insert the time into the example_time column
cursor.execute("INSERT INTO test VALUES (?)", (time, ))
connection.commit()
connection.close()

There is no Date or Time data type in SQLite.
The documentation from the link that you have in your question clearly states that in SQLite you can store datetime in 3 ways: text in ISO-8601 format, integer unix epochs and float julian days.
If you chose the first way then you should pass strings:
booking_date = dt.datetime.now().date().strftime('%Y-%m-%d')
booking_time = dt.datetime.now().time().strftime('%H:%M:00')
sql = "INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?)"
cursor.execute(sql, (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price, booking_date, booking_time))
But, you could also let SQLite get the current date and/or time.
Assuming that in the columns booking_date and booking_time you want the current date and time, you can define these columns as:
booking_date TEXT NOT NULL DEFAULT CURRENT_DATE,
booking_time TEXT NOT NULL DEFAULT CURRENT_TIME
and then you don't need to pass anything for them in the INSERT statement:
sql = "INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?)"
cursor.execute(sql, (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price,))

Checkout the SQLite datatypes documentation
2.2. Date and Time Datatype
SQLite does not have a storage class set aside for storing dates
and/or times. Instead, the built-in Date And Time Functions of SQLite
are capable of storing dates and times as TEXT, REAL, or INTEGER
values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic
Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can choose to store dates and times in any of these
formats and freely convert between formats using the built-in date and
time functions.
Store the dates as TEXT datatypes.

The documentation you refer to mostly discusses how to format column values that representing dates and times. That is, it discusses what you can do with dates and times that already exist in your database.
It does, however, give just enough information to help you here I think. It says:
Date and time values can be stored as
text in a subset of the ISO-8601 format,
numbers representing the Julian day, or
numbers representing the number of seconds since (or before) 1970-01-01 00:00:00 UTC (the unix timestamp).
So you want to define and supply your dates and times as either full ISO-8601 date strings or as numbers. When defining a table, you indicate which of these formats you wish to use by defining a column type as a STRING, REAL or INTEGER respectively.
Here's some documentation that discusses how to store dates and times in one of these formats: https://www.sqlitetutorial.net/sqlite-date/

Date selection using Selenium

So I'm building a program that extracts data from several different websites that will display all our business data on a dashboard.
I've managed to get to the point where I can extract the data I need, and have it set up to select todays data, previous week and month to date by importing datetime, and for the example of previous week the following:
date = datetime.date.today()
yesterday = (date - datetime.timedelta(days = 1)).strftime('%d')
lastWeek = (date - datetime.timedelta(days = 7)).strftime('%d')
today = date.strftime('%d')
Then just using the find element function:
browser.find_element_by_link_text(lastWeek).click()
However I'm running into problems at the beginning of the month, where I can't subtract 7 as it would take me into negative numbers and wouldn't go to the previous months date as each month has different days etc.
Is there any way of getting around this?
TIA

Filter a DataFrame By Date One Week Previous From Current Date

I am reading an .xlsx spreadsheet into a Pandas DataFrame so that I can remove duplicate rows based on all columns and export the DataFrame into a .csv. One of the columns is a date column formatted as MM/DD/YY.
Here is a sample of the unaltered data
This spreadsheet contains abnormal pay hours entries for a payroll that is payed every Friday based on hours from one week previous to the current week. Rows are added each day there is an abnormal function with that day's data. I want to tell pandas to only find duplicates in rows whose date is less than or equal to the Friday date one week previous from the current Friday (This script will only be ran on Fridays). For example, if today is Friday 12/7/18, I want to set a cutoff date of the previous Friday, 11/30/18, and only look at rows whose dates are on or before 11/30/18. How can I trim the DataFrame in this way before executing drop_duplicates?

you can use date and timedelta.
get todays date.
store the date one week from todays date.
filter your data (I'm not sure how you have it stored, but I used generic names)
from datetime import date, timedelta
today = date.today()
week_prior = today - timedelta(weeks=1)
df_last_week = df[df['date'] <= week_prior]

Note that using a fixed time window of 1 week (or 7 days) is fine if you are sure that your script will only ever be run on a Friday.
You can, of course programatically get the date of last Friday, and filter your dataframe on that date:
last_friday = datetime.now().date() - timedelta(days=datetime.now().weekday()) + timedelta(days=4, weeks=-1)
print(df[df['date'] <= pd.Timestamp(last_friday)])

Django: Why Filtering By Year Works, But Not Month and Day?

I have a simple datetime field named date. Stores dates and time like this: 2015-07-04 01:40:00+00:00.
When I run
today = datetime.datetime.now().today()
x = game.objects.filter(date__year=today.year)
It works, however if i run
today = datetime.datetime.now().today()
x = game.objects.filter(date__month=today.month)
I get an empty list.

Current month is July. If you filter by the year part of the date being equal to current month, then you search for records being in the year of 07 AD. I believe you do not have such records. You need to filter by month of the date if you want to filter by month.

Python Count number of records within a given date range

We have a backend table that stores details of transaction including seconds since epoch. I am creating a UI where I collect from-to dates to display counts of transaction occurred in-between the dates.
Assuming that the date range is from 07/01/2012 - 07/30/2012, I am unable to establish a logic that will increment a counter for records that happened within the time period. I should hit the DB only once as hitting for each day will give poor performance.
I am stuck at a logic:
Convert 07/01/2012 & 07/30/2012 to seconds since epoch.
Get the records for start date - end date [as converted to seconds since epoch]
For each record get the month / date
-- now how will we add counters for each date in between 07/01/2012 - 07/30/2012

MySQL has the function FROM_UNIXTIME which will convert your seconds since epoch into datetime and you can then extract the DATE part of it (YYYY-MM-DD format) and group according to it.
SELECT DATE(FROM_UNIXTIME(timestamp_column)), COUNT(*)
FROM table_name
GROUP BY DATE(FROM_UNIXTIME(timestamp_column))
This will return something like
2012-07-01 2
2012-07-03 4
…
(no entries for days without transactions)

Categories