We Keep Coding

Python/SQLite: Error inserting datetime.time variable into column of type Time

I am having trouble passing a datetime.time variable into a SQLite database, I have some very basic code here to show what exactly the variable is.
import datetime as dt
time = dt.datetime.now().time()
time = time.strftime('%H:%M')
time = dt.datetime.strptime(time, '%H:%M').time()
print(time)
print(type(time))
time = dt.datetime.now().time() gets the current time in type datetime.time.
Output:
17:34:48.286215
<class 'datetime.time'>
time = time.strftime('%H:%M') is then retrieving just the hour and minute but is of type str
Output:
17:35
<class 'str'>
I then convert it back to a datetime.time with time = dt.datetime.strptime(time, '%H:%M').time() which gives the the output:
17:32:00
<class 'datetime.time'>
The column of type Time accepts the format of HH:SS as shown in the documentation (SQLite3 DateTime Documentation), so I am not sure why I am getting this error:
sqlite3.InterfaceError: Error binding parameter 11 - probably unsupported type.
From this INSERT statement:
cursor.execute("INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price, booking_date, booking_time, ))
EDIT: As requested, here is a snippet of code to recreate the table with the broken columns:
import datetime as dt
import sqlite3
connection = sqlite3.connect("your_database.db")
cursor = connection.cursor()
# Get the current time
time = dt.datetime.now().time()
# Format the time as a string using the '%H:%M' format
time_str = time.strftime('%H:%M')
# Parse the string back to a time object using the '%H:%M' format
time = dt.datetime.strptime(time_str, '%H:%M').time()
# Create the table
cursor.execute("CREATE TABLE test (example_time Time)")
# Insert the time into the example_time column
cursor.execute("INSERT INTO test VALUES (?)", (time, ))
connection.commit()
connection.close()

There is no Date or Time data type in SQLite.
The documentation from the link that you have in your question clearly states that in SQLite you can store datetime in 3 ways: text in ISO-8601 format, integer unix epochs and float julian days.
If you chose the first way then you should pass strings:
booking_date = dt.datetime.now().date().strftime('%Y-%m-%d')
booking_time = dt.datetime.now().time().strftime('%H:%M:00')
sql = "INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?)"
cursor.execute(sql, (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price, booking_date, booking_time))
But, you could also let SQLite get the current date and/or time.
Assuming that in the columns booking_date and booking_time you want the current date and time, you can define these columns as:
booking_date TEXT NOT NULL DEFAULT CURRENT_DATE,
booking_time TEXT NOT NULL DEFAULT CURRENT_TIME
and then you don't need to pass anything for them in the INSERT statement:
sql = "INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?)"
cursor.execute(sql, (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price,))

Checkout the SQLite datatypes documentation
2.2. Date and Time Datatype
SQLite does not have a storage class set aside for storing dates
and/or times. Instead, the built-in Date And Time Functions of SQLite
are capable of storing dates and times as TEXT, REAL, or INTEGER
values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic
Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can choose to store dates and times in any of these
formats and freely convert between formats using the built-in date and
time functions.
Store the dates as TEXT datatypes.

The documentation you refer to mostly discusses how to format column values that representing dates and times. That is, it discusses what you can do with dates and times that already exist in your database.
It does, however, give just enough information to help you here I think. It says:
Date and time values can be stored as
text in a subset of the ISO-8601 format,
numbers representing the Julian day, or
numbers representing the number of seconds since (or before) 1970-01-01 00:00:00 UTC (the unix timestamp).
So you want to define and supply your dates and times as either full ISO-8601 date strings or as numbers. When defining a table, you indicate which of these formats you wish to use by defining a column type as a STRING, REAL or INTEGER respectively.
Here's some documentation that discusses how to store dates and times in one of these formats: https://www.sqlitetutorial.net/sqlite-date/

How to convert datetime.timedelta to number? Field 'days' expected a number but got datetime.timedelta

My code:
data.days=(form.cleaned_data['checkout'] - form.cleaned_data['checkin'])
The error message:
Field 'days' expected a number but got datetime.timedelta

timedelta has a days property you can use:
data.days = (form.cleaned_data['checkout'] - form.cleaned_data['checkin']).days
# Here -------------------------------------------------------------------^

datetime.timedelta is an object instantiated, besides the standard constructor way, as a result of the difference between two datetime objects. It represents a time duration expressed in days, seconds and microseconds.
timedelta also provides a set of properties to access those values, namely:
timedelta.days
timedelta.seconds
timedelta.microseconds
the first one being the one you're searching for.

Querying postgresql from Python for DateTime values between two dates

I have the following dateTime text type variable in Postgres table
"2016-05-12T23:59:11+00:00"
"2016-05-13T11:00:11+00:00"
"2016-05-13T23:59:11+00:00"
"2016-05-15T10:10:11+00:00"
"2016-05-16T10:10:11+00:00"
"2016-05-17T10:10:11+00:00"
I have to write a Python function to extract the data for a few variables between two dates
def fn(dateTime):
df1=pd.DataFrame()
query = """ SELECT "recordId" from "Table" where "dateTime" BETWEEN %s AND %s """ %(dStart,dEnd)
df1=pd.read_sql_query(query1,con=engine)
return df1
I need to create dStart and dEnd variables and use them as function parameters as below
fn('2016-05-12','2016-05-15')
I tried using to_char("dateTime", 'YYYY-MM-DD') Postgres function but didn't work out. Please let me know how to solve this

When working with sql, you should always use your sql library to substitute parameters into the query, instead of using Python's string operators. This avoids the risk of malformed queries or sql injection attacks. See e.g., this page. Right now your code won't run because it directly inserts dStart and dEnd without any quoting, so they are interpreted as mathematical expressions (2016 - 5 - 12 = 1999).
There's also a secondary problem that your query will exclude dateTime values on the end date, because endDate will be treated as having a time value of 00:00:00 when it is compared to dateTime. And if you use to_char() or some other function to extract just the date from the dateTime column to do the comparison, it will prevent your query from using indexes, making it very inefficient.
Here is some revised code that may work for you:
def fn(dStart, dEnd):
query = """
SELECT "recordId"
FROM "Table"
WHERE "dateTime" >= %(start)s AND "dateTime" < %(end)s + interval '1 day'
"""
query_params = {'start': dStart, 'end': dEnd}
df1 = pd.read_sql_query(query1, con=engine, params=query_params)
return df1
This code relies on a few assumptions (welcome to the wonderful world of datetime querying!):
you will pass dStart and dEnd to fn(), instead of just a single dateTime,
the dateTime column is type timestamp with timezone (not text),
the timezones in the dateTime column are correct, and
the dates given by dStart and dEnd are in the server's timezone or you have used SET TIMEZONE ... with your engine object to select the right time zone to use for this session.
Notes
Different database engines use different placeholders for the parameters, so you will need to check your database driver's documentation to decide what placeholders to use. The code above should work fine for postgresql.
With the code above, dStart and dEnd will be inserted into the query as strings, and postgresql automatically convert them into timestamps when it runs the query. This should work fine for the example dates you gave, but if you need more direct control, you have two options:
call fn() with Python date or datetime values for dStart and dEnd, and the code above will insert them into the query as postgresql dates or timestamps; or
explicitly convert the dStart and dEnd strings into postgresql dates by replacing %(start)s and %(end)s with something like this: to_date(%(start)s, 'YYYY-MM-DD').

I'm not familiar with postgresql, but you can convert the strings to the struct_time class which is part of the built in time package in Python and simply make comparisons between them.
import time
time_data = ["2016-05-12T23:59:11+00:00",
"2016-05-13T11:00:11+00:00",
"2016-05-13T23:59:11+00:00",
"2016-05-15T10:10:11+00:00",
"2016-05-16T10:10:11+00:00",
"2016-05-17T10:10:11+00:00"]
def fn(t_init, t_fin, t_all):
# Convert string inputs to struct_time using time.strptime()
t_init, t_fin = [time.strptime(x, '%Y-%m-%d') for x in [t_init, t_fin]]
t_all = [time.strptime(x, '%Y-%m-%dT%H:%M:%S+00:00') for x in time_all]
out = []
for jj in range(len(t_all)):
if t_init < t_all[jj] < t_fin:
out.append(jj)
return out
out = fn('2016-05-12','2016-05-15', time_data)
print(out)
# [0, 1, 2]
The time.strptime routine uses a format specifiers to specify which parts of the string correspond to different time components.
%Y Year with century as a decimal number.
%m Month as a decimal number [01,12].
%d Day of the month as a decimal number [01,31].
%H Hour (24-hour clock) as a decimal number [00,23].
%M Minute as a decimal number [00,59].
%S Second as a decimal number [00,61].
%z Time zone offset from UTC.
%a Locale's abbreviated weekday name.
%A Locale's full weekday name.
%b Locale's abbreviated month name.
%B Locale's full month name.
%c Locale's appropriate date and time representation.
%I Hour (12-hour clock) as a decimal number [01,12].
%p Locale's equivalent of either AM or PM.

Why does MYSQL DB return a corrupted value when averaging over a Django models.DateTimeField?

I'm running a Django application on top of a MySQL (actually MariaDB) database.
My Django Model looks like this:
from django.db import models
from django.db.models import Avg, Max, Min, Count
class myModel(models.Model):
my_string = models.CharField(max_length=32,)
my_date = models.DateTimeField()
#staticmethod
def get_stats():
logger.info(myModel.objects.values('my_string').annotate(
count=Count("my_string"),
min=Min('my_date'),
max=Max('my_date'),
avg=Avg('my_date'),
)
)
When I run get_stats(), I get the following log line:
[2015-06-21 09:45:40] INFO [all_logs:96] [{'my_string': u'A', 'count': 2, 'avg': 20080507582679.5, 'min': datetime.datetime(2007, 8, 2, 11, 33, 53, tzinfo=<UTC>), 'max': datetime.datetime(2009, 2, 13, 5, 20, 6, tzinfo=<UTC>)}]
The problem I have with this is that the average of the my_date field returned by the database is: 20080507582679.5. Look carefully at that number. It is an invalid date format.
Why doesn't the database return a valid value for the average of these two dates? How do I get the actual average of this field if the way described fails? Is Django DateTimeField not setup to do handle averaging?

Q1: Why doesn't the database return a valid value for the average of these two dates?
A: The value returned is expected, it's well defined MySQL behavior.
MySQL automatically converts a date or time value to a number if the value is used in a numeric context and vice versa.
MySQL Reference Manual: https://dev.mysql.com/doc/refman/5.5/en/date-and-time-types.html
In MySQL, the AVG aggregate function operates on numeric values.
In MySQL, a DATE or DATETIME expression can be evaluated in a numeric context.
As a simple demonstration, performing an numeric addition operation on a DATETIME implicitly converts the datetime value into a number. This query:
SELECT NOW(), NOW()+0
returns a result like:
NOW() NOW()+0
------------------- -----------------------
2015-06-23 17:57:48 20150623175748.000000
Note that the value returned for the expression NOW()+0 is not a DATETIME, it's a number.
When you specify a SUM() or AVG() function on a DATETIME expression, that's equivalent to converting the DATETIME into a number, and then summing
or averaging the number.
That is, the return from this expression AVG(mydatetimecol) is equivalent to the return from this expression: AVG(mydatetimecol+0)
What is being "averaged" is a numeric value. And you have observed, the value returned is not a valid datetime; and even in cases where it happens to look like a valid datetime, it's likely not a value you would consider a true "average".
Q2: How do I get the actual average of this field if the way described fails?
A2: One way to do that is to convert the datetime into a numeric value that can be "accurately" averaged, and then convert that back into a datetime.
For example, you could convert the datetime into a numeric value representing a number of seconds from some fixed point in time, e.g.
TIMESTAMPDIFF(SECOND,'2015-01-01',t.my_date)
You could then "average" those values, to get an average number of seconds from a fixed point in time. (NOTE: beware of adding up an extremely large number of rows, with extremely large values, and exceeding the limit (maximum numeric value), numeric overflow issues.)
AVG(TIMESTAMPDIFF(SECOND,'2015-01-01',t.my_date))
To convert that back to a datetime, add that value as a number of seconds back to a the fixed point in time:
'2015-01-01' + INTERVAL AVG(TIMESTAMPDIFF(SECOND,'2015-01-01',t.my_date)) SECOND
(Note that the DATEIME values are evaluated in the timezone of the MySQL session; so there are edge cases where the setting of the time_zone variable in the MySQL session will have some influence on the value returned.)
MySQL also provides a UNIX_TIMESTAMP() function which returns a unix-style integer value, number of seconds from the beginning of the era (midnight Jan. 1, 1970 UTC). You can use that to accomplish the same operation more concisely:
FROM_UNIXTIME(AVG(UNIX_TIMESTAMP(t.my_date)))
Note that this final expression is really doing the same thing... converting the datetime value into a number of seconds since '1970-01-01 00:00:00' UTC, taking a numeric average of that, and then adding that average number of seconds back to '1970-01-01' UTC, and finally converting that back to a DATETIME value, represented in the current session time_zone.
Q3: Is Django DateTimeField not setup to do handle averaging?
A: Apparently, the authors of Django are satisfied with the value returned from the database for a SQL expression AVG(datetime).

Plan A: Use a TIMESTAMP field instead of a DATETIME field
Plan B: Convert DATETIME to TIMESTAMP during the computation:
FROM_UNIXTIME(ROUND(AVG(UNIX_TIMESTAMP(`my_date`))))
(Sorry, I don't know the Django syntax needed.)

When you use values(), Django will not convert the value it got from the database-python connector. It's up to the connector to determine how the value is returned.
In this case, it seems that the MySQL connector returns a string-representation with the separators removed. You can try to use datetime.strptime() with a matching format to parse it into a datetime object.

How to convert dateutil.relativedelta object to datetime.timedelta object?

How can I convert a dateutil.relativedelta object to a datetime.timedelta object?
e.g.,
# pip install python-dateutil
from dateutil.relativedelta import relativedelta
from datetime import timedelta
rel_delta = relativedelta(months=-2)
# How can I convert rel_delta to a timedelta object so that I can call total_seconds() ?
time_delta = ???(rel_delta)
time_delta.total_seconds() # call the timedelta.total_seconds() method

You can't, for one huge reason: They don't store the same information. datetime.timedelta only stores days, seconds, and milliseconds, whereas dateutil.relativedelta stores every single time component fed to it.
That dateutil.relativedelta does so is important for storing things such as a difference of 1 month, but since the length of a month can vary this means that there is no way at all to express the same thing in datetime.timedelta.

In case someone is looking to convert a relativedelta to a timedelta from a specific date, simply add and subtract the known time:
utcnow = datetime.utcnow()
rel_delta = relativedelta(months=-2)
time_delta = utcnow + rel_delta - utcnow # e.g, datetime.timedelta(days=-62)
As a commenter points out, the resulting timedelta value will differ based on what month it is.

Depending on why you want to call total_seconds, it may be possible to refactor your code to avoid the conversion altogether. For example, consider a check on whether or not a user is over 18 years old:
datetime.date.today() - user['dateOfBirth'] < datetime.timedelta(days=365*18)
This check is not a good idea, because the timedelta object does not account for things like leap years. It's tempting to rewrite as:
datetime.date.today() - user['dateOfBirth'] < dateutil.relativedelta.relativedelta(years=18)
which would require comparing a timedelta (LHS) to a relativedelta (RHS), or converting one to the other. However, you can refactor the check to avoid this conversion altogether:
user['dateOfBirth'] + dateutil.relativedelta.relativedelta(years=18) > datetime.date.today()