The Sqlite documentation states:
SQLite has no DATETIME datatype. Instead, dates and times can be stored in any of these ways:
As a TEXT string in the ISO-8601 format. Example: '2018-04-02 12:13:46'.
As an INTEGER number of seconds since 1970 (also known as "unix time").
...
so I decided to use an INTEGER unix timestamp:
import sqlite3, time
conn = sqlite3.connect(':memory:')
conn.execute("CREATE TABLE data(datetime INTEGER, t TEXT);")
conn.execute("INSERT INTO data VALUES (CURRENT_TIMESTAMP, 'hello')")
Why does the following query return no result?
ts = int(time.time()) + 31*24*3600 # unix timestamp 1 month in the future
print(list(conn.execute("SELECT * FROM data WHERE datetime <= ?", (ts, ))))
More generally, how to do a SELECT query with a comparison with a unix timestamp with Sqlite?
PS:
I have already read SQLite DateTime comparison and similar questions, which offer other comparison methods, but here I'd like to precisely discuss why this unix timestamp comparison does not work.
For performance reasons, I'd like to:
do a query that compares integers (which is super fast if many rows): WHERE datetime <= unix_timestamp,
avoid to convert unix_timestamp into string, and then compare datetime to this string (I guess it'll be far slower)
You use CURRENT_TIMESTAMP when inserting new rows.
This means that in your column the values are not stored as unix timestamps becuase CURRENT_TIMESTAMP returns the current date in the format of YYYY-MM-DD hh:mm:ss.
You can transform the unix timestamp to datetime in the format of YYYY-MM-DD hh:mm:ss with the function datetime() and the unixepoch modifier:
conn.execute("SELECT * FROM data WHERE datetime <= datetime(?, 'unixepoch')", (ts, ))
If your unix timestamp contains milliseconds you must strip them off:
conn.execute("SELECT * FROM data WHERE datetime <= datetime(? / 1000, 'unixepoch')", (ts, ))
Or, you can transform the string datetime in the column datetime to a unix timestamp with the function strftime():
conn.execute("SELECT * FROM data WHERE strftime('%s', datetime) + 0 <= ?", (ts, ))
If you want to store integer values in the column, use strftime() like this:
INSERT INTO data VALUES (strftime('%s', CURRENT_TIMESTAMP) + 0, 'hello')
Related
I am having trouble passing a datetime.time variable into a SQLite database, I have some very basic code here to show what exactly the variable is.
import datetime as dt
time = dt.datetime.now().time()
time = time.strftime('%H:%M')
time = dt.datetime.strptime(time, '%H:%M').time()
print(time)
print(type(time))
time = dt.datetime.now().time() gets the current time in type datetime.time.
Output:
17:34:48.286215
<class 'datetime.time'>
time = time.strftime('%H:%M') is then retrieving just the hour and minute but is of type str
Output:
17:35
<class 'str'>
I then convert it back to a datetime.time with time = dt.datetime.strptime(time, '%H:%M').time() which gives the the output:
17:32:00
<class 'datetime.time'>
The column of type Time accepts the format of HH:SS as shown in the documentation (SQLite3 DateTime Documentation), so I am not sure why I am getting this error:
sqlite3.InterfaceError: Error binding parameter 11 - probably unsupported type.
From this INSERT statement:
cursor.execute("INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price, booking_date, booking_time, ))
EDIT: As requested, here is a snippet of code to recreate the table with the broken columns:
import datetime as dt
import sqlite3
connection = sqlite3.connect("your_database.db")
cursor = connection.cursor()
# Get the current time
time = dt.datetime.now().time()
# Format the time as a string using the '%H:%M' format
time_str = time.strftime('%H:%M')
# Parse the string back to a time object using the '%H:%M' format
time = dt.datetime.strptime(time_str, '%H:%M').time()
# Create the table
cursor.execute("CREATE TABLE test (example_time Time)")
# Insert the time into the example_time column
cursor.execute("INSERT INTO test VALUES (?)", (time, ))
connection.commit()
connection.close()
There is no Date or Time data type in SQLite.
The documentation from the link that you have in your question clearly states that in SQLite you can store datetime in 3 ways: text in ISO-8601 format, integer unix epochs and float julian days.
If you chose the first way then you should pass strings:
booking_date = dt.datetime.now().date().strftime('%Y-%m-%d')
booking_time = dt.datetime.now().time().strftime('%H:%M:00')
sql = "INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?)"
cursor.execute(sql, (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price, booking_date, booking_time))
But, you could also let SQLite get the current date and/or time.
Assuming that in the columns booking_date and booking_time you want the current date and time, you can define these columns as:
booking_date TEXT NOT NULL DEFAULT CURRENT_DATE,
booking_time TEXT NOT NULL DEFAULT CURRENT_TIME
and then you don't need to pass anything for them in the INSERT statement:
sql = "INSERT INTO booked_tickets VALUES (?,?,?,?,?,?,?,?,?,?)"
cursor.execute(sql, (booking_ref, ticket_date, film, showing, ticket_type, num_tickets, cus_name, cus_phone, cus_email, ticket_price,))
Checkout the SQLite datatypes documentation
2.2. Date and Time Datatype
SQLite does not have a storage class set aside for storing dates
and/or times. Instead, the built-in Date And Time Functions of SQLite
are capable of storing dates and times as TEXT, REAL, or INTEGER
values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic
Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can choose to store dates and times in any of these
formats and freely convert between formats using the built-in date and
time functions.
Store the dates as TEXT datatypes.
The documentation you refer to mostly discusses how to format column values that representing dates and times. That is, it discusses what you can do with dates and times that already exist in your database.
It does, however, give just enough information to help you here I think. It says:
Date and time values can be stored as
text in a subset of the ISO-8601 format,
numbers representing the Julian day, or
numbers representing the number of seconds since (or before) 1970-01-01 00:00:00 UTC (the unix timestamp).
So you want to define and supply your dates and times as either full ISO-8601 date strings or as numbers. When defining a table, you indicate which of these formats you wish to use by defining a column type as a STRING, REAL or INTEGER respectively.
Here's some documentation that discusses how to store dates and times in one of these formats: https://www.sqlitetutorial.net/sqlite-date/
The Following function create a table in sqlite3:
def create_table(mycursor):
mycursor.execute('''CREATE TABLE IF NOT EXISTS ch_details(id INTEGER PRIMARY KEY autoincrement,
ch_id TEXT, ch_date DATETIME DEFAULT CURRENT_DATE, ch_no TEXT, cli TEXT, vhs_no TEXT, f_lo TEXT,
t_lo TEXT, qty REAL, adv REAL DEFAULT 0 , oth_de REAL DEFAULT 0, ch_amt REAL DEFAULT 0, mem_id TEXT,
UNIQUE(ch_id));''')
which stores my date in Datetime in ch_date column however when
i try to get the last row of ch_id column in this table stored by providing specific month/year using the following code:
def gen_chid():
dt, mt, yr = cal_gen.get().split("/")
conn = sqlite3.connect('database/u_data.vita')
mycursor = conn.cursor()
mycursor.execute("SELECT ch_id FROM ch_details WHERE strftime('%Y%m', ch_date)",yr, mt)
row = mycursor.fetchone()
The code gets this error:
TypeError: function takes at most 2 arguments (3 given)
The can_gen.get() gets the date from entry box in "07/07/2021" string format
I have also checked this stack answer link but did not get any result.
SQLite's datetime functions like strftime() work only if the datetime values that are passed to them have the format YYYY-MM-DD for dates or YYYY-MM-DD hh:mm:ss for datetimes.
If you stored the dates in the format DD/MM/YYYY then the only way to extract the date parts like day, month or year is by treating the date as a string and use string functions like SUBSTR():
def gen_chid():
conn = sqlite3.connect('database/u_data.vita')
mycursor = conn.cursor()
sql = """
SELECT ch_id
FROM ch_details
WHERE SUBSTR(ch_date, 4) = ?
"""
mycursor.execute(sql, (cal_gen.get()[3:],))
row = mycursor.fetchone()
Here SUBSTR(ch_date, 4) extracts the month/year part in the format MM/YYYY and it is compared to the substring returned from cal_gen.get() after the 3d char which is passed to execute() as the only member of a list.
I have the following dateTime text type variable in Postgres table
"2016-05-12T23:59:11+00:00"
"2016-05-13T11:00:11+00:00"
"2016-05-13T23:59:11+00:00"
"2016-05-15T10:10:11+00:00"
"2016-05-16T10:10:11+00:00"
"2016-05-17T10:10:11+00:00"
I have to write a Python function to extract the data for a few variables between two dates
def fn(dateTime):
df1=pd.DataFrame()
query = """ SELECT "recordId" from "Table" where "dateTime" BETWEEN %s AND %s """ %(dStart,dEnd)
df1=pd.read_sql_query(query1,con=engine)
return df1
I need to create dStart and dEnd variables and use them as function parameters as below
fn('2016-05-12','2016-05-15')
I tried using to_char("dateTime", 'YYYY-MM-DD') Postgres function but didn't work out. Please let me know how to solve this
When working with sql, you should always use your sql library to substitute parameters into the query, instead of using Python's string operators. This avoids the risk of malformed queries or sql injection attacks. See e.g., this page. Right now your code won't run because it directly inserts dStart and dEnd without any quoting, so they are interpreted as mathematical expressions (2016 - 5 - 12 = 1999).
There's also a secondary problem that your query will exclude dateTime values on the end date, because endDate will be treated as having a time value of 00:00:00 when it is compared to dateTime. And if you use to_char() or some other function to extract just the date from the dateTime column to do the comparison, it will prevent your query from using indexes, making it very inefficient.
Here is some revised code that may work for you:
def fn(dStart, dEnd):
query = """
SELECT "recordId"
FROM "Table"
WHERE "dateTime" >= %(start)s AND "dateTime" < %(end)s + interval '1 day'
"""
query_params = {'start': dStart, 'end': dEnd}
df1 = pd.read_sql_query(query1, con=engine, params=query_params)
return df1
This code relies on a few assumptions (welcome to the wonderful world of datetime querying!):
you will pass dStart and dEnd to fn(), instead of just a single dateTime,
the dateTime column is type timestamp with timezone (not text),
the timezones in the dateTime column are correct, and
the dates given by dStart and dEnd are in the server's timezone or you have used SET TIMEZONE ... with your engine object to select the right time zone to use for this session.
Notes
Different database engines use different placeholders for the parameters, so you will need to check your database driver's documentation to decide what placeholders to use. The code above should work fine for postgresql.
With the code above, dStart and dEnd will be inserted into the query as strings, and postgresql automatically convert them into timestamps when it runs the query. This should work fine for the example dates you gave, but if you need more direct control, you have two options:
call fn() with Python date or datetime values for dStart and dEnd, and the code above will insert them into the query as postgresql dates or timestamps; or
explicitly convert the dStart and dEnd strings into postgresql dates by replacing %(start)s and %(end)s with something like this: to_date(%(start)s, 'YYYY-MM-DD').
I'm not familiar with postgresql, but you can convert the strings to the struct_time class which is part of the built in time package in Python and simply make comparisons between them.
import time
time_data = ["2016-05-12T23:59:11+00:00",
"2016-05-13T11:00:11+00:00",
"2016-05-13T23:59:11+00:00",
"2016-05-15T10:10:11+00:00",
"2016-05-16T10:10:11+00:00",
"2016-05-17T10:10:11+00:00"]
def fn(t_init, t_fin, t_all):
# Convert string inputs to struct_time using time.strptime()
t_init, t_fin = [time.strptime(x, '%Y-%m-%d') for x in [t_init, t_fin]]
t_all = [time.strptime(x, '%Y-%m-%dT%H:%M:%S+00:00') for x in time_all]
out = []
for jj in range(len(t_all)):
if t_init < t_all[jj] < t_fin:
out.append(jj)
return out
out = fn('2016-05-12','2016-05-15', time_data)
print(out)
# [0, 1, 2]
The time.strptime routine uses a format specifiers to specify which parts of the string correspond to different time components.
%Y Year with century as a decimal number.
%m Month as a decimal number [01,12].
%d Day of the month as a decimal number [01,31].
%H Hour (24-hour clock) as a decimal number [00,23].
%M Minute as a decimal number [00,59].
%S Second as a decimal number [00,61].
%z Time zone offset from UTC.
%a Locale's abbreviated weekday name.
%A Locale's full weekday name.
%b Locale's abbreviated month name.
%B Locale's full month name.
%c Locale's appropriate date and time representation.
%I Hour (12-hour clock) as a decimal number [01,12].
%p Locale's equivalent of either AM or PM.
I currently store a timestamp like this:
datetime = int(time.mktime(_scheduled_datetime.timetuple()))
> 1172969203.1
I need to then find all objects with a datetime of now().
Unfortunately, my application does not allow me to query ranges, only whole values. Is it possible to store only the days part of a timestamp i.e. 01/01/2016 So I could get all datetime values for today().
you can try:
import time
import datetime
timestamp = int(time.time())
# date timestamp
print int(
time.mktime(
datetime.date.fromtimestamp(timestamp).timetuple()
)
)
# iso format date string
print datetime.date.fromtimestamp(timestamp).isoformat()
I have a date column in a MySQL table. I want to insert a datetime.datetime() object into this column. What should I be using in the execute statement?
I have tried:
now = datetime.datetime(2009,5,5)
cursor.execute("INSERT INTO table
(name, id, datecolumn) VALUES (%s, %s
, %s)",("name", 4,now))
I am getting an error as: "TypeError: not all arguments converted during string formatting"
What should I use instead of %s?
For a time field, use:
import time
time.strftime('%Y-%m-%d %H:%M:%S')
I think strftime also applies to datetime.
You are most likely getting the TypeError because you need quotes around the datecolumn value.
Try:
now = datetime.datetime(2009, 5, 5)
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, '%s')",
("name", 4, now))
With regards to the format, I had success with the above command (which includes the milliseconds) and with:
now.strftime('%Y-%m-%d %H:%M:%S')
Hope this helps.
Try using now.date() to get a Date object rather than a DateTime.
If that doesn't work, then converting that to a string should work:
now = datetime.datetime(2009,5,5)
str_now = now.date().isoformat()
cursor.execute('INSERT INTO table (name, id, datecolumn) VALUES (%s,%s,%s)', ('name',4,str_now))
Use Python method datetime.strftime(format), where format = '%Y-%m-%d %H:%M:%S'.
import datetime
now = datetime.datetime.utcnow()
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, %s)",
("name", 4, now.strftime('%Y-%m-%d %H:%M:%S')))
Timezones
If timezones are a concern, the MySQL timezone can be set for UTC as follows:
cursor.execute("SET time_zone = '+00:00'")
And the timezone can be set in Python:
now = datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc)
MySQL Documentation
MySQL recognizes DATETIME and TIMESTAMP values in these formats:
As a string in either 'YYYY-MM-DD HH:MM:SS' or 'YY-MM-DD HH:MM:SS'
format. A “relaxed” syntax is permitted here, too: Any punctuation
character may be used as the delimiter between date parts or time
parts. For example, '2012-12-31 11:30:45', '2012^12^31 11+30+45',
'2012/12/31 11*30*45', and '2012#12#31 11^30^45' are equivalent.
The only delimiter recognized between a date and time part and a
fractional seconds part is the decimal point.
The date and time parts can be separated by T rather than a space. For
example, '2012-12-31 11:30:45' '2012-12-31T11:30:45' are equivalent.
As a string with no delimiters in either 'YYYYMMDDHHMMSS' or
'YYMMDDHHMMSS' format, provided that the string makes sense as a date.
For example, '20070523091528' and '070523091528' are interpreted as
'2007-05-23 09:15:28', but '071122129015' is illegal (it has a
nonsensical minute part) and becomes '0000-00-00 00:00:00'.
As a number in either YYYYMMDDHHMMSS or YYMMDDHHMMSS format, provided
that the number makes sense as a date. For example, 19830905132800 and
830905132800 are interpreted as '1983-09-05 13:28:00'.
What database are you connecting to? I know Oracle can be picky about date formats and likes ISO 8601 format.
**Note: Oops, I just read you are on MySQL. Just format the date and try it as a separate direct SQL call to test.
In Python, you can get an ISO date like
now.isoformat()
For instance, Oracle likes dates like
insert into x values(99, '31-may-09');
Depending on your database, if it is Oracle you might need to TO_DATE it:
insert into x
values(99, to_date('2009/05/31:12:00:00AM', 'yyyy/mm/dd:hh:mi:ssam'));
The general usage of TO_DATE is:
TO_DATE(<string>, '<format>')
If using another database (I saw the cursor and thought Oracle; I could be wrong) then check their date format tools. For MySQL it is DATE_FORMAT() and SQL Server it is CONVERT.
Also using a tool like SQLAlchemy will remove differences like these and make your life easy.
If you're just using a python datetime.date (not a full datetime.datetime), just cast the date as a string. This is very simple and works for me (mysql, python 2.7, Ubuntu). The column published_date is a MySQL date field, the python variable publish_date is datetime.date.
# make the record for the passed link info
sql_stmt = "INSERT INTO snippet_links (" + \
"link_headline, link_url, published_date, author, source, coco_id, link_id)" + \
"VALUES(%s, %s, %s, %s, %s, %s, %s) ;"
sql_data = ( title, link, str(publish_date), \
author, posted_by, \
str(coco_id), str(link_id) )
try:
dbc.execute(sql_stmt, sql_data )
except Exception, e:
...
dt= datetime.now()
query = """INSERT INTO table1(python_Date_col)
VALUES (%s)
"""
conn = ...... # Connection creating process
cur = conn.cursor()
cur.execute(query,(dt))
Above code will fail as "datetime.now()" produces "datetime.datetime(2014, 2, 11, 1, 16)" as a parameter value to insert statement.
Use the following method to capture the datetime which gives string value.
dt= datetime.now().strftime("%Y%m%d%H%M%S")
I was able to successfully run the code after the change...
for example date is 5/5/22 convert it into mysql date format 2022-05-05 to insert record in mysql database
%m month
%d date
%Y Year of 4 digits
%y 2 digits
Code Below:
from datetime import datetime
now='5/5/22'
print("Before", now)
now= datetime.strptime(dob,'%m/%d/%y').strftime('%Y-%m-%d')
print("After", now)
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, %s)",(name, 4,now))
Output:
Before 5/5/22
After 2022-05-05
(mysql format you can easily insert into database)
when iserting into t-sql
this fails:
select CONVERT(datetime,'2019-09-13 09:04:35.823312',21)
this works:
select CONVERT(datetime,'2019-09-13 09:04:35.823',21)
easy way:
regexp = re.compile(r'\.(\d{6})')
def to_splunk_iso(dt):
"""Converts the datetime object to Splunk isoformat string."""
# 6-digits string.
microseconds = regexp.search(dt).group(1)
return regexp.sub('.%d' % round(float(microseconds) / 1000), dt)