ValueError when querying dates with SQLite - python

A simple example:
import sqlite3, datetime, csv
import pandas.io.sql as sql
from dateutil.parser import parse
my_db = 'test_db.db'
connection=sqlite3.connect(my_db,detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
cursor = connection.cursor()
cursor.execute('''CREATE TABLE test_table (Id INTEGER PRIMARY KEY, Date DATE);''')
date_str = '1/1/2011'
date_parsed = parse(date_str)
cursor.execute('INSERT into test_table (Id, Date) values(?,?)',(1,date_parsed))
cursor.execute('SELECT * FROM test_table')
yields:
ValueError: invalid literal for int() with base 10: '01 00:00:00'
I'm simply trying to have the SQL db return my dates in datetime format so I can then perform filtered queries on them.
I've already read this related post for reference:
SQLite date storage and conversion

Inside your CREATE TABLE, you're using Date DATE
CREATE TABLE test_table (Id INTEGER PRIMARY KEY, Date DATE)
The problem with this is that that tries to map to datetime.date which your format isn't, if you change your format to use a TIMESTAMP, then it works correctly and tries to make it a datetime.datetime...
CREATE TABLE test_table (Id INTEGER PRIMARY KEY, Date TIMESTAMP)

Related

How to select dates in SQLite with python

Im trying to query a table, and need to grab all products that have a date = today date.
Below is my code so far
import sqlite3
from datetime import date
date = date.today()
con = sqlite3.connect('test.db')
cur = con.cursor()
date = date.today()
sql_q = f'''SELECT date, name FROM table WHERE date = {date}'''
table = cur.execute(sql_q)
for row in table:
print(row)
i am using an SQlite 3 db and all data has been entered with the following format:
2022-09-20
However this variable type does not seem to work with SQL.
i know the SQL code should look somthing like this
SELECT name FROM test WHERE date = '2022-09-20'
but i'd like the date to be selected automatically from python rather than typing it in manually.
Use the function date() to get the current date:
SELECT name FROM test WHERE date = date()
or CURRENT_DATE:
SELECT name FROM test WHERE date = CURRENT_DATE
I think you need to convert date to string and then pass it in query.
maybe your datatype of column and date.today() is different.
date = date.strftime('%Y-%m-%d')
try using this.

Sqlite3 search column by providing Year and month

The Following function create a table in sqlite3:
def create_table(mycursor):
mycursor.execute('''CREATE TABLE IF NOT EXISTS ch_details(id INTEGER PRIMARY KEY autoincrement,
ch_id TEXT, ch_date DATETIME DEFAULT CURRENT_DATE, ch_no TEXT, cli TEXT, vhs_no TEXT, f_lo TEXT,
t_lo TEXT, qty REAL, adv REAL DEFAULT 0 , oth_de REAL DEFAULT 0, ch_amt REAL DEFAULT 0, mem_id TEXT,
UNIQUE(ch_id));''')
which stores my date in Datetime in ch_date column however when
i try to get the last row of ch_id column in this table stored by providing specific month/year using the following code:
def gen_chid():
dt, mt, yr = cal_gen.get().split("/")
conn = sqlite3.connect('database/u_data.vita')
mycursor = conn.cursor()
mycursor.execute("SELECT ch_id FROM ch_details WHERE strftime('%Y%m', ch_date)",yr, mt)
row = mycursor.fetchone()
The code gets this error:
TypeError: function takes at most 2 arguments (3 given)
The can_gen.get() gets the date from entry box in "07/07/2021" string format
I have also checked this stack answer link but did not get any result.
SQLite's datetime functions like strftime() work only if the datetime values that are passed to them have the format YYYY-MM-DD for dates or YYYY-MM-DD hh:mm:ss for datetimes.
If you stored the dates in the format DD/MM/YYYY then the only way to extract the date parts like day, month or year is by treating the date as a string and use string functions like SUBSTR():
def gen_chid():
conn = sqlite3.connect('database/u_data.vita')
mycursor = conn.cursor()
sql = """
SELECT ch_id
FROM ch_details
WHERE SUBSTR(ch_date, 4) = ?
"""
mycursor.execute(sql, (cal_gen.get()[3:],))
row = mycursor.fetchone()
Here SUBSTR(ch_date, 4) extracts the month/year part in the format MM/YYYY and it is compared to the substring returned from cal_gen.get() after the 3d char which is passed to execute() as the only member of a list.

sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type when inserting date and time

I am trying to insert date and time into a sqlite table.
Here is my code.
import sqlite3
from datetime import datetime
### Date Time ###
dt = datetime.now()
dates = dt.date()
times = dt.time()
def sql(date, time):
### CREATE DB
con = sqlite3.connect("date.db")
cur = con.cursor()
## CREATE TABLE
cur.execute("CREATE TABLE if NOT EXISTS d_t (datee, timee)")
con.commit()
## INSERT DATA
cur.execute("INSERT INTO d_t (datee, timee) VALUES (?,?)", (date, time))
con.commit()
## VIEW DATA
cur.execute("SELECT * from d_t")
row = cur.fetchall()
print(type((row[0][0]))) # Printing_Date_only
sql(dates, times)
But this is the error I am getting:
Traceback (most recent call last):
File "C:\Users\Hridoy\Documents\GitHub\Covid19\datedb.py", line 26, in <module>
sql(dates, times)
File "C:\Users\Hridoy\Documents\GitHub\Covid19\datedb.py", line 18, in sql
cur.execute("INSERT INTO d_t (datee, timee) VALUES (?,?)", (date, time))
sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.
I don't want to insert the date and time as strings because I need to compare two dates later, and one of them will come from the database.
Kindly looking for solution of this.
Working with SQLite can be a little tricky when trying to store dates/times, and I don't think you can store a time (without a date). Why not just store the date and time together as the complete datetime value?
The below code stores the datetime value and modifies the sqlite3.connect() call to make dealing with datetimes nicer (we'll get a datetime back when we query the database). We also need to specify the type of the d_t table's "date_time" column as a SQLite timestamp when we create the table.
import sqlite3
from datetime import datetime
### Date Time ###
dt = datetime.now()
def sql(dt_value):
### CREATE DB
con = sqlite3.connect("date.db", detect_types=sqlite3.PARSE_DECLTYPES)
cur = con.cursor()
## CREATE TABLE
cur.execute("CREATE TABLE if NOT EXISTS d_t (date_time timestamp)")
con.commit()
## INSERT DATA
cur.execute("INSERT INTO d_t (date_time) VALUES (?)", (dt_value,))
con.commit()
## VIEW DATA
cur.execute("SELECT * from d_t")
row = cur.fetchall()
print(type(row[0][0])) # Print type of datetime
print(row[0][0]) # Print datetime
print(row[0][0].date()) # Print date
print(row[0][0].time()) # Print time
sql(dt)
As an alternative solution, you may also want to try the "easy_db" library I wrote to help out with this kind of problem. Just install it with pip:
pip install easy_db
Then we can accomplish the same task with less/cleaner code.
import easy_db
from datetime import datetime
### Date Time ###
dt = datetime.now()
def insert_and_read_datetime(dt_value):
# Create and connect to SQLite database
db = easy_db.DataBase("date.db")
# Create "d_t" table and add one row of data to it
# From our input dictionary, a "date" column is automatically created and
# this column is given the SQLite timestamp type based on the type of dt_value
db.append("d_t", {"date": dt_value})
# Run a SELECT * query to pull the full "d_t" table
# Returned data is a list with a dictionary for each row
data = db.pull("d_t")
print(type(data[0]["date"])) # Print type of datetime
print(data[0]["date"]) # Print datetime
print(data[0]["date"].date()) # Print date
print(data[0]["date"].time()) # Print time
insert_and_read_datetime(dt)
Best of luck with your Covid-19 project!

inserting date time with microsecond into SQL Server from python

Below code works perfectly:
import pypyodbc
import datetime
connection = pypyodbc.connect('Driver={SQL Server};'
'Server=some_server;'
'Database=some_db')
cur = connection.cursor()
some_time = datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
query = "insert into table_a (date_created) values ('"+some_time+"')"
cur.execute(query)
connection.commit()
connection.close()
But if I change (adding microseconds to date)
some_time = datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')
it generates error:
DataError: ('22007', '[22007] [Microsoft][ODBC SQL Server Driver][SQL Server]Conversion failed when converting date and/or time from character string.')
date_created column is of datetime type and does display microseconds.
Any thoughts?
SQL Server datetime columns are only able to store fractional seconds to millisecond precision (3 decimal places). When you do
some_time = datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')
you get a string formatted to the precision of Python's datetime, which is microseconds (6 decimal places)
>>> some_time
'2018-09-28 16:38:14.344801'
and SQL server doesn't like the extra three decimal places.
The solution is to not format the datetime as a string, just pass the datetime value itself in a proper parameterized query
query = "insert into table_a (date_created) values (?)"
params = (datetime.datetime.utcnow(), )
cur.execute(query, params)

Inserting a Python datetime.datetime object into MySQL

I have a date column in a MySQL table. I want to insert a datetime.datetime() object into this column. What should I be using in the execute statement?
I have tried:
now = datetime.datetime(2009,5,5)
cursor.execute("INSERT INTO table
(name, id, datecolumn) VALUES (%s, %s
, %s)",("name", 4,now))
I am getting an error as: "TypeError: not all arguments converted during string formatting"
What should I use instead of %s?
For a time field, use:
import time
time.strftime('%Y-%m-%d %H:%M:%S')
I think strftime also applies to datetime.
You are most likely getting the TypeError because you need quotes around the datecolumn value.
Try:
now = datetime.datetime(2009, 5, 5)
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, '%s')",
("name", 4, now))
With regards to the format, I had success with the above command (which includes the milliseconds) and with:
now.strftime('%Y-%m-%d %H:%M:%S')
Hope this helps.
Try using now.date() to get a Date object rather than a DateTime.
If that doesn't work, then converting that to a string should work:
now = datetime.datetime(2009,5,5)
str_now = now.date().isoformat()
cursor.execute('INSERT INTO table (name, id, datecolumn) VALUES (%s,%s,%s)', ('name',4,str_now))
Use Python method datetime.strftime(format), where format = '%Y-%m-%d %H:%M:%S'.
import datetime
now = datetime.datetime.utcnow()
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, %s)",
("name", 4, now.strftime('%Y-%m-%d %H:%M:%S')))
Timezones
If timezones are a concern, the MySQL timezone can be set for UTC as follows:
cursor.execute("SET time_zone = '+00:00'")
And the timezone can be set in Python:
now = datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc)
MySQL Documentation
MySQL recognizes DATETIME and TIMESTAMP values in these formats:
As a string in either 'YYYY-MM-DD HH:MM:SS' or 'YY-MM-DD HH:MM:SS'
format. A “relaxed” syntax is permitted here, too: Any punctuation
character may be used as the delimiter between date parts or time
parts. For example, '2012-12-31 11:30:45', '2012^12^31 11+30+45',
'2012/12/31 11*30*45', and '2012#12#31 11^30^45' are equivalent.
The only delimiter recognized between a date and time part and a
fractional seconds part is the decimal point.
The date and time parts can be separated by T rather than a space. For
example, '2012-12-31 11:30:45' '2012-12-31T11:30:45' are equivalent.
As a string with no delimiters in either 'YYYYMMDDHHMMSS' or
'YYMMDDHHMMSS' format, provided that the string makes sense as a date.
For example, '20070523091528' and '070523091528' are interpreted as
'2007-05-23 09:15:28', but '071122129015' is illegal (it has a
nonsensical minute part) and becomes '0000-00-00 00:00:00'.
As a number in either YYYYMMDDHHMMSS or YYMMDDHHMMSS format, provided
that the number makes sense as a date. For example, 19830905132800 and
830905132800 are interpreted as '1983-09-05 13:28:00'.
What database are you connecting to? I know Oracle can be picky about date formats and likes ISO 8601 format.
**Note: Oops, I just read you are on MySQL. Just format the date and try it as a separate direct SQL call to test.
In Python, you can get an ISO date like
now.isoformat()
For instance, Oracle likes dates like
insert into x values(99, '31-may-09');
Depending on your database, if it is Oracle you might need to TO_DATE it:
insert into x
values(99, to_date('2009/05/31:12:00:00AM', 'yyyy/mm/dd:hh:mi:ssam'));
The general usage of TO_DATE is:
TO_DATE(<string>, '<format>')
If using another database (I saw the cursor and thought Oracle; I could be wrong) then check their date format tools. For MySQL it is DATE_FORMAT() and SQL Server it is CONVERT.
Also using a tool like SQLAlchemy will remove differences like these and make your life easy.
If you're just using a python datetime.date (not a full datetime.datetime), just cast the date as a string. This is very simple and works for me (mysql, python 2.7, Ubuntu). The column published_date is a MySQL date field, the python variable publish_date is datetime.date.
# make the record for the passed link info
sql_stmt = "INSERT INTO snippet_links (" + \
"link_headline, link_url, published_date, author, source, coco_id, link_id)" + \
"VALUES(%s, %s, %s, %s, %s, %s, %s) ;"
sql_data = ( title, link, str(publish_date), \
author, posted_by, \
str(coco_id), str(link_id) )
try:
dbc.execute(sql_stmt, sql_data )
except Exception, e:
...
dt= datetime.now()
query = """INSERT INTO table1(python_Date_col)
VALUES (%s)
"""
conn = ...... # Connection creating process
cur = conn.cursor()
cur.execute(query,(dt))
Above code will fail as "datetime.now()" produces "datetime.datetime(2014, 2, 11, 1, 16)" as a parameter value to insert statement.
Use the following method to capture the datetime which gives string value.
dt= datetime.now().strftime("%Y%m%d%H%M%S")
I was able to successfully run the code after the change...
for example date is 5/5/22 convert it into mysql date format 2022-05-05 to insert record in mysql database
%m month
%d date
%Y Year of 4 digits
%y 2 digits
Code Below:
from datetime import datetime
now='5/5/22'
print("Before", now)
now= datetime.strptime(dob,'%m/%d/%y').strftime('%Y-%m-%d')
print("After", now)
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, %s)",(name, 4,now))
Output:
Before 5/5/22
After 2022-05-05
(mysql format you can easily insert into database)
when iserting into t-sql
this fails:
select CONVERT(datetime,'2019-09-13 09:04:35.823312',21)
this works:
select CONVERT(datetime,'2019-09-13 09:04:35.823',21)
easy way:
regexp = re.compile(r'\.(\d{6})')
def to_splunk_iso(dt):
"""Converts the datetime object to Splunk isoformat string."""
# 6-digits string.
microseconds = regexp.search(dt).group(1)
return regexp.sub('.%d' % round(float(microseconds) / 1000), dt)

Categories