I have a date column in a MySQL table. I want to insert a datetime.datetime() object into this column. What should I be using in the execute statement?
I have tried:
now = datetime.datetime(2009,5,5)
cursor.execute("INSERT INTO table
(name, id, datecolumn) VALUES (%s, %s
, %s)",("name", 4,now))
I am getting an error as: "TypeError: not all arguments converted during string formatting"
What should I use instead of %s?
For a time field, use:
import time
time.strftime('%Y-%m-%d %H:%M:%S')
I think strftime also applies to datetime.
You are most likely getting the TypeError because you need quotes around the datecolumn value.
Try:
now = datetime.datetime(2009, 5, 5)
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, '%s')",
("name", 4, now))
With regards to the format, I had success with the above command (which includes the milliseconds) and with:
now.strftime('%Y-%m-%d %H:%M:%S')
Hope this helps.
Try using now.date() to get a Date object rather than a DateTime.
If that doesn't work, then converting that to a string should work:
now = datetime.datetime(2009,5,5)
str_now = now.date().isoformat()
cursor.execute('INSERT INTO table (name, id, datecolumn) VALUES (%s,%s,%s)', ('name',4,str_now))
Use Python method datetime.strftime(format), where format = '%Y-%m-%d %H:%M:%S'.
import datetime
now = datetime.datetime.utcnow()
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, %s)",
("name", 4, now.strftime('%Y-%m-%d %H:%M:%S')))
Timezones
If timezones are a concern, the MySQL timezone can be set for UTC as follows:
cursor.execute("SET time_zone = '+00:00'")
And the timezone can be set in Python:
now = datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc)
MySQL Documentation
MySQL recognizes DATETIME and TIMESTAMP values in these formats:
As a string in either 'YYYY-MM-DD HH:MM:SS' or 'YY-MM-DD HH:MM:SS'
format. A “relaxed” syntax is permitted here, too: Any punctuation
character may be used as the delimiter between date parts or time
parts. For example, '2012-12-31 11:30:45', '2012^12^31 11+30+45',
'2012/12/31 11*30*45', and '2012#12#31 11^30^45' are equivalent.
The only delimiter recognized between a date and time part and a
fractional seconds part is the decimal point.
The date and time parts can be separated by T rather than a space. For
example, '2012-12-31 11:30:45' '2012-12-31T11:30:45' are equivalent.
As a string with no delimiters in either 'YYYYMMDDHHMMSS' or
'YYMMDDHHMMSS' format, provided that the string makes sense as a date.
For example, '20070523091528' and '070523091528' are interpreted as
'2007-05-23 09:15:28', but '071122129015' is illegal (it has a
nonsensical minute part) and becomes '0000-00-00 00:00:00'.
As a number in either YYYYMMDDHHMMSS or YYMMDDHHMMSS format, provided
that the number makes sense as a date. For example, 19830905132800 and
830905132800 are interpreted as '1983-09-05 13:28:00'.
What database are you connecting to? I know Oracle can be picky about date formats and likes ISO 8601 format.
**Note: Oops, I just read you are on MySQL. Just format the date and try it as a separate direct SQL call to test.
In Python, you can get an ISO date like
now.isoformat()
For instance, Oracle likes dates like
insert into x values(99, '31-may-09');
Depending on your database, if it is Oracle you might need to TO_DATE it:
insert into x
values(99, to_date('2009/05/31:12:00:00AM', 'yyyy/mm/dd:hh:mi:ssam'));
The general usage of TO_DATE is:
TO_DATE(<string>, '<format>')
If using another database (I saw the cursor and thought Oracle; I could be wrong) then check their date format tools. For MySQL it is DATE_FORMAT() and SQL Server it is CONVERT.
Also using a tool like SQLAlchemy will remove differences like these and make your life easy.
If you're just using a python datetime.date (not a full datetime.datetime), just cast the date as a string. This is very simple and works for me (mysql, python 2.7, Ubuntu). The column published_date is a MySQL date field, the python variable publish_date is datetime.date.
# make the record for the passed link info
sql_stmt = "INSERT INTO snippet_links (" + \
"link_headline, link_url, published_date, author, source, coco_id, link_id)" + \
"VALUES(%s, %s, %s, %s, %s, %s, %s) ;"
sql_data = ( title, link, str(publish_date), \
author, posted_by, \
str(coco_id), str(link_id) )
try:
dbc.execute(sql_stmt, sql_data )
except Exception, e:
...
dt= datetime.now()
query = """INSERT INTO table1(python_Date_col)
VALUES (%s)
"""
conn = ...... # Connection creating process
cur = conn.cursor()
cur.execute(query,(dt))
Above code will fail as "datetime.now()" produces "datetime.datetime(2014, 2, 11, 1, 16)" as a parameter value to insert statement.
Use the following method to capture the datetime which gives string value.
dt= datetime.now().strftime("%Y%m%d%H%M%S")
I was able to successfully run the code after the change...
for example date is 5/5/22 convert it into mysql date format 2022-05-05 to insert record in mysql database
%m month
%d date
%Y Year of 4 digits
%y 2 digits
Code Below:
from datetime import datetime
now='5/5/22'
print("Before", now)
now= datetime.strptime(dob,'%m/%d/%y').strftime('%Y-%m-%d')
print("After", now)
cursor.execute("INSERT INTO table (name, id, datecolumn) VALUES (%s, %s, %s)",(name, 4,now))
Output:
Before 5/5/22
After 2022-05-05
(mysql format you can easily insert into database)
when iserting into t-sql
this fails:
select CONVERT(datetime,'2019-09-13 09:04:35.823312',21)
this works:
select CONVERT(datetime,'2019-09-13 09:04:35.823',21)
easy way:
regexp = re.compile(r'\.(\d{6})')
def to_splunk_iso(dt):
"""Converts the datetime object to Splunk isoformat string."""
# 6-digits string.
microseconds = regexp.search(dt).group(1)
return regexp.sub('.%d' % round(float(microseconds) / 1000), dt)
Related
The Following function create a table in sqlite3:
def create_table(mycursor):
mycursor.execute('''CREATE TABLE IF NOT EXISTS ch_details(id INTEGER PRIMARY KEY autoincrement,
ch_id TEXT, ch_date DATETIME DEFAULT CURRENT_DATE, ch_no TEXT, cli TEXT, vhs_no TEXT, f_lo TEXT,
t_lo TEXT, qty REAL, adv REAL DEFAULT 0 , oth_de REAL DEFAULT 0, ch_amt REAL DEFAULT 0, mem_id TEXT,
UNIQUE(ch_id));''')
which stores my date in Datetime in ch_date column however when
i try to get the last row of ch_id column in this table stored by providing specific month/year using the following code:
def gen_chid():
dt, mt, yr = cal_gen.get().split("/")
conn = sqlite3.connect('database/u_data.vita')
mycursor = conn.cursor()
mycursor.execute("SELECT ch_id FROM ch_details WHERE strftime('%Y%m', ch_date)",yr, mt)
row = mycursor.fetchone()
The code gets this error:
TypeError: function takes at most 2 arguments (3 given)
The can_gen.get() gets the date from entry box in "07/07/2021" string format
I have also checked this stack answer link but did not get any result.
SQLite's datetime functions like strftime() work only if the datetime values that are passed to them have the format YYYY-MM-DD for dates or YYYY-MM-DD hh:mm:ss for datetimes.
If you stored the dates in the format DD/MM/YYYY then the only way to extract the date parts like day, month or year is by treating the date as a string and use string functions like SUBSTR():
def gen_chid():
conn = sqlite3.connect('database/u_data.vita')
mycursor = conn.cursor()
sql = """
SELECT ch_id
FROM ch_details
WHERE SUBSTR(ch_date, 4) = ?
"""
mycursor.execute(sql, (cal_gen.get()[3:],))
row = mycursor.fetchone()
Here SUBSTR(ch_date, 4) extracts the month/year part in the format MM/YYYY and it is compared to the substring returned from cal_gen.get() after the 3d char which is passed to execute() as the only member of a list.
If I have a datetime object of abc= 2021-03-11 21:14:21.043000+00:00 how do I convert it to 2021-03-11 21:14:21 using python? I am trying to insert this particular value into my postgresql column having column's datatype as timestamp and it gives me an error like
[parameters: (datetime.datetime(2021, 3, 11, 21, 14, 21, 43000, tzinfo=<UTC>))]
The datetime: 2021-03-11 21:14:21.043000+00:00 is coming from python and I am trying to insert this in postgresql.
How do I solve this? Help me out.
Edit:
I also tried to use time() on a datetime object but it gave an attribute error.
here's what I did:
last_file_ran_time = last_file_run_time.rsplit('.')[0]```
values = last_file_run, last_file_ran_time
db.execute("""INSERT INTO gcs_file_info_table (last_file_run, last_file_run_time) VALUES (?,?) """, values)
print("data inserted")
And here's the error:
LINE 1: ...info_table (last_file_run, last_file_run_time) VALUES (?,?)
^
[SQL: INSERT INTO gcs_file_info_table (last_file_run, last_file_run_time) VALUES (?,?) ]
[parameters: ('xyz.json', datetime.time(21, 14, 21, 43000))]
You can format the datetime with f-strings or the .strftime method, but you probably shouldn't need to; depending on which framework/library you're using, it should convert the datetime automatically.
So the options, in the order of preference:
Check your framework/library (you don't say which one you're using) for instructions on how to pass a datetime as a parameter; this is the best option by far.
If that doesn't work, format the date as you might for printing it out, using either an f-string:
f"{the_date:%Y-%m-%d %H:%M:%S}"
or, using the strftime method
the_date.strftime("%Y-%m-%d %H:%M:%S")
Edit: The error you're seeing is not caused by the date; it's caused by using ? as the placeholder, while the library requires %s
db.execute("""INSERT INTO gcs_file_info_table (last_file_run, last_file_run_time) VALUES (%s, %s) """, values)
Variable.time() will give you time factor of datetime
`
import datetime
x = datetime.datetime.now()
x.time()
`
The Sqlite documentation states:
SQLite has no DATETIME datatype. Instead, dates and times can be stored in any of these ways:
As a TEXT string in the ISO-8601 format. Example: '2018-04-02 12:13:46'.
As an INTEGER number of seconds since 1970 (also known as "unix time").
...
so I decided to use an INTEGER unix timestamp:
import sqlite3, time
conn = sqlite3.connect(':memory:')
conn.execute("CREATE TABLE data(datetime INTEGER, t TEXT);")
conn.execute("INSERT INTO data VALUES (CURRENT_TIMESTAMP, 'hello')")
Why does the following query return no result?
ts = int(time.time()) + 31*24*3600 # unix timestamp 1 month in the future
print(list(conn.execute("SELECT * FROM data WHERE datetime <= ?", (ts, ))))
More generally, how to do a SELECT query with a comparison with a unix timestamp with Sqlite?
PS:
I have already read SQLite DateTime comparison and similar questions, which offer other comparison methods, but here I'd like to precisely discuss why this unix timestamp comparison does not work.
For performance reasons, I'd like to:
do a query that compares integers (which is super fast if many rows): WHERE datetime <= unix_timestamp,
avoid to convert unix_timestamp into string, and then compare datetime to this string (I guess it'll be far slower)
You use CURRENT_TIMESTAMP when inserting new rows.
This means that in your column the values are not stored as unix timestamps becuase CURRENT_TIMESTAMP returns the current date in the format of YYYY-MM-DD hh:mm:ss.
You can transform the unix timestamp to datetime in the format of YYYY-MM-DD hh:mm:ss with the function datetime() and the unixepoch modifier:
conn.execute("SELECT * FROM data WHERE datetime <= datetime(?, 'unixepoch')", (ts, ))
If your unix timestamp contains milliseconds you must strip them off:
conn.execute("SELECT * FROM data WHERE datetime <= datetime(? / 1000, 'unixepoch')", (ts, ))
Or, you can transform the string datetime in the column datetime to a unix timestamp with the function strftime():
conn.execute("SELECT * FROM data WHERE strftime('%s', datetime) + 0 <= ?", (ts, ))
If you want to store integer values in the column, use strftime() like this:
INSERT INTO data VALUES (strftime('%s', CURRENT_TIMESTAMP) + 0, 'hello')
I have a Python(3) script that is supposed to run each morning. In it, I call some SQL. However I'm getting an error message:
Error while connecting to PostgreSQL operator does not exist: date = integer
The SQL is based on the concatenation of a string:
ecom_dashboard_query = """
with
days_data as (
select
s.date,
s.user_type,
s.channel_grouping,
s.device_category,
sum(s.sessions) as sessions,
count(distinct s.dimension2) as daily_users,
sum(s.transactions) as transactions,
sum(s.transaction_revenue) as revenue
from ga_flagship_ecom.sessions s
where date = """ + run.start_date + """
group by 1,2,3,4
)
insert into flagship_reporting.ecom_dashboard
select *
from days_data;
"""
Here is the full error:
09:31:25 Error while connecting to PostgreSQL operator does not exist: date = integer
09:31:25 LINE 14: where date = 2020-01-19
09:31:25 ^
09:31:25 HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
I tried wrapping run.start_date within str like so: str(run.start_date) but I received the same error message.
I suspect it may be to do with the way I concatenate the SQL query string, but I am not sure.
The query runs fine in SQL directly with a hard coded date and no concatenation:
where date = '2020-01-19'
How can I get the query string to work correctly?
It's more better to pass query params to cursor.execute method. From docs
Warning Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
So instead of string concatenation pass run.start_date as second argument of cursor.execute.
In your query instead of concatenation use %s:
where date = %s
group by 1,2,3,4
In your python code add second argument to execute method:
cur.execute(ecom_dashboard_query , (run.start_date,))
Your sentece is wrong:
where date = """ + run.start_date + """
try to compare a date and a string and this is not posible, you need to convert "run.start_date" to datetime and compare simply:
date_format = datetime.strptime(your_date_string, '%y-%m-%d')
and with this date converted to datetime do:
where date = date_format
Final code:
date_format = datetime.strptime(your_date_string, '%y-%m-%d')
ecom_dashboard_query = """
with
days_data as (
select
s.date,
s.user_type,
s.channel_grouping,
s.device_category,
sum(s.sessions) as sessions,
count(distinct s.dimension2) as daily_users,
sum(s.transactions) as transactions,
sum(s.transaction_revenue) as revenue
from ga_flagship_ecom.sessions s
where date = {}
group by 1,2,3,4
)
insert into flagship_reporting.ecom_dashboard
select *
from days_data;
""".format(date_format)
I'm trying to INSERT to MySQL from a CSV file, first 'column' in the file is a date in this format:
31/08/2017;
then my column in the table is set as YYYY-MM-DD
this is my code:
import datetime
import csv
import MySQLdb
...
insertionSQL="INSERT INTO transactions (trans_date, trans_desc, trans_amnt) VALUES(" + datetime.datetime.strptime('{0}', '%d/%m/%Y').strftime('%Y-%m-%d') + ",{2},{3}), row)"
cur.execute(insertionSQL)
I get the following error from my python script:
(data_string, format))
ValueError: time data '{0}' does not match format '%d/%m/%Y'
You should perform the date formatting logic during the query execution. You can't just put the logic in your insertion query string and expect it to work (as you can see, strptime is trying to convert the string {0} into a date object according to the date format string, which obviously cannot work!)
insertionSQL = "INSERT INTO transactions (trans_date, trans_desc, trans_amnt) VALUES(%s, %s, %s)"
... # read csv
# Reformat the date on the actual data from the CSV!
csv_date = datetime.datetime.strptime(csv_date, '%d/%m/%Y').strftime('%Y-%m-%d')
# Now all of your input data is formatted correctly, execute the prepared statement.
cursor.execute(insertionSQL, (csv_date, csv_desc, csv_amnt))
I found the issue. I was missing delimiter=";" so the csv reader took entire row as one value