Some code I am writing in Python takes in a date from a server in the struct_time format (with the 9 args).
How can I store this date in an SQL database, and be able to read back this date as a struct_time while keeping the timezone and all additional information coming from struct_time?
I tried putting the struct_time directly in the SQL
struct_date = time.struct_time(tm_year=2020, tm_mon=9, tm_mday=10, tm_hour=22, tm_min=31, tm_sec=4, tm_wday=3, tm_yday=254, tm_isdst=0)
cursor.execute("UPDATE dbo.RSS_Links SET last_update=? WHERE link=?;", struct_date, links)
> "A TVP's rows must be Sequence objects.", 'HY000'
I can put the time in the database using the below, but I don't see where the timezone is kept when converting to strftime.
date_to_store = time.strftime("%Y-%m-%d %H:%M:%S", struct_date)
I'd highly suggest doing one of these (in this specific order):
Use built-in DATETIME data type and store all dates in UTC
Use LONG/BIGINT type to store date in epoch
Use built-in DATETIME format that can store time zone information
Don't store dates as strings, don't couple it with struct_time or any other struct/class, you'll regret it later :)
Your application should have a data layer, which would handle data serialization/deserialization.
Related
I have a string datetime "2017-01-01T20:19:47.922596536+09".
I would like to convert this into snowflake's DATETIME_NTZ date type (which can be found here). Simply put, DATETIME_NTZ is defined as
TIMESTAMP_NTZ
TIMESTAMP_NTZ internally stores “wallclock” time with a specified precision. All operations are performed without taking any time zone into account.
If the output format contains a time zone, the UTC indicator (Z) is displayed.
TIMESTAMP_NTZ is the default for TIMESTAMP.
Aliases for TIMESTAMP_NTZ:
TIMESTAMPNTZ
TIMESTAMP WITHOUT TIME ZONE
I've tried using numpy.datetime64 but I get the following:
> numpy.datetime64("2017-01-01T20:19:47.922596536+09")
numpy.datetime64('2017-01-01T11:19:47.922596536')
This for some reason converts the time to certain timezone.
I've also tried pd.to_datetime:
> pd.to_datetime("2017-01-01T20:19:47.922596536+09")
Timestamp('2017-01-01 20:19:47.922596536+0900', tz='pytz.FixedOffset(540)')
This gives me the correct value but when I try to insert the above value to snowflake db, I get the following error:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 252004: Failed processing pyformat-parameters: 255001: Binding data in type (timestamp) is not supported.
Any suggestions would be much appreciated!
You can do this on the Snowflake side if you want by sending the string format as-is and converting to a timestamp_ntz. This single line shows two ways, one that simply strips off the time zone information, and one that converts the time zone to UTC before stripping off the time zone.
select try_to_timestamp_ntz('2017-01-01T20:19:47.922596536+09',
'YYYY-MM-DD"T"HH:MI:SS.FF9TZH') TS_NTZ
,convert_timezone('UTC',
try_to_timestamp_tz('2017-01-01T20:19:47.922596536+09',
'YYYY-MM-DD"T"HH:MI:SS.FF9TZH'))::timestamp_ntz UTC_TS_NTZ
;
Note that Snowflake UI by default only shows 3 decimal places (milliseconds) unless you specify higher precision for the output display using to_varchar() and a timestamp format string.
TS_NTZ
UTC_TS
2017-01-01 20:19:47.922596536
2017-01-01 11:19:47.922596536
class Test(Model):
time = DateTimeField()
# ...
row = Test.select()[0]
test.time
This returns a string that looks like this: 2017-01-23 01:01:39+01:00. How can I get it as a datetime object instead? Do I have to parse it manually?
Also I would be interested if there is any documentation on how to use the DateTimeField. The official documentation doesn't have anything on it.
Are you using SQLite? If so, SQLite doesn't have a dedicated datetime type, so datetimes are stored as strings in the DB. What peewee will do is recognize certain datetime formats coming out of the DB and convert them to datetime objects. What you need to do is ensure that either:
When you create/save your object, that you assign a datetime object to the field.
When reading back pre-existing data, that the data is in a recognized format.
The formats peewee supports out-of-the-box for datetime field are:
YYYY-mm-dd HH:MM:SS.ffffff
YYYY-mm-dd HH:MM:SS
YYYY-mm-dd
It looks like your has zone info. I'd suggest converting to UTC and dropping the zone info. That should fix it.
Have you tried adding a default like this?
time = DateTimeField(default=datetime.datetime.now())
Or when adding an entry add it as a datetime.datetime object directly:
test = Test(....., time=datetime.datetime.strptime("2018-3-15", '%Y-%m-%d'))
In the second case you don't need to specify anything in the class definition...
I want to insert date and time into mongo ,using pymongo.
However, I can insert datetime but not just date or time .
here is the example code :
now = datetime.datetime.now()
log_date = now.date()
log_time = now.time()
self.logs['test'].insert({'log_date_time': now, 'log_date':log_date, 'log_time':log_time})
it show errors :
bson.errors.InvalidDocument: Cannot encode object: datetime.time(9, 12, 39, 535769)
in fact , i don't know how to insert just date or time in mongo shell too.
i know insert datetime is new Date(), but I just want the date or time filed.
You are experiencing the defined behavior. MongoDB has a single datetime type (datetime). There are no separate, discrete types of just date or just time.
Workarounds: Plenty, but food for thought:
Storing just date is straightforward: assume Z time, use a time component of 00:00:00, and ignore the time offset upon retrieval.
Storing just time is trickier but doable: establish a base date like the epoch and only vary the time component, and ignore the date component upon retrieval.
I am trying to store date into mongodb using python(bottle framework).
I want to store it in the below format:
ISODate("2015-06-08 03:38:28")
Currently I am using the following command:
datetime.strptime(DateField, '%m/%d/%Y %H:%M:%S %p')
it is getting stored like this:
ISODate("2015-06-08T03:38:28Z")
How to store it without "T" and "Z" in it??
You are confusing how something is stored vs. how something is displayed.
In MongoDB, dates are stored as 64 bit integers, what you are seeing is the way it is represented so that we can easily determine what date and time the 64bit number represents.
The ISODate is just a helper method, which formats the date in the ISO date format.
So when you pass it in a normal date and time string, it will convert it into the correct format.
The format adds the T (to separate the time part) and the Z (as you have not identified a time zone, it is defaulted to UTC).
In short - you are not storing it with the T and the Z, that's just how it is displayed back to you.
I am trying to store a Python datetime object in an ORACLE column of type date.
so far, I have used,
rpt_time = time.strftime('%Y-%m-%d %H:%M:%S') or
rpt_time = str(datetime.datetime.now())
but all are giving ORA-01843: not a valid month
I am really confused how to insert ORACLE date type python datetime object
cx_Oracle supports passing objects of class datetime.datetime. Because of this when you already have object of this class (for example when calling datetime.datetime.now()) you should not try to format it and pass as a string but instead pass it directly. This way you prevent all errors caused by wrong format of date and/or time.
Example:
cursor.execute("INSERT INTO SomeTable VALUES(:now)", {'now': datetime.datetime.now()})
Be aware that you have to take additional steps if you want to prevent truncation of fractional part of seconds. For details please read Mastering Oracle+Python, Part 2: Working with Times and Dates article by Przemysław Piotrowski.
As far as my search shows, ORACLE can be picky on dates so this might be what you need to do.
Firstly, check the format of date you have. For example, if you have something like, 2010/01/26:11:00:00AM, then you might want to do following on your cursor execute:
insert into x
values(99, to_date('2010/01/26:11:00:00AM', 'yyyy/mm/dd:hh:mi:ssam'));
You have to convert date from python to oracle by setting nls_date_format in you session
>>> rpt_time = time.strftime('%Y-%m-%d %H:%M:%S')
>>> rpt_time
'2014-05-12 21:06:40'
Then before inserting into oracle, do the following
cursor.execute("ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY-MM-DD HH24:MI:SS'")
datetime.now() in python gives you milliseconds and have to get rid of it before sucessfully writing to Oracle.
from datetime import datetime
....
cursor.execute("INSERT INTO myTable VALUES(to_date('" + str(datetime.now().replace(microsecond=0)) + "','yyyy-mm-dd hh24:mi:ss'))")
....