I am trying to perform a query which contains a date through an API. The date should be in a ISO_8601 date format(yyyy-MM-dd'T'HH:mm:ssZ).
For example, one of the tuples in my table could have the date:
2012-11-11T00:00:00+0000.
In unicode it is queried for as follows:
2012-11-11T00%3A00%3A00%2B0000
All my own checks to see if the date is valid passes but I am getting the following SQL Exception:
SQL Exception: ERROR: operator does not exist: timestamp with time zone >= character varying
My code which carries out the queries through the API is written in python. I'm not sure what the exception means? Is it not possible to query for a timestamp with a String?
To specify a timestamp literal, use the following format:
where ts_column >= timestamp '2012-11-11 T00:00:00+0000'
The keyword timestamp is important to define a timestamp literal, otherwise it's "only" a character value (and that's what the error message tells you). The value must conform to the ISO date style.
Another option would be to use the to_timestamp function:
where ts_column >= to_timestamp('2012-11-11 00:00:00', 'yyyy-mm-dd hh24:mi:ss');
which gives you more control over the actual format of your literal.
For more details see the manual: http://www.postgresql.org/docs/current/static/functions-formatting.html#FUNCTIONS-FORMATTING-DATETIME-TABLE
Related
I have a string datetime "2017-01-01T20:19:47.922596536+09".
I would like to convert this into snowflake's DATETIME_NTZ date type (which can be found here). Simply put, DATETIME_NTZ is defined as
TIMESTAMP_NTZ
TIMESTAMP_NTZ internally stores “wallclock” time with a specified precision. All operations are performed without taking any time zone into account.
If the output format contains a time zone, the UTC indicator (Z) is displayed.
TIMESTAMP_NTZ is the default for TIMESTAMP.
Aliases for TIMESTAMP_NTZ:
TIMESTAMPNTZ
TIMESTAMP WITHOUT TIME ZONE
I've tried using numpy.datetime64 but I get the following:
> numpy.datetime64("2017-01-01T20:19:47.922596536+09")
numpy.datetime64('2017-01-01T11:19:47.922596536')
This for some reason converts the time to certain timezone.
I've also tried pd.to_datetime:
> pd.to_datetime("2017-01-01T20:19:47.922596536+09")
Timestamp('2017-01-01 20:19:47.922596536+0900', tz='pytz.FixedOffset(540)')
This gives me the correct value but when I try to insert the above value to snowflake db, I get the following error:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 252004: Failed processing pyformat-parameters: 255001: Binding data in type (timestamp) is not supported.
Any suggestions would be much appreciated!
You can do this on the Snowflake side if you want by sending the string format as-is and converting to a timestamp_ntz. This single line shows two ways, one that simply strips off the time zone information, and one that converts the time zone to UTC before stripping off the time zone.
select try_to_timestamp_ntz('2017-01-01T20:19:47.922596536+09',
'YYYY-MM-DD"T"HH:MI:SS.FF9TZH') TS_NTZ
,convert_timezone('UTC',
try_to_timestamp_tz('2017-01-01T20:19:47.922596536+09',
'YYYY-MM-DD"T"HH:MI:SS.FF9TZH'))::timestamp_ntz UTC_TS_NTZ
;
Note that Snowflake UI by default only shows 3 decimal places (milliseconds) unless you specify higher precision for the output display using to_varchar() and a timestamp format string.
TS_NTZ
UTC_TS
2017-01-01 20:19:47.922596536
2017-01-01 11:19:47.922596536
I am querying Elasticsearch based on date, passing in a date and time string in this format yyyy-mm-dd hh:mm:ss, but Elasticsearch and DateTime are unable to accept this format.
I am writing a script that takes input and queries Elasticsearch based on those inputs, primarily by index and date-time. I've written the script using command line arguments, entering the date-time in the same format, and the script runs perfectly. However, when I try converting the script running with hardcoded inputs, the error appears:
error elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to parse date field
[2019-07-01 00:00:00] with format
[strict_date_optional_time||epoch_millis]')
#this throws the error
runQueryWithoutCommandLine("log4j-*", "2019-07-01 00:00:00", "csv", "json")
#this does not throw error
def runQueryWithCommandLine(*args):
# "yyyy-mm-dd hh:mm:ss" date-time format is given in commandline
Why is this error appearing, and how can I get rid of it? Thank you!
The Date format "strict_date_optional_time||epoch_millis" in elastic uses the ISO date format standards.
As can be seen in link above, the ISO format for string representation of date is :
date-opt-time = date-element ['T' [time-element] [offset]]
In your case, the time portion is separated by a whitespace and not the 'T' and hence the parsing error.
In addition, as I see the time mentioned is 00:00:00, you can simply omit this as this is what that's taken as default is no time portion is specified.
So, any of below date value will work:
1) 2019-07-01T00:00:00
2) 2019-07-01
I am loading the data using COPY command.
My Dates are in the following format.
D/MM/YYYY eg. 1/12/2016
DD/MM/YYYY eg. 23/12/2016
My target table data type is DATE. I am getting the following error "Invalid Date Format - length must be 10 or more"
As per the AWS Redshift documentation,
The default date format is YYYY-MM-DD. The default time stamp without
time zone (TIMESTAMP) format is YYYY-MM-DD HH:MI:SS.
So, as your date is not in the same format and of different length, you are getting this error. Append the following at the end of your COPY command and it should work.
[[COPY command as you are using right now]] + DATEFORMAT 'DD/MM/YYYY'
Not sure about the single digit case though. You might want to pad the incoming values with a 0 in the beginning to match the format length.
I am moving some of my code onto sqlalchemy from using raw MySQL queries.
The current issue I am having is that the datetime was saved in a string format by a C# tool. Unfortunately, the representation does not match up with Python's (as well as that it has an extra set of single quotes), thus making filtering somewhat cumbersome.
Here is an example of the format that the date was saved in:
'2016-07-01T17:27:01'
Which I was able to convert to a usable datetime using the following MySQL command:
STR_TO_DATE(T.PredicationGeneratedTime, \"'%%Y-%%m-%%dT%%H:%%i:%%s'\")
However, I cannot find any documentation that describes how to invoke built-in functions such as STR_TO_DATE when filtering with sqlalchemy
The following Python code:
session.query(Train.Model).filter(cast(Train.Model.PredicationGeneratedTime, date) < start)
is giving me:
TypeError: Required argument 'year' (pos 1) not found
There does not seem to be a way to specify the format for the conversion.
Note: I realize the solution is to fix the way the datetime is stored, but in the mean time I'd like to run queries against the existing data.
You can try to use func.str_to_date(COLUMN, FORMAT_STRING) instead of cast
In the cast() you should be using sqlalchemy.DateTime, not (what I assume is) a datetime.date - that is the cause of the exception.
However, fixing that will not really help because of the embedded single quotes.
You are fortunate that the dates stored in your table are in ISO format. That means that lexicographic comparisons will work on the date strings themselves, without casting. As long as you use a string for start with the surrounding single quotes, it will work.
from datetime import datetime
start = "'{}'".format(datetime.now().isoformat())
session.query(Train.Model).filter(Train.Model.PredicationGeneratedTime < start)
I am trying to store a Python datetime object in an ORACLE column of type date.
so far, I have used,
rpt_time = time.strftime('%Y-%m-%d %H:%M:%S') or
rpt_time = str(datetime.datetime.now())
but all are giving ORA-01843: not a valid month
I am really confused how to insert ORACLE date type python datetime object
cx_Oracle supports passing objects of class datetime.datetime. Because of this when you already have object of this class (for example when calling datetime.datetime.now()) you should not try to format it and pass as a string but instead pass it directly. This way you prevent all errors caused by wrong format of date and/or time.
Example:
cursor.execute("INSERT INTO SomeTable VALUES(:now)", {'now': datetime.datetime.now()})
Be aware that you have to take additional steps if you want to prevent truncation of fractional part of seconds. For details please read Mastering Oracle+Python, Part 2: Working with Times and Dates article by Przemysław Piotrowski.
As far as my search shows, ORACLE can be picky on dates so this might be what you need to do.
Firstly, check the format of date you have. For example, if you have something like, 2010/01/26:11:00:00AM, then you might want to do following on your cursor execute:
insert into x
values(99, to_date('2010/01/26:11:00:00AM', 'yyyy/mm/dd:hh:mi:ssam'));
You have to convert date from python to oracle by setting nls_date_format in you session
>>> rpt_time = time.strftime('%Y-%m-%d %H:%M:%S')
>>> rpt_time
'2014-05-12 21:06:40'
Then before inserting into oracle, do the following
cursor.execute("ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY-MM-DD HH24:MI:SS'")
datetime.now() in python gives you milliseconds and have to get rid of it before sucessfully writing to Oracle.
from datetime import datetime
....
cursor.execute("INSERT INTO myTable VALUES(to_date('" + str(datetime.now().replace(microsecond=0)) + "','yyyy-mm-dd hh24:mi:ss'))")
....