I am querying Elasticsearch based on date, passing in a date and time string in this format yyyy-mm-dd hh:mm:ss, but Elasticsearch and DateTime are unable to accept this format.
I am writing a script that takes input and queries Elasticsearch based on those inputs, primarily by index and date-time. I've written the script using command line arguments, entering the date-time in the same format, and the script runs perfectly. However, when I try converting the script running with hardcoded inputs, the error appears:
error elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to parse date field
[2019-07-01 00:00:00] with format
[strict_date_optional_time||epoch_millis]')
#this throws the error
runQueryWithoutCommandLine("log4j-*", "2019-07-01 00:00:00", "csv", "json")
#this does not throw error
def runQueryWithCommandLine(*args):
# "yyyy-mm-dd hh:mm:ss" date-time format is given in commandline
Why is this error appearing, and how can I get rid of it? Thank you!
The Date format "strict_date_optional_time||epoch_millis" in elastic uses the ISO date format standards.
As can be seen in link above, the ISO format for string representation of date is :
date-opt-time = date-element ['T' [time-element] [offset]]
In your case, the time portion is separated by a whitespace and not the 'T' and hence the parsing error.
In addition, as I see the time mentioned is 00:00:00, you can simply omit this as this is what that's taken as default is no time portion is specified.
So, any of below date value will work:
1) 2019-07-01T00:00:00
2) 2019-07-01
Related
I have a string datetime "2017-01-01T20:19:47.922596536+09".
I would like to convert this into snowflake's DATETIME_NTZ date type (which can be found here). Simply put, DATETIME_NTZ is defined as
TIMESTAMP_NTZ
TIMESTAMP_NTZ internally stores “wallclock” time with a specified precision. All operations are performed without taking any time zone into account.
If the output format contains a time zone, the UTC indicator (Z) is displayed.
TIMESTAMP_NTZ is the default for TIMESTAMP.
Aliases for TIMESTAMP_NTZ:
TIMESTAMPNTZ
TIMESTAMP WITHOUT TIME ZONE
I've tried using numpy.datetime64 but I get the following:
> numpy.datetime64("2017-01-01T20:19:47.922596536+09")
numpy.datetime64('2017-01-01T11:19:47.922596536')
This for some reason converts the time to certain timezone.
I've also tried pd.to_datetime:
> pd.to_datetime("2017-01-01T20:19:47.922596536+09")
Timestamp('2017-01-01 20:19:47.922596536+0900', tz='pytz.FixedOffset(540)')
This gives me the correct value but when I try to insert the above value to snowflake db, I get the following error:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 252004: Failed processing pyformat-parameters: 255001: Binding data in type (timestamp) is not supported.
Any suggestions would be much appreciated!
You can do this on the Snowflake side if you want by sending the string format as-is and converting to a timestamp_ntz. This single line shows two ways, one that simply strips off the time zone information, and one that converts the time zone to UTC before stripping off the time zone.
select try_to_timestamp_ntz('2017-01-01T20:19:47.922596536+09',
'YYYY-MM-DD"T"HH:MI:SS.FF9TZH') TS_NTZ
,convert_timezone('UTC',
try_to_timestamp_tz('2017-01-01T20:19:47.922596536+09',
'YYYY-MM-DD"T"HH:MI:SS.FF9TZH'))::timestamp_ntz UTC_TS_NTZ
;
Note that Snowflake UI by default only shows 3 decimal places (milliseconds) unless you specify higher precision for the output display using to_varchar() and a timestamp format string.
TS_NTZ
UTC_TS
2017-01-01 20:19:47.922596536
2017-01-01 11:19:47.922596536
I have this date example '2022-08-30T11:53:52.204219' stored in database, when I get it from database it's type is string so I wanted to convert it to a date type by using this python code
datetime.strptime('2022-08-30T11:53:52.204219', "%Y-%m-%d'T'%H:%M:%S.%f")
I also tried this one
datetime.strptime('2022-08-30T11:53:52.204219', "yyyy-MM-dd'T'HH:mm:ssZ")
But I always get this error response 'time data '2022-08-30T11:53:52.204219' does not match format "%Y-%m-%d'T'%H:%M:%S.%f'
I need help to convert this string date to an actual date
As per comment:
from datetime import datetime
print(datetime.strptime('2022-08-30T11:53:52.204219', "%Y-%m-%dT%H:%M:%S.%f"))
Result:
2022-08-30 11:53:52.204219
I am loading the data using COPY command.
My Dates are in the following format.
D/MM/YYYY eg. 1/12/2016
DD/MM/YYYY eg. 23/12/2016
My target table data type is DATE. I am getting the following error "Invalid Date Format - length must be 10 or more"
As per the AWS Redshift documentation,
The default date format is YYYY-MM-DD. The default time stamp without
time zone (TIMESTAMP) format is YYYY-MM-DD HH:MI:SS.
So, as your date is not in the same format and of different length, you are getting this error. Append the following at the end of your COPY command and it should work.
[[COPY command as you are using right now]] + DATEFORMAT 'DD/MM/YYYY'
Not sure about the single digit case though. You might want to pad the incoming values with a 0 in the beginning to match the format length.
I was trying some API e-commerce and i followed their example, but I keep getting Timestamp error / mismatch.
The API requires a timestamp in this format:
2017-08-29T09:54:00+07:00
I tried:
datetime.now().isoformat()
But it returns without the timezone:
'2017-08-29T09:54:57.702000'
then I tried:
datetime.now(pytz.timezone( 'Asia/Jakarta' )).isoformat()
but it returns:
2017-08-29T09:54:00.547000+07:00
Basically the python version doesn't use : for the minutes and seconds and it seems they also gives the microsecond which the API does not need. How do i get this format in python 2?
year-month-day then T hour-minute-second+Timezone
2017-08-29T09:54:00+07:00
I got it
datetime.now(pytz.timezone( 'Asia/Jakarta')).replace(microsecond=0).isoformat()
I am trying to perform a query which contains a date through an API. The date should be in a ISO_8601 date format(yyyy-MM-dd'T'HH:mm:ssZ).
For example, one of the tuples in my table could have the date:
2012-11-11T00:00:00+0000.
In unicode it is queried for as follows:
2012-11-11T00%3A00%3A00%2B0000
All my own checks to see if the date is valid passes but I am getting the following SQL Exception:
SQL Exception: ERROR: operator does not exist: timestamp with time zone >= character varying
My code which carries out the queries through the API is written in python. I'm not sure what the exception means? Is it not possible to query for a timestamp with a String?
To specify a timestamp literal, use the following format:
where ts_column >= timestamp '2012-11-11 T00:00:00+0000'
The keyword timestamp is important to define a timestamp literal, otherwise it's "only" a character value (and that's what the error message tells you). The value must conform to the ISO date style.
Another option would be to use the to_timestamp function:
where ts_column >= to_timestamp('2012-11-11 00:00:00', 'yyyy-mm-dd hh24:mi:ss');
which gives you more control over the actual format of your literal.
For more details see the manual: http://www.postgresql.org/docs/current/static/functions-formatting.html#FUNCTIONS-FORMATTING-DATETIME-TABLE