We Keep Coding

Detect time string format in Python?

I have an extremely large dataset with date/time columns with various formats. I have a validation function to detect the possible date/time string formats that can handle handle 24 hour time as well as 12 hour. The seperator is always :. A sample of the is below. However, after profiling my code, it seems this can become a bottleneck and expensive in terms of the execution time. My question is if there is a better way to do this without affecting the performance.
import datetime
def validate_time(time_str: str):
for time_format in ["%H:%M", "%H:%M:%S", "%H:%M:%S.%f", "%I:%M %p"]:
try:
return datetime.datetime.strptime(time_str, time_format)
except ValueError:
continue
return None
print(validate_time(time_str="9:21 PM"))

Instead of trying to parse using every format string, you could split by colons to obtain the segments of your string that denote hours, minutes, and everything that remains. Then you can parse the result depending on the number of values the split returns:
def validate_time_new(time_str: str):
time_vals = time_str.split(':')
try:
if len(time_vals) == 1:
# No split, so invalid time
return None
elif len(time_vals) == 2:
if time_vals[-1][::-2].lower() in ["am", "pm"]:
# if last element contains am or pm, try to parse as 12hr time
return datetime.datetime.strptime(time_str, "%I:%M %p")
else:
# try to parse as 24h time
return datetime.datetime.strptime(time_str, "%H:%M")
elif len(time_vals) == 3:
if "." in time_vals[-1]:
# If the last element has a decimal point, try to parse microseconds
return datetime.datetime.strptime(time_str, "%H:%M:%S.%f")
else:
# try to parse without microseconds
return datetime.datetime.strptime(time_str, "%H:%M:%S")
else: return None
except ValueError:
# If any of the attempts to parse throws an error, return None
return None
To test this, let's time both methods for a bunch of test strings:
import timeit
print("old\t\t\tnew\t\t\t\told/new\t\ttest_string")
for s in ["12:24", "12:23:42", "13:53", "1:53 PM", "12:24:43.220", "not a date", "54:23:21"]:
t1 = timeit.timeit('validate_time(s)', 'from __main__ import datetime, validate_time, s', number=100)
t2 = timeit.timeit('validate_time_new(s)', 'from __main__ import datetime, validate_time_new, s', number=100)
print(f"{t1:.6f}\t{t2:.6f}\t\t{t1/t2:.6f}\t\t{s}")
old new old/new test_string
0.001628 0.001143 1.424322 12:24
0.001567 0.001012 1.548661 12:23:42
0.000935 0.000979 0.955177 13:53
0.003004 0.000722 4.161657 1:53 PM
0.004523 0.001396 3.241204 12:24:43.220
0.002148 0.000025 84.897370 not a date
0.002262 0.000622 3.638629 54:23:21

convert double to time in python

I have a TimeField() in my django models. I want to convert the sum of the record of this to hours.
Here's the actual code in my view:
hours_week_decimal = Timesheet.objects.filter(owner = request.user.pk, week = datetime.datetime.now().isocalendar()[1]).aggregate(total=Sum('working_hour')) # this method return a dict of decimal
total_hours_week = convertDecimalToHours(hours_week_decimal)
and the related function:
def convertDecimalToHours(times):
total_time = 0
for k, v in times.items():
total_time += int(v)
print("type: {} - value: {}".format(type(total_time), total_time))
This returned me:
type: int - value: 166000
I have two hours:
Monday (08:30) and Tuesday(08:30)
It must have returned me "17:00"
Hope you can help me in solving this problem :)

The problem is that 30m+30m = 60m that yes, is 1h, but you expected that the calculator was able to understand that you want 30+30 = 1h
So, in your case, you have to explicit convert 8:30 to 8.5
A fast but not graceful approach to extend your example with few lines can be:
Convert your ints to strings.
Cut out the hours (positions 0 and 1) and multiply for 60 to obtain minutes.
Sum this result to the minutes (positions 2 and 3)
After doing this for every TimeField you have, sum all your times converted in minutes and then reconvert in hours.
In your example:
def convertDecimalToHours(times):
total_time = 0
for k, v in times.items():
tmp_time = str(int(v))
minutes = int(tmp_time[2:4]) + int(tmp_time[0:2])*60
total_time += minutes / 60 # Now your 8:30 is 8.5
print("type: {} - value: {}".format(type(total_time), total_time))
Your output here will be 17.
I suggest you to use this as example to understand the concept, not as a packed solution to your problem.

thanks for your answer,
In my database, i have a column of type "Time". The value's stored like this: 08:30:00.000000.
with my previous code, the times variables (passed in parameters) are like that:
{'total': Decimal('166000.000000')}

datetime: Round/trim number of digits in microseconds

Currently I am logging stuff and I am using my own formatter with a custom formatTime():
def formatTime(self, _record, _datefmt):
t = datetime.datetime.now()
return t.strftime('%Y-%m-%d %H:%M:%S.%f')
My issue is that the microseconds, %f, are six digits. Is there anyway to spit out less digits, like the first three digits of the microseconds?

The simplest way would be to use slicing to just chop off the last three digits of the microseconds:
def format_time():
t = datetime.datetime.now()
s = t.strftime('%Y-%m-%d %H:%M:%S.%f')
return s[:-3]
I strongly recommend just chopping. I once wrote some logging code that rounded the timestamps rather than chopping, and I found it actually kind of confusing when the rounding changed the last digit. There was timed code that stopped running at a certain timestamp yet there were log events with that timestamp due to the rounding. Simpler and more predictable to just chop.
If you want to actually round the number rather than just chopping, it's a little more work but not horrible:
def format_time():
t = datetime.datetime.now()
s = t.strftime('%Y-%m-%d %H:%M:%S.%f')
head = s[:-7] # everything up to the '.'
tail = s[-7:] # the '.' and the 6 digits after it
f = float(tail)
temp = "{:.03f}".format(f) # for Python 2.x: temp = "%.3f" % f
new_tail = temp[1:] # temp[0] is always '0'; get rid of it
return head + new_tail
Obviously you can simplify the above with fewer variables; I just wanted it to be very easy to follow.

As of Python 3.6 the language has this feature built in:
def format_time():
t = datetime.datetime.now()
s = t.isoformat(timespec='milliseconds')
return s

This method should always return a timestamp that looks exactly like this (with or without the timezone depending on whether the input dt object contains one):
2016-08-05T18:18:54.776+0000
It takes a datetime object as input (which you can produce with datetime.datetime.now()). To get the time zone like in my example output you'll need to import pytz and pass datetime.datetime.now(pytz.utc).
import pytz, datetime
time_format(datetime.datetime.now(pytz.utc))
def time_format(dt):
return "%s:%.3f%s" % (
dt.strftime('%Y-%m-%dT%H:%M'),
float("%.3f" % (dt.second + dt.microsecond / 1e6)),
dt.strftime('%z')
)
I noticed that some of the other methods above would omit the trailing zero if there was one (e.g. 0.870 became 0.87) and this was causing problems for the parser I was feeding these timestamps into. This method does not have that problem.

An easy solution that should work in all cases:
def format_time():
t = datetime.datetime.now()
if t.microsecond % 1000 >= 500: # check if there will be rounding up
t = t + datetime.timedelta(milliseconds=1) # manually round up
return t.strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
Basically you do manual rounding on the date object itself first, then you can safely trim the microseconds.

Edit: As some pointed out in the comments below, the rounding of this solution (and the one above) introduces problems when the microsecond value reaches 999500, as 999.5 is rounded to 1000 (overflow).
Short of reimplementing strftime to support the format we want (the potential overflow caused by the rounding would need to be propagated up to seconds, then minutes, etc.), it is much simpler to just truncate to the first 3 digits as outlined in the accepted answer, or using something like:
'{:03}'.format(int(999999/1000))
-- Original answer preserved below --
In my case, I was trying to format a datestamp with milliseconds formatted as 'ddd'. The solution I ended up using to get milliseconds was to use the microsecond attribute of the datetime object, divide it by 1000.0, pad it with zeros if necessary, and round it with format. It looks like this:
'{:03.0f}'.format(datetime.now().microsecond / 1000.0)
# Produces: '033', '499', etc.

You can subtract the current datetime from the microseconds.
d = datetime.datetime.now()
current_time = d - datetime.timedelta(microseconds=d.microsecond)
This will turn 2021-05-14 16:11:21.916229 into 2021-05-14 16:11:21

This method allows flexible precision and will consume the entire microsecond value if you specify too great a precision.
def formatTime(self, _record, _datefmt, precision=3):
dt = datetime.datetime.now()
us = str(dt.microsecond)
f = us[:precision] if len(us) > precision else us
return "%d-%d-%d %d:%d:%d.%d" % (dt.year, dt.month, dt.day, dt.hour, dt.minute, dt.second, int(f))
This method implements rounding to 3 decimal places:
import datetime
from decimal import *
def formatTime(self, _record, _datefmt, precision='0.001'):
dt = datetime.datetime.now()
seconds = float("%d.%d" % (dt.second, dt.microsecond))
return "%d-%d-%d %d:%d:%s" % (dt.year, dt.month, dt.day, dt.hour, dt.minute,
float(Decimal(seconds).quantize(Decimal(precision), rounding=ROUND_HALF_UP)))
I avoided using the strftime method purposely because I would prefer not to modify a fully serialized datetime object without revalidating it. This way also shows the date internals in case you want to modify it further.
In the rounding example, note that the precision is string-based for the Decimal module.

Here is my solution using regexp:
import re
# Capture 6 digits after dot in a group.
regexp = re.compile(r'\.(\d{6})')
def to_splunk_iso(dt):
"""Converts the datetime object to Splunk isoformat string."""
# 6-digits string.
microseconds = regexp.search(dt.isoformat()).group(1)
return regexp.sub('.%d' % round(float(microseconds) / 1000), dt.isoformat())

Fixing the proposed solution based on Pablojim Comments:
from datetime import datetime
dt = datetime.now()
dt_round_microsec = round(dt.microsecond/1000) #number of zeroes to round
dt = dt.replace(microsecond=dt_round_microsec)

If once want to get the day of the week (i.e, 'Sunday)' along with the result, then by slicing '[:-3]' will not work. At that time you may go with,
dt = datetime.datetime.now()
print("{}.{:03d} {}".format(dt.strftime('%Y-%m-%d %I:%M:%S'), dt.microsecond//1000, dt.strftime("%A")))
#Output: '2019-05-05 03:11:22.211 Sunday'
%H - for 24 Hour format
%I - for 12 Hour format
Thanks,

Adding my two cents here as this method will allow you to write your microsecond format as you would a float in c-style. It takes advantage that they both use %f.
import datetime
import re
def format_datetime(date, format):
"""Format a ``datetime`` object with microsecond precision.
Pass your microsecond as you would format a c-string float.
e.g "%.3f"
Args:
date (datetime.datetime): You input ``datetime`` obj.
format (str): Your strftime format string.
Returns:
str: Your formatted datetime string.
"""
# We need to check if formatted_str contains "%.xf" (x = a number)
float_format = r"(%\.\d+f)"
has_float_format = re.search(float_format, format)
if has_float_format:
# make microseconds be decimal place. Might be a better way to do this
microseconds = date.microsecond
while int(microseconds): # quit once it's 0
microseconds /= 10
ms_str = has_float_format.group(1) % microseconds
format = re.sub(float_format, ms_str[2:], format)
return date.strftime(format)
print(datetime.datetime.now(), "%H:%M:%S.%.3f")
# '17:58:54.424'

Python if block executes if the condition expression evaluates to -1

I wrote a method to convert a date string to a date in python. When I get the date from an external method, the millisecond precision is lost, whereas when I do the casting within the method the precision is preserved. Could someone let me know what is the problem here? Thanks a lot!
from datetime import datetime
from pytz import timezone
def getUTCTimeFromString(date_string):
#allow time with Z in it
if date_string:
if date_string.find('Z'):
date_string = date_string[:len(date_string)-1]
return datetime.strptime(date_string,"%Y-%m-%dT%H:%M:%S.%f").replace(tzinfo=timezone('UTC'))
return None
def getStringFromDate(dateObject):
return dateObject.strftime('%Y-%m-%d %H:%M:%S.%f')
#Method being tested
# Input 2012-02-27T05:32:10.607Z
def getEasternTimeFromString(date_string):
if date_string:
if date_string.find('Z'):
date_string = date_string[:len(date_string)-1]
local_date = datetime.strptime(date_string,"%Y-%m-%dT%H:%M:%S.%f").replace(tzinfo=timezone('UTC'))
utc_date = getUTCTimeFromString(date_string)
print 'utc date from external method --> '+getStringFromDate(utc_date)
print 'utc date calculated locally -->' +getStringFromDate(local_date)
return utc_date.astimezone(timezone('US/Eastern'))
return None

This is the problem:
if date_string.find('Z'):
date_string = date_string[:len(date_string)-1]
The problem is that string.find() returns -1 if the target is not found. Since -1 is not zero, the if statement is true, and then your code chops off the last character of the string (whether it was a Z or not).
You're doing this truncation twice in the case of utc_date and once in the case of local_date, thus your different results.
I would suggest:
if date_string.endswith('Z'):
date_string = date_string[:-1]

Handling the different results from parsedatetime

I'm trying to learn python after spending the last 15 or so years working only in Perl and only occasionally.
I can't understand how to handle the two different kinds of results from the parse method of Calendar.parse() from parsedatetime
Given this script:
#!/usr/bin/python
import parsedatetime.parsedatetime as pdt
import parsedatetime.parsedatetime_consts as pdc
import sys
import os
# create an instance of Constants class so we can override some of the defaults
c = pdc.Constants()
# create an instance of the Calendar class and pass in our Constants # object instead of letting it create a default
p = pdt.Calendar(c)
while True:
reply = raw_input('Enter text:')
if reply == 'stop':
break
else:
result = p.parse(reply)
print result
print
And this sample run:
Enter text:tomorrow
(time.struct_time(tm_year=2009, tm_mon=11, tm_mday=28, tm_hour=9, tm_min=0, tm_sec=0, tm_wday=5, tm_yday=332, tm_isdst=-1), 1)
Enter text:11/28
((2009, 11, 28, 14, 42, 55, 4, 331, 0), 1)
I can't figure out how to get the output such that I can consisently use result like so:
print result[0].tm_mon, result[0].tm_mday
That won't work in the case where the input is "11/28" because the output is just a tuple and not a struct_time.
Probably a simple thing.. but not for this newbie. From my perspective the output of Calendar.parse() is unpredictable and hard to use. Any help appreciated. Tia.

I know this is an old question but I ran into this yesterday and the answer here is incomplete (it will fail in the case that parse() returns a datetime).
From the parsedatetime docs:
parse() returns a tuple ( result, type ) where type specifies one of:
0 = not parsed at all
1 = parsed as a date (of type struct_time)
2 = parsed as a time (of type struct_time)
3 = parsed as a datetime (of type datetime.datetime)
Which is a little weird and maybe not the clearest way to do it, but it works and is pretty useful.
Here's a little chunk of code that will convert whatever it returns to a proper python datetime:
import parsedatetime.parsedatetime as pdt
def datetimeFromString( s ):
c = pdt.Calendar()
result, what = c.parse( s )
dt = None
# what was returned (see http://code-bear.com/code/parsedatetime/docs/)
# 0 = failed to parse
# 1 = date (with current time, as a struct_time)
# 2 = time (with current date, as a struct_time)
# 3 = datetime
if what in (1,2):
# result is struct_time
dt = datetime.datetime( *result[:6] )
elif what == 3:
# result is a datetime
dt = result
if dt is None:
# Failed to parse
raise ValueError, ("Don't understand date '"+s+"'")
return dt

Use x = time.struct_time(result[0]) and you'll get a struct_time (so that you can check x.tm_mon and x.tm_mday) no matter whether that result[0] is a struct_time itself, or just a 9-tuple (I've never heard of parsedatetime so I don't know why it's inconsistent in its return type, but with this simple approach you can neutralize that inconsistency).