We Keep Coding

Stanford NLP's SUTime: Unable to capture certain date formats

I am using the python wrapper of NLP Stanford's SUTime.
So far comparing the results to other date parsers like duckling, dateparser's search_dates, parsedatetime and natty, SUTime gives the most reliable results.
However, it fails to capture some obvious dates from documents.
Following are the 2 types of documents that I am having difficult parsing for dates using SUTime.
I am out and I won't be available until 9/19
I am out and I won't be available between (September 18-September 20)
It gives no results in case of the first document.
However, for the second document, it only captures the month but not the date or date range.
I tried wrapping my head around the java's code to see if I could alter or add some rules to make this work, but couldn't figure it out.
If someone can suggest a way to make this work with SUTime, it would be really helpful.
Also, I tried dateparser's search_dates, and it is unreliable as it captures anything and everything. Like for the first document it would parse a date on text "am out" (which is not required) and "9/19" (which is okay). So if there is a way to control this behavior it would work as well.

Question: Unable to capture certain date formats
This solution does use datetime instead of SUTime
import datetime
def datetime_from_string(datestring):
rules = [('(\d{1,2}\/\d{1,2})', '%m/%d', {'year': 2018}), ('(\w+ \d{1,2})-(\w+ \d{1,2})', '%B %d', {'year': 2018})]
result = None
for rule in rules:
match = re.match(rule[0], datestring)
if match:
result = []
for part in match.groups():
try:
date = datetime.strptime(part, rule[1])
if rule[2]:
for key in rule[2]:
if key == 'year':
date = datetime(rule[2][key], date.month, date.day)
result.append(date)
except ValueError:
pass
return result
# If you reach heare, NO matching rule
raise ValueError("Datestring '{}', does not match any rule!".format(datestring))
# Usage
for datestring in ['9/19', 'September 18-September 20', '2018-09-01']:
result = datetime_from_string(datestring)
print("str:{} result:{}".format(datestring, result))
Output:
str:'9/19' result:[datetime.datetime(2018, 9, 19, 0, 0)]
str:'September 18-September 20' result:[datetime.datetime(2018, 9, 18, 0, 0), datetime.datetime(2018, 9, 20, 0, 0)
ValueError: Datestring '2018-09-01', does not match any rule!
Tested with Python: 3.4.2

What is an efficient way to test a string for a specific datetime format like "m%/d%/Y%" in python 3.6?

In my Python 3.6 application, from my input data I can receive datatimes in two different formats:
"datefield":"12/29/2017" or "datefield":"2017-12-31"
I need to make sure the that I can handle either datetime format and convert them to (or leave it in) the iso 8601 format. I want to do something like this:
#python pseudocode
import datetime
if datefield = "m%/d%/Y%":
final_date = datetime.datetime.strptime(datefield, "%Y-%m-%d").strftime("%Y-%m-%d")
elif datefield = "%Y-%m-%d":
final_date = datefield
The problem is I don't know how to check the datefield for a specific datetime format in that first if-statement in my pseudocode. I want a true or false back. I read through the Python docs and some tutorials. I did see one or two obscure examples that used try-except blocks, but that doesn't seem like an efficient way to handle this. This question is unique from other stack overflow posts because I need to handle and validate two different cases, not just one case, where I can simply fail it if it does validate.

You can detect the first style of date by a simple string test, looking for the / separators. Depending on how "loose" you want the check to be, you could check a specific index or scan the whole string with a substring test using the in operator:
if "/" in datefield: # or "if datefield[2] = '/'", or maybe "if datefield[-5] = '/'"
final_date = datetime.datetime.strptime(datefield, "%m/%d/%Y").strftime("%Y-%m-%d")

Since you'll only ever deal with two date formats, just check for a / or a - character.
import datetime
# M/D/Y
if '/' in datefield:
final_date = datetime.datetime.strpdate(date, '%M/%D/%Y').isoformat()
# Y-M-D
elif '-' in datefield:
final_date = datetime.datetime.strpdate(date, '%Y-%M-%D').isoformat()

A possible approach is to use the dateutil library. It contains many of the commonest datetime formats and can automagically detect these formats for you.
>>> from dateutil.parser import parse
>>> d1 = "12/29/2017"
>>> d2 = "2017-12-31"
>>> parse(d1)
datetime.datetime(2017, 12, 29, 0, 0)
>>> parse(d2)
datetime.datetime(2017, 12, 31, 0, 0)
NOTE: dateutil is a 3rd party library so you may need to install it with something like:
pip install python-dateutil
It can be found on pypi:
https://pypi.python.org/pypi/python-dateutil/2.6.1
And works with Python2 and Python3.
Alternate Examples:
Here are a couple of alternate examples of how well dateutil handles random date formats:
>>> d3 = "December 28th, 2017"
>>> parse(d3)
datetime.datetime(2017, 12, 28, 0, 0)
>>> d4 = "27th Dec, 2017"
>>> parse(d4)
datetime.datetime(2017, 12, 27, 0, 0)

I went with the advice of #ChristianDean and used the try-except block in effort to be Pythonic. The first format %m/%d/%Y appears a bit more in my data, so I lead the try-except with that datetime formatting attempt.
Here is my final solution:
import datetime
try:
final_date = datetime.datetime.strptime(datefield, "%m/%d/%Y").strftime("%Y-%m-%d")
except ValueError:
final_date = datetime.datetime.strptime(datefield, "%Y-%m-%d").strftime("%Y-%m-%d")

Return ISO Week list where it crosses over another year using python

I am trying to convert a yyyyww which is actually starting as a string object I think at is like 201731. I want to capture the last 6 weeks including its self.
def generate_6_last_week(yearweek):
weeks = [int(yearweek)]
date = time.mktime(datetime.strptime(str(yearweek)+"0","%Y%W%w").timetuple())
for i in range(1,6):
date = date-604800 # 60*60*24*7
weeks.append(int(datetime.fromtimestamp(int(date)).strftime('%Y%W')))
return weeks
generate_6_last_week(201731)
so the output for 201731 should be:
[201731, 201730, 201729, 201728, 201727, 201726]
and this seems to work, the problem is if i test it with a crossover year like 201702 it returns this:
[201702, 201701, 201700, 201651, 201650, 201649]
Which also looks pretty good, but I need it in ISO weeks so there shouldnt be a week 00 I think and the last week of a year should be either 53 or 52 but not the 51.
Any ideas how to adapt this?

This requires the "isoweek" package to be installed but gave me what i want with some manipulation of the format of my str YYYYWW and works well with the crossover years.
from isoweek import Week
yearweek = "201702"
weeks = [int(yearweek)]
x = 1
for i in range(5):
week = int(str(Week(int(yearweek[:4]), int(yearweek[-2:])-x)).replace("W",""))
weeks.append(week)
x +=1
print(weeks)
or in a function format.
def generate_6_last_week(yearweek):
weeks = [int(yearweek)]
x = 1
for i in range(5):
week = int(str(Week(int(yearweek[:4]), int(yearweek[-2:])-x)).replace("W",""))
weeks.append(week)
x +=1
print(weeks)
generate_6_last_week("201702")

I do not know anything about striptime(). So I have solved the problem by writing my own code. Here's the code:
date=input()
a=int(date[0:4])
b=int(date[4:])
k=5-b
finallist=[]
def m(x,y):
return(x-y)
if b>=5:
date=int(date)
finallist=[date,date-1,date-2,date-3,date-4,date-5]
print(finallist)
else:
date=int(date)
for i in range(b+1):
finallist.append(date)
date-=1
a-=1
b=52
date=int(str(a)+str(b))
for i in range(k):
finallist.append(date)
date-=1
print(finallist)
201700 gives [201700, 201652, 201651, 201650, 201649, 201648]
201702 gives [201702, 201701, 201700, 201652, 201651, 201650]
ALITER: There's a much easier way to do this. Just add 1 to those specific list items in your list :p

How to trim spaces within timestamps using 'm/d/yy' format

I have a Python script that generates .csv files from other data sources.
Currently, an error happens when the user manually adds a space to a date by accident. Instead of inputting the date as "1/13/17", a space may be added at the front (" 1/13/17") so that there's a space in front of the month.
I've included the relevant part of my Python script below:
def processDateStamp(sourceStamp):
matchObj = re.match(r'^(\d+)/(\d+)/(\d+)\s', sourceStamp)
(month, day, year) = (matchObj.group(1), matchObj.group(2), matchObj.group(3))
return "%s/%s/%s" % (month, day, year)
How do I trim the space issue in front of month and possibly on other components of the date (the day and year) as well for the future?
Thanks in advance.

Since you're dealing with dates, it might be more appropriate to use datetime.strptime than regex here. There are two advantages of this approach:
It makes it slightly clearer to anyone reading that you're trying to parse dates.
Your code will be more prone to throw exceptions when trying to parse data that doesn't represent dates, or represent dates in an incorrect format - this is good because it helps you catch and address issues that might otherwise go unnoticed.
Here's the code:
from datetime import datetime
def processDateStamp(sourceStamp):
date = datetime.strptime(sourceStamp.replace(' ', ''), '%M/%d/%y')
return '{}/{}/{}'.format(date.month, date.day, date.year)
if __name__ == '__main__':
print(processDateStamp('1/13/17')) # 1/13/17
print(processDateStamp(' 1/13/17')) # 1/13/17
print(processDateStamp(' 1 /13 /17')) # 1/13/17

You also can use parser from python-dateutil library. The main benefit you will get - it can recognize the datetime format for you (sometimes it may be useful):
from dateutil import parser
from datetime import datetime
def processDateTimeStamp(sourceStamp):
dt = parser.parse(sourceStamp)
return dt.strftime("%m/%d/%y")
processDateTimeStamp(" 1 /13 / 17") # returns 01/13/17
processDateTimeStamp(" jan / 13 / 17")
processDateTimeStamp(" 1 - 13 - 17")
processDateTimeStamp(" 1 .13 .17")

Once again, a perfect opportunity to use split, strip, and join:
def remove_spaces(date_string):
date_list = date_string.split('/')
result = '/'.join(x.strip() for x in date_list)
return result
Examples
In [7]: remove_spaces('1/13/17')
Out[7]: '1/13/17'
In [8]: remove_spaces(' 1/13/17')
Out[8]: '1/13/17'
In [9]: remove_spaces(' 1/ 13/17')
Out[9]: '1/13/17'

Convert Military Time from text file to Standard time Python

I am having problems with logic on how to convert military time from a text file to standard time and discard all the wrong values. I have only got to a point where the user is asked for the input file and the contents are displayed from the text file entered. Please help me

Python's datetime.time objects use "military" time. You can do things like this:
>>> t = datetime.time(hour=15, minute=12)
>>> u = datetime.time(hour=16, minute=44)
>>> t = datetime.datetime.combine(datetime.datetime.today(), t)
>>> t
datetime.datetime(2011, 5, 11, 15, 12)
>>> u = datetime.datetime.combine(datetime.datetime.today(), u)
>>> t - u
datetime.timedelta(-1, 80880)
With a little twiddling, conversions like the ones you describe should be pretty simple.
Without seeing any code, it's hard to tell what exactly you want. But I assume you could do something like this:
raw_time = '2244'
converted_time = datetime.time(hour=int(raw_time[0:2]), minute=int(raw_time[2:4]))
converted_time = datetime.datetime.combine(datetime.datetime.today(), converted_time)
Now you can work with converted_time, adding and subtracting timedelta objects. Fyi, you can create a timedelta like so:
td = datetime.timedelta(hours=4)
The full list of possible keyword arguments to the timedelta constructor is here.

from dateutil.parser import parse
time_today = parse("08:00")
from dateutil.relativedelta import relativedelta
required_time = time_today-relativedelta(minutes=35)
print required_time
datetime.datetime(2011, 5, 11, 7, 25)

It's not a true answer like the other two, but the philosophy I use when dealing with dates and times in python: convert to a datetime object as soon as possible after getting from the user, and only convert back into a string when you are presenting the value back to the user.
Most of the date/time math you will need can be done by datetime objects, and their cousins the timedelta objects. Unless you need ratios of timedeltas to other timedeltas, but that's another story.