If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the right format string?
Obviously, generating every possible format string to do an exhaustive search is not a feasible idea. I also don't really want to write one function with many format strings hardcoded into it.
Does anyone have any thoughts on how this can be accomplished (perhaps some sort of regex?)?
What about fuzzyparsers:
Sample inputs:
jan 12, 2003
jan 5
2004-3-5
+34 -- 34 days in the future (relative to todays date)
-4 -- 4 days in the past (relative to todays date)
Example usage:
>>> from fuzzyparsers import parse_date
>>> parse_date('jun 17 2010') # my youngest son's birthday
datetime.date(2010, 6, 17)
Install with:
$ pip install fuzzyparsers
You can use parser from dateutil
Example usage:
from dateutil import parser
dt = parser.parse("Aug 28 1999 12:00AM")
Related
I need to return the date format from a string. Currently I am using parser to parse a string as a date, then replacing the year with a yyyy or yy. Similarly for other dates items. Is there some function I could use that would return mm-dd-yyyy when I send 12-05-2018?
Technically, it is an impossible question. If you send in 12-05-2018, there is no way for me to know whether you are sending in a mm-dd-yyyy (Dec 5, 2018) or dd-mm-yyyy (May 12, 2018).
One approach might be to do a regex replacement of anything which matches your expected date pattern, e.g.
date = "Here is a date: 12-05-2018 and here is another one: 10-31-2010"
date_masked = re.sub(r'\b\d{2}-\d{2}-\d{4}\b', 'mm-dd-yyyy', date)
print(date)
print(date_masked)
Here is a date: 12-05-2018 and here is another one: 10-31-2010
Here is a date: mm-dd-yyyy and here is another one: mm-dd-yyyy
Of course, the above script makes no effort to check whether the dates are actually valid. If you require that, you may use one of the date libraries available in Python.
I don't really understand what you plan to do with the format. There are two reasons I can think of why you might want it. (1) You want at some future point to convert a normalized datetime back into the original string. If that is what you want you would be better off just storing the normalized datetime and the original string. Or (2) you want to draw (dodgy) conclusions about person sending the data, because different nationalities will tend to use different formats. But, whatever you want it for, you can do it this way:
from dateutil import parser
def get_date_format(date_input):
date = parser.parse(date_input)
for date_format in ("%m-%d-%Y", "%d-%m-%Y", "%Y-%m-%d"):
# You can extend the list above to include formats with %y in addition to %Y, etc, etc
if date.strftime(date_format) == date_input:
return date_format
>>> date_input = "12-05-2018"
>>> get_date_format(date_input)
'%m-%d-%Y'
You mention in a comment you are prepared to make assumptions about ambiguous dates like 12-05-2018 (could be May or December) and 05-12-18 (could be 2018 or 2005). You can pass those assumptions to dateutil.parser.parse. It accepts boolean keyword parameters dayfirst and yearfirst which it will use in ambiguous cases.
Take a look at the datetime library. There you will find the function strptime(), which is exactly what you are looking for.
Here is the documentation: https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior
I tried multiple packages to extract timestamp from a given string, but no package gives correct results. I did use dateutils, datefinder, parsedatetime, etc. for this task. They extract some datetimes which are in certain formats but not all formats, sometimes they extract some unwanted numbers also as timestamps.
Is there any python package which extracts datetime from a given string.
Assume, I have 2 strings like these:
scala> val xorder= new order(1,"2016-02-22 00:00:00.00",100,"COMPLETED")
and
Fri, 10 Jun 2011 11:04:17 +0200 (CEST)
and want to extract only datetime. Is there any function which extracts both formats of datetimes from above strings. In other cases formats may be different, still it should pick out datetime strings
You can use the datetime function strptime() as follows
dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M")
You can create your own formatting and use the function as well.
I created a small python package datetime_extractor to pull out timestamps from a given strings. It can extract many datetime formats from given strings. Hope it will be useful.
pip install datetime-extractor
from datetime_extractor import DateTimeExtractor
import re
samplestring1 = 'scala> val xorder= new order(1,"2016-02-22 00:00:00.00",100,"COMPLETED")'
DateTimeExtractor(samplestring1)
Out: ['2016-02-22 00:00:00.00']
samplestring2 = 'Fri, 10 Jun 2011 11:04:17 +0200 (CEST)'
DateTimeExtractor(samplestring2)
Out: ['10 Jun 2011 11:04:17']
#Allan & #Manmeet Singh, Let me know your comments.
Here's an array of datetime values:
array = np.array(['2016-05-01T00:00:59.3+10:00', '2016-05-01T00:02:59.4+10:00',
'2016-05-01T00:03:59.4+10:00', '2016-05-01T00:13:00.1+10:00',
'2016-05-01T00:22:00.5+10:00', '2016-05-01T00:31:01.1+10:00'],
dtype=object)
pd.to_datetime is very good at inferring datetime formats.
array = pd.to_datetime(array)
print(array)
DatetimeIndex(['2016-04-30 14:00:59.300000', '2016-04-30 14:02:59.400000',
'2016-04-30 14:03:59.400000', '2016-04-30 14:13:00.100000',
'2016-04-30 14:22:00.500000', '2016-04-30 14:31:01.100000'],
dtype='datetime64[ns]', freq=None)
How can I dynamically figure out what datetime format pd.to_datetime inferred? Something like: %Y-%m-%dT... (sorry, my datetime foo is really bad).
I don't think it's possible to do this in full generality in pandas.
As mentioned in other comments and answers, the internal function _guess_datetime_format is close to being what you ask for, but it has strict criteria for what constitutes a guessable format and so it will only work for a restricted class of datetime strings.
These criteria are set out in the _guess_datetime_format function on these lines and you can also see some examples of good and bad formats in the test_parsing script.
Some of the main points are:
year, month and day must each be present and identifiable
the year must have four digits
exactly six digits must be used if using microseconds
you can't specify a timezone
This means that it will fail to guess the format for datetime strings in the question despite them being a valid ISO 8601 format:
>>> from pandas.core.tools.datetimes import _guess_datetime_format_for_array
>>> array = np.array(['2016-05-01T00:00:59.3+10:00'])
>>> _guess_datetime_format_for_array(array)
# returns None
In this case, dropping the timezone and padding the microseconds to six digits is enough to make pandas to recognise the format:
>>> array = np.array(['2016-05-01T00:00:59.300000']) # six digits, no tz
>>> _guess_datetime_format_for_array(array)
'%Y-%m-%dT%H:%M:%S.%f'
This is probably as good as it gets.
If pd.to_datetime is not asked to infer the format of the array, or given a format string to try, it will just try and parse each string separately and hope that it is successful. Crucially, it does not need to infer a format in advance to do this.
First, pandas parses the string assuming it is (approximately) a ISO 8601 format. This begins in a call to _string_to_dts and ultimately hits the low-level parse_iso_8601_datetime function that does the hard work.
You can check if your string is able to be parsed in this way using the _test_parse_iso8601 function. For example:
from pandas._libs.tslib import _test_parse_iso8601
def is_iso8601(string):
try:
_test_parse_iso8601(string)
return True
except ValueError:
return False
The dates in the array you give are recognised as this format:
>>> is_iso8601('2016-05-01T00:00:59.3+10:00')
True
But this doesn't deliver what the question asks for and I don't see any realistic way to recover the exact format that is recognised by the parse_iso_8601_datetime function.
If parsing the string as a ISO 8601 format fails, pandas falls back to using the parse() function from the third-party dateutil library (called by parse_datetime_string). This allows a fantastic level of parsing flexibility but, again, I don't know of any good way to extract the recognised datetime format from this function.
If both of these two parsers fail, pandas either raises an error, ignores the string or defaults to NaT (depending on what the user specifies). No further attempt is made to parse the string or guess the format of the string.
DateInfer (PyDateInfer) library allows to infer dates based on the sequence of available dates:
github.com/wdm0006/dateinfer
Usage from docs:
>>> import dateinfer
>>> dateinfer.infer(['Mon Jan 13 09:52:52 MST 2014', 'Tue Jan 21 15:30:00 EST 2014'])
'%a %b %d %H:%M:%S %Z %Y'
>>>
Disclaimer: I have used and then contributed to this library
You can use _guess_datetime_format from core.tools to get the format. ie
from pandas.core.tools import datetimes as tools
tools._guess_datetime_format(pd.to_datetime(array).format()[0][:10])
Output :
'%Y-%m-%d'
To know more about this method you can see here. Hope it helps.
For example I give the date as:
2/12/2015
The result should be:
February/Thursday/2015
I tried to do with if but I'm not getting the result. It would be nice if you could tell me the long way (without using built in functions (like datetime and others) too much). I'm new to python and not much is taught in my school.
You don't have to use datetime too much, simply parse the date and output it in whatever format you want
from datetime import datetime
d = "2/12/2015"
print(datetime.strptime(d,"%m/%d/%Y").strftime("%B/%A/%Y"))
February/Thursday/2015
A = Locale’s full weekday name.
B = Locale’s full month name.
Y = Year with century as a decimal number.
All the format directives are here
You could create a dict mapping but you will find datetime is lot simpler.
I need to create a script that will run every 5 minutes via cron and parse syslog, but only match the entries within the last 5 minutes of the execution time.
If there a module I can leverage for part of this? If so, will it work when running at midnight when the date is one day ahead? The date format of my log is:
Jun 20 14:04:47 <log msg details>
You can use datetime's strptime function:
>>> datetime.strptime("Jun 20 14:04:47", "%b %d %H:%M:%S")
# => datetime.datetime(1900, 6, 20, 14, 4, 47)
You can then use simple operators like < to compare datetimes. (You'll have to be aware that if a year is not specified, it will default to 1900.)
I had to write a script that parses the syslog continuously (think "tail -f"), so the case is slightly different, but maybe this helps anyway. I parsed each line using this regex. depending your your syslog implementation you'll have to adapt of course.
pattern=re.compile("^(?P<month>[a-zA-Z]{3})\s+(?P<day>\d\d?)\s(?P<hour>\d\d)\:(?P<minute>\d\d):(?P<second>\d\d)(?:\s(?P<suppliedhost>[a-zA-Z0-9_-]+))?\s(?P<host>[a-zA-Z0-9_-]+)\s(?P<process>[a-zA-Z0-9\/_-]+)(\[(?P<pid>\d+)\])?:\s(?P<message>.+)$")
then I could convert it into a datetime object
minute=int(m.group('minute'))
hour=int(m.group('hour'))
day=int(m.group('day'))
second=int(m.group('second'))
month=monthindex[m.group('month')]
now=datetime.datetime.utcnow()
year=now.year
ts=datetime.datetime(year,month,day,hour,minute,second)
you could then check for each line if that object is within the last 5 minutes.
your problem will be that you'll have to parse the whole syslog in each run unless you store the last file position somewhere.