String to time stamp conversion - python

I have a data file with about 5.6million time-stamps in the format "2016-10-17 15:00:40.739". They are all strings at the moment for some reason and I need to convert them all to date times as I will later need to calculate the difference between groups of them (e.g: stamp1 -> stamp2 = 2hours, 4minutes etc).
I found another question "Converting string into datetime" but mine are in a different format and I cannot get that answer to work for me.
Any help is much appreciated.

Use numpy's datetime64:
>>> np.datetime64('2016-10-17 15:00:40.739')
numpy.datetime64('2016-10-17T15:00:40.739')
You can easily find differences by simply subtracting, or using numpy's timedelta64:
>>> np.datetime64('2016-10-17 15:00:40.739') - np.datetime64('2016-10-15 15:00:40.739')
numpy.timedelta64(172800000,'ms')
>>> np.datetime64('2016-10-17 15:00:40.739') + np.timedelta64(1,'D')
numpy.datetime64('2016-10-18T15:00:40.739')

Try this:
from datetime import datetime
a = "2016-10-17 15:00:40.739"
b = datetime.strptime(a,'%Y-%m-%d %H:%M:%S.%f')
print(b)
>>> datetime.datetime(2016, 10, 17, 15, 0, 40, 739000)
To define the format of your dates. Follow this guide: https://www.tutorialspoint.com/python/time_strptime.htm

You can use the dateutil module to convert the string date to datetime object.
from dateutil import parser
dt = parser.parse("2016-10-17 15:00:40.739")
print dt
print type(dt)
Output:
2016-10-17 15:00:40.739000
<type 'datetime.datetime'>

Related

How to parse time string without date and date string without time?

Is there any way to automatically parse strings with time only to datetime.time object (or something similar)? Same for datetime.date.
I've tried dateutil, arrow, moment, pandas.to_datetime.
All these parsers create timestamps with a current date.
>>> from dateutil.parser import parse
>>> parse('23:53')
datetime.datetime(2019, 1, 8, 23, 53) # datetime.time(23, 53) expected
>>> parse('2018-01-04')
datetime.datetime(2018, 1, 4, 0, 0) # datetime.date(2018, 1, 4) expected
UPD:
Thanks for the responses. Think that I should clarify the problem.
The program doesn't know what will be in the input (timestamp, date or time), and it should decide to set appropriate type. The problem is to distinguish these types.
For example, I can parse 23:53 and get a timestamp. How can I decide to extract the time from it or not?
You can use fromisoformat() from datetime.
import datetime
datetime.time.fromisoformat('23:53')
datetime.date.fromisoformat('2018-01-04')
What you basically want is for '23:53' to become a datetime.time object and for '2018-01-04' to become a datetime.date object. This cannot be achieved by using dateutil.parser.parse():
Returns a datetime.datetime object or, if the fuzzy_with_tokens option is True, returns a tuple, the first element being a datetime.datetime object, the second a tuple containing the fuzzy tokens.
From the documentation. So you'll always get a datetime.datetime object when using dateutil.parser.parse()
I would guess you need to interpret the input string yourself to define wether you're trying to parse a time or a date. When you do that, you can still use the dateutil.parser.parse() function to get the object you want:
from dateutil.parser import parse
my_time = parse('23:53')
my_time.time() # datetime.time(23, 53)
my_time.date() # datetime.date(2019, 1, 8)
Here you have an example. Just set the date attributes with replace, and select the output with strftime.
import datetime
date = datetime.datetime.now()
newdate = date.replace(hour=11, minute=59)
print(newdate.strftime('%H:%M'))
newdate2 = date.replace(year=2014, month=1, day=3)
print(newdate2.strftime('%Y-%m-%d'))
You can use either time or datetime modules, but one thing to bear in mind, is that these always create an object, that specifies a moment in time. (Also, if parsing strings, consider using the strptime function and displaying as string, strftime function respectively)
e.g.
>>> hours = time.strptime("23:59", "%H:%M")
>>> days = time.strptime("2018-01-04", "%Y-%m-%d")
>>> time.strftime("%H:%M", hours)
'23:59'
>>> time.strftime("%H:%M %Y", hours)
'23:59 1900'
Not recommended, but if you wish to separate these two object for some reason and wish to only care for a specific portion of your assignement, you can still adress the respective numbers with
>>> hours.tm_hour
23
>>> hours.tm_min
59
>>> days.tm_mon
1
>>> days.tm_mday
4
>>> days.tm_year
2018
A far better approach, in my opinion would be formatting the complete date string and using the strptime to form a complete timestamp - even if you get the time and date as separate inputs:
>>> ttime = "22:45"
>>> dday = "2018-01-04"
You can use the % formatter, or the "new" python f-Strings
>>> complete_t_string = "{} {}".format(dday, ttime)
>>> complete_t_string
'2018-01-04 22:45'
Now that we have a complete string, we can specify how it should be read and create a complete timestamp:
>>> complete_time = time.strptime(complete_t_string, "%Y-%m-%d %H:%M")
>>> complete_time
time.struct_time(tm_year=2018, tm_mon=1, tm_mday=4, tm_hour=22, tm_min=45, tm_sec=0, tm_wday=3, tm_yday=4, tm_isdst=-1)
EDIT:
Somebody will probably kill me, but if you absolutely know that you will only get two types of values, you could just do a simple try / except construct. It can probably be written more Pythonically:
try:
time.strptime(t_string, "%H:%M")
except ValueError:
time.strptime(t_string, "%Y-%m-%d")

Convert string to NumPy datetime64 dtype

I program gets a string of current time every minute as date = '201711081750'
I want to store these strings as np.datetime64 into an array.
I think I could convert this kind of strings as
>>> date = '201711081750'
>>> np.datetime64( date[:4] +'-'+date[4:6]+'-'+date[6:8]+' ' +date[8:10]+':'+date[10:] , 'm' )
numpy.datetime64('2017-11-08T17:50')
But it looks complicated and I think it might engender errors later.
Are there simpler ways to do this?
pd.to_datetime
import pandas as pd
pd.to_datetime(date, format='%Y%m%d%H%M')
Timestamp('2017-11-08 17:50:00')
The important bit here is the format string '%Y%m%d%H%M'.
datetime.datetime equivalent in python.
from datetime import datetime as dt
dt.strptime(date, '%Y%m%d%H%M')
datetime.datetime(2017, 11, 8, 17, 50)

compare date and time objects and get older date in python

I have 3 dates (in hours and sec and mins)
2016-11-30T13:27:04-05:00
2016-11-30T13:27:41-05:00
2017-03-01T22:16:35-05:00
How can i get older date which is 2016-11-30T13:27:04-05:00
as output
This python script is not giving correct results
import time
find_a = min(a)
print find_a
Try this
dat = ['2016-11-30T13:27:04-05:02', '2016-11-30T13:27:41-05:02','2016-11-30T13:27:04-05:00']
print(min(dat))
You can use dateutil.parser to parse dates and a min to compare them. Here is an example:
In [1]: from dateutil.parser import parse
In [4]: dates = ['2016-11-30T13:27:04-05:00', '2016-11-30T13:27:41-05:00', '2017-03-01T22:16:35-05:00']
In [5]: min([parse(s) for s in dates])
Out[5]: datetime.datetime(2016, 11, 30, 13, 27, 4, tzinfo=tzoffset(None, -18000))
a = ['2016-11-30T13:27:04-05:00', '2016-11-30T13:27:41-05:00','2017-03-01T22:16:35-05:00']
>>> min(a) '2016-11-30T13:27:04-05:00
It works, just try it as normal strings. You can avoid importing time
Do you want to convert them datetime objects and then find the minimum?
use the method specified here https://stackoverflow.com/a/12282040/5334188 to datetime objects and find minimum

Converting dates in Python

I have dates in the form 26/11/2015. How can I convert them into the format 26-Nov-2015 and still keep them as dates and not strings?
Your question does not make much sense. If you keep them as dates, they have no format. The format is only manifested when you convert them to strings.
So the answer is: Store the dates as date (or datetime) objects, and use datetime.strftime with some specific format whenever you need them as a string:
>>> from datetime import date
>>> d = date(2016, 11, 26)
>>> d.strftime("%Y/%m/%d")
'2016/11/26'
>>> d.strftime("%d-%b-%Y")
'26-Nov-2016'
Conversely, use strptime to parse strings in different formats to dates:
>>> datetime.datetime.strptime("26-Nov-2015", "%d-%b-%Y")
datetime.datetime(2015, 11, 26, 0, 0)
from datetime import datetime
date = datetime.strptime('26/11/2015', '%d/%m/%Y')
print date.strftime("%d-%B-%Y")
In the above example, we are taking your input string 'dd/mm/yyyy' and turning it into a python datetime saving it to a variable called date (for future usage as per your request), and then printing it out in the format requested.
You want to use the datetime module I think. For example:
from datetime import date
a = date(2015, 11, 26)
a.strftime("%A %d of %B, %Y")
should give you 'Thursday 26 of November, 2015'
Or for your specific formatting request:
a.strftime("%d-%b-%Y") #'26-Nov-2015'
Hope this helps, good luck!

Python datetime to string without microsecond component

I'm adding UTC time strings to Bitbucket API responses that currently only contain Amsterdam (!) time strings. For consistency with the UTC time strings returned elsewhere, the desired format is 2011-11-03 11:07:04 (followed by +00:00, but that's not germane).
What's the best way to create such a string (without a microsecond component) from a datetime instance with a microsecond component?
>>> import datetime
>>> print unicode(datetime.datetime.now())
2011-11-03 11:13:39.278026
I'll add the best option that's occurred to me as a possible answer, but there may well be a more elegant solution.
Edit: I should mention that I'm not actually printing the current time – I used datetime.now to provide a quick example. So the solution should not assume that any datetime instances it receives will include microsecond components.
If you want to format a datetime object in a specific format that is different from the standard format, it's best to explicitly specify that format:
>>> import datetime
>>> datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
'2011-11-03 18:21:26'
See the documentation of datetime.strftime() for an explanation of the % directives.
Starting from Python 3.6, the isoformat() method is flexible enough to also produce this format:
datetime.datetime.now().isoformat(sep=" ", timespec="seconds")
>>> import datetime
>>> now = datetime.datetime.now()
>>> print unicode(now.replace(microsecond=0))
2011-11-03 11:19:07
In Python 3.6:
from datetime import datetime
datetime.now().isoformat(' ', 'seconds')
'2017-01-11 14:41:33'
https://docs.python.org/3.6/library/datetime.html#datetime.datetime.isoformat
This is the way I do it. ISO format:
import datetime
datetime.datetime.now().replace(microsecond=0).isoformat()
# Returns: '2017-01-23T14:58:07'
You can replace the 'T' if you don't want ISO format:
datetime.datetime.now().replace(microsecond=0).isoformat(' ')
# Returns: '2017-01-23 15:05:27'
Yet another option:
>>> import time
>>> time.strftime("%Y-%m-%d %H:%M:%S")
'2011-11-03 11:31:28'
By default this uses local time, if you need UTC you can use the following:
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
'2011-11-03 18:32:20'
Keep the first 19 characters that you wanted via slicing:
>>> str(datetime.datetime.now())[:19]
'2011-11-03 14:37:50'
I usually do:
import datetime
now = datetime.datetime.now()
now = now.replace(microsecond=0) # To print now without microsecond.
# To print now:
print(now)
output:
2019-01-13 14:40:28
Since not all datetime.datetime instances have a microsecond component (i.e. when it is zero), you can partition the string on a "." and take only the first item, which will always work:
unicode(datetime.datetime.now()).partition('.')[0]
As of Python 3.6+, the best way of doing this is by the new timespec argument for isoformat.
isoformat(timespec='seconds', sep=' ')
Usage:
>>> datetime.now().isoformat(timespec='seconds')
'2020-10-16T18:38:21'
>>> datetime.now().isoformat(timespec='seconds', sep=' ')
'2020-10-16 18:38:35'
We can try something like below
import datetime
date_generated = datetime.datetime.now()
date_generated.replace(microsecond=0).isoformat(' ').partition('+')[0]
>>> from datetime import datetime
>>> dt = datetime.now().strftime("%Y-%m-%d %X")
>>> print(dt)
'2021-02-05 04:10:24'
f-string formatting
>>> import datetime
>>> print(f'{datetime.datetime.now():%Y-%m-%d %H:%M:%S}')
2021-12-01 22:10:07
This I use because I can understand and hence remember it better (and date time format also can be customized based on your choice) :-
import datetime
moment = datetime.datetime.now()
print("{}/{}/{} {}:{}:{}".format(moment.day, moment.month, moment.year,
moment.hour, moment.minute, moment.second))
I found this to be the simplest way.
>>> t = datetime.datetime.now()
>>> t
datetime.datetime(2018, 11, 30, 17, 21, 26, 606191)
>>> t = str(t).split('.')
>>> t
['2018-11-30 17:21:26', '606191']
>>> t = t[0]
>>> t
'2018-11-30 17:21:26'
>>>
You can also use the following method
import datetime as _dt
ts = _dt.datetime.now().timestamp()
print("TimeStamp without microseconds: ", int(ts)) #TimeStamp without microseconds: 1629275829
dt = _dt.datetime.now()
print("Date & Time without microseconds: ", str(dt)[0:-7]) #Date & Time without microseconds: 2021-08-18 13:07:09
Current TimeStamp without microsecond component:
timestamp = list(str(datetime.timestamp(datetime.now())).split('.'))[0]

Categories