I have a variable in a df that looks like this
Datetime
10/27/2020 2:28:28 PM
8/2/2020 3:30:18 AM
6/15/2020 5:38:19 PM
How can I change it to this using python?
Date Time
10/27/2020 14:28:28
8/2/2020 3:30:18
6/15/2020 17:38:19
I understand how to separate date and time, but unsure of how to convert it to 24 hour time.
I think this is source you want:
from dateutil.parser import parse
dt = parse("10/27/2020 2:28:28")
print(dt)
# 2020-10-27 02:28:28
# Create Date
date=f"{str(dt.year)}/{str(dt.month)}/{str(dt.day)}"
# Create Time
time=f"{str(dt.hour)}:{str(dt.minute)}:{str(dt.second)}"
You can use pd.to_datetime to convert a scalar, array-like, Series or DataFrame/dict-like to a pandas datetime object. Then, you can use the accessor object for datetimelike properties of the Series values (Series.dt()) to obtain the time, that will be already in the desired format.
You can also use dt.strftime to format the output string which supports the same string format as the python standard library.
df['Datetime'] = pd.to_datetime(df.Datetime)
df['Date'] = df.Datetime.dt.strftime('%m/%d/%Y')
df['Time'] = df.Datetime.dt.time
print(df)
Datetime Date Time
0 2020-10-27 14:28:28 10/27/2020 14:28:28
1 2020-08-02 03:30:18 08/02/2020 03:30:18
2 2020-06-15 17:38:19 06/15/2020 17:38:19
Related
I'm trying to convert a column in a dataframe to timeseries, the values in the column are strings and they are in the following form:
12/10/202110:42:05.397
which means 12-10-2021 at 10:42:05 and 397 milliseconds. This is the format that Labview is saving the data into a file.
I'm trying to use the following command, but I can't figure out how to define the format for my case:
pd.to_datetime(df.DateTime, format=???)
Note that there is no space between year 2021 and hour 10
Use:
df['dt'] = pd.to_datetime(df['DateTime'], format='%d/%m/%Y%H:%M:%S.%f')
print(df)
# Output
DateTime dt
0 12/10/202110:42:05.397 2021-10-12 10:42:05.397
Setup:
df = pd.DataFrame({'DateTime': ['12/10/202110:42:05.397']})
As suggested by #RaymondKwok, use the documentation:
strftime() and strptime() Format Codes
I am dealing with a csv file containing a column called startTime, containing times.
When opening this file with Excel, the times appear as AM/PM times in the formula bar, although the timestamps in the column appear improperly formatted:
startTime
16:02.0
17:45.0
18:57.0
20:23.0
When reading this file using pandas' read_csv, I am unable to format these timestamps properly:
import pandas as pd
df = pd.read_csv('example_file.csv')
print(df.startTime)
Simply yields:
0 16:02.0
1 17:45.0
2 18:57.0
3 20:23.0
I first attempted to convert the output Series using pd.to_datetime(df.startTime,format=" %H%M%S") but this yields the following error message:
time data '16:02.0' does not match format ' %H%M%S' (match)
I then tried pd.to_datetime(df.startTime,format=" %I:%M:%S %p") based on this answer, in order to account for the AM/PM convention, but this returned the same error message.
How can I use pandas to format these timestamps like Excel automatically does?
Your csv file has text, not datetime, so you need to first convert text stored in this column to pandas datetime object, then you can convert this pandas datetime object to the kind of format that you want via a strftime method:
pd.to_datetime(df['startTime']).dt.strftime(date_format = '%I:%M:%S %p')
Outputs:
0 04:02:00 PM
1 05:45:00 PM
2 06:57:00 PM
3 08:23:00 PM
Note: these values are string values, not datetime.
Edit for this specific issue:
A quick format to add 00h to your timestamp before converting to get midnight AM:
pd.to_datetime(df['startTime'].apply(lambda x: f'00:{x}')).dt.strftime(date_format = '%I:%M:%S %p')
Outputs:
0 00:16:02 AM
1 00:17:45 AM
2 00:18:57 AM
3 00:20:23 AM
Try:
>>> pd.to_datetime(df['startTime'].str.strip(), format='%H:%M.%S')
0 1900-01-01 16:02:00
1 1900-01-01 17:45:00
2 1900-01-01 18:57:00
3 1900-01-01 20:23:00
Name: startTime, dtype: datetime64[ns]
Coerce to datetetime and extract time using dt.strftime
df['startTime']=pd.to_datetime(df['startTime']).dt.strftime('%I:%M.%S%p')
I work with data from Datalogger and the timestap is not supported by datetime in the Pandas Dataframe.
I would like to convert this timestamp into a format pandas knows and the then convert the datetime into seconds, starting with 0.
>>>df.time
0 05/20/2019 19:20:27:374
1 05/20/2019 19:20:28:674
2 05/20/2019 19:20:29:874
3 05/20/2019 19:20:30:274
Name: time, dtype: object
I tried to convert it from the object into datetime64[ns]. with %m or %b for month.
df_time = pd.to_datetime(df["time"], format = '%m/%d/%y %H:%M:%S:%MS')
df_time = pd.to_datetime(df["time"], format = '%b/%d/%y %H:%M:%S:%MS')
with error: redefinition of group name 'M' as group 7; was group 5 at position 155
I tried to reduce the data set and remove the milliseconds without success.
df['time'] = pd.to_datetime(df['time'],).str[:-3]
ValueError: ('Unknown string format:', '05/20/2019 19:20:26:383')
or is it possible to just subtract the first time line from all the other values in the column time?
Use '%m/%d/%Y %H:%M:%S:%f' as format instead of '%m/%d/%y %H:%M:%S:%MS'
Here is the format documentation for future reference
I am not exactly sure what you are looking for but you can use the above example to format your output and then you can remove items from your results like the microseconds this way:
date = str(datetime.now())
print(date)
2019-07-28 14:04:28.986601
print(date[11:-7])
14:04:28
time = date[11:-7]
print(time)
14:04:28
I am working with an Excel file in Pandas where I am trying to deal with a
Date column where the Date is listed in ISO 8601 format. I want to take this column and store the date and time in two different columns.The values in these two columns need to be stored in Eastern Daylight Savings. This is what they are supposed to look like
Date Date (New) Time (New)
1999-01-01T00:00:29.75 12/31/1998 6:59:58 PM
1999-01-01T00:00:30.00 12/31/1998 6:59:59 PM
1999-01-01T00:00:32.25 12/31/1998 7:00:00 PM
1999-01-01T00:00:30.50 12/31/1998 6:59:58 PM
I have achieved this, partially.
I have converted the values to Eastern Daylight savings time and successfully stored the Date value correctly. However, I want the time value to be stored in the 12 hours format and not in the 24 hours format as it is being right now?
This is what my output looks like so far.
Date Date (New) Time (New)
1999-01-01T00:00:29.75 1998-12-31 19:00:30
1999-01-01T00:00:30.00 1998-12-31 19:00:30
1999-01-01T00:00:32.25 1998-12-31 19:00:32
1999-01-01T00:00:30.50 1998-12-31 19:00:31
Does anyone have any idea what i can do for this?
from pytz import timezone
import dateutil.parser
from pytz import UTC
import datetime as dt
df3['Day']=pd.to_datetime(df['Date'], format='%Y-%m-%d %H:%M: %S.%f',errors='coerce').dt.tz_localize('UTC')
df3['Day']= df3['Day'].dt.tz_convert('US/Eastern')
df3['Date(New)'], df3['Time(New)'] = zip(*[(d.date(), d.time()) for d in df3['Day']])
You should use d.time().strftime("%I:%M:%S %p") which will format the date as requested.
strftime() and strptime() Behavior
You can set the time format used for outputting - the time value itself is (and should be) stored as datetime.time() - if you want a specific string representation you can create a string-type column in the format you want:
from pytz import timezone
import pandas as pd
import datetime as dt
df= pd.DataFrame([{"Date":dt.datetime.now()}])
df['Day']=pd.to_datetime( df['Date'], format='%Y-%m-%d %H:%M: %S.%f',
errors='coerce').dt.tz_localize('UTC')
df['Day']= df['Day'].dt.tz_convert('US/Eastern')
df['Date(New)'], df['Time(New)'] = zip(*[(d.date(), d.time()) for d in df['Day']])
# create strings with specific formatting
df['Date(asstring)'] = df['Day'].dt.strftime("%Y-%m-%d")
df['Time(asstring)'] = df["Day"].dt.strftime("%I:%M:%S %p")
# show resulting column / cell types
print(df.dtypes)
print(df.applymap(type))
# show df
print(df)
Output:
# df.dtypes
Date datetime64[ns]
Day datetime64[ns, US/Eastern]
Date(New) object
Time(New) object
Date(asstring) object
Time(asstring) object
# from df.applymap(type)
Date <class 'pandas._libs.tslib.Timestamp'>
Day <class 'pandas._libs.tslib.Timestamp'>
Date(New) <class 'datetime.date'>
Time(New) <class 'datetime.time'>
Date(asstring) <class 'str'>
Time(asstring) <class 'str'>
# from print(df)
Date Day Date(New) Time(New)
0 2019-01-04 00:40:02.802606 2019-01-03 19:40:02.802606-05:00 2019-01-03 19:40:02.802606
Date(asstring) Time(asstring)
2019-01-03 07:40:02 PM
It looks like you are very close. %H is the 24 hour format. You should use %I instead.
How can I account for period (AM/PM) with datetime.strptime?
I have a data frame with a column with values in millisecond time stamp.(df['Millisecond'])
What I want to do is to convert all values of that column into normal dates. Ex. 2017-04-27 04:55:00
You need python's datetime package to do that:
import datetime
date = datetime.datetime.fromtimestamp(milliseconds/1000.0)
date = date.strftime('%Y-%m-%d %H:%M:%S')
you can do this by using to_datetime function https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html.
df['Millisecond'] = pd.to_datetime(df['Millisecond'], unit='ms')