Hey so I have seen several questions about this, however, I have yet to successful solve my problem.
I have a single column Time in the format:
2014-07-17 00:59:27.400189+00
I want to split this into a two columns, Date and Hour.
I used
posts['Date']=pd.to_datetime(posts['Time'],format='%Y-%m-%d %H:%M:%S')
However, I get an error
ValueError: unconverted data remains: 400189+00
I am not sure what to label the last bit of information. I tried added %o but received another error
ValueError: 'o' is a bad directive in format '%Y-%m-%d %H:%M:%S.%o'
Any ideas on how I can split these two values into two columns?
Thanks!
the following worked for me:
In [18]:
import pandas as pd
df = pd.DataFrame({'Date':['2014-07-17 00:59:27.400189+00']})
df.dtypes
Out[18]:
Date object
dtype: object
In [19]:
df['Date'] = pd.to_datetime(df['Date'])
df.dtypes
Out[19]:
Date datetime64[ns]
dtype: object
In [20]:
df['Time'],df['Date']= df['Date'].apply(lambda x:x.time()), df['Date'].apply(lambda x:x.date())
df
Out[20]:
Date Time
0 2014-07-17 00:59:27.400189
[1 rows x 2 columns]
This worked for me
import pandas as pd
data = pd.DataFrame({'Date':['2014-07-17 00:59:27.400189+00']})
data['Dates'] = pd.to_datetime(data['Date'], format='%Y:%M:%D').dt.date
data['Hours'] = pd.to_datetime(data['Date'], format='%Y:%M:%D').dt.time
You have to have
print(data)
Dates Hours
2014-07-17 00:59:27.400189+00
import pandas as pd
data = pd.DataFrame({'Date':['2014-07-17 00:59:27.400189+00']})
data['Dates'] = pd.to_datetime(data['Date'], format='%Y:%M:%D').dt.date
data['Hours'] = pd.to_datetime(data['Date'], format='%Y:%M:%D').dt.time
This gives me object type Date and Time. The expected column should be in date format
Related
I have a dataframe called pomi that looks like this
date time sub
2019-09-20 00:00:00 25.0 org
I want to convert the values in the column 'date' to datetime.date, so that I'm left with only the dates (ie '2019-09-20').
I have tried:
pomi['date'] = pd.to_datetime(pomi['date'])
pomi['just_date'] = pomi['date'].dt.date
pomi.date = pd.to_datetime(pomi.date,dayfirst=True)
pomi['date'] = pd.to_datetime(pomi["date"].astype(str)).dt.time
pomi['date'] = pd.to_datetime(pomi['date']).dt.date
pomi['date'] = pd.to_datetime(pomi['date']).dt.normalize()
None of them have worked.
Most often I get the error message "TypeError: <class 'datetime.time'> is not convertible to datetime"
All help appreciated. Thanks.
Full disclosure, I am not 100% sure what is the issue, your code was working fine at my end. But there is something you can try as convert to Timestamp & than check. This & your code both works at my end giving required out.
import pandas as pd
df = pd.DataFrame({'date': ['2019-09-20 00:00:00'], 'time':[25], 'sub':['org']})
df['date'] = df['date'].apply(pd.Timestamp)
df['just_date'] = df['date'].dt.date
df
I am pulling a time series from a csv file which has dates in "mm/dd/yyyy" format
df = pd.read_csv(lib_file.csv)
df['Date'] = df['Date'].apply(lambda x:datetime.strptime(x,'%m/%d/%Y').strftime('%d/%m/%Y'))
below is the output
I convert dtypes for ['Date'] from object to datetime64
df['Date'] = pd.to_datetime(df['Date'])
but that changes my dates as well
how do I fix it?
Try this:
df['Date'] = pd.to_datetime(df['Date'], infer_datetime_format=True)
This will infer your dates based on the first non-NaN element which is being correctly parsed in your case and will not infer the format for each and every row of the dataframe.
just using the below code helped
df = pd.read_csv(lib_file.csv)
df['Date'] = pd.to_datetime(df['Date])
I have a pandas dataframe that has a column of type int64 but this columns represets date, e.g. 20180501. I'd like to convert this column to datetime and I'm having the following code but it returns an error message
df['new_date'] = pd.to_datetime(df['old_date'].astype('str'), format = '%y%m%d')
I'm getting the following error message
ValueError: unconverted data remains: 0501
How can I fix my code?
You need a capital Y. See Python's strftime directives for a complete reference.
df = pd.DataFrame({'old_date': [20180501, 20181230, 20181001]})
df['new_date'] = pd.to_datetime(df['old_date'].astype(str), format='%Y%m%d')
print(df)
old_date new_date
0 20180501 2018-05-01
1 20181230 2018-12-30
2 20181001 2018-10-01
It could be that the problem arises due to a format error at some places in the dataframe.
You could try setting the parameter errors="coerce" to avoid converting those entries and setting them to NaT.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html
I have a dataframe with strings that I am converting to datetimes. They all look like "12/20/17 5:45:30" (month/day/year hour:minute:second). This is my code:
for col in cols:
df[col] = pd.to_datetime(df[col], format='%m/%d/%Y %H:%M:%S')
But I get the following error:
ValueError: time data '4/19/16 1:05:30' does not match format '%m/%d/%Y %H:%M:%S'
The date shown in the error is the very first date in the dataframe, so it is not working at all. Can someone explain what's wrong with my datetime format? How does that datetime not match the format? By the way, before I was doing this with a file that had no seconds, and my format was %m/%d/%Y %H:%M, which worked fine, but now with seconds it does not.
Your format string is not working because your format uses a Y where it needed a y. But pandas to the rescue, it can often figure this stuff out for you by using the infer_datetime_format parameter to pandas.to_datetime()
Code:
df[col] = pd.to_datetime(df[col], infer_datetime_format=True)
Test Code:
df = pd.DataFrame(["12/20/17 5:45:30", "4/19/16 1:05:30"], columns=['date'])
print(df)
for col in df.columns:
df[col] = pd.to_datetime(df[col], infer_datetime_format=True)
print(df)
Results:
date
0 12/20/17 5:45:30
1 4/19/16 1:05:30
date
0 2017-12-20 05:45:30
1 2016-04-19 01:05:30
I am trying to find the day difference between today, and dates in my dataframe.
Below is my conversion of dates in my dataframe
df['Date']=pd.to_datetime(df['Date'])
Below is my code to get today
today1=dt.datetime.today().strftime('%Y-%m-%d')
today1=pd.to_datetime(today1)
Both are converted to pandas.to_datetime, but when I do subtraction, the below error came out.
ValueError: Cannot add integral value to Timestamp without offset.
Can someone help to advise? Thanks!
This is a simple example how you can do this:
import pandas
import datetime as dt
First, you have to get today.
today1=dt.datetime.today().strftime('%Y-%m-%d')
today1=pd.to_datetime(today1)
Then, you can construct the data frame:
df = pandas.DataFrame({'Date':'2016-11-24 11:03:10.050000', 'today1': today1 }, index = [0])
In this example I just have 2 columns, each with one value.
Next, you should check the data types:
print(df.dtypes)
Date datetime64[ns]
today1 datetime64[ns]
If both data types are datetime64[ns], you can then subtract df.Date from df.today1.
print(df.today1 - df.Date)
The output:
0 19 days 12:56:49.950000
dtype: timedelta64[ns]