Remove date from datatime index in pandas - python

I want to remove the date from datetime function in pandas and the following code works just fine.
df= pd.read_csv('data.csv')
df['Value']= df.Value.astype(float)
df['Time'] = pd.to_datetime(df['Time']).dt.time
df.set_index('Time',inplace=True)
But after that when I try to select rows based on the time using .loc function it gives me the following error.
df_to_plot = df.loc['09:43:00':'13:54:00']
TypeError: '<' not supported between instances of 'datetime.time' and 'str'
But the same code works fine without .dt.time as follows:
df= pd.read_csv('data.csv')
df['Value']= df.Value.astype(float)
df['Time'] = pd.to_datetime(df['Time'])
df.set_index('Time',inplace=True)
df_to_plot = df.loc['2022-07-28 09:43':'2022-07-28 13:54']
How can I remove date and still select rows based on time?
Thank you.

The TypeError arrises because df['Time'] = pd.to_datetime(df['Time']).dt.time turns df['Time'] into a datetime.time object, whereas in your loc statement, '09:43:00':'13:54:00' is a string.
Try this:
df['Time'] = pd.to_datetime(df['Time']).dt.time.astype(str)

try using df.index = df.index.time

Related

How to convert datetime.time into datetime.date

I have a dataframe called pomi that looks like this
date time sub
2019-09-20 00:00:00 25.0 org
I want to convert the values in the column 'date' to datetime.date, so that I'm left with only the dates (ie '2019-09-20').
I have tried:
pomi['date'] = pd.to_datetime(pomi['date'])
pomi['just_date'] = pomi['date'].dt.date
pomi.date = pd.to_datetime(pomi.date,dayfirst=True)
pomi['date'] = pd.to_datetime(pomi["date"].astype(str)).dt.time
pomi['date'] = pd.to_datetime(pomi['date']).dt.date
pomi['date'] = pd.to_datetime(pomi['date']).dt.normalize()
None of them have worked.
Most often I get the error message "TypeError: <class 'datetime.time'> is not convertible to datetime"
All help appreciated. Thanks.
Full disclosure, I am not 100% sure what is the issue, your code was working fine at my end. But there is something you can try as convert to Timestamp & than check. This & your code both works at my end giving required out.
import pandas as pd
df = pd.DataFrame({'date': ['2019-09-20 00:00:00'], 'time':[25], 'sub':['org']})
df['date'] = df['date'].apply(pd.Timestamp)
df['just_date'] = df['date'].dt.date
df

How do I make a column with iso weeks?

I have a data frame (df) with a column of dates (DATUM).
i then try to make a new column with iso weeks on the dates.
But as a beginner in python, I have run into a problem.
When I try to use:
df ['iso_week_num'] = df ["DATUM"]. isocalendar () [1]
I get the following error message:
AttributeError: 'Series' object has no attribute 'isocalendar'
What am I doing wrong?
Notice that isocalendar must be applied to a timestamp (check documentation) - it is not an attribute as the error raised. Said that, this should work for your case:
# Generating some data (you may use rand too):
dates = ['2021-01-03', '2021-01-04', '2021-01-05', '2021-01-06', '2021-01-07']
dataframe = pd.DataFrame(data = dates, columns = ['date'])
dataframe['date'] = pd.to_datetime(dataframe['date'])
# Applying isocalendar for each element in the series:
dataframe['year'] = dataframe['date'].apply(lambda x: x.isocalendar()[0])
You should use pandas.Series.dt() to access object for datetimelike properties of the Series values.
df['week'] = df['date'].dt.isocalendar().week

py pd DataFrame Timestamp to string conversion error (ValueError: cannot set a Timestamp with a non-timestamp str)

Last week the below code worked well to convert timestamp to string within a DataFrame:
df.at[i, 'VB12.GMA_DOC']
Timestamp('2022-01-12 00:00:00')
len_df = len(df.index)
df['GMA_DOC'] = ''
for i in range(0,len_df):
df.at[i, 'VB12.GMA_DOC'] = df.at[i, 'VB12.GMA_DOC'].strftime('%Y-%m-%d')
Today, no changes to libraries or other parts of the code, I have the error:
ValueError: cannot set a Timestamp with a non-timestamp str
I noticed that directly from the shell there is no problem:
df.at[i, 'VB12.GMA_DOC'].strftime('%Y-%m-%d')
'2022-01-12'
After some tentatives I solved modifying the code as below:
len_df = len(df.index)
df['GMA_DOC'] = ''
for i in range(0,len_df):
df.at[i, 'GMA_DOC'] = df.at[i, 'VB12.GMA_DOC'].strftime('%Y-%m-%d')
del df['VB12.GMA_DOC']
df['VB12.GMA_DOC'] = df['GMA_DOC']
del df['GMA_DOC']
The problem apparently is the direct assignment of the df_string to the previous df_timestamp column.
Is that normal or do you see a better solution to avoid the error ?
I think the problem is that the type of your column is a Timestamp and you try to add a string to it. pandas tries to convert the string to a Timestamp, but it is unable to do so. In order to change both the values and the data type in one go, I'd recommend using a vectorized solution:
import pandas as pd
# Create dataframe with the timestamp values
df = pd.DataFrame(data=[{'VB12.GMA_DOC':'2022-01-12 00:00:01'}, {'VB12.GMA_DOC':'2022-01-11 00:00:00'}])
df['VB12.GMA_DOC'] = pd.to_datetime(df['VB12.GMA_DOC'], format="%Y-%m-%d %H:%M:%S")
print(df.dtypes) # datetime64[ns]
# Change timestamps to str
df['VB12.GMA_DOC'] = df['VB12.GMA_DOC'].dt.strftime('%Y-%m-%d')
print(df.dtypes) # object
df
Output:
VB12.GMA_DOC
0 2022-01-12
1 2022-01-11

Converting dates to datetime64 results in day and month places getting swapped

I am pulling a time series from a csv file which has dates in "mm/dd/yyyy" format
df = pd.read_csv(lib_file.csv)
df['Date'] = df['Date'].apply(lambda x:datetime.strptime(x,'%m/%d/%Y').strftime('%d/%m/%Y'))
below is the output
I convert dtypes for ['Date'] from object to datetime64
df['Date'] = pd.to_datetime(df['Date'])
but that changes my dates as well
how do I fix it?
Try this:
df['Date'] = pd.to_datetime(df['Date'], infer_datetime_format=True)
This will infer your dates based on the first non-NaN element which is being correctly parsed in your case and will not infer the format for each and every row of the dataframe.
just using the below code helped
df = pd.read_csv(lib_file.csv)
df['Date'] = pd.to_datetime(df['Date])

Pandas Dataframe asFreq changing datatype of index

I'm having an issue when using asfreq to resample a dataframe. My dataframe, df, has an index of type Datetime.Date(). After using df.asfreq('d','pad'), my dataframe index has been changed to type pandas.tslib.Timestamp. I've tried the following to change it back but I'm having no luck...
df = df.set_index(df.index.to_datetime())
df.index = df.index.to_datetime()
df.index = pd.to_datetime(df.index)
Any thoughts?
Thanks!
use pd.to_datetime
df.index = pd.to_datetime(df.index)
This is the canonical approach to creating datetime indices. If you want your index indices to all be of type datetime.datetime then you can do this following.
df.index = pd.Index([i.to_datetime() for i in df.index], name=df.index.name, dtype=object)
I just don't know why you'd want to.
Why is this a problem? If you really need a datetime.date you can try df.index = df.index.map(lambda x: x.date() since pandas.TimeStamp subclasses datetime.datetime

Categories