We Keep Coding

Python-PANDAS dataframe- Datetime format

I am trying to get a specific format i.e. date-month-year eg: 01-12-2023 12:00:00;00 AM.
In my dataframe i have a column which is in a datetime format i.e. 2023-12-01.
Here i tried to convert the datetime format into the format that i needed by using strftime ,however didnt want the datatype to be changed so after converting using strftime , converted back again in datetime.
After doing so , again it went back datetime format i.e. 2023-12-01 instead of 01-12-2023 12:00:00;00 AM.
How can i resolve this by getting the required format and not changing the datatype.
Checked few examples but didnt work.
Code:
Below code creates a new column DOB1 from the Month column which has a datetime format :
Dataframe bkp:
Month
2022-05-01
2023-06-01
bkp['DOB1'] = bkp['Month'].dt.strftime('%d-%m-%Y %I:%M:%S %p')
Output:
Month DOB1
2022-05-01 01-05-2022 12:00:00:00 AM
2023-06-01 01-06-2023 12:00:00:00 AM
bkp.dtypes >>>> Month : datetime64[ns] , DOB1 : object
# Converting to datetime format , to retain the original format
bkp['DOB1'] = pd.to_datetime(bkp['DOB1'])
Output:
Month DOB1
2022-05-01 2022-05-01
2023-06-01 2023-06-01
Could anyone suggest how to get the required format and have the datetime format still.

How to read AM/PM times in pandas?

I am dealing with a csv file containing a column called startTime, containing times.
When opening this file with Excel, the times appear as AM/PM times in the formula bar, although the timestamps in the column appear improperly formatted:
startTime
16:02.0
17:45.0
18:57.0
20:23.0
When reading this file using pandas' read_csv, I am unable to format these timestamps properly:
import pandas as pd
df = pd.read_csv('example_file.csv')
print(df.startTime)
Simply yields:
0 16:02.0
1 17:45.0
2 18:57.0
3 20:23.0
I first attempted to convert the output Series using pd.to_datetime(df.startTime,format=" %H%M%S") but this yields the following error message:
time data '16:02.0' does not match format ' %H%M%S' (match)
I then tried pd.to_datetime(df.startTime,format=" %I:%M:%S %p") based on this answer, in order to account for the AM/PM convention, but this returned the same error message.
How can I use pandas to format these timestamps like Excel automatically does?

Your csv file has text, not datetime, so you need to first convert text stored in this column to pandas datetime object, then you can convert this pandas datetime object to the kind of format that you want via a strftime method:
pd.to_datetime(df['startTime']).dt.strftime(date_format = '%I:%M:%S %p')
Outputs:
0 04:02:00 PM
1 05:45:00 PM
2 06:57:00 PM
3 08:23:00 PM
Note: these values are string values, not datetime.
Edit for this specific issue:
A quick format to add 00h to your timestamp before converting to get midnight AM:
pd.to_datetime(df['startTime'].apply(lambda x: f'00:{x}')).dt.strftime(date_format = '%I:%M:%S %p')
Outputs:
0 00:16:02 AM
1 00:17:45 AM
2 00:18:57 AM
3 00:20:23 AM

Try:
>>> pd.to_datetime(df['startTime'].str.strip(), format='%H:%M.%S')
0 1900-01-01 16:02:00
1 1900-01-01 17:45:00
2 1900-01-01 18:57:00
3 1900-01-01 20:23:00
Name: startTime, dtype: datetime64[ns]

Coerce to datetetime and extract time using dt.strftime
df['startTime']=pd.to_datetime(df['startTime']).dt.strftime('%I:%M.%S%p')

csv Pandas datetime convert time to seconds

I work with data from Datalogger and the timestap is not supported by datetime in the Pandas Dataframe.
I would like to convert this timestamp into a format pandas knows and the then convert the datetime into seconds, starting with 0.
>>>df.time
0 05/20/2019 19:20:27:374
1 05/20/2019 19:20:28:674
2 05/20/2019 19:20:29:874
3 05/20/2019 19:20:30:274
Name: time, dtype: object
I tried to convert it from the object into datetime64[ns]. with %m or %b for month.
df_time = pd.to_datetime(df["time"], format = '%m/%d/%y %H:%M:%S:%MS')
df_time = pd.to_datetime(df["time"], format = '%b/%d/%y %H:%M:%S:%MS')
with error: redefinition of group name 'M' as group 7; was group 5 at position 155
I tried to reduce the data set and remove the milliseconds without success.
df['time'] = pd.to_datetime(df['time'],).str[:-3]
ValueError: ('Unknown string format:', '05/20/2019 19:20:26:383')
or is it possible to just subtract the first time line from all the other values in the column time?

Use '%m/%d/%Y %H:%M:%S:%f' as format instead of '%m/%d/%y %H:%M:%S:%MS'
Here is the format documentation for future reference

I am not exactly sure what you are looking for but you can use the above example to format your output and then you can remove items from your results like the microseconds this way:
date = str(datetime.now())
print(date)
2019-07-28 14:04:28.986601
print(date[11:-7])
14:04:28
time = date[11:-7]
print(time)
14:04:28

Pandas convert datetime string column to datetime without offset applied

I'm new to Python and Pandas, so dont be hard with me :)
I have multiple Columns in the form of "2014-01-01 00:00:00-06:00". Now i want to convert the columns name into a pandas datetime. But i struggle with the format i need to use. I already tried
date = pd.to_datetime("2014-01-01 00:00:00-06:00", format='%Y-%m-%d %H:%M:%S%z')
But here i get a error with "ValueError: time data '2014-01-01 00:00:00-06:00' does not match format '%Y-%m-%d %H:%M:%S%Z' (match)"
I dont want the time to get converted into my timezone. I need it for the Timezone -06:00
For this Input:
2014-01-01 00:00:00-06:00
The Output should be:
2014-01-01 00:00:00
I want to use the date variable of the Output so i can split my data into seasons. Something like this:
date > springBegining
Thanks for all help

You don't need a format string, pandas is man/woman enough to handle this:
In[2]:
pd.to_datetime('2014-01-01 00:00:00-06:00')
Out[2]: Timestamp('2014-01-01 06:00:00')
besides your format string has numerous issues:
%b is month as locale abbreviated form, you have a numerical representation so it should be %m
%z requires a UTC offset in the form '+HHMM'/-HHMM
So you'd need to reformat the datetime string to:
'2014-01-01 00:00:00-0600'
If you don't want the offset to be applied and the offset is always the same you can strip this from the string:
In[25]:
pd.to_datetime('2014-01-01 00:00:00-06:00'.rsplit('-',1)[0])
Out[25]: Timestamp('2014-01-01 00:00:00')
Or you could slice the string:
In[26]:
pd.to_datetime('2014-01-01 00:00:00-06:00'[:-6])
Out[26]: Timestamp('2014-01-01 00:00:00')
So to do the above on an entire column:
pd.to_datetime(df[col].str[:-6])
Example:
In[27]:
df = pd.DataFrame({'date':['2014-01-01 00:00:00-06:00','2014-01-01 00:00:00+06:00']})
df
Out[27]:
date
0 2014-01-01 00:00:00-06:00
1 2014-01-01 00:00:00+06:00
In[28]:
pd.to_datetime(df['date'].str[:-6])
Out[28]:
0 2014-01-01
1 2014-01-01
Name: date, dtype: datetime64[ns]
Here we use the string accessor .str to slice all the columns in the same manner and pass this to to_datetime to convert the entire column

python parse string in date format to get the date

There is a string and a date format. I want to get the date based on format.
If date format is YYYY.MM.dd and string is 2017.01.01. It should transform to a valid date object.
How can I find the date.

You can use datetime module something like this :
from datetime import datetime
date_object = datetime.strptime('2017.01.01', '%Y.%m.%d') # Converting the given date string into a datetime object.
formatted_date = date_object.strftime('%c') #User Defined Output Format
print(formatted_date)
This will result in :
Sun Jan 1 00:00:00 2017
You can refer to the documentation here.