I tried to convert two columns to the same format, datetime in this case.
a['sale_date'] = pd.to_datetime(a['sale_date'])
a['last_date'] = pd.to_datetime(a['last'])
a[a.last_date>a.sale_date]
When I output the dtypes they both show up as the same:
sale_date datetime64[ns]
last_date datetime64[ns]
But I get an error from the comparison of sale_date with last that says:
Invalid comparison between dtype=datetime64[ns] and method
Does this mean they are different types? Why does this not show up when I use .dtypes? Visually the outputs look comparable.
last is the name of an existing pandas method. So, it is better to avoid using last as a column name. If you can't avoid it, then you have to select the column using square brackets.
a = pd.DataFrame({'sale_date': pd.date_range('2018-04-09', periods=4, freq='3D'),
'last': pd.date_range('2018-04-12', periods=4, freq='1D')})
a[a["last"] > a.sale_date]
# sale_date last
# 0 2018-04-09 2018-04-12
# 1 2018-04-12 2018-04-13
Related
When using pd.to_datetime on my data frame I get this error:
Out of bounds nanosecond timestamp: 30-04-18 00:00:00
Now from looking on StackO I know I can simply use the coerce option:
pd.to_datetime('13000101', format='%Y%m%d', errors='coerce')
But I was wondering if anyone had an idea on how I might replace these values with a fixed value? Say 1900-01-01 00:00:00 (or maybe 1955-11-12 for anyone who gets the reference!)
Reason being that this data frame is part of a process that handles thousands and thousands of JSONs per day. I want to be able to see in the dataset easily the incorrect ones by filtering for said fixed date.
It is just as invalid for the JSON to contain any date before 2010 so using an earlier date is fine and it is also perfectly acceptable to have a blank (NA) date value so I can't rely on just blanking the data.
Replace missing values by some default datetime value in Series.mask only for missing values generated by to_datetime with errors='coerce':
df=pd.DataFrame({"date": [np.nan,'20180101','20-20-0']})
t = pd.to_datetime('1900-01-01')
date = pd.to_datetime(df['date'], format='%Y%m%d', errors='coerce')
df['date'] = date.mask(date.isna() & df['date'].notna(), t)
print (df)
date
0 NaT
1 2018-01-01
2 1900-01-01
I have a dataframe with a date column:
data['Date']
0 1/1/14
1 1/8/14
2 1/15/14
3 1/22/14
4 1/29/14
...
255 11/21/18
256 11/28/18
257 12/5/18
258 12/12/18
259 12/19/18
But, when I try to get the max date out of that column, I get:
test_data.Date.max()
'9/9/15'
Any idea why this would happen?
Clearly the column is of type object. You should try using pd.to_datetime() and then performing the max() aggregator:
data['Date'] = pd.to_datetime(data['Date'],errors='coerce') #You might need to pass format
print(data['Date'].max())
The .max() understands it as a date (like you want), if it is a datetime object. Building upon Seshadri's response, try:
type(data['Date'][1])
If it is a datetime object, this returns this:
pandas._libs.tslibs.timestamps.Timestamp
If not, you can make that column a datatime object like so:
data['Date'] = pd.to_datetime(data['Date'],format='%m/%d/%y')
The format argument makes sure you get the right formatting. See the full list of formatting options here in the python docs.
Your date may be stored as a string. First convert the column from string to datetime. Then, max() should work.
test = pd.DataFrame(['1/1/2010', '2/1/2011', '3/4/2020'], columns=['Dates'])
Dates
0 1/1/2010
1 2/1/2011
2 3/4/2020
pd.to_datetime(test['Dates'], format='%m/%d/%Y').max()
Timestamp('2020-03-04 00:00:00')
That timestamp can be cleaned up using .dt.date:
pd.to_datetime(test['Dates'], format='%m/%d/%Y').dt.date.max()
datetime.date(2020, 3, 4)
to_datetime format argument table python docs
pandas to_datetime pandas docs
I have tried many things and cannot seem to get this to work. In essence, I want to do this because an error occurs when I'm trying to convert this ndarray to a DataFrame. The following error occurs when finding missing Datetime64 values within the Dataframe:
"Out of bounds nanosecond timestamp: 1-01-01 00:00:00"
Therefore I wish to convert these DateTime64 columns into Strings and Recode '1-01-01 00:00:00' within the ndarray, then convert them back to DateTime variables in a DataFrame in order to avoid facing the error shown above.
with sRW.SavReaderNp('C:/Users/Sam/Downloads/data.sav') as reader:
record = reader.all()
prints:
[(b'61D8894E-7FB0-3DE6-E053-6C04A8C01207', 250000., '2019-08-05T00:00:00.000000',
(b'61D8894E-7FB0-3DE6-E053-6C04A8C01207', 250000., '2019-08-05T00:00:00.000000',
(b'61D8894E-7FB0-3DE6-E053-6C04A8C01207', 250000., '0001-01-01T00:00:00.000000',)]
First of all please check if your post is valid, i.e. contains runnable code.
Your example returns a syntax error and the code where you tried what you explained is simply not there.
However, I assume your data looks like
arr = [(b'61D8894E-7FB0-3DE6-E053-6C04A8C01207', 250000., '2019-08-05T00:00:00.000000'),
(b'61D8894E-7FB0-3DE6-E053-6C04A8C01207', 250000., '2019-08-05T00:00:00.000000'),
(b'61D8894E-7FB0-3DE6-E053-6C04A8C01207', 250000., '0001-01-01T00:00:00.000000')]
which looks converted to a dataframe like
df = pd.DataFrame(arr, columns=['ID', 'value', 'date'])
# ID ... date
# 0 b'61D8894E-7FB0-3DE6-E053-6C04A8C01207' ... 2019-08-05T00:00:00.000000
# 1 b'61D8894E-7FB0-3DE6-E053-6C04A8C01207' ... 2019-08-05T00:00:00.000000
# 2 b'61D8894E-7FB0-3DE6-E053-6C04A8C01207' ... 0001-01-01T00:00:00.000000
Then your attempt to convert the date strings into datetime objects was probably
df.date = pd.to_datetime(df.date)
# OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1-01-01 00:00:00
which results in the error message you posted in your question.
You can catch these parsing errors with the errors kwarg of pd.to_datetime:
df.date = pd.to_datetime(df.date, 'coerce')
# ID value date
# 0 b'61D8894E-7FB0-3DE6-E053-6C04A8C01207' 250000.0 2019-08-05
# 1 b'61D8894E-7FB0-3DE6-E053-6C04A8C01207' 250000.0 2019-08-05
# 2 b'61D8894E-7FB0-3DE6-E053-6C04A8C01207' 250000.0 NaT
Beginner python (and therefore pandas) user. I am trying to import some data into a pandas dataframe. One of the columns is the date, but in the format "YYYYMM". I have attempted to do what most forum responses suggest:
df_cons['YYYYMM'] = pd.to_datetime(df_cons['YYYYMM'], format='%Y%m')
This doesn't work though (ValueError: unconverted data remains: 3). The column actually includes an additional value for each year, with MM=13. The source used this row as an average of the past year. I am guessing to_datetime is having an issue with that.
Could anyone offer a quick solution, either to strip out all of the annual averages (those with the last two digits "13"), or to have to_datetime ignore them?
pass errors='coerce' and then dropna the NaT rows:
df_cons['YYYYMM'] = pd.to_datetime(df_cons['YYYYMM'], format='%Y%m', errors='coerce').dropna()
The duff month values will get converted to NaT values
In[36]:
pd.to_datetime('201613', format='%Y%m', errors='coerce')
Out[36]: NaT
Alternatively you could filter them out before the conversion
df_cons['YYYYMM'] = pd.to_datetime(df_cons.loc[df_cons['YYYYMM'].str[-2:] != '13','YYYYMM'], format='%Y%m', errors='coerce')
although this could lead to alignment issues as the returned Series needs to be the same length so just passing errors='coerce' is a simpler solution
Clean up the dataframe first.
df_cons = df_cons[~df_cons['YYYYMM'].str.endswith('13')]
df_cons['YYYYMM'] = pd.to_datetime(df_cons['YYYYMM'])
May I suggest turning the column into a period index if YYYYMM column is unique in your dataset.
First turn YYYYMM into index, then convert it to monthly period.
df_cons = df_cons.reset_index().set_index('YYYYMM').to_period('M')
I have a pandas dataframe as follows:
Symbol Date
A 02/20/2015
A 01/15/2016
A 08/21/2015
I want to sort it by Date, but the column is just an object.
I tried to make the column a date object, but I ran into an issue where that format is not the format needed. The format needed is 2015-02-20, etc.
So now I'm trying to figure out how to have numpy convert the 'American' dates into the ISO standard, so that I can make them date objects, so that I can sort by them.
How would I convert these american dates into ISO standard, or is there a more straight forward method I'm missing within pandas?
You can use pd.to_datetime() to convert to a datetime object. It takes a format parameter, but in your case I don't think you need it.
>>> import pandas as pd
>>> df = pd.DataFrame( {'Symbol':['A','A','A'] ,
'Date':['02/20/2015','01/15/2016','08/21/2015']})
>>> df
Date Symbol
0 02/20/2015 A
1 01/15/2016 A
2 08/21/2015 A
>>> df['Date'] =pd.to_datetime(df.Date)
>>> df.sort('Date') # This now sorts in date order
Date Symbol
0 2015-02-20 A
2 2015-08-21 A
1 2016-01-15 A
For future search, you can change the sort statement:
>>> df.sort_values(by='Date') # This now sorts in date order
Date Symbol
0 2015-02-20 A
2 2015-08-21 A
1 2016-01-15 A
sort method has been deprecated and replaced with sort_values. After converting to datetime object using df['Date']=pd.to_datetime(df['Date'])
df.sort_values(by=['Date'])
Note: to sort in-place and/or in a descending order (the most recent first):
df.sort_values(by=['Date'], inplace=True, ascending=False)
#JAB's answer is fast and concise. But it changes the DataFrame you are trying to sort, which you may or may not want.
(Note: You almost certainly will want it, because your date columns should be dates, not strings!)
In the unlikely event that you don't want to change the dates into dates, you can also do it a different way.
First, get the index from your sorted Date column:
In [25]: pd.to_datetime(df.Date).order().index
Out[25]: Int64Index([0, 2, 1], dtype='int64')
Then use it to index your original DataFrame, leaving it untouched:
In [26]: df.ix[pd.to_datetime(df.Date).order().index]
Out[26]:
Date Symbol
0 2015-02-20 A
2 2015-08-21 A
1 2016-01-15 A
Magic!
Note: for Pandas versions 0.20.0 and later, use loc instead of ix, which is now deprecated.
Since pandas >= 1.0.0 we have the key argument in DataFrame.sort_values. This way we can sort the dataframe by specifying a key and without adjusting the original dataframe:
df.sort_values(by="Date", key=pd.to_datetime)
Symbol Date
0 A 02/20/2015
2 A 08/21/2015
1 A 01/15/2016
The data containing the date column can be read by using the below code:
data = pd.csv(file_path,parse_dates=[date_column])
Once the data is read by using the above line of code, the column containing the information about the date can be accessed using pd.date_time() like:
pd.date_time(data[date_column], format = '%d/%m/%y')
to change the format of date as per the requirement.
data['Date'] = data['Date'].apply(pd.to_datetime) # non-null datetime64[ns]