We Keep Coding

Does auto arima predict function return series

I have been following this tutorial:
https://www.analyticsvidhya.com/blog/2021/11/basic-understanding-of-time-series-modelling-with-auto-arimax/
I was fitting it on my dataset.
It was working fine but when I ran it again the next day model.predict function started returning series rather than a list. And it is not giving proper results like it did previously.
model = pm.auto_arima(df_train["farm rate"], exogenous=df_train[exogenous_features], trace=True, error_action="ignore", suppress_warnings=True)
model.fit(df_train["farm rate"], exogenous=df_train[exogenous_features])
forecast = model.predict(n_periods=len(df_valid), exogenous=df_valid[exogenous_features])
df_valid["Forecast_ARIMAX"] = forecast
My df looks like this
date day doc farm rate open close price month year day_of_week week doc_mean_3 open_mean_3 close_mean_3 price_mean_3
date
2015-01-01 2015-01-01 1.0 51.5 165.0 170.0 170.0 2375.0 1.0 2015.0 3.0 1.0 38.574580 155.158973 150.355472 3418.872350
2015-01-02 2015-01-02 2.0 48.5 165.0 170.0 160.0 2375.0 1.0 2015.0 4.0 1.0 38.574580 155.158973 150.355472 3418.872350
2015-01-03 2015-01-03 3.0 44.5 150.0 180.0 167.5 2375.0 1.0 2015.0 5.0 1.0 48.166667 173.333333 165.833333 2375.000000
2015-01-05 2015-01-05 5.0 40.5 150.0 165.0 155.0 2375.0 1.0 2015.0 0.0 2.0 44.500000 171.666667 160.833333 2375.000000
2015-01-06 2015-01-06 6.0 36.5 140.0 155.0 152.5 2375.0 1.0 2015.0 1.0 2.0 40.500000 166.666667 158.333333 2375.000000
2015-01-07 2015-01-07 7.0 35.5 140.0 155.0 150.0 2375.0 1.0 2015.0 2.0 2.0 37.500000 158.333333 152.500000 2375.000000
2015-01-08 2015-01-08 8.0 35.5 140.0 162.5 155.0 2375.0 1.0 2015.0 3.0 2.0 35.833333 157.500000 152.500000 2375.000000
2015-01-09 2015-01-09 9.0 37.5 145.0 165.0 157.5 2375.0 1.0 2015.0 4.0 2.0 36.166667 160.833333
I have data from 2015 to 2022.Exogenous features are : ["doc", "open","close","doc_mean_3", "open_mean_3", "close_mean_3", "price_mean_3"]
They are calculated using this method:
def calc_lag_features(df, feature, window):
df[f"{feature}_mean_{window}"] = df[feature].rolling(window=window).mean()
return df
Edit:
These are the results I get. Completely flat graph. Adding or removing exogenous variables isnt making any difference. Previously, it was able to capture the true graph to quite an extent.
OUTPUT
Depending on exogenous variables straight line only goes up and down and is not able to capture the true trend.

View data after transformation

Is it possible to see the data after Altair applies transformations and aggregations?
For example, can you access the underlying data after the following transformations?
import altair as alt
from vega_datasets import data
source = data.seattle_weather.url
step = 20
overlap = 1
alt.Chart(source, height=step).transform_timeunit(
Month='month(date)'
).transform_joinaggregate(
mean_temp='mean(temp_max)', groupby=['Month']
).transform_bin(
['bin_max', 'bin_min'], 'temp_max'
).transform_aggregate(
value='count()', groupby=['Month', 'mean_temp', 'bin_min', 'bin_max']
).transform_impute(
impute='value', groupby=['Month', 'mean_temp'], key='bin_min', value=0
).mark_area(...
)
Above code from the Ridgeplot example

Transforms are evaluated in Javascript, and there is not any built-in way to access data in the javascript frontend from the Python backend. However, there is an experimental package called altair_transform that is able to evaluate most Vega expressions in Python.
For your chart you can use it like this:
import altair as alt
from vega_datasets import data
source = data.seattle_weather()
step = 20
overlap = 1
chart = alt.Chart(source, height=step).transform_timeunit(
Month='month(date)'
).transform_joinaggregate(
mean_temp='mean(temp_max)', groupby=['Month']
).transform_bin(
['bin_max', 'bin_min'], 'temp_max'
).transform_aggregate(
value='count()', groupby=['Month', 'mean_temp', 'bin_min', 'bin_max']
).transform_impute(
impute='value', groupby=['Month', 'mean_temp'], key='bin_min', value=0
).mark_area().encode(
x='Month:T',
y='value:Q'
)
import altair_transform
data = altair_transform.extract_data(chart)
print(data)
bin_min Month mean_temp bin_max value
0 0.0 1900-01-01 8.229032 -5.0 2.0
1 5.0 1900-01-01 8.229032 0.0 19.0
2 10.0 1900-01-01 8.229032 5.0 72.0
3 15.0 1900-01-01 8.229032 10.0 29.0
4 20.0 1900-01-01 8.229032 15.0 2.0
.. ... ... ... ... ...
103 20.0 1900-12-01 8.194355 15.0 4.0
104 25.0 1900-12-01 8.194355 NaN 0.0
105 30.0 1900-12-01 8.194355 NaN 0.0
106 35.0 1900-12-01 8.194355 NaN 0.0
107 40.0 1900-12-01 8.194355 NaN 0.0
[108 rows x 5 columns]

How to disable calculating with nans while pandas resample().mean() and resample().sum()?

I need to calculate the annual mean from monthly data. If there is a nan value in my monthly data, I want the whole year to be nan as well.
This is my code so far:
station_data = pd.read_csv(station_data_files[0], sep=';', header=0)
station_data = station_data.replace(-999, np.nan)
station_data = station_data.set_index("MESS_DATUM_BEGINN") # it is a row with time dates
station_data_anual = pd.DataFrame()
station_data_anual["Y_TT"] = station_data["MO_TT"].resample("A").mean()
station_data_anual["Y_RR"] = station_data["MO_RR"].resample("A").sum()
The problem is, that it ignores the nans. Which means e.g. that station_data_anual["Y_RR"] values are to low. For years in which I have only nans as monthly values, it returns 0.
Note: There are some questions similar to mine, but they didn't help me.
Note: Python
some clarifications:
Input Data:
station_data
Out[235]:
STATIONS_ID MESS_DATUM_ENDE QN_4 ... MO_RR MX_RS eor
MESS_DATUM_BEGINN ...
1981-01-01 403.0 1981-01-31 10.0 ... 51.5 10.0 eor
1981-02-01 403.0 1981-02-28 10.0 ... 23.8 5.4 eor
1981-03-01 403.0 1981-03-31 10.0 ... 116.5 28.0 eor
1981-04-01 403.0 1981-04-30 10.0 ... 24.1 9.5 eor
1981-05-01 403.0 1981-05-31 10.0 ... 29.4 8.4 eor
... ... ... ... ... ... ...
2010-08-01 403.0 2010-08-31 10.0 ... NaN 29.1 eor
2010-09-01 403.0 2010-09-30 10.0 ... NaN 29.8 eor
2010-10-01 403.0 2010-10-31 10.0 ... NaN 5.5 eor
2010-11-01 403.0 2010-11-30 10.0 ... NaN 17.5 eor
2010-12-01 403.0 2010-12-31 10.0 ... NaN 8.2 eor
[360 rows x 16 columns]
have a closer look:
station_data["MO_RR"][276:288]
Out[242]:
MESS_DATUM_BEGINN
2004-01-01 66.3
2004-02-01 NaN
2004-03-01 NaN
2004-04-01 NaN
2004-05-01 NaN
2004-06-01 NaN
2004-07-01 NaN
2004-08-01 NaN
2004-09-01 NaN
2004-10-01 NaN
2004-11-01 NaN
2004-12-01 NaN
Name: MO_RR, dtype: float64
Output Data:
station_data_anual
Out[238]:
Y_TT Y_RR
MESS_DATUM_BEGINN
...
2003-12-31 9.866667 430.5
2004-12-31 9.620833 66.3
2005-12-31 9.665833 0.0
2006-12-31 10.158333 0.0
2007-12-31 10.555000 0.0
2008-12-31 10.361667 0.0
2009-12-31 9.587500 0.0
2010-12-31 8.207500 0.0
my result has to look like:
Y_TT Y_TX Y_TN Y_RR
MESS_DATUM_BEGINN
...
Y_TT Y_RR
MESS_DATUM_BEGINN
...
2003-12-31 9.866667 430.5
2004-12-31 9.620833 nan # getting nan instead of 66.3 is especially important
2005-12-31 9.665833 nan
2006-12-31 10.158333 nan
2007-12-31 10.555000 nan
2008-12-31 10.361667 nan
2009-12-31 9.587500 nan
2010-12-31 8.207500 nan

I have never used sampling and there might be better solutions out there which could simply ignore the "group" based on "condition". But a very simple solution could be to use a custom mean function after resample.
def very_mean(array_like):
if any(pd.isnull(array_like)):
return np.nan
else:
return array_like.mean()
station_data_anual["Y_TT"] = station_data["MO_TT"].resample("A").apply(very_mean)

Could you try to remove the nan values first using?
station_data_anual = pd.DataFrame()
station_data_anual["Y_TT"] = station_data["MO_TT"].dropna().resample("A").mean()
station_data_anual["Y_RR"] = station_data["MO_RR"].dropna().resample("A").sum()

It seems that NaN values are not included in the mean, considering the following experiment:
df_ = pd.DataFrame(index=pd.date_range("2022","2023",periods=12))
df_['a'] = np.ones(12)
df_.iloc[1]['a'] = np.NaN
df_.resample("2M").mean()
All of the averaged 2 month periods still have 1.0 as the mean value in the output of the mean() calculation above:
a
2022-01-31 1.0
2022-03-31 1.0
2022-05-31 1.0
2022-07-31 1.0
2022-09-30 1.0
2022-11-30 1.0
2023-01-31 1.0

Problems understanding the logic when creating code using groupby, list comprehensions and custom functions

I want to calculate a rolling mean of different window sizes for each ticker in my dataframe. Ideally I could pass a list of window sizes and for each ticker I would get new columns (one for each rolling mean size). So if I wanted a rolling mean of 2 and one of 3, the output would be two columns for each ticker.
import datetime as dt
import numpy as np
import pandas as pd
Dt_df = pd.DataFrame({"Date":pd.date_range('2018-07-01', periods=5, freq='D')})
Tick_df = pd.DataFrame({"Ticker":['ABC',"HIJ","XYZ"]})
Mult_df = pd.merge(Tick_df.assign(key='x'), Dt_df.assign(key='x') on='key').drop('key', 1)
df2 = pd.DataFrame(np.random.randint(low=5, high=10, size=(15, 1)), columns=['Price'])
df3 = Mult_df.join(df2, how='outer')
df3.set_index(['Ticker','Date'],inplace = True)
Here is the Example Dataset:
When I try to apply this function:
def my_RollMeans(x):
w = [1,2,3]
s = pd.Series(x)
Bob = pd.DataFrame([s.rolling(w1).mean() for w1 in w]).T
return Bob
to my dataframe df3 using various versions of apply or transform I get errors.
NewDF = df3.groupby('Ticker').Price.transform(my_RollMeans).fillna(0)
The latest error is:
Data must be 1-dimensional

IIUC try using apply and I made a modification to your custom function:
def my_RollMeans(x):
w = [1,2,3]
s = pd.Series(x)
Bob = pd.DataFrame([s.rolling(w1).mean().rename('Price_'+str(w1)) for w1 in w]).T
return Bob
df3.groupby('Ticker').apply(lambda x : my_RollMeans(x.Price)).fillna(0)
Output:
Price_1 Price_2 Price_3
Ticker Date
ABC 2018-07-01 9.0 0.0 0.000000
2018-07-02 8.0 8.5 0.000000
2018-07-03 7.0 7.5 8.000000
2018-07-04 8.0 7.5 7.666667
2018-07-05 8.0 8.0 7.666667
HIJ 2018-07-01 8.0 0.0 0.000000
2018-07-02 9.0 8.5 0.000000
2018-07-03 5.0 7.0 7.333333
2018-07-04 6.0 5.5 6.666667
2018-07-05 7.0 6.5 6.000000
XYZ 2018-07-01 9.0 0.0 0.000000
2018-07-02 5.0 7.0 0.000000
2018-07-03 9.0 7.0 7.666667
2018-07-04 8.0 8.5 7.333333
2018-07-05 6.0 7.0 7.666667

How do I merge 2 dataframes with an index in only 1 dataframe?

I have created 2 panda data frames, the first called 'dfmas' with an index 'Date', then dates, data and 3 moving average columns;
OPEN HIGH LOW LAST ma5 ma8 ma21
Date
11/23/2009 88.84 89.19 88.58 88.97 NaN NaN NaN
11/24/2009 88.97 89.07 88.36 88.50 NaN NaN NaN
11/25/2009 88.50 88.63 87.22 87.35 NaN NaN NaN
11/26/2009 87.35 87.48 86.30 86.59 NaN NaN NaN
11/27/2009 86.59 87.02 84.83 86.53 87.588 NaN NaN
11/30/2009 87.17 87.17 85.87 86.41 87.076 NaN NaN
12/1/2009 86.41 87.53 86.17 86.68 86.712 NaN NaN
12/2/2009 86.68 87.49 86.59 87.39 86.720 87.302 NaN
12/3/2009 87.39 88.48 87.32 88.26 87.054 87.214 NaN
12/4/2009 88.26 90.77 88.00 90.56 87.860 87.471 NaN
the second dataframe is made from the above data looking at when the moving averages crossover;
ma = [0,]
ma5Last = ma5[0]
ma8Last = ma8[0]
for ma5Curr, ma8Curr in zip(ma5[1:], ma8[1:]):
if ma5Curr > ma5Last and ma8Curr > ma8Last:
ma.append(1)
elif ma5Curr < ma5Last and ma8Curr < ma8Last:
ma.append(-1)
else:
ma.append(0)
ma5Last = ma5Curr
ma8Last = ma8Curr
maX = pd.DataFrame(ma).astype('float')
maX.columns = ['maX']
and is called 'maX' below;
maX
0 0.0
1 0.0
2 0.0
3 0.0
4 0.0
5 0.0
6 0.0
7 0.0
8 0.0
9 1.0
However I'm unable to merge/concat the 2 data frames.
How do I add the 'Date" index to the second 'maX'dataframe and then merge/concat/combine the two dataframes together? Many thanks in advance.

Is this what you are after?
df['maX'] = maX.maX.values
df
Out[1263]:
OPEN HIGH LOW LAST ma5 ma8 ma21 maX
Date
11/23/2009 88.84 89.19 88.58 88.97 NaN NaN NaN 0.0
11/24/2009 88.97 89.07 88.36 88.50 NaN NaN NaN 0.0
11/25/2009 88.50 88.63 87.22 87.35 NaN NaN NaN 0.0
11/26/2009 87.35 87.48 86.30 86.59 NaN NaN NaN 0.0
11/27/2009 86.59 87.02 84.83 86.53 87.588 NaN NaN 0.0
11/30/2009 87.17 87.17 85.87 86.41 87.076 NaN NaN 0.0
12/1/2009 86.41 87.53 86.17 86.68 86.712 NaN NaN 0.0
12/2/2009 86.68 87.49 86.59 87.39 86.720 87.302 NaN 0.0
12/3/2009 87.39 88.48 87.32 88.26 87.054 87.214 NaN 0.0
12/4/2009 88.26 90.77 88.00 90.56 87.860 87.471 NaN 1.0

If dataframes have same length simply add index from original DataFrame for align indexes:
maX = pd.DataFrame(ma, index=df.index).astype('float')