We Keep Coding

Matrix and vector shape in TVP-VAR in the statespace mlemodels

Thanks to everyone in advance for their time!
I am trying to run a TVP-VAR (time varying papramenter) for a panel in the statespace mlemodels in statsmodel. I get an error while trying to fit the model. My understanding is that mostly is regarding the dimensions for state covariance matrix. I suspect will get the same error later when I will deal with column shape.
The error is :
ValueError: Invalid dimensions for state covariance matrix matrix: requires 702 rows, got 3
How could I solve that ? The type error is showing is as below, highlighted in bold both the error and the Traceback :
preliminary = tvppanelvarmodel.fit(maxiter=1000)
Traceback (most recent call last):
File "/var/folders/m6/68zljfsj2t9_dzgpwwslj29r0000gp/T/ipykernel_6232/3038987883.py", line 1, in <module>
preliminary = tvppanelvarmodel.fit(maxiter=1000)
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 704, in fit
mlefit = super(MLEModel, self).fit(start_params, method=method,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/base/model.py", line 563, in fit
xopt, retvals, optim_settings = optimizer._fit(f, score, start_params,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/base/optimizer.py", line 241, in _fit
xopt, retvals = func(objective, gradient, start_params, fargs, kwargs,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/base/optimizer.py", line 651, in _fit_lbfgs
retvals = optimize.fmin_l_bfgs_b(func, start_params, maxiter=maxiter,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/lbfgsb.py", line 197, in fmin_l_bfgs_b
res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/lbfgsb.py", line 306, in _minimize_lbfgsb
sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/optimize.py", line 261, in _prepare_scalar_function
sf = ScalarFunction(fun, x0, args, grad, hess,
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 140, in __init__
self._update_fun()
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 233, in _update_fun
self._update_fun_impl()
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in update_fun
self.f = fun_wrapped(self.x)
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 134, in fun_wrapped
return fun(np.copy(x), *args)
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/base/model.py", line 531, in f
return -self.loglike(params, *args) / nobs
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 933, in loglike
self.update(params, transformed=True, includes_fixed=True,
File "/var/folders/m6/68zljfsj2t9_dzgpwwslj29r0000gp/T/ipykernel_6232/3786466608.py", line 104, in update
self['state_cov'] = np.diag([params[2]**2, params[3]**2, params[4]**2]) # W
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 239, in __setitem__
return self.ssm.__setitem__(key, value)
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/representation.py", line 420, in __setitem__
setattr(self, key, value)
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/representation.py", line 54, in __set__
value = self._set_matrix(obj, value, shape)
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/representation.py", line 68, in _set_matrix
validate_matrix_shape(
File "/opt/anaconda3/envs/spyder-env/lib/python3.10/site-packages/statsmodels/tsa/statespace/tools.py", line 1474, in validate_matrix_shape
raise ValueError('Invalid dimensions for %s matrix: requires %d'
ValueError: Invalid dimensions for state covariance matrix matrix: requires 702 rows, got 3
When I defined the class initially, I did the following
class TVPVAR(sm.tsa.statespace.MLEModel):
def __init__(self, y):
# Create a matrix with [y_t' : y_{t-1}'] for t = 2, ..., T
augmented = sm.tsa.lagmat(y, 1, trim='both', original='in', use_pandas=True)
# Separate into y_t and z_t = [1 : y_{t-1}']
p = y.shape[1]
y_t = augmented.iloc[:, :p]
z_t = sm.add_constant(augmented.iloc[:, p:])
nobs = y.shape[0]
T=y.shape[0]
# Recall that the length of the state vector is p * (p + 1)
k_states = p * (p + 1)
super(TVPVAR,self).__init__(y_t, exog=None, k_states=k_states,k_posdef=k_states)
self.k_y = p
self.k_states = p * (p + 1)
self.nobs = T
self['design'] = np.zeros((self.k_y, self.k_states, 1))
self['transition'] = np.eye(k_states) # G
self['selection'] = np.eye(k_states) # R=1
def update_variances(self, obs_cov, state_cov_diag):
self['obs_cov'] = obs_cov
self['state_cov'] = np.diag(state_cov_diag) # W
init = initialization.Initialization(self.k_states)
init.set((0, 2), 'diffuse')
init.set((2, 4), 'stationary')
self.ssm.initialize(init)
OTHER CODE
def update(self, params, **kwargs):
params = super().update(params, **kwargs)
self['transition', 2,2] = params[0]
self['transition', 3,2] = params[1]
self['state_cov'] = np.diag([params[2]**2, params[3]**2, params[4]**2]) # W
How can I define the dimensions for state covariance matrix and the vector shape? Thanks your inputs.

python Error: memoryError when apply ARIMA.fit with high order q=367 for MA

hi All stackoverflow Forum experts
i am using the software pyCharm2018.1.1
i have tried to build ARIMA model in python, my model has been identified by the parameters
(p=0, d=0, q=367), here is the code:
enter code here def arima_Model_Static_PlotErrorAC_PAC(series):
train, expctd =series , series
arima_orders = (0, 0, 367)
model = ARIMA(series, order=arima_orders)
results_MA = model.fit(disp=-1, start_params=[.1 for i in range(1 + arima_orders[2])])
yhatList=results_MA.fittedvalues
residuals = [expctd[i] - yhatList[i] for i in range(len(expctd))]
mse = mean_squared_error(expctd, yhatList)
rmse = sqrt(mse)
print(results_MA.summary())
print(rmse)
this model is called as follow:
enter code here series=DataSetDiff #DataSetDiff is a series with a length of 3652 values outputResidualError=arima_Model_Static_PlotErrorAC_PAC(series)
an error is loaded with this high q order which is:
Blockquote C:\109_personel\112_pyCharmArima\venv\Scripts\python.exe C:/109_personel/112_pyCharmArima/Presentation_Vers2_ModelOneFunct_3_5.py
Traceback (most recent call last):
File "C:/109_personel/112_pyCharmArima/Presentation_Vers2_ModelOneFunct_3_5.py", line 243, in arima_Model_Static_PlotErrorAC_PAC
results_MA = model.fit(disp=-1, start_params=[.1 for i in range(1 + arima_orders[2])], solver='bfgs')
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 959, in fit
callback=callback, **kwargs)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\model.py", line 466, in fit
full_output=full_output)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\optimizer.py", line 191, in _fit
hess=hessian)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\optimizer.py", line 327, in _fit_bfgs
disp=disp, retall=retall, callback=callback)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\scipy\optimize\optimize.py", line 916, in fmin_bfgs
res = _minimize_bfgs(f, x0, args, fprime, callback=callback, **opts)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\scipy\optimize\optimize.py", line 970, in _minimize_bfgs
gfk = myfprime(x0)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\scipy\optimize\optimize.py", line 300, in function_wrapper
return function(*(wrapper_args + args))
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\model.py", line 451, in score
return -self.score(params, *args) / nobs
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 583, in score
return approx_fprime_cs(params, self.loglike, args=(False,))
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tools\numdiff.py", line 202, in approx_fprime_cs
for i, ih in enumerate(increments)]
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tools\numdiff.py", line 202, in
for i, ih in enumerate(increments)]
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 780, in loglike
return self.loglike_kalman(params, set_sigma2)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 790, in loglike_kalman
return KalmanFilter.loglike(params, self, set_sigma2)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\kalmanf\kalmanfilter.py", line 654, in loglike
R_mat, T_mat)
File "kalman_loglike.pyx", line 359, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_loglike_complex
File "kalman_loglike.pyx", line 228, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_filter_complex
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\numpy\core\numeric.py", line 2200, in identity
return eye(n, dtype=dtype)
File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\numpy\lib\twodim_base.py", line 186, in eye
m = zeros((N, M), dtype=dtype, order=order)
MemoryError
Process finished with exit code 1
my model is well running and forecasting until q order MA 150 that mean(0,0,150). the error memoryError raised on the selection of q=367 as order
is any one can help me to solve this error, i have googled this error many time and i did not found a suitable solutions
Thank you for any help.

How to implement div and grad in sympy?

Playing around a lot with sympy lately, I came up with the problem of calculating divergence and gradient for scalar and vector fields. What I want to do for now is solving the heat equation, i.e.
d/dt u(x,t) - a * lap(u(x,t)) = 0, with lap(x) = (div(grad(x))
on a scalar field. Since I could not find lap, div and grad in sympy.physics.mechanics, I tried to implement them by myself
from sympy import *
from sympy.physics.mechanics import *
o = ReferenceFrame('o')
x,y,z = symbols('x y z')
class div(Function):
#classmethod
def eval(cls, F):
return F.diff(x, o).dot(o.x)+F.diff(y, o).dot(o.y)+F.diff(z, o).dot(o.z)
class grad(Function):
#classmethod
def eval(cls, F):
return o.x * F.diff(x) + o.y * F.diff(y) + o.z * F.diff(z)
f = x**2 + y**3 + z*0.5
print grad(f)
print type(grad(f))
print div(grad(f))
unluckily, this gives
2*x*o.x + 3*y**2*o.y + 0.500000000000000*o.z
Traceback (most recent call last):
File "/home/fortmeier/Desktop/autokernel/autokernel/tools/GenerateCode.py", line 24, in <module>
print div(grad(f))
File "/usr/local/lib/python2.7/dist-packages/sympy/core/cache.py", line 93, in wrapper
r = func(*args, **kw_args)
File "/usr/local/lib/python2.7/dist-packages/sympy/core/function.py", line 368, in __new__
result = super(Function, cls).__new__(cls, *args, **options)
File "/usr/local/lib/python2.7/dist-packages/sympy/core/cache.py", line 93, in wrapper
r = func(*args, **kw_args)
File "/usr/local/lib/python2.7/dist-packages/sympy/core/function.py", line 188, in __new__
args = list(map(sympify, args))
File "/usr/local/lib/python2.7/dist-packages/sympy/core/sympify.py", line 313, in sympify
expr = parse_expr(a, local_dict=locals, transformations=transformations, evaluate=evaluate)
File "/usr/local/lib/python2.7/dist-packages/sympy/parsing/sympy_parser.py", line 757, in parse_expr
return eval_expr(code, local_dict, global_dict)
File "/usr/local/lib/python2.7/dist-packages/sympy/parsing/sympy_parser.py", line 691, in eval_expr
code, global_dict, local_dict) # take local objects in preference
File "<string>", line 1, in <module>
AttributeError: 'Symbol' object has no attribute 'x'
[Finished in 0.3s with exit code 1]
I know that I could do something with the galgebra module, but first I'd like to understand more whats going on here. The question thus is what am I missing?

This looks like a bug in SymPy. sympify(ReferenceFrame('o').x) does not work.

newton raphson method using dataframes

I am using newton() function from the scipy to solve a particular non linear equation which works for a single values
def iv(p):
I=p
diode=Il-(Io*(np.exp((v+(I*RS))/a)-1))-((v+(I*RS))/Rsh)-I
return diode
I=[opt.newton(iv,2)for v in np.arange(0,44.5,0.1)]
I am trying to do the same for 8760 values of (I,Io,RS,a,Rsh) which are individual dataframes
def elec():
I=DataFrame(zeros(3898200).reshape((8760,445)),index=pd.date_range('1/1/2001 00:00','12/31/2001 23:59',freq='1h'),dtype=float)
for i in np.arange(0,8761,1):
def power(u):
I=u
diode=DataFrame(zeros(3898200).reshape((8760,445)),index=pd.date_range('1/1/2001 00:00','12/31/2001 23:59',freq='1h'),dtype=float)
diode[i]=IL[i]-(Io[i]*(np.exp((v+(I*RS[i]))/a[i]-1)))-((v+(I*RS[i]))/Rsh[i])-I
return diode
I[i]=[opt.newton(power,2)for v in np.arange(0,44.5,0.1)]
return I
I=elec()
I am getting a error of index error:index out of bounds
I should get 8760*445 values of I
Traceback (most recent call last):
File "<ipython-input-176-18d774ef131a>", line 5, in <module>
elec()
File "<ipython-input-174-996ccd566a20>", line 18, in elec
I[i]=[opt.newton(power,2)for v in np.arange(0,44.5,0.1)]
File "C:\Python27\lib\site-packages\scipy\optimize\zeros.py", line 143, in newton
q0 = func(*((p0,) + args))
File "<ipython-input-174-996ccd566a20>", line 14, in power
diode_east[i]=IL_east[0][i]-(Io_east[0][i]*(np.exp((v+(I*RS_ref[i]))/a_east[0][i]-1)))- ((v+(I*RS_ref[i]))/Rsh_east[0][i])-I
File "C:\Python27\lib\site-packages\pandas\core\series.py", line 613, in __getitem__
return self.index.get_value(self, key)
File "C:\Python27\lib\site-packages\pandas\tseries\index.py", line 1132, in get_value
return Index.get_value(self, series, key)
File "C:\Python27\lib\site-packages\pandas\core\index.py", line 769, in get_value
return tslib.get_value_box(series, key)
File "tslib.pyx", line 364, in pandas.tslib.get_value_box (pandas\tslib.c:8228)
File "tslib.pyx", line 379, in pandas.tslib.get_value_box (pandas\tslib.c:8075)
IndexError: index out of bounds
<type 'exceptions.IndexError'>

scipy.optimize.minimize() standard function error in example?

I will shortly explain what happened. I was learning (scipy.optimize tutorial) example (Nelder-Mead Simplex algorithm) to understand the minimize function, but error occures as shown below.
How can I fix it? I did not add any new values I just copied sample values and example.
from scipy import optimize
def rosen(x):
return sum(100.0*(x[1:]-x[:-1]**2.0) + (1 - x[:-1])**2.0)
x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])
res = optimize.minimize(rosen, x0, method='nelder-mead',
options={'xtol': 1e-8, 'disp': True})
Command line: C:\Python27\python.exe C:\Users\Desktop\FSFDSD~1.PY
Working directory: C:\Users\Desktop
Timeout: 0 ms
Traceback (most recent call last):
File "C:\Users\Desktop\FSFDSD~1.PY", line 49, in <module>
main()
File "C:\Users\Desktop\FSFDSD~1.PY", line 40, in main
options={'xtol': 1e-8, 'disp': True})
File "C:\Python27\lib\site-packages\scipy\optimize\_minimize.py", line 338, in minimize
return _minimize_neldermead(fun, x0, args, callback, **options)
File "C:\Python27\lib\site-packages\scipy\optimize\optimize.py", line 424, in
_minimize_neldermead
fsim[0] = func(x0)
File "C:\Python27\lib\site-packages\scipy\optimize\optimize.py", line 268, in function_wrapper
return function(x, args)
TypeError: rosen() takes exactly 1 argument (2 given)
Process "Python Interpreter" terminated, ExitCode: 00000001