I am trying to optimize a function in python, using optimize.fmin from scipy. The function should optimize a vector of parameters, given initial conditions and arguments. However, I keep receiving the following error when I try to run the optimization, while running the function itself works:
IndexError: too many indices for array, line 1, in parametrization
In brief, my code is like:
import numpy as np # import numpy library
import pandas as pd # import pandas library
from scipy import optimize # import optimize from scipy library
from KF_GATSM import KF_GATSM # import script with Kalman filter
yields=pd.read_excel('data.xlsx',index_col=None,header=None) # Import observed yields
Omega0=pd.read_excel('parameters.xlsx') # Import initial parameters
# Function to optimize
def GATSM(Omega,yields,N):
# recover parameters
Omega=np.matrix(Omega)
muQ,muP=parametrization(N,Omega) # run parametrization
Y=muQ+muP # or any other function
return Y
# Parametrization of the function
def parametrization(nstate,N,Omega):
muQ=np.matrix([[Omega[0,0],0,0]]).T # intercept risk-neutral world
muP=np.matrix([[Omega[1,0],Omega[2,0],Omega[3,0]]]).T # intercept physical world
return muQ,muP
# Run optimization
def MLE(data,Omega0):
# extract number of observations and yields maturities
N=np.shape(yields)[1]
# local optimization
omega_opt=optimize.fmin(GATSM,np.array(Omega0)[:,0],args=(yields,N))
return Y
I solved the issue. It seems that I cannot select the element of an array as follows in Scipy (although it works in Numpy):
Omega[0,0]
Omega[0]
The trick is to use:
Omega.item(0)
Related
I have the following code, but it shows error:
IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
If increasing the limit yields no improvement it is advised to analyze
the integrand in order to determine the difficulties. If the position of a
local difficulty can be determined (singularity, discontinuity) one will
probably gain from splitting up the interval and calling the integrator
on the subranges. Perhaps a special-purpose integrator should be used.
potC=sc.integrate.quad(lambda r: Psi(r,n2)*(-1/r)Psi(r,n1)(r**2),0,np.inf)
How to fix it?
import scipy as sc
import numpy as np
def Psi(r,n):
return 2*np.exp(-r/n)*np.sqrt(n)*sc.special.hyp1f1(1-n, 2, 2*r/n)/n**2
def p(n1,n2):
potC=sc.integrate.quad(lambda r: Psi(r,n2)*(-1/r)*Psi(r,n1)*(r**2),0,np.inf)
pot=potC[0]
return pot
print(p(15,15))
The error literally says what your problem is. Your function is not "well behaved" in some regions.
For example with n = 15 and r = 50 your special.hyp1f1(-14, 2, 2*50/15) result in NaN. I am not familiar with this function, so I do not know if this is the expected behaviour, but this is what happens.
You can try to isolate these points and exclude them from the integration (if you know lower and upper bounds of the function's value (if it is defined) you can also update the expected error of your integration) and just integrate in the well behaved regions.
If it is a bug in scipy, then please report it to them.
Ps.: Tested with scipy 1.8.0
Ps2.: With some reading I found, that you can get the values correctly, if you do your calculations with complex number, so the following code gives you a value:
import scipy as sc
from scipy import integrate
from scipy import special
import numpy as np
def Psi(r,n):
r = np.array(r,dtype=complex)
return 2*np.exp(-r/n)*np.sqrt(n)*special.hyp1f1(1-n, 2, 2*r/n)/n**2
def p(n1,n2):
potC=integrate.quad(lambda r: Psi(r,n2)*(-1/r)*Psi(r,n1)*(r**2),0,np.inf)
pot=potC[0]
return pot
print(p(15,15))
I have been trying to solve the following system of delay differential equations using JiTCDDE:
Model equations
And this is my code:
from jitcdde import y, t
from jitcdde import jitcdde
import matplotlib.pyplot as plt
import numpy as np
from pylab import *
N1=30290000
a1=0.98*N1
eps=1/5
b1=0.000024246
eta=0.3
B1=0.7
m=0.0005
chi=0.071
k1=0.185
alpha1=0.1155
delta=0.0225
phi1=0.26
omega1=0.26
d=3
model=[b1-((y(0)*B1*(y(2)+y(3)+(eta*y(5))))/a1)-b1*y(0)-m*y(0,t-d),
((y(0)*B1*(y(2)+y(3)+(eta*y(5))))/a1)-(k1+eps)*y(1)-m*y(1,t-d),
k1*eps*y(1)-(alpha1+chi)*y(2)-m*y(2,t-d),
(1-k1)*eps*y(1)-(phi1+omega1)*y(3)-m*y(3,t-d),
k1*y(1)+alpha1*y(2)-chi*y(4),
(phi1+omega1)*y(3)-(chi+delta)*y(5),
chi*(y(4)+y(5))-b1*y(6)-m*y(6,t-d),
delta*y(5)]
I=jitcdde(model)
I.constant_past([(0.98*N1-13),0,5,7,0,1,0,0], time=0.0)
I.step_on_discontinuities()
e=[]
for i in range(50):
e.append(I.integrate(i)[1])
print(e)
The problem is, for the second array of the solution (which I am trying to access), the first few values are negative values when I have specified that for t<0, the value is is 0. I have tried out this same model using ddeint and it gives a monotonically increasing curve with positive values, which is what I expect.
I want jitcdde to work though, since this model should run even when there is no delay term.
The first array seems fine and I have checked my model to see if I had made any typos but everything looks good to me. I have also tried using adjust_diff and integrate_blindly, but the issue remains.
In the module _denoise.py of skimage I found the following piece of code:
def estimate_sigma(image, average_sigmas=False, multichannel=False):
# some more code here
sigmas = [estimate_sigma(image[..., c], multichannel=False)...
return _sigma_est_dwt(detail_coeffs, distribution='Gaussian')
Inside estimate_sigma there's an estimate_sigma? How and why does that work? The imports are
import scipy.stats
import numpy as np
from math import ceil
from .. import img_as_float
from ..restoration._denoise_cy import _denoise_bilateral, _denoise_tv_bregman
from .._shared.utils import skimage_deprecation, warn
import pywt
import skimage.color as color
import numbers
which doesn't seem to sneak in any new functions.
Note that the recursive call of estimate_sigma is inside an if-clause:
if multichannel:
sigmas = [estimate_sigma(image[..., c], multichannel=False)...
...
return _sigma_est_dwt(detail_coeffs, distribution='Gaussian')
Case A) If we call estimate_sigma with multichannel=False, the function won't get inside the if-clause, thus won't invoke itself and will return reaching the end of its body.
Case B) If we call estimate_sigma with multichannel=True the condition will succeed, so estimate_sigma will call itself. As seen from the piece of source above, when estimate_sigma calls itself, it passes multichannel as False. It means that during the recursive invocation "case A" will happen. This time the program will not enter the above if block and recursion will end, ending execution of the function and returning.
Basically the idea is: if we've got multiple channels, let's divide them into separate ones and perform sigma estimation on each channel
i used wavelet decomposition command in python using pywt library but it does not return any coefficients. my code is given below .
import numpy as np
import pywt as pywt
(e,f)=pywt.wavedec(y,'db12' ,level=2)
print("e:"+str(e))
print("f:"+str(f))
I also tried with pywt.dwt(y,' db12', level=2) it is also not returning any coefficients
it returns a null output, where y is a matrix contains my input
I tried reproducing your results with a random (discrete) signal like so:
import numpy as np
import pyw
x = np.random.randint(0,100,500)
y = pywt.wavedec(x, 'db12', level=2)
(e,f) = pywt.dwt(x, 'db12')
I noticed two things: For a 1D signal, wavedec returns more than two coefficient arrays, as also mentioned in the docs. Similarly, the dwt function does not know the keyword level=, but works just fine with the command specified above.
Hope that helps
I am new to python and programming and am working my way through "Machine Learning: An Algorithmic Perspective." I was told to normalise data, seperate it into training and testing data, recover the beta vector, and then use the sum-of-least squares error. I keep getting,
File "/Users/shaune/Dropbox/Shaune PhD/auto-mpg.py", line 34, in
beta=linreg.linreg(trainin,traintgt)
AttributeError: 'function' object has no attribute 'linreg'
when running the following:
import os
import pylab as pl
import numpy as np
from pylab import *
from numpy import *
import linreg
os.chdir('/Users/shaune/Dropbox/Shaune PhD')
auto=np.loadtxt('auto-mpg.data.txt',comments='"')
#normalise the data
auto=(auto-auto.mean(axis=0))/auto.var(axis=0)
#seperate the training and testing data
trainin=auto[::2,:8]
testin=auto[1::2,:8]
traintgt=auto[::2,1:2]
testtgt=auto[1::2,1:2]
#recover the beta vector
def linreg(trainin,traintgt):
trainin=np.concatenate((trainin,-np.ones((np.shape(trainin)[0],1))),axis=1)
beta=np.dot(np.dot(np.linalg.inv(np.dot(np.transpose(trainin),trainin)),np.transpose(trainin)),traintgt)
traintgt=np.dot(trainin, beta)
#sum of squares error to get predicted values on test set (want small values)
beta=linreg.linreg(trainin,traintgt)
testin=concatenate((testin,-np.ones((np.shape(testin)[0],1))),axis=1)
testout=dot(testin,beta)
error=sum((testout-testtgt)**2)
print error
Please help! Thanks.
The definition of this function
def linreg(trainin,traintgt):
is overwriting the name linreg that you imported with
import linreg
Rename the function. The comment says recover the beta vector, so perhaps a better name is recover_beta. That is, change the def statement to
def recover_beta(trainin,traintgt):
You'll probably want to add a return statement to the function while you are at it. Currently it doesn't return anything.