I have been using lmfit for about a day now and needless to say I know very little about the library. I have been using several built-in models for curve fitting and all of them work flawlessly with the data except the Lognormal Model.
Here is my code:
from numpy import *
from lmfit.models import LognormalModel
import pandas as pd
import scipy.integrate as integrate
import matplotlib.pyplot as plt
data = pd.read_csv('./data.csv', delimiter = ",")
x = data.ix[:, 0]
y = data.ix[:, 1]
print (x)
print (y)
mod = LognormalModel()
pars = mod.guess(y, x=x)
out = mod.fit(y, pars , x=x)
print(out.best_values)
print(out.fit_report(min_correl=0.25))
out.plot()
plt.plot(x, y, 'bo')
plt.plot(x, out.init_fit, 'k--')
plt.plot(x, out.best_fit, 'r-')
plt.show()
and the error output is:
Traceback (most recent call last):
File "Cs_curve_fit.py", line 17, in <module>
pvout = pvmod.fit(y, amplitude= 1, center = 1, sigma =1 , x=x)
File "C:\Users\NAME\Anaconda3\lib\site-packages\lmfit\model.py", line 731, in fit
output.fit(data=data, weights=weights)
File "C:\Users\NAME\Anaconda3\lib\site-packages\lmfit\model.py", line 944, in fit
self.init_fit = self.model.eval(params=self.params, **self.userkws)
File "C:\Users\NAME\Anaconda3\lib\site-packages\lmfit\model.py", line 569, in eval
return self.func(**self.make_funcargs(params, kwargs))
File "C:\Users\NAME\Anaconda3\lib\site-packages\lmfit\lineshapes.py", line 162, in lognormal
x[where(x <= 1.e-19)] = 1.e-19
File "C:\Users\NAME\Anaconda3\lib\site-packages\pandas\core\series.py", line 773, in __setitem__
setitem(key, value)
File "C:\Users\NAME\Anaconda3\lib\site-packages\pandas\core\series.py", line 755, in setitem
raise ValueError("Can only tuple-index with a MultiIndex")
ValueError: Can only tuple-index with a MultiIndex
First, the error message you show cannot have come from the code you post. The error message says that line 17 of the file "Cs_curve_fit.py" reads
pvout = pvmod.fit(y, amplitude= 1, center = 1, sigma =1 , x=x)
but that is not anywhere in your code. Please post the actual code and the actual output.
Second, the problem appears to happening because the data for x is cannot be turned into a 1D numpy array. Not being able to trust your code or output, I would just suggest converting the data to 1D numpy arrays yourself as a first test. Lmfit should be able to handle Pandas series, but it just does a simple coercion to 1D numpy arrays.
Related
I'm trying to do some plots of some symbolic data. I have some expression from a regression in the form:
expr = '(((((((((1.0)*(2.0)))-(ER)))-(-0.37419122066665467))*0.006633039574629684)*(0.006633039574629684*((((T)-(((1.0)+(P)))))-(P))))+0.1451920626347467)'
Where expr here is some prediction: f = f(T, P, ER). I know this particular example is a crazy expression but it's not really super important. Basically, supposing I have some dataframe, plotdata, I am trying to produce plots with:
import pandas
import sympy
import numexpr
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
expr = '(((((((((1.0)*(2.0)))-(ER)))-(-0.37419122066665467))*0.006633039574629684)*(0.006633039574629684*((((T)-(((1.0)+(P)))))-(P))))+0.1451920626347467)'
#Extract some data for surface plot but fixing one variable
plotdata = plotdata.loc[(plotdata.P == 1)]
#Extract data as lists for plotting
x = list(plotdata['T'])
y = list(plotdata['ER'])
f_real = list(plotdata['f'])
T_sympy = sympy.Symbol('T')
P_sympy = sympy.Symbol('P')
ER_sympy = sympy.Symbol('ER')
f_pred = numexpr.evaluate(expr)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_trisurf(x,y,f_real, alpha = 0.3)
ax.plot_surface(x,y,f_pred)
However, I am getting an error with f_pred.
numexpr.evaluate(expr)
Traceback (most recent call last):
File "/anaconda3/lib/python3.7/site-packages/numexpr/necompiler.py", line 744, in getArguments
a = local_dict[name]
KeyError: 'ER'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<ipython-input-100-c765b0f1e5ce>", line 1, in <module>
numexpr.evaluate(expr)
File "/anaconda3/lib/python3.7/site-packages/numexpr/necompiler.py", line 818, in evaluate
arguments = getArguments(names, local_dict, global_dict)
File "/anaconda3/lib/python3.7/site-packages/numexpr/necompiler.py", line 746, in getArguments
a = global_dict[name]
KeyError: 'ER'
I am not super familiar with the numexpr package. However, I have been building this up from a 1D regression to now a 3D regression. ER was my 1D variable and was working fine. I have obviously slightly altered my code since the 1D case but I am still slightly at a loss as to why this error is popping up.
Any pointers would be greatly appreciated.
I've figured it out. Pretty silly error in the end. I needed to change:
#Extract data as lists for plotting
x = list(plotdata['T'])
y = list(plotdata['ER'])
to:
T = list(plotdata['T'])
ER = list(plotdata['ER'])
P = list(plotdata['P'])
i.e. numexpr.evaluate was looking for the input data, not the symbol!
This is my code to plot a Power spectral density signal. I need some help as im having error coding in python.
import math
import numpy as np
import matplotlib.pyplot as plt
from numpy.fft import fft, ifft
from scipy import signal
t=np.linspace(0,1,106)
Fs=1000
y=np.sin(2*math.pi*15*t) + np.sin(2*math.pi*30*t)
Y=fft(y,512)
f = np.arange(0,len(Y)-1)*(Fs-1)/len(Y)
P = abs(Y)*2/len(Y)
plt.plot(f,P)
plt.show()
This is the error i have when i try to run it.
Traceback (most recent call last):
File "C:\Users\keehu\Desktop\trial.py", line 13, in <module>
plt.plot(f,P)
File "C:\Users\keehu\AppData\Local\Programs\Python\Python38-32\lib\site-packages\matplotlib\pyplot.py", line 2761, in plot
return gca().plot(
File "C:\Users\keehu\AppData\Local\Programs\Python\Python38-32\lib\site-packages\matplotlib\axes\_axes.py", line 1646, in plot
lines = [*self._get_lines(*args, data=data, **kwargs)]
File "C:\Users\keehu\AppData\Local\Programs\Python\Python38-32\lib\site-packages\matplotlib\axes\_base.py", line 216, in __call__
yield from self._plot_args(this, kwargs)
File "C:\Users\keehu\AppData\Local\Programs\Python\Python38-32\lib\site-packages\matplotlib\axes\_base.py", line 342, in _plot_args
raise ValueError(f"x and y must have same first dimension, but "
ValueError: x and y must have same first dimension, but have shapes (511,) and (512,)
>>>
Your f is of size len(Y)-1 as you requested, but P has size len(Y).
The size of f is 511 and Y is 512. I have changed the code to solve that.
import math
import numpy as np
import matplotlib.pyplot as plt
from numpy.fft import fft, ifft
from scipy import signal
t=np.linspace(0,1,106)
Fs=1000
y=np.sin(2*math.pi*15*t) + np.sin(2*math.pi*30*t)
Y=fft(y,512)
f = np.arange(0,len(Y))*(Fs-1)/len(Y) #changed here
P = abs(Y)*2/len(Y)
plt.plot(f,P)
plt.show()
"ValueError: x and y must have same first dimension, but have shapes (511,) and (512,)" means that your arrays, f and P don't have the same size. f is one element to small. Use:
f = np.arange(0,len(Y))*(Fs-1)/len(Y)
I am trying to fit a function y(x,T,p) to get the coefficients a,b,c,d,e,f. The data for y,x,T,p are known. With a global optimizer I want to find a good starting point. shgo seems to be the only one that accept constraints.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import shgo
# test data
x = np.array([0.1,0.2,0.3,1])
T = np.array([300,300,300,300])
p = np.array([67.2,67.2,67.2,67.2])
y = np.array([30,50,55,67.2])
# function
def func(pars,x,T,p):
a,b,c,d,e,f = pars
return x*p+x*(1-x)*(a+b*T+c*T**2+d*x+e*x*T+f*x*T**2)*p
# residual
def resid(pars):
return ((func(pars,x,T,p) - y) ** 2).sum()
# constraint: derivation is positive in every data point
def der(pars):
a,b,c,d,e,f = pars
return -p*((3*f*T**2+3*e*T+3*d)*x**2+((2*c-2*f)*T**2+(2*b-2*e)*T-2*d+2*a)*x-c*T**2-b*T-a-1)
con1 = ({'type':'ineq', 'fun':der})
# minimizer shgo
bounds = [(-1,1),(-1,1),(-1,1),(-1,1),(-1,1),(-1,1)]
res = shgo(resid, bounds, constraints=con1)
print("a = %f , b = %f, c = %f, d = %f, e = %f, f = %f" % (res[0], res[1], res[2], res[3], res[4], res[5]))
# plotting
x0 = np.linspace(0, 1, 100)
fig, ax = plt.subplots()
fig.dpi = 80
ax.plot(x,y,'ro',label='data')
for i,txt in enumerate(T):
ax.annotate(txt,(x[i],y[i]))
ax.plot(x0, func(res.x, x0, 300,67.2), '-', label='fit1')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
With this I am getting ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I do not know what that Error means and other threads with the same Error does not realy help me to understand. When I use a local minimizer (scipy.optimize.minimize with the method cobyla) the error does not appear.
Can someone help me to understand my problem or even help to fix it?
Thanks
EDIT:
Traceback (most recent call last):
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo_lib\triangulation.py", line 759, in __getitem__
return self.cache[x]
KeyError: (0, 0, 0, 0, 0, 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/.../test.py", line 70, in <module>
res = shgo(resid, bounds, constraints=con1)
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo.py", line 423, in shgo
shc.construct_complex()
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo.py", line 726, in construct_complex
self.iterate()
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo.py", line 869, in iterate
self.iterate_complex()
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo.py", line 890, in iterate_hypercube
self.g_args)
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo_lib\triangulation.py", line 121, in __init__
self.n_cube(dim, symmetry=symmetry)
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo_lib\triangulation.py", line 172, in n_cube
self.C0.add_vertex(self.V[origintuple])
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo_lib\triangulation.py", line 767, in __getitem__
index=self.index)
File "C:\Users\...\Python\Python36\site-packages\scipy\optimize\_shgo_lib\triangulation.py", line 681, in __init__
if g(self.x_a, *args) < 0.0:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
The problem is, that der returns an array instead of a scalar value. Changing
con1 = ({'type':'ineq', 'fun':der})
to
con_list = [{'type':'ineq', 'fun': lambda x: der(x)[i_out]} for i_out in range(T.shape[0])]
removes the error. This transforms each output of der into its own inequality constraint.
Also, since your constraints are all written so that der(x)>=0, one can simply keep the definition of oyur constraint with vector output and then fetch the minimum of the outputs, i.e., take a scalar value constraint x -> \min (der(x)).
# Python code to demonstrate SQL to fetch data.
# importing the module
import sqlite3
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
from scipy.stats import chisquare
# connect withe the myTable database
connection = sqlite3.connect(r"C:\Users\Aidan\Desktop\CEP_DB.db")
# cursor object
crsr = connection.cursor()
dog= crsr.execute("Select s, ei, ki FROM cep_db_lite1_vc WHERE s IN ('d')")
ans= crsr.fetchall()
filtered_data = ans[~np.isnan(ans["ki"])]
dogData = np.array(filtered_data)
dogData.astype(float)
FdogData= dogData[:, [1,2]]
x, y = FdogData[:,0], FdogData[:,1]
# Reshaping
x, y = x.reshape(-1,1), y.reshape(-1, 1)
# Linear Regression Object
lin_regression = LinearRegression()
# Fitting linear model to the data
lin_regression.fit(x,y)
# Get slope of fitted line
m = lin_regression.coef_
# Get y-Intercept of the Line
b = lin_regression.intercept_
# Get Predictions for original x values
# you can also get predictions for new data
predictions = lin_regression.predict(x)
chi= chisquare(predictions, y)
# following slope intercept form
print ("formula: y = {0}x + {1}".format(m, b))
print(chi)
# Plot the Original Model (Black) and Predictions (Blue)
plt.scatter(x, y, color='black')
plt.plot(x, predictions, color='blue',linewidth=3)
plt.show()
Error: runfile('C:/Users/Aidan/.spyder-py3/temp.py',
wdir='C:/Users/Aidan/.spyder-py3') Traceback (most recent call last):
File "", line 1, in
runfile('C:/Users/Aidan/.spyder-py3/temp.py', wdir='C:/Users/Aidan/.spyder-py3')
File
"C:\Users\Aidan\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 705, in runfile
execfile(filename, namespace)
File
"C:\Users\Aidan\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Aidan/.spyder-py3/temp.py", line 22, in
filtered_data = ans[~np.isnan(ans["ki"])]
TypeError: list indices must be integers or slices, not str
I believe I am recieving errors in this section:
dog= crsr.execute("Select s, ei, ki FROM cep_db_lite1_vc WHERE s IN ('d')")
ans= crsr.fetchall()
filtered_data = ans[~np.isnan(ans["ki"])]
dogData = np.array(filtered_data)
dogData.astype(float)
FdogData= dogData[:, [1,2]]
x, y = FdogData[:,0], FdogData[:,1]
How can I fix this script? I have been struggling with this error for weeks to no avail!
TypeError: list indices must be integers or slices, not str
That means you can't index a list with str, only integers and slices. With that, you can find that in this line:
filtered_data = ans[~np.isnan(ans["ki"])]
ans is a list, and can't be indexed by a string.
I am having problems interpolating some data points using Scipy. I guess that it might depend on the fact that the function I'm trying to interpolate is discontinuous at x roughly 4.
Here is the code I'm using to interpolate:
from scipy import *
y_interpolated = interp1d(x,y,buonds_error=False,fill_value=0.,kind='cubic')
new_x_array = arange(min(x),max(x),0.05)
plot(new_x_array,x_interpolated(new_x_array),'r-')
The error I get is
File "<stdin>", line 2, in <module>
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 357, in __call__
out_of_bounds = self._check_bounds(x_new)
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 415, in _check_bounds
raise ValueError("A value in x_new is above the interpolation "
ValueError: A value in x_new is above the interpolation range.
These are my data points:
1.56916432074 -27.9998263169
1.76773750527 -27.6198430485
1.98360238449 -27.2397962268
2.25133982943 -26.8596491107
2.49319293195 -26.5518194791
2.77823462692 -26.1896935372
3.07201297519 -25.9540514619
3.46090507092 -25.7362456112
3.65968688527 -25.6453922172
3.84116464506 -25.53652509
3.97070419447 -25.3374215879
4.03087127145 -24.8493356465
4.08217147954 -24.0540196233
4.12470899596 -23.0960856364
4.17612639206 -22.4634289328
4.19318305992 -22.1380894034
4.2708234589 -21.902951035
4.3745696768 -21.9027079759
4.52158254627 -21.9565591238
4.65985875536 -21.8839570732
4.80666329863 -21.6486676004
4.91026629192 -21.4496126386
5.05709528961 -21.2685401725
5.29054655428 -21.2860476871
5.54129211534 -21.3215908912
5.73174988353 -21.6645019816
6.06035782465 -21.772138994
6.30243916407 -21.7715483093
6.59656410998 -22.0238656166
6.86481948673 -22.3665921479
7.01182409559 -22.4385289076
7.17609125906 -22.4200564296
7.37494987052 -22.4376476472
7.60844044988 -22.5093814451
7.79869207061 -22.5812017094
8.00616642549 -22.5445612485
8.17903446593 -22.4899243886
8.29141325457 -22.4715846981
What version of scipy are you using?
The script you posted has some syntax errors (I assume due to wrong copy and paste).
This script works, with scipy.__version__ == 0.9.0. .
import sys
from scipy import *
from scipy.interpolate import *
from pylab import plot
x = []
y = []
for line in sys.stdin:
a, b = line.split()
x.append(float(a))
y.append(float(b))
y_interpolated = interp1d(x,y,bounds_error=False,fill_value=0.,kind='cubic')
new_x_array = arange(min(x),max(x),0.05)
plot(new_x_array,y_interpolated(new_x_array),'r-')