We Keep Coding

try to interpolate data from a csv file, getting error 'x and y arrays must be equal in length along interpolation axis'. found solution doesn't work

I'm completely new to python and data science with it, so please bear with me here. I appreciate every help and try to understand it as much as possible. Following code is what I got so far.
import pandas as pd
import numpy as np
from pandas import read_csv
from matplotlib import pyplot
import matplotlib.pyplot as plt
from scipy import interpolate
from scipy.interpolate import interp1d
from scipy.interpolate import interp2d
dataset = pd.read_csv(r"C:\Users\...\csv\Test1K.csv", sep=';', skiprows=0)
x = dataset.iloc[0:1420, 0:1].values
y = dataset.iloc[0:1420, 3:4].values
f = interpolate.interp1d(x, y, kind = 'cubic')
xn =270
t_on = f(xn)
print(t_on)
first rows of output from the csv file looks like this:
0 [s] [Celsius] [Celsius] [Celsius] [Celsius] [Celsius]
1 0 22.747 22.893 0.334 22.898 22.413
2 60 22.769 22.902 22.957 22.907 -0.187
3 120 22.78 22.895 25.519 22.911 -2.739
4 180 22.794 22.956 33.62 22.918 -10.827
short thing about what I try to do and where the problem is. I have this csv file, where there is a alot of data in it, with temperature readings every 60 seconds, for like 1400 readings. Now I want to interpolate that, so I can get a specific data point between each 60 seconds and possible even further than the 1400 iterations. (maybe up to 1600)
The first dataset I want, is the third celsius one. The code above is how far I got so far. Now I get the error code
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_8008/3128405253.py in <module>
7
8
----> 9 f = interpolate.interp1d(x, y, kind = 'cubic')
10 yn =270 # Wert auf den Interpoliert werden soll
11 t_on = f(yn)
~\AppData\Roaming\Python\Python38\site-packages\scipy\interpolate\interpolate.py in __init__(self, x, y, kind, axis, copy, bounds_error, fill_value, assume_sorted)
434 assume_sorted=False):
435 """ Initialize a 1-D linear interpolation class."""
--> 436 _Interpolator1D.__init__(self, x, y, axis=axis)
437
438 self.bounds_error = bounds_error # used by fill_value setter
~\AppData\Roaming\Python\Python38\site-packages\scipy\interpolate\polyint.py in __init__(self, xi, yi, axis)
52 self.dtype = None
53 if yi is not None:
---> 54 self._set_yi(yi, xi=xi, axis=axis)
55
56 def __call__(self, x):
~\AppData\Roaming\Python\Python38\site-packages\scipy\interpolate\polyint.py in _set_yi(self, yi, xi, axis)
122 shape = (1,)
123 if xi is not None and shape[axis] != len(xi):
--> 124 raise ValueError("x and y arrays must be equal in length along "
125 "interpolation axis.")
126
ValueError: x and y arrays must be equal in length along interpolation axis.
I searched for solutions and got this for example:
x = np.linspace(0, 4, 13)
y = np.linspace(0, 4, 13)
X, Y = np.meshgrid(x, y)
z = np.arccos(-np.cos(2*X) * np.cos(2*Y))
f = interpolate.interp2d(x, y, z, kind = 'cubic')
I read at other problems, that the 2d solution should help, but when I put it like this I get:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_8008/1890924245.py in <module>
14 f = interpolate.interp2d(x, y, z, kind = 'cubic')
15 yn =270 # Wert auf den Interpoliert werden soll
---> 16 t_on = f(yn)
17 print(t_on)
18
TypeError: __call__() missing 1 required positional argument: 'y'
Now I get it, I need something for y, but wouldnt that ruin the whole thing I try to do? My result should be my y and not one of my inputs.
If anyone can help me with my code or even has completely different other solution, I appreciate everthing. Thank you all for the help
Edit: the whole error message

I think your issue is here:
x = dataset.iloc[0:1420, 0:1].values
y = dataset.iloc[0:1420, 3:4].values
The result of a 0:1 slice will be a DataFrame, and you will end up with a two-dimensional array with the shape (1420, 1).
You need a 1d array for interp1d, so you should just do
x = dataset.iloc[0:1420, 0].values
y = dataset.iloc[0:1420, 3].values
The shape of x and y will be (1420,) (1d array).

QuantLib Python Hull White Model - RuntimeError: time (20) is past max curve time (19)

I tried using QuantLib-python to run several iterations of a Hull-White model. I followed along with the code and blog here:
http://gouthamanbalaraman.com/blog/hull-white-simulation-quantlib-python.html
I made some edits from Balaraman's code on the site. Namely, I changed the spot_curve from being a FlatForward to a ZeroCurve. Now I keep getting an error. I am trying to model the zero curve data as seen in my code below.
Does anyone know how to fix this and implement the zero curve in QuantLib-python?
from QuantLib import *
import utils
import numpy as np
%matplotlib inline
##Assign all variables
sigma = 0.015
a = 0.1
timestep = 30
length = 30 # in years
day_count = Thirty360()
start_date = Date(19, 11, 1989)
calendar = UnitedStates()
interpolation = Linear()
compounding = Compounded
compoundingFrequency = Annual
dates = [Date(19,11,1990), Date(19,11,1991), Date(19,11,1992),
Date(19,11,1993), Date(19,11,1994), Date(19,11,1995),
Date(19,11,1996), Date(19,11,1997), Date(19,11,1998),
Date(19,11,1999), Date(19,11,2000), Date(19,11,2001),
Date(19,11,2002), Date(19,11,2003), Date(19,11,2004),
Date(19,11,2005), Date(19,11,2006), Date(19,11,2007),
Date(19,11,2008), Date(19,11,2009)]
zeros = [0.115974,0.118913,0.120676,0.121751,0.122455,0.122988,
0.12347,0.123972,0.124527,0.125147,0.125831,0.126573,
0.127359,0.128178,0.129016,0.129863,0.130708,0.131544,
0.132364,0.133162]
#setup spot curve. Notable difference is the ZeroCurve instead of FlatForward
spot_curve = ZeroCurve(dates, zeros, day_count, calendar, interpolation, compounding, compoundingFrequency)
spot_curve_handle = YieldTermStructureHandle(spot_curve)
#The Hull-White process is constructed by passing the term-structure, a and sigma.
#To create the path generator, one has to provide a random sequence generator along
#with other simulation inputs such as timestep and `length.
hw_process = HullWhiteProcess(spot_curve_handle, a, sigma)
rng = GaussianRandomSequenceGenerator(
UniformRandomSequenceGenerator(timestep, UniformRandomGenerator()))
seq = GaussianPathGenerator(hw_process, length, timestep, rng, False)
#define generate paths function
def generate_paths(num_paths, timestep):
arr = np.zeros((num_paths, timestep+1))
for i in range(num_paths):
sample_path = seq.next()
path = sample_path.value()
time = [path.time(j) for j in range(len(path))]
value = [path[j] for j in range(len(path))]
arr[i, :] = np.array(value)
return np.array(time), arr
#plotting short rates
num_paths = 100
paths = generate_paths(num_paths, timestep)
fig, ax = utils.plot()
for i in range(num_paths):
ax.plot(time, paths[i, :], lw=0.8, alpha=0.6)
ax.set_title("Hull-White Short Rate Simulation");
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-3-366fe665a669> in <module>
62 #plotting short rates
63 num_paths = 100
---> 64 paths = generate_paths(num_paths, timestep)
65 fig, ax = utils.plot()
66 for i in range(num_paths):
<ipython-input-3-366fe665a669> in generate_paths(num_paths, timestep)
52 arr = np.zeros((num_paths, timestep+1))
53 for i in range(num_paths):
---> 54 sample_path = seq.next()
55 path = sample_path.value()
56 time = [path.time(j) for j in range(len(path))]
~/opt/anaconda3/lib/python3.7/site-packages/QuantLib/QuantLib.py in next(self)
22328
22329 def next(self):
> 22330 return _QuantLib.GaussianPathGenerator_next(self)
22331
22332 def antithetic(self):
RuntimeError: time (20) is past max curve time (19)

That error means you are trying to get a point (date) from your yield curve that is past the maximum maturity. Your yield curve has a maximum maturity of 19 years and your simulation is for 30 years...
To avoid that error, either build your curve with extra maturities, or enable extrapolation:
spot_curve.enableExtrapolation()

What to do with: ValueError: x and y can be no greater than 2-D, but have shapes (1L,) and (1L, 28L, 28L)

My goal is to plot a profile of a 2D FITS file that I already asked the user for with a try/except. I want to slice a part of this FITS file and then plot this part. I will just put my whole code here. I don't think there is anything wrong with the try/except part but the slicing and plotting have some erros.
My main problem is that the programme is giving me the following error ValueError: x and y can be no greater than 2-D, but have shapes (1L,) and (1L, 28L, 28L)
import os
files = [f for f in os.listdir('.') if os.path.isfile(f)]
for f in files:
if f.endswith(".fits"):
print(f)
which gives me the two FITS files i have in my folder
from astropy.io import fits
valid = False
while not valid:
filename = input("Name of 2D FITS file:")
hdulist = fits.open(filename)
hdr= hdulist[0].header
try:
naxis = hdr['NAXIS']
valid = (naxis == 2)
if not valid:
print("Data structure is not two dimensional")
except Exception as e:
print(e)
valid = False
print("Valid input:", valid)
hdulist = fits.open(filename)
hdr= hdulist[0].header
Here ends my try/except and starts the messy part where i try to slice the FITS file over an angle
from matplotlib.pyplot import figure, show
from astropy.io import fits
data=hdulist[0].data
B=input("Enter the y value of the pixel defined as your starting value of the slice B=")
C=input("Enter the y value of the pixel defined as your stopping value of the slice C=")
D=input("Enter the x value of the pixel defined as your starting value of the slice D=")
E=input("Enter the x value of the pixel defined as your stopping value of the slice E=")
A = data[B:C,D:E]
astd = A.std()
print(astd)
fig = figure()
frame = fig.add_subplot(1,1,1)
mappable = frame.imshow(A, interpolation='none', origin="lower", cmap='jet', vmin=1500, vmax=8000)
cbar = fig.colorbar(mappable, ax=frame, fraction=0.046, pad=0.04)
show()
#the figure below always adjusts the axis such that the starting value is in the
#lower left corner and the stopping value in the upper right corner
import math
R= math.degrees(math.atan((C-B)/(E-D)))
L= ((E-D)**2+(C-B)**2)**(0.5)
print("The line from the starting to the stopping point has a length of:", L)
print("The line from the starting to the stopping point has an angle in degrees of:", R)
F=int(L)
print(F)
This gives me a slice, now i want to put this in a profile and plot the whole thing.
# construct interpolation function
import numpy
import scipy
from scipy import interpolate
x = numpy.arange(data.shape[1])
y = numpy.arange(data.shape[0])
f = scipy.interpolate.interp2d(x, y, data)
# extract values on line from D, B to E, C (degrees are not even necessary)
num_points = F
xvalues = numpy.linspace(D, E, num_points)
yvalues = numpy.linspace(B, C, num_points)
zvalues = f(xvalues, yvalues)
print(zvalues)
import numpy as np
from numpy import random
from matplotlib.pyplot import figure, show
from scipy import stats
from numpy import pi
profile=np.array([zvalues])
This is where i start to design the plot i used a previous design where velocity was used therefore the name "vels"
#Plot of profile
vels = np.linspace(0, 100, len(profile))
gam = profile.sum()
XY = (vels*profile).sum()
X0 = XY/gam
var = (1/gam)*(profile*(vels-X0)**2).sum()
sigma = var**0.5
skew = (1/gam)*(profile*(vels-X0)**3).sum()/sigma**3
kurt = (1/gam)*(profile*(vels-X0)**4).sum()/sigma**4
print("Mean:", X0)
print("Standard deviation:", sigma)
print("Skewness:", skew)
print("Fisher Kurtosis:", kurt-3)
a=profile.max
N=1000
def f(x,mu,sigma,a):
return a*(np.exp)((-(x-mu)**2)/(2*sigma**2))
fig = figure()
frame = fig.add_subplot(1,1,1)
frame.plot(vels, profile,)
frame.plot(vels,f(vels,X0, sigma,profile.max()), color="pink")
frame.set_xlabel('x-axis')
frame.set_ylabel('y-axis')
frame.grid(True)
show()
This is what the programme returns once I try to execute it.
('Mean:', 0.0)
('Standard deviation:', 0.0)
('Skewness:', nan)
('Fisher Kurtosis:', nan)
C:\Users\Thomas\Anaconda2\lib\site-packages\ipykernel\__main__.py:10: RuntimeWarning: invalid value encountered in double_scalars
C:\Users\Thomas\Anaconda2\lib\site-packages\ipykernel\__main__.py:11: RuntimeWarning: invalid value encountered in double_scalars
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-13-80786b6f31c3> in <module>()
24 fig = figure()
25 frame = fig.add_subplot(1,1,1)
---> 26 frame.plot(vels, profile,)
27 frame.plot(vels,f(vels,X0, sigma,profile.max()), color="pink")
28 frame.set_xlabel('x-axis')
C:\Users\Thomas\Anaconda2\lib\site-packages\matplotlib\__init__.pyc in inner(ax, *args, **kwargs)
1889 warnings.warn(msg % (label_namer, func.__name__),
1890 RuntimeWarning, stacklevel=2)
-> 1891 return func(ax, *args, **kwargs)
1892 pre_doc = inner.__doc__
1893 if pre_doc is None:
C:\Users\Thomas\Anaconda2\lib\site-packages\matplotlib\axes\_axes.pyc in plot(self, *args, **kwargs)
1404 kwargs = cbook.normalize_kwargs(kwargs, _alias_map)
1405
-> 1406 for line in self._get_lines(*args, **kwargs):
1407 self.add_line(line)
1408 lines.append(line)
C:\Users\Thomas\Anaconda2\lib\site-packages\matplotlib\axes\_base.pyc in _grab_next_args(self, *args, **kwargs)
405 return
406 if len(remaining) <= 3:
--> 407 for seg in self._plot_args(remaining, kwargs):
408 yield seg
409 return
C:\Users\Thomas\Anaconda2\lib\site-packages\matplotlib\axes\_base.pyc in _plot_args(self, tup, kwargs)
383 x, y = index_of(tup[-1])
384
--> 385 x, y = self._xy_from_xy(x, y)
386
387 if self.command == 'plot':
C:\Users\Thomas\Anaconda2\lib\site-packages\matplotlib\axes\_base.pyc in _xy_from_xy(self, x, y)
245 if x.ndim > 2 or y.ndim > 2:
246 raise ValueError("x and y can be no greater than 2-D, but have "
--> 247 "shapes {} and {}".format(x.shape, y.shape))
248
249 if x.ndim == 1:
ValueError: x and y can be no greater than 2-D, but have shapes (1L,) and (1L, 28L, 28L)

Cubic spline memory error

On a computer with 4GB of memory this simple interpolation leads to a memory error:
(based on: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html)
import numpy as np
from scipy.interpolate import interp1d
x = np.linspace(0, 10, 80000)
y = np.cos(-x**2/8.0)
f2 = interp1d(x, y, kind='cubic')
I thought about cutting the data into chunks, but is there a way I can perform this cubic spline interpolation without requiring so much memory?
Why does it even get in trouble?

If you look at the traceback when the error occurs, you'll see something like:
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-4-1e538e8d766e> in <module>()
----> 1 f2 = interp1d(x, y, kind='cubic')
/home/warren/local_scipy/lib/python2.7/site-packages/scipy/interpolate/interpolate.py in __init__(self, x, y, kind, axis, copy, bounds_error, fill_value)
390 else:
391 minval = order + 1
--> 392 self._spline = splmake(x, y, order=order)
393 self._call = self.__class__._call_spline
394
/home/warren/local_scipy/lib/python2.7/site-packages/scipy/interpolate/interpolate.py in splmake(xk, yk, order, kind, conds)
1754
1755 # the constraint matrix
-> 1756 B = _fitpack._bsplmat(order, xk)
1757 coefs = func(xk, yk, order, conds, B)
1758 return xk, coefs, order
MemoryError:
The function that is failing is scipy.interpolate._fitpack._bsplmat(order, xk). This function creates a 2-d array of 64-bit floats with shape (len(xk), len(xk) + order - 1). In your case, this is over 51GB.
Instead of interp1d, see if InterpolatedUnivariateSpline works for you. For example,
import numpy as np
from scipy.interpolate import InterpolatedUnivariateSpline
x = np.linspace(0, 10, 80000)
y = np.cos(-x**2/8.0)
f2 = InterpolatedUnivariateSpline(x, y, k=3)
I don't get a memory error with this.

"argument must be a callable function" when using scipy.integrate.quad

I've written some code that enables me to take data from DICOM files and separate the data into the individual FID signals.
from pylab import *
import dicom
import numpy as np
import scipy as sp
plan=dicom.read_file("1.3.46.670589.11.38085.5.22.3.1.4792.2013050818105496124")
all_points = array(plan.SpectroscopyData)
cmplx_data = all_points[0::2] -1j*all_points[1::2]
frames = int(plan.NumberOfFrames)
fid_pts = len(cmplx_data)/frames
del_t = plan.AcquisitionDuration / (frames * fid_pts)
fid_list = []
for fidN in arange(frames):
offset = fidN * fid_pts
current_fid = cmplx_data[offset:offset+fid_pts]
fid_list.append(current_fid)
I'd now like to quantify some of this data so, after applying a Fourier transform and shift I've tried to use Scipy's quad function and encountered the following error:
spec = fftshift(fft(fid_list[0])))
sp.integrate.quad(spec, 660.0, 700.0)
error Traceback (most recent call last)
/home/dominicc/Experiments/In Vitro/Glu, Cr/Phantom 1: 10mM Cr, 5mM Glu/WIP_SV_PRESS_ME_128TEs_5ms_spacing_1828/<ipython-input-107-17cb50e45927> in <module>()
----> 1 sp.integrate.quad(fid, 660.0, 700.0)
/usr/lib/python2.7/dist-packages/scipy/integrate/quadpack.pyc in quad(func, a, b, args, full_output, epsabs, epsrel, limit, points, weight, wvar, wopts, maxp1, limlst)
243 if type(args) != type(()): args = (args,)
244 if (weight is None):
--> 245 retval = _quad(func,a,b,args,full_output,epsabs,epsrel,limit,points)
246 else:
247 retval = _quad_weight(func,a,b,args,full_output,epsabs,epsrel,limlst,limit,maxp1,weight,wvar,wopts)
/usr/lib/python2.7/dist-packages/scipy/integrate/quadpack.pyc in _quad(func, a, b, args, full_output, epsabs, epsrel, limit, points)
307 if points is None:
308 if infbounds == 0:
--> 309 return _quadpack._qagse(func,a,b,args,full_output,epsabs,epsrel,limit)
310 else:
311 return _quadpack._qagie(func,bound,infbounds,args,full_output,epsabs,epsrel,limit)
error: First argument must be a callable function.
Could somebody please suggest a way to make this object callable? I'm still not particularly clear after reading What is a "callable" in Python?. After reading that I tried
def fid():
spec = fftshift(fft(fid_list[0]))
return spec
Which returned the same error.
Any help would be greatly appreciated, thanks.

Since you are integrating a function defined only on a grid (i.e., an array of numbers), you need to use one of the routines from "Integration, given fixed samples".
In fact, the first argument of quad() must be a function, something you can call (a "callable"). For instance, you can do:
>>> from scipy.integrate import quad
>>> def f(x):
... return x**2
...
>>> quad(f, 0, 1)
(0.33333333333333337, 3.700743415417189e-15)
This can be done instead by sampling f():
>>> from scipy.integrate import simps
>>> x_grid = numpy.linspace(0, 1, 101) # 101 numbers between 0 and 1
>>> simps(x_grid**2, dx=x_grid[1]-x_grid[0]) # x_grid**2 is an *array*. dx=0.01 between x_grid values
0.33333333333333337
Your situation is like in this second example: you integrate a sampled function, so it is natural to use one or cumtrapz(), simps() or romb().