I have a function findMaxEval which I invoke in a following way:
eMax0,var0=findMaxEval(np.diag(eVal0),q,bWidth=.01)
where np.diag(eVal0) is an ndarray of shape (1000,), q is a number (10).
findMaxEval has the following definition:
def findMaxEval(eVal,q,bWidth):
out=minimize(lambda *x:errPDFs(*x),.5,args= (eVal,q,bWidth),bounds=((1E-5,1-1E-5),))
if out['success']:var=out['x'][0]
else:var=1
eMax=var*(1+(1./q)**.5)**2
return eMax,var
This funtion tries to minimize errPDFs which is defined as follows:
def errPDFs(var,eVal,q,bWidth,pts=1000):
pdf0=mpPDF(var,q,pts)
pdf1=fitKDE(eVal,bWidth,x=pdf0.index.values)
sse=np.sum((pdf1-pdf0)**2)
return sse
var is a number which I pass in the findMaxEval function in minimize, initial value is 0.5.
Further, mpPDF and fitKDE are defined:
def mpPDF(var,q,pts):
eMin,eMax=var*(1-(1./q)**.5)**2,var*(1+(1./q)**.5)**2
eVal=np.linspace(eMin,eMax,pts)
pdf=q/(2*np.pi*var*eVal)*((eMax-eVal)*(eVal-eMin))**.5
pdf=pd.Series(pdf,index=eVal)
return pdf
def fitKDE(obs,bWidth=.25,kernel='gaussian',x=None):
if len(obs.shape)==1:obs=obs.reshape(-1,1)
kde=KernelDensity(kernel=kernel,bandwidth=bWidth).fit(obs)
if x is None:x=np.unique(obs).reshape(-1,1)
if len(x.shape)==1:x=x.reshape(-1,1)
logProb=kde.score_samples(x) # log(density)
pdf=pd.Series(np.exp(logProb),index=x.flatten())
return pdf
When I invoke findMaxEval (first line in the description), I get the following error:
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-25-abd7cf64e843> in <module>
----> 1 eMax0,var0=findMaxEval(np.diag(eVal0),q,bWidth=.01)
2 nFacts0=eVal0.shape[0]-np.diag(eVal0)[::-1].searchsorted(eMax0)
<ipython-input-24-f44a1e9d84b1> in findMaxEval(eVal, q, bWidth)
1 def findMaxEval(eVal,q,bWidth):
2 # Find max random eVal by fitting Marcenko’s dist
----> 3 out=minimize(lambda *x:errPDFs(*x),.5,args= (eVal,q,bWidth),bounds=((1E-5,1-1E-5),))
4 if out['success']:var=out['x'][0]
5 else:var=1
/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
598 return _minimize_neldermead(fun, x0, args, callback, **options)
599 elif meth == 'powell':
--> 600 return _minimize_powell(fun, x0, args, callback, **options)
601 elif meth == 'cg':
602 return _minimize_cg(fun, x0, args, jac, callback, **options)
/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)
333
334 while 1:
--> 335 # x, f, g, wa, iwa, task, csave, lsave, isave, dsave = \
336 _lbfgsb.setulb(m, x, low_bnd, upper_bnd, nbd, f, g, factr,
337 pgtol, wa, iwa, task, iprint, csave, lsave,
/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py in func_and_grad(x)
278 # unbounded variables must use None, not +-inf, for optimizer to work properly
279 bounds = [(None if l == -np.inf else l, None if u == np.inf else u) for l, u in bounds]
--> 280
281 if disp is not None:
282 if disp == 0:
/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/optimize.py in function_wrapper(*wrapper_args)
324
325 def function_wrapper(*wrapper_args):
--> 326 ncalls[0] += 1
327 return function(*(wrapper_args + args))
328
<ipython-input-24-f44a1e9d84b1> in <lambda>(*x)
1 def findMaxEval(eVal,q,bWidth):
2 # Find max random eVal by fitting Marcenko’s dist
----> 3 out=minimize(lambda *x:errPDFs(*x),.5,args= (eVal,q,bWidth),bounds=((1E-5,1-1E-5),))
4 if out['success']:var=out['x'][0]
5 else:var=1
<ipython-input-23-24070a331535> in errPDFs(var, eVal, q, bWidth, pts)
1 def errPDFs(var,eVal,q,bWidth,pts=1000):
2 # Fit error
----> 3 pdf0=mpPDF(var,q,pts) # theoretical pdf
4 pdf1=fitKDE(eVal,bWidth,x=pdf0.index.values) # empirical pdf
5 sse=np.sum((pdf1-pdf0)**2)
<ipython-input-17-565d70018af2> in mpPDF(var, q, pts)
10 eVal=np.linspace(eMin,eMax,pts)
11 pdf=q/(2*np.pi*var*eVal)*((eMax-eVal)*(eVal-eMin))**.5
---> 12 pdf=pd.Series(pdf,index=eVal)
13 return pdf
/opt/anaconda3/lib/python3.7/site-packages/pandas/core/series.py in __init__(self, data, index, dtype, name, copy, fastpath)
312
313 def _init_dict(self, data, index=None, dtype=None):
--> 314 """
315 Derive the "_data" and "index" attributes of a new Series from a
316 dictionary input.
/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/construction.py in sanitize_array(data, index, dtype, copy, raise_cast_failure)
Exception: Data must be 1-dimensional
I don't understand what should be 1-Dimensional. np.diag(eVal0) is of shape (1000,).
I looked at all the other similar questions, but none seems to help me solve this.
Thanks.
The error is unrelated to bounds.
For some reason minimize() calls the custom function errPDFs() with the argument to be optimized - minimize() calls this for x0 - which is an array. So if you redefine the function errPDFs() to extract the first element of an array:
def errPDFs(var, eVal, q, bWidth, pts=1000):
print("var:"+var)
pdf0 = mpPDF(var[0], q, pts) #theoretical pdf
pdf1 = fitKDE(eVal, bWidth, x=pdf0.index.values) #empirical pdf
sse = np.sum((pdf1-pdf0)**2)
print("sse:"+str(sse))
return sse
It should work.
Sample output:
>>> out = minimize(lambda *x: errPDFs(*x), .5, args=(eVal, q, bWidth),bounds=
((1E-5, 1-1E-5),))
var:[0.5]
sse:743.6200749295413
var:[0.50000001]
sse:743.6199819531047
var:[0.99999]
sse:289.1462047531385
...
Updated 6/29 ... I got it to run this way, which is strange because it is the same thing, must be a bug in the library or casting explicitly like this gets it into the precise format desired:
import numpy as np
import pandas as pd
from scipy.optimize import minimize
from sklearn.neighbors import KernelDensity
def findMaxEval(eVal, q, bWidth):
bnds = ((float(1e5/10000000000), float(0.99999*-1)),)
print(bnds)
out = minimize(lambda *x: errPDFs(*x), .5, args=(eVal, q, bWidth), bounds=bnds)
if out['success']: var = out['x'][0]
else: var = 1
eMax = var*(1+(1./q)**.5)**2
return eMax, var
def errPDFs(var, eVal, q, bWidth, pts = 1000):
pdf0 = mpPDF(var, q, pts)
pdf1 = fitKDE(eVal, bWidth, x=pdf0.index.values)
sse=np.sum((pdf1-pdf0)**2)
return sse
def mpPDF(var, q, pts):
eMin, eMax=var*(1-(1./q)**.5)**2,var*(1+(1./q)**.5)**2
eVal = np.linspace(eMin, eMax, pts)
pdf = q/(2*np.pi*var*eVal)*((eMax-eVal)*(eVal-eMin))**.5
pdf = pd.Series(pdf, index=eVal)
return pdf
def fitKDE(obs, bWidth = .25, kernel='gaussian', x=None):
if len(obs.shape) == 1: obs = obs.reshape(-1, 1)
kde=KernelDensity(kernel=kernel, bandwidth=bWidth).fit(obs)
if x is None: x = np.unique(obs).reshape(-1, 1)
if len(x.shape) == 1: x = x.reshape(-1, 1)
logProb = kde.score_samples(x)
pdf=pd.Series(np.exp(logProb), index=x.flatten())
return pdf
eMax0, var0 = findMaxEval((1000,), 10, bWidth=.01)
print(eMax0)
print(var0)
Here is the updated output on Macbook in PyCharm Community, Python version 3.8.1:
Related
I have data that comes from numerical simulations (around 7000 points, more in other simulations), this data is plotted in the image bellow. As you can see it has a small interval (between 0.35 and 0.45) where this data is linear (it is theoretically linear on that region, but also i measured it and indeed it is in that region).
I want to calculate the best interval where the slope/fit is maximal/best automatically.
For the moment i do it by hand, i chose by eye an interval, then calculate the fit. Then repeat until i find a slope/fit that looks good (This is what i did for this image).
I know this can be done by brute force by calculating all possible intervals in my data and comparing the slopes. However i want to doit a little more intelligently.
I tried to do it whit scipy minimize, since this is a function i have used multiple times and know how to use it. For this i take my data and pass it to miminize, with hoppes that it will search for 2 numbers the min and max of the interval.
I tried by searching for the values on x (ie 0.35 and 0.45), and also the index of the data (ie 3000 and 3700). As follows,
Mini = minimize(wrapper(MinFit, x_dat, y_dat), (0,nPoints), method='Powell')
def wrapper(func, x, y):
def newfunc(t):
return func(*t, x, y)
return(newfunc)
def MinFit(iMin, iMax, x_dat, y_dat):
iMin, iMax = np.int(np.ceil(np.abs(iMin))), np.int(np.ceil(np.abs(iMax)))
if(iMin == iMax):
print('i = ', iMin, iMax)
FitParam = [0., 0.]
else:
Min, Max = np.min([x_dat[iMin], x_dat[iMax]]), np.max([x_dat[iMin], x_dat[iMax]])
print('i = ', iMin, iMax)
print(x_dat[iMin], x_dat[iMax])
x_dat_short, y_dat_short = CutData(x_dat, y_dat, Max, Min)
FitParam, FitCovar = curve_fit(Linear, x_dat_short, y_dat_short, p0=(0,0), maxfev= 5000)
return(-FitParam[0])
Where MinFit is a function that takes 4 inputs (indexMin, indexMax, x_data, y_data), wrapper just an intermediary function to properly use minimize (reduce inputs to the once i want to search) and nPoints is the index of the last data point.
However the code properly calculates the first iteration and then it breaks (for both index and float search) and outputs the following error:
Mini = minimize(wrapper(MinFit, x_dat, y_dat), (0,nPoints), method='Powell')
i = 0 58333
0.0 0.699996
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-117-295df9b333b7> in <module>
----> 1 Mini = minimize(wrapper(MinFit, x_dat, y_dat), (0,nPoints), method='Powell')
/.../_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
588 return _minimize_neldermead(fun, x0, args, callback, **options)
589 elif meth == 'powell':
--> 590 return _minimize_powell(fun, x0, args, callback, **options)
591 elif meth == 'cg':
592 return _minimize_cg(fun, x0, args, jac, callback, **options)
/.../optimize.py in _minimize_powell(func, x0, args, callback, xtol, ftol, maxiter, maxfev, disp, direc, return_all, **unknown_options)
2610 fx2 = fval
2611 fval, x, direc1 = _linesearch_powell(func, x, direc1,
-> 2612 tol=xtol * 100)
2613 if (fx2 - fval) > delta:
2614 delta = fx2 - fval
/.../optimize.py in _linesearch_powell(func, p, xi, tol)
2396 def myfunc(alpha):
2397 return func(p + alpha*xi)
-> 2398 alpha_min, fret, iter, num = brent(myfunc, full_output=1, tol=tol)
2399 xi = alpha_min*xi
2400 return squeeze(fret), p + xi, xi
/.../optimize.py in brent(func, args, brack, tol, full_output, maxiter)
2107 options = {'xtol': tol,
2108 'maxiter': maxiter}
-> 2109 res = _minimize_scalar_brent(func, brack, args, **options)
2110 if full_output:
2111 return res['x'], res['fun'], res['nit'], res['nfev']
/.../optimize.py in _minimize_scalar_brent(func, brack, args, xtol, maxiter, **unknown_options)
2139 full_output=True, maxiter=maxiter)
2140 brent.set_bracket(brack)
-> 2141 brent.optimize()
2142 x, fval, nit, nfev = brent.get_result(full_output=True)
2143 return OptimizeResult(fun=fval, x=x, nit=nit, nfev=nfev,
/.../optimize.py in optimize(self)
1923 # set up for optimization
1924 func = self.func
-> 1925 xa, xb, xc, fa, fb, fc, funcalls = self.get_bracket_info()
1926 _mintol = self._mintol
1927 _cg = self._cg
/.../optimize.py in get_bracket_info(self)
1897 ### carefully DOCUMENT any CHANGES in core ##
1898 if brack is None:
-> 1899 xa, xb, xc, fa, fb, fc, funcalls = bracket(func, args=args)
1900 elif len(brack) == 2:
1901 xa, xb, xc, fa, fb, fc, funcalls = bracket(func, xa=brack[0],
/.../optimize.py in bracket(func, xa, xb, args, grow_limit, maxiter)
2322 _gold = 1.618034 # golden ratio: (1.0+sqrt(5.0))/2.0
2323 _verysmall_num = 1e-21
-> 2324 fa = func(*(xa,) + args)
2325 fb = func(*(xb,) + args)
2326 if (fa < fb): # Switch so fa > fb
/.../optimize.py in myfunc(alpha)
2395 """
2396 def myfunc(alpha):
-> 2397 return func(p + alpha*xi)
2398 alpha_min, fret, iter, num = brent(myfunc, full_output=1, tol=tol)
2399 xi = alpha_min*xi
TypeError: can only concatenate tuple (not "float") to tuple
(I deleted the paths for privacy reasons and put ...)
Is there a way to do what i want? Can minimize do it? If so how?
Second question, is there a way to constrain scipy minimize? (ie. only search for values between a=1 and b=nPoints, or any other value and type)
Thanks.
I am trying to calibrate a model using pykalman and the scipy optimiser. For some reasons scipy seem to think that my input is a masked array, but it is not. I have added the code below:
k = 0.00000000000001 #small sarting value
T = np.array([1, 2, 3, 4, 5] , dtype=int) #maturities
delta = 1e-9
x = k
transition_covariance=delta / (1 - delta) * np.eye(1)
def obj(x):
k = x[0]
Q = (1 - k) *np.eye(1)
Z = np.exp(-k*T)
Z =Z[:, None] #reshape to 2 dim
transition_covariance=delta / (1 - delta) * np.eye(1)
observation_covariance=0.001*np.eye(len(T))
kf = KalmanFilter(transition_matrices=np.eye(1)*(1-k),
observation_matrices= Z,
transition_covariance=transition_covariance,
observation_covariance=observation_covariance)
lk = kf.loglikelihood(X) # X is my observed data, 1265 rows × 5 columns
return -lk
x = np.array([k])
k =scipy.optimize.minimize(obj, x, method= 'TNC')
Running the scipy optimiser I get the error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-16-8d87c9f2ced0> in <module>
1 # method that works: TNC, Nelder-Mead
----> 2 k =scipy.optimize.minimize(obj, x, method= 'TNC')
~\Anaconda3\lib\site-packages\scipy\optimize\_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
618 callback=callback, **options)
619 elif meth == 'tnc':
--> 620 return _minimize_tnc(fun, x0, args, jac, bounds, callback=callback,
621 **options)
622 elif meth == 'cobyla':
~\Anaconda3\lib\site-packages\scipy\optimize\tnc.py in _minimize_tnc(fun, x0, args, jac, bounds, eps, scale, offset, mesg_num, maxCGit, maxiter, eta, stepmx, accuracy, minfev, ftol, xtol, gtol, rescale, disp, callback, finite_diff_rel_step, maxfun, **unknown_options)
373 messages = MSG_NONE
374
--> 375 sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
376 finite_diff_rel_step=finite_diff_rel_step,
377 bounds=new_bounds)
~\Anaconda3\lib\site-packages\scipy\optimize\optimize.py in _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
259 # ScalarFunction caches. Reuse of fun(x) during grad
260 # calculation reduces overall function evaluations.
--> 261 sf = ScalarFunction(fun, x0, args, grad, hess,
262 finite_diff_rel_step, bounds, epsilon=epsilon)
263
~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py in __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step, finite_diff_bounds, epsilon)
74
75 self._update_fun_impl = update_fun
---> 76 self._update_fun()
77
78 # Gradient evaluation
~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py in _update_fun(self)
164 def _update_fun(self):
165 if not self.f_updated:
--> 166 self._update_fun_impl()
167 self.f_updated = True
168
~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py in update_fun()
71
72 def update_fun():
---> 73 self.f = fun_wrapped(self.x)
74
75 self._update_fun_impl = update_fun
~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py in fun_wrapped(x)
68 def fun_wrapped(x):
69 self.nfev += 1
---> 70 return fun(x, *args)
71
72 def update_fun():
<ipython-input-13-48731c614f15> in obj(x)
19 observation_covariance=observation_covariance)
20
---> 21 lk = kf.loglikelihood(X) # X is my observed data, 1016 rows × 17 columns
22 return -lk
~\Anaconda3\lib\site-packages\pykalman\standard.py in loglikelihood(self, X)
1470
1471 # get likelihoods for each time step
-> 1472 loglikelihoods = _loglikelihoods(
1473 observation_matrices, observation_offsets, observation_covariance,
1474 predicted_state_means, predicted_state_covariances, Z
~\Anaconda3\lib\site-packages\pykalman\standard.py in _loglikelihoods(observation_matrices, observation_offsets, observation_covariance, predicted_state_means, predicted_state_covariances, observations)
165 + observation_covariance
166 )
--> 167 loglikelihoods[t] = log_multivariate_normal_density(
168 observation[np.newaxis, :],
169 predicted_observation_mean[np.newaxis, :],
~\Anaconda3\lib\site-packages\pykalman\utils.py in log_multivariate_normal_density(X, means, covars, min_covar)
71 lower=True)
72 cv_log_det = 2 * np.sum(np.log(np.diagonal(cv_chol)))
---> 73 cv_sol = solve_triangular(cv_chol, (X - mu).T, lower=True).T
74 log_prob[:, c] = - .5 * (np.sum(cv_sol ** 2, axis=1) + \
75 n_dim * np.log(2 * np.pi) + cv_log_det)
~\Anaconda3\lib\site-packages\scipy\linalg\basic.py in solve_triangular(a, b, trans, lower, unit_diagonal, overwrite_b, debug, check_finite)
332
333 a1 = _asarray_validated(a, check_finite=check_finite)
--> 334 b1 = _asarray_validated(b, check_finite=check_finite)
335 if len(a1.shape) != 2 or a1.shape[0] != a1.shape[1]:
336 raise ValueError('expected square matrix')
~\Anaconda3\lib\site-packages\scipy\_lib\_util.py in _asarray_validated(a, check_finite, sparse_ok, objects_ok, mask_ok, as_inexact)
259 if not mask_ok:
260 if np.ma.isMaskedArray(a):
--> 261 raise ValueError('masked arrays are not supported')
262 toarray = np.asarray_chkfinite if check_finite else np.asarray
263 a = toarray(a)
ValueError: masked arrays are not supported
Maybe something to do with an update in the scipy library? It worked with a different setup, so that's all I can think. My versions are:
Python 3.8.5
Scipy 1.5.2
Pykalman 0.9.5
Thanks!
I found the solution, which involves a small change in the utils.py file in the pykalman library (line 73):
try:
cv_sol = solve_triangular(cv_chol, (X - mu).T, lower=True).T
except ValueError:
cv_sol = np.linalg.solve(cv_chol, (X - mu).T).T
Source:
https://github.com/pykalman/pykalman/issues/83
i'm working on portefolio optimization so i want to minimize a function which computes the variance of its net result subject to its mean is equal or larger than a certain expected result exp :
minimize Var (Net Result) subject to E(Net result) >=exp
i defined my objective function as :
R=[]
length_to_split=[26,56,46,38,48,38,45,55,59,47]
from itertools import accumulate
def objective(z):
for i in range(0, len(dataframe)):
R.append((1- max(0, (dataframe.iloc[i,1]- z) /dataframe.iloc[i,1]) )*(dataframe.iloc[i,2]-dataframe.iloc[i,1]))
R_ann=[R[x - y: x] for x, y in zip(accumulate(length_to_split), length_to_split)] #each sublist
#contains results of one year
R_avg=sum([sum(r) for r in R_ann ] )/len([sum(r) for r in R_ann ]) #[sum(r) for r in R_ann ] will
#return 10 results corresponding to net results for each year from 2010 to 2019.
var = sum((Rj-R_avg)**2 for Rj in [sum(r) for r in R_ann ]) / len([sum(r) for r in R_ann ])
return var
dataframe is a data which contains 4 columns where the first one represents costs and the second one premiums and then i defined my constraint function as :
K=[]
from statistics import mean
def constraint(z):
exp=1000000
for i in range(0, len(dataframe)):
if dataframe.iloc[i,1]>z:
K.append((1- ((dataframe.iloc[i,1]-z) /dataframe.iloc[i,1])) * (dataframe.iloc[i,2]-dataframe.iloc[1,1]))
else:
K.append((dataframe.iloc[i,2]-dataframe.iloc[i,1]))
K_ann=[K[v - w: v] for v, w in zip(accumulate(length_to_split), length_to_split)]
return mean([sum(j) for j in K_ann]) -exp
When i tried to minimize the objective function as,
from scipy.optimize import minimize
con = {'type': 'ineq', 'fun': constraint}
guess=500000
minimize(objective,guess,constraints=con)
it gave me the error below :
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-404-ff80aabebc42> in <module>
2 con = {'type': 'ineq', 'fun': constraint}
3 guess=500000
----> 4 minimize(objective,guess,constraints=con)
5 # initial guesses
~\Anaconda3\lib\site-packages\scipy\optimize\_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
607 elif meth == 'slsqp':
608 return _minimize_slsqp(fun, x0, args, jac, bounds,
--> 609 constraints, callback=callback, **options)
610 elif meth == 'trust-constr':
611 return _minimize_trustregion_constr(fun, x0, args, jac, hess, hessp,
~\Anaconda3\lib\site-packages\scipy\optimize\slsqp.py in _minimize_slsqp(func, x0, args, jac, bounds, constraints, maxiter, ftol, iprint, disp, eps, callback, **unknown_options)
313 for c in cons['eq']]))
314 mieq = sum(map(len, [atleast_1d(c['fun'](x, *c['args']))
--> 315 for c in cons['ineq']]))
316 # m = The total number of constraints
317 m = meq + mieq
~\Anaconda3\lib\site-packages\scipy\optimize\slsqp.py in <listcomp>(.0)
313 for c in cons['eq']]))
314 mieq = sum(map(len, [atleast_1d(c['fun'](x, *c['args']))
--> 315 for c in cons['ineq']]))
316 # m = The total number of constraints
317 m = meq + mieq
<ipython-input-401-fe7f26be36f9> in constraint(z)
10 K.append((dataframe.iloc[i,2]-dataframe.iloc[i,1]))
11 K_ann=[K[v - w: v] for v, w in zip(accumulate(length_to_split), length_to_split)]
---> 12 return mean([sum(j) for j in K_ann]) -exp
13 #contrainte(900000)
14
~\Anaconda3\lib\statistics.py in mean(data)
309 if n < 1:
310 raise StatisticsError('mean requires at least one data point')
--> 311 T, total, count = _sum(data)
312 assert count == n
313 return _convert(total/n, T)
~\Anaconda3\lib\statistics.py in _sum(data, start)
145 for typ, values in groupby(data, type):
146 T = _coerce(T, typ) # or raise TypeError
--> 147 for n,d in map(_exact_ratio, values):
148 count += 1
149 partials[d] = partials_get(d, 0) + n
~\Anaconda3\lib\statistics.py in _exact_ratio(x)
227 return (x, None)
228 msg = "can't convert type '{}' to numerator/denominator"
--> 229 raise TypeError(msg.format(type(x).__name__))
230
231
TypeError: can't convert type 'ndarray' to numerator/denominator
Could someone help me solve this problem please? i could not get the correct result i think the problem is with my constraint function even when i fix this type of error.
I am using fmin_l_bfgs_b function while doing neural style transfer and keep getting
TypeError: 'numpy.float32' object is not callable
Details of the error block are:
TypeError Traceback (most recent call last)
<ipython-input-10-4699dceebbd9> in <module>()
----> 1 generate_art('/content/nst_images/5.jpg', '/content/nst_images/a.jpg',1, img_height=400)
4 frames
<ipython-input-9-67cbbed20548> in generate_art(content_image_path, style_image_path, iterations, img_height)
38 for i in range(iterations):
39
---> 40 x, min_val, info = fmin_l_bfgs_b(evaluator.loss(x,img_height,img_width,fetch_loss_and_grads),x, fprime=evaluator.grads, maxfun=20)
41 img = x.copy().reshape((img_height, img_width, 3))
42 img = deprocess_image(img)
/usr/local/lib/python3.6/dist-packages/scipy/optimize/lbfgsb.py in fmin_l_bfgs_b(func, x0, fprime, args, approx_grad, bounds, m, factr, pgtol, epsilon, iprint, maxfun, maxiter, disp, callback, maxls)
197
198 res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
--> 199 **opts)
200 d = {'grad': res['jac'],
201 'task': res['message'],
/usr/local/lib/python3.6/dist-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)
343 # until the completion of the current minimization iteration.
344 # Overwrite f and g:
--> 345 f, g = func_and_grad(x)
346 elif task_str.startswith(b'NEW_X'):
347 # new iteration
/usr/local/lib/python3.6/dist-packages/scipy/optimize/lbfgsb.py in func_and_grad(x)
293 else:
294 def func_and_grad(x):
--> 295 f = fun(x, *args)
296 g = jac(x, *args)
297 return f, g
/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py in function_wrapper(*wrapper_args)
325 def function_wrapper(*wrapper_args):
326 ncalls[0] += 1
--> 327 return function(*(wrapper_args + args))
328
329 return ncalls, function_wrapper
TypeError: 'numpy.float32' object is not callable
Any suggestions on how to troubleshoot this? thanks!
This was a silly error. for fmin_l_bfgs_b, arguments of the loss function have to be passed separately as args = ()
There is an answer to this very question here however the solution doesn't seem to work for me.
my function is the following:
def slopefunction_cf(xin,amp,phase,off,freq,intercept,t):
fitted =np.zeros((4,12))
for im in t:
fitted[:,im-1] = intercept[im-1]+[amp*np.sin(freq*(im)+phase)+off]*xin
return fitted.ravel()
where intercept and t are additional arguments, that in leastsq I would pass as args=
I tried:
popt, pcov = curve_fit(slopefunction_cf, np.arange(0,4), datain, p0=paramin, args=(intercept,t) )
and get the error:
TypeError: func_wrapped() takes exactly 1 argument (3 given)
Then I tried what suggested in the answer I linked and transformed it to:
def define_var_args(interceptin,tin):
def slopefunction_cf(xin,amp,phase,off,freq):
intercept = interceptin
t = tin
fitted =np.zeros((4,12))
for im in t:
fitted[:,im-1] = intercept[im-1]+[amp*np.sin(freq*(im)+phase)+off]*xin
return fitted.ravel()
and then called curve_fit:
popt, pcov = curve_fit(define_var_args(intercept, t), np.arange(0,4), datain, p0=paramin )
but I get the following error (i copied everything)
--> 115 popt, pcov = curve_fit(define_var_args(interceptconus, t), np.arange(0,4), datain.ravel(), p0=paramin )
116
117 print popt
/software/centos6/x86_64/canopy-1.7.4/Canopy_64bit/User/lib/python2.7/site-packages/scipy/optimize/minpack.pyc in curve_fit(f, xdata, ydata, p0, sigma, absolute_sigma, check_finite, bounds, method, jac, **kwargs)
734 # Remove full_output from kwargs, otherwise we're passing it in twice.
735 return_full = kwargs.pop('full_output', False)
--> 736 res = leastsq(func, p0, Dfun=jac, full_output=1, **kwargs)
737 popt, pcov, infodict, errmsg, ier = res
738 cost = np.sum(infodict['fvec'] ** 2)
/software/centos6/x86_64/canopy-1.7.4/Canopy_64bit/User/lib/python2.7/site-packages/scipy/optimize/minpack.pyc in leastsq(func, x0, args, Dfun, full_output, col_deriv, ftol, xtol, gtol, maxfev, epsfcn, factor, diag)
375 if not isinstance(args, tuple):
376 args = (args,)
--> 377 shape, dtype = _check_func('leastsq', 'func', func, x0, args, n)
378 m = shape[0]
379 if n > m:
/software/centos6/x86_64/canopy-1.7.4/Canopy_64bit/User/lib/python2.7/site-packages/scipy/optimize/minpack.pyc in _check_func(checker, argname, thefunc, x0, args, numinputs, output_shape)
24 def _check_func(checker, argname, thefunc, x0, args, numinputs,
25 output_shape=None):
---> 26 res = atleast_1d(thefunc(*((x0[:numinputs],) + args)))
27 if (output_shape is not None) and (shape(res) != output_shape):
28 if (output_shape[0] != 1):
/software/centos6/x86_64/canopy-1.7.4/Canopy_64bit/User/lib/python2.7/site-packages/scipy/optimize/minpack.pyc in func_wrapped(params)
452 if transform is None:
453 def func_wrapped(params):
--> 454 return func(xdata, *params) - ydata
455 elif transform.ndim == 1:
456 def func_wrapped(params):
TypeError: 'NoneType' object is not callable
I am not even sure what the error is - if the code or something in my data? it's a bit obscure.
Is there another way to pass additional parameters?
EDIT: Adding some data and slight change to the above function
t = np.arange(1,13)
paramin = [0.40569370873086791, 1, -0.88134718450091365, 0.5]
datain = [-3.45083921, -3.405628 , -2.57401114, -1.7632551 , -1.29342567,
-1.44200248, -2.05366981, -2.51467815, -2.79597983, -2.95528074,
-2.90766603, -3.60850134, -3.90036107, -3.87760971, -3.18549774,
-2.43304533, -2.1734368 , -2.43953997, -3.1183866 , -3.57696154,
-3.72915335, -3.67344888, -3.29697388, -4.10385938, -4.88945667,
-4.82385939, -4.10568684, -3.50093464, -3.26766599, -3.78206157,
-4.51782818, -4.95149472, -4.93689182, -4.75850421, -4.22849458,
-4.91811193, -5.34080606, -5.40676402, -4.66050702, -4.29447163,
-4.16430688, -4.75008652, -5.57106708, -5.58801663, -5.96637981,
-5.51549722, -4.94026303, -5.42975354]
intercept = [-3.39651633, -3.33601661, -2.55447417, -1.69869584, -1.26867791,
-1.41340658, -2.02249291, -2.56860547, -2.74926044, -2.91082705,
-2.78895263, -3.57335517]
I already "raveled" datain, so I modified the function at the beginning of my question.
using these data I reproduce my error in a new notebook.
SOLUTION:
I had two issues in my code:
1) I was not returning the nested function
2) I did not put xin as an argument of the nested function
3) I had to put the star before the p
this now works
def define_extravar(interceptin,tin):
def slopefunction_cf(xin,*p):
intercept = interceptin
t = tin
fitted =np.zeros((4,12))
amp,phase,off,freq = p
for im in t:
fitted[:,im-1] = intercept[im-1]+[amp*np.sin(freq*(im)+phase)+off]*xin
return fitted.ravel()
return slopefunction_cf