We Keep Coding

TypeError: numpy boolean subtract, the `-` operator, is not supported in Scipy.Optimize

Using scipy.optimise (code below) - for a battery optimisation problem
Getting this error:
TypeError: numpy boolean subtract, the - operator, is not supported, use the bitwise_xor, the ^ operator, or the logical_xor function instead.
Which came from the minimize function directly, so I'm not sure exactly where its coming from.
line 70, in <module>
sol = minimize(objective, B0, method='SLSQP', \
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_minimize.py", line 708, in minimize
res = _minimize_slsqp(fun, x0, args, jac, bounds,
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_slsqp_py.py", line 418, in _minimize_slsqp
a = _eval_con_normals(x, cons, la, n, m, meq, mieq)
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_slsqp_py.py", line 486, in _eval_con_normals
a_eq = vstack([con['jac'](x, *con['args'])
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_slsqp_py.py", line 486, in <listcomp>
a_eq = vstack([con['jac'](x, *con['args'])
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_slsqp_py.py", line 301, in cjac
return approx_derivative(fun, x, method='2-point',
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_numdiff.py", line 505, in approx_derivative
return _dense_difference(fun_wrapped, x0, f0, h,
File "AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\scipy\optimize\_numdiff.py", line 576, in _dense_difference
df = fun(x) - f0
TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
TIME = 2
MAX_BATTERY_CHARGE_RATE = 4 #kwh
MAX_BATTERY_CAPACITY = 12 #kw
INITIAL_BATTERY_CHARGE = 5
BUY_RATE = [0.1, 0.3]
SELL_RATE = [0, 0]
L = [2, 5]
S = [1, 1]
def objective(B):
cost = 0
for i in range(TIME):
isum = (L[i] - S[i] + B[i])
if isum > 0 :
cost += BUY_RATE[i] * isum
else:
cost += SELL_RATE[i] * isum
return cost
# BOUNDS
# cannot exceed charge rate
b = (-1 * MAX_BATTERY_CHARGE_RATE, MAX_BATTERY_CHARGE_RATE)
bnds = [b for i in range(TIME)]
print(bnds)
# CONSTRAINTS
# Sum of B up to any point in time cannot be less than 0
# or greater than battery capacity
def constraint1(x):
for i in range(TIME + 1):
array = x[:i]
print(array)
if (np.sum(array) < 0) or (np.sum(array) > MAX_BATTERY_CAPACITY):
return False
return True
con1 = {'type': 'ineq', 'fun': constraint1}
# Battery charge at initial time is set
def constraint2(x):
return x[0] == INITIAL_BATTERY_CHARGE
con2 = {'type': 'eq', 'fun': constraint2}
cons = [con1, con2]
# SOLUTION
B0 = np.ones(TIME)
sol = minimize(objective, B0, method='SLSQP', \
bounds=bnds, constraints=cons)
print(sol)

Is the SLSQP method compatible with functions that return boolean values true and false? I would rework function constraint1 to return real values instead of boolean values. If SLSQP is compatible with boolean functions, could you point me to the documentation that states this?
When I saw this message, "df = fun(x) - f0 TypeError: numpy boolean subtract," I found this odd because why would the computation of the derivative involve substraction of boolean values. After looking through your code, I noticed that the function constraint1 is returning boolean values true and false. Traditionally, the objective and constraint functions return real values. I would rework function constraint1 to return real values.

i keep getting this error in python (spyder) and i have no idea how to solve it , index 1 is out of bounds for axis 0 with size 1

im trying to estimate the gjr garch model using this code below and the dataset im using is the bitcoin returns (daily) total of 1600 observations. im getting this error ''index 1 is out of bounds for axis 0 with size 1''. and above it it says :
File "C:\Users\georgios\Downloads\untitled1.py", line 97, in
estimates = fmin_slsqp(gjr_garch_likelihood, startingVals,
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize\slsqp.py", line 207, in fmin_slsqp
res = _minimize_slsqp(func, x0, args, jac=fprime, bounds=bounds,
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize\slsqp.py", line 375, in _minimize_slsqp
sf = _prepare_scalar_function(func, x, jac=jac, args=args, epsilon=eps,
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize\optimize.py", line 261, in _prepare_scalar_function
sf = ScalarFunction(fun, x0, args, grad, hess,
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize_differentiable_functions.py", line 136, in init
self._update_fun()
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize_differentiable_functions.py", line 226, in _update_fun
self._update_fun_impl()
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize_differentiable_functions.py", line 133, in update_fun
self.f = fun_wrapped(self.x)
File "C:\Users\georgios\Nieuwe map\lib\site-packages\scipy\optimize_differentiable_functions.py", line 130, in fun_wrapped
return fun(x, *args)
File "C:\Users\georgios\Downloads\untitled1.py", line 21, in gjr_garch_likelihood
sigma2[t] = (omega + alpha * eps[t-1]**2
File "C:\Users\georgios\Nieuwe map\lib\site-packages\pandas\core\series.py", line 977, in setitem
values[key] = value
IndexError: index 1 is out of bounds for axis 0 with size 1
My dataset is fine from
What I see , I only have one excel file with only one column with the returns
And the garch model is Univariate
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from numpy import size, log, pi, sum, array, zeros, diag, mat, asarray, sqrt, \
copy
from numpy.linalg import inv
from scipy.optimize import fmin_slsqp
def gjr_garch_likelihood(parameters, data, sigma2, out=None):
''' Returns negative log-likelihood for GJR-GARCH(1,1,1) model.'''
mu = parameters[0]
omega = parameters[1]
alpha = parameters[2]
gamma = parameters[3]
beta = parameters[4]
T = size(data,0)
eps = data - mu
# Data and sigma2 are T by 1 vectors
for t in range(1,T):
sigma2[t] = (omega + alpha * eps[t-1]**2
+ gamma * eps[t-1]**2 * (eps[t-1]<0) + beta * sigma2[t-1])
logliks = 0.5*(log(2*pi) + log(sigma2) + eps**2/sigma2)
loglik = sum(logliks)
if out is None:
return loglik
else:
return loglik, logliks, copy(sigma2)
def gjr_constraint(parameters, data, sigma2, out=None):
''' Constraint that alpha+gamma/2+beta<=1'''
alpha = parameters[2]
gamma = parameters[3]
beta = parameters[4]
return array([1-alpha-gamma/2-beta])
def hessian_2sided(fun, theta, args):
f = fun(theta, *args)
h = 1e-5*np.abs(theta)
thetah = theta + h
h = thetah - theta
K = size(theta,0)
h = np.diag(h)
fp = zeros(K)
fm = zeros(K)
for i in range(K):
fp[i] = fun(theta+h[i], *args)
fm[i] = fun(theta-h[i], *args)
fpp = zeros((K,K))
fmm = zeros((K,K))
for i in range(K):
for j in range(i,K):
fpp[i,j] = fun(theta + h[i] + h[j], *args)
fpp[j,i] = fpp[i,j]
fmm[i,j] = fun(theta - h[i] - h[j], *args)
fmm[j,i] = fmm[i,j]
hh = (diag(h))
hh = hh.reshape((K,1))
hh = hh # hh.T
H = zeros((K,K))
for i in range(K):
for j in range(i,K):
H[i,j] = (fpp[i,j] - fp[i] - fp[j] + f
+ f - fm[i] - fm[j] + fmm[i,j])/hh[i,j]/2
H[j,i] = H[i,j]
return H
# Import data
FTSEreturn = pd.read_csv('1.csv')
# Starting values
startingVals = array([FTSEreturn.mean(),
FTSEreturn.var() * .01,
.03, .09, .90])
# Estimate parameters
finfo = np.finfo(np.float64)
bounds = [(-10*FTSEreturn.mean(), 10*FTSEreturn.mean()),
(finfo.eps, 2*FTSEreturn.var() ),
(0.0,1.0), (0.0,1.0), (0.0,1.0)]
T = FTSEreturn.shape[0]
sigma2 = T * FTSEreturn.var()
# Pass a NumPy array, not a pandas Series
args = (np.asarray(FTSEreturn), sigma2)
estimates = fmin_slsqp(gjr_garch_likelihood, startingVals,
f_ieqcons=gjr_constraint, bounds = bounds,
args = args)
loglik, logliks, sigma2final = gjr_garch_likelihood(estimates, FTSEreturn,
sigma2, out=True)
step = 1e-5 * estimates
scores = zeros((T,5))
for i in range(5):
h = step[i]
delta = np.zeros(5)
delta[i] = h
loglik, logliksplus, sigma2 = gjr_garch_likelihood(estimates + delta, \
np.asarray(FTSEreturn), sigma2, out=True)
loglik, logliksminus, sigma2 = gjr_garch_likelihood(estimates - delta, \
np.asarray(FTSEreturn), sigma2, out=True)
scores[:,i] = (logliksplus - logliksminus)/(2*h)
I = (scores.T # scores)/T
J = hessian_2sided(gjr_garch_likelihood, estimates, args)
J = J/T
Jinv = mat(inv(J))
vcv = Jinv*mat(I)*Jinv/T
vcv = asarray(vcv)
output = np.vstack((estimates,sqrt(diag(vcv)),estimates/sqrt(diag(vcv)))).T
print('Parameter Estimate Std. Err. T-stat')
param = ['mu','omega','alpha','gamma','beta']
for i in range(len(param)):
print('{0:<11} {1:>0.6f} {2:0.6f} {3: 0.5f}'.format(param[i],
output[i,0], output[i,1], output[i,2]))
any help would be appreciated because im stuck on this one month now and i cant solve it

The error says there's an indexing error during a setitem in
sigma2[t] = (omega + alpha * eps[t-1]**2
In other words, t is too large for the array sigma2.
That's in the gjr_garch_likelihood function. t is an iterator that starts at 1. sigma2 is an argument. So we need to look at how the function is called, and the corresponding argument is.
That's more complicated, since it's a function used by fmin_slsqp. So the next step is to review that function's docs, to understand how it calls the func, and especially what arguments it provides to your gjr.... I won't do that for you!
But it's a good idea when encountering errors like this to add some diagnostic prints to your gjr... function. You need to clearly understand what gets passed to it, paying particular attention to array shapes. Don't make assumptions. Verify.
You may need to change gjr ... to accomodate the arguments, or you may need to modify how you call fmin....
Question - how much of this did you already figure out in the past month of struggle? If you already figured this out, you should have included that information in the question.

How to pass an array of input parameters in scipy.optimize.minimize?

I want to use scipy.optimize.minimize to solve for a set of parameters by minimizing an error function.
The function called "error" returns the squared error for the function that I am trying to find z1,z2,z3(the parameters) for.
I have an array of x(called "b" in the function) and y(called "real" in the function) values.
The code below works fine if I set x and y to some integer, but not if I try to pass in an array of x and y values, to act as the variable "b" and "real" in the equation to be minimized.
Trying to pass in an array of X and Y values results in the error pasted below.
Is there a way to pass in arrays to act as a variable in an equation for the minimize function, instead of just a single integer?
Here is what my code looks like:
import numpy as np
import pandas as pd
from scipy.optimize import minimize
#dataset file, with a column named x,y with 1000 rows
f = pd.read_csv('Test_Data_.txt', sep='\t')
#initial guess
x0 = [1, 2, 3]
#f['x'] and f['y'] are columns with 1000 rows
#x = f['x'].values
#y = f['y'].values
x = 1 #these parameters work fine
y = 4
#a function called inside the function to be minimized
def est(z1, z2, z3, b):
return z1 * b**2 + z2 * b + z3
#function to minimize
def error(x, real, b):
return (real - est(x[0], x[1], x[2], b))**2
print(minimize(error, x0, args = ( x, y), method='BFGS', tol=1e-6))
Feeding in the array of x and y values produces the error:
Traceback (most recent call last):
File "problem1.py", line 24, in <module>
minimize(error, x0, args = ( np.array(list(f['y'].values)), np.array(list(f['x'].values))), method='BFGS', tol=1e-6)
File "/usr/local/lib/python3.5/dist-packages/scipy/optimize/_minimize.py", line 595, in minimize
return _minimize_bfgs(fun, x0, args, jac, callback, **options)
File "/usr/local/lib/python3.5/dist-packages/scipy/optimize/optimize.py", line 970, in _minimize_bfgs
gfk = myfprime(x0)
File "/usr/local/lib/python3.5/dist-packages/scipy/optimize/optimize.py", line 300, in function_wrapper
return function(*(wrapper_args + args))
File "/usr/local/lib/python3.5/dist-packages/scipy/optimize/optimize.py", line 730, in approx_fprime
return _approx_fprime_helper(xk, f, epsilon, args=args)
File "/usr/local/lib/python3.5/dist-packages/scipy/optimize/optimize.py", line 670, in _approx_fprime_helper
grad[k] = (f(*((xk + d,) + args)) - f0) / d[k]
ValueError: setting an array element with a sequence.

logistic regression and numpy: ValueError: operands could not be broadcast together with shapes

Machine learning beginner here.
In python 3.7, I keep getting this error when trying to run numpy.optimize's fmin_tnc.
I know this type of question has been asked several times, but despite having checked my matrix dimensions and the code several times, I can't find my mistake.
Here is the function:
def compute_cost(theta, X, y, lambda_):
m = len(y)
mask = np.eye(len(theta))
mask[0,0] = 0
hypo = sigmoid(X # theta)
func = y.T # np.log(hypo) + (1-y.T) # np.log(1-hypo)
cost = -1/m * func
reg_cost = cost + lambda_/(2*m) * (mask#theta).T # (mask#theta)
grad = 1/m * X.T#(hypo-y) + lambda_/m * (mask#theta)
return reg_cost.item(), grad
Here are my dimensions:
X: (118, 3)
y: (118, 1)
theta: (3, 1)
The function call,
initial_theta = np.zeros((3,1))
lambda_ = 1
thetopt, nfeval, rc = opt.fmin_tnc(
func=compute_cost,
x0=initial_theta,
args=(X, y, 1)
)
And the error.
File "<ipython-input-21-f422f885412a>", line 16, in compute_cost
grad = 1/m * X.T#(hypo-y) + lambda_/m * (mask#theta)
ValueError: operands could not be broadcast together with shapes (3,118) (3,)
Thanks for your help!

In scipy.optimize.tnc, fmin_tnc function calls to _minimize_tnc, which seems to do the heavy lifting. In this function, almost the first thing it does (line 348) it to flatten x0:
x0 = asfarray(x0).flatten()
So what you need to do, is to reshape it in your function. Just add this line in the begging of your compute_cost function:
theta = theta.reshape((3, 1))

fmin_cg: Desired error not necessarily achieved due to precision loss

I have the following code to minimize the Cost Function with its gradient.
def trainLinearReg( X, y, lamda ):
# theta = zeros( shape(X)[1], 1 )
theta = random.rand( shape(X)[1], 1 ) # random initialization of theta
result = scipy.optimize.fmin_cg( computeCost, fprime = computeGradient, x0 = theta,
args = (X, y, lamda), maxiter = 200, disp = True, full_output = True )
return result[1], result[0]
But I am having this warning:
Warning: Desired error not necessarily achieved due to precision loss.
Current function value: 8403387632289934651424768.000000
Iterations: 0
Function evaluations: 15
Gradient evaluations: 3
My computeCost and computeGradient are defined as
def computeCost( theta, X, y, lamda ):
theta = theta.reshape( shape(X)[1], 1 )
m = shape(y)[0]
J = 0
grad = zeros( shape(theta) )
h = X.dot(theta)
squaredErrors = (h - y).T.dot(h - y)
# theta[0] = 0.0
J = (1.0 / (2 * m)) * (squaredErrors) + (lamda / (2 * m)) * (theta.T.dot(theta))
return J[0]
def computeGradient( theta, X, y, lamda ):
theta = theta.reshape( shape(X)[1], 1 )
m = shape(y)[0]
J = 0
grad = zeros( shape(theta) )
h = X.dot(theta)
squaredErrors = (h - y).T.dot(h - y)
# theta[0] = 0.0
J = (1.0 / (2 * m)) * (squaredErrors) + (lamda / (2 * m)) * (theta.T.dot(theta))
grad = (1.0 / m) * (X.T.dot(h - y)) + (lamda / m) * theta
return grad.flatten()
I have reviewed these similar questions:
scipy.optimize.fmin_bfgs: “Desired error not necessarily achieved due to precision loss”
scipy.optimize.fmin_cg: "'Desired error not necessarily achieved due to precision loss.'
scipy is not optimizing and returns "Desired error not necessarily achieved due to precision loss"
But still cannot have the solution to my problem. How to let the minimization function process converge instead of being stuck at first?
ANSWER:
I solve this problem based on #lejlot 's comments below.
He is right. The data set X is to large since I did not properly return the correct normalized value to the correct variable. Even though this is a small mistake, it indeed can give you the thought where should we look at when encountering such problems. The Cost Function value is too large leads to the possibility that there are some wrong with my data set.
The previous wrong one:
X_poly = polyFeatures(X, p)
X_norm, mu, sigma = featureNormalize(X_poly)
X_poly = c_[ones((m, 1)), X_poly]
The correct one:
X_poly = polyFeatures(X, p)
X_poly, mu, sigma = featureNormalize(X_poly)
X_poly = c_[ones((m, 1)), X_poly]
where X_poly is actually used in the following traing as
cost, theta = trainLinearReg(X_poly, y, lamda)

ANSWER:
I solve this problem based on #lejlot 's comments below.
He is right. The data set X is to large since I did not properly return the correct normalized value to the correct variable. Even though this is a small mistake, it indeed can give you the thought where should we look at when encountering such problems. The Cost Function value is too large leads to the possibility that there are some wrong with my data set.
The previous wrong one:
X_poly = polyFeatures(X, p)
X_norm, mu, sigma = featureNormalize(X_poly)
X_poly = c_[ones((m, 1)), X_poly]
The correct one:
X_poly = polyFeatures(X, p)
X_poly, mu, sigma = featureNormalize(X_poly)
X_poly = c_[ones((m, 1)), X_poly]
where X_poly is actually used in the following traing as
cost, theta = trainLinearReg(X_poly, y, lamda)

For my implementation scipy.optimize.fmin_cg also failed with the above-mentioned error in some initial guesses. Then I changed it to the BFGS method and converged.
scipy.optimize.minimize(fun, x0, args=(), method='BFGS', jac=None, tol=None, callback=None, options={'disp': False, 'gtol': 1e-05, 'eps': 1.4901161193847656e-08, 'return_all': False, 'maxiter': None, 'norm': inf})
seems that this error in cg is inevitable still as,
CG ends up with a non-descent direction

I too faced this problem and even after searching a lot for solutions nothing happened as the solutions were not clearly defined.
Then I read the documentation from scipy.optimize.fmin_cg where it is clearly mentioned that parameter x0 must be a 1-D array.
My approach was same as that of you wherein I passed 2-D matrix as x0 and I always got some precision error or divide by zero error and same warning as you got.
Then I changed my approach and passed theta as a 1-D array and converted that array into 2-D matrix inside the computeCost and computeGradient function which worked for me and I got the results as expected.
My solutiion for Logistic Regression
def sigmoid(z):
return 1 / (1 + np.exp(-z))
theta = np.zeros(features)
def computeCost(theta,X, Y):
x = np.matrix(X.values)
y = np.matrix(Y.values)
theta = np.matrix(theta)
xtheta = np.matmul(x,theta.T)
hx = sigmoid(xtheta)
cost = (np.multiply(y,np.log(hx)))+(np.multiply((1-y),np.log(1-hx)))
return -(np.sum(cost))/m
def computeGradient(theta, X, Y):
x = np.matrix(X.values)
y = np.matrix(Y.values)
theta = np.matrix(theta)
grad = np.zeros(features)
xtheta = np.matmul(x,theta.T)
hx = sigmoid(xtheta)
error = hx-Y
for i in range(0,features,1):
term = np.multiply(error,x[:,i])
grad[i] = (np.sum(term))/m
return grad
import scipy.optimize as opt
result = opt.fmin_tnc(func=computeCost, x0=theta, fprime=computeGradient, args=(X, Y))
print cost(result[0],X, Y)
Note Again that theta has to be a 1-D array
So in your code modify theta in trainLinearReg to theta = random.randn(features)

I today faced this problem.
I then noticed that my cost function was implemented wrong way and was producing high scaled errors due to which scipy was asking for more data. Hope this helps for someone like me.