I am trying to find a vector that minimizes the residual sum of squares when multiplying a matrix.
I know of scipy's optimize package (which has a minimize function). However, there is an extra constraint for my code. The sum of all entries of w (see function below) must equal 1, and no entry of w can be less than 0. Is there a package that does this for me? If not, how can I do this?
Trying to minimize w:
def w_rss(w,x0,x1):
predictions = np.dot(x0,w)
errors = x1 - predictions
rss = np.dot(errors.transpose(),errors).item(0)
return rss
X0 = np.array([[3,4,5,3],
[1,2,2,4],
[6,5,3,7],
[1,0,5,2]])
X1 = np.array([[4],
[2],
[4],
[2]])
W = np.array([[.0],
[.5],
[.5],
[.0]])
print w_rss(W,X0,X1)
So far this is my best attempt at looping through possible values of w, but it's not working properly.
def get_w(x0,x1):
J = x0.shape[1]
W0 = np.matrix([[1.0/J]*J]).transpose()
rss0 = w_rss(W0,x0,x1)
loop = range(J)
for i in loop:
W1 = W0
rss1 = rss0
while rss0 == rss1:
den = len(loop)-1
W1[i][0] += 0.01
for j in loop:
if i == j:
continue
W1[j][0] -= 0.01/den
if W1[j][0] <= 0:
loop.remove(j)
rss1 = w_rss(W1,x0,x1)
if rss1 < rss0:
#print W1
W0 = W1
rss0 = rss1
print '--'
print rss0
print W0
return W0,rss0
The SLSQP code in scipy can do this. You can use scipy.optimize.minimize with method='SLSQP, or you can use the function fmin_slsqp directly. In the following, I use fmin_slsqp.
The scipy solvers generally pass a one-dimensional array to the objective function, so to be consistent, I'll change W and X1 to be 1-d arrays, and I'll write the objective function (now called w_rss1) to expect a 1-d argument w.
The condition that all the elements in w must be between 0 and 1 is specified using the bounds argument, and the condition that the sum must be 1 is specified using the f_eqcons argument. The constraint function returns np.sum(w) - 1, so it is 0 when the sum of the elements is 1.
Here's the code:
import numpy as np
from scipy.optimize import fmin_slsqp
def w_rss1(w, x0, x1):
predictions = np.dot(x0, w)
errors = x1 - predictions
rss = (errors**2).sum()
return rss
def sum1constraint(w, x0, x1):
return np.sum(w) - 1
X0 = np.array([[3,4,5,3],
[1,2,2,4],
[6,5,3,7],
[1,0,5,2]])
X1 = np.array([4, 2, 4, 2])
W = np.array([.0, .5, .5, .0])
result = fmin_slsqp(w_rss1, W, f_eqcons=sum1constraint, bounds=[(0.0, 1.0)]*len(W),
args=(X0, X1), disp=False, full_output=True)
Wopt, fW, its, imode, smode = result
if imode != 0:
print("Optimization failed: " + smode)
else:
print(Wopt)
When I run this, the output is
[ 0.05172414 0.55172414 0.39655172 0. ]
Related
I'm struggling with using scipy to deal with lasso regression.
I wrote a code that could theoretically compute the problem:
where p = 1.
However it returns such kind of error and I'm wondering where did I go wrong. The dimension of x is (50,3) and y is (50, 1):
ValueError: The user-provided objective function must return a scalar value.
My code goes like this:
def norm_one_regression(w, X, y, lambda_):
a = y - X # w
a = a.T # a
return a + lambda_ * np.linalg.norm(w, 1)
# using scipy
w0 = np.ones((3, 1))
X0 = data['x']
y0 = data['y']
lambda_0 = 1
minimum = sop.fmin(norm_one_regression, w0, args=(X0, y0, lambda_0))
print(minimum)
Thanks for your consideration and help.
Below I implemented the gradient descent function found here. (Method: batch_gradient_descent) I wanted to try and implement it without numpy because I don't know numpy arrays well and I want to intuitively understand the required operations.
I created a simple linear relation in lists x and y such that y0 = 5x0+1. x is initially a list of two element lists of n elements and y is a st of n elements. The second element of x is 1 to account for the bias weight. The weights are initialized to [0,0].
I think my issue is in GetGradient() as I have tested GetHypothesis() - a dot product operation between the weights and each 2-element input list- and Subtract1d() - both of which are easy enough to verify on run 1 of the loop when the weights are 0.
GetGradient() calculates X.T.dot(loss) / m found in their tutorial, which is what this method implements. But I don't know where I am going wrong. Gradient shuffles off to nan-land pretty soon after 10 iterations.
First ever posted question, so I appreciate the help.
def dot(K, L):
if len(K) != len(L):
return False
return sum(i[0] * i[1] for i in zip(K, L))
def GetHypothesis(input,weights):
hyp = []
for i in range(len(input)):
d = dot(input[i],weights)
hyp.append(d)
return hyp
def Subtract1D(hyp,act):
loss = []
for i in range (len(hyp)):
loss.append(hyp[i]-act[i])
return loss
def GetGradient(x,loss):
col1 = []
col2 = []
for ele in x:
col1.append(ele[0])
col2.append(ele[1])
w1 = dot(col1,loss)
w2 = dot(col2,loss)
w1 = w1/len(x)
w2 = w2/len(x)
return [w1, w2]
def GradientDescent(x,y,w,lr=0.1):
for i in range(10):
# hypothesis
h = GetHypothesis(x,w)
loss = Subtract1D(h,y)
gradientVector = []
grad = GetGradient(x,loss)
grad[0] = grad[0]*lr
grad[1] = grad[1]*lr
w[0] = (w[0] - grad[0])
w[1] = (w[1] - grad[1])
print(grad)
return w
x = list(range(1,50))
y = [i*5+1 for i in x]
for i in range(len(x)):
x[i] = [x[i],1]
w = [0,0]
GradientDescent(x,y,w)
I'm trying to maximize the minimum between two function using exhaustive research, this solution work but loop in python consumes a lot of computing time. is there an efficient way to use numpy (mesh grid or vectorize) to solve this problem?
Code :
Functions below are used in the exhaustive research method
import numpy as np
def F1(x):
return (x/11)**10
def F2(x,y,z):
return z+x/y
def F3(x,y,z,a,b,c):
return ((x+y)**z)/((a-b)**c)
Exhaustive research method take 6 parameter (scalar or 1D array). for the moment I just want to compute my code on scalar, then I can use another function to browse those parameter if they are 1D array.
def B_F(P1, P2, P3,P4, P5, P6) :
# initializing my optimal parameters
a_Opt, b_opt, c_opt, obj_opt = 0, 0, 0, 0
# feasible set
a = np.linspace(0.0,1.0,10)
b = np.linspace(0.0,100.0,100)
c = np.linspace(0.0,100.0,100)
for i in a:
for j in b:
for k in c:
#if constraint is respected
if P1*k+P2*j+2*(i*k*j) <= F1(P3):
# calculate the max min of the two function
f_1 = F2(i,k,P6)
f_2 = F3(i,k,j,10,P4,P5)
min_f = np.minimum(f_1, f_2)
# extract optimal parameters and objective function
if obj_opt <= min_f :
a_Opt = i
b_opt = j
c_opt = k
obj_opt = min_f
exhaustive_research = np.array([[obj_opt, a_Opt, b_opt, c_opt]])
return exhaustive_research
You can do it this way:
A,B,C = np.meshgrid(a,b,c)
mask = P1*C+P2*B+2*(A*B*C) <= F1(P3)
A = A[mask]
B = B[mask]
C = C[mask]
f_1 = F2(A,C,P6)
f_2 = F3(A,C,B,10,P4,P5)
min_f = np.minimum(f_1, f_2)
ind = np.argmax(min_f)
obj_opt, a_Opt, b_opt, c_opt = min_f[ind], A[ind], B[ind], C[ind]
I have a function what takes params as vector.
I need to restrict any of the vector variables be less than 0 and vector should be summary equal to 1
I've tried to find something in goolge and scipy docs. No luck so far.
def portfolio_optimization(weight_vector):
return np.sqrt(cov_table.dot(weight_vector).sum())
bound what I need to apply:
sum(weight_vector) = 1
0 < weight_vector[i] < 1
The first condition is a constraint (sum(w)=1), as for the second you can use bounds for it. Here is a small example on how to use scipy.optimize.minimize with a weights vector having 4 elements:
import numpy as np
from scipy.optimize import minimize
# objective function
func = lambda w: np.sqrt(cov_table.dot(w).sum())
# constraint: sum(weights) = 1
fconst = lambda w: 1 - sum(w)
cons = ({'type':'eq','fun':fconst})
# initial weights
w0 = [0, 0, 0, 0]
# define bounds
b = (0.0, 1.0)
bnds = (b, b, b, b)
# minimize
sol = minimize(func,
w0,
bounds = bnds,
constraints = cons)
print(sol)
*Don't forget to assign a value to cov_table for the code to work.
I am trying to find the minimum of a natural cubic spline. I have written the following code to find the natural cubic spline. (I have been given test data and have confirmed this method is correct.) Now I can not figure out how to find the minimum of this function.
This is the data
xdata = np.linspace(0.25, 2, 8)
ydata = 10**(-12) * np.array([1,2,1,2,3,1,1,2])
This is the function
import scipy as sp
import numpy as np
import math
from numpy.linalg import inv
from scipy.optimize import fmin_slsqp
from scipy.optimize import minimize, rosen, rosen_der
def phi(x, xd,yd):
n = len(xd)
h = np.array(xd[1:n] - xd[0:n-1])
f = np.divide(yd[1:n] - yd[0:(n-1)],h)
q = [0]*(n-2)
for i in range(n-2):
q[i] = 3*(f[i+1] - f[i])
A = np.zeros(((n-2),(n-2)))
#define A for j=0
A[0,0] = 2*(h[0] + h[1])
A[0,1] = h[1]
#define A for j = n-2
A[-1,-2] = h[-2]
A[-1,-1] = 2*(h[-2] + h[-1])
#define A for in the middle
for j in range(1,(n-3)):
A[j,j-1] = h[j]
A[j,j] = 2*(h[j] + h[j+1])
A[j,j+1] = h[j+1]
Ainv = inv(A)
B = Ainv.dot(q)
b = (n)*[0]
b[1:(n-1)] = B
# now we find a, b, c and d
a = [0]*(n-1)
c = [0]*(n-1)
d = [0]*(n-1)
s = [0]*(n-1)
for r in range(n-1):
a[r] = 1/(3*h[r]) * (b[r + 1] - b[r])
c[r] = f[r] - h[r]*((2*b[r] + b[r+1])/3)
d[r] = yd[r]
#solution 1 start
for m in range(n-1):
if xd[m] <= x <= xd[m+1]:
s = a[m]*(x - xd[m])**3 + b[m]*(x-xd[m])**2 + c[m]*(x-xd[m]) + d[m]
return(s)
#solution 1 end
I want to find the minimum on the domain of my xdata, so a fmin didn't work as you can not define bounds there. I tried both fmin_slsqp and minimize. They are not compatible with the phi function I wrote so I rewrote phi(x, xd,yd) and added an extra variable such that phi is phi(x, xd,yd, m). M indicates in which subfunction of the spline we are calculating a solution (from x_m to x_m+1). In the code we replaced #solution 1 by the following
# solution 2 start
return(a[m]*(x - xd[m])**3 + b[m]*(x-xd[m])**2 + c[m]*(x-xd[m]) + d[m])
# solution 2 end
To find the minimum in a domain x_m to x_(m+1) we use the following code: (we use an instance where m=0, so x from 0.25 to 0.5. The initial guess is 0.3)
fmin_slsqp(phi, x0 = 0.3, bounds=([(0.25,0.5)]), args=(xdata, ydata, 0))
What I would then do (I know it's crude), is iterate this with a for loop to find the minimum on all subdomains and then take the overall minimum. However, the function fmin_slsqp constantly returns the initial guess as the minimum. So there is something wrong, which I do not know how to fix. If you could help me this would be greatly appreciated. Thanks for reading this far.
When I plot your function phi and the data you feed in, I see that its range is of the order of 1e-12. However, fmin_slsqp is unable to handle that level of precision and fails to find any change in your objective.
The solution I propose is scaling the return of your objective by the same order of precision like so:
return(s*1e12)
Then you get good results.
>>> sol = fmin_slsqp(phi, x0=0.3, bounds=([(0.25, 0.5)]), args=(xdata, ydata))
>>> print(sol)
Optimization terminated successfully. (Exit mode 0)
Current function value: 1.0
Iterations: 2
Function evaluations: 6
Gradient evaluations: 2
[ 0.25]