I know how to solve A.X = B by least squares using Python:
Example:
A=[[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,0,0]]
B=[1,1,1,1,1]
X=numpy.linalg.lstsq(A, B)
print X[0]
# [ 5.00000000e-01 5.00000000e-01 -1.66533454e-16 -1.11022302e-16]
But what about solving this same equation with a weight matrix not being Identity:
A.X = B (W)
Example:
A=[[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,0,0]]
B=[1,1,1,1,1]
W=[1,2,3,4,5]
I found another approach (using W as a diagonal matrix, and matricial products) :
A=[[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,0,0]]
B = [1,1,1,1,1]
W = [1,2,3,4,5]
W = np.sqrt(np.diag(W))
Aw = np.dot(W,A)
Bw = np.dot(B,W)
X = np.linalg.lstsq(Aw, Bw)
Same values and same results.
I don't know how you have defined your weights, but you could try this if appropriate:
import numpy as np
A=np.array([[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,1,1],[1,1,0,0]])
B = np.array([1,1,1,1,1])
W = np.array([1,2,3,4,5])
Aw = A * np.sqrt(W[:,np.newaxis])
Bw = B * np.sqrt(W)
X = np.linalg.lstsq(Aw, Bw)
scikit package offers weighted regression directly ..
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression.fit
import numpy as np
# generate random data
N = 25
xp = [-5.0, 5.0]
x = np.random.uniform(xp[0],xp[1],(N,1))
e = 2*np.random.randn(N,1)
y = 2*x+e
w = np.ones(N)
# make the 3rd one outlier
y[2] += 30.0
w[2] = 0.0
from sklearn.linear_model import LinearRegression
# fit WLS using sample_weights
WLS = LinearRegression()
WLS.fit(x, y, sample_weight=w)
from matplotlib import pyplot as plt
plt.plot(x,y, '.')
plt.plot(xp, xp*WLS.coef_[0])
plt.show()
Related
I am trying to solve a Weighted Least Squares problem on Python (using numpy) but I am unsure on how to apply the following weight
on
This is what i have done so far
import numpy as np
import matplotlib.pyplot as plt
X = np.random.rand(50) #Generate X values
Y = 2 + 3*X + np.random.rand(50) #Y Values
plt.plot(X,Y,'o')
plt.xlabel('X')
plt.ylabel('Y')
W = ??
X_b = np.c_[np.ones((50,1)), X] #generate [1,x]
beta = np.linalg.inv(X_b.T.dot(W).dot(X_b)).dot(X_b.T).dot(W).dot(Y)
You can use a diagonal matrix for W.
import numpy as np
sigma = # array of [nx1] sigmas
W = np.zeros((len(sigma), len(sigma)) # nxn matrix containing zeros
np.fill_diagonal(W, sigma)
https://numpy.org/doc/stable/reference/generated/numpy.fill_diagonal.html
I have a vector and the Eigenvectors of the matrix H, id like to find the dot product on V1 with every vector in the array.
I want to multiply the vector PhiZero with every eigenvector that's calculated for H
import numpy as np
import scipy.linalg as la
import math
import networkx as nx
Alpha = []
n=3
p=0.5
G = nx.gnp_random_graph(n,p)
A = nx.to_numpy_matrix(G)
w = np.zeros(shape=(n,n))
w[1,2] = 1
gamma = 1/(n*p)
H = (-w) - (gamma * A)
ive chosen a random position in w,
evals, evecs = la.eig(H)
PhiZero = np.reciprocal(math.sqrt(n)) * np.ones((n,1), dtype=int)
i've tried to calculate it two ways, the first way, I get a 3x3 matrix
Alpha = np.dot(PhiZero.transpose(), evecs)
the other way, I tried it with a for loop:
for y in evecs:
alphaJ = np.dot(PhiZero.transpose(), evecs)
Alpha.append(alphaJ)
i've taken the transpose PhiZero to make the dimensions align with evecs (1x3 & 3x1)
In your second approach, should you not have:
alphaJ = np.dot(PhiZero.transpose(), y)
rather than
alphaJ = np.dot(PhiZero.transpose(), evecs)
?
If I try your first example, it works:
import numpy as np
import scipy.linalg as la
import math
import networkx as nx
Alpha = []
n=3
p=0.5
G = nx.gnp_random_graph(n,p)
A = nx.to_numpy_matrix(G)
w = np.zeros(shape=(n,n))
w[1,2] = 1
gamma = 1/(n*p)
H = (-w) - (gamma * A)
evals, evecs = la.eig(H)
PhiZero = np.reciprocal(math.sqrt(n)) * np.ones((n,1), dtype=int)
Alpha = np.dot(PhiZero.transpose(), evecs)
print(Alpha)
gives
[[ 0.95293215 -0.32163376 0.03179978]]
Are you sure you get a 3x3 if you run this?
I have been trying to implement my own Linear Regression from scratch using python but have been facing a issue during the last days.
This is the code I am using :
Import modules
import pandas as pd
import numpy as np
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt
Initialize parameters
def initialize_parameters(n):
w = np.zeros(n,)
b = 0.0
return w,b
Predictor/Hypothesis
def predictor(x, w, b):
return np.dot(x,w) + b
Cost function
def calculate_cost(X, y, theta, b):
m = len(y)
predictions = np.dot(X, theta)
error = predictions - y
cost = (1/2*m) * np.sum(np.power(error,2))
return cost
Gradient descent
def gradient_descent(X, W, b, y, learning_rate = 0.0001, epochs = 25):
m = len(y)
final_cost = 0
for _ in range(epochs):
predictions = predictor(X, W, b)
error = predictions - y
derivate = np.dot(error, X)
print(derivate)
W = W - (learning_rate/m) * derivate
b = b - (learning_rate/m) * error.sum()
Test run :
# Load dataset
boston = load_boston()
data = pd.DataFrame(boston.data)
data.columns = boston.feature_names
data['PRICE'] = boston.target
# Split dataset
X = data.drop(columns=['PRICE']).values
Y = data['PRICE'].values
w, b = initialize_parameters(X.shape[1])
gradient_descent(X, w, b, Y)
During the test run, I can see that the values for the derivate is growing insanely fast :
[1.41239553e+06 3.20162679e+06 3.84829686e+06 2.17737688e+04
1.81667467e+05 1.99565485e+06 2.27660208e+07 1.15045731e+06
3.50107975e+06 1.40396525e+08 5.96494458e+06 1.14447329e+08
4.25947931e+06]
[-4.33362969e+07 -9.66008831e+07 -1.16941872e+08 -6.62733008e+05
-5.50761913e+06 -6.04452389e+07 -6.90425672e+08 -3.46792848e+07
-1.06967561e+08 -4.26847914e+09 -1.80579130e+08 -3.45024565e+09
-1.29016170e+08]
...
[-2.01209195e+34 -4.47742185e+34 -5.42629282e+34 -3.07294644e+32
-2.55503032e+33 -2.80363423e+34 -3.20314565e+35 -1.60824109e+34
-4.96433806e+34 -1.98052568e+36 -8.37673498e+34 -1.60024763e+36
-5.98654489e+34]
[6.09700758e+35 1.35674093e+36 1.64426623e+36 9.31159124e+33
7.74221040e+34 8.49552585e+35 9.70611871e+36 4.87326542e+35
1.50428547e+36 6.00135600e+37 2.53830431e+36 4.84904376e+37
1.81403288e+36]
[-1.84750510e+37 -4.11117381e+37 -4.98242821e+37 -2.82158290e+35
-2.34603173e+36 -2.57430013e+37 -2.94113196e+38 -1.47668879e+37
-4.55826082e+37 -1.81852092e+39 -7.69152754e+37 -1.46934918e+39
-5.49685229e+37]
[5.59827926e+38 1.24576106e+39 1.50976712e+39 8.54991361e+36
7.10890636e+37 7.80060146e+38 8.91216919e+39 4.47463782e+38
1.38123662e+39 5.51045187e+40 2.33067389e+39 4.45239747e+40
1.66564705e+39]
[-1.69638128e+40 -3.77488445e+40 -4.57487122e+40 -2.59078061e+38
-2.15412899e+39 -2.36372529e+40 -2.70055070e+41 -1.35589732e+40
-4.18540025e+40 -1.66976797e+42 -7.06236930e+40 -1.34915808e+42
-5.04721600e+40]
And then, the gradient descent run stops before all interactions due to the high values.
At a certain point, the values form the derivate assume values as NaN.
As expected, when I try to predict a test case, I get 0.0 as output:
sample_house = [[2.29690000e-01, 0.00000000e+00, 1.05900000e+01, 0.00000000e+00, 4.89000000e-01,
6.32600000e+00, 5.25000000e+01, 4.35490000e+00, 4.00000000e+00, 2.77000000e+02,
1.86000000e+01, 3.94870000e+02, 1.09700000e+01]]
test_predict = predictor(sample_house, w, b)
test_predict
------------------------------------------------
out : array([0.])
Thanks!
Your cost function is wrong, it should be:
cost = 1/(2*m) * np.sum(np.power(error,2))
Also, try to initialize your weights as random values between 0 an 1 and scale your inputs to range 0-1.
I had the same issue which I resolved by normalizing the x values.
I think that you are making a mistake in the gradient descent algorithm. When updating the values for "W" vector it should be:
W = W - (learning_rate/m) * derivate.sum()
The learning rate is too large.
I try learning_rate = 0.000001, and it converges normally.
I am using Scipy's odrpack to fit a linear function to some data that has uncertainties in both the x and y dimensions. Each data point has it's own uncertainty that is asymmetric.
I can fit a function using symmetric uncertainties, but this is not a true representation of my data.
How can I perform the fit with this in mind?
This is my code so far. It receives input data as a command line argument, and the uncertainties i'm using are just random numbers at the moment. (also, two fits are happening, one for positive data points another for the negative. The reasons are unrelated to this question)
import sys
import numpy as np
import scipy.odr.odrpack as odrpack
def f(B, x):
return B[0]*x + B[1]
xdata = sys.argv[1].split(',')
xdata = [float(i) for i in xdata]
xdata = np.array(xdata)
#find indices of +/- data
zero_ind = np.where(xdata >= 0)[0][0]
x_p = xdata[zero_ind:]
x_m = xdata[:zero_ind+1]
ydata = sys.argv[2].split(',')
ydata = [float(i) for i in ydata]
ydata = np.array(ydata)
y_p = ydata[zero_ind:]
y_m = ydata[:zero_ind+1]
sx_m = np.random.random(len(x_m))
sx_p = np.random.random(len(x_p))
sy_m = np.random.random(len(y_m))
sy_p = np.random.random(len(y_p))
linear = odrpack.Model(f)
data_p = odrpack.RealData(x_p, y_p, sx=sx_p, sy=sy_p)
odr_p = odrpack.ODR(data_p, linear, beta0=[1.,2.])
out_p = odr_p.run()
data_m = odrpack.RealData(x_m, y_m, sx=sx_m, sy=sy_m)
odr_m = odrpack.ODR(data_m, linear, beta0=[1.,2.])
out_m = odr_m.run()
Thanks!
I will just give you solution with random data,I could not bother to import your data
import numpy as np
import scipy.odr.odrpack as odrpack
np.random.seed(1)
N = 10
x = np.linspace(0,5,N)*(-1)
y = 2*x - 1 + np.random.random(N)
sx = np.random.random(N)
sy = np.random.random(N)
def f(B, x):
return B[0]*x + B[1]
linear = odrpack.Model(f)
# mydata = odrpack.Data(x, y, wd=1./np.power(sx,2), we=1./np.power(sy,2))
mydata = odrpack.RealData(x, y, sx=sx, sy=sy)
myodr = odrpack.ODR(mydata, linear, beta0=[1., 2.])
myoutput = myodr.run()
myoutput.pprint()
Than we got
Beta: [ 1.92743947 -0.94409236]
Beta Std Error: [ 0.03117086 0.11273067]
Beta Covariance: [[ 0.02047196 0.06690713]
[ 0.06690713 0.26776027]]
Residual Variance: 0.04746112419196648
Inverse Condition #: 0.10277763521624257
Reason(s) for Halting:
Sum of squares convergence
i'm trying to write function called plotting which takes i/p parameters Z, p and q and plots the function
f(y) = det(Z − yI) on the interval [p, q]
(Note: I is the identity matrix.) det() is the determinant.
For finding det(), numpy.linalg.det() can be used
and for indentity matrix , np.matlib.identity(n)
Is there a way to write such functions in python? and plot them?
import numpy as np
def f(y):
I2 = np.matlib.identity(y)
x = Z-yI2
numpy.linalg.det(x)
....
Is what i am tryin correct? any alternative?
You could use the following implementation.
import numpy as np
import matplotlib.pyplot as plt
def f(y, Z):
n, m = Z.shape
assert(n==m)
I = np.identity(n)
x = Z-y*I
return np.linalg.det(x)
Z = np.matrix('1 2; 3 4')
p = -15
q = 15
y = np.linspace(p, q)
w = np.zeros(y.shape)
for i in range(len(y)):
w[i] = f(y[i], Z)
plt.plot(y, w)
plt.show()