We Keep Coding

Why does np.linalg.lstsq not show me the residues? I checked the Paramater sizes but the second array is empty

a_test = (Phi_train_M.T.dot(Phi_train_M) +lambda_reg*IdendityMatrix)
b_test = Phi_train_M.T.dot(data_train[:,1])
error_reg_test = np.linalg.lstsq (a_test,b_test,rcond=None)[1]
I want to check for the regularized sum of squared errors/residues. a_test dimensions are (16,16) and b_test dimensions are (16,) the doc says
Blockquote
residuals{(1,), (K,), (0,)} ndarray
Sums of squared residuals: Squared Euclidean 2-norm for each column in b - a # x. If the rank of a is < N or M <= N, this is an empty array. If b is 1-dimensional, this is a (1,) shape array. Otherwise the shape is (K,).
and as i typed it i noticed this doesnt work if 'a' is an N x N matrix. Does someone know a workaround? or how to solve this?

Compare elements of 2D array to the product of two 1D arrays?

I'm trying to compare a 2D array to the product of two 1D arrays (joint-probability density v.s. product of its individual probability densities) in order to determine if variables x and y are independent, where independence is given as ρ(x,y)=ρ(x)*ρ(y).
Let's say I called the 2D array h, and the 1D lists n and m. How would I go about iterating over h to check if it's elements are equivalent to n*m?

To test for exact equality, just use np.all()
import numpy as np
m = np.random.rand(10)
n = np.random.rand(20)
h = m.reshape(1, -1) * n.reshape(-1, 1)
print(np.all(h == m.reshape(1, -1) * n.reshape(-1, 1))) # True
To test whether the numbers are all close, you could use:
print(np.all(np.isclose(h, m.reshape(1, -1) * n.reshape(-1, 1))))

Two different cost in Logistic Regression cost function

I am implementing logistic Regression algorithm with two feature x1 and x2. I am writing the code of cost function in logistic regression.
def computeCost(X,y,theta):
J =((np.sum(-y*np.log(sigmoid(np.dot(X,theta)))-(1-y)*(np.log(1-sigmoid(np.dot(X,theta))))))/m)
return J
Here My X is the training set matrix, y is the output.
the shape of X is (100,3) and shape of y is (100,) as determined by shape attribute of numpy library. my theta is initially contained all zero entry with shape (3,1). When I calculate cost with these parameters I got the cost 69.314. But it is incorrect. The correct Cost is 0.69314. Actually, I got this correct cost when I reshape my y vector as y = numpy.reshape(y,(-1,1)) .
But I actually didn't get how this reshaping corrects my cost.
Here m(numbers of the training set) is 100.

First of all never in future simply dump your code! You post(code + explanation) should be as descriptive as it could be! (not verbose, nobody will read it). Here is what your code is doing! Please post readable code in future! Else it's hard to read & answer!
def computeCost(X,y,theta):
'''
Using Mean Absolute Error
X:(100,3)
y: (100,1)
theta:(3,1)
Returns 1D matrix of predictions
Cost = ( log(predictions) + (1-labels)*log(1-predictions) ) / len(labels)
'''
m = len(y)
# calculate the prediction
predictions = sigmoid(np.dot(X,theta))
# error for when label is of class1
class1_cost= -y * np.log(predictions)
# error for when label is of class1
class2_cost= (1-y)*np.log(1-predictions)
# total cost
cost = class1_cost-class2_cost
# averaging cost
cost =cost.sum() / m
return cost
You should first understand How dot product works in math and what shape of input you algorithm would take to give you the correct answer! Don't throw random shapes! Your feature_vector is of shape(100,3) which when multiplied by your theta which of shape(3,1) will output a prediction vector of shape (100,1).
Matrix multiplication: The product of an M x N matrix and an N x K matrix is an M x K matrix. The new matrix takes the rows of the 1st and columns of the 2nd
So, your y dimension should be in (100,1) shape and not (100,). Huge difference! One is [[3],[4],[6],[7],[9],...] and another [3,4,6,7,9,.....].
Your dimension should match for correct output!
A better way of asking the question would be, how to calculate error/cost in logistic regression using the correct dimensions of my labels.!
For additional understanding!
import numpy as np
label_type1= np.random.rand(100,1)
label_type2= np.random.rand(100,)
predictions= np.random.rand(100,1)
print(label_type1.shape, label_type2.shape, predictions.shape)
# When you mutiply (100,1) with (100,1) --> (100,1)
print((label_type1 * predictions).shape)
# When you do a dot product (100,1) with (100,1) --> Error, for which you have to take a transpose which isn't relavant to the context!
# print( np.dot(label_type1,predictions).shape) # error: shapes (100,1) and (100,1) not aligned: 1 (dim 1) != 100 (dim 0)
print( np.dot(label_type1.T,predictions).shape) #
print('*'*5)
# When you mutiply (100,) with (100,1) --> (100,100) !
print((label_type2 * predictions).shape) #
# When you do a dot product (100,) with (100,1) --> (1,) !
print(np.dot(label_type2, predictions).shape)
print('*'*5)
# what you are doin
label_type1_addDim = np.reshape(label_type2,(-1,1))
print(label_type1_transpose.shape)
So, coming straight to the point, What you wanna achieve is a cost with dim (100,1)! so either you do 1st which you aren't doing! or you do the fifth, where you unknowingly adding a dimension to your y
making it from (100,) to (100,1) and doing the same * operation as of 1st case! to get dim (100,1).

Pairwise Euclidean distances between two binary tensors

I am trying to compute the pairwise distances between all points in two binary areas/volume/hypervolume in Tensorflow.
E.g. In 2D the areas are defined as binary tensors with ones and zeros:
input1 = tf.constant(np.array([[1,0,0], [0,1,0], [0,0,1]))
input2 = tf.constant(np.array([[0,1,0], [0,0,1], [0,1,0]))
input1 has 3 points and input2 has 2 points.
So far I have managed to convert the binary tensors into arrays of spatial coordinates:
coord1 = tf.where(tf.cast(input1, tf.bool))
coord2 = tf.where(tf.cast(input2, tf.bool))
Where, coord1 will have shape=(3,2) and coord2 will have shape=(2,2). The first dimension refers to the number of points and the second to their spatial coordinates (in this case 2D).
The result that I want is a tensor with shape=(6, ) with the pairwise Euclidean distances between all of the points in the areas.
Example (the order of the distances might be incorrect):
output = [1, sqrt(5), 1, 1, sqrt(5), 1]
Since TensorFlow isn't great with loops and in my real application the number of points in each tensor is unknown, I think I might be missing some linear algebra here.

I'm not familiar with Tensorflow, but my understanding from reading this is that the underlying NumPy arrays should be easy to extract from your data. So I will provide a solution which shows how to calculate pairwise Euclidean distances between points of 3x2 and 2x2 NumPy arrays, and hopefully it helps.
Generating random NumPy arrays in the same shape as your data:
coord1 = np.random.random((3, 2))
coord2 = np.random.random((2, 2))
Import the relevant SciPy function and run:
from scipy.spatial.distance import cdist
distances = cdist(coord1, coord2, metric='euclidean')
This will return a 3x2 array, but you can use distances.flatten() to get your desired 1-dimensional array of length 6.

I have come up with an answer using only matrix multiplies and transposition. This makes use of the fact that distances can be expressed with inner products (d^2 = x^2 + y^2 - 2xy):
input1 = np.array([[1,0,0],[0,1,0],[0,0,1]])
input2 = np.array([[1,1,0],[0,0,1],[1,0,0]])
c1 = tf.cast(tf.where(tf.cast(input1, tf.bool)), tf.float32)
c2 = tf.cast(tf.where(tf.cast(input2, tf.bool)), tf.float32)
distances = tf.sqrt(-2 * tf.matmul(c1, tf.transpose(c2)) + tf.reduce_sum(tf.square(c2), axis=1)
+ tf.expand_dims(tf.reduce_sum(tf.square(c1), axis=1), axis=1))
with tf.Session() as sess:
d = sess.run(distances)
Since Tensorflow has broadcast by default the fact the arrays have different dimensions doesn't matter.
Hope it helps somebody.

compute matrix product for multiple inputs

I am trying to compute a transform given by b = A*x. A is a (3,4) matrix. If x is one (4,1) vector the result is b (3,1).
Instead, for x I have a bunch of vectors concatenated into a matrix and I am trying to evaluate the transform for each value of x. So x is (20, 4). How do I broadcast this in numpy such that I get 20 resulting values for b (20,3)?
I could loop over each input and compute the output but it feels like there must be a better way using broadcasting.
Eg.
A = [[1,0,0,0],
[2,0,0,0],
[3,0,0,0]]
if x is:
x = [[1,1,1,1],
[2,2,2,2]]
b = [[1,2,3],
[2,4,6]]
Each row of x is multiplied with A and result is stored as a row in b.

numpy dot
import numpy as np
A = np.random.normal(size=(3,4))
x = np.random.normal(size=(4,20))
y = np.dot(A,x)
print y.shape
Result: (3, 20)
And of course if you want (20,3) you can use np.transpose()