I have a strange bug in this naive code for Gram-schmidt orthogonalization I wrote for numpy (v1.21.2).
def gram_schmidt(basis):
"""
Compute orthogonal basis from given basis.
Parameters:
basis <- an mxn array-like of vectors in m-space
Returns:
orthogonalized basis as array of shape (m,n)
"""
m, n = basis.shape
orthobasis = np.zeros_like(basis)
quadrances = np.ones(n)
for i, b in enumerate(basis.T):
# initial iteration
if i == 0:
orthobasis[:, i] = b
quadrances[i] = b.T # b
continue
# subsequent iterations
# get the orthogonal complement vector as the next basis vector in the set
# 1: project onto subspace generated by basis accumulated so far
P = orthobasis[:, :i]
proj_b = P # ((P.T # b) / quadrances[:i])
# 2: get the orthogonal vector to the projection
e = b - proj_b
# 3: add this orthogonal vector to the basis being constructed
orthobasis[:, i] = e
# 4: add the quadrance of the new vector
quadrances[i] = e.T # e
print(f"**iteration {i}**\nb = {b}\nP = {P}\nproj_b = {proj_b}\ne = {e}\northobasis =\n{orthobasis}")
print("-" * 100)
return orthobasis
When I run this code with the following input:
basis = np.stack([ np.array(v) for v in
[[1,1,1,1],
[0,1,1,1],
[0,0,1,1]] ],
axis=1)
gram_schmidt(basis)
I get the output:
**iteration 1**
b = [0 1 1 1]
P = [[1]
[1]
[1]
[1]]
proj_b = [0.75 0.75 0.75 0.75]
e = [-0.75 0.25 0.25 0.25]
orthobasis =
[[1 0 0]
[1 0 0]
[1 0 0]
[1 0 0]]
----------------------------------------------------------------------------------------------------
**iteration 2**
b = [0 0 1 1]
P = [[1 0]
[1 0]
[1 0]
[1 0]]
proj_b = [0.5 0.5 0.5 0.5]
e = [-0.5 -0.5 0.5 0.5]
orthobasis =
[[1 0 0]
[1 0 0]
[1 0 0]
[1 0 0]]
So the unexpected behavior is that step 3 in the gram_schmidt function does not assign the computed array e to the i-th column of orthobasis. You can ignore the correctness of gram_schmidt as the problem is related to array assignment: the first iteration assigns successfully to the first column, yet the subsequent iterations silently fail. What is even more odd is that if I change e in step 3 to a hard-coded array (of conforming shape with columns of orthobasis), it assigns successfully! In the output you can see that starting from the second iteration (iteration 1), there is no change to the columns of orthobasis; they are always 0!
I am so confused and have lost hours debugging this. I appreciate any assistance!
Each numpy array has a specified data type and can contain elements only of this type. In your example basis is an array of integers. Then you are initializing orthobasis using
orthobasis = np.zeros_like(basis)
This makes the data type of orthobasis the same as basis, i.e. it is an array of integers too. When you try to modify entries of this array by assigning to them float values, these floats are first converted to integers by rounding them toward 0. Thus the array [-0.5 -0.5 0.5 0.5] becomes an array of zeros.
To fix it, you can initialize orthobasis as an array of floats:
orthobasis = np.zeros_like(basis, dtype=float)
Related
Given a matrix S and a binary matrix W, I want to create a submatrix of S corresponding to the non zero coordinates of W.
For example:
S = [[1,1],[1,2],[1,3],[1,4],[1,5]]
W = [[1,0,0],[1,1,0],[1,1,1],[0,1,1],[0,0,1]]
I want to get matrices
S_1 = [[1,1],[1,2],[1,3]]
S_2 = [[1,2],[1,3],[1,4]]
S_3 = [[1,3],[1,4],[1,5]]
I couldn't figure out a slick way to do this in python. The best I could do for each S_i is
S_1 = S[0,:]
for i in range(np.shape(W)[0]):
if W[i, 0] == 1:
S_1 = np.vstack((S_1, S[i, :]))
but if i want to change the dimensions of the problem and have, say, 100 S_i's, writing a for loop for each one seems a bit ugly. (Side note: S_1 should be initialized to some empty 2d array but I couldn't get that to work, so initialized it to S[0,:] as a placeholder).
EDIT: To clarify what I mean:
I have a matrix S
1 1
1 2
1 3
1 4
1 5
and I have a binary matrix
1 0 0
1 1 0
1 1 1
0 1 1
0 0 1
Given the first column of the binary matrix W
1
1
1
0
0
The 1's are in the first, second, and third positions. So I want to create a corresponding submatrix of S with just the first, second and third positions of every column, so S_1 (corresponding to the 1st column of W) is
1 1
1 2
1 3
Similarly, if we look at the third column of W
0
0
1
1
1
The 1's are in the last three coordinates and so I want a submatrix of S with just the last three coordinates of every column, called S_3
1 3
1 4
1 5
So given any ith column of the binary matrix, I'm looking to generate a submatrix S_i where the columns of S_i contain the columns of S, but only the entries corresponding to the positions of the 1's in the ith column of the binary matrix.
It probably is more useful to work with the transpose of W rather than W itself, both for human-readability and to facilitate writing the code. This means that the entries that affect each S_i are grouped together in one of the inner parentheses of W, i.e. in a row of W rather than a column as you have it now.
Then, S_i = np.array[S[j,:] for j in np.shape(S)[0] if W_T[i,j] == 1], where W_T is the transpose of W. If you need/want to stick with W as is, you need to reverse the indices i and j.
As for the outer loop, you could try to nest this in another similar comprehension without an if statement--however this might be awkward since you aren't actually building one output matrix (the S_i can easily be different dimensions, unless you're somehow guaranteed to have the same number of 1s in every column of W). This in fact raises the question of what you want--a list of these arrays S_i? Otherwise if they are separate variables as you have it written, there's no good way to refer to them in a generalizable way as they don't have indices.
Numpy can do this directly.
import numpy as np
S = np.array([[1,1],[1,2],[1,3],[1,4],[1,5]])
W = np.array([[1,0,0],[1,1,0],[1,1,1],[0,1,1],[0,0,1]])
for row in range(W.shape[1]):
print(S[W[:,row]==1])
Output:
[[1 1]
[1 2]
[1 3]]
[[1 2]
[1 3]
[1 4]]
[[1 3]
[1 4]
[1 5]]
This is part of an algorithm to generate concentric rings of points on a hexagonal lattice that I'm in the process of rewriting.
I'd thought it's all integer math, but I discovered that in some cases arrays are unexpectedly created as floats!
In the sequence below p0 is a float64 for n=1 but int64 for n>1 and I simply can not figure out why this is happening.
I'm running numpy version 1.17.3, Anaconda installation of Python 3.7.3 on MacOS
import numpy as np
n_max = 3
for n in range(1, n_max+1):
seq = np.arange(n, -n-1, -1, dtype=int)
p0 = np.hstack((seq, (n-1)*[-n], seq[::-1], (n-1)*[n]))
print('n: ', n)
print('seq: ', seq)
print('p0: ', p0.dtype, p0)
print('')
returns
n: 1
seq: [ 1 0 -1]
p0: float64 [ 1. 0. -1. -1. 0. 1.]
n: 2
seq: [ 2 1 0 -1 -2]
p0: int64 [ 2 1 0 -1 -2 -2 -2 -1 0 1 2 2]
n: 3
seq: [ 3 2 1 0 -1 -2 -3]
p0: int64 [ 3 2 1 0 -1 -2 -3 -3 -3 -3 -2 -1 0 1 2 3 3 3]
Is that expected behavior?
update 1: okay np.hstack(([1, 0, -1], 1*[7])) returns int64 but np.hstack(([1, 0, -1], 0*[7])) returns float64 so it's the occurrence of the 0*[n] in the tuple on which np.hstack operates that's triggering the upcast to float64.
update 2: Just asked in Code Review: Is there a better, cleaner or otherwise “less tricky” way to get these hexagonal arrays of dots arranged in this spiral pattern?
What's triggering the entire array to be cast to np.float64, is the empty list obtained when n=0 with (n-1)*[n] and (n-1)*[-n]:
print((n-1)*[n])
# []
np.hstack constructs an array from each of its input arrays to be concatenated. For each array there's a call to np.atleast_1d, which by default casts empty arrays to np.float64 dtype:
np.atleast_1d([])
# array([], dtype=float64)
The reason for this is that NumPy creates ndarrays from all inputs before concatenating them.
[0]*n evaluates to [], which is an empty list and thus has no numeric type, therefore when being cast to an array it becomes an empty array with the default data type, which is to use a float.
You can avoid this by using casting the input to ndarrays yourself and specifying the datatype as int, for example like so:
import numpy as np
n_max = 3
for n in range(1, n_max+1):
seq = np.arange(n, -n-1, -1, dtype=int)
p0 = np.hstack((seq, np.array((n-1)*[-n], dtype=np.int32), seq[::-1], np.array((n-1)*[-n], dtype=np.int32)))
print('n: ', n)
print('seq: ', seq)
print('p0: ', p0.dtype, p0)
print('')
I can't really speak to whether it is expected behavior or not, but it does make some sense intrinsically.
I am given two numpy-arrays: One of dimensions i x mand the other of dimensions j x m. What I want to do is, loop through the FirstArray and compare each of its elements with each of the elements of the SecondArray. When I say 'compare', I mean: I want to compute the Euclidean distance between the elements of FirstArray and SecondArray. Then, I want to store the index of the element of SecondArray that is closest to the corresponding element of FirstArray, and I also want to store the index of the element of SecondArray that is second closest to the element of the FirstArray.
In code this would look somewhat similar to this:
smallest = None
idx = 0
for i in range(0, FirstArrayRows):
for j in range(0, SecondArrayRows):
EuclideanDistance = np.sqrt(np.sum(np.square(FirstArray[i,:] - SecondArray[j,:])))
if smallest is None or EuclideanDistance < smallest:
smallest = EuclideanDistance
idx_second = idx
idx = j
Closest[i] = idx
SecondClosest[i] = idx_second
And I think this works. However, there are two cases when this code fails to give the correct index for the second closest element of SecondArray:
when the element of SecondArray that is closest to the element of FirstArray is at j = 0.
when the element of SecondArray that is closest to the element of FirstArray is at j = 1.
So I wonder: Is there a better way of implementing this?
I know there is. Maybe someone can help me see it?
You could use numpy's broadcasting to your advantage. Compute the Euclidean distance with all elements of the second array in a single operation. Then, you can find the two smallest distances using argpartition.
import numpy as np
i, j, m = 3, 4, 5
a = np.random.choice(10,(i,m))
b = np.random.choice(10,(j,m))
print('First array:\n',a)
print('Second array:\n',b)
closest, second_closest = np.zeros(i), np.zeros(i)
for i in range(a.shape[0]):
dist = np.sqrt(((a[i,:] - b)**2).sum(axis=1))
closest[i], second_closest[i] = np.argpartition(dist, 2)[:2]
print('Closest:', closest)
print('Second Closest:', second_closest)
Output:
First array:
[[3 9 0 2 2]
[1 2 9 9 7]
[4 0 6 6 4]]
Second array:
[[9 9 2 2 3]
[9 9 0 2 3]
[1 1 6 7 7]
[5 7 0 4 4]]
Closest: [3. 2. 2.]
Second Closest: [1. 3. 3.]
Assume I have a matrix which is N items long x M columns long (where M<=N). I want to know the average rank for each of the N across the M columns.
arr = np.array([
[0,1],
[2,0],
[1,2]
])
I could loop through each of the N values and do something like the following, but I'm wondering whether there's a better approach to this
for n in range(3):
np.where(arr==n)[0].mean()
Edit
Sorry, it seems my choice of example has caused some confusion. To better illustrate, let me swap in letters since the values in the matrix are identifiers, not numbers to be calculated on.
arr = np.array([
['A','B'],
['C','A'],
['B','C']
])
I am not trying to do a simple row-wise average. I'm trying to say that
A average rank is 0.5 (0 + 1) / 2
B average rank is 1.0 (0 + 2) / 2
C average rank is 1.5 (1 + 2) / 2
Hopefully this clarified my request
It looks like you want to get the mean of your array along a certain axis. You can do this using the axis= argument of numpy.mean:
import numpy as np
arr = np.array([
[0,1],
[2,0],
[1,2]
])
np.mean(arr, axis=1)
# [ 0.5 1. 1.5]
If you want row wise mean
>>> np.mean(arr, axis=1)
array([ 0.5, 1. , 1.5])
To get rank (as OP's description)
First generate 2D array of indices
import numpy as np
M = 5
N = 7
narray = np.array(np.tile(np.arange(N), M)).reshape(N, M)
print(narray)
Output:
[[0 1 2 3 4]
[5 6 0 1 2]
[3 4 5 6 0]
[1 2 3 4 5]
[6 0 1 2 3]
[4 5 6 0 1]
[2 3 4 5 6]]
Now take row wise mean to get rank
mean_value = np.mean(narray, axis=1)
print(mean_value)
Output
[ 2. 2.8 3.6 3. 2.4 3.2 4. ]
If each of the N items appears exactly 1 time in each column,
(i.e each column is a ranking) , you can simply do :
#arr = np.array([['A','B'],['C','A'],['B','C']])
means = arr.argsort(0).mean(1)
#array([ 0.5, 1. , 1.5])
Here is my attempt to "improve" your original solution. My solution has the benefit of not needing to perform two (possibly very time consuming) operations all over again for each value in the array: np.where(arr==n) (1. find all values equal to n; 2. find indices of elements for which previous equality is true).
values, inverse, counts = np.unique(arr, return_inverse=True, return_counts=True)
rows = np.argsort(inverse) // len(arr[0])
cumsum = np.cumsum(counts)
avranks = np.add.reduceat(rows, cumsum - cumsum[0]) / counts
Then, for your original data,
>>> print(avranks)
[0.5 1. 1.5]
I wrote the following code that does: (1) Generate root matrix of shape (3, 16), and (2) Generate 1000 binary vectors C such that each vector is added at the end of the root matrix iteratively one at a time.
The code is the following (python 2.7):
# STEP1: Generate root matrix
root = [random.randint(0, 2 ** 16 - 1) for _ in range(16 - 1)]
root = [('{0:0' + str(16) + 'b}').format(x) for x in root]
root = np.asarray([list(map(int, list(x))) for x in root], dtype=np.uint8)
# STEP2: Generate 1000 binary vectors
C = [random.randint(0, 2 ** 16 - 1) for _ in range(1000)]
C = list(set(C))
C = [('{0:0' + str(16) + 'b}').format(x) for x in C]
C = np.asarray([list(map(int, list(x))) for x in C], dtype=np.uint8)
# Step3: For each vector of C, append it at the end of root matrix
for i in range(1000):
batch = root[i:(i + (len(root)))]
batch = np.append(batch, C[i])
print(batch)
# Continue to process batch after being extended by adding a new vector
The matrix C looks like this:
[[0 1 1 ..., 0 1 1]
[1 0 1 ..., 1 0 1]
[0 1 0 ..., 1 1 0]
...,
[1 1 0 ..., 1 0 0]
[0 0 1 ..., 1 1 0]
[1 1 1 ..., 1 1 1]]
The problem is that np.append(batch, C[i])merges all vectors into a single one but this is not what I want. My goal is to extend the root matrix from (3, 16) to be (4,16) by just adding a vector C[i] iteratively. How can I achieve that?
Thank you
If you can swap this:
batch = root[i:(i + (len(root)))]
batch = np.append(batch, C[i])
For this:
batch = np.append(batch, [C[i]], axis=0)
axis allows you to append two matrices on a particular dimension. So we make C[i] into a matrix and append it in dimension 0.
I am not sure what the intention of this batch = root[i:(i + (len(root)))] is but it shortens batch to a matrix the size of root every time so it will not increase in size. It actually shrinks as you near the end of root.
Also C is not always 1000 vectors. Making them unique removes some.