We Keep Coding

Subtract means collected from tf.math.unsorted_segment_mean from a tensor in Tensorflow

So I have a 2D tf.float32 Tensor of xyz coords and a 1D tf.int32 Tensor of segment_ids.
I want to subtract every point from the mean of the corresponding segment_id.
Please check the code below:
x_index = tf.constant([1, 1, 2, 2])
y_index = tf.constant([1, 1, 3, 4])
points = tf.constant([[0.1, 0.1, 0.1],
[0.11, 0.11, 0.11],
[0.2, 0.3, 0.1],
[0.2, 0.4, 0.1]])
points_x_y_indices = tf.transpose(tf.stack([x_index, y_index]))
uniques, idx = tf.raw_ops.UniqueV2(x=points_x_y_indices, axis=[0], out_idx=tf.dtypes.int32)
n_pillars = int(tf.reduce_max(idx))+1
x_means = tf.math.unsorted_segment_mean(points[:, 0], idx, n_pillars)
y_means = tf.math.unsorted_segment_mean(points[:, 1], idx, n_pillars)
z_means = tf.math.unsorted_segment_mean(points[:, 2], idx, n_pillars)
Now, I have the means over every segment_id in x_means, y_means and z_means. How can I subtract those values from original points tensor?? of course without looping as I am trying to avoid tf.py_func
Thanks!

I figured it out, you can use
full_x_means = tf.gather(x_means, idx)
full_y_means = tf.gather(y_means, idx)
full_z_means = tf.gather(z_means, idx)
Then
pillar_points_xc = points[:, 0] - full_x_means
pillar_points_yc = points[:, 1] - full_y_means
pillar_points_zc = points[:, 2] - full_z_means

Mapping z's in numpy array A = [[x0, y0, z0], [x1, y1, z1]] for 3rd column of array B = [[x1, y1, ?], [x0, y0, ?]] based off matching (x,y)?

I have a numpy array T whose rows have the following column structure: [x, y, value], where x, y, value are integers. A sample T array would look like:
[[1, 0, 4],
[0, 2, 3],
[1, 2, 7]]
This data comes from a model where the third column specifies the value of a variable for the tuple (x, y). In the model, this tuple corresponds to a label for the value. For example, my label T_10 (subscript 10) has value 4, T_02 has value 3 and T_12 has value 7.
Now, I want to swap a pair of labels. For example, I want to replace all labels 2 with 1 (and vice versa), to get T_20, T_01 and T_21 for the previous examples respectively. So, this new data is
U = [[2, 0, ?],
[0, 1, ?],
[2, 1, ?]]
My issue is that I do not know how to make my new data look like this:
U = [[2, 0, -3]
[0, 1, -4],
[2, 1, -7]]
This new data should follow two rules:
First, it should correctly identify the row of T whose first and second columns (x, y) are the same as the new (x, y) in U. For each row of U, if the ordered pair (x, y) = (x, y) of T, then the appropriate '?' value in the third column of U should be the corresponding value in T.
Second: If, on the other hand, (x, y) of U = (y, x) of T, then it should be the negative of the corresponding value.
My attempt involved first extracting the columns of T, and then swapping the pair of labels using the following function:
def swap_indices(a, pair):
for n, i in enumerate(a):
if i == pair[0]: # check whether a0's element is = swap element 1
a[n] = pair[1]
elif i == pair[1]:
a[n] = pair[0]
return a
For example, I will swap label 0 with 1 and vice versa for column x and column y using:
pair = (0, 1)
a0 = swap_indices(T[:,0], pair) # column x
a1 = swap_indices(T[:,1], pair) # column y
Then I iterate over the number of rows of T; num_rows_of_T:
for k in range(num_rows_of_T):
temp = np.where((T[k, 0] == a0[k]) & (T[k, 1] == a1[k]) | ((T[k, 0] == a1[k]) & (T[k, 1] == a0[k])))
Above, I am trying to get the index of the row where either (x, y) of U = (x, y) of T, or (x, y) of U = (y, x) of T. However this is where I get stuck. I don't think the above is correct. Also, this approach will not let me apply the second rule where I take the negative of the value of T if (x, y) = (y, x). I also tried using set() for starters (to get an unordered pair), but I cannot correctly find the corresponding value of T even then.
Basically, I want to find the values of T that match with the new labels in U. My data is nice, in that there may only exist one possible set of coordinates, and that there is always a bijective mapping between the (x,y) of T and U (given my two rules).
Any advice?
Please help edit the question as necessary. It was very difficult for me to ask.
Here is a minimal working example:
import numpy as np
# swap index labels if match swap pair
def swap_indices(a, pair):
for n, i in enumerate(a):
if i == pair[0]: # check whether a0's element is = swap element 1
a[n] = pair[1]
elif i == pair[1]:
a[n] = pair[0]
return a
def find_valid_swaps(The1 = np.array([1, 0, -1, 1, 0, 1]), headers = np.array(['10', '20', '21', '30', '31', '32'])):
num_indices = len(The1)
T = np.zeros((num_indices,3)); U = T;
# match format given for T in question
for i in range(num_indices):
T[i,:] = [int(list(headers[i])[0]), int(list(headers[i])[1]), The1[i]]
pair = (0, 1) # label pair to swap
a0 = swap_indices(T[:, 0], pair) # column 0 of U
a1 = swap_indices(T[:, 1], pair) # column 1 of U
# try to extract correct 'value' from T based on new labels in U
for k in range(num_indices):
temp = np.where((T[k, 0] == a0[k]) & (T[k, 1] == a1[k]) | ((T[k, 0] == a1[k]) & (T[k, 1] == a0[k])))
print("temp",temp[0][0])
U[k, :] = [a0[k], a1[k], T[temp[0][0], 2]] # here, I would finally create the new U matrix, applying both rules
print(U)
find_valid_swaps()
More involved example using #MadPhysicist 's answer:
# swap index labels if match swap pair
def swap_indices(a, pair):
for n, i in enumerate(a):
if i == pair[0]: # check whether a0's element is = swap element 1
a[n] = pair[1]
elif i == pair[1]:
a[n] = pair[0]
return a
def key(arr, m):
return arr[:, 0] * m + arr[:, 1]
def find_valid_swaps(Thetas1 = np.array([1, 1, 0, 0, -1, -1]), Thetas2 = np.array([1, 0, -1, 1, 0, 1]), num_bands = 4, headers = np.array(['10', '20', '21', '30', '31', '32'])):
import itertools # for permutations: https://stackoverflow.com/questions/40092474/get-all-pairwise-combinations-from-a-list
if (Thetas1==Thetas2).all():
print("Warning: Input sets of indices are equal to each other. Will check other possible permutations regardless.")
else:
print("Input sets of indices are unique. Will proceed checking other viable permutations.")
num_indices = len(Thetas1)
T = np.zeros((num_indices,3))
U = np.zeros((num_indices,3))
for i in range(num_indices):
T[i,:] = [int(list(headers[i])[0]), int(list(headers[i])[1]), Thetas2[i]]
print("input T")
print(T)
pair = (2,3)
a0 = swap_indices(T[:,0], pair) # column 1
a1 = swap_indices(T[:,1], pair) # column 2
for k in range(num_indices):
U[k, :] = [a0[k], a1[k], 0]
# below code due to #MadPhysicist from https://stackoverflow.com/questions/67223782/mapping-zs-in-numpy-array-a-x0-y0-z0-x1-y1-z1-for-3rd-column-of-ar/67235030?noredirect=1#67235030
y_max = T[:, 1].max() + 1
Tkey = key(T, y_max)
s = np.argsort(Tkey)
Ukey = key(U, y_max)
i = np.searchsorted(Tkey, Ukey, sorter=s)
i[i == len(i)] -= 1 # cleanup indices that won't match anyway
mask = (Ukey == Tkey[s[i]])
U2key = key(U[~mask, 1::-1], y_max)
j = np.searchsorted(Tkey, U2key, sorter=s)
U[mask, -1] = T[s[i[mask]], -1]
U[~mask, -1] = -T[s[j], -1]
print("reordered U")
print(U)
The above gives output:
input T
[[ 1., 0., 1.]
[ 2., 0., 0.]
[ 2., 1., -1.]
[ 3., 0., 1.]
[ 3., 1., 0.]
[ 3., 2., 1.]]
reordered U
[[ 1., 0., 1.]
[ 3., 0., 0.]
[ 3., 1., -1.]
[ 2., 0., 1.]
[ 2., 1., 0.]
[ 2., 3., 1.]]

You can boil your algorithm down to three big steps:
Sort Txy
Binary search for Uxy in Txy
Binary search for remaining Uyx in Txy
Combining the result is clearly pretty trivial. The whole operation should be quite doable in O(N log N) time because that's how long each step takes.
Since np.searchsorted is the prime candidate for steps 2 and 3, let's say that you can turn the first two columns into a unique key. For example, let's say that y <= y_max in all cases, and that y_max has a sane bound such that x * y_max + y <= 2**32-1 for all x. You can play with using np.int64 or using x_max instead of y_max at your leisure.
So now you do:
def key(arr, m):
return arr[:, 0] * m + arr[:, 1]
y_max = T[:, :1].max(None) + 1
Tkey = key(T, y_max)
s = np.argsort(Tkey)
To find which elements of U match:
Ukey = key(U, y_max)
i = np.searchsorted(Tkey, Ukey, sorter=s)
i[i == len(i)] -= 1 # cleanup indices that won't match anyway
mask = (Ukey == Tkey[s[i]])
Now find the reverse index.
U2key = key(U[~mask, 1::-1], y_max)
j = np.searchsorted(Tkey, U2key, sorter=s)
Since the mapping is bijective, this step only searched for elements that are guaranteed to exist, and there's no need to verify the indices.
Now you can combine the indices. If U doesn't already have a third column, add one:
U = np.concatenate((U, np.empty_like(T[:, :1])), axis=1)
Using the indices we computed, extract the elements of Tsort that you want:
U[mask, -1] = T[s[i[mask]], -1]
U[~mask, -1] = -T[s[j], -1]
Now if you can't get a mapping like key working, things can be a bit more complicated. If nothing else works, first try
def key(arr):
return arr[:, 0] + 1j * arr[:, 1]
The complex values will only be used as sort keys and nothing else. If that fails, you may have to define a structured datatype and view your array through that to get the search working. You can of course implement a hierarchical search, but I feel that that's out of scope here.
Here is a complete toy example based off your T, with a slightly modified U that shows both positive and negative numbers in the last column:
>>> T = np.array([[1, 0, 4],
[0, 2, 3],
[1, 2, 7]])
>>> U = np.array([[2, 1, 0],
[1, 0, 0],
[2, 0, 0]])
>>> def key(arr, m):
... return arr[:, 0] * m + arr[:, 1]
>>> y_max = T[:, :1].max(None) + 1
>>> Tkey = key(T, y_max)
>>> s = np.argsort(Tkey)
>>> Ukey = key(U, y_max)
>>> i = np.searchsorted(Tkey, Ukey, sorter=s)
>>> i[i == len(i)] -= 1 # cleanup indices that won't match anyway
>>> mask = (Ukey == Tkey[s[i]])
>>> U2key = key(U[~mask, 1::-1], y_max)
>>> j = np.searchsorted(Tkey, U2key, sorter=s)
>>> U[mask, -1] = T[s[i[mask]], -1]
>>> U[~mask, -1] = -T[s[j], -1]
>>> print(U)
[[ 2 1 -7]
[ 1 0 4]
[ 2 0 -3]]

PCA: TypeError: can only concatenate list (not "int") to list

I was trying to fit a PCA on my dataset following suggested solution from this post. The code worked for iris data of shape (150, 8), which looks like this:
array([[ 1.7837721 , -1.23464679, 4.27808537, ..., 0.63061657,
-1.79849625, -1.41574397],
[-0.35396307, -0.13400175, 3.91751182, ..., -0.58928302,
-0.15735542, -0.99157312],
[-0.20380491, -1.06074392, 4.65814864, ..., 2.19686369,
0.14920164, 2.33371106],
...,
[-1.05079672, 1.46836264, 5.41970214, ..., 0.32847349,
0.27133141, 1.01266607],
[ 0.19569856, 0.57739573, 3.84749973, ..., 0.02400556,
-0.08193678, 0.51223263],
[ 0.04905765, 0.66314259, 6.22608157, ..., 0.60076934,
-0.56890579, -0.23642103]])
However, using my data of shape (3475, 29) it caught error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-292-5661ffbde57b> in <module>
38 # data = array([randn(8) for k in range(150)])
39 data[:50, 2:4] += 5
---> 40 data[50:, 2:5] += 5
41
42 """ visualize """
TypeError: can only concatenate list (not "int") to list
My data(shape (3475, 29)) looks like this:
array([[58.5, 27.0, 88.5, ..., nan, 0.0, -3.0],
[58.5, 27.0, 88.5, ..., nan, 0.0, -3.0],
[47.0, 45.0, 92.0, ..., 1.6, -0.649519052838329,
-1.1249999999999998],
...,
[46.0, 44.5, 98.0, ..., 2.5, 0.0, -1.3],
[46.0, 40.0, 98.0, ..., 2.5, 0.0, -1.3],
[46.5, 44.5, 76.5, ..., 17.767857142857142, -0.4788774197473401,
-1.4219984343829701]], dtype=object)
Code used:
# SO - doug - my data
from numpy import array, dot, mean, std, empty, argsort
from numpy.linalg import eigh, solve
from numpy.random import randn
from matplotlib.pyplot import subplots, show
def cov(X):
"""
Covariance matrix
note: specifically for mean-centered data
note: numpy's `cov` uses N-1 as normalization
"""
return dot(X.T, X) / X.shape[0]
# N = data.shape[1]
# C = empty((N, N))
# for j in range(N):
# C[j, j] = mean(data[:, j] * data[:, j])
# for k in range(j + 1, N):
# C[j, k] = C[k, j] = mean(data[:, j] * data[:, k])
# return C
def pca(data, pc_count = None):
"""
Principal component analysis using eigenvalues
note: this mean-centers and auto-scales the data (in-place)
"""
data -= mean(data, 0)
data /= std(data, 0)
C = cov(data)
E, V = eigh(C)
key = argsort(E)[::-1][:pc_count]
E, V = E[key], V[:, key]
U = dot(data, V) # used to be dot(V.T, data.T).T
return U, E, V
""" test data """
# data = array([randn(8) for k in range(150)])
data = my_data1 # Using my own data
data[:50, 2:4] += 5
data[50:, 2:5] += 5
""" visualize """
trans = pca(data, 3)[0]
fig, (ax1, ax2) = subplots(1, 2)
ax1.scatter(data[:50, 0], data[:50, 1], c = 'r')
ax1.scatter(data[50:, 0], data[50:, 1], c = 'b')
ax2.scatter(trans[:50, 0], trans[:50, 1], c = 'r')
ax2.scatter(trans[50:, 0], trans[50:, 1], c = 'b')
show()
What does
data[:50, 2:4] += 5
data[50:, 2:5] += 5
do?
I tried to replace these two lines with
data = [data[:50, 2:4] += 5]
data = [data[50:, 2:5] += 5]
based on this answer, and it returned
File "<ipython-input-296-5d80e1852b4e>", line 42
data = [data[:50, 2:4] += 5]
^
SyntaxError: invalid syntax
Any advice appreciated!

If data is a 2d numeric array,
data[:50, 2:4]
selects a slice (technically a view) of the array, and
data[:50, 2:4] += 5
adds 5 to all elements of that slice - and modifying data.
But
TypeError: can only concatenate list (not "int") to list
means that data contains one or more lists, not (just) numbers. For a list, + is not (mathematical) addition, but rather a join/concatenate. [1,2,3]+[4].
As for your second try:
[data[:50, 2:4] += 5]
is very different from
[i + 1]
[] makes a list. You cannot perform assignment, such as = or += inside a list. Hence the syntax error.
The (3475, 29) array that you show lists the dtype as object. That's a strong indicator that it contains things other than numbers (or in addition to). And according to the error, that must be a list(s).
So you need to clean up mydata.

Expectation Maximization Algorithm (EM) for Gaussian Mixture Models (GMMs)

I'm trying to apply the Expectation Maximization Algorithm (EM) to a Gaussian Mixture Model (GMM) using Python and NumPy. The PDF document I am basing my implementation on can be found here.
Below are the equations:
When applying the algorithm I get the mean of the first and second cluster equal to:
array([[2.50832195],
[2.51546208]])
When the actual vector means for the first and second cluster are, respectively:
array([[0],
[0]])
and:
array([[5],
[5]])
The same thing happens when getting the values of the covariance matrices I get:
array([[7.05168736, 6.17098629],
[6.17098629, 7.23009494]])
When it should be:
array([[1, 0],
[0, 1]])
for both clusters.
Here is the code:
np.random.seed(1)
# first cluster
X_11 = np.random.normal(0, 1, 1000)
X_21 = np.random.normal(0, 1, 1000)
# second cluster
X_12 = np.random.normal(5, 1, 1000)
X_22 = np.random.normal(5, 1, 1000)
X_1 = np.concatenate((X_11,X_12), axis=None)
X_2 = np.concatenate((X_21,X_22), axis=None)
# data matrix of k x n dimensions (2 x 2000 dimensions)
X = np.concatenate((np.array([X_1]),np.array([X_2])), axis=0)
# multivariate normal distribution function gives n x 1 vector (2000 x 1 vector)
def normal_distribution(x, mu, sigma):
mvnd = []
for i in range(np.shape(x)[1]):
gd = (2*np.pi)**(-2/2) * np.linalg.det(sigma)**(-1/2) * np.exp((-1/2) * np.dot(np.dot((x[:,i:i+1]-mu).T, np.linalg.inv(sigma)), (x[:,i:i+1]-mu)))
mvnd.append(gd)
return np.reshape(np.array(mvnd), (np.shape(x)[1], 1))
# Initialized parameters
sigma_1 = np.array([[10, 0],
[0, 10]])
sigma_2 = np.array([[10, 0],
[0, 10]])
mu_1 = np.array([[10],
[10]])
mu_2 = np.array([[10],
[10]])
pi_1 = 0.5
pi_2 = 0.5
Sigma_1 = np.empty([2000, 2, 2])
Sigma_2 = np.empty([2000, 2, 2])
for i in range(10):
# E-step:
w_i1 = (pi_1*normal_distribution(X, mu_1, sigma_1))/(pi_1*normal_distribution(X, mu_1, sigma_1) + pi_2*normal_distribution(X, mu_2, sigma_2))
w_i2 = (pi_2*normal_distribution(X, mu_2, sigma_2))/(pi_1*normal_distribution(X, mu_1, sigma_1) + pi_2*normal_distribution(X, mu_2, sigma_2))
# M-step:
pi_1 = np.sum(w_i1)/2000
pi_2 = np.sum(w_i2)/2000
mu_1 = np.array([(1/(np.sum(w_i1)))*np.sum(w_i1.T*X, axis=1)]).T
mu_2 = np.array([(1/(np.sum(w_i2)))*np.sum(w_i2.T*X, axis=1)]).T
for i in range(2000):
Sigma_1[i:i+1, :, :] = w_i1[i:i+1,:]*np.dot((X[:,i:i+1]-mu_1), (X[:,i:i+1]-mu_1).T)
Sigma_2[i:i+1, :, :] = w_i2[i:i+1,:]*np.dot((X[:,i:i+1]-mu_2), (X[:,i:i+1]-mu_2).T)
sigma_1 = (1/(np.sum(w_i1)))*np.sum(Sigma_1, axis=0)
sigma_2 = (1/(np.sum(w_i2)))*np.sum(Sigma_2, axis=0)
Would really appreciate if someone could point out the mistake in my code or in my misunderstanding of the algorithm..

Weighted masking in TensorFlow

I have the following task: having two vectors
[v_1, ..., v_n] and [w_1, ..., w_n] build new vector [v_1] * w_1 + ... + [v_n] * w_n.
For exmaple for v = [0.5, 0.1, 0.7] and w = [2, 3, 0] the result will be
[0.5, 0.5, 0.1, 0.1, 0.1].
In case of using vanilla python, the solution would be
v, w = [...], [...]
res = []
for i in range(len(v)):
res += [v[i]] * w[i]
Is it possible to build such code within TensorFlow function? It seems to be an extension of tf.boolean_mask with additional argument like weights or repeats.

Here is a simple solution using tf.sequence_mask:
import tensorflow as tf
v = tf.constant([0.5, 0.1, 0.7])
w = tf.constant([2, 3, 0])
m = tf.sequence_mask(w)
v2 = tf.tile(v[:, None], [1, tf.shape(m)[1]])
res = tf.boolean_mask(v2, m)
sess = tf.InteractiveSession()
print(res.eval())
# array([0.5, 0.5, 0.1, 0.1, 0.1], dtype=float32)