We Keep Coding

Visualize documents embeddings and clustering

I have the following dataframe:
print(df)
document embeddings
1 [-1.1132643 , 0.793635 , 0.8664889]
2 [-1.1132643 , 0.793635 , 0.8664889]
3 [-0.19276126, -0.48233205, 0.17549737]
4 [0.2080252 , 0.01567003, 0.0717131]
I want to cluster and visualize them to see the similarities between the documents. What is the best method/steps to do this?
This is just a small dataframe, the original dataframe has more than 20k documents.

Document vectors in your case reside in a 768-dimensional euclidean space. Meaning in a 768-dimensional coordinate space, each point represents a document. Assuming these have been trained correctly, it's safe to imagine that contextually similar documents should be closer to each other in this space as compared to different ones. This may allow you to apply a clustering method to club similar documents together.
For clustering, you can use multiple clustering techniques such as -
Kmeans (clusters based on euclidean distances)
Dbscan (clusters with the notion of density)
Gaussian mixtures (clusters based on a mixture of k gaussians)
You can use Silhouette score to find the optimal number of clusters for the clustering algorithm to best create separations in clusters.
For visualization, you can ONLY visualize in 3D or 2D space. This means you will have to use some dimensionality reduction methods to reduce the 768 dimensions to 3 dimensions or 2 dimensions.
This can be achieved with the following algorithms set to 2 or 3 components -
PCA
T-SNE
LDA (requires labels)
Once you have clustered the data AND reduced the dimensionality of the data separately, you can use matplotlib to plot each of the points in a 2D/3D space and color each point based on its cluster (0-7) to visualize documents and clusters.
#process flow
(20k,768) -> K-clusters (20k,1) ------|
|--- Visualize (3 axis, k colors)
(20k,768) -> Dim reduction (20k,3)----|
Here is an example of the goal you are trying to achieve -
Here, you see the first 2 components of data from T-SNE, and each color represents the clusters you have created from your clustering method of choice (deciding the number of clusters using silhouette score)
EDIT: You can apply dimensionality reduction to project your 768-dimensional data into a 3D or 2D space and THEN cluster using a clustering method. This would reduce the amount of computation you have to handle since now you are clustering only on 3 dimensions instead of 768, but at cost of information that might help you discriminate clusters better.
#process flow
|------------------------|
(20k,768) -> Dim reduction (20k,3)--| |-- Visualize
|--- K-Clusters (20k,1)--|

MNIST dataset conversion

I've trained a decision tree on a dataset (handwritten) which contains 8 x-y points sampled along the length of the number (number digit dataset). The test dataset given (assignment), is the MNIST dataset, which is the pixel intensities in a 28x28 bitmap image. I need to sample 8 points and along the trajectory of the number so that it performs well.
I'm doing this in Python. I don't know what to do with the image to sample those points. Any package/procedure will help.

Simply index the array as you would any other array. The pixel intensities are merely ints. e.g val = arr[3,9].

You do not have the stroke direction in mnist.
Hence, you cannot reliably infer such positions.
You can do the opposite though: render the stroke information as pixel image, train a classifier on that, and then test it with mnist.

There's an MNIST sequence dataset from Edwin de Jong:
Paper: https://arxiv.org/pdf/1611.03068.pdf
Github: https://edwin-de-jong.github.io/blog/mnist-sequence-data/
Blog: https://github.com/edwin-de-jong/mnist-digits-as-stroke-sequences/
and an MNIST classification using RNN by Ryan Epp:
https://www.ryanepp.com/blog/mnist-classification-using-stroke-paths
In both projects, the direction strokes will take at a T-junction depends on the algorithm and is often counterintuitive. This means that there's more to learn for sequences since many stroke patterns will produce the same image.

Proper way of setting up my input and output data for training CNN

I am currently working on the structuring my input and output for a CNN in which the input is different sized images, and for the different sized images are different sized vectors.
Example:
Name: mfaa-b-an165
Output: (278, 13)
Input: (276, 846, 3)
-->
Name: mdms2-b-an405
Output: (538, 13)
Input: (276, 1491, 3)
-->
Here are some shapes of how the data is currently stored. I have to create 13 different CNN for each different feature, in which each CNN extract a separate feature from the feature vector. The problem is though there isn't that much consistency in the number of output compared to input.
The only constant is the number of rows, and color dimension, and the number of columns somehow has to be linked number of features extracted from that image.
My idea was to resize the x-axis of the image, such that would fit with the number of features extracted, but sometimes is this not possible..
Eg.
First example (mfaa-b-an165):
278*3 = 834 => meaning removing 12 columns can provide me 3 output for the pooling layer.
second exmaple (mdms2-b-an405):
538*3 = 1614 => Image should be larger => data should be added not possible.
This clearly doesn't seem to work.. Is there some other way i can make this work?..

t-SNE High Dimension Data Visualisation

I have a twitter corpus which I am using to build sentiment analysis application. The corpus has 5k tweets which have been hand labelled as - negative, neutral or positive
To represent the text - I am using gensim word2vec pretrained vectors. Each word is mapped to 300 dimensions. For a tweet, I add all the word vectors to get a single 300 dim vectors. Thus every tweet is mapped to a single vector of 300 dimension.
I am visualizing my data using t-SNE (tsne python package). See attached image 1 - Red points = negative tweets, Blue points = neutral tweets and Green points = Positive tweets
Question:
In the plot there no clear separation (boundary) among the data points. Can I assume this will also be the case with the original points in 300 Dimensions ?
i.e if points overlap in t-SNE graph then they also overlap in original space and vice-versa ?

Question: In the plot there no clear separation (boundary) among the data points. Can I assume this will also be the case with the original points in 300 Dimensions ?
In most cases NO. By reducing dimensions you will probably loose some information.
The case where you may reduce dimension without losing information is when or data in some dimensions is zero(for example line in 3dimensional space) or when some dimensions linearly dependent on other.
There are few tricks to test how good some dimensionality reductions techniques works. For example:
You may use PCA to reduce dimension form 300 to for example 10. You can calculate sum of 300 eigenvalues(original space) and sum of 10 biggest eigenvalues(these 10 eigenvalues represent eigenvectors that will be used for dimension reduction) and calculate percentage of lost information sum(top-10-eigenvalues)/sum(300-eigenvalues) .This value is not exactly "information" lost, but it is close to that.

Subtract mean from image

I'm implementing a CNN with Theano. In the paper, I have to do this image preprocess before train the CNN
We extracted RGB patches of 61x61 dimensions associated with each poselet activation, subtracted the mean and used this data to train the convnet model shown in Table 1
Can you tell me what does it mean with "subtracted the mean"? Tell me if these steps are correct (it is what I understood)
1) Compute the mean for Red Channel, Green Channel and Blue Channel for the whole image
2) For each pixel, subtract from red value the mean of red channel, from green value the mean of green channel and the same for the blue channel
3) Is it correct to have negative value or do I have use the abs?
Thanks all!!

You should read paper carefully, but what is the most probable is that they mean mean of the patches, so you have N matrices 61x61 pixels, which is equivalent of a vector of length 61^2 (if there are three channels then 3*61^2). What they do - they simple compute mean of each dimension, so they calculate mean over these N vectors in respect to each of the 3*61^2 dimensions. As the result they obtain a mean vector of length 3*61^2 (or mean matrix/mean patch if you prefer) and they substract it from all of these N patches. Resulting patches will have negatives values, it is perfectly fine, you should not take abs value, neural networks prefer this kind of data.

I would assume the mean mentioned in the paper is the mean over all images used in the training set (computed separately for each channel).
Several indications:
Caffe is a lib for ConvNets. In their tutorial they mention the compute image mean part: http://caffe.berkeleyvision.org/gathered/examples/imagenet.html
For this they use the following script: https://github.com/BVLC/caffe/blob/master/examples/imagenet/make_imagenet_mean.sh
which does what I indicated.
Google played around with ConvNets and published their code here: https://github.com/google/deepdream/blob/master/dream.ipynb and they do also use the mean of the training set.
This is of course only indirect evidence since I can not explain you why this happens. In fact I stumbled over this question while trying to figure out precisely that.
//EDIT:
In the mean time I found a source confirming my claim (Highlighting added by me):
There are three common forms of data preprocessing a data matrix X [...]
Mean subtraction is the most common form of preprocessing. It
involves subtracting the mean across every individual feature in the
data, and has the geometric interpretation of centering the cloud of
data around the origin along every dimension. In numpy, this operation
would be implemented as: X -= np.mean(X, axis = 0). With images
specifically, for convenience it can be common to subtract a single
value from all pixels (e.g. X -= np.mean(X)), or to do so separately
across the three color channels.
As we can see, the whole data is used to compute the mean.