We Keep Coding

How do I use scipy.interpolate.griddata similar to griddata in MATLAB?

I am performing some 3D linear interpolation to a set of 3D images (96 * 96 * 60) to apply affine transform. I used MATLAB griddata function:
% T = [4*4] affine transform matrix
[X_org,Y_org,Z_org]=meshgrid(1:imgWidth,1:imgHeight,1:sliceNum);
x=X_org(:);
y=Y_org(:);
z=Z_org(:);
% Fill in v with input image data
for i = 1:length(x)
v(i)=img_in(y(i),x(i),z(i));
end
v = v';
tempsource = [X_org(:) Y_org(:) Z_org(:) ones(length(Z_org(:)),1)]';
sourceCoor =inv(T) * tempsource; % TargetCoor=T*sourceCoor, padding 1s to the matrix
%interpolation method = 'linear';
xq=sourceCoor(1,:)';
yq=sourceCoor(2,:)';
zq=sourceCoor(3,:)';
vq = griddata(x,y,z,v,xq,yq,zq);
img_out = reshape(vq,imgWidth,imgHeight,sliceNum); % the correct output
However, if I tried in Python, scipy.interpolate.griddata behaves differently. I first create same inputs, x,y,z,v,xq,yq,zq to the function:
# T = [4*4] affine transform matrix
Y_org, X_org, Z_org = np.mgrid[1:imgWidth+1, 1:imgHeight+1, 1:sliceNum+1]
x = X_org.flatten(order='F')
y = Y_org.flatten(order='F')
z = Z_org.flatten(order='F')
v = np.zeros_like(x, dtype=np.float32)
for i in range(len(x)):
v[i] = img_in[x[i]-1, y[i]-1, z[i]-1]
v = v.T
tempSoure = np.vstack((x, y, z, np.ones(z.shape[0])))
sourceCoor = np.linalg.inv(T) # tempSoure
xq = sourceCoor[0, :].T
yq = sourceCoor[1, :].T
zq = sourceCoor[2, :].T
# every variable before this line is same as in MATLAB
# vq = griddata((x, y, z), v, (xq, yq, zq))
# vq = griddata((xq, yq, zq), v, (x, y, z))
img_out = np.reshape(vq, (imgHeight, imgWidth, sliceNum))
Two approaches in last three lines of the Python code snippet do not work. The upper one takes forever to execute, while the lower one returns a different and weird result. I see that Python is 0-indexed and MATLAB is 1-indexed so I made some modification in Python version (v[i] = img_in[x[i]-1, y[i]-1, z[i]-1]{The order of indexes I passed in also changed to maintain the original v}, etc.), but I am worried the griddata function in Python will still try to fit with 0-indexed style, resulting to no solution for my 1-indexed style input. Is that the cause of my outcomes? And how can I manage to get the correct output image, assuming 0-indexed?

Dimensionality Reduction – PCA explanation

I don't think I have a good understanding of PCA, can someone help with my confusion below please?
Take iris dataset as an example, I have 4 covariates, x1:sepal length; x2:sepal width; x3:petal length; x4:petal width. And the formula can be seen below, a1,a2,a3,a4 are the weightings for the covariates. And PCA will try to maximise the variance using different linear transformations. While also follows the rule of a1^2 + a2^2 + a3^2 + a4^2=1. I'm interested in knowing the value of a1,a2,a3,a4.
a1*x1 + a2*x2 + a3*x3 + a4*x4
I have below code on python which I think is correct?
# load libraries
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
import seaborn as sns
import pandas as pd
import numpy as np
iris = load_iris()
X = iris.data
df = pd.DataFrame(X,columns=iris.feature_names)
pca = decomposition.PCA(n_components = 4)
digits_pca_4 = pca.fit(X)
digits_pca_4.explained_variance_ratio_
And the result is
array([0.92461872, 0.05306648, 0.01710261, 0.00521218])
My question is:
Am I correct to assume that a1=sqrt(0.92), a2=sqrt(0.05), a3=sqrt(0.02), a4=sqrt(0.005)?
Second question:
And if I were to choose the linear combination of a1=a2=a3=a4=0.5, what's the variance of this compared to the variance from the PCA (I'm assuming it's less than the PCA result since PCA maximise the variance?)? How can I get the variance of when a1=a2=a3=a4=0.5 in python? And is the variance from PCA the code below?
pca.explained_variance_.sum()
Many thanks!

To answer directly your question: no, your initial interpretation is not correct.
Explanation
The actual projection done by PCA is a matrix multiplication Y = (X - u) W where u is the mean of X (u = X.mean(axis=0)) and W is the projection matrix found by PCA: a n x p orthonormal matrix, where n is the original data dimension and p the desired output dimensions. The expression you give (a1*x1 + a2*x2 + a3*x3 + a4*x4) does not mean anything with all values being scalars. At best, it could mean the calculation of a single component, using one column j of W below as the a_k: Y[i, j] == sum(W[k, j] * (X[i, k] - u[k]) for k in range(n)).
In any case, you can inspect all the variables of the result of pca = PCA.fit(...) with vars(pca). In particular, the projection matrix described above can be found as W = pca.components_.T. The following statements can be verified:
# projection
>>> u = pca.mean_
... W = pca.components_.T
... Y = (X - u).dot(W)
... np.allclose(Y, pca.transform(X))
True
>>> np.allclose(X.mean(axis=0), u)
True
# orthonormality
>>> np.allclose(W.T.dot(W), np.eye(W.shape[1]))
True
# explained variance is the sample variation (not population variance)
# of the projection (i.e. the variance along the proj axes)
>>> np.allclose(Y.var(axis=0, ddof=1), pca. explained_variance_)
True
Graphical demo
The simplest way to understand PCA is that it is purely a rotation in n-D (after mean removal) while retaining only the first p-dimensions. The rotation is such that your data's directions of largest variance become aligned with the natural axes in the projection.
Here is a little demo code to help you visualize what's going on. Please also read the Wikipedia page on PCA.
def pca_plot(V, W, idx, ax):
# plot only first 2 dimensions of W along with axes W
colors = ['k', 'r', 'b', 'g', 'c', 'm', 'y']
u = V.mean(axis=0) # n
axes_lengths = 1.5*(V - u).dot(W).std(axis=0)
axes = W * axes_lengths # n x p
axes = axes[:2].T # p x 2
ax.set_aspect('equal')
ax.scatter(V[:, 0], V[:, 1], alpha=.2)
ax.scatter(V[idx, 0], V[idx, 1], color='r')
hlen = np.max(np.linalg.norm((V - u)[:, :2], axis=1)) / 25
for k in range(axes.shape[0]):
ax.arrow(*u[:2], *axes[k], head_width=hlen/2, head_length=hlen, fc=colors[k], ec=colors[k])
def pca_demo(X, p):
n = X.shape[1] # input dimension
pca = PCA(n_components=p).fit(X)
u = pca.mean_
v = pca.explained_variance_
W = pca.components_.T
Y = pca.transform(X)
assert np.allclose((X - u).dot(W), Y)
# plot first 2D of both input space and output space
# for visual identification: select a point that's as far as possible
# in the direction of the diagonal of the axes cube, after normalization
# Z: variance-1 projection
Z = (X - u).dot(W/np.sqrt(v))
idx = np.argmax(Z.sum(axis=1) / np.sqrt(np.linalg.norm(Z, axis=1)))
fig, ax = plt.subplots(ncols=2, figsize=(12, 6))
# input space
pca_plot(X, W, idx, ax[0])
ax[0].set_title('input data (first 2D)')
# output space
pca_plot(Y, np.eye(p), idx, ax[1])
ax[1].set_title('projection (first 2D)')
return Y, W, u, pca
Examples
Iris data
# to better understand the shape of W, we project onto
# a space of dimension p=3
X = load_iris().data
Y, W, u, pca = pca_demo(X, 3)
Note that the projection is really just (X - u) W:
>>> np.allclose((X - u).dot(W), Y)
True
Synthetic ellipsoid data
A = np.array([
[20, 10, 7],
[-1, 3, 7],
[5, 1, 2],
])
X = np.random.normal(size=(1000, A.shape[0])).dot(A)
Y, W, u, pca = pca_demo(X, 3)

Whitening transformation does NOT return a unit covariance matrix

For this question, I am using the following Wiki definition of Matrix whitening:
From the definition, I expect the covariance matrix of Y to be the identity matrix. However, this is far from the truth!
Here is the reproduction:
import numpy as np
# random matrix
dim1 = 512 # dimentionality_of_features
dim2 = 100 # no_of_samples
X = np.random.rand(dim1, dim2)
# centering to have mean 0
X = X - np.mean(X, axis=1, keepdims=True)
# covariance of X
Xcov = np.dot(X, X.T) / (X.shape[1] - 1)
# SVD decomposition
# Eigenvecors and eigenvalues
Ec, wc, _ = np.linalg.svd(Xcov)
# get only the first positive ones (for numerical stability)
k_c = (wc > 1e-5).sum()
# Diagonal Matrix of eigenvalues
Dc = np.diag((wc[:k_c]+1e-6)**-0.5)
# E D ET should be the whitening matrix
W = Ec[:,:k_c].dot(Dc).dot(Ec[:,:k_c].T)
# SVD decomposition End
Y = W.dot(X)
# Now apply the same to the whitened X
Ycov = np.dot(Y, Y.T) / (Y.shape[1] - 1)
print(Ycov)
>> [[ 0.19935189 -0.00740203 -0.00152036 ... 0.00133161 -0.03035149
0.02638468] ...
It seems that it won't give me a unit diagonal matrix, unless, dim2 >> dim1.
If I take dim2=1 then I get a vector (although in the example I get an error due to division by 0), and by the Wikis definition, it is incorrect?

Cost Function and Gradient Seem to be Working, but scipy.optimize functions are not

I'm working through my Matlab code for the Andrew NG Coursera course and turning it into python. I am working on non-regularized logistic regression and after writing my gradient and cost functions I needed something similar to fminunc and after some googling, I found a couple options. They are both returning the same results, but they do not match what is in Andrew NG's expected results code. Others seem to be getting this to work correctly, but I'm wondering why my specific code does not seem to return the desired result when using scipy.optimize functions, but does for the cost and gradient pieces earlier in the code.
The data I'm using can be found at the link below;
ex2data1
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as op
#Machine Learning Online Class - Exercise 2: Logistic Regression
#Load Data
#The first two columns contains the exam scores and the third column contains the label.
data = pd.read_csv('ex2data1.txt', header = None)
X = np.array(data.iloc[:, 0:2]) #100 x 3
y = np.array(data.iloc[:,2]) #100 x 1
y.shape = (len(y), 1)
#Creating sub-dataframes for plotting
pos_plot = data[data[2] == 1]
neg_plot = data[data[2] == 0]
#==================== Part 1: Plotting ====================
#We start the exercise by first plotting the data to understand the
#the problem we are working with.
print('Plotting data with + indicating (y = 1) examples and o indicating (y = 0) examples.')
plt.plot(pos_plot[0], pos_plot[1], "+", label = "Admitted")
plt.plot(neg_plot[0], neg_plot[1], "o", label = "Not Admitted")
plt.xlabel('Exam 1 score')
plt.ylabel('Exam 2 score')
plt.legend()
plt.show()
def sigmoid(z):
'''
SIGMOID Compute sigmoid function
g = SIGMOID(z) computes the sigmoid of z.
Instructions: Compute the sigmoid of each value of z (z can be a matrix,
vector or scalar).
'''
g = 1 / (1 + np.exp(-z))
return g
def costFunction(theta, X, y):
'''
COSTFUNCTION Compute cost and gradient for logistic regression
J = COSTFUNCTION(theta, X, y) computes the cost of using theta as the
parameter for logistic regression and the gradient of the cost
w.r.t. to the parameters.
'''
m = len(y) #number of training examples
h = sigmoid(X.dot(theta)) #logisitic regression hypothesis
J = (1/m) * np.sum((-y*np.log(h)) - ((1-y)*np.log(1-h)))
#h is 100x1, y is %100x1, these end up as 2 vector we subtract from each other
#then we sum the values by rows
#cost function for logisitic regression
return J
def gradient(theta, X, y):
m = len(y)
grad = np.zeros((theta.shape))
h = sigmoid(X.dot(theta))
for i in range(len(theta)): #number of rows in theta
XT = X[:,i]
XT.shape = (len(X),1)
grad[i] = (1/m) * np.sum((h-y)*XT) #updating each row of the gradient
return grad
#============ Part 2: Compute Cost and Gradient ============
#In this part of the exercise, you will implement the cost and gradient
#for logistic regression. You neeed to complete the code in costFunction.m
#Add intercept term to x and X_test
Bias = np.ones((len(X), 1))
X = np.column_stack((Bias, X))
#Initialize fitting parameters
initial_theta = np.zeros((len(X[0]), 1))
#Compute and display initial cost and gradient
(cost, grad) = costFunction(initial_theta, X, y), gradient(initial_theta, X, y)
print('Cost at initial theta (zeros): %f' % cost)
print('Expected cost (approx): 0.693\n')
print('Gradient at initial theta (zeros):')
print(grad)
print('Expected gradients (approx):\n -0.1000\n -12.0092\n -11.2628')
#Compute and display cost and gradient with non-zero theta
test_theta = np.array([[-24], [0.2], [0.2]]);
(cost, grad) = costFunction(test_theta, X, y), gradient(test_theta, X, y)
print('\nCost at test theta: %f' % cost)
print('Expected cost (approx): 0.218\n')
print('Gradient at test theta:')
print(grad)
print('Expected gradients (approx):\n 0.043\n 2.566\n 2.647\n')
result = op.fmin_tnc(func = costFunction, x0 = initial_theta, fprime = gradient, args = (X,y))
result[1]
Result = op.minimize(fun = costFunction,
x0 = initial_theta,
args = (X, y),
method = 'TNC',
jac = gradient, options={'gtol': 1e-3, 'disp': True, 'maxiter': 1000})
theta = Result.x
theta
test = np.array([[1, 45, 85]])
prob = sigmoid(test.dot(theta))
print('For a student with scores 45 and 85, we predict an admission probability of %f,' % prob)
print('Expected value: 0.775 +/- 0.002\n')

This was a very difficult problem to debug, and illustrates a poorly documented aspect of the scipy.optimize interface. The documentation vaguely indicates that theta will be passed around as a vector:
Minimization of scalar function of one or more variables.
In general, the optimization problems are of the form:
minimize f(x) subject to
g_i(x) >= 0, i = 1,...,m
h_j(x) = 0, j = 1,...,p
where x is a vector of one or more variables.
What's important is that they really mean vector in the most primitive sense, a 1-dimensional array. So you have to expect that whenever theta is passed into one of your callbacks, it will be passed in as a 1-d array. But in numpy, 1-d arrays sometimes behave differently from 2-d row arrays (and, obviously, from 2-d column arrays).
I don't know exactly why it's causing a problem in your case, but it's easily fixed regardless. You just have to add the following at the top of both your cost function and your gradient function:
theta = theta.reshape(-1, 1)
This guarantees that theta will be a 2-d column array, as expected. Once you've done this, the results are correct.

I have had similar issues with Scipy dealing with the same problem as you. As senderle points out the interface is not the easiest to deal with, especially combined with the numpy array interface... Here is my implementation which works as expected.
Defining the cost and gradient functions
Note that initial_theta is passed as a simple array of shape (3,) and converted to a column vector of shape (3,1) within the function. The gradient function then returns the grad.ravel() which has shape (3,) again. This is important as doing otherwise caused an error message with various optimization methods in Scipy.optimize.
Note that different methods have different behaviours but returning .ravel() seems to fix most issues...
import pandas as pd
import numpy as np
import scipy.optimize as opt
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def CostFunc(theta,X,y):
#Initializing variables
m = len(y)
J = 0
grad = np.zeros(theta.shape)
#Vectorized computations
z = X # theta
h = sigmoid(z)
J = (1/m) * ( (-y.T # np.log(h)) - (1 - y).T # np.log(1-h));
return J
def Gradient(theta,X,y):
#Initializing variables
m = len(y)
theta = theta[:,np.newaxis]
grad = np.zeros(theta.shape)
#Vectorized computations
z = X # theta
h = sigmoid(z)
grad = (1/m)*(X.T # ( h - y));
return grad.ravel() #<-- This is the trick
Initializing variables and parameters
Note that initial_theta.shape returns (3,)
X = data1.iloc[:,0:2].values
m,n = X.shape
X = np.concatenate((np.ones(m)[:,np.newaxis],X),1)
y = data1.iloc[:,-1].values[:,np.newaxis]
initial_theta = np.zeros((n+1))
Calling Scipy.optimize
model = opt.minimize(fun = CostFunc, x0 = initial_theta, args = (X, y), method = 'TNC', jac = Gradient)
Any comments from more knowledgeable people are welcome, this Scipy interface is a mystery to me, thanks

How to implement ZCA Whitening? Python

Im trying to implement ZCA whitening and found some articles to do it, but they are a bit confusing.. can someone shine a light for me?
Any tip or help is appreciated!
Here is the articles i read :
http://courses.media.mit.edu/2010fall/mas622j/whiten.pdf
http://bbabenko.tumblr.com/post/86756017649/learning-low-level-vision-feautres-in-10-lines-of
I tried several things but most of them i didnt understand and i got locked at some step.
Right now i have this as base to start again :
dtype = np.float32
data = np.loadtxt("../inputData/train.csv", dtype=dtype, delimiter=',', skiprows=1)
img = ((data[1,1:]).reshape((28,28)).astype('uint8')*255)

Here is a python function for generating the ZCA whitening matrix:
def zca_whitening_matrix(X):
"""
Function to compute ZCA whitening matrix (aka Mahalanobis whitening).
INPUT: X: [M x N] matrix.
Rows: Variables
Columns: Observations
OUTPUT: ZCAMatrix: [M x M] matrix
"""
# Covariance matrix [column-wise variables]: Sigma = (X-mu)' * (X-mu) / N
sigma = np.cov(X, rowvar=True) # [M x M]
# Singular Value Decomposition. X = U * np.diag(S) * V
U,S,V = np.linalg.svd(sigma)
# U: [M x M] eigenvectors of sigma.
# S: [M x 1] eigenvalues of sigma.
# V: [M x M] transpose of U
# Whitening constant: prevents division by zero
epsilon = 1e-5
# ZCA Whitening matrix: U * Lambda * U'
ZCAMatrix = np.dot(U, np.dot(np.diag(1.0/np.sqrt(S + epsilon)), U.T)) # [M x M]
return ZCAMatrix
And an example of the usage:
X = np.array([[0, 2, 2], [1, 1, 0], [2, 0, 1], [1, 3, 5], [10, 10, 10] ]) # Input: X [5 x 3] matrix
ZCAMatrix = zca_whitening_matrix(X) # get ZCAMatrix
ZCAMatrix # [5 x 5] matrix
xZCAMatrix = np.dot(ZCAMatrix, X) # project X onto the ZCAMatrix
xZCAMatrix # [5 x 3] matrix
Hope it helps!
Details for why Edgar Andrés Margffoy Tuay's answer is not correct: As pointed out in R.M's comment, Edgar Andrés Margffoy Tuay's ZCA whitening function contains a small, but crucial mistake: the np.diag(S) should be removed. Numpy returns S as a m x 1 vector and not a m x m matrix (as is common to other svd implementations, e.g. Matlab). Hence the ZCAMatrix variable becomes a m x 1 vector and not a m x m matrix as it should be (when the input is m x n). (Also, the covariance matrix in Andfoy's answer is only valid if X is pre-centered, i.e mean 0).
Other references for ZCA: You can see the full answer, in Python, to the Stanford UFLDL ZCA Whitening exercise here.

Is your data stored in an mxn matrix? Where m is the dimension of the data and n are the total number of cases? If that's not the case, you should resize your data. For instance if your images are of size 28x28 and you have only one image, you should have a 1x784 vector. You could use this function:
import numpy as np
def flatten_matrix(matrix):
vector = matrix.flatten(1)
vector = vector.reshape(1, len(vector))
return vector
Then you apply ZCA Whitening to your training set using:
def zca_whitening(inputs):
sigma = np.dot(inputs, inputs.T)/inputs.shape[1] #Correlation matrix
U,S,V = np.linalg.svd(sigma) #Singular Value Decomposition
epsilon = 0.1 #Whitening constant, it prevents division by zero
ZCAMatrix = np.dot(np.dot(U, np.diag(1.0/np.sqrt(np.diag(S) + epsilon))), U.T) #ZCA Whitening matrix
return np.dot(ZCAMatrix, inputs) #Data whitening
It is important to save the ZCAMatrix matrix, you should multiply your test cases if you want to predict after training the Neural Net.
Finally, I invite you to take the Stanford UFLDL Tutorials at http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial or http://ufldl.stanford.edu/tutorial/ . They have pretty good explanations and also some programming exercises on MATLAB, however, almost all the functions found on MATLAB are on Numpy by the same name. I hope this may give an insight.

I may be a little late to the discussion, but I found this thread recently as I struggled to implement ZCA in TensorFlow because my poor PC processor was too slow to process large volume of data.
If anyone is interested, I have made a gist of my implementation of the ZCA in TensorFlow:
import tensorflow as tf
from keras.datasets import mnist
import numpy as np
tf.enable_eager_execution()
assert tf.executing_eagerly()
class ZCA(object):
"""
Simple ZCA aka Mahalanobis transformation class made in TensorFlow.
The code was largely ported from Keras ImageDataGenerator
"""
def __init__(self, epsilon=1e-5, dtype='float64'):
"""epsilon is the normalization constant, dtype refers to the data type used in the computation.
WARNING: the default precision is set to float64 as i have found that when computing the mean tensorflow'
and numpy results can differ by a substantial amount.
Usage: fit method computes the principal components and should be called first,
compute method returns the actual transformed tensor
NOTE : The input to both methods must be a 4D tensor.
"""
assert dtype is 'float32' or 'float64', "precision must be float32 or float64"
self.epsilon = epsilon
self.dtype = dtype
self.princ_comp = None
self.mean = None
def _featurewise_center(self, images_tensor):
if self.mean is None:
self.mean, _ = tf.nn.moments(images_tensor, axes=(0, 1, 2))
broadcast_shape = [1, 1, 1]
broadcast_shape[2] = images_tensor.shape[3]
self.mean = tf.reshape(self.mean, broadcast_shape)
norm_images = tf.subtract(images_tensor, self.mean)
return norm_images
def fit(self, images_tensor):
assert images_tensor.shape[3], "The input should be a 4D tensor"
if images_tensor.dtype is not self.dtype: # numerical error for float32
images_tensor = tf.cast(images_tensor, self.dtype)
images_tensor = self._featurewise_center(images_tensor)
flat = tf.reshape(images_tensor, (-1, np.prod(images_tensor.shape[1:].as_list())))
sigma = tf.div(tf.matmul(tf.transpose(flat), flat), tf.cast(flat.shape[0], self.dtype))
s, u, _ = tf.svd(sigma)
s_inv = tf.div(tf.cast(1, self.dtype), (tf.sqrt(tf.add(s[tf.newaxis], self.epsilon))))
self.princ_comp = tf.matmul(tf.multiply(u, s_inv), tf.transpose(u))
def compute(self, images_tensor):
assert images_tensor.shape[3], "The input should be a 4D tensor"
assert self.princ_comp is not None, "Fit method should be called first"
if images_tensor.dtype is not self.dtype:
images_tensor = tf.cast(images_tensor, self.dtype)
images_tensors = self._featurewise_center(images_tensor)
flatx = tf.cast(tf.reshape(images_tensors, (-1, np.prod(images_tensors.shape[1:]))), self.dtype)
whitex = tf.matmul(flatx, self.princ_comp)
x = tf.reshape(whitex, images_tensors.shape)
return x
def main():
import matplotlib.pyplot as plt
train_set, test_set = mnist.load_data()
x_train, y_train = train_set
zca1 = ZCA(epsilon=1e-5, dtype='float64')
# input should be a 4D tensor
x_train = x_train.reshape(*x_train.shape, 1)
zca1.fit(x_train)
x_train_transf = zca1.compute(x_train)
# reshaping to 28*28 and casting to uint8 for plotting
x_train_transf = tf.reshape(x_train_transf, x_train_transf.shape[0:3])
fig, axes = plt.subplots(3, 3)
for i, ax in enumerate(axes.flat):
# Plot image.
ax.imshow(x_train_transf[i],
cmap='binary'
)
xlabel = "True: %d" % y_train[i]
ax.set_xlabel(xlabel)
ax.set_xticks([])
ax.set_yticks([])
plt.show()
if __name__ == '__main__':
main()
I know this isn't a proper answer to the original question, but still it may be useful to anyone who is looking for a GPU implementation of ZCA but couldn't find one.

Although both answers refer to the UFLDL tutorial, none of them seems to use the steps as described in it.
Therefore, I thought it might not be bad idea to just provide an answer that simply implements PCA/ZCA-whitening according to the tutorial:
import numpy as np
# generate some random, 2D data
x = np.random.randn(1000, 2)
# and center it
x_c = x - np.mean(x, 0)
# compute the 2x2 covariance matrix
# (remember that covariance matrix is symmetric)
sigma = np.cov(x, rowvar=False)
# and extract eigenvalues and eigenvectors
# using the algorithm for symmetric matrices
l,u = np.linalg.eigh(sigma)
# NOTE that for symmetric matrices,
# eigenvalues and singular values are the same.
# u, l, _ = np.linalg.svd(sigma) should thus give equivalent results
# rotate the (centered) data to decorrelate it
x_rot = np.dot(x_c, u)
# check that the covariance is diagonal (indicating decorrelation)
np.allclose(np.cov(x_rot.T), np.diag(np.diag(np.cov(x_rot.T))))
# scale the data by eigenvalues to get unit variance
x_white = x_rot / np.sqrt(l)
# have the whitened data be closer to the original data
x_zca = np.dot(x_white, u.T)
I assume you can wrap this in a function by yourself...
For completeness, different implementation flavours and their runtime (evaluated on a centred version of CIFAR10):
x = np.random.randn(10_000, 3, 32, 32)
x_ = np.reshape(x, (len(x), -1))
x_c = x_ - np.mean(x_, axis=0)
def zca1(x):
s, u = np.linalg.eigh(x.T # x)
scale = np.sqrt(len(x) / s)
return (u * scale) # u.T
def zca2(x):
u, s, _ = np.linalg.svd(x.T # x, hermitian=True)
scale = np.sqrt(len(x) / s)
return (u * scale) # u.T
def zca3(x):
_, s, v = np.linalg.svd(x, full_matrices=False)
scale = np.sqrt(len(x)) / s
return (v.T * scale) # v
%timeit zca1(x_c)
# 4.57 s ± 14.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit zca2(x_c)
# 4.62 s ± 22.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit zca3(x_c)
# 20.2 s ± 1.2 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
For the mathematics behind this, I refer to this excellent answer from cross validated.

This works with an array of 48x48:
def flatten_matrix(matrix):
vector = matrix.flatten(order='F')
vector = vector.reshape(1, len(vector))
return vector
def zca_whitening(inputs):
sigma = np.dot(inputs, inputs.T)/inputs.shape[1] #Correlation matrix
U,S,V = np.linalg.svd(sigma) #Singular Value Decomposition
epsilon = 0.1 #Whitening constant, it prevents division by zero
ZCAMatrix = np.dot(np.dot(U, np.diag(1.0/np.sqrt(np.diag(S) + epsilon))), U.T) #ZCA Whitening matrix
return np.dot(ZCAMatrix, inputs) #Data whitening
def global_contrast_normalize(X, scale=1., subtract_mean=True, use_std=True,
sqrt_bias=10, min_divisor=1e-8):
"""
__author__ = "David Warde-Farley"
__copyright__ = "Copyright 2012, Universite de Montreal"
__credits__ = ["David Warde-Farley"]
__license__ = "3-clause BSD"
__email__ = "wardefar#iro"
__maintainer__ = "David Warde-Farley"
.. [1] A. Coates, H. Lee and A. Ng. "An Analysis of Single-Layer
Networks in Unsupervised Feature Learning". AISTATS 14, 2011.
http://www.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf
"""
assert X.ndim == 2, "X.ndim must be 2"
scale = float(scale)
assert scale >= min_divisor
mean = X.mean(axis=1)
if subtract_mean:
X = X - mean[:, np.newaxis]
else:
X = X.copy()
if use_std:
ddof = 1
if X.shape[1] == 1:
ddof = 0
normalizers = np.sqrt(sqrt_bias + X.var(axis=1, ddof=ddof)) / scale
else:
normalizers = np.sqrt(sqrt_bias + (X ** 2).sum(axis=1)) / scale
normalizers[normalizers < min_divisor] = 1.
X /= normalizers[:, np.newaxis] # Does not make a copy.
return X
def ZeroCenter(data):
data = data - np.mean(data,axis=0)
return data
def Zerocenter_ZCA_whitening_Global_Contrast_Normalize(data):
numpy_data = np.array(data).reshape(48,48)
data2 = ZeroCenter(numpy_data)
data3 = zca_whitening(flatten_matrix(data2)).reshape(48,48)
data4 = global_contrast_normalize(data3)
data5 = np.rot90(data4,3)
return data5
for example from this image:
returns:
Here is the code:
https://gist.github.com/m-alcu/45f4a083cb5e388d2ed26ace4392ed66, needs to put fer2013.csv file in the same directory (https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data)