We Keep Coding

Performing a random addition between two arrays

What I'm trying to do is get two different arrays, where the first array is just filled with zeros and second array would be populated by random numbers. I would like to perform an operation where only certain elements from the latter array are added to the array filled with zeros and the rest of elements within the former array remain as zero. I'm trying to get the addition done in a random way as well. I just added the code below as an example. I honestly don't know how to perform something like this and I would be very grateful for any help or suggestions! Thank you!
shape = (6, 3)
empty_array = np.zeros(shape)
random_array = 0.1 * np.random.randn(*empty_array)
sum = np.add(empty_array, random_array)

You can use a binary mask with the density P:
P = 0.5
# Repeat the next two lines as needed
mask = np.random.binomial(1, P, size = empty_array.size)\
.reshape(shape).astype(bool)
empty_array[mask] += random_array[mask]
If you plan to add more random elements, you may want to re-generate the mask at each further iteration.

If I understand you correctly from your comments, you want to create random numbers at random indices based on the threshold of some percent of whole array (you do not need to create a whole random array and use only a percent of it, such random number generation is usually costly in larger scales):
sz = shape[0]*shape[1]
#this is your for example 20% threshold
threshold = 0.2
#create random numbers and random indices
random_array = np.random.rand(int(threshold*sz))
random_idx = np.random.randint(0,sz,int(threshold*sz))
#now you can add this random_array to random indices of your desired array
empty_array.reshape(-1)[random_idx] += random_array
or another solution:
sz = shape[0]*shape[1]
#this is your for example 20% threshold
threshold = 0.2
random_array = np.random.rand(int(threshold*sz))
#pad with enough zeros and randomly shuffle and finally reshape it
random_array.resize(sz)
np.random.shuffle(random_array)
#now you can add this random_array to any array of your choice
empty_array += random_array.reshape(shape)
sample output:
[[0. 0. 0. ]
[0. 0. 0. ]
[0. 0. 0.7397274 ]
[0. 0. 0. ]
[0. 0. 0.79541551]
[0.75684113 0. 0. ]]

How to blur 3D array of points, while maintaining their original values? (Python)

I have a sparse 3D array of values. I am trying to turn each "point" into a fuzzy "sphere", by applying a Gaussian filter to the array.
I would like the original value at the point (x,y,z) to remain the same. I just want to create falloff values around this point... But applying the Gaussian filter changes the original (x,y,z) value as well.
I am currently doing this:
dataCube = scipy.ndimage.filters.gaussian_filter(dataCube, 3, truncate=8)
Is there a way for me to normalize this, or do something so that my original values are still in this new dataCube? I am not necessarily tied to using a Gaussian filter, if that is not the best approach.

You can do this using a convolution with a kernel that has 1 as its central value, and a width smaller than the spacing between your data points.
1-d example:
import numpy as np
import scipy.signal
data = np.array([0,0,0,0,0,5,0,0,0,0,0])
kernel = np.array([0.5,1,0.5])
scipy.signal.convolve(data, kernel, mode="same")
gives
array([ 0. , 0. , 0. , 0. , 2.5, 5. , 2.5, 0. , 0. , 0. , 0. ])
Note that fftconvolve might be much faster for large arrays. You also have to specify what should happen at the boundaries of your array.
Update: 3-d example
import numpy as np
from scipy import signal
# first build the smoothing kernel
sigma = 1.0 # width of kernel
x = np.arange(-3,4,1) # coordinate arrays -- make sure they contain 0!
y = np.arange(-3,4,1)
z = np.arange(-3,4,1)
xx, yy, zz = np.meshgrid(x,y,z)
kernel = np.exp(-(xx**2 + yy**2 + zz**2)/(2*sigma**2))
# apply to sample data
data = np.zeros((11,11,11))
data[5,5,5] = 5.
filtered = signal.convolve(data, kernel, mode="same")
# check output
print filtered[:,5,5]
gives
[ 0. 0. 0.05554498 0.67667642 3.0326533 5. 3.0326533
0.67667642 0.05554498 0. 0. ]

Need explanation how specgram function work in python (matplotlib - MATLAB compatible functions)

I'm working on converting my code from python to objective c.
Inside matplotlib.mlab.specgram function I see 3 important functions before fft :
result = stride_windows(x, NFFT, noverlap, axis=0)
result = detrend(result, detrend_func, axis=0)
result, windowVals = apply_window(result, window, axis=0,
return_window=True)
result = np.fft.fft(result, n=pad_to, axis=0)[:numFreqs, :]
I tried to debug to understand purpose of each. For example I have array of input:
x = [1,2,3,4,5,6,7,8,9,10,11,12]
After first function stride_windows (this one to prevent leakage?), if NFFT = 4, noverlap = 2 then:
x = [ [1,3,5,7,9],
[2,4,6,8,10],
[3,5,7,9,11],
[4,6,8,10,12]
]
After detrend nothing changes (I understand of detrend before fft)
Inside apply_window (I don't understand this step):
xshape = list(x.shape)
xshapetarg = xshape.pop(axis) // =4
windowVals = window(np.ones(xshapetarg, dtype=x.dtype))
//result of 4 elements [0.0, 0.75, 0.75, 0.0]
xshapeother = xshape.pop() // =5
otheraxis = (axis+1) % 2 // =1
windowValsRep = stride_repeat(windowVals, xshapeother, axis=otheraxis)
// result windowValsRep = [
[ 0. ,0. ,0. ,0. ,0. ,],
[0.75, 0.75, 0.75, 0.75,
[0.75, 0.75, 0.75, 0.75,
[ 0. ,0. ,0. ,0. ,0. ,]
]
then multiply it with x
windowValsRep * x
Now
x = [
[ 0. , 0. , 0. , 0. , 0. ],
[ 1.5 , 3 , 4.5 , 6. , 7.5 ],
[ 2.25, 3.75 , 5.25 , 6.75 , 8.25 ],
[ 0. , 0. , 0. , 0. , 0. ]
]
And then final is fft, as I know fft only need a single array but here it processes 2 dimension array. Why ?
result = np.fft.fft(x, n=pad_to, axis=0)[:numFreqs, :]
Could anyone explain for me step by step why data need to be processed like this before fft ?
Thanks,

Spectrograms and FFTs are not the same thing. The purpose of a spectogram is to take the FFT of small, equal-sized time chunks. This produces a 2D fourier transform where the X axis is the start time of the time chunk and the Y axis is the energy (or power, etc.) in each frequency in that time chunk. This allows you to see how the frequency components change over time.
This is explained in the documentation for the specgram function:
Data are split into NFFT length segments and the spectrum of each section is computed. The windowing function window is applied to each segment, and the amount of overlap of each segment is specified with noverlap.
As for the individual functions, a lot of what you are asking is described in the documentation for reach function, but I will try to explain in a bit more detail.
The purpose of stride_windows, as described in the documentation, is to convert the 1D array of data into a 2D array of successive time chunks. These are the time chunks that will have their FFT calculated in the final spectrogram. In your case they are length-4 (NFFT=4) time chunks (notice the 4 elements per column). Because you set noverlap=2, the last 2 elements of each column are the same as the first 2 elements of the next column (that is what the overlap means). It is called "stride" because it uses a trick regarding the internal storage of numpy arrays to allow it to create an array with the overlapping windows without taking any additional memory.
The detrend function, as its name implies and as is described in its documentation, removes the trend from a signal. By default it uses the mean, which as the detrend_mean documentation describes, removes the mean (DC offset) of the signal.
The apply_window function does exactly what its name implies, and what its documentation says: it applies a window function to each of the time chunks. This is needed because suddenly cutting of the signal at the beginning and end of the time chunks causes large bursts of broadband energy called transients that will mess up the spectrogram. Windowing the signal reduces those transients. By default the spectrogram function uses the hanning window. This attenuates the beginning and end of each time chunk.
The FFT isn't really 2D. The numpy FFT function allows you to specify an axis to take an FFT over. So in this case, we have a 2D array, and we take the FFT of each column of that array. It is much cleaner and a little faster to do this in one step rather than manually looping over each column.

Can numpy diagonalise a skew-symmetric matrix with real arithmetic?

Any skew-symmetric matrix (A^T = -A) can be turned into a Hermitian matrix (iA) and diagonalised with complex numbers. But it is also possible to bring it into block-diagonal form with a special orthogonal transformation and find its eigevalues using only real arithmetic. Is this implemented anywhere in numpy?

Let's take a look at the dgeev() function of the LAPACK librarie. This routine computes the eigenvalues of any real double-precison square matrix. Moreover, this routine is right behind the python function numpy.linalg.eigvals() of the numpy library.
The method used by dgeev() is described in the documentation of LAPACK. It requires the reduction of the matrix A to its real Schur form S.
Any real square matrix A can be expressed as:
A=QSQ^t
where:
Q is a real orthogonal matrix: QQ^t=I
S is a real block upper triangular matrix. The blocks on the diagonal of S are of size 1×1 or 2×2.
Indeed, if A is skew-symmetric, this decomposition seems really close to a block diagonal form obtained by a special orthogonal transformation of A. Moreover, it is really to see that the Schur form S of the skew symmetric matrix A is ... skew-symmetric !
Indeed, let's compute the transpose of S:
S^t=(Q^tAQ)^t
S^t=Q^t(Q^tA)^t
S^t=Q^tA^tQ
S^t=Q^t(-A)Q
S^t=-Q^tAQ
S^t=-S
Hence, if Q is special orthogonal (det(Q)=1), S is a block diagonal form obtained by a special orthogonal transformation. Else, a special orthogonal matrix P can be computed by permuting the first two columns of Q and another Schur form Sd of the matrix A is obtained by changing the sign of S_{12} and S_{21}. Indeed, A=PSdP^t. Then, Sd is a block diagonal form of A obtained by a special orthogonal transformation.
In the end, even if numpy.linalg.eigvals() applied to a real matrix returns complex numbers, there is little complex computation involved in the process !
If you just want to compute the real Schur form, use the function scipy.linalg.schur() with argument output='real'.
Just a piece of code to check that:
import numpy as np
import scipy.linalg as la
a=np.random.rand(4,4)
a=a-np.transpose(a)
print "a= "
print a
#eigenvalue
w, v =np.linalg.eig(a)
print "eigenvalue "
print w
print "eigenvector "
print v
# Schur decomposition
#import scipy
#print scipy.version.version
t,z=la.schur(a, output='real', lwork=None, overwrite_a=True, sort=None, check_finite=True)
print "schur form "
print t
print "orthogonal matrix "
print z

Yes you can do it via sticking a unitary transformation in the middle of the product hence we get
A = V * U * V^-1 = V * T' * T * U * T' * T * V^{-1}.
Once you get the idea you can optimize the code by tiling things but let's do it the naive way by forming T explicitly.
If the matrix is even-sized then all blocks are complex conjugates. Otherwise we get a zero as the eigenvalue. The eigenvalues are guaranteed to have zero real parts so the first thing is to clean up the noise and then order such that the zeros are on the upper left corner (arbitrary choice).
n = 5
a = np.random.rand(n,n)
a=a-np.transpose(a)
[u,v] = np.linalg.eig(a)
perm = np.argsort(np.abs(np.imag(u)))
unew = 1j*np.imag(u[perm])
Obviously, we need to reorder the eigenvector matrix too to keep things equivalent.
vnew = v[:,perm]
Now so far we did nothing other than reordering the middle eigenvalue matrix in the eigenvalue decomposition. Now we switch from complex form to real block diagonal form.
First we have to know how many zero eigenvalues there are
numblocks = np.flatnonzero(unew).size // 2
num_zeros = n - (2 * numblocks)
Then we basically, form another unitary transformation (complex this time) and stick it the same way
T = sp.linalg.block_diag(*[1.]*num_zeros,np.kron(1/np.sqrt(2)*np.eye(numblocks),np.array([[1.,1j],[1,-1j]])))
Eigs = np.real(T.conj().T.dot(np.diag(unew).dot(T)))
Evecs = np.real(vnew.dot(T))
This gives you the new real valued decomposition. So the code all in one place
n = 5
a = np.random.rand(n,n)
a=a-np.transpose(a)
[u,v] = np.linalg.eig(a)
perm = np.argsort(np.abs(np.imag(u)))
unew = 1j*np.imag(u[perm])
vnew = v[perm,:]
numblocks = np.flatnonzero(unew).size // 2
num_zeros = n - (2 * numblocks)
T = sp.linalg.block_diag(*[1.]*num_zeros,np.kron(1/np.sqrt(2)*np.eye(numblocks),np.array([[1.,1j],[1,-1j]])))
Eigs = np.real(T.conj().T.dot(np.diag(unew).dot(T)))
Evecs = np.real(vnew.dot(T))
print(np.allclose(Evecs.dot(Eigs.dot(np.linalg.inv(Evecs))) - a,np.zeros((n,n))))
gives True. Note that this is the naive way of obtaining the real spectral decomposition. There are lots of places where you need to keep track of numerical error accumulation.
Example output
Eigs
Out[379]:
array([[ 0. , 0. , 0. , 0. , 0. ],
[ 0. , 0. , -0.61882847, 0. , 0. ],
[ 0. , 0.61882847, 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , -1.05097581],
[ 0. , 0. , 0. , 1.05097581, 0. ]])
Evecs
Out[380]:
array([[-0.15419078, -0.27710323, -0.39594838, 0.05427001, -0.51566173],
[-0.22985364, 0.0834649 , 0.23147553, -0.085043 , -0.74279915],
[ 0.63465436, 0.49265672, 0. , 0.20226271, -0.38686576],
[-0.02610706, 0.60684296, -0.17832525, 0.23822511, 0.18076858],
[-0.14115513, -0.23511356, 0.08856671, 0.94454277, 0. ]])

Faster convolution of probability density functions in Python

Suppose the convolution of a general number of discrete probability density functions needs to be calculated. For the example below there are four distributions which take on values 0,1,2 with the specified probabilities:
import numpy as np
pdfs = np.array([[0.6,0.3,0.1],[0.5,0.4,0.1],[0.3,0.7,0.0],[1.0,0.0,0.0]])
The convolution can be found like this:
pdf = pdfs[0]
for i in range(1,pdfs.shape[0]):
pdf = np.convolve(pdfs[i], pdf)
The probabilities of seeing 0,1,...,8 are then given by
array([ 0.09 , 0.327, 0.342, 0.182, 0.052, 0.007, 0. , 0. , 0. ])
This part is the bottleneck in my code and it seems there must be something available to vectorize this operation. Does anyone have a suggestion for making it faster?
Alternatively, a solution where you could use
pdf1 = np.array([[0.6,0.3,0.1],[0.5,0.4,0.1]])
pdf2 = np.array([[0.3,0.7,0.0],[1.0,0.0,0.0]])
convolve(pd1,pd2)
and get the pairwise convolutions
array([[ 0.18, 0.51, 0.24, 0.07, 0. ],
[ 0.5, 0.4, 0.1, 0. , 0. ]])
would also help tremendously.

You can compute the convolution of all your PDFs efficiently using fast fourier transforms (FFTs): the key fact is that the FFT of the convolution is the product of the FFTs of the individual probability density functions. So transform each PDF, multiply the transformed PDFs together, and then perform the inverse transform. You'll need to pad each input PDF with zeros to the appropriate length to avoid effects from wraparound.
This should be reasonably efficient: if you have m PDFs, each containing n entries, then the time to compute the convolution using this method should grow as (m^2)n log(mn). The time is dominated by the FFTs, and we're effectively computing m + 1 independent FFTs (m forward transforms and one inverse transform), each of an array of length no greater than mn. But as always, if you want real timings you should profile.
Here's some code:
import numpy.fft
def convolve_many(arrays):
"""
Convolve a list of 1d float arrays together, using FFTs.
The arrays need not have the same length, but each array should
have length at least 1.
"""
result_length = 1 + sum((len(array) - 1) for array in arrays)
# Copy each array into a 2d array of the appropriate shape.
rows = numpy.zeros((len(arrays), result_length))
for i, array in enumerate(arrays):
rows[i, :len(array)] = array
# Transform, take the product, and do the inverse transform
# to get the convolution.
fft_of_rows = numpy.fft.fft(rows)
fft_of_convolution = fft_of_rows.prod(axis=0)
convolution = numpy.fft.ifft(fft_of_convolution)
# Assuming real inputs, the imaginary part of the output can
# be ignored.
return convolution.real
Applying this to your example, here's what I get:
>>> convolve_many([[0.6, 0.3, 0.1], [0.5, 0.4, 0.1], [0.3, 0.7], [1.0]])
array([ 0.09 , 0.327, 0.342, 0.182, 0.052, 0.007])
That's the basic idea. If you want to tweak this, you might also look at numpy.fft.rfft (and its inverse, numpy.fft.irfft), which take advantage of the fact that the input is real to produce more compact transformed arrays. You might also be able to gain some speed by padding the rows array with zeros so that the total number of columns is optimal for performing an FFT. The definition of "optimal" here would depend on the FFT implementation, but powers of two would be good targets, for example. Finally, there are some obvious simplifications that can be made when creating rows if all the input arrays have the same length. But I'll leave these potential enhancements to you.