We Keep Coding

Why numpy.fft.rfft/irfft transforms do not bring the input back?

I have a list, when I transform it by np.fft.rfft and bring it back by np.fft.irfft it does not work for ex(2) but work with ex(1). What should I do to make it work with ex(2)?
ex(1):
import NumPy as np
z=[[1,2,34,45],[1,2,5,6],[7,8,9,10]]
x1=np.fft.rfft(z)
x2=np.fft.irfft(x1)
print(x2)
print(z)
out:
[[ 1. 2. 34. 45.]
[ 1. 2. 5. 6.]
[ 7. 8. 9. 10.]]
[[1, 2, 34, 45], [1, 2, 5, 6], [7, 8, 9, 10]]
ex(2):
import NumPy as np
z1=[[5,8,6],[45,6,3],[847,5847,6]]
x3=np.fft.rfft(z1)
x4=np.fft.irfft(x3)
print(x4)
print(z1)
out:
[[ 8.5 10.5 ]
[ 47.25 6.75]
[2310.25 4389.75]]
[[5, 8, 6], [45, 6, 3], [847, 5847, 6]]
Please help.

The isn't an error but the intended behaviour of np.rfft for input of odd length:
The truncated or zero-padded input, transformed along the axis
indicated by axis, or the last one if axis is not specified. If n is
even, the length of the transformed axis is (n/2)+1. If n is odd, the
length is (n+1)/2.
This is a consequence of Nyquist-Shannon sampling theorem.
In order to solve this, you can simply add a zero at the end of every rows of z1 if there is an odd number of columns (i.e. zero-padding) by specifying an appropriate n kwarg in np.rfft call which gives:
import numpy as np
z1 = np.array([[5,8,6],[45,6,3],[847,5847,6]])
n = z1.shape[1]
if n%2:
# zero padding if n odd
n += 1
x3 = np.fft.rfft(z1,n,axis=-1)
x4 = np.fft.irfft(x3)
which gives the initial input:
print(x4)
>>>[[5.000e+00 8.000e+00 6.000e+00 0.000e+00]
[4.500e+01 6.000e+00 3.000e+00 0.000e+00]
[8.470e+02 5.847e+03 6.000e+00 0.000e+00]]
print(z1)
>>>[[ 5 8 6]
[ 45 6 3]
[ 847 5847 6]]
Feel free to discard the last column of zeros of x4 after going back from the frequency domain.

Getting a column index in numpy

I'm pretty new to NumPy and I'm looking for a way to get the index of a current column I'm iterating over in a matrix.
import numpy as np
#sum of elements in each column
def p_b(mtrx):
b = []
for c in mtrx.T:
summ = 0
for i in c:
summ += i
b.append(summ)
return b
#return modified matrix where each element is equal to itself divided by
#the sum of the current column in the original matrix
def a_div_b(mtrx):
for c in mtrx:
for i in c:
#change i to be i/p_b(mtrx)[index_of_a_current_column]
return mtrx
For the input ([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) the result would be
([[1/12, 2/12, 3/12], [4/15, 5/15, 6/15], [7/18, 8/18, 9/18]]).
Any ideas about how I can achieve that?

You don't need those functions and loops to do that. Those will not be efficient. When using numpy, go for vectorized operations whenever is possible (in most cases it is possible). numpy broadcasting rules are used to perform mathematical operation between arrays of different dimensions, when possible, such that you can use vectorization, which is much more efficient than python loops.
In your case, say that your array arr is:
arr = np.arange(1, 10)
arr.shape = (3, 3)
#arr is:
>>> arr
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
you can achieve the desired result with:
res = (arr.T / arr.sum(axis=0)).T
>>> res
array([[0.08333333, 0.16666667, 0.25 ],
[0.26666667, 0.33333333, 0.4 ],
[0.38888889, 0.44444444, 0.5 ]])
numpy sum allows you to sum your array along a given axis if the axis parameter is given. 0 is the inner axis, the one you want to sum.
.T gives the transposed matrix. You need to transpose to perform the division on the correct axis and then transpose back.

tensorflow: how to interleave columns of two tensors (e.g. using tf.scatter_nd)?

I've read the tf.scatter_nd documentation and run the example code for 1D and 3D tensors... and now I'm trying to do it for a 2D tensor. I want to 'interleave' the columns of two tensors. For 1D tensors, one can do this via
'''
We want to interleave elements of 1D tensors arr1 and arr2, where
arr1 = [10, 11, 12]
arr2 = [1, 2, 3, 4, 5, 6]
such that
desired result = [1, 2, 10, 3, 4, 11, 5, 6, 12]
'''
import tensorflow as tf
with tf.Session() as sess:
updates1 = tf.constant([1,2,3,4,5,6])
indices1 = tf.constant([[0], [1], [3], [4], [6], [7]])
shape = tf.constant([9])
scatter1 = tf.scatter_nd(indices1, updates1, shape)
updates2 = tf.constant([10,11,12])
indices2 = tf.constant([[2], [5], [8]])
scatter2 = tf.scatter_nd(indices2, updates2, shape)
result = scatter1 + scatter2
print(sess.run(result))
(aside: is there a better way to do this? I'm all ears.)
This gives the output
[ 1 2 10 3 4 11 5 6 12]
Yay! that worked!
Now lets' try to extend this to 2D.
'''
We want to interleave the *columns* (not rows; rows would be easy!) of
arr1 = [[1,2,3,4,5,6],[1,2,3,4,5,6],[1,2,3,4,5,6]]
arr2 = [[10 11 12], [10 11 12], [10 11 12]]
such that
desired result = [[1,2,10,3,4,11,5,6,12],[1,2,10,3,4,11,5,6,12],[1,2,10,3,4,11,5,6,12]]
'''
updates1 = tf.constant([[1,2,3,4,5,6],[1,2,3,4,5,6],[1,2,3,4,5,6]])
indices1 = tf.constant([[0], [1], [3], [4], [6], [7]])
shape = tf.constant([3, 9])
scatter1 = tf.scatter_nd(indices1, updates1, shape)
This gives the error
ValueError: The outer 1 dimensions of indices.shape=[6,1] must match the outer 1
dimensions of updates.shape=[3,6]: Dimension 0 in both shapes must be equal, but
are 6 and 3. Shapes are [6] and [3]. for 'ScatterNd_2' (op: 'ScatterNd') with
input shapes: [6,1], [3,6], [2].
Seems like my indices is specifying row indices instead of column indices, and given the way that arrays are "connected" in numpy and tensorflow (i.e. row-major order), does that mean
I need to explicitly specify every single pair of indices for every element in updates1?
Or is there some kind of 'wildcard' specification I can use for the rows? (Note indices1 = tf.constant([[:,0], [:,1], [:,3], [:,4], [:,6], [:,7]]) gives syntax errors, as it probably should.)
Would it be easier to just do a transpose, interleave the rows, then transpose back?
Because I tried that...
scatter1 = tf.scatter_nd(indices1, tf.transpose(updates1), tf.transpose(shape))
print(sess.run(tf.transpose(scatter1)))
...and got a much longer error message, that I don't feel like posting unless someone requests it.
PS- I searched to make sure this isn't a duplicate -- I find it hard to imagine that someone else hasn't asked this before -- but turned up nothing.

This is pure slicing but I didn't know that syntax like arr1[0:,:][:,:2] actually works. It seems it does but not sure if it is better.
This may be the wildcard slicing mechanism you are looking for.
arr1 = tf.constant([[1,2,3,4,5,6],[1,2,3,4,5,7],[1,2,3,4,5,8]])
arr2 = tf.constant([[10, 11, 12], [10, 11, 12], [10, 11, 12]])
with tf.Session() as sess :
sess.run( tf.global_variables_initializer() )
print(sess.run(tf.concat([arr1[0:,:][:,:2], arr2[0:,:] [:,:1],
arr1[0:,:][:,2:4],arr2[0:, :][:, 1:2],
arr1[0:,:][:,4:6],arr2[0:, :][:, 2:3]],axis=1)))
Output is
[[ 1 2 10 3 4 11 5 6 12]
[ 1 2 10 3 4 11 5 7 12]
[ 1 2 10 3 4 11 5 8 12]]
So, for example,
arr1[0:,:] returns
[[1 2 3 4 5 6]
[1 2 3 4 5 7]
[1 2 3 4 5 8]]
and arr1[0:,:][:,:2] returns the first two columns
[[1 2]
[1 2]
[1 2]]
axis is 1.

Some moderators might have regarded my question as a duplicate of this one, not because the questions are the same, but only because the answers contain parts one can use to answer this question -- i.e. specifying every index combination by hand.
A totally different method would be to multiply by a permutation matrix as shown in the last answer to this question. Since my original question was about scatter_nd, I'm going to post this solution but wait to see what other answers come in... (Alternatively, I or someone could edit the question to make it about reordering columns, not specific to scatter_nd --EDIT: I have just edited the question title to reflect this).
Here, we concatenate the two different arrays/tensors...
import numpy as np
import tensorflow as tf
sess = tf.Session()
# the ultimate application is for merging variables which should be in groups,
# e.g. in this example, [1,2,10] is a group of 3, and there are 3 groups of 3
n_groups = 3
vars_per_group = 3 # once the single value from arr2 (below) is included
arr1 = 10+tf.range(n_groups, dtype=float)
arr1 = tf.stack((arr1,arr1,arr1),0)
arr2 = 1+tf.range(n_groups * (vars_per_group-1), dtype=float)
arr2 = tf.stack((arr2,arr2,arr2),0)
catted = tf.concat((arr1,arr2),1) # concatenate the two arrays together
print("arr1 = \n",sess.run(arr1))
print("arr2 = \n",sess.run(arr2))
print("catted = \n",sess.run(catted))
Which gives output
arr1 =
[[10. 11. 12.]
[10. 11. 12.]
[10. 11. 12.]]
arr2 =
[[1. 2. 3. 4. 5. 6.]
[1. 2. 3. 4. 5. 6.]
[1. 2. 3. 4. 5. 6.]]
catted =
[[10. 11. 12. 1. 2. 3. 4. 5. 6.]
[10. 11. 12. 1. 2. 3. 4. 5. 6.]
[10. 11. 12. 1. 2. 3. 4. 5. 6.]]
Now we build the permutation matrix and multiply...
start_index = 2 # location of where the interleaving begins
# cml = "column map list" is the list of where each column will get mapped to
cml = [start_index + x*(vars_per_group) for x in range(n_groups)] # first array
for i in range(n_groups): # second array
cml += [x + i*(vars_per_group) for x in range(start_index)] # vars before start_index
cml += [1 + x + i*(vars_per_group) + start_index \
for x in range(vars_per_group-start_index-1)] # vars after start_index
print("\n cml = ",cml,"\n")
# Create a permutation matrix using p
np_perm_mat = np.zeros((len(cml), len(cml)))
for idx, i in enumerate(cml):
np_perm_mat[idx, i] = 1
perm_mat = tf.constant(np_perm_mat,dtype=float)
result = tf.matmul(catted, perm_mat)
print("result = \n",sess.run(result))
Which gives output
cml = [2, 5, 8, 0, 1, 3, 4, 6, 7]
result =
[[ 1. 2. 10. 3. 4. 11. 5. 6. 12.]
[ 1. 2. 10. 3. 4. 11. 5. 6. 12.]
[ 1. 2. 10. 3. 4. 11. 5. 6. 12.]]
Even though this doesn't use scatter_nd as the original question asked, one thing I like about this is, you can allocate the perm_mat once in some __init__() method, and hang on to it, and after that initial overhead it's just matrix-matrix multiplication by a sparse, constant matrix, which should be pretty fast. (?)
Still happy to wait and see what other answers might come in.

Vectorized form to get count of elements greater than a reference

...and that reference comes from a separate matrix.
This question is an extension of an earlier answered question where the reference element came directly from the same column it was being compared against. Some clever sorting and referencing the index of the sort seemed to solve that one.
Broadcasting has been suggested in both the original and this new question. I run out of memory at around n ~ 3000 and need another order of magnitude larger yet.
The Target ( Production-grade ) Scaling Definitions:
So as to let proposed solutions' approaches fair and mutually comparable, both in the [SPACE]- and the [TIME]-domains,
let's assume n = 50000; m = 20; k = 50; a = np.random.rand( n, m ); ...
I'm now interested in a more general form where the reference value comes from another matrix of reference values.
Original question:
Vectorized pythonic way to get count of elements greater than current element
New question: Can we write a vectorized form to perform the following role.
Function receives as input 2 2-d arrays.
A = n x m
B = k x m
and returns
C = k x m.
C[i,j] is the proportion of observations in A[:,j] ( just the j-th column ) that are larger than B[i,j]
Here is my embarrasingly slow double for loop implementation.
import numpy as np
n = 100
m = 20
k = 50
a = np.random.rand(n,m)
b = np.random.rand(k,m)
c = np.zeros((k,m))
for j in range(0,m): #cols
for i in range(0,k): # rows
r = b[i,j]
c[i,j] = ( ( a[:,j] > r).sum() ) / (n)

Approach #1
We could again use the argsort trick as discussed in this solution but in a bit twisted manner. We would concatenate the second array into the first array and then perform argsort-ing. We need to use argsort for both the concatenated array and the second one and have our desired output. The implementation would look something like this -
ab = np.vstack((a,b))
len_a, len_b = len(a), len(b)
b_argoffset = b.argsort(0).argsort(0)
total_args = ab.argsort(0).argsort(0)[-len_b:]
out = len_a - total_args + b_argoffset
Explanation
Concatenate second array whose values are to be computed into the first array.
Now, since we are appending, we would have their index positions later on, after the first array length has ended.
We use one argsort to get the relative positions of the second array w.r.t to the entire concatenated array and one more argsort to trace back those indices w.r.t the original order.
We need to repeat the double argsort-ing for the second array on itself, so as to compensate for the concatenation.
These indices are for each element in b with the comparison : a[:,j] > b[i,j]. Now, these indices orders are 0-based, i.e. an index closer to 0 represent greater number of elements in a[:,j] than the current element b[i,j], so a greater count and vice versa. So, we need to subtract those indices from the length of a[:,j] for the final output.
Approach #1 - Improvement
We would optimize it further by using array-assignment, again inspired by Approach #2 from the same solution. So, those arg outputs : b_argoffset and total_args could be alternatively computed, like so -
def unqargsort(a):
n,m = a.shape
idx = a.argsort(0)
out = np.zeros((n,m),dtype=int)
out[idx, np.arange(m)] = np.arange(n)[:,None]
return out
b_argoffset = unqargsort(b)
total_args = unqargsort(ab)[-len_b:]
Approach #2
We could also leverage searchsorted for an altogether different approach -
k,m = b.shape
sidx = a.argsort(0)
out = np.empty((k,m), dtype=int)
for i in range(m): #cols
out[:,i] = np.searchsorted(a[:,i], b[:,i],sorter=sidx[:,i])
out = len(a) - out
Explanation
We get the sorted order indices for each column of a.
Then, use those indices to get how we could place values off b into the sorted a with searcshorted. This gives us same as the output from step#3,4 in Approach#1.
Note that these approaches give us the count. So, for the final output, divide the output thus obtained by n.

I think you can use broadcasting:
c = (a[:,None,:] > b).mean(axis=0)
Demo:
In [207]: n = 5
In [208]: m = 3
In [209]: a = np.random.randint(10, size=(n,m))
In [210]: b = np.random.randint(10, size=(n,m))
In [211]: c = np.zeros((n,m))
In [212]: a
Out[212]:
array([[2, 2, 8],
[5, 0, 8],
[2, 5, 7],
[4, 4, 4],
[2, 6, 7]])
In [213]: b
Out[213]:
array([[3, 6, 8],
[2, 7, 5],
[8, 9, 2],
[9, 8, 7],
[2, 7, 2]])
In [214]: for j in range(0,m): #cols
...: for i in range(0,n): # rows
...: r = b[i,j]
...: c[i,j] = ( ( a[:,j] > r).sum() ) / (n)
...:
...:
In [215]: c
Out[215]:
array([[0.4, 0. , 0. ],
[0.4, 0. , 0.8],
[0. , 0. , 1. ],
[0. , 0. , 0.4],
[0.4, 0. , 1. ]])
In [216]: (a[:,None,:] > b).mean(axis=0)
Out[216]:
array([[0.4, 0. , 0. ],
[0.4, 0. , 0.8],
[0. , 0. , 1. ],
[0. , 0. , 0.4],
[0.4, 0. , 1. ]])
check:
In [217]: ((a[:,None,:] > b).mean(axis=0) == c).all()
Out[217]: True

Efficient Numpy computation of pairwise squared differences

The following code does exactly what I want, which is to compute the pairwise sum of squares of differences between elements of a vector (length three in the example), of which I have a long series (limited to five here). The desired result is shown at the bottom.
But the implementation feels kludgy for two reasons:
1) the need to add a phantom dimension, changing the shape from (5, 3) to (5,1,3) to avoid broadcast problems, and
2) the apparent necessity of an explicit 'for' loop, which I'm sure is why it's taking hours to execute on my much larger data set (a million vectors of length 2904).
Is there a more efficient and/or pythonic way to achieve the same result?
a = np.array([[ 4, 2, 3], [-1, -5, 4], [ 2, 1, 4], [-5, -1, 4], [6, -3, 3]])
a = a.reshape((5,1,3))
m = a.shape[0]
n = a.shape[2]
d = np.zeros((n,n))
for i in range(m):
c = a[i,:] - np.transpose(a[i,:])
c = c**2
d += c
print d
[[ 0. 118. 120.]
[ 118. 0. 152.]
[ 120. 152. 0.]]

If you don't mind the dependency on scipy, you can use functions from the scipy.spatial.distance library:
In [17]: from scipy.spatial.distance import pdist, squareform
In [18]: a = np.array([[ 4, 2, 3], [-1, -5, 4], [ 2, 1, 4], [-5, -1, 4], [6, -3, 3]])
In [19]: d = pdist(a.T, metric='sqeuclidean')
In [20]: d
Out[20]: array([ 118., 120., 152.])
In [21]: squareform(d)
Out[21]:
array([[ 0., 118., 120.],
[ 118., 0., 152.],
[ 120., 152., 0.]])

You could eliminate the for-loop by using:
In [48]: ((a - a.swapaxes(1,2))**2).sum(axis=0)
Out[48]:
array([[ 0, 118, 120],
[118, 0, 152],
[120, 152, 0]])
Note that if a has shape (N, 1, M) then (a - a.swapaxes(1,2)) has shape (N, M, M). Make sure you have enough RAM to accommodate an array of this size. Page swapping can also slow the calculation to a crawl.
If you do have too little memory, you will have to break up the calculation in chunks:
m, _, n = a.shape
chunksize = 10**4
d = np.zeros((n,n))
for i in range(0, m, chunksize):
b = a[i:i+chunksize]
d += ((b - b.swapaxes(1,2))**2).sum(axis=0)
This is a compromise between performing the calculation on the entire array and
calculating row-by-row. If there are a million rows, and the chunksize is 10**4, then there will be only 100 iterations of the loop instead of a million.
Thus, it should be significantly faster than calculating row-by-row. Choose the largest value of chunksize you can which allows the calculation to be performed in RAM.