We Keep Coding

Extract upper diagonal of a block matrix in numpy - like np.triu_idx, but for block matrices

I want to extract off-block-diagonal elements from a block-diagonal matrix, i.e.
import numpy as np
import scipy as sp
A = np.array([
[1, 1, 1],
[1, 1, 1],
[1, 1, 1]
])
B = np.array([
[2, 2],
[2, 2]
])
C = np.array([
[3]
])
D = sp.linalg.block_diag(A, B, C)
print(D)
>>> array([[1, 1, 1, 0, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 0, 0, 0],
[0, 0, 0, 2, 2, 0],
[0, 0, 0, 2, 2, 0],
[0, 0, 0, 0, 0, 3]])
So, I need to extract the elements over the diagonal that do not belong to blocks, i.e. that ones which are zeros in D.
How to achieve that?

A straightforwad solution based on loops, can one make it better avoiding loops?
Taking upper triangle and then taking non-zero values is not what I want as it will not allow me to get indices for the original block matrix D, but only for it's upper triangle.
def block_triu_indices(block_sizes=None):
n = np.sum(block_sizes)
blocks = []
for block_size in block_sizes:
blocks.append(np.ones((block_size, block_size)))
A = sp.linalg.block_diag(*blocks)
row_idx = []
col_idx = []
for i in range(n):
for j in range(i+1, n):
if A[i,j]==0:
row_idx.append(i)
col_idx.append(j)
return (np.array(row_idx), np.array(col_idx))
block_triu_idx = block_triu_indices([3, 2, 1])
print(block_triu_idx)
>>> (array([0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 4]),
array([3, 4, 5, 3, 4, 5, 3, 4, 5, 5, 5]))
print(D[block_triu_idx])
>>> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

How to change all values to the left of a particular cell in every row

I have an array which contains 1's and 0's. A very small section of it looks like this:
arr=[[0,0,0,0,1],
[0,0,1,0,0],
[0,1,0,0,0],
[1,0,1,0,0]]
I want to change the value of every cell to 1, if it is to the left of a cell with a value of 1. I want all other cells to keep their value of 0, i.e:
arrOut=[[1,1,1,1,1],
[1,1,1,0,0],
[1,1,0,0,0]
[1,1,1,0,0]
Some rows have >1 cell with a value =1.
I have managed to do this using a very ugly double for-loop:
for i in range(len(arr)):
for j in range(len(arr[i])):
if arr[i][j]==1:
arrOut[i][0:j]=1
Does anyone know of another way to do this with using for loops? I'm relatively comfortable with numpy and pandas, but also open to other libraries.
Thanks!

You can flip using it, and use np.cumsum:
>>> arr[:, ::-1].cumsum(axis=1)[:, ::-1]
array([[1, 1, 1, 1, 1],
[1, 1, 1, 0, 0],
[1, 1, 0, 0, 0]], dtype=int32)
Or the same using np.fliplr,
>>> np.fliplr(np.fliplr(arr).cumsum(axis=1))
array([[1, 1, 1, 1, 1],
[1, 1, 1, 0, 0],
[1, 1, 0, 0, 0]], dtype=int32)
Using np.where:
>>> np.where(arr.cumsum(1)==0, 1, arr)
array([[1, 1, 1, 1, 1],
[1, 1, 1, 0, 0],
[1, 1, 0, 0, 0]], dtype=int32)
If array has more than one 1, use np.clip:
>>> arr
array([[0, 0, 0, 0, 1],
[0, 0, 1, 0, 0],
[0, 1, 0, 1, 0]])
>>> np.clip(arr[:, ::-1].cumsum(axis=1)[:, ::-1], 0, 1)
array([[1, 1, 1, 1, 1],
[1, 1, 1, 0, 0],
[1, 1, 1, 1, 0]], dtype=int32)
# If you want to make all 0s before the leftmost 1 to 1:
>>> np.where(arr.cumsum(1)==0, 1, arr)
array([[1, 1, 1, 1, 1],
[1, 1, 1, 0, 0],
[1, 1, 0, 1, 0]])

A possibility without using libraries:
for subarray in arr:
#Get the index of the 1 starting from behind so that if there are 2 1s,
#you get the index of the rightmost one
indexof1 = len(subarray) -1 - subarray[::-1].index(1)
#Until the 1, replace with 1s
subarray[:indexof1] = [1]*len(subarray[:indexof1])

Another solution using np.maximum.accumulate:
np.maximum.accumulate(arr[:,::-1],axis=1)[:,::-1]
Where np.maximum.accumulate simply apply a cumulative maximum (cummax).

Count indices to array to produce heatmap

I'd like to accumulate indices that point to a m-by-n array to another array of that very shape to produce a heatmap. For example, these indices:
[
[0, 1, 2, 0, 1, 2]
[0, 1, 0, 0, 0, 2]
]
would produce the following array:
[
[2, 0, 0]
[1, 1, 0]
[1, 0, 1]
]
I've managed to succesfully implement an algorithm, but I started wondering, whether there is already a built-in NumPy solution for this kind of problem.
Here's my code:
a = np.array([[0, 1, 2, 0, 1, 2], [0, 1, 0, 0, 0, 2]])
def _gather_indices(indices: np.ndarray, shape: tuple):
heat = np.zeros(shape)
for i in range(indices.shape[-1]):
heat[tuple(indices[:, i])] += 1

Two methods could be suggested.
With np.add.at -
heat = np.zeros(shape,dtype=int)
np.add.at(heat,(a[0],a[1]),1)
Or with tuple() based one for a more aesthetic one -
np.add.at(heat,tuple(a),1)
With bincount -
idx = np.ravel_multi_index(a,shape)
np.bincount(idx,minlength=np.prod(shape)).reshape(shape)
Additionally, we could compute shape using the max-limits of the indices in a -
shape = a.max(axis=1)+1
Sample run -
In [147]: a
Out[147]:
array([[0, 1, 2, 0, 1, 2],
[0, 1, 0, 0, 0, 2]])
In [148]: shape = (3,3)
In [149]: heat = np.zeros(shape,dtype=int)
...: np.add.at(heat,(a[0],a[1]),1)
In [151]: heat
Out[151]:
array([[2, 0, 0],
[1, 1, 0],
[1, 0, 1]])
In [173]: idx = np.ravel_multi_index(a,shape)
In [174]: np.bincount(idx,minlength=np.prod(shape)).reshape(shape)
Out[174]:
array([[2, 0, 0],
[1, 1, 0],
[1, 0, 1]])

Most efficient way to get sorted indices based on two numpy arrays

How can i get the sorted indices of a numpy array (distance), only considering certain indices from another numpy array (val).
For example, consider the two numpy arrays val and distance below:
val = np.array([[10, 0, 0, 0, 0],
[0, 0, 10, 0, 10],
[0, 10, 10, 0, 0],
[0, 0, 0, 10, 0],
[0, 0, 0, 0, 0]])
distance = np.array([[4, 3, 2, 3, 4],
[3, 2, 1, 2, 3],
[2, 1, 0, 1, 2],
[3, 2, 1, 2, 3],
[4, 3, 2, 3, 4]])
the distances where val == 10 are 4, 1, 3, 1, 0, 2. I would like to get these sorted to be 0, 1, 1, 2, 3, 4 and return the respective indices from distance array.
Returning something like:
(array([2, 1, 2, 3, 1, 0], dtype=int64), array([2, 2, 1, 3, 4, 0], dtype=int64))
or:
(array([2, 2, 1, 3, 1, 0], dtype=int64), array([2, 1, 2, 3, 4, 0], dtype=int64))
since the second and third element both have distance '1', so i guess the indices can be interchangable.
Tried using combinations of np.where, np.argsort, np.argpartition, np.unravel_index but cant seem to get it working right

Here's one way with masking -
In [20]: mask = val==10
In [21]: np.argwhere(mask)[distance[mask].argsort()]
Out[21]:
array([[2, 2],
[1, 2],
[2, 1],
[3, 3],
[1, 4],
[0, 0]])

Using np.tile to tile 10 images of each image in a batch of images

Take the array: arr = [0, 1, 2]
np.tile(arr,[10,1])
array([[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2],
[0, 1, 2]])
>>> np.tile(arr,[10,2])
array([[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2],
[0, 1, 2, 0, 1, 2]])
Similar to this, I want to use the tile function to create 10 copies of an image batch of size 10x227x227x3 (the batch already has 10 images)) For each image I want to create a tile. So I should get 100x227x227x3
However when I do this M=10):
images = np.tile(img_batch, [M, 1])
I get 10x227x2270x3 instead, images = np.tile(img_batch, [M]) doesn't work either and brings a value of size 10x227x227x30
I can't get around my head on how to get the tiles I need. Any recommendations are welcome.

Your img_batch has 4 dimensions. Make the reps of size 4:
np.tile(img_batch, [M, 1, 1, 1])
Otherwise, it will be equivalent to np.tile(img_batch, [1, 1, M, 1] in your first case according to the docs:
If A.ndim > d, reps is promoted to A.ndim by pre-pending 1’s to it.
Thus for an A of shape (2, 3, 4, 5), a reps of (2, 2) is treated as
(1, 1, 2, 2).