We Keep Coding

Is there a function to add a buffer of (NaN) values around a NumPy ndarray?

I am working with a thematic raster of land use classes. The goal is to split the raster into smaller tiles of a given size. For example, I have a raster of 1490 pixels and I want to split it into tiles of 250x250 pixels. To get tiles of equal size, I would want to increase the width of the raster to 1500 pixels to fit in exactly 6 tiles. To do so, I need to increase the size of the raster by 10 pixels.
I am currently opening the raster with the rasterio library, which returns a NumPy ndarray. Is there a function to add a buffer around this array? The goal would be something like this:
import numpy as np
a = np.array([
[1,4,5],
[4,5,5],
[1,2,2]
])
a_with_buffer = a.buffer(a, 1) # 2nd argument refers to the buffer size
Then a_with_buffer would look as following:
[0,0,0,0,0]
[0,1,4,5,0],
[0,4,5,5,0],
[0,1,2,2,0],
[0,0,0,0,0]

You can use np.pad:
>>> np.pad(a, 1)
array([[0, 0, 0, 0, 0],
[0, 1, 4, 5, 0],
[0, 4, 5, 5, 0],
[0, 1, 2, 2, 0],
[0, 0, 0, 0, 0]])

you can create np.zeros then insert a in the index what you want like below.
Try this:
>>> a = np.array([[1,4,5],[4,5,5],[1,2,2]])
>>> b = np.zeros((5,5))
>>> b[1:1+a.shape[0],1:1+a.shape[1]] = a
>>> b
array([[0., 0., 0., 0., 0.],
[0., 1., 4., 5., 0.],
[0., 4., 5., 5., 0.],
[0., 1., 2., 2., 0.],
[0., 0., 0., 0., 0.]])

Numpy add smaller matrix to a bigger one

I have big 3D matrices indicating the position of agents in a 3D space. The values of the matrix are 0 if there is not agent on it and 1 if there is an agent on it.
Then, my problem is that I want the agents to 'grow' in the sense that I want them to be determined by lets say a cube (3x3x3) of ones. If already gotten a way to do it but I'm having trouble when the agent is close to the borders.
For example, I have a matrix of positions 100x100x100, if I know my agent is at position (x, y, z) I will do:
positions_matrix = numpy.zeros((100, 100, 100))
positions_matrix[x - 1: x + 2, y - 1: y + 2, z - 1: z + 2] += numpy.ones((3, 3, 3))
Of course in my real code I'm looping over more positions but this is basically it. This works but the problem comes when the agent is to close to the border in which the sum can't be made because the resultant matrix from slicing would be smaller than the ones matrix.
Any idea how to solve it or if numpy or any other package have an implementation for this? I couldn't manage to find it although I'm pretty sure I'm not the first one to face against this.

A slightly more programmatic way of solving the problem:
import numpy as np
m = np.zeros((100, 100, 100))
slicing = tuple(
slice(max(0, x_i - 1), min(x_i + 2, d - 1))
for x_i, d in zip((x, y, z), m.shape))
ones_shape = tuple(s.stop - s.start for s in slicing)
m[slicing] += np.ones(ones_shape)
But it is otherwise the same as the accepted answer.

You should cut at the lower and upper bounds, using something like:
import numpy as np
m = np.zeros((100, 100, 100))
x_min, x_max = np.max([0, x-1]), np.min([x+2, m.shape[0]-1])
y_min, y_max = np.max([0, y-1]), np.min([y+2, m.shape[1]-1])
z_min, z_max = np.max([0, z-1]), np.min([z+2, m.shape[2]-1])
m[x_min:x_max, y_min:y_max, z_min:z_max] += np.ones((x_max-x_min, y_max-y_min, z_max-z_min))

There is a solution using np.put, and its 'clip' option.
It just requires a little gymnastics because the function requires indices in the flattened matrix; fortunately, the function np.ravel_multi_index does the job:
import itertools
import numpy as np
x, y, z = 2, 0, 4
positions_matrix = np.zeros((100,100,100))
indices = np.array( list( itertools.product( (x-1, x, x+1), (y-1, y, y+1), (z-1, z, z+1)) ))
flat_indices = np.ravel_multi_index(indices.T, positions_matrix.shape, mode='clip')
positions_matrix.put(flat_indices, 1+positions_matrix.take(flat_indices))
# positions_matrix[2,1,4] is now 1.0
The nice thing about this solution is that you can play with other modes, for instance 'wrap' (if your agents live on a donut ;-) or in a periodic space).
I'll explain how it works on a smaller 2D matrix:
import itertools
import numpy as np
positions_matrix = np.zeros((8,8))
ones = np.ones((3,3))
x, y = 0, 4
indices = np.array( list( itertools.product( (x-1, x, x+1), (y-1, y, y+1) )))
# array([[-1, 3],
# [-1, 4],
# [-1, 5],
# [ 0, 3],
# [ 0, 4],
# [ 0, 5],
# [ 1, 3],
# [ 1, 4],
# [ 1, 5]])
flat_indices = np.ravel_multi_index(indices.T, positions_matrix.shape, mode='clip')
# array([ 3, 4, 5, 3, 4, 5, 11, 12, 13])
positions_matrix.put(flat_indices, ones, mode='clip')
# positions_matrix is now:
# array([[0., 0., 0., 1., 1., 1., 0., 0.],
# [0., 0., 0., 1., 1., 1., 0., 0.],
# [0., 0., 0., 0., 0., 0., 0., 0.],
# [ ...
By the way, in this case mode='clip' was redundant for put.
Well, I just cheated put does an assignment. The +=1 requires both take and put:
positions_matrix.put(flat_indices, ones.flat + positions_matrix.take(flat_indices))
# notice that ones has to be flattened, or alternatively the result of take could be reshaped (3,3)
# positions_matrix is now:
# array([[0., 0., 0., 2., 2., 2., 0., 0.],
# [0., 0., 0., 2., 2., 2., 0., 0.],
# [0., 0., 0., 0., 0., 0., 0., 0.],
# [ ...
There is one important difference in this solution compared to the others: the ones matrix is always (3,3),
which may or may not be an advantage.
The trick is in this flat_indices list, that has repeating entries (result of clip).
It may thus require some precautions, if you add a non constant sub-matrix at max indices:
x, y = 1, 7
values = 1 + np.arange(9)
indices = np.array( list( itertools.product( (x-1, x, x+1), (y-1, y, y+1) )))
flat_indices = np.ravel_multi_index(indices.T, positions_matrix.shape, mode='clip')
positions_matrix.put(flat_indices, values, mode='clip')
# positions_matrix is now:
# array([[0., 0., 0., 2., 2., 2., 1., 3.],
# [0., 0., 0., 2., 2., 2., 4., 6.],
# [0., 0., 0., 0., 0., 0., 7., 9.],
... you were probably expecting the last column to be 2 5 8.
Currently, you could work on flat_indices, for example by putting -1 in the out-of-bounds locations.
But it'd all be easier if np.put accepted non-flat indices, or if there was a clip mode='ignore'.

numpy arange for list (vectorized calculation)

I want to create a 2d matrix b from an array a, where a contains range_stop values for each matrix column.
For example, with a = [2,3], I want to obtain
b = [[0, 0],
[1, 1],
[2, 2],
[NaN, 3]]
What's the most efficient way (for vectorized calculation) to do it? My current code is:
a = [2,3]
b = np.zeros((max(a)+1,len(a)))
b.fill(np.nan)
for i,ai in enumerate(a):
b[:ai, i] = np.arange(ai)

You can first create the 2D arange using repeat
a = np.asarray([2, 3])
b = np.repeat(np.arange(np.max(a) + 1, dtype=float)[:, None], len(a), axis=1)
# array([[0., 0.],
# [1., 1.],
# [2., 2.],
# [3., 3.]])
and then compare each column with a to fill in np.nans
b[b > a] = np.nan
# array([[ 0., 0.],
# [ 1., 1.],
# [ 2., 2.],
# [nan, 3.]])

Convolving a periodic image with python

I want to convolve an n-dimensional image which is conceptually periodic.
What I mean is the following: if I have a 2D image
>>> image2d = [[0,0,0,0],
... [0,0,0,1],
... [0,0,0,0]]
and I want to convolve it with this kernel:
>>> kernel = [[ 1,1,1],
... [ 1,1,1],
... [ 1,1,1]]
then I want the result to be:
>>> result = [[1,0,1,1],
... [1,0,1,1],
... [1,0,1,1]]
How to do this in python/numpy/scipy?
Note that I am not interested in creating the kernel, but mainly the periodicity of the convolution, i.e. the three leftmost ones in the resulting image (if that makes sense).

This is already built in, with scipy.signal.convolve2d's optional boundary='wrap' which gives periodic boundary conditions as padding for the convolution. The mode option here is 'same' to make the output size match the input size.
In [1]: image2d = [[0,0,0,0],
... [0,0,0,1],
... [0,0,0,0]]
In [2]: kernel = [[ 1,1,1],
... [ 1,1,1],
... [ 1,1,1]]
In [3]: from scipy.signal import convolve2d
In [4]: convolve2d(image2d, kernel, mode='same', boundary='wrap')
Out[4]:
array([[1, 0, 1, 1],
[1, 0, 1, 1],
[1, 0, 1, 1]])
The only disadvantage here is that you cannot use scipy.signal.fftconvolve which is often faster.

image2d = [[0,0,0,0,0],
[0,0,0,1,0],
[0,0,0,0,0],
[0,0,0,0,0]]
kernel = [[1,1,1],
[1,1,1],
[1,1,1]]
image2d = np.asarray(image2d)
kernel = np.asarray(kernel)
img_f = np.fft.fft2(image2d)
krn_f = np.fft.fft2(kernel, s=image2d.shape)
conv = np.fft.ifft2(img_f*krn_f).real
>>> conv.round()
array([[ 0., 0., 0., 0., 0.],
[ 1., 0., 0., 1., 1.],
[ 1., 0., 0., 1., 1.],
[ 1., 0., 0., 1., 1.]])
Note that the kernel is placed with its top left corner at the position of the 1 in the image. You would need to roll the result to get what you are after:
k_rows, k_cols = kernel.shape
conv2 = np.roll(np.roll(conv, -(k_cols//2), axis=-1),
-(k_rows//2), axis=-2)
>>> conv2.round()
array([[ 0., 0., 1., 1., 1.],
[ 0., 0., 1., 1., 1.],
[ 0., 0., 1., 1., 1.],
[ 0., 0., 0., 0., 0.]])

This kind of 'periodic convolution' is better known as circular or cyclic convolution. See http://en.wikipedia.org/wiki/Circular_convolution.
In the case of an n-dimensional image, as is the case in this question, one can use the scipy.ndimage.convolve function. It has a parameter mode which can be set to wrap for a circular convolution.
result = scipy.ndimage.convolve(image,kernel,mode='wrap')
>>> import numpy as np
>>> image = np.array([[0, 0, 0, 0],
... [0, 0, 0, 1],
... [0, 0, 0, 0]])
>>> kernel = np.array([[1, 1, 1],
... [1, 1, 1],
... [1, 1, 1]])
>>> from scipy.ndimage import convolve
>>> convolve(image, kernel, mode='wrap')
array([[1, 0, 1, 1],
[1, 0, 1, 1],
[1, 0, 1, 1]])

Map arrays with duplicate indexes?

Assume three arrays in numpy:
a = np.zeros(5)
b = np.array([3,3,3,0,0])
c = np.array([1,5,10,50,100])
b can now be used as an index for a and c. For example:
In [142]: c[b]
Out[142]: array([50, 50, 50, 1, 1])
Is there any way to add up the values connected to the duplicate indexes with this kind of slicing? With
a[b] = c
Only the last values are stored:
array([ 100., 0., 0., 10., 0.])
I would like something like this:
a[b] += c
which would give
array([ 150., 0., 0., 16., 0.])
I'm mapping very large vectors onto 2D matrices and would really like to avoid loops...

The += operator for NumPy arrays simply doesn't work the way you are hoping, and I'm not aware of a away of making it work that way. As a work-around I suggest using numpy.bincount():
>>> numpy.bincount(b, c)
array([ 150., 0., 0., 16.])
Just append zeros as needed.

You could do something like:
def sum_unique(label, weight):
order = np.lexsort(label.T)
label = label[order]
weight = weight[order]
unique = np.ones(len(label), 'bool')
unique[:-1] = (label[1:] != label[:-1]).any(-1)
totals = weight.cumsum()
totals = totals[unique]
totals[1:] = totals[1:] - totals[:-1]
return label[unique], totals
And use it like this:
In [110]: coord = np.random.randint(0, 3, (10, 2))
In [111]: coord
Out[111]:
array([[0, 2],
[0, 2],
[2, 1],
[1, 2],
[1, 0],
[0, 2],
[0, 0],
[2, 1],
[1, 2],
[1, 2]])
In [112]: weights = np.ones(10)
In [113]: uniq_coord, sums = sum_unique(coord, weights)
In [114]: uniq_coord
Out[114]:
array([[0, 0],
[1, 0],
[2, 1],
[0, 2],
[1, 2]])
In [115]: sums
Out[115]: array([ 1., 1., 2., 3., 3.])
In [116]: a = np.zeros((3,3))
In [117]: x, y = uniq_coord.T
In [118]: a[x, y] = sums
In [119]: a
Out[119]:
array([[ 1., 0., 3.],
[ 1., 0., 3.],
[ 0., 2., 0.]])
I just thought of this, it might be easier:
In [120]: flat_coord = np.ravel_multi_index(coord.T, (3,3))
In [121]: sums = np.bincount(flat_coord, weights)
In [122]: a = np.zeros((3,3))
In [123]: a.flat[:len(sums)] = sums
In [124]: a
Out[124]:
array([[ 1., 0., 3.],
[ 1., 0., 3.],
[ 0., 2., 0.]])