Related
Assume I have a matrix like
a = np.array([[[ 1, 2], [ 3, 4]],
[[ 5, 6], [ 7, 8]],
[[ 9, 10], [11, 12]],
[[13, 14], [15, 16]]])
The shape is (4, 2, 2). I want to sum the first two and the 2nd two matrices to each other. Final output size should have shape (2, 2, 2) and the output should be
output = np.array([[[ 6, 8], [10, 12]],
[[22, 24], [26, 28]]])
You can see my attempt below:
import numpy as np
a = np.array([[[ 1, 2], [ 3, 4]],
[[ 5, 6], [ 7, 8]],
[[ 9, 10], [11, 12]],
[[13, 14], [15, 16]]])
output = np.add(a[:2], a[2:])
Break the first dimension up into two using a reshape, sum along the second axis:
a.reshape(2, 2, *a.shape[1:]).sum(axis=1)
Your current approach is equivalent to a.reshape(2, 2, *a.shape[1:]).sum(axis=0). The correct way would be to slice every other row of the entire array, rather than every other block of the entire array:
a[::2] + a[1::2]
The latter approach does not generalize well. If you had to add up say every block of seven, you would get
a[::7] + a[1::7] + a[2::7] + a[3::7] + ... + a[6::7]
The former approach is quite flexible, however:
a.reshape(-1, 7, *a.shape[1:]).sum(axis=1)
Numpy's rot90 function promises to rotate a 2d or higher array by 90 degrees, taking an axes parameter. The method:
numpy.rot90(m, k=1, axes=(0, 1))[source]
Rotate an array by 90 degrees in the plane specified by axes.
Rotation direction is from the first towards the second axis.
I'm very confused about the axes part. An object can be rotated around the x, y, or z axis. Typically, this is defined by something such as a Vector3f, with 3 floats defining the axis value (ex, (0, 0, 1) to rotate around z axis.) I do not understand how these two numbers can be used to rotate a 3d object, shouldn't it be 3 like a Vector3f? Can anyone help me understand what these two axes mean, and what two numbers would be used for, respectively, rotation around the x, y, and z axis? I've tried many different combinations of numbers and they all have various results (I can't put in two of the same numbers), but I have no idea how it's possible to have enough information with two numbers (k represents amount of times to rotate.)
I like to work with a sample array with distinct values and dimensions, such as np.arange(24).reshape(2,3,4).
In this case I also looked at the code. After some preliminaries to make sure axes and k are right it does different things depending on the 4 possible values of k.
axes define a "plane". With a 3d array, axes=(0,1) can be thought of as rotation about the third axes (2), imagining a "vector" perpendicular to that "plane". But it's these axes values that used in the code. While I suspect we could construct equivalent operations with trig based rotation matrices, this code does not do any calculations. (Note that integers are not changed to floats.)
k=0 changes nothing:
In [104]: np.rot90(m,k=0, axes=(0,1))
Out[104]:
array([[[ 0, 1, 2, 3], # shape (2,3,4)
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
k=1 involves a flip followed by a transpose
In [105]: np.rot90(m,k=1, axes=(0,1))
Out[105]:
array([[[ 8, 9, 10, 11], # shape (3,2,4)
[20, 21, 22, 23]],
[[ 4, 5, 6, 7],
[16, 17, 18, 19]],
[[ 0, 1, 2, 3],
[12, 13, 14, 15]]])
k=2 is simpler - just a flip on both axes. This is easy to visualize. Last dimension is unchanged (across rows), but planes and rows are reversed:
In [106]: np.rot90(m,k=2, axes=(0,1))
Out[106]:
array([[[20, 21, 22, 23],
[16, 17, 18, 19],
[12, 13, 14, 15]],
[[ 8, 9, 10, 11],
[ 4, 5, 6, 7],
[ 0, 1, 2, 3]]])
k=3 does the flip first, then the transpose
In [107]: np.rot90(m,k=3, axes=(0,1))
Out[107]:
array([[[12, 13, 14, 15], # again (3,2,4)
[ 0, 1, 2, 3]],
[[16, 17, 18, 19],
[ 4, 5, 6, 7]],
[[20, 21, 22, 23],
[ 8, 9, 10, 11]]])
Looking at the strides:
In [111]: m.strides
Out[111]: (96, 32, 8)
In [112]: np.rot90(m,k=2, axes=(0,1)).strides
Out[112]: (-96, -32, 8) # the double flip
The transpose switches the order, while the flip again changes the sign:
In [113]: np.rot90(m,k=1, axes=(0,1)).strides
Out[113]: (-32, 96, 8)
In [114]: np.rot90(m,k=3, axes=(0,1)).strides
Out[114]: (32, -96, 8)
Because it just uses flip and transpose the result is a view.
Simpler (1,3,4) array
It may be easier to visualize in an array that represents values in one plane, a (3,4) array:
In [120]: m = np.arange(12).reshape(1,3,4)
In [121]: m
Out[121]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]])
In [122]: np.rot90(m,k=2, axes=(1,2))
Out[122]:
array([[[11, 10, 9, 8],
[ 7, 6, 5, 4],
[ 3, 2, 1, 0]]])
In [123]: np.rot90(m,k=1, axes=(1,2)) # visualize a counterclockwise rotation
Out[123]:
array([[[ 3, 7, 11],
[ 2, 6, 10],
[ 1, 5, 9],
[ 0, 4, 8]]])
In [124]: np.rot90(m,k=3, axes=(1,2)) # clockwise
Out[124]:
array([[[ 8, 4, 0],
[ 9, 5, 1],
[10, 6, 2],
[11, 7, 3]]])
k=3 can also be visualized as successive counterclockwise rotations through 1 and 2.
I created a sample array:
a = np.arange(18).reshape(9,2)
On printing, I get this as output:
[[ 0 1]
[ 2 3]
[ 4 5]
[ 6 7]
[ 8 9]
[10 11]
[12 13]
[14 15]
[16 17]]
On executing this reshaping:
b = a.reshape(2,3,3).swapaxes(0,2)
I get:
[[[ 0 9]
[ 3 12]
[ 6 15]]
[[ 1 10]
[ 4 13]
[ 7 16]]
[[ 2 11]
[ 5 14]
[ 8 17]]]
I went through this question, but it does not solve my problem.
Reshape an array in NumPy
The documentation is not useful either.
https://docs.scipy.org/doc/numpy/reference/generated/numpy.swapaxes.html
I need to know how the swapping is working(which is x-axis, y-axis, z-axis). A diagrammatic explanation would be most helpful.
Here is my understanding of swapaxes
Suppose you have an array
In [1]: arr = np.arange(16).reshape((2, 2, 4))
In [2]: arr
Out[2]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]]])
And the shape of arr is (2, 2, 4), for the value 7, you can get the value by
In [3]: arr[0, 1, 3]
Out[3]: 7
There are 3 axes 0, 1 and 2, now, we swap axis 0 and 2
In [4]: arr_swap = arr.swapaxes(0, 2)
In [5]: arr_swap
Out[5]:
array([[[ 0, 8],
[ 4, 12]],
[[ 1, 9],
[ 5, 13]],
[[ 2, 10],
[ 6, 14]],
[[ 3, 11],
[ 7, 15]]])
And as you can guess, the index of 7 is (3, 1, 0), with axis 1 unchanged,
In [6]: arr_swap[3, 1, 0]
Out[6]: 7
So, now from the perspective of the index, swapping axis is just change the index of values. For example
In [7]: arr[0, 0, 1]
Out[7]: 1
In [8]: arr_swap[1, 0, 0]
Out[8]: 1
In [9]: arr[0, 1, 2]
Out[9]: 6
In [9]: arr_swap[2, 1, 0]
Out[9]: 6
So, if you feel difficult to get the swapped-axis array, just change the index, say arr_swap[2, 1, 0] = arr[0, 1, 2].
Start with the reshape
In [322]: a = np.arange(18).reshape(2,3,3)
In [323]: a
Out[323]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]]])
This displays as 2 planes, and each plane is a 3x3. Is that part clear? The fact that the array was shaped (9,2) at one point isn't significant. Reshaping doesn't change the order of elements.
Apply the swapaxes. Shape is now (3,3,2). 3 planes, each is 3x2. This particular swap is the same as a transpose
np.arange(18).reshape(2,3,3).transpose(2,1,0)
The middle axis is unchanged. There are still columns of [0,3,6], [9,12,15], etc.
It may be easier to visualize the change with 3 different sized axes
In [335]: a=np.arange(2*3*4).reshape(2,3,4)
In [336]: a
Out[336]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [337]: a.swapaxes(0,2)
Out[337]:
array([[[ 0, 12],
[ 4, 16],
[ 8, 20]],
[[ 1, 13],
[ 5, 17],
[ 9, 21]],
[[ 2, 14],
[ 6, 18],
[10, 22]],
[[ 3, 15],
[ 7, 19],
[11, 23]]])
Notice what happens when I flatten the array
In [338]: a.swapaxes(0,2).ravel()
Out[338]:
array([ 0, 12, 4, 16, 8, 20, 1, 13, 5, 17, 9, 21, 2, 14, 6, 18, 10,
22, 3, 15, 7, 19, 11, 23])
the order of terms has been shuffled. As created it was [0,1,2,3...]. Now the 1 is the 6th term (2x3).
Under the covers numpy actually performs the swap or transpose by changing shape, strides and order, without changing the data buffer (i.e. it's a view). But further reshaping, including raveling, forces it to make a copy. But that might be more confusing than helpful at this stage.
In numpy axes are numbered. Terms like x,y,z or planes, rows, columns may help you map those on to constructs that you can visualize, but they aren't 'built-in'. Describing the swap or transpose in words is tricky.
As stated in scipy lecture notes, this will not work as expected:
a = np.random.randint(0, 10, (1000, 1000))
a += a.T
assert np.allclose(a, a.T)
But why? How does being a view affect this behavior?
a += a.T
does in-place summing up (it's using view a.T while processing), so you end up with a non-symmetric matrix
you can easily check this, i.e. I got:
In [3]: a
Out[3]:
array([[ 6, 15, 7, ..., 8, 9, 2],
[15, 6, 9, ..., 14, 9, 7],
[ 7, 9, 0, ..., 9, 5, 8],
...,
[ 8, 23, 15, ..., 6, 4, 10],
[18, 13, 8, ..., 4, 2, 6],
[ 3, 9, 9, ..., 16, 8, 4]])
You can see it's not symmetric, right? (compare right-top and left-bottom items)
if you do a real copy:
a += np.array(a.T)
it works fine, i.e.:
In [6]: a
Out[6]:
array([[ 2, 11, 8, ..., 9, 15, 5],
[11, 4, 14, ..., 10, 3, 13],
[ 8, 14, 14, ..., 10, 9, 3],
...,
[ 9, 10, 10, ..., 16, 7, 6],
[15, 3, 9, ..., 7, 14, 1],
[ 5, 13, 3, ..., 6, 1, 2]])
To better understand why it does so, you can imagine you wrote the loop yourself as following:
In [8]: for i in xrange(1000):
for j in xrange(1000):
a[j,i] += a[i,j]
....:
In [9]: a
Out[9]:
array([[ 4, 5, 14, ..., 12, 16, 13],
[ 3, 2, 16, ..., 16, 8, 8],
[ 9, 12, 10, ..., 7, 7, 23],
...,
[ 8, 10, 6, ..., 14, 13, 23],
[10, 4, 6, ..., 9, 16, 21],
[11, 8, 14, ..., 16, 12, 12]])
It adds a[999,0] to calculate a[0,999] but a[999,0] already has sum of a[999,0] + a[0,999] -- so below the main diagonal you add the values twice.
This problem is due to internal designs of numpy.
It basically boils down to that the inplace operator will change the values as it goes, and then those changed values will be used where you were actually intending for the original value to be used.
This is discussed in this bug report, and it does not seem to be fixable.
The reason why it works for smaller size arrays seems to be because of how the data is buffered while being worked on.
To exactly understand why the issue crops up, I am afraid you will have to dig into the internals of numpy.
For large arrays, the in-place operator causes you to apply the addition to the operator currently being operated on. For example:
>>> a = np.random.randint(0, 10, (1000, 1000))
>>> a
array([[9, 4, 2, ..., 7, 0, 6],
[8, 4, 1, ..., 3, 5, 9],
[6, 4, 9, ..., 6, 9, 7],
...,
[6, 2, 5, ..., 0, 4, 6],
[5, 7, 9, ..., 8, 0, 5],
[2, 0, 1, ..., 4, 3, 5]])
Notice that the top-right and bottom-left elements are 6 and 2.
>>> a += a.T
>>> a
array([[18, 12, 8, ..., 13, 5, 8],
[12, 8, 5, ..., 5, 12, 9],
[ 8, 5, 18, ..., 11, 18, 8],
...,
[19, 7, 16, ..., 0, 12, 10],
[10, 19, 27, ..., 12, 0, 8],
[10, 9, 9, ..., 14, 11, 10]])
Now notice that the top-right element is correct (8 = 6 + 2). However, the bottom-left element is the result not of 6 + 2, but of 8 + 2. In other words, the addition that was applied to the bottom-left element is the top-right element of the array after the addition. You'll notice this is true for all the other elements below the first row as well.
I imagine this works this way because you do not need to make a copy of your array. (Though it then looks like it does make a copy if the array is small.)
assert np.allclose(a, a.T)
I just understood now that you are generating a symmetric matrix by summing a with it's transponse a.T resulting in a symmetric matrix)
Which makes us rightfully expect np.allclose(a, a.T) to return true (resulting matrix being symmetric so it should be equal to its transpose)
a += a.T # How being a view affects this behavior?
I've just narrowed it down to this
TL;DR a = a + a.T is fine for larger matrices, while a += a.T gives strange results starting from 91x91 size
>>> a = np.random.randint(0, 10, (1000, 1000))
>>> a += a.T # using the += operator
>>> np.allclose(a, a.T)
False
>>> a = np.random.randint(0, 10, (1000, 1000))
>>> a = a + a.T # using the + operator
>>> np.allclose(a, a.T)
True
I've got same cut-off at size 90x90 like #Hannes (I am on Python 2.7 and Numpy 1.11.0, so there are at least two environments that can produce this)
>>> a = np.random.randint(0, 10, (90, 90))
>>> a += a.T
>>> np.allclose(a, a.T)
True
>>> a = np.random.randint(0, 10, (91, 91))
>>> a += a.T
>>> np.allclose(a, a.T)
False
I have 2-D data containing bad values (0 indicates bad). My goal is to replace each bad value with its nearest neighbor that isn't bad.
SciPy's NearestNDInterpolator seems like a nice way to do this. In the 2-D case, it accepts a (number of points) x 2 array of indices and a (number of points) x 1 array of corresponding values to interpolate from.
So, I need to get a subset of the indices and values: those that are "good." The code below achieves this, but coordinates = array(list(ndindex(n_y, n_x))) and where(values != 0)[0] are both messy. Is there a cleaner way to do this?
# n_y and n_x are the number of points along each dimension.
coordinates = array(list(ndindex(n_y, n_x)))
values = data.flatten()
nonzero_ind = where(values != 0)[0]
nonzero_coordinates = coordinates[nonzero_ind, :]
nonzero_values = values[nonzero_ind]
Thanks.
nonzero_coordinates = np.argwhere(data != 0)
nonzero_values = np.extract(data, data)
or simply:
nonzero_values = data[data!=0]
I initially rather missed the obvious nonzero_values method, but thanks to #askewchan in the comments for that.
So, I need to get a subset of the indices and values: those that are "good."
If you've created a "mask" of the bad indices, you can take the negation of that mask ~ and then find the indices from the mask using np.where. For example:
import numpy as np
# Sample array
Z = np.random.random(size=(5,5))
# Use whatever criteria you have to mark the bad indices
bad_mask = Z<.2
good_mask = ~bad_mask
good_idx = np.where(good_mask)
print good_mask
print good_idx
Gives, as an example:
[[ True True True True False]
[ True False False True True]
[ True False True True True]
[ True True True True True]
[ True True True True True]]
(array([0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4]), array([0, 1, 2, 3, 0, 3, 4, 0, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4]))
Another way to approach this problem altogether is to just run your array through an image filter that would automatically 'close' those holes. There is such a filter in scipy.ndimage called grey_closing:
>>> from scipy import ndimage
>>> a = np.arange(1,26).reshape(5,5)
>>> a[2,2] = 0
>>> a
array([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 0, 14, 15],
[16, 17, 18, 19, 20],
[21, 22, 23, 24, 25]])
>>> a = np.arange(1,26).reshape(5,5)
>>> ndimage.grey_closing(a, size=2)
array([[ 7, 7, 8, 9, 10],
[ 7, 7, 8, 9, 10],
[12, 12, 13, 14, 15],
[17, 17, 18, 19, 20],
[22, 22, 23, 24, 25]])
But this has unfortunate edge affects (which you can change a bit with the mode paramenter). To avoid this, you could just take the new values from where the original array was 0 and place them into the original array:
>>> np.where(a, a, ndimage.grey_closing(a, size=2))
array([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 12, 14, 15],
[16, 17, 18, 19, 20],
[21, 22, 23, 24, 25]])
Alternatively, you could use scikit-image:
>>> from skimage.morphology import closing, square
>>> a = np.arange(1,10, dtype=np.uint8).reshape(3,3)
>>> a[1,1] = 0
>>> a
array([[1, 2, 3],
[4, 0, 6],
[7, 8, 9]], dtype=uint8)
>>> closing(a, square(2))
array([[1, 2, 3],
[4, 4, 6],
[7, 8, 9]], dtype=uint8)
>>> a
array([[1, 2, 3],
[4, 0, 6],
[7, 8, 9]], dtype=uint8)
Give it a as the output array and the closing is done in-place:
>>> closing(a, square(2), a)
>>> a
array([[1, 2, 3],
[4, 4, 6],
[7, 8, 9]], dtype=uint8)
Use a larger square (or any shape from skimage.morphology) if you have big gaps of zeros. The disadvantage of this (aside from the dependency) is that it seems to only work for uint8.