We Keep Coding

Figuring out correct numpy transpose for 3*3*3 array

Suppose I have a 3*3*3 array x. I would like to find out an array y, such that such that y[0,1,2] = x[1,2,0], or more generally, y[a,b,c]= x[b,c,a]. I can try numpy.transpose
import numpy as np
x = np.arange(27).reshape((3,3,3))
y = np.transpose(x, [2,0,1])
print(x[0,1,2],x[1,2,0])
print(y[0,1,2])
The output is
5 15
15
The result 15,15 is what I expected (the first 15 is the reference value from x[1,2,0]; the second is from y[0,1,2]) . However, I found the transpose [2,0,1] by drawing in a paper.
B C A
A B C
by inspection, the transpose should be [2,0,1], the last entry in the upper row goes to 1st in the lower row; the middle goes last; the first go middle. Is there any automatic and hopefully efficient way to do it (like any standard function in numpy/sympy)?
Given the input y[a,b,c]= x[b,c,a], output [2,0,1]?

I find easier to explore tranpose with a example with shape like (2,3,4), each axis is different.
But sticking with your (3,3,3)
In [23]: x = np.arange(27).reshape(3,3,3)
In [24]: x
Out[24]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
In [25]: x[0,1,2]
Out[25]: 5
Your sample transpose:
In [26]: y = x.transpose(2,0,1)
In [27]: y
Out[27]:
array([[[ 0, 3, 6],
[ 9, 12, 15],
[18, 21, 24]],
[[ 1, 4, 7],
[10, 13, 16],
[19, 22, 25]],
[[ 2, 5, 8],
[11, 14, 17],
[20, 23, 26]]])
We get the same 5 with
In [28]: y[2,0,1]
Out[28]: 5
We could get that (2,0,1) by applying the same transposing values:
In [31]: idx = np.array((0,1,2)) # use an array for ease of indexing
In [32]: idx[[2,0,1]]
Out[32]: array([2, 0, 1])
The way I think about the trapose (2,0,1), we are moving the last axis, 2, to the front, and preserving the order of the other 2.
With differing dimensions, it's easier to visualize the change:
In [33]: z=np.arange(2*3*4).reshape(2,3,4)
In [34]: z
Out[34]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [35]: z.transpose(2,0,1)
Out[35]:
array([[[ 0, 4, 8],
[12, 16, 20]],
[[ 1, 5, 9],
[13, 17, 21]],
[[ 2, 6, 10],
[14, 18, 22]],
[[ 3, 7, 11],
[15, 19, 23]]])
In [36]: _.shape
Out[36]: (4, 2, 3)
np.swapaxes is another compiled function for making these changes. np.rollaxis is another, though it's python code that ends up calling transpose.
I haven't tried to follow all of your reasoning, though I think you want a kind reverse of the transpose numbers, one where you specify the result order, and want how to get them.

numpy 3d array and 1d array addition on first axis

i have a 1d np array "array1d" and a 3d np array "array3d", i want to sum them so the n'th value in "array1d" will be added to each of the elements of the n'th plane in array3d.
this can be done in the following loop
for i, value in enumerate(array1d):
array3d[i] += value
question is, how can this be done in a single numpy line?
example arrays:
arr1d = np.array(range(3))
>>>array([0, 1, 2])
arr3d = np.array(range(27)).reshape(3, 3, 3)
>>>array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
wanted result:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 10, 11, 12],
[13, 14, 15],
[16, 17, 18]],
[[20, 21, 22],
[23, 24, 25],
[26, 27, 28]]])

Use Numpy's broadcasting features:
In [23]: arr1d[:, None, None] + arr3d
Out[23]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[10, 11, 12],
[13, 14, 15],
[16, 17, 18]],
[[20, 21, 22],
[23, 24, 25],
[26, 27, 28]]])
This basically copies the content of arr1d across the other two dimensions (without actually copying, it just provides a view of the memory which looks like it). Instead of None, you can also use numpy.newaxis.
Alternatively, you can also use reshape:
In [32]: arr1d.reshape(3, 1, 1) + arr3d
Out[32]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[10, 11, 12],
[13, 14, 15],
[16, 17, 18]],
[[20, 21, 22],
[23, 24, 25],
[26, 27, 28]]])

Find all n-dimensional lines and diagonals with NumPy

Using NumPy, I would like to produce a list of all lines and diagonals of an n-dimensional array with lengths of k.
Take the case of the following three-dimensional array with lengths of three.
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
For this case, I would like to obtain all of the following types of sequences. For any given case, I would like to obtain all of the possible sequences of each type. Examples of desired sequences are given in parentheses below, for each case.
1D lines
x axis (0, 1, 2)
y axis (0, 3, 6)
z axis (0, 9, 18)
2D diagonals
x/y axes (0, 4, 8, 2, 4, 6)
x/z axes (0, 10, 20, 2, 10, 18)
y/z axes (0, 12, 24, 6, 12, 18)
3D diagonals
x/y/z axes (0, 13, 26, 2, 13, 24)
The solution should be generalized, so that it will generate all lines and diagonals for an array, regardless of the array's number of dimensions or length (which is constant across all dimensions).

This solution generalized over n
Lets rephrase this problem as "find the list of indices".
We're looking for all of the 2d index arrays of the form
array[i[0], i[1], i[2], ..., i[n-1]]
Let n = arr.ndim
Where i is an array of shape (n, k)
Each of i[j] can be one of:
The same index repeated n times, ri[j] = [j, ..., j]
The forward sequence, fi = [0, 1, ..., k-1]
The backward sequence, bi = [k-1, ..., 1, 0]
With the requirements that each sequence is of the form ^(ri)*(fi)(fi|bi|ri)*$ (using regex to summarize it). This is because:
there must be at least one fi so the "line" is not a point selected repeatedly
no bis come before fis, to avoid getting reversed lines
def product_slices(n):
for i in range(n):
yield (
np.index_exp[np.newaxis] * i +
np.index_exp[:] +
np.index_exp[np.newaxis] * (n - i - 1)
)
def get_lines(n, k):
"""
Returns:
index (tuple): an object suitable for advanced indexing to get all possible lines
mask (ndarray): a boolean mask to apply to the result of the above
"""
fi = np.arange(k)
bi = fi[::-1]
ri = fi[:,None].repeat(k, axis=1)
all_i = np.concatenate((fi[None], bi[None], ri), axis=0)
# inedx which look up every possible line, some of which are not valid
index = tuple(all_i[s] for s in product_slices(n))
# We incrementally allow lines that start with some number of `ri`s, and an `fi`
# [0] here means we chose fi for that index
# [2:] here means we chose an ri for that index
mask = np.zeros((all_i.shape[0],)*n, dtype=np.bool)
sl = np.index_exp[0]
for i in range(n):
mask[sl] = True
sl = np.index_exp[2:] + sl
return index, mask
Applied to your example:
# construct your example array
n = 3
k = 3
data = np.arange(k**n).reshape((k,)*n)
# apply my index_creating function
index, mask = get_lines(n, k)
# apply the index to your array
lines = data[index][mask]
print(lines)
array([[ 0, 13, 26],
[ 2, 13, 24],
[ 0, 12, 24],
[ 1, 13, 25],
[ 2, 14, 26],
[ 6, 13, 20],
[ 8, 13, 18],
[ 6, 12, 18],
[ 7, 13, 19],
[ 8, 14, 20],
[ 0, 10, 20],
[ 2, 10, 18],
[ 0, 9, 18],
[ 1, 10, 19],
[ 2, 11, 20],
[ 3, 13, 23],
[ 5, 13, 21],
[ 3, 12, 21],
[ 4, 13, 22],
[ 5, 14, 23],
[ 6, 16, 26],
[ 8, 16, 24],
[ 6, 15, 24],
[ 7, 16, 25],
[ 8, 17, 26],
[ 0, 4, 8],
[ 2, 4, 6],
[ 0, 3, 6],
[ 1, 4, 7],
[ 2, 5, 8],
[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 13, 17],
[11, 13, 15],
[ 9, 12, 15],
[10, 13, 16],
[11, 14, 17],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 22, 26],
[20, 22, 24],
[18, 21, 24],
[19, 22, 25],
[20, 23, 26],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26]])
Another good set of test data is np.moveaxis(np.indices((k,)*n), 0, -1), which gives an array where every value is its own index
I've solved this problem before to implement a higher dimensional tic-tac-toe

In [1]: x=np.arange(27).reshape(3,3,3)
Selecting individual rows is easy:
In [2]: x[0,0,:]
Out[2]: array([0, 1, 2])
In [3]: x[0,:,0]
Out[3]: array([0, 3, 6])
In [4]: x[:,0,0]
Out[4]: array([ 0, 9, 18])
You could iterate over dimensions with an index list:
In [10]: idx=[slice(None),0,0]
In [11]: x[idx]
Out[11]: array([ 0, 9, 18])
In [12]: idx[2]+=1
In [13]: x[idx]
Out[13]: array([ 1, 10, 19])
Look at the code for np.apply_along_axis to see how it implements this sort of iteration.
Reshape and split can also produce a list of rows. For some dimensions this might require a transpose:
In [20]: np.split(x.reshape(x.shape[0],-1),9,axis=1)
Out[20]:
[array([[ 0],
[ 9],
[18]]), array([[ 1],
[10],
[19]]), array([[ 2],
[11],
...
np.diag can get diagonals from 2d subarrays
In [21]: np.diag(x[0,:,:])
Out[21]: array([0, 4, 8])
In [22]: np.diag(x[1,:,:])
Out[22]: array([ 9, 13, 17])
In [23]: np.diag?
In [24]: np.diag(x[1,:,:],1)
Out[24]: array([10, 14])
In [25]: np.diag(x[1,:,:],-1)
Out[25]: array([12, 16])
And explore np.diagonal for direct application to the 3d. It's also easy to index the array directly, with range and arange, x[0,range(3),range(3)].
As far as I know there isn't a function to step through all these alternatives. Since dimensions of the returned arrays can differ, there's little point to producing such a function in compiled numpy code. So even if there was a function, it would step through the alternatives as I outlined.
==============
All the 1d lines
x.reshape(-1,3)
x.transpose(0,2,1).reshape(-1,3)
x.transpose(1,2,0).reshape(-1,3)
y/z diagonal and anti-diagonal
In [154]: i=np.arange(3)
In [155]: j=np.arange(2,-1,-1)
In [156]: np.concatenate((x[:,i,i],x[:,i,j]),axis=1)
Out[156]:
array([[ 0, 4, 8, 2, 4, 6],
[ 9, 13, 17, 11, 13, 15],
[18, 22, 26, 20, 22, 24]])

np.einsum can be used to build all these kind of expressions; for instance:
# 3d diagonals
print(np.einsum('iii->i', a))
# 2d diagonals
print(np.einsum('iij->ij', a))
print(np.einsum('iji->ij', a))

splitting ND arrays using numpy

I have a 3D numpy array and I want to partition it by the first 2 dimensions (and select all elements in the last one). Is there a simple way I can do that using numpy?
Example: given array
a = array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
I would like to split it N ways by the first two axes (while retaining all elements in the last one), e.g.,:
a[0:2, 0:2, :], a[2:3, 2:3, :]
But it doesn't need to be evenly split. Seems like numpy.array_split will split on all axes?

In [179]: np.array_split(a,2,0)
Out[179]:
[array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]]]),
array([[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])]
is the same as [a[:2,:,:], a[2:,:,:]]
You could loop on those 2 arrays and apply split on the next axis.
In [182]: a2=[np.array_split(aa,2,1) for aa in a1]
In [183]: a2 # edited for clarity
Out[183]:
[[array([[[ 0, 1, 2],
[ 3, 4, 5]],
[[ 9, 10, 11],
[12, 13, 14]]]), # (2,2,3)
array([[[ 6, 7, 8]],
[[15, 16, 17]]])], # (2,1,3)
[array([[[18, 19, 20],
[21, 22, 23]]]), # (1,2,3)
array([[[24, 25, 26]]])]] # (1,1,3)
In [184]: a2[0][0].shape
Out[184]: (2, 2, 3)
In [185]: a2[0][1].shape
Out[185]: (2, 1, 3)
In [187]: a2[1][0].shape
Out[187]: (1, 2, 3)
In [188]: a2[1][1].shape
Out[188]: (1, 1, 3)
With the potential of splitting in uneven arrays in each dimension, it is hard to do this in a full vectorized form. And even if the splits were even it's tricky to do this sort of grid splitting because values are not contiguous. In this example there's a gap between 5 and 9 in the first subarray.

A quick list comprehension will do the trick
[np.array_split(arr, 2, axis=1)
for arr in np.array_split(a, 2, axis=0)]
This will result in a list of lists, the items of which contain the arrays you're looking for.

Reshaping array into a square array Python

I have an array of numbers whose shape is 26*43264. I would like to reshape this into an array of shape 208*208 but in chunks of 26*26.
[[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10,11,12,13,14,15,16,17,18,19]]
becomes something like:
[[0, 1, 2, 3, 4],
[10,11,12,13,14],
[ 5, 6, 7, 8, 9],
[15,16,17,18,19]]

This kind of reshaping question has come up before. But rather than search I'll quickly demonstate a numpy approach
make your sample array:
In [473]: x=np.arange(20).reshape(2,10)
In [474]: x
Out[474]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])
Use reshape to split it into blocks of 5
In [475]: x.reshape(2,2,5)
Out[475]:
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9]],
[[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]]])
and use transpose to reorder dimensions, and in effect reorder those rows
In [476]: x.reshape(2,2,5).transpose(1,0,2)
Out[476]:
array([[[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14]],
[[ 5, 6, 7, 8, 9],
[15, 16, 17, 18, 19]]])
and another shape to consolidate the 1st 2 dimensions
In [477]: x.reshape(2,2,5).transpose(1,0,2).reshape(4,5)
Out[477]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[ 5, 6, 7, 8, 9],
[15, 16, 17, 18, 19]])
If x is already a numpy array, these transpose and reshape operations are cheap (time wise). If x was really nested lists, then the other solution with list operations will be faster, since making a numpy array has overhead.

A little ugly, but here's a one-liner for the small example that you should be able to modify for the full size one:
In [29]: from itertools import chain
In [30]: np.array(list(chain(*[np.arange(20).reshape(4,5)[i::2] for i in xrange(2)])))
Out[30]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[ 5, 6, 7, 8, 9],
[15, 16, 17, 18, 19]])
EDIT: Here's a more generalized version in a function. Uglier code, but the function just takes an array and a number of segments you'd like to end up with.
In [57]: def break_arr(arr, chunks):
....: to_take = arr.shape[1]/chunks
....: return np.array(list(chain(*[arr.take(xrange(x*to_take, x*to_take+to_take), axis=1) for x in xrange(chunks)])))
....:
In [58]: arr = np.arange(40).reshape(4,10)
In [59]: break_arr(arr, 5)
Out[59]:
array([[ 0, 1],
[10, 11],
[20, 21],
[30, 31],
[ 2, 3],
[12, 13],
[22, 23],
[32, 33],
[ 4, 5],
[14, 15],
[24, 25],
[34, 35],
[ 6, 7],
[16, 17],
[26, 27],
[36, 37],
[ 8, 9],
[18, 19],
[28, 29],
[38, 39]])
In [60]: break_arr(arr, 2)
Out[60]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[ 5, 6, 7, 8, 9],
[15, 16, 17, 18, 19],
[25, 26, 27, 28, 29],
[35, 36, 37, 38, 39]])