python/numpy combine subarrays 4 rows at a time - python

I have a numpy array that is split by each row:
splitArray:
[[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]
I was hoping to merge said splitArray every 4 rows, and the last subarray not necessarily having to be 4, but just the remainder of what's left.
Below is the array I hope to have:
joinedArray:
[[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]]


Using a list-comp:
[a[i:i+4] for i in range(0, len(a), 4)]
#[array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]]),
# array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]]),
# array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]]),
# array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]])]


As a pure Numpythonic approach you can find all the desired indexes for splitting your array by creating a range from chunking number to number of rows with the chunking number as thestep arg of the range. Then use np.split() to split your array:
In [24]: def chunk_array(arr, ch):
...: x = arr.shape[0]
...: return np.split(a, np.arange(ch, x, ch))
...:
...:
Demo:
In [25]: chunk_array(a, 4)
Out[25]:
[array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]])]
In [26]: chunk_array(a, 3)
Out[26]:
[array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]])]
If you want the chunked arrays to be concatenated you can use #jpp's answer with np.concatenate() and map or slightly different in a list comprehension.
In [75]: def chunk_array(arr, ch):
...: x = arr.shape[0]
...: return [np.concatenate(subs) for subs in np.split(arr, np.arange(ch, x, ch))]


That can be done using the infamous grouper recipe.
>>> from itertools import zip_longest
>>> import numpy as np
>>>
>>> data = [7 * [0] for i in range(14)]
>>> i=iter(data); list(map(np.concatenate, zip_longest(*4*(i,), fillvalue=[])))
[array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0]), array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0]), array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0]), array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])]


You can use np.concatenate with np.split. If required, you can adjust the below example to output a list of lists instead of a list of arrays.
As mentioned, a single jagged numpy array is not a good idea.
A = np.zeros((14, 3))
res = list(map(np.concatenate, np.split(A, np.arange(4, A.shape[0], 4))))
print(res)
[array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([ 0., 0., 0., 0., 0., 0.])]

Related

Have created a 3D array of chess boards, how can I add more boards to this?

neighboringStates = np.array([
[[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0]],
[[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0]]
])
Wondering how I can add another 8x8 of zeros and ones and add it to this already existing 3D array. Thanks!

Use np.concatenate() to add a new array to the existing 3D array
Example Code:
import numpy as np
neighboringStates = np.array([
[[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0]],
[[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0]]
])
new_array = np.array([
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1]
])
neighboringStates = np.concatenate((neighboringStates, np.array([new_array])))

How to add label to image?

Hi I used this line of code to see the format of images in mnist dataset:
print(mnsit.load_data())
The output is this:
(array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8), array([5, 0, 4, ..., 5, 6, 8], dtype=uint8))
The main array contains an array at the end. This array is a set of labels that I give to input of a neural network.
Now I need to add an array of labels at the end of images to create a dataset like mnist dataset. Are there functions I can use to do this? I tried append and insert but it didn't work.

You could do something like that:
dataset = [image_array, label_array]
That creates an list with the two array as a elements.

Concatenating or adding elements of an array

i have a numpy array of 27 elements,Im trying concatenate or add all the elements inside the array,but i cant come up with anything right,
I tried,
for index,value in enumerate(array):
np.concatenate(array[index],array[index])
but this throws
TypeError: only integer scalar arrays can be converted to a scalar index
I tried
array[1]+array[2]+array[3]
this works for me, but im not sure how to put this in a loop,
Any suggestions on this front would be really helpful
Thanks in advance.
EDIT:
array looks like this
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,

You want to sum over one axis (the first, I think). Like this:
array.sum(axis=0)

numpy 3D dot product

I have two 3dim numpy matrices and I want to do a dot product according to one axis without using a loop:
a=[ [[ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]],
[[ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]],
[ [ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]],
[ [ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]],
[[ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]],
[[ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0.]],
[[ 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],
[ 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[ 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[ 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]]]
b=[[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]],
[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]],
[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]],
[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]],
[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]],
[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]],
[[ 0, 0, 1, 0, 0.],
[ 1, 0, 0, 0, 0.],
[ 0, 0, 0, 0, 0.],
[ 0, 1, 0, 0, 0.]]]
dt = np.dtype(np.float32)
a=np.asarray(a,dtype=dt)
b=np.asarray(b,dtype=dt)
print(a.shape)
print(b.shape)
a has the shape of (7, 4, 15) and b has the shape of (7, 4, 5).
I want the c=np.dot(a,b) be in the size of (7,5,15) as below:
c = np.zeros((7,15,5))
for i in range(7):
c[i,:,:] = np.dot(a[i,:,:].T , b[i,:,:])
But I am looking for a solution without a for-loop. something like:
c = np.tensordot(a.reshape(4,7,5),b.reshape(7,4,15),axes=([1,0],[0,1]))
but this one doesn't work as expected.
I also tried this:
newaxes_a=[2,0,1]
newaxes_b=[1,0,2]
newshape_a=(-1,28)
newshape_b=(28,-1)
a_t = a.transpose(newaxes_a).reshape(newshape_a)
b_t = b.transpose(newaxes_b).reshape(newshape_b)
c = np.dot(a_t, b_t)
which didn't work as expected.
Any ideas?

You can use np.einsum -
#to match the given example
c2 = np.einsum('ijk,ijl->ikl',a,b)
print np.allclose(c, c2)
Another one using broadcasting -
c = (a[:,:,None,:]*b[...,None]).sum(1)

scipy block_diag does not preserve complex numbers

I wrote a function that puts tensor products of Pauli matrices on the diagonal, using the block_diag function.
When I implement the function , I obtain:
array([[ 1, 0, 0, 0, 0, 0, 0, 0],
[ 0, 1, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 1, 0, 0, 0, 0],
[ 0, 0, 1, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 1, 0],
[ 0, 0, 0, 0, 0, 0, 0, -1]])
as you can see, the matrix
array([[ 0 , -1j],[1j,0]])
is missing, being replaced by a 2x2 0 matrix.
The warning it gives me is :
/usr/lib/python2.7/dist-packages/scipy/linalg/special_matrices.py:541: ComplexWarning: Casting complex values to real discards the imaginary part
out[r:r + rr, c:c + cc] = arrs[i]
Any ideas on how I can overcome this?

Categories