We Keep Coding

Python - How to split an array based on the first column?

I have below fake data. After reading it into array it will have shape (8, 3). Now I want to split the data based on the first column(ID) and return a list of array whose shape will be:[(3,3),(2,3),(3,3)]. I think np.split could do the job by assigning a 1-D array to "indices_or_sections" argument. But is there any more convenient way to do this?
1 700 35
1 700 35
1 700 35
2 680 25
2 680 25
3 750 40
3 750 40
3 750 40

You can achieve this by using a combination of np.split, sort, np.unique and np.cumsum.
>>> a = [[1, 700, 35],
... [1, 700, 35],
... [1, 700, 35],
... [2, 680, 25],
... [2, 680, 25],
... [3, 750, 40],
... [3, 750, 40],
... [3, 750, 40]]
>>> a = np.array(a)
>>> # sort the array by first column.
>>> a = a[a[:,0].argsort()]
>>> np.split(a, np.cumsum(np.unique(a[:, 0], return_counts=True)[1])[:-1])
[array([[ 1, 700, 35],
[ 1, 700, 35],
[ 1, 700, 35]]), array([[ 2, 680, 25],
[ 2, 680, 25]]), array([[ 3, 750, 40],
[ 3, 750, 40],
[ 3, 750, 40]])]

2D cross-correlation of a 6x6 array with three 3x3 kernels

I have a 6x6 matrix: e.g. matrix A
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
I also have a 3x3x3 matrix: e.g. matrix B
array([[[ 1, 7, 2],
[ 5, 9, 3],
[ 2, 8, 6]],
[[ 3, 4, 6],
[ 6, 8, 9],
[ 4, 2, 8]],
[[ 6, 4, 7],
[ 8, 7, 8],
[ 4, 4, 7]]])
Finally, I have a 3x4x4 matrix C, (4 rows, 4 columns, 3 dimensions), that's empty (filled with 0s)
I want to multiply each "3rd dimension" of B (i.e. [1,:,:],[2,:,:],[3,:,:]) with A. However, for each dimension I want to multiply B in "windows", sliding by 1 each time across A till I cannot go further, at which point I move back to the beginning, slide 1 unit down and again sliding across one-by-one multiplying B with A, till the end, then move down and repeat till you don't go over the border. The results being stored in the respective "3rd dimension" of matrix C. So my result would be a [3x4x4] matrix.
Ex. (multiplication is dot product giving a scalar value, np.sum((np.multiply(x,y)))), so...
imagining B "overtop" of A, starting in the right corner, I multiply that 3x3 part of A with Bs [1x3x3] part storing the result in C...
referring to 1st unit (located in 1st row and 1st column) in the 1st dimension of C...
C[1,0,0] = 340. because [[0,1,2],[6,7,8],[12,13,4]] dot product [[1,7,2],[5,9,3],[2,8,6]]
sliding B matrix over by 1 on A, and storing my 2nd result in C...
C[1,0,1] = 383. because [[1,2,3],[7,8,9],[13,14,15]] dot product [[1,7,2],[5,9,3],[2,8,6]]
Then repeat this procedure of sliding across and down and across and ..., for B[2,:,:] and B[3,:,:] over A again, storing in C2,:,:] and C[3,:,:] respectively.
What is a good way to do this?

I think you're asking about 2D cross-correlation with three different kernels, rather than straightforward matrix multiplication.
The following piece of code is not the most efficient way to do this, but does this give you the answer you are looking for? I'm using scipy.signal.correlate2d to achieve 2D correlation here...
>>> from scipy.signal import correlate2d
>>> C = np.dstack([correlate2d(A, B[:, :, i], 'valid') for i in range(B.shape[2])])
>>> C.shape
(4, 4, 3)
>>> C
array([[[ 333, 316, 464],
[ 372, 369, 520],
[ 411, 422, 576],
[ 450, 475, 632]],
[[ 567, 634, 800],
[ 606, 687, 856],
[ 645, 740, 912],
[ 684, 793, 968]],
[[ 801, 952, 1136],
[ 840, 1005, 1192],
[ 879, 1058, 1248],
[ 918, 1111, 1304]],
[[1035, 1270, 1472],
[1074, 1323, 1528],
[1113, 1376, 1584],
[1152, 1429, 1640]]])
Here's a more "fun" way of doing this which doesn't use scipy, but using stride_tricks instead. I'm not sure if it's more efficient:
>>> import numpy.lib.stride_tricks as st
>>> s, t = A.strides
>>> i, j = A.shape
>>> k, l, m = B.shape
>>> D = st.as_strided(A, shape=(i-k+1, j-l+1, k, l), strides=(s, t, s, t))
>>> E = np.einsum('ijkl,klm->ijm', D, B)
>>> (E == C).all()
True

Tensorflow, how to multiply a 2D tensor (matrix) by corresponding elements in a 1D vector

I have a 2D matrix M of shape [batch x dim], I have a vector V of shape [batch]. How can I multiply each of the columns in the matrix by the corresponding element in the V? That is:
I know an inefficient numpy implementation would look like this:
import numpy as np
M = np.random.uniform(size=(4, 10))
V = np.random.randint(4)
def tst(M, V):
rows = []
for i in range(len(M)):
col = []
for j in range(len(M[i])):
col.append(M[i][j] * V[i])
rows.append(col)
return np.array(rows)
In tensorflow, given two tensors, what is the most efficient way to achieve this?
import tensorflow as tf
sess = tf.InteractiveSession()
M = tf.constant(np.random.normal(size=(4,10)), dtype=tf.float32)
V = tf.constant([1,2,3,4], dtype=tf.float32)

In NumPy, we would need to make V 2D and then let broadcasting do the element-wise multiplication (i.e. Hadamard product). I am guessing, it should be the same on tensorflow. So, for expanding dims on tensorflow, we can use tf.newaxis (on newer versions) or tf.expand_dims or a reshape with tf.reshape -
tf.multiply(M, V[:,tf.newaxis])
tf.multiply(M, tf.expand_dims(V,1))
tf.multiply(M, tf.reshape(V, (-1, 1)))

In addition to #Divakar's answer, I would like to make a note that the order of M and V don't matter. It seems that tf.multiply also does broadcasting during multiplication.
Example:
In [55]: M.eval()
Out[55]:
array([[1, 2, 3, 4],
[2, 3, 4, 5],
[3, 4, 5, 6]], dtype=int32)
In [56]: V.eval()
Out[56]: array([10, 20, 30], dtype=int32)
In [57]: tf.multiply(M, V[:,tf.newaxis]).eval()
Out[57]:
array([[ 10, 20, 30, 40],
[ 40, 60, 80, 100],
[ 90, 120, 150, 180]], dtype=int32)
In [58]: tf.multiply(V[:, tf.newaxis], M).eval()
Out[58]:
array([[ 10, 20, 30, 40],
[ 40, 60, 80, 100],
[ 90, 120, 150, 180]], dtype=int32)

How to break numpy array into smaller chunks/batches, then iterate through them

Suppose i have this numpy array
[[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]]
And i want to split it in 2 batches and then iterate:
[[1, 2, 3], Batch 1
[4, 5, 6]]
[[7, 8, 9], Batch 2
[10, 11, 12]]
What is the simplest way to do it?
EDIT: I'm deeply sorry i missed putting such info: Once i intend to carry on with the iteration, the original array would be destroyed due to splitting and iterating over batches. Once the batch iteration finished, i need to restart again from the first batch hence I should preserve that the original array wouldn't be destroyed. The whole idea is to be consistent with Stochastic Gradient Descent algorithms which require iterations over batches. In a typical example, I could have a 100000 iteration For loop for just 1000 batch that should be replayed again and again.

You can use numpy.split to split along the first axis n times, where n is the number of desired batches. Thus, the implementation would look like this -
np.split(arr,n,axis=0) # n is number of batches
Since, the default value for axis is 0 itself, so we can skip setting it. So, we would simply have -
np.split(arr,n)
Sample runs -
In [132]: arr # Input array of shape (10,3)
Out[132]:
array([[170, 52, 204],
[114, 235, 191],
[ 63, 145, 171],
[ 16, 97, 173],
[197, 36, 246],
[218, 75, 68],
[223, 198, 84],
[206, 211, 151],
[187, 132, 18],
[121, 212, 140]])
In [133]: np.split(arr,2) # Split into 2 batches
Out[133]:
[array([[170, 52, 204],
[114, 235, 191],
[ 63, 145, 171],
[ 16, 97, 173],
[197, 36, 246]]), array([[218, 75, 68],
[223, 198, 84],
[206, 211, 151],
[187, 132, 18],
[121, 212, 140]])]
In [134]: np.split(arr,5) # Split into 5 batches
Out[134]:
[array([[170, 52, 204],
[114, 235, 191]]), array([[ 63, 145, 171],
[ 16, 97, 173]]), array([[197, 36, 246],
[218, 75, 68]]), array([[223, 198, 84],
[206, 211, 151]]), array([[187, 132, 18],
[121, 212, 140]])]

consider array a
a = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]])
Option 1
use reshape and //
a.reshape(a.shape[0] // 2, -1, a.shape[1])
array([[[ 1, 2, 3],
[ 4, 5, 6]],
[[ 7, 8, 9],
[10, 11, 12]]])
Option 2
if you wanted groups of two rather than two groups
a.reshape(-1, 2, a.shape[1])
array([[[ 1, 2, 3],
[ 4, 5, 6]],
[[ 7, 8, 9],
[10, 11, 12]]])
Option 3
Use a generator
def get_every_n(a, n=2):
for i in range(a.shape[0] // n):
yield a[n*i:n*(i+1)]
for sa in get_every_n(a, n=2):
print sa
[[1 2 3]
[4 5 6]]
[[ 7 8 9]
[10 11 12]]

To avoid the error "array split does not result in an equal division",
np.array_split(arr, n, axis=0)
is better than np.split(arr, n, axis=0).
For example,
a = np.array([[170, 52, 204],
[114, 235, 191],
[ 63, 145, 171],
[ 16, 97, 173]])
then
print(np.array_split(a, 2))
[array([[170, 52, 204],
[114, 235, 191]]), array([[ 63, 145, 171],
[ 16, 97, 173]])]
print(np.array_split(a, 3))
[array([[170, 52, 204],
[114, 235, 191]]), array([[ 63, 145, 171]]), array([[ 16, 97, 173]])]
However, print(np.split(a, 3)) will raise an error since 4/3 is not an integer.

This is what I have used to iterate through. I use b.next() method to generate the indices, then pass the output to slice a numpy array, for example a[b.next()] where a is a numpy array.
class Batch():
def __init__(self, total, batch_size):
self.total = total
self.batch_size = batch_size
self.current = 0
def next(self):
max_index = self.current + self.batch_size
indices = [i if i < self.total else i - self.total
for i in range(self.current, max_index)]
self.current = max_index % self.total
return indices
b = Batch(10, 3)
print(b.next()) # [0, 1, 2]
print(b.next()) # [3, 4, 5]
print(b.next()) # [6, 7, 8]
print(b.next()) # [9, 0, 1]
print(b.next()) # [2, 3, 4]
print(b.next()) # [5, 6, 7]

Improving previous answer, to split based on batch size, you can use:
def split_by_batchsize(arr, batch_size):
return np.array_split(arr, (arr.shape[0]/batch_size)+1)
or with extra safety:
def split_by_batch_size(arr, batch_size):
nbatches = arr.shape[0]//batch_size
if nbatches != arr.shape[0]/batch_size:
nbatches += 1
return np.array_split(arr, nbatches)
example:
import numpy as np
ncols = 17
batch_size= 2
split_by_batchsize(np.random.random((ncols, 2)), batch_size)
# [array([[0.60482079, 0.81391257],
# [0.00175093, 0.25126441]]),
# array([[0.48591974, 0.77793401],
# [0.72128946, 0.3606879 ]]),
# array([[0.95649328, 0.24765806],
# [0.78844782, 0.56304567]]),
# array([[0.07310456, 0.76940976],
# [0.92163079, 0.90803845]]),
# array([[0.77838703, 0.98460593],
# [0.88397437, 0.39227769]]),
# array([[0.87599421, 0.7038426 ],
# [0.19780976, 0.12763436]]),
# array([[0.14263759, 0.9182901 ],
# [0.40523958, 0.0716843 ]]),
# array([[0.9802908 , 0.01067808],
# [0.53095143, 0.74797636]]),
# array([[0.7596607 , 0.97923229]])]
Sadly, simple iteration is faster than this fancy method. Thus, I did not suggest you to use this approach.
batch_size = 3
nrows = 1000
arr = np.random.random((nrows, 2))
%%timeit
for i in range((arr.shape[0] // batch_size) + 1):
idx = i*batch_size
foo = arr[idx:idx+batch_size,:]
# 345 µs ± 119 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%%timeit
for foo in split_by_batch_size(arr, batch_size):
pass
# 1.84 ms ± 377 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
The speed different seems came from np.array_split create list of array first.

do like this:
a = [[1, 2, 3],[4, 5, 6],
[7, 8, 9],[10, 11, 12]]
b = a[0:2]
c = a[2:4]

Add dimensions to an array from a list

The structure of my array 'cama' is the following:
shape(cama)
>>>(365, 720, 1440)
And the shape of my 'lon_list' is the following:
shape(lon_list)
>>>(1440,)
What I want is to add or append lon_list to cama. So that I end up with an array with the following dimensions:
shape(new_cama)
>>>(365, 720, 1440, 1440)
I've tried:
new_cama = np.concatenate((cama, lon_list))
ValueError: all the input arrays must have same number of dimensions
Any suggestions?

Dimensions are multiplicative. This means if you have:
a = [[1, 2, 3], [4, 5, 6]]
You have a (2 x 3) array of 6 elements. If you want to add another dimension, say b=[10, 20, 30, 40], you will end up with an array of (2 x 3 x 4) = 24 elements, so you need to provide a way to complete these missing (24 - 4 - 6) = 14 elements. You cannot 'append' a dimension: increasing the dimension means appending newdimiension - 1 elements in all dimensions.
So what can we do with shapes like this ? Well, you can broadcast your arrays to match the corresponding dimension and combine them in some way:
In[52]: a = np.array([[1,2,3], [4,5,6]])
In[53]: b = np.array([10, 20, 30, 40])
In[54]: a
Out[54]:
array([[1, 2, 3],
[4, 5, 6]])
In[55]: b
Out[55]: array([10, 20, 30, 40])
In[56]: c = a[:, :, None] * b[None, None, :]
In[57]: c
Out[57]:
array([[[ 10, 20, 30, 40],
[ 20, 40, 60, 80],
[ 30, 60, 90, 120]],
[[ 40, 80, 120, 160],
[ 50, 100, 150, 200],
[ 60, 120, 180, 240]]])
In[58]: c.shape
Out[58]: (2L, 3L, 4L)
Here I have multiplied them, but it is of course possible to use any other combination: the key thing to understand is that:
You can only apply operations to arrays of similar shapes
You can 'emulate' bigger shapes via broadcasting.
This way, we can combine a (2, 3) and a (4,) arrays by 'viewing' them both as (2, 3, 4) arrays and combining them in some way. But you have to have them make sense as arrays of similar shapes.
Alternatively, you can concatenate arrays, if all their dimensions but one differ. Say you have a (2, 3, 4, 5) array and a (2, 3, 7, 5) array: then it makes sense to concatenate them in a (2, 3, 4+7, 5) array. You can again do that by 'emulating' shapes (basically by repeating the pattern along any missing dimension) but from your question, it's unclear this is what you're trying to achieve...