We Keep Coding

How does slice indexing work in numpy array

Suppose we have an array
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
Now I have below
row_r1 = a[1, :]
row_r2 = a[1:2, :]
print(row_r1.shape)
print(row_r2.shape)
I don't understand why row_r1.shape is (4,) and row_r2.shape is (1,4)
Shouldn't their shape all equal to (4,)?

I like to think of it this way. The first way row[1, :], states go get me all values on row 1 like this:
Returning:
array([5, 6, 7, 8])
shape
(4,) Four values in a numpy array.
Where as the second row[1:2, :], states go get me a slice of data between index 1 and index 2:
Returning:
array([[5, 6, 7, 8]]) Note: the double brackets
shape
(1,4) Four values in on one row in a np.array.

Their shapes are different because they aren't the same thing. You can verify by printing them:
import numpy as np
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
row_r1 = a[1, :]
row_r2 = a[1:2, :]
print("{} is shape {}".format(row_r1, row_r1.shape))
print("{} is shape {}".format(row_r2, row_r2.shape))
Yields:
[5 6 7 8] is shape (4,)
[[5 6 7 8]] is shape (1, 4)
This is because indexing will return an element, whereas slicing will return an array. You can however manipulate them to be the same thing using the .resize() function available to numpy arrays.
The code:
import numpy as np
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
row_r1 = a[1, :]
row_r2 = a[1:2, :]
print("{} is shape {}".format(row_r1, row_r1.shape))
print("{} is shape {}".format(row_r2, row_r2.shape))
# Now resize row_r1 to be the same shape
row_r1.resize((1, 4))
print("{} is shape {}".format(row_r1, row_r1.shape))
print("{} is shape {}".format(row_r2, row_r2.shape))
Yields
[5 6 7 8] is shape (4,)
[[5 6 7 8]] is shape (1, 4)
[[5 6 7 8]] is shape (1, 4)
[[5 6 7 8]] is shape (1, 4)
Showing that you are in fact now dealing with the same shaped object. Hope this helps clear it up!

Array size difference Python

My question is about Python array shape.
What is the difference between array size (2, ) and (2, 1)?
I tried to add those two arrays together. However, I got an error as follows:
Non-broadcastable output operant with shape (2, ) doesn't match the broadcast shape (2, 2)

There is no difference in the raw memory. But logically, one is a one-dimensional array of two values, the other is a 2D array (where one of the dimensions just happens to be size 1).
The logical distinction is important to numpy; when you try to add them, it wants to make a new 2x2 array where the top row is the sum of the (2, 1) array's top "row" with each value in the (2,) array. If you use += to do that though, you're indicating that you expect to be able to modify the (2,) array in place, which is not possible without resizing (which numpy won't do). If you change your code from:
arr1 += arr2
to:
arr1 = arr1 + arr2
it will happily create a new (2, 2) array. Or if the goal was that the 2x1 array should act like a flat 1D array, you can flatten it:
alreadyflatarray += twodarray.flatten()

(2,) is an unidimensional array, (2,1) is a matrix with only one column
You can easily see the difference by crating arrays full of zeros using np.zero passing the desired shape:
>>> np.zeros((2,))
array([0., 0.])
>>> np.zeros((2,1))
array([[0.],
[0.]])

#yx131, you can have a look at the below code to just have a clear picture about tuples and it's use in defining the shape of numpy arrays.
Note: Do not forget to see the code below as it has explanation of the problems related to Broadcasting in numpy.
Also check numpy's broadcasting rule at
https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html.
There's a difference between (2) and (2,). The first one is a literal value 2 where as the 2nd one is a tuple.
(2,) is 1 item tuple and (2, 2) is 2 items tuple. It is clear in the code example.
Note: In case of numpy arrays, (2,) denotes shape of 1 dimensional array of 2 items and (2, 2) denotes the shape of 2 dimensional array (matrix) with 2 rows and 2 colums. If you want to add 2 arrays then their shape should be same.
v = (2) # Assignment of value 2
t = (2,) # Comma is necessary at the end of item to define 1 item tuple, it is not required in case of list
t2 = (2, 1) # 2 item tuple
t3 = (3, 4) # 2 item tuple
print(v, type(v))
print(t, type(t))
print(t2, type(t2))
print(t3, type(t3))
print(t + t2)
print(t2 + t3)
"""
2 <class 'int'>
(2,) <class 'tuple'>
(2, 1) <class 'tuple'>
(3, 4) <class 'tuple'>
(2, 2, 1)
(2, 1, 3, 4)
"""
Now, let's have a look at the below code to figure out the error related to broadcasting. It's all related to dimensions.
# Python 3.5.2
import numpy as np
arr1 = np.array([1, 4]);
arr2 = np.array([7, 6, 3, 8]);
arr3 = np.array([3, 6, 2, 1]);
print(arr1, ':', arr1.shape)
print(arr2, ":", arr2.shape)
print(arr3, ":", arr3.shape)
print ("\n")
"""
[1 4] : (2,)
[7 6 3 8] : (4,)
[3 6 2 1] : (4,)
"""
# Changing shapes (dimensions)
arr1.shape = (2, 1)
arr2.shape = (2, 2)
arr3.shape = (2, 2)
print(arr1, ':', arr1.shape)
print(arr2, ":", arr2.shape)
print(arr3, ":", arr3.shape)
print("\n")
print(arr1 + arr2)
"""
[[1]
[4]] : (2, 1)
[[7 6]
[3 8]] : (2, 2)
[[3 6]
[2 1]] : (2, 2)
[[ 8 7]
[ 7 12]]
"""
arr1.shape = (2, )
print(arr1, arr1.shape)
print(arr1 + arr2)
"""
[1 4] (2,)
[[ 8 10]
[ 4 12]]
"""
# Code with error(Broadcasting related)
arr2.shape = (4,)
print(arr1+arr2)
"""
Traceback (most recent call last):
File "source_file.py", line 53, in <module>
print(arr1+arr2)
ValueError: operands could not be broadcast together
with shapes (2,) (4,)
"""
So in your case, the problem is related to the mismatched dimensions (acc. to numpy's broadcasting ) to be added. Thanks.

Make an array that has shape (2,)
In [164]: a = np.array([3,6])
In [165]: a
Out[165]: array([3, 6])
In [166]: a.shape
Out[166]: (2,)
In [167]: a.reshape(2,1)
Out[167]:
array([[3],
[6]])
In [168]: a.reshape(1,2)
Out[168]: array([[3, 6]])
The first displays like a simple list [3,6]. The second as a list with 2 nested lists. The third as a list with one nested list of 2 items. So there is a consistent relation between shape and list nesting.
In [169]: a + a
Out[169]: array([ 6, 12]) # shape (2,)
In [170]: a + a.reshape(1,2)
Out[170]: array([[ 6, 12]]) # shape (1,2)
In [171]: a + a.reshape(2,1)
Out[171]:
array([[ 6, 9], # shape (2,2)
[ 9, 12]])
Dimensions behave as:
(2,) + (2,) => (2,)
(2,) + (1,2) => (1,2) + (1,2) => (1,2)
(2,) + (2,1) => (1,2) + (2,1) => (2,2) + (2,2) => (2,2)
That is a lower dimensional array can be expanded to the matching number of dimensions with the addition of leading size 1 dimensions.
And size 1 dimensions can be changed to match the corresponding dimension.
I suspect you got the error when doing a a += ... (If so you should have stated that clearly.)
In [172]: a += a
In [173]: a += a.reshape(1,2)
....
ValueError: non-broadcastable output operand with shape (2,)
doesn't match the broadcast shape (1,2)
In [175]: a += a.reshape(2,1)
...
ValueError: non-broadcastable output operand with shape (2,)
doesn't match the broadcast shape (2,2)
With the a+=... addition, the result shape is fixed at (2,), the shape of a. But as noted above the two additions generate (1,2) and (2,2) results, which aren't compatible with (2,).
The same reasoning can explain these additions and errors:
In [176]: a1 = a.reshape(1,2)
In [177]: a1 += a
In [178]: a1
Out[178]: array([[12, 24]])
In [179]: a2 = a.reshape(2,1)
In [180]: a2 += a
...
ValueError: non-broadcastable output operand with shape (2,1)
doesn't match the broadcast shape (2,2)
In [182]: a1 += a2
...
ValueError: non-broadcastable output operand with shape (1,2)
doesn't match the broadcast shape (2,2)

In numpy.sum() there is parameter called "keepdims". What does it do?

In numpy.sum() there is parameter called keepdims. What does it do?
As you can see here in the documentation:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.sum.html
numpy.sum(a, axis=None, dtype=None, out=None, keepdims=False)[source]
Sum of array elements over a given axis.
Parameters:
...
keepdims : bool, optional
If this is set to True, the axes which are reduced are left in the result as
dimensions with size one. With this option, the result will broadcast
correctly against the input array.
...

#Ney
#hpaulj is correct, you need to experiment, but I suspect you don't realize that summation for some arrays can occur along axes. Observe the following which reading the documentation
>>> a
array([[0, 0, 0],
[0, 1, 0],
[0, 2, 0],
[1, 0, 0],
[1, 1, 0]])
>>> np.sum(a, keepdims=True)
array([[6]])
>>> np.sum(a, keepdims=False)
6
>>> np.sum(a, axis=1, keepdims=True)
array([[0],
[1],
[2],
[1],
[2]])
>>> np.sum(a, axis=1, keepdims=False)
array([0, 1, 2, 1, 2])
>>> np.sum(a, axis=0, keepdims=True)
array([[2, 4, 0]])
>>> np.sum(a, axis=0, keepdims=False)
array([2, 4, 0])
You will notice that if you don't specify an axis (1st two examples), the numerical result is the same, but the keepdims = True returned a 2D array with the number 6, whereas, the second incarnation returned a scalar.
Similarly, when summing along axis 1 (across rows), a 2D array is returned again when keepdims = True.
The last example, along axis 0 (down columns), shows a similar characteristic... dimensions are kept when keepdims = True.
Studying axes and their properties is critical to a full understanding of the power of NumPy when dealing with multidimensional data.

An example showing keepdims in action when working with higher dimensional arrays. Let's see how the shape of the array changes as we do different reductions:
import numpy as np
a = np.random.rand(2,3,4)
a.shape
# => (2, 3, 4)
# Note: axis=0 refers to the first dimension of size 2
# axis=1 refers to the second dimension of size 3
# axis=2 refers to the third dimension of size 4
a.sum(axis=0).shape
# => (3, 4)
# Simple sum over the first dimension, we "lose" that dimension
# because we did an aggregation (sum) over it
a.sum(axis=0, keepdims=True).shape
# => (1, 3, 4)
# Same sum over the first dimension, but instead of "loosing" that
# dimension, it becomes 1.
a.sum(axis=(0,2)).shape
# => (3,)
# Here we "lose" two dimensions
a.sum(axis=(0,2), keepdims=True).shape
# => (1, 3, 1)
# Here the two dimensions become 1 respectively

advanced 3d indexind in theano

My question is very similar to
Indexing tensor with index matrix in theano?
except that I have 3 dimensions. At first I want to got it working in numpy. With 2 dimensions there is no problem:
>>> idx = np.random.randint(3, size=(4, 2, 3))
>>> d = np.random.rand(4*2*3).reshape((4, 2, 3))
>>> d[1]
array([[ 0.37057415, 0.73066383, 0.76399376],
[ 0.12155831, 0.12552545, 0.87648523]])
>>> idx[1]
array([[2, 0, 1],
[2, 2, 2]])
>>> d[1][np.arange(d.shape[1])[:, np.newaxis], idx[1]]
array([[ 0.76399376, 0.37057415, 0.73066383],
[ 0.87648523, 0.87648523, 0.87648523]]) #All correct
But I have no idea how to make it works with all 3 dimensions. Example of failed try:
>>> d[np.arange(d.shape[0])[:, np.newaxis], np.arange(d.shape[1]), idx]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (4,1) (2,) (4,2,3)

Does this work?
d[
np.arange(d.shape[0])[:, np.newaxis, np.newaxis],
np.arange(d.shape[1])[:, np.newaxis],
idx
]
You need the index arrays to collectively have broadcastable dimensions

Numpy concatenate 2D arrays with 1D array

I am trying to concatenate 4 arrays, one 1D array of shape (78427,) and 3 2D array of shape (78427, 375/81/103). Basically this are 4 arrays with features for 78427 images, in which the 1D array only has 1 value for each image.
I tried concatenating the arrays as follows:
>>> print X_Cscores.shape
(78427, 375)
>>> print X_Mscores.shape
(78427, 81)
>>> print X_Tscores.shape
(78427, 103)
>>> print X_Yscores.shape
(78427,)
>>> np.concatenate((X_Cscores, X_Mscores, X_Tscores, X_Yscores), axis=1)
This results in the following error:
Traceback (most recent call last):
File "", line 1, in
ValueError: all the input arrays must have same number of dimensions
The problem seems to be the 1D array, but I can't really see why (it also has 78427 values). I tried to transpose the 1D array before concatenating it, but that also didn't work.
Any help on what's the right method to concatenate these arrays would be appreciated!

Try concatenating X_Yscores[:, None] (or X_Yscores[:, np.newaxis] as imaluengo suggests). This creates a 2D array out of a 1D array.
Example:
A = np.array([1, 2, 3])
print A.shape
print A[:, None].shape
Output:
(3,)
(3,1)

I am not sure if you want something like:
a = np.array( [ [1,2],[3,4] ] )
b = np.array( [ 5,6 ] )
c = a.ravel()
con = np.concatenate( (c,b ) )
array([1, 2, 3, 4, 5, 6])
OR
np.column_stack( (a,b) )
array([[1, 2, 5],
[3, 4, 6]])
np.row_stack( (a,b) )
array([[1, 2],
[3, 4],
[5, 6]])

You can try this one-liner:
concat = numpy.hstack([a.reshape(dim,-1) for a in [Cscores, Mscores, Tscores, Yscores]])
The "secret" here is to reshape using the known, common dimension in one axis, and -1 for the other, and it automatically matches the size (creating a new axis if needed).