We Keep Coding

Efficiently slicing an array using indexes from another array

(I apologize in advance if this is a duplicate question, though I looked at many similar questions on SO but didn't find a matching solution)
Suppose you have an array
A = np.array([
[0, 1, 2],
[3, 4, 5],
[6, 7, 8]
])
and another array
I = np.array([1, 1, 2])
For each row in A, I want to get the i-th element of it, where i is the row-th element of I.
In this case, the output I'd like to have is array([1, 4, 8]).
My most intuitive attempt to do so is:
A[:, I]
then I figured that the desired output is actually the diagonal of it, so A[:, I].diagonal() would do the trick.
But it feels that there's some waste of space and time by doing this way, because it requires an intermediate "big" matrix, which diagonal will be extracted from.
Is there a more efficient to perform this slicing?

This would do the trick:
res = A[np.arange(A.shape[0]), I]

Slicing with :-1 and None - What does each of the statement mean?

I came across a code snippet where I could not understand two of the statements, though I could see the end result of each.
I will create a variable before giving the statements:
train = np.random.random((10,100))
One of them read as :
train = train[:-1, 1:-1]
What does this slicing mean? How to read this? I know that that -1 in slicing denotes from the back. But I cannot understand this.
Another statement read as follows:
la = [0.2**(7-j) for j in range(1,t+1)]
np.array(la)[:,None]
What does slicing with None as in [:,None] mean?
For the above two statements, along with how each statement is read, it will be helpful to have an alternative method along, so that I understand it better.

One of Python's strengths is its uniform application of straightforward principles. Numpy indexing, like all indexing in Python, passes a single argument to the indexed object's (i.e., the array's) __getitem__ method, and numpy arrays were one of the primary justifications for the slicing mechanism (or at least one of its very early uses).
When I'm trying to understand new behaviours I like to start with a concrete and comprehensible example, so rather than 10x100 random values I'll start with a one-dimensional 4-element vector and work up to 3x4, which should be big enough to understand what's going on.
simple = np.array([1, 2, 3, 4])
train = np.array([[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12]])
The interpreter shows these as
array([1, 2, 3, 4])
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
The expression simple[x] is equivalent to (which is to say the interpreter ends up executing) simple.__getitem__(x) under the hood - note this call takes a single argument.
The numpy array's __getitem__ method implements indexing with an integer very simply: it selects a single element from the first dimension. So simple[1] is 2, and train[1] is array([5, 6, 7, 8]).
When __getitem__ receives a tuple as an argument (which is how Python's syntax interprets expressions like array[x, y, z]) it applies each element of the tuple as an index to successive dimensions of the indexed object. So result = train[1, 2] is equivalent (conceptually - the code is more complex in implementation) to
temp = train[1] # i.e. train.__getitem__(1)
result = temp[2] # i.e. temp.__getitem__(2)
and sure enough we find that result comes out at 7. You could think of array[x, y, z] as equivalent to array[x][y][z].
Now we can add slicing to the mix. Expressions containing a colon can be regarded as slice literals (I haven't seen a better name for them), and the interpreter creates slice objects for them. As the documentation notes, a slice object is mostly a container for three values, start, stop and slice, and it's up to each object's __getitem__ method how it interprets them. You might find this question helpful to understand slicing further.
With what you now know, you should be able to understand the answer to your first question.
result = train[:-1, 1:-1]
will call train.__getitem__ with a two-element tuple of slices. This is equivalent to
temp = train[:-1]
result = temp[..., 1:-1]
The first statement can be read as "set temp to all but the last row of train", and the second as "set result to all but the first and last columns of temp". train[:-1] is
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
and applying the [1:-1] subscripting to the second dimension of that array gives
array([[2, 3],
[6, 7]])
The ellipsis on the first dimension of the temp subscript says "pass everything," so the subscript expression[...]can be considered equivalent to[:]. As far as theNonevalues are concerned, a slice has a maximum of three data points: _start_, _stop_ and _step_. ANonevalue for any of these gives the default value, which is0for _start_, the length of the indexed object for _stop_, and1for _step. Sox[None:None:None]is equivalent tox[0:len(x):1]which is equivalent tox[::]`.
With this knowledge under your belt you should stand a bit more chance of understanding what's going on.

Rationale for numpy.split returning a list and not an array

I was surprised that numpy.split yields a list and not an array. I would have thought it would be better to return an array, since numpy has put a lot of work into making arrays more useful than lists. Can anyone justify numpy returning a list instead of an array? Why would that be a better programming decision for the numpy developers to have made?

A comment pointed out that if the slit is uneven, the result can't be a array, at least not one that has the same dtype. At best it would be an object dtype.
But lets consider the case of equal length subarrays:
In [124]: x = np.arange(10)
In [125]: np.split(x,2)
Out[125]: [array([0, 1, 2, 3, 4]), array([5, 6, 7, 8, 9])]
In [126]: np.array(_) # make an array from that
Out[126]:
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
But we can get the same array without split - just reshape:
In [127]: x.reshape(2,-1)
Out[127]:
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
Now look at the code for split. It just passes the task to array_split. Ignoring the details about alternative axes, it just does
sub_arys = []
for i in range(Nsections):
# st and end from `div_points
sub_arys.append(sary[st:end])
return sub_arys
In other words, it just steps through array and returns successive slices. Those (often) are views of the original.
So split is not that sophisticate a function. You could generate such a list of subarrays yourself without a lot of numpy expertise.
Another point. Documentation notes that split can be reversed with an appropriate stack. concatenate (and family) takes a list of arrays. If give an array of arrays, or a higher dim array, it effectively iterates on the first dimension, e.g. concatenate(arr) => concatenate(list(arr)).

Actually you are right it returns a list
import numpy as np
a=np.random.randint(1,30,(2,2))
b=np.hsplit(a,2)
type(b)
it will return type(b) as list so, there is nothing wrong in the documentation, i also first thought that the documentation is wrong it doesn't return a array, but when i checked
type(b[0])
type(b[1])
it returned type as ndarray.
it means it returns a list of ndarrary's.

Seemingly inconsistent slicing behavior in numpy arrays

I ran across something that seemed to me like inconsistent behavior in Numpy slices. Specifically, please consider the following example:
import numpy as np
a = np.arange(9).reshape(3,3) # a 2d numpy array
y = np.array([1,2,2]) # vector that will be used to index the array
b = a[np.arange(len(a)),y] # a vector (what I want)
c = a[:,y] # a matrix ??
I wanted to obtain a vector such that the i-th element is a[i,y[i]]. I tried two things (b and c above) and was surprised that b and c are not the same... in fact one is a vector and the other is a matrix! I was under the impression that : was shorthand for "all elements" but apparently the meaning is somewhat more subtle.
After trial and error I somewhat understand the difference now (b == np.diag(c)), but would appreciate clarification on why they are different, what exactly using : implies, and how to understand when to use either case.
Thanks!

It's hard to understand advanced indexing (with lists or arrays) without understanding broadcasting.
In [487]: a=np.arange(9).reshape(3,3)
In [488]: idx = np.array([1,2,2])
Index with a (3,) and (3,) producing shape (3,) result:
In [489]: a[np.arange(3),idx]
Out[489]: array([1, 5, 8])
Index with (3,1) and (3,), result is (3,3)
In [490]: a[np.arange(3)[:,None],idx]
Out[490]:
array([[1, 2, 2],
[4, 5, 5],
[7, 8, 8]])
The slice : does basically the same thing. There are subtle differences, but here it's the same.
In [491]: a[:,idx]
Out[491]:
array([[1, 2, 2],
[4, 5, 5],
[7, 8, 8]])
ix_ does the same thing, converting the (3,) & (3,) to (3,1) and (1,3):
In [492]: np.ix_(np.arange(3),idx)
Out[492]:
(array([[0],
[1],
[2]]), array([[1, 2, 2]]))
A broadcasted sum might help visualize the two cases:
In [495]: np.arange(3)*10+idx
Out[495]: array([ 1, 12, 22])
In [496]: np.sum(np.ix_(np.arange(3)*10,idx),axis=0)
Out[496]:
array([[ 1, 2, 2],
[11, 12, 12],
[21, 22, 22]])

When you pass
np.arange(len(a)), y
You can view the result as being all the indexed pairs for the zipped elements you passed. In this case, indexing by np.arange(len(a)) and y
np.arange(len(a))
# [0, 1, 2]
y
# [1, 2, 2]
effectively takes elements: (0, 1), (1, 2), and (2, 2).
print(a[0, 1], a[1, 2], a[2, 2]) # 0th, 1st, 2nd elements from each indexer
# 1 5 8
In the second case, you take the entire slice along the first dimension. (Nothing before the colon.) So this is all elements along the 0th axis. You then specify with y that you want the 1st, 2nd, and 2nd element along each row. (0-indexed.)
As you pointed out, it may seem a bit unintuitive that the results are different given that the individual elements of the slice are equivalent:
a[:] == a[np.arange(len(a))]
and
a[:y] == a[:y]
However, NumPy advanced indexing cares what type of data structure you pass when indexing (tuples, integers, etc). Things can become hairy very quickly.
The detail behind that is this: first consider all NumPy indexing to be of the general form x[obj], where obj is the evaluation of whatever you passed. How NumPy "behaves" depends on what type of object obj is:
Advanced indexing is triggered when the selection object, obj, is a
non-tuple sequence object, an ndarray (of data type integer or bool),
or a tuple with at least one sequence object or ndarray (of data type
integer or bool).
...
The definition of advanced indexing means that x[(1,2,3),] is
fundamentally different than x[(1,2,3)]. The latter is equivalent to
x[1,2,3] which will trigger basic selection while the former will
trigger advanced indexing. Be sure to understand why this occurs.
In your first case, obj = np.arange(len(a)),y, a tuple that fits the bill in bold above. This triggers advanced indexing and forces the behavior described above.
As for the second case, [:,y]
When there is at least one slice (:), ellipsis (...) or np.newaxis in
the index (or the array has more dimensions than there are advanced
indexes), then the behaviour can be more complicated. It is like
concatenating the indexing result for each advanced index element.
Demonstrated:
# Concatenate the indexing result for each advanced index element.
np.vstack((a[0, y], a[1, y], a[2, y]))

Function application over numpy's matrix row/column

I am using Numpy to store data into matrices. Coming from R background, there has been an extremely simple way to apply a function over row/columns or both of a matrix.
Is there something similar for python/numpy combination? It's not a problem to write my own little implementation but it seems to me that most of the versions I come up with will be significantly less efficient/more memory intensive than any of the existing implementation.
I would like to avoid copying from the numpy matrix to a local variable etc., is that possible?
The functions I am trying to implement are mainly simple comparisons (e.g. how many elements of a certain column are smaller than number x or how many of them have absolute value larger than y).

Almost all numpy functions operate on whole arrays, and/or can be told to operate on a particular axis (row or column).
As long as you can define your function in terms of numpy functions acting on numpy arrays or array slices, your function will automatically operate on whole arrays, rows or columns.
It may be more helpful to ask about how to implement a particular function to get more concrete advice.
Numpy provides np.vectorize and np.frompyfunc to turn Python functions which operate on numbers into functions that operate on numpy arrays.
For example,
def myfunc(a,b):
if (a>b): return a
else: return b
vecfunc = np.vectorize(myfunc)
result=vecfunc([[1,2,3],[5,6,9]],[7,4,5])
print(result)
# [[7 4 5]
# [7 6 9]]
(The elements of the first array get replaced by the corresponding element of the second array when the second is bigger.)
But don't get too excited; np.vectorize and np.frompyfunc are just syntactic sugar. They don't actually make your code any faster. If your underlying Python function is operating on one value at a time, then np.vectorize will feed it one item at a time, and the whole
operation is going to be pretty slow (compared to using a numpy function which calls some underlying C or Fortran implementation).
To count how many elements of column x are smaller than a number y, you could use an expression such as:
(array['x']<y).sum()
For example:
import numpy as np
array=np.arange(6).view([('x',np.int),('y',np.int)])
print(array)
# [(0, 1) (2, 3) (4, 5)]
print(array['x'])
# [0 2 4]
print(array['x']<3)
# [ True True False]
print((array['x']<3).sum())
# 2

Selecting elements from a NumPy array based on one or more conditions is straightforward using NumPy's beautifully dense syntax:
>>> import numpy as NP
>>> # generate a matrix to demo the code
>>> A = NP.random.randint(0, 10, 40).reshape(8, 5)
>>> A
array([[6, 7, 6, 4, 8],
[7, 3, 7, 9, 9],
[4, 2, 5, 9, 8],
[3, 8, 2, 6, 3],
[2, 1, 8, 0, 0],
[8, 3, 9, 4, 8],
[3, 3, 9, 8, 4],
[5, 4, 8, 3, 0]])
how many elements in column 2 are greater than 6?
>>> ndx = A[:,1] > 6
>>> ndx
array([False, True, False, False, True, True, True, True], dtype=bool)
>>> NP.sum(ndx)
5
how many elements in last column of A have absolute value larger than 3?
>>> A = NP.random.randint(-4, 4, 40).reshape(8, 5)
>>> A
array([[-4, -1, 2, 0, 3],
[-4, -1, -1, -1, 1],
[-1, -2, 2, -2, 3],
[ 1, -4, -1, 0, 0],
[-4, 3, -3, 3, -1],
[ 3, 0, -4, -1, -3],
[ 3, -4, 0, -3, -2],
[ 3, -4, -4, -4, 1]])
>>> ndx = NP.abs(A[:,-1]) > 3
>>> NP.sum(ndx)
0
how many elements in the first two rows of A are greater than or equal to 2?
>>> ndx = A[:2,:] >= 2
>>> NP.sum(ndx.ravel()) # 'ravel' just flattens ndx, which is originally 2D (2x5)
2
NumPy's indexing syntax is pretty close to R's; given your fluency in R, here are the key differences between R and NumPy in this context:
NumPy indices are zero-based, in R, indexing begins with 1
NumPy (like Python) allows you to index from right to left using negative indices--e.g.,
# to get the last column in A
A[:, -1],
# to get the penultimate column in A
A[:, -2]
# this is a big deal, because in R, the equivalent expresson is:
A[, dim(A)[0]-2]
NumPy uses colon ":" notation to denote "unsliced", e.g., in R, to
get the first three rows in A, you would use, A[1:3, ]. In NumPy, you
would use A[0:2, :] (in NumPy, the "0" is not necessary, in fact it
is preferable to use A[:2, :]

I also come from a more R background, and bumped into the lack of a more versatile apply which could take short customized functions. I've seen the forums suggesting using basic numpy functions because many of them handle arrays. However, I've been getting confused over the way "native" numpy functions handle array (sometimes 0 is row-wise and 1 column-wise, sometimes the opposite).
My personal solution to more flexible functions with apply_along_axis was to combine them with the implicit lambda functions available in python. Lambda functions should very easy to understand for the R minded who uses a more functional programming style, like in R functions apply, sapply, lapply, etc.
So for example I wanted to apply standardisation of variables in a matrix. Tipically in R there's a function for this (scale) but you can also build it easily with apply:
(R code)
apply(Mat,2,function(x) (x-mean(x))/sd(x) )
You see how the body of the function inside apply (x-mean(x))/sd(x) is the bit we can't type directly for the python apply_along_axis. With lambda this is easy to implement FOR ONE SET OF VALUES, so:
(Python)
import numpy as np
vec=np.random.randint(1,10,10) # some random data vector of integers
(lambda x: (x-np.mean(x))/np.std(x) )(vec)
Then, all we need is to plug this inside the python apply and pass the array of interest through apply_along_axis
Mat=np.random.randint(1,10,3*4).reshape((3,4)) # some random data vector
np.apply_along_axis(lambda x: (x-np.mean(x))/np.std(x),0,Mat )
Obviously, the lambda function could be implemented as a separate function, but I guess the whole point is to use rather small functions contained within the line where apply originated.
I hope you find it useful !

Pandas is very useful for this. For instance, DataFrame.apply() and groupby's apply() should help you.