I have a dataframe df with a single column that contains arrays of length 3. Now, I want to transform this column to a numpy array of the correct shape. However, applying np.reshape does not work. How can I do this?
Here is a brief example:
import pandas as pd
import numpy as np
df = pd.DataFrame(columns=['col'])
for i in range(10):
df.loc[i,'col'] = np.zeros(3)
arr = np.array(df['col'])
np.reshape(arr, (10,3)) # This does not work
Here are two approaches using np.vstack and np.concatenate -
np.vstack(df.col)
np.concatenate(df.col).reshape(df.shape[0],-1) # for performance
For best performance, we could use the underlying data with df.col.values instead.
Sample run -
In [116]: df
Out[116]:
col
0 [7, 5, 2]
1 [1, 1, 3]
2 [6, 1, 4]
3 [7, 0, 0]
4 [8, 8, 0]
5 [7, 8, 0]
6 [0, 5, 8]
7 [8, 3, 1]
8 [6, 6, 8]
9 [8, 2, 3]
In [117]: np.vstack(df.col)
Out[117]:
array([[7, 5, 2],
[1, 1, 3],
[6, 1, 4],
[7, 0, 0],
[8, 8, 0],
[7, 8, 0],
[0, 5, 8],
[8, 3, 1],
[6, 6, 8],
[8, 2, 3]])
Related
I have tried to find a similar question but so far it seems only half my question can be answered.
I have a 2D numpy array, e.g.:
a= np.array([[6, 4, 5],
[4, 7, 8],
[2, 8, 9]])
And i also have 2 further numpy arrays, indicating the rows, and columns where i would like to rearrange (or not):
rows= np.array([[0, 0, 0],
[1, 0, 1],
[2, 2, 2]])
cols= np.array([[0, 1, 2],
[0, 0, 2],
[0, 1, 2]])
now i would like to rearrange the array "a" based on these indices, so that the result is:
result= np.array([[6, 4, 5],
[4, 6, 8],
[2, 8, 9]])
Doing this only for columns or only for rows is easy, e.g. see this Thread:
np.array(list(map(lambda x, y: y[x], cols, a)))
This is a typical case of fancy/array indexing:
result = a[rows, cols]
Output:
array([[6, 4, 5],
[4, 6, 8],
[2, 8, 9]])
Let's say I have data structured in a 2D array like this:
[[1, 3, 4, 6],
[1, 4, 8, 2],
[1, 3, 2, 9],
[2, 2, 4, 8],
[2, 4, 9, 1],
[2, 2, 9, 3]]
The first column denotes a third dimension, so I want to convert this to the following 3D array:
[[[3, 4, 6],
[4, 8, 2],
[3, 2, 9]],
[[2, 4, 8],
[4, 9, 1],
[2, 9, 3]]]
Is there a built-in numpy function to do this?
You can try code below:
import numpy as np
array = np.array([[1, 3, 4, 6],
[1, 4, 8, 2],
[1, 3, 2, 9],
[2, 2, 4, 8],
[2, 4, 9, 1],
[2, 2, 9, 3]])
array = np.delete(array, 0, 1)
array.reshape(2,3,-1)
Output
array([[[3, 4, 6],
[4, 8, 2],
[3, 2, 9]],
[[2, 4, 8],
[4, 9, 1],
[2, 9, 3]]])
However, this code can be used when you are aware of the array's shape. But if you are sure that the number of columns in the array is a multiple of 3, you can simply use code below to show the array in the desired format.
array.reshape(array.shape[0]//3,3,-3)
Use numpy array slicing with reshape function.
import numpy as np
arr = [[1, 3, 4, 6],
[1, 4, 8, 2],
[1, 3, 2, 9],
[2, 2, 4, 8],
[2, 4, 9, 1],
[2, 2, 9, 3]]
# convert the list to numpy array
arr = np.array(arr)
# remove first column from numpy array
arr = arr[:,1:]
# reshape the remaining array to desired shape
arr = arr.reshape(len(arr)//3,3,-1)
print(arr)
Output:
[[[3 4 6]
[4 8 2]
[3 2 9]]
[[2 4 8]
[4 9 1]
[2 9 3]]]
You list a non numpy array. I am unsure if you are just suggesting numpy as a means to get a non numpy result, or you are actually looking for a numpy array as result. If you don't actually need numpy, you could do something like this:
arr = [[1, 3, 4, 6],
[1, 4, 8, 2],
[1, 3, 2, 9],
[2, 2, 4, 8],
[2, 4, 9, 1],
[2, 2, 9, 3]]
# Length of the 3rd and 2nd dimension.
nz = arr[-1][0] + (arr[0][0]==0)
ny = int(len(arr)/nz)
res = [[arr[ny*z_idx+y_idx][1:] for y_idx in range(ny)] for z_idx in range(nz)]
OUTPUT:
[[[3, 4, 6], [4, 8, 2], [3, 2, 9]], [[2, 4, 8], [4, 9, 1], [2, 9, 3]]]
Note that the calculation of nz takes into account that the 3rd dimension index in your array is either 0-based (as python is per default) or 1-based (as you show in your example).
Let's say I have this numpy matrix:
>>> mat = np.matrix([[3,4,5,2,1], [1,2,7,6,5], [8,9,4,5,2]])
>>> mat
matrix([[3, 4, 5, 2, 1],
[1, 2, 7, 6, 5],
[8, 9, 4, 5, 2]])
Now let's say I have some indexes in this form:
>>> ind = np.matrix([[0,2,3], [0,4,2], [3,1,2]])
>>> ind
matrix([[0, 2, 3],
[0, 4, 2],
[3, 1, 2]])
What I would like to do is to get three values from each row of the matrix, specifically values at columns 0, 2, and 3 for the first row, values at columns 0, 4 and 2 for the second row, etc. This is the expected output:
matrix([[3, 5, 2],
[1, 5, 7],
[5, 9, 4]])
I've tried using np.take but it doesn't seem to work. Any suggestion?
This is take_along_axis.
>>> np.take_along_axis(mat, ind, axis=1)
matrix([[3, 5, 2],
[1, 5, 7],
[5, 9, 4]])
This will do it: mat[np.arange(3).reshape(-1, 1), ind]
In [245]: mat[np.arange(3).reshape(-1, 1), ind]
Out[245]:
matrix([[3, 5, 2],
[1, 5, 7],
[5, 9, 4]])
(but take_along_axis in #user3483203's answer is simpler).
I have two matrices of the same size, A, B. I want to use the columns of B to acsses the columns of A, on a per column basis. For example,
A = np.array([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])
and
B = np.array([[0, 0, 2],
[1, 2, 1],
[2, 1, 0]])
I want something like:
A[B] = [[1, 4, 9],
[2, 6, 8],
[3, 5, 7]]
I.e., I've used the j'th column of B as indices to the j'th column of A.
Is there any effiecnt way of doing so?
Thanks!
You can use advanced indexing:
A[B, np.arange(A.shape[0])]
array([[1, 4, 9],
[2, 6, 8],
[3, 5, 7]])
Or with np.take_along_axis:
np.take_along_axis(A, B, axis=0)
array([[1, 4, 9],
[2, 6, 8],
[3, 5, 7]])
Suppose I have a matrix A with some arbitrary values:
array([[ 2, 4, 5, 3],
[ 1, 6, 8, 9],
[ 8, 7, 0, 2]])
And a matrix B which contains indices of elements in A:
array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
How do I select values from A pointed by B, i.e.:
A[B] = [[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]]
EDIT: np.take_along_axis is a builtin function for this use case implemented since numpy 1.15. See #hpaulj 's answer below for how to use it.
You can use NumPy's advanced indexing -
A[np.arange(A.shape[0])[:,None],B]
One can also use linear indexing -
m,n = A.shape
out = np.take(A,B + n*np.arange(m)[:,None])
Sample run -
In [40]: A
Out[40]:
array([[2, 4, 5, 3],
[1, 6, 8, 9],
[8, 7, 0, 2]])
In [41]: B
Out[41]:
array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
In [42]: A[np.arange(A.shape[0])[:,None],B]
Out[42]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
In [43]: m,n = A.shape
In [44]: np.take(A,B + n*np.arange(m)[:,None])
Out[44]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
More recent versions have added a take_along_axis function that does the job:
A = np.array([[ 2, 4, 5, 3],
[ 1, 6, 8, 9],
[ 8, 7, 0, 2]])
B = np.array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
np.take_along_axis(A, B, 1)
Out[]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
There's also a put_along_axis.
I know this is an old question, but another way of doing it using indices is:
A[np.indices(B.shape)[0], B]
output:
[[2 2 4 5]
[1 9 8 6]
[2 0 7 8]]
Following is the solution using for loop:
outlist = []
for i in range(len(B)):
lst = []
for j in range(len(B[i])):
lst.append(A[i][B[i][j]])
outlist.append(lst)
outarray = np.asarray(outlist)
print(outarray)
Above can also be written in more succinct list comprehension form:
outlist = [ [A[i][B[i][j]] for j in range(len(B[i]))]
for i in range(len(B)) ]
outarray = np.asarray(outlist)
print(outarray)
Output:
[[2 2 4 5]
[1 9 8 6]
[2 0 7 8]]