For example, I have a matrix:
[ [1 2 3 4 5],
[6 7 8 9 10],
[11 12 13 14 15],
[16 17 18 19 20],
[21 22 23 24 25] ]
I want to insert [ [-1 -1 -1], [0 5 0] ] in some position, like:
[ [1 2 3 4 5],
[6 7 8 9 10],
[11 -1 -1 -1 15],
[16 0 5 0 20],
[21 22 23 24 25] ]
Use numpy insert!
Here is an example from the numpy reference at scipy:
>>> a = np.array([[1, 1], [2, 2], [3, 3]])
>>> a
array([[1, 1],
[2, 2],
[3, 3]])
>>> np.insert(a, 1, 5)
array([1, 5, 1, 2, 2, 3, 3])
>>> np.insert(a, 1, 5, axis=1)
array([[1, 5, 1],
[2, 5, 2],
[3, 5, 3]]
Read more here: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.insert.html
Based on the example, I would say you are trying to replace or modify
part of the existing array rather than insert an array.
You could use basic slicing to get a view of the part of the array you want to overwrite,
and assign the value of that slice to a new matrix of the same size
as the slice.
For example:
>>> x=np.matrix([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
>>> x
matrix([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12],
[13, 14, 15, 16]])
>>> x[1:3,1:4]=np.matrix([[-1,-2,-3],[-4,-5,-6]])
>>> x
matrix([[ 1, 2, 3, 4],
[ 5, -1, -2, -3],
[ 9, -4, -5, -6],
[13, 14, 15, 16]])
In general, to describe a submatrix of m rows and n columns with its upper left corner at row r and column c of the original matrix,
index the slice as x[r:r+m,c:c+n].
This method take the matrix m, and replace the elements with array n starting from row r, column c
def replace(m, n, r, c):
i = 0
if len(n) + c > len(m[r]):
return
for each in n:
m[r][c] = n[i]
c += 1
i += 1
you have to check the index boundaries for the matrix
Related
I created a code that gets values of df based on list of indices in another column:
import numpy as np
import pandas as pd
d = {'myvalues': [11, 13, 0, -1, 10, 14], 'neighbours': [[1,2],[0,2,3],[0,1,3],[1,2,4],[3,5],[4]]}
df = pd.DataFrame(data=d)
df['neighboring_idxs'] = df['neighbours']+pd.Series(([[x] for x in df.index.values]))
df['neighboring_myvalues'] = df.apply(lambda row: df.myvalues.values[row.neighboring_idxs], axis=1)
Result is:
myvalues neighbours neighboring_idxs neighboring_myvalues
0 11 [1, 2] [1, 2, 0] [13, 0, 11]
1 13 [0, 2, 3] [0, 2, 3, 1] [11, 0, -1, 13]
2 0 [0, 1, 3] [0, 1, 3, 2] [11, 13, -1, 0]
3 -1 [1, 2, 4] [1, 2, 4, 3] [13, 0, 10, -1]
4 10 [3, 5] [3, 5, 4] [-1, 14, 10]
5 14 [4] [4, 5] [10, 14]
However on large dataset using apply is really time-consuming. Is there a smarter way to achieve the same df['neighboring_myvalues'], without using apply?
I don't know if it's faster but you can try to explode your list of list:
df['neighboring_myvalues'] = (
df.explode('neighboring_idxs').reset_index()
.assign(vals=lambda x: df.loc[x['neighboring_idxs'], 'myvalues'].tolist())
.groupby('index')['vals'].agg(list)
)
Output:
>>> df
myvalues neighbours neighboring_idxs neighboring_myvalues
0 11 [1, 2] [1, 2, 0] [13, 0, 11]
1 13 [0, 2, 3] [0, 2, 3, 1] [11, 0, -1, 13]
2 0 [0, 1, 3] [0, 1, 3, 2] [11, 13, -1, 0]
3 -1 [1, 2, 4] [1, 2, 4, 3] [13, 0, 10, -1]
4 10 [3, 5] [3, 5, 4] [-1, 14, 10]
5 14 [4] [4, 5] [10, 14]
I am going to simulate a 2d area using a matrix in python and I should change the indexes somehow!. I need to find a relation between the original matrix and a new axis coordinate. I think it can be more clarified if I explain by a piece of my code:
originalMatrix=np.random.randint(0, 10, size=(5,5))
the output is:
[[4 8 3 2 5]
[2 2 2 6 5]
[2 4 7 9 9]
[6 2 6 6 6]
[2 8 3 8 2]]
we can access to number '7' by originalMatrix[2][2]. but in the coordinate system the index must be (0,0).Index of '4' in the left side of '7' in originalMatrix is [2][1] but in the new coordinate should be (-1,0) and so on... I hope I could explain well!
An easy way is to roll your array:
a = np.array([[4, 8, 3, 2, 5],
[2, 2, 2, 6, 5],
[2, 4, 7, 9, 9],
[6, 2, 6, 6, 6],
[2, 8, 3, 8, 2]])
center = np.array(a.shape)//2
# b will be indexed with the previous center being 0,0
b = np.roll(a, -center, axis=(0,1))
b[0, -1]
# 4
b[0, 0]
# 7
NB. You have to decide what is the center in case you have an even number as a dimension.
Here is another possible approach, assuming the matrix is nxn where n is odd:
import numpy as np
def access_coordinate(matrix, x, y):
center = (len(matrix)//2, len(matrix)//2)
translated_coordinates = (center[0]+y, center[1] + x)
return matrix[translated_coordinates[0], translated_coordinates[1]]
originalMatrix=np.array([[4, 8, 3, 2, 5],
[2, 2, 2 ,6, 5],
[2, 4, 7, 9, 9],
[6, 2, 6, 6, 6],
[2, 8, 3, 8, 2]])
print(access_coordinate(originalMatrix, -1, 0)) #4
There is no way to do that with standard numpy arrays, but you could define a function that does it for you
def arrayByCoordinates(array, coords):
coords = np.array(coords)
center = np.array(array.shape) // 2
return array[tuple(coords + center)]
>>> x
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
>>> arrayByCoordinates(x,(0,0))
12
But keep in mind that if the shape of the array is even, there is no real center, so this will convert to the lowest one
I'm currently porting a MATLAB library over to Python. As of right now, I am trying to keep the code as one-to-one as possible. I'm noticing some differences between reshape in Matlab vs Python that is causing some issues.
I've heard people talk about the difference in 'C' and 'Fortran' order. How numpy defaults to 'C' order and Matlab uses 'Fortran'. Below are two Python examples using both orders.
>>> a = np.arange(12).reshape((2,3,2))
>>> a
array([[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]]])
>>> b = np.arange(12).reshape((2,3,2), order='F')
>>> b
array([[[ 0, 6],
[ 2, 8],
[ 4, 10]],
[[ 1, 7],
[ 3, 9],
[ 5, 11]]])
Below is the matlab/octave equivalent to the above python code.
octave:12> a = reshape((0:11), [3,2,2])
a =
ans(:,:,1) =
0 3
1 4
2 5
ans(:,:,2) =
6 9
7 10
8 11
Notice that each example yields a different result.
These examples are meant to illustrate the discrepancy that I'm referring to. The datasets that I'm working on in my project are significantly larger. I need to be able to reshape arrays in Python and be confident that it is performing the same reshape operations as it would in Matlab. Any help would be appreciated.
Why are you using a (2,3,2) shape in one, and (3,2,2) in the other?
In [82]: arr = np.arange(12).reshape((3,2,2), order='F')
In [83]: arr
Out[83]:
array([[[ 0, 6],
[ 3, 9]],
[[ 1, 7],
[ 4, 10]],
[[ 2, 8],
[ 5, 11]]])
In [84]: arr[:,:,0]
Out[84]:
array([[0, 3],
[1, 4],
[2, 5]])
In [85]: arr[:,:,1]
Out[85]:
array([[ 6, 9],
[ 7, 10],
[ 8, 11]])
===
Looking a strides may help identify the differences between c and f orders
In [86]: arr.shape
Out[86]: (3, 2, 2)
In [87]: arr.strides
Out[87]: (8, 24, 48)
Notice how the smallest steps, 1 element (8 bytes) is taken in first dimension.
Contrast that with a C order:
In [89]: np.arange(12).reshape(2,2,3)
Out[89]:
array([[[ 0, 1, 2],
[ 3, 4, 5]],
[[ 6, 7, 8],
[ 9, 10, 11]]])
In [90]: np.arange(12).reshape(2,2,3).strides
Out[90]: (48, 24, 8)
===
OK lets try the (2,3,2) shape:
>> a = reshape((0:11),[2,3,2])
a =
ans(:,:,1) =
0 2 4
1 3 5
ans(:,:,2) =
6 8 10
7 9 11
Samething with order 'F':
In [94]: arr = np.arange(12).reshape((2,3,2), order='F')
In [95]: arr
Out[95]:
array([[[ 0, 6],
[ 2, 8],
[ 4, 10]],
[[ 1, 7],
[ 3, 9],
[ 5, 11]]])
In [96]: arr[:,:,0]
Out[96]:
array([[0, 2, 4],
[1, 3, 5]])
>> squeeze(a(1,:,:))
ans =
0 6
2 8
4 10
In [98]: arr[0,:,:]
Out[98]:
array([[ 0, 6],
[ 2, 8],
[ 4, 10]])
I have the following numpy array:
[[[1], [2], [3], [1], [2], [3]],
[[4], [5], [6], [4], [5], [6]],
[[7], [8], [9], [7], [8], [9]]]
And I want each of the elements in the last dimension, [1], [2], [3] etc. to be concatenate with the following n arrays in the second dimension. In case of overflow, elements can be filled with 0. For example, for n = 2:
[[[1, 2, 3], [2, 3, 1], [3, 1, 2], [1, 2, 3], [2, 3, 0], [3, 0, 0]],
[[4, 5, 6], [5, 6, 4], [6, 4, 5], [4, 5, 6], [5, 6, 0], [6, 0, 0]],
[[7, 8, 9], [8, 9, 7], [9, 7, 8], [7, 8, 9], [8, 9, 0], [9, 0, 0]]]
I want to do this with the built in numpy functions for good performance and also want to do it in reverse i.e., a shift of n = -2 is fair game. How to do this?
For n = -2:
[[[0, 0, 1], [0, 1, 2], [1, 2, 3], [2, 3, 1], [3, 1, 2], [1, 2, 3]],
[[0, 0, 4], [0, 4, 5], [4, 5, 6], [5, 6, 4], [6, 4, 5], [4, 5, 6]],
[[0, 0, 7], [0, 7, 8], [7, 8, 9], [8, 9, 7], [9, 7, 8], [7, 8, 9]]]
For n = 3
[[[1, 2, 3, 1], [2, 3, 1, 2], [3, 1, 2, 3], [1, 2, 3, 0], [2, 3, 0, 0], [3, 0, 0, 0]],
[[4, 5, 6, 4], [5, 6, 4, 5], [6, 4, 5, 6], [4, 5, 6, 0], [5, 6, 0, 0], [6, 0, 0, 0]],
[[7, 8, 9, 7], [8, 9, 7, 8], [9, 7, 8, 9], [7, 8, 9, 0], [8, 9, 0, 0], [9, 0, 0, 0]]]
If the current shape of the array is (height, width, 1), after the operation, the shape will be (height, width, abs(n) + 1).
How to generalize this so that the numbers 1, 2, 3 etc. can themselves be numpy arrays?
This sounds like a textbook application for the monster that is as_strided. One of the nice things about it is that it does not require any additional imports. The general idea is this:
You have an array with shape (3, 6, 1) and strides (6, 1, 1) * element_size.
x = ...
n = ... # Must not be zero, but you can special-case it to return the original array
You want to transform this into an array that has shape (3, 6, |n| + 1) and therefore strides (6 * (|n| + 1), |n| + 1, 1) * element_size.
To do this, you first pad the left or the right with |n| zeros:
pad = np.zeros((x.shape[0], np.abs(n), x.shape[2]))
x_pad = np.concatenate([x, pad][::np.sign(n)], axis=1)
Now you can index directly into the buffer with a custom shape and strides to get the result you want. Instead of using the proper strides (6 * (|n| + 1), |n| + 1, 1) * element_size, we will index each repeated element directly into the same buffer of the original array, meaning that the strides will be adjusted. The middle dimension will move by one element, rather than the proper |n| + 1. That way, the columns can start exactly where you want them to:
new_shape = (x.shape[0], x.shape[1], x.shape[2] + np.abs(n))
new_strides = (x_pad.strides[0], x_pad.strides[2], x_pad.strides[2])
result = np.lib.stride_tricks.as_strided(x_pad, shape=new_shape, strides=new_strides)
There are many caveats here. The biggest thing to be aware of is that multiple array elements access the same memory. My advice is to make a proper fleshed-out copy if you plan to do anything besides just reading the data:
result = result.copy()
This will give you a buffer of the correct size rather than a crazy view into the original data with padding.
Here is a way to do it:
from skimage.util import view_as_windows
if n>=0:
a = np.pad(a.reshape(*a.shape[:-1]),((0,0),(0,n)))
else:
n *= -1
a = np.pad(a.reshape(*a.shape[:-1]),((0,0),(n,0)))
b = view_as_windows(a,(1,n+1))
b = b.reshape(*b.shape[:-2]+(n+1,))
a is your input array and b is your output:
n=2:
[[[1 2 3]
[2 3 1]
[3 1 2]
[1 2 3]
[2 3 0]
[3 0 0]]
[[4 5 6]
[5 6 4]
[6 4 5]
[4 5 6]
[5 6 0]
[6 0 0]]
[[7 8 9]
[8 9 7]
[9 7 8]
[7 8 9]
[8 9 0]
[9 0 0]]]
n=-2:
[[[0 0 1]
[0 1 2]
[1 2 3]
[2 3 1]
[3 1 2]
[1 2 3]]
[[0 0 4]
[0 4 5]
[4 5 6]
[5 6 4]
[6 4 5]
[4 5 6]]
[[0 0 7]
[0 7 8]
[7 8 9]
[8 9 7]
[9 7 8]
[7 8 9]]]
Explanation:
np.pad(a.reshape(*a.shape[:-1]),((0,0),(0,n))) pads enough zeros to the right side of array for overflow of windows (similarly padding left side for negative n)
view_as_windows(a,(1,n+1)) creates windows of shape (1,n+1) from the array as desired by the question.
b.reshape(*b.shape[:-2]+(n+1,)) gets rid of the extra dimension of length 1 created by (1,n+1)-shaped windows and reshape b to desired shape. Note the argument *b.shape[:-2]+(n+1,) is simply concatenation of two tuples to create a single tuple as shape.
You can also do this (requires numpy version 1.20.x+
import numpy as np
arr = np.array([[[1], [2], [3], [1], [2], [3]],
[[4], [5], [6], [4], [5], [6]],
[[7], [8], [9], [7], [8], [9]]])
n = 2 # your n value
nslices = n+1
# reshape into 2d array
arr = arr.reshape((3, -1)) # <-- add n zeros as padding here
# perform the slicing
slices = np.lib.stride_tricks.sliding_window_view(arr, (1, nslices))
slices = slices[:, :, 0]
print(slices)
I have a list.
l = [[1, 2, 8] [8, 2, 7] [7, 2, 5]]
I want first element to be zero and then I need to subtract values columnwise.
explanation :
1 2 8
8 2 7
7 2 5
subtraction as,
0 1 6
0 -6 5
0 -5 3
I want output as :
l = [[0, 1, 6], [0, -6, 5], [0, -5, 3]]
which is the faster way to perform this operation if I have large list?
I am using numpy but I changed here so that easy to understand
my numpy array object is
l = [[1 2 8] [8 2 7] [7 2 5]]
>>> l = np.array([[1, 2, 8], [8, 2, 7], [7, 2, 5]])
>>> l[:, 1:] -= l[:, :-1]
>>> l[:, 0] = 0
>>> l
array([[ 0, 1, 6],
[ 0, -6, 5],
[ 0, -5, 3]])
Using numpy.insert and numpy.diff:
>>> import numpy as np
>>> a = np.array([[1, 2, 8], [8, 2, 7], [7, 2, 5]])
>>> np.insert(np.diff(a), 0, 0, axis=1)
array([[ 0, 1, 6],
[ 0, -6, 5],
[ 0, -5, 3]])
Without numpy, you can get away with this
l = [[1, 2, 8], [8, 2, 7], [7, 2, 5]]
def minus(rest, val):
rest[-1] -= val
rest.append(val)
return rest
def myReduce(l):
l2 = reduce(minus, l[-2::-1], [l[-1]])
l2.reverse()
l2[0] = 0
return l2
l2 = map(myReduce, l)
print l2
I guess it's quite straightforward and easy to understand.