This is the array I have at hand:
[array([[[ 4, 9, 1, -3],
[-2, 0, 8, 6],
[ 1, 3, 7, 9 ],
[ 2, 5, 0, -7],
[-1, -6, -5, -8]]]),
array([[[ 0, 2, -1, 6 ],
[9, 8, 0, 3],
[ -1, 2, 5, -4],
[0, 5, 9, 6],
[ 6, 2, 9, 4]]]),
array([[[ 1, 2, 0, 9],
[3, 4, 8, -1],
[5, 6, 9, 0],
[ 7, 8, -3, -],
[9, 0, 8, -2]]])]
But the goal is obtain arrays A from first columns of nested arrays, B from second columns of nested arrays, Cfrom third columns of nested array etc.
Such that:
A = array([4, -2, 1, 2, -1, 0, 9, -1 ,0, 6, 1, 3, 5, 7, 9])
B = array([9, 0, 3, 5, -6, 2, 8, 2, 5, 2, 2,, 4, 6, 8, 0])
How should I do this?
You can do this with a single hstack() and use squeeze() to remove the extra dimension. With that you can use regular numpy indexing to pull out columns (or anything else you want):
import numpy as np
l = [np.array([[[ 4, 9, 1, -3],
[-2, 0, 8, 6],
[ 1, 3, 7, 9 ],
[ 2, 5, 0, -7],
[-1, -6, -5, -8]]]),
np.array([[[ 0, 2, -1, 6 ],
[9, 8, 0, 3],
[ -1, 2, 5, -4],
[0, 5, 9, 6],
[ 6, 2, 9, 4]]]),
np.array([[[ 1, 2, 0, 9],
[3, 4, 8, -1],
[5, 6, 9, 0],
[ 7, 8, -3, -1],
[9, 0, 8, -2]]])]
arr = np.hstack(l).squeeze()
A = arr[:,0]
print(A)
# [ 4 -2 1 2 -1 0 9 -1 0 6 1 3 5 7 9]
B = arr[:,1]
print(B)
#[ 9 0 3 5 -6 2 8 2 5 2 2 4 6 8 0]
# etc...
IIUC,
l = [np.array([[[ 4, 9, 1, -3],
[-2, 0, 8, 6],
[ 1, 3, 7, 9 ],
[ 2, 5, 0, -7],
[-1, -6, -5, -8]]]),
np.array([[[ 0, 2, -1, 6 ],
[9, 8, 0, 3],
[ -1, 2, 5, -4],
[0, 5, 9, 6],
[ 6, 2, 9, 4]]]),
np.array([[[ 1, 2, 0, 9],
[3, 4, 8, -1],
[5, 6, 9, 0],
[ 7, 8, -3, -9],
[9, 0, 8, -2]]])]
a = np.hstack([i[0][:, 0] for i in l])
b = np.hstack([i[0][:, 1] for i in l])
Output:
array([ 4, -2, 1, 2, -1, 0, 9, -1, 0, 6, 1, 3, 5, 7, 9])
array([ 9, 0, 3, 5, -6, 2, 8, 2, 5, 2, 2, 4, 6, 8, 0])
Related
I would like to ask you how to copy the rows to the next rows in matrix by using python. I tried however the result is not as expected. Here is the coding and the expected result. Thank you in advance.
import numpy as np
Total_T = 6
Total_P = 8
M0 = np.array([[0,0,0,0,0,1,1,1]])
M = np.random.randint(10,size = (Total_T, Total_P))
for k in range(0,Total_T):
for l in range (0,Total_P):
if k == Total_T-1:
M[Total_T-1][l] = M0[0][l]
#if k == 0:
#M[0][l] = M0[0][l]
else:
M
M_prev = np.empty((Total_T, Total_P), dtype = object)
if M is not None:
for k in range(0,Total_T):
for l in range (0,Total_P):
if k == 0:
M_prev[0][l] = M0[0][l]
else:
M_prev = M[:].copy()
The result of M:
array([[4, 8, 1, 6, 2, 4, 9, 6],
[3, 9, 1, 6, 9, 2, 7, 1],
[9, 4, 5, 9, 3, 5, 1, 2],
[8, 3, 5, 2, 8, 5, 4, 8],
[6, 3, 4, 3, 7, 9, 8, 4],
[0, 0, 0, 0, 0, 1, 1, 1]])
The expected result of M_prev:
array([[0, 0, 0, 0, 0, 1, 1, 1],
[4, 8, 1, 6, 2, 4, 9, 6],
[3, 9, 1, 6, 9, 2, 7, 1],
[9, 4, 5, 9, 3, 5, 1, 2],
[8, 3, 5, 2, 8, 5, 4, 8],
[6, 3, 4, 3, 7, 9, 8, 4]])
However the result of M_prev from the coding:
array([[4, 8, 1, 6, 2, 4, 9, 6],
[3, 9, 1, 6, 9, 2, 7, 1],
[9, 4, 5, 9, 3, 5, 1, 2],
[8, 3, 5, 2, 8, 5, 4, 8],
[6, 3, 4, 3, 7, 9, 8, 4],
[0, 0, 0, 0, 0, 1, 1, 1]])
It looks like you're overcomplicating things a lot…
import numpy as np
Total_T = 6
Total_P = 8
M0 = np.array([[0,0,0,0,0,1,1,1]])
# for reproducibility
np.random.seed(0)
# generate random 2D array
M = np.random.randint(10, size=(Total_T, Total_P))
# replace last row with M0
M[-1] = M0
# roll the array
M_prev = np.roll(M, 1, axis=0)
output:
# M
array([[5, 0, 3, 3, 7, 9, 3, 5],
[2, 4, 7, 6, 8, 8, 1, 6],
[7, 7, 8, 1, 5, 9, 8, 9],
[4, 3, 0, 3, 5, 0, 2, 3],
[8, 1, 3, 3, 3, 7, 0, 1],
[0, 0, 0, 0, 0, 1, 1, 1]])
# M_prev
array([[0, 0, 0, 0, 0, 1, 1, 1],
[5, 0, 3, 3, 7, 9, 3, 5],
[2, 4, 7, 6, 8, 8, 1, 6],
[7, 7, 8, 1, 5, 9, 8, 9],
[4, 3, 0, 3, 5, 0, 2, 3],
[8, 1, 3, 3, 3, 7, 0, 1]])
I have a m-dimensional NumPy array A and a n-dimensional NumPy array B
I want to create a m x n matrix C such that C[i, j] = B[j] - A[i]
Is there a efficient/vectorized way to do this in NumPy?
Currently I am using:
C = np.zeros((M, N))
for i in range(0, M):
C[i, :] = (B - A[i])
Edit:
m, n are big numbers, thus, C is a even bigger matrix (of m*n entries)
I tried np.repeat and np.subtract.outer but both of those crash my RAM
I think you are looking for ǹp.subtract.outer
M = 5
N = 10
A = np.arange(N)
B = np.arange(M)
np.subtract.outer(A,B)
output:
array([[ 0, -1, -2, -3, -4],
[ 1, 0, -1, -2, -3],
[ 2, 1, 0, -1, -2],
[ 3, 2, 1, 0, -1],
[ 4, 3, 2, 1, 0],
[ 5, 4, 3, 2, 1],
[ 6, 5, 4, 3, 2],
[ 7, 6, 5, 4, 3],
[ 8, 7, 6, 5, 4],
[ 9, 8, 7, 6, 5]])
It will apply subtraction to all pairs in A and B. For 1 dimension it is equivalent to
C = empty(len(A),len(B))
for i in range(len(A)):
for j in range(len(B)):
C[i,j] = A[i] - B[j]
EDIT
If you have memory issues you could try allocating the output buffer before doing the operation and setting the out keyword appropriately:
C = np.zeros((M, N))
np.subtract.outer(B, A, out=C)
You could repeat one of the arrays on a new axis, and then subtract the other array.
Example:
m = 10
n = 20
A = np.array(range(m))
B = np.array(range(n))
C = np.repeat(B[:, np.newaxis], m, axis=1) - A
Then C would contain:
array([[ 0, -1, -2, -3, -4, -5, -6, -7, -8, -9],
[ 1, 0, -1, -2, -3, -4, -5, -6, -7, -8],
[ 2, 1, 0, -1, -2, -3, -4, -5, -6, -7],
[ 3, 2, 1, 0, -1, -2, -3, -4, -5, -6],
[ 4, 3, 2, 1, 0, -1, -2, -3, -4, -5],
[ 5, 4, 3, 2, 1, 0, -1, -2, -3, -4],
[ 6, 5, 4, 3, 2, 1, 0, -1, -2, -3],
[ 7, 6, 5, 4, 3, 2, 1, 0, -1, -2],
[ 8, 7, 6, 5, 4, 3, 2, 1, 0, -1],
[ 9, 8, 7, 6, 5, 4, 3, 2, 1, 0],
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1],
[11, 10, 9, 8, 7, 6, 5, 4, 3, 2],
[12, 11, 10, 9, 8, 7, 6, 5, 4, 3],
[13, 12, 11, 10, 9, 8, 7, 6, 5, 4],
[14, 13, 12, 11, 10, 9, 8, 7, 6, 5],
[15, 14, 13, 12, 11, 10, 9, 8, 7, 6],
[16, 15, 14, 13, 12, 11, 10, 9, 8, 7],
[17, 16, 15, 14, 13, 12, 11, 10, 9, 8],
[18, 17, 16, 15, 14, 13, 12, 11, 10, 9],
[19, 18, 17, 16, 15, 14, 13, 12, 11, 10]])
This is a simple broadcasting task:
In [31]: A =np.arange(3); B=np.arange(4)
In [32]: C = B - A[:,None]
In [33]: C.shape
Out[33]: (3, 4)
In [34]: C
Out[34]:
array([[ 0, 1, 2, 3],
[-1, 0, 1, 2],
[-2, -1, 0, 1]])
This is like the https://stackoverflow.com/a/66842410/901925 answer, but there's no need to use repeat. That should cut down the memory usage a bit, but if M*N*8 is anywhere close to your memory limits, this or subsequent operations using C could hit that limit.
Let's say I have a list of elements with values
[1, 2, 3, 5, 6, 7, 9, 12]
Basically, the elements in the array may max have a difference of n, in this case three, in increasing order.
The array above would work like:
2-1 = 1 | difference of 1
3-2 = 1 | difference of 1
5-3 = 2 | difference of 2
6-5 = 1 | difference of 1
And so on.
How would I go about to find all permutations of an array with length x and a max difference of n?
Assuming you are looking for differences in absolute value, you can do this recursively by progressively adding each eligible element to the result:
here's an example using a recursive generator function:
def permuteSort(A,maxDiff,previous=None):
if not A: yield []; return
for i,a in enumerate(A):
if previous is not None and abs(a-previous) > maxDiff:
continue
yield from ([a]+p for p in permuteSort(A[:i]+A[i+1:],maxDiff,a))
Output
for p in permuteSort([1, 2, 3, 5, 6, 7, 9, 12],3):
print(p,"differences:",[b-a for a,b in zip(p,p[1:])])
[1, 2, 3, 5, 6, 7, 9, 12] differences: [1, 1, 2, 1, 1, 2, 3]
[1, 2, 3, 5, 7, 6, 9, 12] differences: [1, 1, 2, 2, -1, 3, 3]
[1, 2, 3, 6, 5, 7, 9, 12] differences: [1, 1, 3, -1, 2, 2, 3]
[1, 2, 5, 3, 6, 7, 9, 12] differences: [1, 3, -2, 3, 1, 2, 3]
[1, 3, 2, 5, 6, 7, 9, 12] differences: [2, -1, 3, 1, 1, 2, 3]
[1, 3, 2, 5, 7, 6, 9, 12] differences: [2, -1, 3, 2, -1, 3, 3]
[2, 1, 3, 5, 6, 7, 9, 12] differences: [-1, 2, 2, 1, 1, 2, 3]
[2, 1, 3, 5, 7, 6, 9, 12] differences: [-1, 2, 2, 2, -1, 3, 3]
[2, 1, 3, 6, 5, 7, 9, 12] differences: [-1, 2, 3, -1, 2, 2, 3]
[3, 1, 2, 5, 6, 7, 9, 12] differences: [-2, 1, 3, 1, 1, 2, 3]
[3, 1, 2, 5, 7, 6, 9, 12] differences: [-2, 1, 3, 2, -1, 3, 3]
[5, 2, 1, 3, 6, 7, 9, 12] differences: [-3, -1, 2, 3, 1, 2, 3]
[6, 3, 1, 2, 5, 7, 9, 12] differences: [-3, -2, 1, 3, 2, 2, 3]
[7, 5, 2, 1, 3, 6, 9, 12] differences: [-2, -3, -1, 2, 3, 3, 3]
[12, 9, 6, 3, 1, 2, 5, 7] differences: [-3, -3, -3, -2, 1, 3, 2]
[12, 9, 6, 7, 5, 2, 1, 3] differences: [-3, -3, 1, -2, -3, -1, 2]
[12, 9, 6, 7, 5, 2, 3, 1] differences: [-3, -3, 1, -2, -3, 1, -2]
[12, 9, 6, 7, 5, 3, 1, 2] differences: [-3, -3, 1, -2, -2, -2, 1]
[12, 9, 6, 7, 5, 3, 2, 1] differences: [-3, -3, 1, -2, -2, -1, -1]
[12, 9, 7, 5, 2, 1, 3, 6] differences: [-3, -2, -2, -3, -1, 2, 3]
[12, 9, 7, 5, 6, 3, 1, 2] differences: [-3, -2, -2, 1, -3, -2, 1]
[12, 9, 7, 5, 6, 3, 2, 1] differences: [-3, -2, -2, 1, -3, -1, -1]
[12, 9, 7, 6, 3, 1, 2, 5] differences: [-3, -2, -1, -3, -2, 1, 3]
[12, 9, 7, 6, 3, 5, 2, 1] differences: [-3, -2, -1, -3, 2, -3, -1]
[12, 9, 7, 6, 5, 2, 1, 3] differences: [-3, -2, -1, -1, -3, -1, 2]
[12, 9, 7, 6, 5, 2, 3, 1] differences: [-3, -2, -1, -1, -3, 1, -2]
[12, 9, 7, 6, 5, 3, 1, 2] differences: [-3, -2, -1, -1, -2, -2, 1]
[12, 9, 7, 6, 5, 3, 2, 1] differences: [-3, -2, -1, -1, -2, -1, -1]
Try this as a recursion. Should print out all allowed values.
x is described as required_number and n is diff.
def getPermutations(cur_permutation, arr_to_check, required_number, diff):
if len(arr_to_check) == 0:
return
if required_number == 0:
print cur_permutation
return
cur_last_item = cur_permutation[-1]
for index, arr_item in enumerate(arr_to_check):
if arr_item - cur_last_item <= diff:
new_copy = cur_permutation[:]
new_copy.append(arr_item)
return getPermutations(new_copy, arr_to_check[1:], required_number - 1, diff)
(This could also be done with keeping the require length and checking if cur_permutation is of that length).
If i have an numpy array like:
x= [[3, 3], [2, 2]]
I want to add an element -1 to the end of each the rows to be like this:
x= [[3, 3, -1], [2, 2, -1]]
any simple way to do that ?
A simple way would be with np.insert -
np.insert(x,x.shape[1],-1,axis=1)
We can also use np.column_stack -
np.column_stack((x,[-1]*x.shape[0]))
Sample run -
In [161]: x
Out[161]:
array([[0, 8, 7, 0, 1],
[0, 1, 8, 6, 8],
[3, 4, 7, 0, 2]])
In [162]: np.insert(x,x.shape[1],-1,axis=1)
Out[162]:
array([[ 0, 8, 7, 0, 1, -1],
[ 0, 1, 8, 6, 8, -1],
[ 3, 4, 7, 0, 2, -1]])
In [163]: np.column_stack((x,[-1]*x.shape[0]))
Out[163]:
array([[ 0, 8, 7, 0, 1, -1],
[ 0, 1, 8, 6, 8, -1],
[ 3, 4, 7, 0, 2, -1]])
I want to add the previous n rows as columns to a NumPy array.
For example, if n=2, the array below...
[[ 1, 2]
[ 3, 4]
[ 5, 6]
[ 7, 8]
[ 9, 10]
[11, 12]]
...should be turned into the following one:
[[ 1, 2, 0, 0, 0, 0]
[ 3, 4, 1, 2, 0, 0]
[ 5, 6, 3, 4, 1, 2]
[ 7, 8, 5, 6, 3, 4]
[ 9, 10, 7, 8, 5, 6]
[11, 12, 9, 10, 7, 8]]
Any ideas how I could do that without going over the entire array in a for loop?
Here's a vectorized approach -
def vectorized_app(a,n):
M,N = a.shape
idx = np.arange(a.shape[0])[:,None] - np.arange(n+1)
out = a[idx.ravel(),:].reshape(-1,N*(n+1))
out[N*(np.arange(1,M+1))[:,None] <= np.arange(N*(n+1))] = 0
return out
Sample run -
In [255]: a
Out[255]:
array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10, 11, 12],
[13, 14, 15],
[16, 17, 18]])
In [256]: vectorized_app(a,3)
Out[256]:
array([[ 1, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 4, 5, 6, 1, 2, 3, 0, 0, 0, 0, 0, 0],
[ 7, 8, 9, 4, 5, 6, 1, 2, 3, 0, 0, 0],
[10, 11, 12, 7, 8, 9, 4, 5, 6, 1, 2, 3],
[13, 14, 15, 10, 11, 12, 7, 8, 9, 4, 5, 6],
[16, 17, 18, 13, 14, 15, 10, 11, 12, 7, 8, 9]])
Runtime test -
I am timing #Psidom's loop-comprehension based method and the vectorized method listed in this post on a 100x scaled up version (in terms of size) of the sample posted in the question :
In [246]: a = np.random.randint(0,9,(600,200))
In [247]: n = 200
In [248]: %timeit np.column_stack(mypad(a, i) for i in range(n + 1))
1 loops, best of 3: 748 ms per loop
In [249]: %timeit vectorized_app(a,n)
1 loops, best of 3: 224 ms per loop
Here is a way to pad 0 in the beginning of the array and then column stack them:
import numpy as np
n = 2
def mypad(myArr, n):
if n == 0:
return myArr
else:
return np.pad(myArr, ((n,0), (0,0)), mode = "constant")[:-n]
np.column_stack(mypad(arr, i) for i in range(n + 1))
# array([[ 1, 2, 0, 0, 0, 0],
# [ 3, 4, 1, 2, 0, 0],
# [ 5, 6, 3, 4, 1, 2],
# [ 7, 8, 5, 6, 3, 4],
# [ 9, 10, 7, 8, 5, 6],
# [11, 12, 9, 10, 7, 8]])