Case 1 (solved): Array A has shape (say) (300,50). Array B is an indices array with the shape (300,5), such that B[i,j] indicate for the row i the index of another row to "concate" next to the row i. The end result is an array C with the shape (300,5,50), such that C[i,j,:] = A[B[i,j],:]. This can be done by calling A[B,:].
Here is small script example for case 1:
import numpy as np
## A is the data array
A = np.arange(20).reshape((5,4))
## B indicate for each row which rows to pull together
B = np.array([[0,2],[1,2],[2,0],[3,4],[4,1]])
A[B,:] #The desired result
Case 2 (unsolved): Same problem, only now A is shaped (100,300,50). If B is the indicies matrix shaped (100,300,5), the end result would be an array C with the shape (100,300,5,50) such that C[i,j,k,:] = A[i,B[i,j,k],:]. A[B,:] doesn't work anymore, because it result with a shape (100,300,5,300,50), due to broadcasting.
How should I approach this with indexing?
One approach would be reshaping to 2D keeping the number of columns intact and then indexing into the first axis with the flattened B indices and finally reshaping back to the desired one.
Thus, the implementation would be -
A.reshape(-1,A.shape[-1])[B.ravel()].reshape(100,300,5,50)
Those reshaping being merely views into the arrays, should be quite efficient.
This solves both cases. Here's a sample run for the case #1 -
1) Inputs :
In [667]: A = np.random.rand(3,4)
...: B = np.random.randint(0,3,(3,5))
...:
2) Original method :
In [668]: A[B,:]
Out[668]:
array([[[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.1 , 0.91, 0.1 , 0.98]],
[[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02]],
[[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.45, 0.16, 0.02, 0.02]]])
3) Proposed method :
In [669]: A.reshape(-1,A.shape[-1])[B.ravel()].reshape(3,5,4)
Out[669]:
array([[[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.1 , 0.91, 0.1 , 0.98]],
[[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02]],
[[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.45, 0.16, 0.02, 0.02]]])
Related
The question was answered by #QuangHoang in the comments. Using np.tile(array, (repeats, 1)) or slower: np.vstack([array]*repeats).
I have what seems to be a simple problem. I have a NumPy array I want to replicate column-wise, and keep it a 2D array, just repeated over and over again say 50x. I try with a np.repeat but the order isn't preserved - it just copies each row however many times I specify. broadcast_to I can't seem to get to work either. I'm thinking maybe I need to flatten the array with a .reshape(-1) then do something with it and reshape it back, but I can't seem to find the right commands. I know I can do it with .append and a for loop, but I'm looking for a vectorized solution. Here's an example with the initial array and then what it should look like afterwards (I'm only repeating it 3x due to space):
array = np.array([(1.00, 0.80, 0.95, 0.88, 0.97, 0.85),
(0.80, 1.00, 0.87, 0.97, 0.80, 0.92),
(0.95, 0.87, 1.00, 0.85, 0.92, 0.89),
(0.88, 0.97, 0.85, 1.00, 0.85, 0.95),
(0.97, 0.80, 0.92, 0.85, 1.00, 0.88),
(0.85, 0.92, 0.89, 0.95, 0.88, 1.00)])
Repeat 3x:
array([[1. , 0.8 , 0.95, 0.88, 0.97, 0.85],
[0.8 , 1. , 0.87, 0.97, 0.8 , 0.92],
[0.95, 0.87, 1. , 0.85, 0.92, 0.89],
[0.88, 0.97, 0.85, 1. , 0.85, 0.95],
[0.97, 0.8 , 0.92, 0.85, 1. , 0.88],
[0.85, 0.92, 0.89, 0.95, 0.88, 1. ],
[1. , 0.8 , 0.95, 0.88, 0.97, 0.85],
[0.8 , 1. , 0.87, 0.97, 0.8 , 0.92],
[0.95, 0.87, 1. , 0.85, 0.92, 0.89],
[0.88, 0.97, 0.85, 1. , 0.85, 0.95],
[0.97, 0.8 , 0.92, 0.85, 1. , 0.88],
[0.85, 0.92, 0.89, 0.95, 0.88, 1. ],
[1. , 0.8 , 0.95, 0.88, 0.97, 0.85],
[0.8 , 1. , 0.87, 0.97, 0.8 , 0.92],
[0.95, 0.87, 1. , 0.85, 0.92, 0.89],
[0.88, 0.97, 0.85, 1. , 0.85, 0.95],
[0.97, 0.8 , 0.92, 0.85, 1. , 0.88],
[0.85, 0.92, 0.89, 0.95, 0.88, 1. ]])
Appreciate the help!
Generation of a list of many lists each with different ranges
Isc_act = [0.1, 0.2, 0.3]
I_cel = []
a = []
for i in range(0,len(Isc_act)):
a = np.arange(0, Isc_act[i], 0.1*Isc_act[i])
I_cel[i].append(a)
print(I_cel)
Output is:
IndexError: list index out of range
My code is giving error. But, I want to get I_cel = [[0,0.01,..,0.1],[0,0.02,0.04,...,0.2],[0, 0.03, 0.06,...,0.3]]. Hence, the 'nested list' I_cel has three lists and each list has 10 values.
The simplest fix to your code, probably what you were intending to do:
Isc_act = [0.1, 0.2, 0.3]
I_cel = []
for i in range(0,len(Isc_act)):
a = np.arange(0, Isc_act[i], 0.1*Isc_act[i])
I_cel.append(a)
print(I_cel)
Note that the endpoint will be one step less than you wanted! For example row zero, you have to pick two of the below:
Steps of size 0.01
Start point 0.0 and end point 0.1
10 elements total
You can not have all three.
More numpythonic approach:
>>> Isc_act = [0.1, 0.2, 0.3]
>>> (np.linspace(0, 1, 11).reshape(11,1) # [Isc_act]).T
array([[0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 ],
[0. , 0.02, 0.04, 0.06, 0.08, 0.1 , 0.12, 0.14, 0.16, 0.18, 0.2 ],
[0. , 0.03, 0.06, 0.09, 0.12, 0.15, 0.18, 0.21, 0.24, 0.27, 0.3 ]])
linspace gives better control of the end point when dealing with floats:
In [84]: [np.linspace(0,x,11) for x in [.1,.2,.3]]
Out[84]:
[array([0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 ]),
array([0. , 0.02, 0.04, 0.06, 0.08, 0.1 , 0.12, 0.14, 0.16, 0.18, 0.2 ]),
array([0. , 0.03, 0.06, 0.09, 0.12, 0.15, 0.18, 0.21, 0.24, 0.27, 0.3 ])]
Or we could scale just one array (arange with integers is predictable):
In [86]: np.array([.1,.2,.3])[:,None]*np.arange(0,11)
Out[86]:
array([[0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ],
[0. , 0.2, 0.4, 0.6, 0.8, 1. , 1.2, 1.4, 1.6, 1.8, 2. ],
[0. , 0.3, 0.6, 0.9, 1.2, 1.5, 1.8, 2.1, 2.4, 2.7, 3. ]])
I have one solution for particular problem as
[[0.34 0.26 0.76 ]
[0.79 0.82 0.37 ]
[0.93 0.87 0.94]]
I have another solution for same problem as
[[0.21 0.73 0.69 ]
[0.35 0.24 0.53]
[0.01 0.42 0.50]]
Now I have to merge their ith position together so the resultant array would be like
[[0.34 0.21]
[0.26 0.73]
[0.76 0.69]
[0.79 0.35]
..........
..........
Setup
x = np.array([[0.34, 0.26, 0.76 ], [0.79, 0.82, 0.37 ], [0.93, 0.87, 0.94]])
y = np.array([[0.21, 0.73, 0.69 ], [0.35, 0.24, 0.53], [0.01, 0.42, 0.50]])
dstack and ravel
np.dstack([x.ravel(), y.ravel()])
array([[[0.34, 0.21],
[0.26, 0.73],
[0.76, 0.69],
[0.79, 0.35],
[0.82, 0.24],
[0.37, 0.53],
[0.93, 0.01],
[0.87, 0.42],
[0.94, 0.5 ]]])
If you're concerned with the extra dimension this introduces, you can vstack and transpose:
np.vstack([x.ravel(), y.ravel()]).T
array([[0.34, 0.21],
[0.26, 0.73],
[0.76, 0.69],
[0.79, 0.35],
[0.82, 0.24],
[0.37, 0.53],
[0.93, 0.01],
[0.87, 0.42],
[0.94, 0.5 ]])
Another alternative using np.column_stack
np.column_stack([x.ravel(), y.ravel()])
You can use vstack on your 2 arrays and reshape appropriately:
np.vstack([arr1,arr2]).reshape(2,-1).T
Example:
>>> arr1
array([[ 0.34, 0.26, 0.76],
[ 0.79, 0.82, 0.37],
[ 0.93, 0.87, 0.94]])
>>> arr2
array([[ 0.21, 0.73, 0.69],
[ 0.35, 0.24, 0.53],
[ 0.01, 0.42, 0.5 ]])
>>> np.vstack([arr1,arr2]).reshape(2,-1).T
array([[ 0.34, 0.21],
[ 0.26, 0.73],
[ 0.76, 0.69],
[ 0.79, 0.35],
[ 0.82, 0.24],
[ 0.37, 0.53],
[ 0.93, 0.01],
[ 0.87, 0.42],
[ 0.94, 0.5 ]])
Here's a one-liner that doesn't need numpy:
[list(a) for a in zip(sum(x, []), sum(y, []))]
sum(x, []) flattens the list of list into a single flat list. Then we zip the two lists together and list out the elements.
You may do it like this by using ravel() and numpy.concatenate(x,y,axis):
np.concatenate((np.reshape(x.ravel(),(-1,1)),np.reshape(y.ravel(),(-1,1))),axis=1)
[[ 0.34 0.21]
[ 0.26 0.73]
[ 0.76 0.69]
[ 0.79 0.35]
[ 0.82 0.24]
[ 0.37 0.53]
[ 0.93 0.01]
[ 0.87 0.42]
[ 0.94 0.5 ]]
Here are some more ways to do the same thing. In terms of readability, numpy.ndarray.flatten is more straightforward.
Input arrays:
In [207]: arr1
Out[207]:
array([[0.34, 0.26, 0.76],
[0.79, 0.82, 0.37],
[0.93, 0.87, 0.94]])
In [208]: arr2
Out[208]:
array([[0.21, 0.73, 0.69],
[0.35, 0.24, 0.53],
[0.01, 0.42, 0.5 ]])
As a first step, flatten them:
In [209]: arr1_flattened = arr1.flatten()[:, np.newaxis]
In [210]: arr1_flattened
Out[210]:
array([[0.34],
[0.26],
[0.76],
[0.79],
[0.82],
[0.37],
[0.93],
[0.87],
[0.94]])
In [211]: arr2_flattened = arr2.flatten()[:, np.newaxis]
In [212]: arr2_flattened
Out[212]:
array([[0.21],
[0.73],
[0.69],
[0.35],
[0.24],
[0.53],
[0.01],
[0.42],
[0.5 ]])
Then concatenate or stack them:
# just horizontally stack (np.hstack) the flattened arrays
In [213]: np.hstack([arr1_flattened, arr2_flattened])
Out[213]:
array([[0.34, 0.21],
[0.26, 0.73],
[0.76, 0.69],
[0.79, 0.35],
[0.82, 0.24],
[0.37, 0.53],
[0.93, 0.01],
[0.87, 0.42],
[0.94, 0.5 ]])
In one-line:
In [205]: np.hstack([arr1.flatten()[:, None], arr2.flatten()[:, None]])
Out[205]:
array([[0.34, 0.21],
[0.26, 0.73],
[0.76, 0.69],
[0.79, 0.35],
[0.82, 0.24],
[0.37, 0.53],
[0.93, 0.01],
[0.87, 0.42],
[0.94, 0.5 ]])
# same thing can be done using np.concatenate
In [206]: np.concatenate([arr1.flatten()[:, None], arr2.flatten()[:, None]], axis=1)
Out[206]:
array([[0.34, 0.21],
[0.26, 0.73],
[0.76, 0.69],
[0.79, 0.35],
[0.82, 0.24],
[0.37, 0.53],
[0.93, 0.01],
[0.87, 0.42],
[0.94, 0.5 ]])
Note that all the stacking methods (stack, hstack, vstack, dstack, column_stack), call numpy.concatenate() under the hood.
I am trying to do a union of two numpy arrays in the following manner
np.union1d( np.arange(0.1, 0.91, 0.1), np.arange(0.4, 0.81, 0.01) )
The output reads:
array([ 0.1 , 0.2 , 0.3 , 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45,
0.46, 0.47, 0.48, 0.49, 0.5 , 0.5 , 0.51, 0.52, 0.53,
0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.6 , 0.61,
0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 ,
0.7 , 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78,
0.79, 0.8 , 0.8 , 0.9 ])
In the output of this union, the number 0.5 features twice. Even when I use the unique function in numpy, this replication of the number 0.5 doesn't go away. Meaning:
np.unique( np.union1d( np.arange(0.1, 0.91, 0.1), np.arange(0.4, 0.81, 0.01) ) )
also gives the same output. What am I doing wrong? How can I correct this and get the desired output (i.e. have only one occurrence of the number 0.5 in my array?
Given the input array is sorted, using the same philosophy as in this post -
a[np.r_[True,~np.isclose(a[1:] , a[:-1])]]
Sample run -
In [20]: a = np.union1d( np.arange(0.1, 0.91, 0.1), np.arange(0.4, 0.81, 0.01) )
In [21]: a
Out[21]:
array([ 0.1 , 0.2 , 0.3 , 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45,
0.46, 0.47, 0.48, 0.49, 0.5 , 0.5 , 0.51, 0.52, 0.53,
0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.6 , 0.61,
0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 ,
0.7 , 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78,
0.79, 0.8 , 0.8 , 0.9 ])
In [22]: a[np.r_[True,~np.isclose(a[1:] , a[:-1])]]
Out[22]:
array([ 0.1 , 0.2 , 0.3 , 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45,
0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54,
0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63,
0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72,
0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.9 ])
As stated by #ImNt in the comments, this might be due to floating point comparision/precision (probably they are not 0.5 in memory, but 0.500000000001)
You can make a workaround, though. You know your numbers will be at most 2 digits long. Then, you can first np.round the array before applying np.unique.
x = np.union1d( np.arange(0.1, 0.91, 0.1), np.arange(0.4, 0.81, 0.01) )
x = np.round(x, 2) # Round 2 floating points
x = np.unique(x)
Output:
array([ 0.1 , 0.2 , 0.3 , 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45,
0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54,
0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63,
0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72,
0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.9 ])
As I have written in my comment, it will be an issue due to floating point precision and their comparison. If applicable in your particular case I would suggest working with integers and normalizing later on.
For example
x = np.union1d( np.arange(10, 91, 10), np.arange(40, 81, 1) )
x = x/100.0
Output:
[ 0.1 0.2 0.3 0.4 0.41 0.42 0.43 0.44 0.45 0.46 0.47 0.48
0.49 0.5 0.51 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6
0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7 0.71 0.72
0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.8 0.9 ]
Or you could use Fractions:
>>> import numpy as np
>>> from fractions import Fraction
>>> np.union1d( np.arange(Fraction(1,10), Fraction(91,100), Fraction(1,10)), np.arange(Fraction(4,10), Fraction(81,100),Fraction(1,100)))
array([Fraction(1, 10), Fraction(1, 5), Fraction(3, 10), Fraction(2, 5),
Fraction(41, 100), Fraction(21, 50), Fraction(43, 100),
Fraction(11, 25), Fraction(9, 20), Fraction(23, 50),
Fraction(47, 100), Fraction(12, 25), Fraction(49, 100),
Fraction(1, 2), Fraction(51, 100), Fraction(13, 25),
Fraction(53, 100), Fraction(27, 50), Fraction(11, 20),
Fraction(14, 25), Fraction(57, 100), Fraction(29, 50),
Fraction(59, 100), Fraction(3, 5), Fraction(61, 100),
Fraction(31, 50), Fraction(63, 100), Fraction(16, 25),
Fraction(13, 20), Fraction(33, 50), Fraction(67, 100),
Fraction(17, 25), Fraction(69, 100), Fraction(7, 10),
Fraction(71, 100), Fraction(18, 25), Fraction(73, 100),
Fraction(37, 50), Fraction(3, 4), Fraction(19, 25),
Fraction(77, 100), Fraction(39, 50), Fraction(79, 100),
Fraction(4, 5), Fraction(9, 10)], dtype=object)
>>> _.astype(float)
array([ 0.1 , 0.2 , 0.3 , 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45,
0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54,
0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63,
0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72,
0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.9 ])
I have an array
x1 = tf.Variable([[0.51, 0.52, 0.53, 0.94, 0.35],
[0.32, 0.72, 0.83, 0.74, 0.55],
[0.23, 0.72, 0.63, 0.64, 0.35],
[0.11, 0.02, 0.03, 0.14, 0.15],
[0.01, 0.72, 0.73, 0.04, 0.75]],tf.float32)
I want to sort the elements in each row from min to max. Is there any function for doing such ?
In the example here they are using tf.nn.top_k2d array,using this I can loop to create the max to min.
def sort(instance):
sorted = []
rows = tf.shape(instance)[0]
col = tf.shape(instance)[1]
for i in range(rows.eval()):
matrix.append([tf.gather(instance[i], tf.nn.top_k(instance[i], k=col.eval()).indices)])
return matrix
Is there any thing similar for finding the min to max or how to reverse the array in each row ?
As suggested by #Yaroslav you can just use the top_k values.
a = tf.Variable([[0.51, 0.52, 0.53, 0.94, 0.35],
[0.32, 0.72, 0.83, 0.74, 0.55],
[0.23, 0.72, 0.63, 0.64, 0.35],
[0.11, 0.02, 0.03, 0.14, 0.15],
[0.01, 0.72, 0.73, 0.04, 0.75]],tf.float32)
row_size = a.get_shape().as_list()[-1]
top_k = tf.nn.top_k(-a, k=row_size)
sess.run(-top_k.values)
this prints for me
array([[ 0.34999999, 0.50999999, 0.51999998, 0.52999997, 0.94 ],
[ 0.31999999, 0.55000001, 0.72000003, 0.74000001, 0.82999998],
[ 0.23 , 0.34999999, 0.63 , 0.63999999, 0.72000003],
[ 0.02 , 0.03 , 0.11 , 0.14 , 0.15000001],
[ 0.01 , 0.04 , 0.72000003, 0.73000002, 0.75 ]], dtype=float32)