Related
i have a numpy array like the XY coordinates here below:
2d_coords = [
[1,2]
[1,1]
[2,1]
[3,1]
...
]
either [1,1] or [1,2] need to go (doesn't care which one) , only one point on the X coordinate is possible.
How can I do that ?
numpy.unique would be helpful. For example,
import numpy as np
l = np.asarray([
[1, 2],
[1, 1],
[2, 1],
[3, 1],
])
_, unique_indices = np.unique(l[:, 0], return_index=True) # get the indices with unique x coordinates
print(l[unique_indices])
The example output:
[[1 2]
[2 1]
[3 1]]
You can use NumPy and matplotlib:
import numpy as np
import matplotlib.pyplot as plt
coords = np.array([[1, 2], [1, 1], [2, 1], [3, 1]])
plot_coords = coords[np.unique(coords[:,0])].T
plt.plot(plot_coords[0], plot_coords[1])
plt.show()
What about pandas?
pd.DataFrame(coords).drop_duplicates(0).values
array([[1, 2],
[2, 1],
[3, 1]])
Without using any external library, you can use a conditional list comprehension:
d_coords = [[1,2],[1,1],[2,1],[3,1]]
new_list = [d_coords[i] for i in range(len(d_coords)) if d_coords[i][0] not in [k[0] for k in d_coords[:i]]]
# new_list: [[1, 2], [2, 1], [3, 1]]
NOTE: don't start variable names with numbers
I'm new with Python and programming in general.
I want to create a function that multiplies two np.array of the same size and get their scalar value, for example:
matrix_1 = np.array([[1, 1], [0, 1], [1, 0]])
matrix_2 = np.array([[1, 2], [1, 1], [0, 0]])
I want to get 4 as output ((1 * 1) + (1 * 2) + (0 * 1) + (1 * 1) + (1 * 0) + (0 * 0))
Thanks!
Multiply two matrices element-wise
Sum all the elements
multiplied_matrix = np.multiply(matrix_1,matrix_2)
sum_of_elements = np.sum(multiplied_matrix)
print(sum_of_elements) # 4
Or in one shot:
print(np.sum(np.multiply(matrix_1, matrix_2))) # 4
You can make use of np.multiply() to multiply the two arrays elementwise, then we call np.sum() on this matrix. So we thus can calculate the result with:
np.multiply(matrix_1, matrix_2).sum()
For your given sample matrix, we thus obtain:
>>> matrix_1 = np.array([[1, 1], [0, 1], [1, 0]])
>>> matrix_2 = np.array([[1, 2], [1, 1], [0, 0]])
>>> np.multiply(matrix_1, matrix_2)
array([[1, 2],
[0, 1],
[0, 0]])
>>> np.multiply(matrix_1, matrix_2).sum()
4
There are a couple of ways to do it (Frobenius inner product) using numpy, e.g.
np.sum(A * B)
np.dot(A.flatten(), B.flatten())
np.trace(np.dot(A, B.T))
np.einsum('ij,ij', A, B)
One recommended way is using numpy.einsum, since it can be adapted to not only matrices but also multiway arrays (i.e., tensors).
Matrices of the same size
Take the matrices what you give as an example,
>>> import numpy as np
>>> matrix_1 = np.array([[1, 1], [0, 1], [1, 0]])
>>> matrix_2 = np.array([[1, 2], [1, 1], [0, 0]])
then, we have
>>> np.einsum('ij, ij ->', matrix_1, matrix_2)
4
Vectors of the same size
An example like this:
>>> vector_1 = np.array([1, 2, 3])
>>> vector_2 = np.array([2, 3, 4])
>>> np.einsum('i, i ->', vector_1, vector_2)
20
Tensors of the same size
Take three-way arrays (i.e., third-order tensors) as an example,
>>> tensor_1 = np.array([[[1, 2], [3, 4]], [[2, 3], [4, 5]], [[3, 4], [5, 6]]])
>>> print(tensor_1)
[[[1 2]
[3 4]]
[[2 3]
[4 5]]
[[3 4]
[5 6]]]
>>> tensor_2 = np.array([[[2, 3], [4, 5]], [[3, 4], [5, 6]], [[6, 7], [8, 9]]])
>>> print(tensor_2)
[[[2 3]
[4 5]]
[[3 4]
[5 6]]
[[6 7]
[8 9]]]
then, we have
>>> np.einsum('ijk, ijk ->', tensor_1, tensor_2)
248
For more usage about numpy.einsum, I recommend:
Understanding NumPy's einsum
I am learning Python and solving a machine learning problem.
class_ids=np.arange(self.x.shape[0])
np.random.shuffle(class_ids)
self.x=self.x[class_ids]
This is a shuffle function in NumPy but I can't understand what self.x=self.x[class_ids] means. because I think it gives the value of the array to a variable.
It's a very complicated way to shuffle the first dimension of your self.x. For example:
>>> x = np.array([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5]])
>>> x
array([[1, 1],
[2, 2],
[3, 3],
[4, 4],
[5, 5]])
Then using the mentioned approach
>>> class_ids=np.arange(x.shape[0]) # create an array [0, 1, 2, 3, 4]
>>> np.random.shuffle(class_ids) # shuffle the array
>>> x[class_ids] # use integer array indexing to shuffle x
array([[5, 5],
[3, 3],
[1, 1],
[4, 4],
[2, 2]])
Note that the same could be achieved just by using np.random.shuffle because the docstring explicitly mentions:
This function only shuffles the array along the first axis of a multi-dimensional array. The order of sub-arrays is changed but their contents remains the same.
>>> np.random.shuffle(x)
>>> x
array([[5, 5],
[3, 3],
[1, 1],
[2, 2],
[4, 4]])
or by using np.random.permutation:
>>> class_ids = np.random.permutation(x.shape[0]) # shuffle the first dimensions indices
>>> x[class_ids]
array([[2, 2],
[4, 4],
[3, 3],
[5, 5],
[1, 1]])
Assuming self.x is a numpy array:
class_ids is a 1-d numpy array that is being used as an integer array index in the expression: x[class_ids]. Because the previous line shuffled class_ids, x[class_ids] evaluates to self.x shuffled by rows.
The assignment self.x=self.x[class_ids] assigns the shuffled array to self.x
Consider the array a
np.random.seed([3,1415])
a = np.random.randint(0, 10, (10, 2))
a
array([[0, 2],
[7, 3],
[8, 7],
[0, 6],
[8, 6],
[0, 2],
[0, 4],
[9, 7],
[3, 2],
[4, 3]])
What is a vectorized way to get the cumulative argmax?
array([[0, 0], <-- both start off as max position
[1, 1], <-- 7 > 0 so 1st col = 1, 3 > 2 2nd col = 1
[2, 2], <-- 8 > 7 1st col = 2, 7 > 3 2nd col = 2
[2, 2], <-- 0 < 8 1st col stays the same, 6 < 7 2nd col stays the same
[2, 2],
[2, 2],
[2, 2],
[7, 2], <-- 9 is new max of 2nd col, argmax is now 7
[7, 2],
[7, 2]])
Here is a non-vectorized way to do it.
Notice that as the window expands, argmax applies to the growing window.
pd.DataFrame(a).expanding().apply(np.argmax).astype(int).values
array([[0, 0],
[1, 1],
[2, 2],
[2, 2],
[2, 2],
[2, 2],
[2, 2],
[7, 2],
[7, 2],
[7, 2]])
Here's a vectorized pure NumPy solution that performs pretty snappily:
def cumargmax(a):
m = np.maximum.accumulate(a)
x = np.repeat(np.arange(a.shape[0])[:, None], a.shape[1], axis=1)
x[1:] *= m[:-1] < m[1:]
np.maximum.accumulate(x, axis=0, out=x)
return x
Then we have:
>>> cumargmax(a)
array([[0, 0],
[1, 1],
[2, 2],
[2, 2],
[2, 2],
[2, 2],
[2, 2],
[7, 2],
[7, 2],
[7, 2]])
Some quick testing on arrays with thousands to millions of values suggests that this is anywhere between 10-50 times faster than looping at the Python level (either implicitly or explicitly).
I cant think of a way to vectorize this over both columns easily; but if the number of columns is small relative to the number of rows, that shouldn't be an issue and a for loop should suffice for that axis:
import numpy as np
import numpy_indexed as npi
a = np.random.randint(0, 10, (10))
max = np.maximum.accumulate(a)
idx = npi.indices(a, max)
print(idx)
I would like to make a function that computes cumulative argmax for 1d array and then apply it to all columns. This is the code:
import numpy as np
np.random.seed([3,1415])
a = np.random.randint(0, 10, (10, 2))
def cumargmax(v):
uargmax = np.frompyfunc(lambda i, j: j if v[j] > v[i] else i, 2, 1)
return uargmax.accumulate(np.arange(0, len(v)), 0, dtype=np.object).astype(v.dtype)
np.apply_along_axis(cumargmax, 0, a)
The reason for converting to np.object and then converting back is a workaround for Numpy 1.9, as mentioned in generalized cumulative functions in NumPy/SciPy?
I have three rather large NumPy arrays with varying numbers of rows, whose first columns are all integers. My hope is to filter these arrays such that the only rows left are those for whom the value in the first column is shared by all three. This would leave three arrays of the same size. The entries in the other columns are not necessarily shared across arrays.
So, with input:
A =
[[1, 1],
[2, 2],
[3, 3],]
B =
[[2, 1],
[3, 2],
[4, 3],
[5, 4]]
C =
[[2, 2],
[3, 1]
[5, 2]]
I hope to get back as output:
A =
[[2, 2],
[3, 3]]
B =
[[2, 1],
[3, 2]]
C =
[[2, 2],
[3, 1]]
My current approach is to:
Find the intersection of the three first columns using numpy.intersect1d()
Use numpy.in1d() on this intersection and the first columns of each array to find the row indices that are not shared in each array (converting boolean to index using a modified version of the method found here: Python: intersection indices numpy array )
Finally using numpy.delete() with each of these indices and its respective array to remove rows with non-shared entries in the first column.
I'm wondering if there might be a faster or more elegantly Pythonic way to go about this however, something that is suited to very large arrays.
Your indices in your example are sorted and unique. Assuming this is no coincidence (and this situation often arises, or can easily be enforced), the following works:
import numpy as np
A = np.array(
[[1, 1],
[2, 2],
[3, 3],])
B = np.array(
[[2, 1],
[3, 2],
[4, 3],
[5, 4]])
C = np.array(
[[2, 2],
[3, 1],
[5, 2],])
I = reduce(
lambda l,r: np.intersect1d(l,r,True),
(i[:,0] for i in (A,B,C)))
print A[np.searchsorted(A[:,0], I)]
print B[np.searchsorted(B[:,0], I)]
print C[np.searchsorted(C[:,0], I)]
and in case the first column is not in sorted order (but is still unique):
C = np.array(
[[9, 2],
[1,6],
[5, 1],
[2, 5],
[3, 2],])
def index_by_first_column_entry(M, keys):
colkeys = M[:,0]
sorter = np.argsort(colkeys)
index = np.searchsorted(colkeys, keys, sorter = sorter)
return M[sorter[index]]
print index_by_first_column_entry(C, I)
and make sure to change the true to false in
I = reduce(
lambda l,r: np.intersect1d(l,r,False),
(i[:,0] for i in (A,B,C)))
generalization to duplicate values can be made using np.unique
One way to do this is to build an indicator array, or a hash table if you like, to indicate which integers are in all your input arrays. Then you can use boolean indexing based on this indicator array to get the subarrays. Something like this:
import numpy as np
# Setup
A = np.array(
[[1, 1],
[2, 2],
[3, 3],])
B = np.array(
[[2, 1],
[3, 2],
[4, 3],
[5, 4]])
C = np.array(
[[2, 2],
[3, 1],
[5, 2],])
def take_overlap(*input):
n = len(input)
maxIndex = max(array[:, 0].max() for array in input)
indicator = np.zeros(maxIndex + 1, dtype=int)
for array in input:
indicator[array[:, 0]] += 1
indicator = indicator == n
result = []
for array in input:
# Look up each integer in the indicator array
mask = indicator[array[:, 0]]
# Use boolean indexing to get the sub array
result.append(array[mask])
return result
subA, subB, subC = take_overlap(A, B, C)
This should be quite fast and this method does not assume the elements of the input arrays are unique or sorted. However this method could take a lot of memory, and might e a bit slower, if the indexing integers are sparse, ie [1, 10, 10000], but should be close to optimal if the integers are more or less dense.
This works but I'm not sure if it is faster than any of the other answers:
import numpy as np
A = np.array(
[[1, 1],
[2, 2],
[3, 3],])
B = np.array(
[[2, 1],
[3, 2],
[4, 3],
[5, 4]])
C = np.array(
[[2, 2],
[3, 1],
[5, 2],])
a = A[:,0]
b = B[:,0]
c = C[:,0]
ab = np.where(a[:, np.newaxis] == b[np.newaxis, :])
bc = np.where(b[:, np.newaxis] == c[np.newaxis, :])
ab_in_bc = np.in1d(ab[1], bc[0])
bc_in_ab = np.in1d(bc[0], ab[1])
arows = ab[0][ab_in_bc]
brows = ab[1][ab_in_bc]
crows = bc[1][bc_in_ab]
anew = A[arows, :]
bnew = B[brows, :]
cnew = C[crows, :]
print(anew)
print(bnew)
print(cnew)
gives:
[[2 2]
[3 3]]
[[2 1]
[3 2]]
[[2 2]
[3 1]]