python indexing and assign by row ndarray - python

Something is wrong in my script and I found the error, but I'm completely stuck.
there is array b which contains two elements:
b = np.zeros ((1,2))
b[0,0] = 272
b[0,1] = 1578
I want to check if there are elements in the second columns, greater than a value, and if so, assign that value a zero.
the command
b[ b[:,1] >= 1000 ] = 0
changes both elements to 0 instead of b[0,1]
what am I missing?
thanks in advance
C

If I understand you correctly, you only want to set the second column to zero (if its value is >1000)? I extended your example to have at least two rows, but just tested it to work also with one:
b = np.array([[123, 456],
[789, 101112]])
mask = b[:,1] > 1000
b[mask,1] = 0
print b
I defined mask to explain it better - you can substitute it in.
mask is then a boolean vector with one element for each row, in this case [False, True]. In the last step, that mask is used to mask out the selected rows and assign zero to the first column element.

I think you could loop over b ?
b_rows = 1 #number of your rows
for i in range(b_rows):
if b[i,1] >= 1000:
b[i,1]=0

Related

How to identify / find a 1D numpy array pattern in a 2D numpy array?

I have a 2D numpy array, and am trying to find the entries where it is equal to a 1D array, but the dimensions of these two arrays prohibit broadcasting. Specifically, my 2D array is like 300x400, and I want to see where it's equal to a 2 element row vector [1, -1].
I am trying to find the location of pixels in an image that are on the border of segmentations. This is denoted in this mask by a 1 being adjacent to the foreground and -1 to the background. So I need to find locations where [1,-1] occurs in the rows of the mask, let's say a.
I have tried a == [1,-1] but this just performs object-level equality and returns False.
I guess I could do this with
for i in range(a.shape[0]):
for j in range(a.shape[1]-1):
if a[i,j] == 1:
if a[i,j+1] == -1:
print(i)
but is there not some cute way to do this with a numpy method or something? I hate loops
arr = np.array([[1,1,-1],[1,-1,-1]])
arr_idx = (arr==1)[:,:-1] & (arr==-1)[:,1:]
Gives
>>> arr_idx
array([[False, True],
[ True, False]])
Which is an index for things that meet your criteria. Note that this is shaped with one fewer column than your input matrix (for obvious reasons).
You can add a column on one side or the other to change the indexing to either side of the pair you're looking for.
arr_idx = np.concatenate((np.zeros(shape=(2, 1), dtype=bool), arr_idx), axis=1)
>>> arr_idx
array([[False, False, True],
[False, True, False]])
Sticking a new column on the left gives the index for the -1 component of the pair.
You can use a 2d cross-correlation for this.
There is a function in scipy for this: signal.correlate2d()
import numpy as np
from scipy import signal
arr = np.array([[1,1,-1],
[1,-1,-1]])
# the pattern you are looking for, has to be 2d
krn = np.array([[1,-1]])
res = signal.correlate2d(arr, krn, mode='same')
print(res)
The result is
[[ 0 2 -1]
[ 2 0 -1]]
it is the higher the better the match. In your case the 2 indicates the positions where your pattern is found.
Check for a 1 starting with the first index (drop the last as no -1 can be after it)
tmp1 = a[:,0:-1]
t1 = tmp1==1
t1 is True wherever the shifted a is 1
Check for -1 starting from the first till the last index
tmp2 = a[:,1:]
t2 = tmp2 ==-1
t1 and t2 contain True and False (shifted according to your goal)
t1*t2 will give you True in the rows where your condition is fulfilled. Summing over the rows gives you a number above zero at every row index i
res = np.sum(t1*t2,axis=1)
desired_Indices = np.where(res>0)
... everybody hates loops ;)

Using numpy where on multidimensional array but need to control indexing

I need to modify elements of an 3D array if they exceed some threshold value. The modification is based upon related elements of another array. More concretely:
A_ijk = A_ijk if A_ijk < threshold value
= (B_(i-1)jk + B_ijk) / 2, otherwise
Numpy.where provides most of the functionality I need, but I don't know how to iterate over the first index without an explicit loop. The follow code does what I want, but uses a loop. Is there a better way? Assume A and B are same shape.
for i in xrange(A.shape[0]):
A[i] = numpy.where(A[i] <= threshold, A[i], (B[i - 1] + B[i]) / 2)
To address the comments below: The first few rows of A are guaranteed to be below threshold. This keeps the i index from looping over to the last entry of A.
You can vectorize your operation by using boolean indexing to replace the elements of A that are above the threshold. A little care has to be taken, since the auxiliary array corresponding to (B[i-1] + B[i])/2 has one less size along the first dimension than A, so we have to explicitly ignore the first row of A (knowing that they are all below the threshold, as explained in the question):
import numpy as np
# some dummy data
A = np.random.rand(3,4,5)
B = np.random.rand(3,4,5)
threshold = 0.5
A[0,:] *= threshold # put the first dummy row below threshhold
mask = A[1:] > threshold # to be overwritten, shape (2,4,5)
replace = (B[:-1] + B[1:])/2 # to overwrite elements in A from, shape (2,4,5)
# replace corresponding elements where `mask` is True
A[1:][mask] = replace[mask]
You can use where to directly index into ndarray:
a = np.random.rand(4,3,10)
b = np.zeros(a.shape)
idx = np.where(a < .1)
print(a)
a[idx] = b[idx]
print(a)
If a for-loop is needed however, just convert the ravel the indices and update.
a = np.random.rand(4,3,10)
b = np.zeros(a.shape)
idx = [np.ravel_multi_index(i, a.shape) for i in zip(*np.where(a < .1))]
print(a, idx)
for i in idx:
a.ravel()[i] = b.ravel()[i]
print(a)

Calculation between values in numpy array

New to python and need some help. I have a numpy array tuple with a shape of (1, 8760) with numbers within each of the 8760 locations. I've been trying to calculate if the values is between -2 and 2 then my new variable will be 0 else just keep the same value in the new variable. Here is what I tried and many others but I probably don't understand the array concept fully.
for x in flow:
if 2 > x < -2:
lflow = 0
else:
lflow = flow
I get this error:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
From what I read those functions gives me a true or false but I want to calculate of value instead of telling me if it matches or not. Please help.
Thanks
If your shape is (1,8760) an array of 8760 elements is assigned to x in your iteration, because the loop iterates the first axis containing one element of size 8760.
Furthermore I'd suggest to use "where" function instead of a loop:
# create a random array with 100 values in the range [-5,5]
a = numpy.random.random(100)*10 - 5
# return an array with all elements within that range set to 0
print numpy.where((a < -2) | (a > 2), a, 0)
Numpy uses boolean masks to select or assign values to arrays in bulk. For example, given the array
A = np.array([-3,-1,-2,0,1,5,2])
This mask represents all the values in A that are less than -2 or greater than 2
mask = (A < -2) | (A > 2)
Then use it to assign those values to 0
A[mask] = 0
This is much faster than using a regular loop in python, since numpy will perform this operation in c or fortran code

Walk through each column in a numpy matrix efficiently in Python

I have a very big two-dimensions array in Python, using numpy library. I want to walk through each column efficiently and check each time if elements are different from 0 to count their number in every column.
Suppose I have the following matrix.
M = array([[1,2], [3,4]])
The following code enables us to walk through each row efficiently, for example (it is not what I intend to do of course!):
for row_idx, row in enumerate(M):
print "row_idx", row_idx, "row", row
for col_idx, element in enumerate(row):
print "col_idx", col_idx, "element", element
# update the matrix M: square each element
M[row_idx, col_idx] = element ** 2
However, in my case I want to walk through each column efficiently, since I have a very big matrix.
I've heard that there is a very efficient way to achieve this using numpy, instead of my current code:
curr_col, curr_row = 0, 0
while (curr_col < numb_colonnes):
result = 0
while (curr_row < numb_rows):
# If different from 0
if (M[curr_row][curr_col] != 0):
result += 1
curr_row += 1
.... using result value ...
curr_col += 1
curr_row = 0
Thanks in advance!
In the code you showed us, you treat numpy's arrays as lists and for what you can see, it works! But arrays are not lists, and while you can treat them as such it wouldn't make sense to use arrays, or even numpy.
To really exploit the usefulness of numpy you have to operate directly on arrays, writing, e.g.,
M = M*M
when you want to square the elements of an array and using the rich set of numpy functions to operate directly on arrays.
That said, I'll try to get a bit closer to your problem...
If your intent is to count the elements of an array that are different from zero, you can use the numpy function sum.
Using sum, you can obtain the sum of all the elements in an array, or you can sum across a particular axis.
import numpy as np
a = np.array(((3,4),(5,6)))
print np.sum(a) # 18
print np.sum(a, axis=0) # [8, 10]
print np.sum(a, axis=1) # [7, 11]
Now you are protesting: I don't want to sum the elements, I want to count the non-zero elements... but
if you write a logical test on an array, you obtain an array of booleans, e.g, we want to test which elements of a are even
print a%2==0
# [[False True]
# [False True]]
False is zero and True is one, at least when we sum it...
print np.sum(a%2==0) # 2
or, if you want to sum over a column, i.e., the index that changes is the 0-th
print np.sum(a%2==0, axis=0) # [0 2]
or sum across a row
print np.sum(a%2==0, axis=1) # [1 1]
To summarize, for your particular use case
by_col = np.sum(M!=0, axis=0)
# use the counts of non-zero terms in each column, stored in an array
...
# if you need the grand total, use sum again
total = np.sum(by_col)

Array element evaluation from reverse

I'm still very new to python and programing and I'm trying to figure out if I'm going about this problem in the correct fashion. I tend to have a matlab approach to things but here I'm just struggling...
Context:
I have two numpy arrays plotted in this image on flickr since I can't post photos here :(. They are of equal length properties (both 777x1600) and I'm trying to use the red array to help return the index(value on the x-axis of plot) and element value(y-axis) of the point in the blue plot indicated by the arrow for each row of the blue array.
The procedure I've been tasked with was to:
a) determine max value of red array (represented with red dot in figure and already achieved)
and b) Start at the end of the blue array with the final element and count backwards, comparing element to preceding element. Goal being to determine where the preceding value decreases. (for example, when element -1 is greater than element -2, indicative of the last peak in the image). Additionally, to prevent selecting "noise" at the tail end of the section with elevated values, I also need to constrain the selected value to be larger than the maximum of the red array.
Here's what I've got so far, but I'm stuck on line two where I have to evaluate the selected row of the array from the (-1) position in the row to the beginning, or (0) position:
for i,n in enumerate(blue): #select each row of blue in turn to analyze
for j,m in enumerate(n): #select each element of blue ??how do I start from the end of array and work backwards??
if m > m-1 and m > max_val_red[i]:
indx_m[i] = j
val_m[i] = m
To answer you question directly, you can use n[::-1] to reverse the arrray n.
So the code is :
for j, m in enumerate(n[::-1]):
j = len(n)-j-1
# here is your code
But to increase calculation speed, you should avoid python loop:
import numpy as np
n = np.array([1,2,3,4,2,5,7,8,3,2,3,3,0,1,1,2])
idx = np.nonzero(np.diff(n) < 0)[0]
peaks = n[idx]
mask = peaks > 3 # peak muse larger than 3
print "index=", idx[mask]
print "value=", peaks[mask]
the output is:
index= [3 7]
value= [4 8]
I assume you mean:
if m > n[j-1] and m > max_val_red[i]:
indx_m[i] = j
val_m[i] = m
because m > m - 1 is always True
To reverse an array on an axis you can index the array using ::-1 on that axis, for example to reverse blue on axis 1 you can use:
blue_reverse = blue[:, ::-1]
Try and see you can write your function as a set of array operations instead of loops (that tends to be much faster). This is similar to the other answer, but it should allow you avoid both loops you're currently using:
threshold = red.max(1)
threshold = threshold[:, np.newaxis] #this makes threshold's shape (n, 1)
blue = blue[:, ::-1]
index_from_end = np.argmax((blue[:, :-1] > blue[:, 1:]) & (blue[:, :-1] > threshold), 1)
value = blue[range(len(blue)), index_from_end]
index = blue.shape[1] - 1 - index_from_end
Sorry, I didn't read all of it but you can possibly look into the built in function reversed.
so instead of enumerate( n ). you can do reversed( enumerate( n ) ). But then your index would be wrong the correct index would be eval to len( n ) - j

Categories