Im trying to create a function that will transform a regular Matrix into CSR form (I don't want to use the scipy.sparse one).
To do this, I'm using a nested for-loop to run through a given matrix to create a new matrix with three rows.
The first row ('Values') should contain all non-zero values. The second ('Cols') should contain the column index for each number in 'Values'. The third row should contain the index value in 'Values' for the first non-zero value on each row.
My question regards the second and third rows:
Is there a way of getting the column ID for the element 'i' in the for-loop?
M=array([[4,0,39],
[0,5,0],
[0,0,7]])
def Convert(x):
CSRMatrix = []
Values = []
Cols = []
Rows = []
for k in x:
for i in k:
if i != 0:
Values.append(i)
Cols.append({#the column index value of 'i'})
Rows.append[#theindex in 'Values' of the first non-zero element on each row]
CSRMatrix.append(Values)
CSRMatrix.append(Cols)
CSRMatrix.append(Rows)
return(CSRMatrix)
Convert(M)
I'm not sure of what you want exactly for Cols.append() because of the way you commented it in the code between curly braces.
Is it a dict containing the index:value of all non 0 value? Or a list of sets containing the indexes of all non 0 values (which would be weird), or is it all the indexes of each row in your array?
Anyway I put the 2 most likely candidates (dict and list of indexes for each row) test each one and delete the unwanted one and if none are right please add some more specifics:
import numpy as np
m = np.array([[4,0,39],
[0,5,0],
[0,0,7]])
def Convert(x):
CSRMatrix = []
Values = []
Cols = []
Rows = []
for num in x:
for i in range(len(num)):
if num[i] != 0:
Values.append(num[i])
Cols.append({i:num[i]}) # <- if dict. Remove if not what you wanted
Rows.append(i)
Cols.append(i) # <- list of all indexes in the array for each row. Remove if not what you wanted
CSRMatrix.append(Values)
CSRMatrix.append(Cols)
CSRMatrix.append(Rows)
return(CSRMatrix)
x = Convert(m)
print(x)
enumerate() passes an index for every iteration.
Thereby the second row can be easily created by appending num2.
For the third row you have to check again if you have already added a value in that row. If not append num2 and set the non_zero check to False. For the next row non_zero check is set to True again.
def Convert(x):
CSRMatrix = []
Values = []
Cols = []
Rows = []
for num, k in enumerate(x):
non_zero = True
for num2, i in enumerate(k):
if i != 0:
Values.append(i)
Cols.append(num2)
if non_zero:
Rows.append(num2)
non_zero = False
CSRMatrix.append(Values)
CSRMatrix.append(Cols)
CSRMatrix.append(Rows)
return (CSRMatrix)
Here is a numpythonic implementation, use the nonzero method to directly obtain the row and column index of non-zero elements, and then use a comparison to generate a mask. Finally, use nonzero for the mask to get the row indices:
>>> M = np.array([[ 4, 0, 39],
... [ 0, 5, 0],
... [ 0, 0, 7]])
>>> r, c = M.nonzero()
>>> mask = np.concatenate(([True], r[1:] != r[:-1]))
>>> [M[r, c], c, *mask.nonzero()]
[array([ 4, 39, 5, 7]), array([0, 2, 1, 2]), array([0, 2, 3])]
Test of a larger array:
>>> a = np.random.choice(10, size=(8, 8), p=[0.73] + [0.03] * 9)
>>> a
array([[0, 0, 0, 0, 8, 0, 0, 1],
[1, 0, 5, 4, 0, 0, 9, 0],
[0, 0, 9, 0, 0, 0, 0, 1],
[0, 0, 0, 8, 9, 0, 0, 4],
[0, 0, 5, 0, 0, 6, 0, 0],
[0, 8, 0, 0, 0, 0, 0, 9],
[0, 0, 0, 0, 0, 0, 0, 9],
[0, 9, 0, 0, 0, 4, 0, 0]])
>>> r, c = a.nonzero()
>>> mask = np.concatenate(([True], r[1:] != r[:-1]))
>>> pp([a[r, c], c, *mask.nonzero()])
[array([8, 1, 1, 5, 4, 9, 9, 1, 8, 9, 4, 5, 6, 8, 9, 9, 9, 4]),
array([4, 7, 0, 2, 3, 6, 2, 7, 3, 4, 7, 2, 5, 1, 7, 7, 1, 5], dtype=int64),
array([ 0, 2, 6, 8, 11, 13, 15, 16], dtype=int64)]
Related
I would like to know the fastest way to extract the indices of the first n non zero values per column in a 2D array.
For example, with the following array:
arr = [
[4, 0, 0, 0],
[0, 0, 0, 0],
[0, 4, 0, 0],
[2, 0, 9, 0],
[6, 0, 0, 0],
[0, 7, 0, 0],
[3, 0, 0, 0],
[1, 2, 0, 0],
With n=2 I would have [0, 0, 1, 1, 2] as xs and [0, 3, 2, 5, 3] as ys. 2 values in the first and second columns and 1 in the third.
Here is how it is currently done:
x = []
y = []
n = 3
for i, c in enumerate(arr.T):
a = c.nonzero()[0][:n]
if len(a):
x.extend([i]*len(a))
y.extend(a)
In practice I have arrays of size (405, 256).
Is there a way to make it faster?
Here is a method, although quite confusing as it uses a lot of functions, that does not require sorting the array (only a linear scan is necessary to get non null values):
n = 2
# Get indices with non null values, columns indices first
nnull = np.stack(np.where(arr.T != 0))
# split indices by unique value of column
cols_ids= np.array_split(range(len(nnull[0])), np.where(np.diff(nnull[0]) > 0)[0] +1 )
# Take n in each (max) and concatenate the whole
np.concatenate([nnull[:, u[:n]] for u in cols_ids], axis = 1)
outputs:
array([[0, 0, 1, 1, 2],
[0, 3, 2, 5, 3]], dtype=int64)
Here is one approach using argsort, it gives a different order though:
n = 2
m = arr!=0
# non-zero values first
idx = np.argsort(~m, axis=0)
# get first 2 and ensure non-zero
m2 = np.take_along_axis(m, idx, axis=0)[:n]
y,x = np.where(m2)
# slice
x, idx[y,x]
# (array([0, 1, 2, 0, 1]), array([0, 2, 3, 3, 5]))
Use dislocation comparison for the row results of the transposed nonzero:
>>> n = 2
>>> i, j = arr.T.nonzero()
>>> mask = np.concatenate([[True] * n, i[n:] != i[:-n]])
>>> i[mask], j[mask]
(array([0, 0, 1, 1, 2], dtype=int64), array([0, 3, 2, 5, 3], dtype=int64))
I have a mask array: [0,0,0,0,1,1,0,0,1,1,0,1,0].
And a values array: [3,4,5,6,7]
Which is the best way that I can replace all value 1 in mask array into the values array?
Expected result: [0,0,0,0,3,4,0,0,5,6,0,7,0]
I am working with large array.
Assuming numpy, and a values length equal to the number of 1s.
Use boolean indexing:
mask = np.array([0,0,0,0,1,1,0,0,1,1,0,1,0])
values = [3,4,5,6,7]
mask[mask==1] = values
If values can be longer than the sum of 1s:
m = mask==1
mask[m] = values[:m.sum()]
Output:
array([0, 0, 0, 0, 3, 4, 0, 0, 5, 6, 0, 7, 0])
You can use iterator:
mask = [0,0,0,0,1,1,0,0,1,1,0,1,0]
nums = iter([3,4,5,6,7])
output = [next(nums) if m else 0 for m in mask]
print(output) # [0, 0, 0, 0, 3, 4, 0, 0, 5, 6, 0, 7, 0]
Let's say we have a 1d numpy array filled with some int values. And let's say that some of them are 0.
Is there any way, using numpy array's power, to fill all the 0 values with the last non-zero values found?
for example:
arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])
fill_zeros_with_last(arr)
print arr
[1 1 1 2 2 4 6 8 8 8 8 8 2]
A way to do it would be with this function:
def fill_zeros_with_last(arr):
last_val = None # I don't really care about the initial value
for i in range(arr.size):
if arr[i]:
last_val = arr[i]
elif last_val is not None:
arr[i] = last_val
However, this is using a raw python for loop instead of taking advantage of the numpy and scipy power.
If we knew that a reasonably small number of consecutive zeros are possible, we could use something based on numpy.roll. The problem is that the number of consecutive zeros is potentially large...
Any ideas? or should we go straight to Cython?
Disclaimer:
I would say long ago I found a question in stackoverflow asking something like this or very similar. I wasn't able to find it. :-(
Maybe I missed the right search terms, sorry for the duplicate then. Maybe it was just my imagination...
Here's a solution using np.maximum.accumulate:
def fill_zeros_with_last(arr):
prev = np.arange(len(arr))
prev[arr == 0] = 0
prev = np.maximum.accumulate(prev)
return arr[prev]
We construct an array prev which has the same length as arr, and such that prev[i] is the index of the last non-zero entry before the i-th entry of arr. For example, if:
>>> arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])
Then prev looks like:
array([ 0, 0, 0, 3, 3, 5, 6, 7, 7, 7, 7, 7, 12])
Then we just index into arr with prev and we obtain our result. A test:
>>> arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])
>>> fill_zeros_with_last(arr)
array([1, 1, 1, 2, 2, 4, 6, 8, 8, 8, 8, 8, 2])
Note: Be careful to understand what this does when the first entry of your array is zero:
>>> fill_zeros_with_last(np.array([0,0,1,0,0]))
array([0, 0, 1, 1, 1])
Inspired by jme's answer here and by Bas Swinckels' (in the linked question) I came up with a different combination of numpy functions:
def fill_zeros_with_last(arr, initial=0):
ind = np.nonzero(arr)[0]
cnt = np.cumsum(np.array(arr, dtype=bool))
return np.where(cnt, arr[ind[cnt-1]], initial)
I think it's succinct and also works, so I'm posting it here for the record. Still, jme's is also succinct and easy to follow and seems to be faster, so I'm accepting it :-)
If the 0s only come in strings of 1, this use of nonzero might work:
In [266]: arr=np.array([1,0,2,3,0,4,0,5])
In [267]: I=np.nonzero(arr==0)[0]
In [268]: arr[I] = arr[I-1]
In [269]: arr
Out[269]: array([1, 1, 2, 3, 3, 4, 4, 5])
I can handle your arr by applying this repeatedly until I is empty.
In [286]: arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])
In [287]: while True:
.....: I=np.nonzero(arr==0)[0]
.....: if len(I)==0: break
.....: arr[I] = arr[I-1]
.....:
In [288]: arr
Out[288]: array([1, 1, 1, 2, 2, 4, 6, 8, 8, 8, 8, 8, 2])
If the strings of 0s are long it might be better to look for those strings and handle them as a block. But if most strings are short, this repeated application may be the fastest route.
So let's say I have a list that looks like:
x = [1, 0, 0, 1, 1, 1, 0, 0, 0, 0]
I then have another list with indices that needs to be removed from list x:
x_remove = [1, 4, 5]
I can then use the numpy command delete to remove this from x and end up with:
x_final = np.delete(x, x_remove)
>>> x_final = [0, 0, 1, 0, 0, 0, 0]
So far so good. Now I then figure out that I don't want to use the entire list x, but start perhaps from index 2. So basically:
x_new = x[2:]
>>> x_new = [0, 1, 1, 1, 0, 0, 0, 0]
I do however still need to remove the indices from the x_remove list, but now, as you can see, the indices are not the same placement as before, so the wrong items are removed. And same thing will happen if I do it the other way around (i.e. first removing the indices, and then use slice to start at index 2). So basically it will/should look like:
x_new_final = [0, 1, 1, 0, 0] (first use slice, and the remove list)
x_new_final_v2 = [1, 0, 0, 0, 0] (first use remove list, and then slice)
x_new_final_correct_one = [0, 1, 0, 0, 0, 0] (as it should be)
So is there some way in which I can start my list at various indices (through slicing), and still use the delete command to remove the correct indices that would correspond to the full list ?
You could change the x_remove list depending on the slice location. For example:
slice_location = 2
x = [1, 0, 0, 1, 1, 1, 0, 0, 0, 0]
x_remove = [1, 4, 5]
x_new=x[slice_location:]
x_remove = [x-slice_location for x in x_remove if x-slice_location>0]
x_new = np.delete(x, x_remove)
x = [1, 0, 0, 1, 1, 1, 0, 0, 0, 0]
x_remove = [1, 4, 5]
for index,value in enumerate(x):
for remove_index in x_remove:
if(index == remove_index-1):
x[index] = ""
final_list = [final_value for final_value in x if(final_value != "")]
print(final_list)
Try it this simple way...
First let's explore alternatives for the simple removal (with out this change in starting position issue):
First make an x with unique and easily recognized values:
In [787]: x = list(range(10))
In [788]: x
Out[788]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
A list comprehension method - maybe not the fastest, but fairly clear and bug free:
In [789]: [v for i,v in enumerate(x) if i not in x_remove]
Out[789]: [0, 2, 3, 6, 7, 8, 9]
Your np.delete approach:
In [790]: np.delete(x, x_remove)
Out[790]: array([0, 2, 3, 6, 7, 8, 9])
That has a downside of converting x to an array, which is not a trivial task (time wise). It also makes a new array. My guess is that it is slower.
Try in place removeal:
In [791]: y=x[:]
In [792]: for i in x_remove:
...: del y[i]
...:
In [793]: y
Out[793]: [0, 2, 3, 4, 6, 8, 9]
oops - wrong. We need to start from the end (largest index). This is a well known Python 'recipe':
In [794]: y=x[:]
In [795]: for i in x_remove[::-1]:
...: del y[i]
...:
...:
In [796]: y
Out[796]: [0, 2, 3, 6, 7, 8, 9]
Under the covers np.delete is taking a masked approach:
In [797]: arr = np.array(x)
In [798]: arr
Out[798]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [799]: mask = np.ones(arr.shape, bool)
In [800]: mask[x_remove] = False
In [801]: mask
Out[801]:
array([ True, False, True, True, False, False, True, True, True,
True])
In [802]: arr[mask]
Out[802]: array([0, 2, 3, 6, 7, 8, 9])
Now to the question of applying x_remove to a slice of x. The slice of x does not have a record of the slice parameters. That is you can't readily determine that y = x[2:] is missing two values. (Well, I could deduce it by comparing some attributes of x and y, but not from y alone).
So regardless of how you do the delete, you will have to first adjust the values of x_remove.
In [803]: x2 = np.array(x_remove)-2
In [804]: x2
Out[804]: array([-1, 2, 3])
In [805]: [v for i,v in enumerate(x[2:]) if i not in x2]
Out[805]: [2, 3, 6, 7, 8, 9]
This works ok, but that -1 is potentially a problem. We don't want it mean the last element. So we have to first filter out the negative indicies to be safe.
In [806]: np.delete(x[2:], x2)
/usr/local/bin/ipython3:1: FutureWarning: in the future negative indices will not be ignored by `numpy.delete`.
#!/usr/bin/python3
Out[806]: array([2, 3, 6, 7, 8, 9])
If delete didn't ignore negative indices, it could get a mask like this - with a False at the end:
In [808]: mask = np.ones(arr[2:].shape, bool)
In [809]: mask[x2] = False
In [810]: mask
Out[810]: array([ True, True, False, False, True, True, True, False])
I have the following array
a = [1, 2, 3, 0, 0, 0, 0, 0, 0, 4, 5, 6, 0, 0, 0, 0, 9, 8, 7,0,10,11]
I would like to find the start and the end index of the array where the values are zeros consecutively. For the array above the output would be as follows
[3,8],[12,15],[19]
I want to achieve this as efficiently as possible.
Here's a fairly compact vectorized implementation. I've changed the requirements a bit, so the return value is a bit more "numpythonic": it creates an array with shape (m, 2), where m is the number of "runs" of zeros. The first column is the index of the first 0 in each run, and the second is the index of the first nonzero element after the run. (This indexing pattern matches, for example, how slicing works and how the range function works.)
import numpy as np
def zero_runs(a):
# Create an array that is 1 where a is 0, and pad each end with an extra 0.
iszero = np.concatenate(([0], np.equal(a, 0).view(np.int8), [0]))
absdiff = np.abs(np.diff(iszero))
# Runs start and end where absdiff is 1.
ranges = np.where(absdiff == 1)[0].reshape(-1, 2)
return ranges
For example:
In [236]: a = [1, 2, 3, 0, 0, 0, 0, 0, 0, 4, 5, 6, 0, 0, 0, 0, 9, 8, 7, 0, 10, 11]
In [237]: runs = zero_runs(a)
In [238]: runs
Out[238]:
array([[ 3, 9],
[12, 16],
[19, 20]])
With this format, it is simple to get the number of zeros in each run:
In [239]: runs[:,1] - runs[:,0]
Out[239]: array([6, 4, 1])
It's always a good idea to check the edge cases:
In [240]: zero_runs([0,1,2])
Out[240]: array([[0, 1]])
In [241]: zero_runs([1,2,0])
Out[241]: array([[2, 3]])
In [242]: zero_runs([1,2,3])
Out[242]: array([], shape=(0, 2), dtype=int64)
In [243]: zero_runs([0,0,0])
Out[243]: array([[0, 3]])
You can use itertools to achieve your expected result.
from itertools import groupby
a= [1, 2, 3, 0, 0, 0, 0, 0, 0, 4, 5, 6, 0, 0, 0, 0, 9, 8, 7,0,10,11]
b = range(len(a))
for group in groupby(iter(b), lambda x: a[x]):
if group[0]==0:
lis=list(group[1])
print [min(lis),max(lis)]
Here is a custom function, not sure the most efficient but works :
def getZeroIndexes(li):
begin = 0
end = 0
indexes = []
zero = False
for ind,elt in enumerate(li):
if not elt and not zero:
begin = ind
zero = True
if not elt and zero:
end = ind
if elt and zero:
zero = False
if begin == end:
indexes.append(begin)
else:
indexes.append((begin, end))
return indexes