We Keep Coding

Is there a fast way to create a bool matrix from another matrix in python?

i would like to know if there is a faster way, not O(n^2), to create a bool matrix out of an integer nxn-matrix.
Example:
given is the matrix:
matrix_int = [[-5,-8,6],[4,6,-9],[7,8,9]]
after transformation i want this:
matrix_bool = [[False,False,True],[True,True,False],[True,True,True]]
so all negative values should be False and all positive values should be True.
The brute force way is O(n^2) and this is too slow for me, too you have any ideas how to make this faster?

matrix_int = [[-5,-8,6],[4,6,-9],[7,8,9]]
matrix_int = np.array(matrix_int)
bool_mat = matrix_int > 0
result:
array([[False, False, True],
[ True, True, False],
[ True, True, True]])

matrix_int = [[-5,-8,6],[4,6,-9],[7,8,9]]
matrix_bool = [[num > 0 for num in row] for row in matrix_int]
# [[False, False, True], [True, True, False], [True, True, True]]

Optimizing a complex list comprehension statement

How can I optimize the list comprehension statement in Step 3?
Background:
In the real world:
r contains about ~500 elements and
a contains about ~1 million elements
Please note that Step 3 is a nested loop over both r and a. Hence, it takes a lot of time. In the below code r and a are shortened for simplicity.
I am also mentioning this function, some_heavy_calculation(), for background purposes. This function is not revealed here, but since it also is called len(r) * len(a) times, it also consumes a lot of time.
In an effort to speed things up, I have noticed that I can avoid 90-95 % of all calls to some_heavy_calculation() by introducing the "faster" alternative. The only problem is that Step 3 now takes a lot of time. In fact, this step consumes more time than I am able to save.
def some_heavy_calculation(rules, data) -> list:
# ...
return []
# r = input rules
r = ['x', 'y', 'z']
# a = input data
a = [7, 7, 7, 4, 4, 2, 2, 8, 2, 9, 4, 4, 8, 7 ]
#########
# Slow alternative: b = result of some_heavy_calculation(r, a)
# b = expected result, size: [ r x a ]
b = [[True, True, True, True, True, True, True, True, True, False, True, True, True, True],
[True, True, True, False, False, True, True, True, True, False, False, False, True, True],
[False, False, False, True, True, False, False, True, False, False, True, True, True, False]]
#########
#########
# Faster:
# Since these steps avoids 90-95 % of all the calls to some_heavy_calculation()
#
# Step 1: c = a in order, but without duplicates
c = [7, 4, 2, 8, 9 ]
# Step 2: d = result of calculation, size: [ r x c ]
d = [[True, True, True, True, False ],
[True, False, True, True, False ],
[False, True, False, True, False ]]
# Step 3: e = should equal b
e = [[d[ri][next(ci for ci, cv in enumerate(c) if cv == av)] for ai, av in enumerate(a)] for ri, rv in enumerate(r)]
#########
str(b) == str(e) # <--- returns True

It seems to me that what you need is a pattern called memoization.
There is a functools.cache decorator (for Python < 3.9 you can use lru_cache) that you can use this way:
import functools
#functools.cache
def some_heavy_calculation_per_item(rules, value) -> bool:
# ...
return []
def some_heavy_calculation(rules, data) -> list:
# ...
returned = []
for value in data:
returned.append(some_heavy_calculation_per_item(rules, value))
return returned
Using memoization is effectively doing the calculations once per each value (the savings of 90-95% as you noted) but also in a memory efficient way (no need to combine many large lists or arrays).
Another potential optimization is to use yield instead of constructing the list in some_heavy_calculation function but this depends on the way you consume the result - if value by value then yielding will improve performance. If you need the list in its entirety - then it won't help at all.

How can I shorten this 2D array code?

I have the following snippet which works fine.
P=im1.copy()
for i in range(P.shape[0]):
for j in range(P.shape[1]):
if (n_dens.data[i][j]==-5000 or T_k.data[i][j]==-5000):
P.data[i][j]=-5000
else :
P.data[i][j]=n_dens.data[i][j]*T_k.data[i][j]
where P is a 2D array.
I was wondering how to trim this down to something along the following lines:
P.data=n_dens.data*T_k.data
P.data=[foo-2.5005*10**7 if n_dens.data==-5000 or T_k.data==-5000 else foo for foo in P.data]
For my trial above I get the following error:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
How can I correct the error? Or is there another method to trim it down?

The n_dens.data==-5000 produces an array of true/false values, not a single value. So, the if can't handle it. You are close to the idea though. You can use logical indexing in numpy.
Also logical operators cannot be overloaded in python. So, numpy does not handle them as you would wish. So, you have to do something like
index = np.logical_or(n_dens.data ==-5000, T_k.data==-5000)
P.data[index] = -5000
Similarly, P.data[np.logical_not(index)] = n_dens.data * T.data for the second branch of if-else.

You can try this:
P.data[(n_dens.data == -5000) | (T_k.data == -5000)] = -5000
cond = ~(n_dens.data == -5000) & ~(T_k.data == -5000) # 2D array of booleans
P.data[cond] = n_dens.data[cond] * T_k.data[cond]
A complete example:
import numpy as np
from copy import deepcopy
class IMAGE:
def __init__(self, data):
self.data = data
self.shape = self.data.shape
np.random.seed(0)
P, n_dens, T_k = IMAGE(np.zeros((5,5))), IMAGE(np.reshape(np.random.choice([-5000,1,2],25), (5,5))), IMAGE(3*np.ones((5,5)))
P1 = deepcopy(P)
# with loop
for i in range(P.shape[0]):
for j in range(P.shape[1]):
if (n_dens.data[i][j]==-5000 or T_k.data[i][j]==-5000):
P.data[i][j]=-5000
else :
P.data[i][j]=n_dens.data[i][j]*T_k.data[i][j]
# vectorized
P1.data[(n_dens.data == -5000) | (T_k.data == -5000)] = -5000
cond = ~(n_dens.data == -5000) & ~(T_k.data == -5000) # 2D array of booleans
P1.data[cond] = n_dens.data[cond] * T_k.data[cond]
cond
# array([[False, True, False, True, True],
# [ True, False, True, False, False],
# [False, True, True, True, True],
# [False, True, True, True, True],
# [False, True, False, False, True]], dtype=bool)
# with same output for both
P.data == P1.data
# array([[ True, True, True, True, True],
# [ True, True, True, True, True],
# [ True, True, True, True, True],
# [ True, True, True, True, True],
# [ True, True, True, True, True]], dtype=bool)

Python: How to pass subarrays of array into array function

The ultimate goal of my question is that I want to generate a new array 'output' by passing the subarrays of an array into a function, where the return of the function for each subarray generates a new element into 'output'.
My input array was generated as follows:
aggregate_input = np.random.rand(100, 5)
input = np.split(aggregate_predictors, 1, axis=1)[0]
So now input appears as follows:
print(input[0:2])
>>[[ 0.61521025 0.07407679 0.92888063 0.66066605 0.95023826]
>> [ 0.0666379 0.20007622 0.84123138 0.94585421 0.81627862]]
Next, I want to pass each element of input (so the array of 5 floats) through my function 'condition' and I want the return of each function call to fill in a new array 'output'. Basically, I want 'output' to contain 100 values.
def condition(array):
return array[4] < 0.5
How do I pass each element of input into condition without using any nasty loops?
========
Basically, I want to do this, but optimized:
lister = []
for i in range(100):
lister.append(condition(input[i]))
output = np.array(lister)

That initial split and index does nothing. It just wraps the array in list, and then takes out again:
In [76]: x=np.random.rand(100,5)
In [77]: y = np.split(x,1,axis=1)
In [78]: len(y)
Out[78]: 1
In [79]: y[0].shape
Out[79]: (100, 5)
The rest just tests if the 4th element of each row is <.5:
In [81]: def condition(array):
...:
...: return array[4] < 0.5
...:
In [82]: lister = []
...:
...: for i in range(100):
...: lister.append(condition(x[i]))
...:
...: output = np.array(lister)
...:
In [83]: output
Out[83]:
array([ True, False, False, True, False, True, True, False, False,
True, False, True, False, False, True, False, False, True,
False, True, False, True, False, False, False, True, False,
...], dtype=bool)
We can do just as easily with column indexing
In [84]: x[:,4]<.5
Out[84]:
array([ True, False, False, True, False, True, True, False, False,
True, False, True, False, False, True, False, False, True,
False, True, False, True, False, False, False, True, False,
...], dtype=bool)
In other words, operate on the whole 4th column of the array.

You are trying to make a very simple indexing expression very convoluted. If you read the docs for np.split very carefully, you will see that passing a second argument of 1 does absolutely nothing: it splits the array into one chunk. The following line is literally a no-op and should be removed:
input = np.split(aggregate_predictors, 1, axis=1)[0]
You have a 2D numpy array of shape 100, 5 (you can check that with aggregate_predictors.shape). Your function returns whether or not the fifth column contains a value less than 0.5. You can do this with a single vectorized expression:
output = aggregate_predictors[:, 4] < 0.5
If you want to find the last column instead of the fifth, use index -1 instead:
output = aggregate_predictors[:, -1] < 0.5
The important thing to remember here is that all the comparison operators are vectorized element-wise in numpy. Usually, vectorizing an operation like this involves finding the correct index in the array. You should never have to convert anything to a list: numpy arrays are iterable as it is, and there are more complex iterators available.
That being said, your original intent was probably to do something like
input = split(aggregate_predictors, len(aggregate_predictors), axis=0)
OR
input = split(aggregate_predictors, aggregate_predictors.shape[0])
Both expressions are equivalent. They split aggregate_predictors into a list of 100 single-row matrices.

Fill scipy / numpy matrix based on indices and values

I have a graph of nodes which each represent about 100 voxels in the brain. I partitioned the graph into communities, but now I need to make a correlation matrix where every voxel in a node is connected to every voxel in the nodes that are in the same community. In other words, if nodes 1 and 2 are in the same community, I need a 1 in the matrix between every voxel in node 1 and every voxel in node 2. This takes a very long time with the code below. Does anyone know how to speed this up?
for edge in combinations(graph.nodes(),2):
if partition.get_node_community(edge[0]) == partition.get_node_community(edge[1]): # if nodes are in same community
voxels1 = np.argwhere(flat_parcel==edge[0]+1) # this is where I find the voxels in each node, and I get the indices for the matrix where I want them.
voxels2 = np.argwhere(flat_parcel==edge[1]+1)
for voxel1 in voxels1:
voxel_matrix[voxel1,voxels2] = 1
Thanks for the responses, I think the easiest and fastest solution is to replace the last loop with
voxel_matrix[np.ix_(voxels1, voxels2)] = 1

Here's an approach that I expect to work for you. It's a stretch on my machine -- even storing two copies of the voxel adjacency matrix (using dtype=bool) pushes my (somewhat old) desktop right to the edge of its memory capacity. But I'm assuming that you have a machine capable of handling at least two (300 * 100) ** 2 = 900 MB arrays -- otherwise, you would probably have run into problems before this stage. It takes my desktop about 30 minutes to process 30000 voxels.
This assumes that voxel_communities is a simple array containing a community label for each voxel at index i. It sounds like you can generate that pretty quickly. It also assumes that voxels are present in only one node.
def voxel_adjacency(voxel_communities):
n_voxels = voxel_communities.size
comm_labels = sorted(set(voxel_communities))
comm_counts = [(voxel_communities == l).sum() for l in comm_labels]
blocks = numpy.zeros((n_voxels, n_voxels), dtype=bool)
start = 0
for c in comm_counts:
blocks[start:start + c, start:start + c] = 1
start += c
ix = numpy.empty_like(voxel_communities)
ix[voxel_communities.argsort()] = numpy.arange(n_voxels)
blocks[:] = blocks[ix,:]
blocks[:] = blocks[:,ix]
return blocks
Here's a quick explanation. This uses an inverse indexing trick to reorder the columns and rows of an array of diagonal blocks into the desired matrix.
n_voxels = voxel_communities.size
comm_labels = sorted(set(voxel_communities))
comm_counts = [(voxel_communities == l).sum() for l in comm_labels]
blocks = numpy.zeros((n_voxels, n_voxels), dtype=bool)
start = 0
for c in comm_counts:
blocks[start:start + c, start:start + c] = 1
start += c
These lines are used to construct the initial block matrix. So for example, say you have six voxels and three communities, and each community contains two voxels. Then the initial block matrix will look like this:
array([[ True, True, False, False, False, False],
[ True, True, False, False, False, False],
[False, False, True, True, False, False],
[False, False, True, True, False, False],
[False, False, False, False, True, True],
[False, False, False, False, True, True]], dtype=bool)
This is essentially the same as the desired adjacency matrix after the voxels have been sorted by community membership. So we need to reverse that sorting. We do so by constructing an inverse argsort array.
ix = numpy.empty_like(voxel_communities)
ix[voxel_communities.argsort()] = numpy.arange(n_voxels)
Now ix will reverse the sorting process when used as an index. And since this is a symmetric matrix, we can perform the reverse sorting operation separately on columns and then on rows:
blocks[:] = blocks[ix,:]
blocks[:] = blocks[:,ix]
return blocks
Here's an example of the result it generates for a small input:
>>> voxel_adjacency(numpy.array([0, 3, 1, 1, 0, 2]))
array([[ True, False, False, False, True, False],
[False, True, False, False, False, False],
[False, False, True, True, False, False],
[False, False, True, True, False, False],
[ True, False, False, False, True, False],
[False, False, False, False, False, True]], dtype=bool)
It seems to me that this does something quite similar to voxel_matrix[np.ix_(voxels1, voxels2)] = 1 as suggested by pv., except it does it all at once, instead of tracking each possible combination of nodes.
There may be a better solution, but this should at least be an improvement.
Also, note that if you can simply accept the new ordering of voxels as canonical, then this solution becomes as simple as creating the block array! That takes all of about 300 milliseconds.