We Keep Coding

faster way to erode/dilate images

I'm making a script thats does some mathemagical morphology on images (mainly gis rasters). Now, I've implemented erosion and dilation, with opening/closing with reconstruction still on the TODO but thats not the subject here.
My implementation is very simple with nested loops, which I tried on a 10900x10900 raster and it took an absurdly long amount of time to finish, obviously.
Before I continue with other operations, I'd like to know if theres a faster way to do this?
My implementation:
def erode(image, S):
(m, n) = image.shape
buffer = np.full((m, n), 0).astype(np.float64)
for i in range(S, m - S):
for j in range(S, n - S):
buffer[i, j] = np.min(image[i - S: i + S + 1, j - S: j + S + 1]) #dilation is just np.max()
return buffer
I've heard about vectorization but I'm not quite sure I understand it too well. Any advice or pointers are appreciated. Also I am aware that opencv has these morphological operations, but I want to implement my own to learn about them.

The question here is do you want a more efficient implementation because you want to learn about numpy or do you want a more efficient algorithm.
I think there are two obvious things that could be improved with your approach. One is you want to avoid looping on the python level because that is slow. The other is that your taking a maximum of overlapping parts of arrays and you can make it more efficient if you reuse all the effort you put in finding the last maximum.
I will illustrate that with 1d implementations of erosion.
Baseline for comparison
Here is basically your implementation just a 1d version:
def erode(image, S):
n = image.shape[0]
buffer = np.full(n, 0).astype(np.float64)
for i in range(S, n - S):
buffer[i] = np.min(image[i - S: i + S + 1]) #dilation is just np.max()
return buffer
You can make this faster using stride_tricks/sliding_window_view. I.e. by avoiding the loops and doing that at the numpy level.
Faster Implementation
np.lib.stride_tricks.sliding_window_view(arr,2*S+1).min(1)
Notice that it's not quite doing the same since it only starts calculating values once there are 2S+1 values to take the maximum of. But for this illustration I will ignore this problem.
Faster Algorithm
A completely different approach would be to not start calculating the min from scratch but keeping the values ordered and only adding one and removing one when considering the next window one to the right.
Here is a ruff implementation of that:
def smart_erode(arr, m):
n = arr.shape[0]
sd = SortedDict()
for new in arr[:m]:
if new in sd:
sd[new] += 1
else:
sd[new] = 1
for to_remove,new in zip(arr[:-m+1],arr[m:]):
yield sd.keys()[0]
if new in sd:
sd[new] += 1
else:
sd[new] = 1
if sd[to_remove] > 1:
sd[to_remove] -= 1
else:
sd.pop(to_remove)
yield sd.keys()[0]
Notice that an ordered set wouldn't work and an ordered list would have to have a way to remove just one element with a specific value sind you could have repeated values in your array. I am using an ordered dict to store the amount of items present for a value.
A Ruff Benchmark
I want to illustrate how the 3 implementations compare for different window sizes. So I am testing them with an array of 10^5 random integers for different window sizes ranging from 10^3 to 10^4.
arr = np.random.randint(0,10**5,10**5)
sliding_window_times = []
op_times = []
better_alg_times = []
for m in np.linspace(0,10**4,11)[1:].astype('int'):
x = %timeit -o -n 1 -r 1 np.lib.stride_tricks.sliding_window_view(arr,2*m+1).min(1)
sliding_window_times.append(x.best)
x = %timeit -o -n 1 -r 1 erode(arr,m)
op_times.append(x.best)
x = %timeit -o -n 1 -r 1 tuple(smart_erode(arr,2*m+1))
better_alg_times.append(x.best)
print("")
pd.DataFrame({"Baseline Comparison":op_times,
'Faster Implementation':sliding_window_times,
'Faster Algorithm':better_alg_times,
},
index = np.linspace(0,10**4,11)[1:].astype('int')
).plot.bar()
Notice that for very small window sizes the raw power of the numpy implementation wins out but very quickly the amount of work we are saving by not calculating the min from scratch is more important.

Analyzing the complexity matrix path-finding

Recently in my homework, I was assinged to solve the following problem:
Given a matrix of order nxn of zeros and ones, find the number of paths from [0,0] to [n-1,n-1] that go only through zeros (they are not necessarily disjoint) where you could only walk down or to the right, never up or left. Return a matrix of the same order where the [i,j] entry is the number of paths in the original matrix that go through [i,j], the solution has to be recursive.
My solution in python:
def find_zero_paths(M):
n,m = len(M),len(M[0])
dict = {}
for i in range(n):
for j in range(m):
M_top,M_bot = blocks(M,i,j)
X,Y = find_num_paths(M_top),find_num_paths(M_bot)
dict[(i,j)] = X*Y
L = [[dict[(i,j)] for j in range(m)] for i in range(n)]
return L[0][0],L
def blocks(M,k,l):
n,m = len(M),len(M[0])
assert k<n and l<m
M_top = [[M[i][j] for i in range(k+1)] for j in range(l+1)]
M_bot = [[M[i][j] for i in range(k,n)] for j in range(l,m)]
return [M_top,M_bot]
def find_num_paths(M):
dict = {(1, 1): 1}
X = find_num_mem(M, dict)
return X
def find_num_mem(M,dict):
n, m = len(M), len(M[0])
if M[n-1][m-1] != 0:
return 0
elif (n,m) in dict:
return dict[(n,m)]
elif n == 1 and m > 1:
new_M = [M[0][:m-1]]
X = find_num_mem(new_M,dict)
dict[(n,m-1)] = X
return X
elif m == 1 and n>1:
new_M = M[:n-1]
X = find_num_mem(new_M, dict)
dict[(n-1,m)] = X
return X
new_M1 = M[:n-1]
new_M2 = [M[i][:m-1] for i in range(n)]
X,Y = find_num_mem(new_M1, dict),find_num_mem(new_M2, dict)
dict[(n-1,m)],dict[(n,m-1)] = X,Y
return X+Y
My code is based on the idea that the number of paths that go through [i,j] in the original matrix is equal to the product of the number of paths from [0,0] to [i,j] and the number of paths from [i,j] to [n-1,n-1]. Another idea is that the number of paths from [0,0] to [i,j] is the sum of the number of paths from [0,0] to [i-1,j] and from [0,0] to [i,j-1]. Hence I decided to use a dictionary whose keys are matricies of the form [[M[i][j] for j in range(k)] for i in range(l)] or [[M[i][j] for j in range(k+1,n)] for i in range(l+1,n)] for some 0<=k,l<=n-1 where M is the original matrix and whose values are the number of paths from the top of the matrix to the bottom. After analizing the complexity of my code I arrived at the conclusion that it is O(n^6).
Now, my instructor said this code is exponential (for find_zero_paths), however, I disagree.
The recursion tree (for find_num_paths) size is bounded by the number of submatrices of the form above which is O(n^2). Also, each time we add a new matrix to the dictionary we do it in polynomial time (only slicing lists), SO... the total complexity is polynomial (poly*poly = poly). Also, the function 'blocks' runs in polynomial time, and hence 'find_zero_paths' runs in polynomial time (2 lists of polynomial-size times a function which runs in polynomial time) so all in all the code runs in polynomial time.
My question: Is the code polynomial and my O(n^6) bound is wrong or is it exponential and I am missing something?

Unfortunately, your instructor is right.
There is a lot to unpack here:
Before we start, as quick note. Please don't use dict as a variable name. It hurts ^^. Dict is a reserved keyword for a dictionary constructor in python. It is a bad practice to overwrite it with your variable.
First, your approach of counting M_top * M_bottom is good, if you were to compute only one cell in the matrix. In the way you go about it, you are unnecessarily computing some blocks over and over again - that is why I pondered about the recursion, I would use dynamic programming for this one. Once from the start to end, once from end to start, then I would go and compute the products and be done with it. No need for O(n^6) of separate computations. Sine you have to use recursion, I would recommend caching the partial results and reusing them wherever possible.
Second, the root of the issue and the cause of your invisible-ish exponent. It is hidden in the find_num_mem function. Say you compute the last element in the matrix - the result[N][N] field and let us consider the simplest case, where the matrix is full of zeroes so every possible path exists.
In the first step, your recursion creates branches [N][N-1] and [N-1][N].
In the second step, [N-1][N-1], [N][N-2], [N-2][N], [N-1][N-1]
In the third step, you once again create two branches from every previous step - a beautiful example of an exponential explosion.
Now how to go about it: You will quickly notice that some of the branches are being duplicated over and over. Cache the results.

Efficient Particle-Pair Interactions Calculation

I have an N-body simulation that generates a list of particle positions, for multiple timesteps in the simulation. For a given frame, I want to generate a list of the pairs of particles' indices (i, j) such that dist(p[i], p[j]) < masking_radius. Essentially I'm creating a list of "interaction" pairs, where the pairs are within a certain distance of each other. My current implementation looks something like this:
interaction_pairs = []
# going through each unique pair (order doesn't matter)
for i in range(num_particles):
for j in range(i + 1, num_particles):
if dist(p[i], p[j]) < masking_radius:
interaction_pairs.append((i,j))
Because of the large number of particles, this process takes a long time (>1 hr per test), and it is severely limiting to what I need to do with the data. I was wondering if there was any more efficient way to structure the data such that calculating these pairs would be more efficient instead of comparing every possible combination of particles. I was looking into KDTrees, but I couldn't figure out a way to utilize them to compute this more efficiently. Any help is appreciated, thank you!

Since you are using python, sklearn has multiple implementations for nearest neighbours finding:
http://scikit-learn.org/stable/modules/neighbors.html
There is KDTree and Balltree provided.
As for KDTree the main point is to push all the particles you have into KDTree, and then for each particle ask query: "give me all particles in range X". KDtree usually do this faster than bruteforce search.
You can read more for example here: https://www.cs.cmu.edu/~ckingsf/bioinfo-lectures/kdtrees.pdf
If you are using 2D or 3D space, then other option is to just cut the space into big grid (which cell size of masking radius) and assign each particle into one grid cell. Then you can find possible candidates for interaction just by checking neighboring cells (but you also have to do a distance check, but for much fewer particle pairs).

Here's a fairly simple technique using plain Python that can reduce the number of comparisons required.
We first sort the points along either the X, Y, or Z axis (selected by axis in the code below). Let's say we choose the X axis. Then we loop over point pairs like your code does, but when we find a pair whose distance is greater than the masking_radius we test whether the difference in their X coordinates is also greater than the masking_radius. If it is, then we can bail out of the inner j loop because all points with a greater j have a greater X coordinate.
My dist2 function calculates the squared distance. This is faster than calculating the actual distance because computing the square root is relatively slow.
I've also included code that behaves similar to your code, i.e., it tests every pair of points, for speed comparison purposes; it also serves to check that the fast code is correct. ;)
from random import seed, uniform
from operator import itemgetter
seed(42)
# Make some fake data
def make_point(hi=10.0):
return [uniform(-hi, hi) for _ in range(3)]
psize = 1000
points = [make_point() for _ in range(psize)]
masking_radius = 4.0
masking_radius2 = masking_radius ** 2
def dist2(p, q):
return (p[0] - q[0])**2 + (p[1] - q[1])**2 + (p[2] - q[2])**2
pair_count = 0
test_count = 0
do_fast = 1
if do_fast:
# Sort the points on one axis
axis = 0
points.sort(key=itemgetter(axis))
# Fast
for i, p in enumerate(points):
left, right = i - 1, i + 1
for j in range(i + 1, psize):
test_count += 1
q = points[j]
if dist2(p, q) < masking_radius2:
#interaction_pairs.append((i, j))
pair_count += 1
elif q[axis] - p[axis] >= masking_radius:
break
if i % 100 == 0:
print('\r {:3} '.format(i), flush=True, end='')
total_pairs = psize * (psize - 1) // 2
print('\r {} / {} tests'.format(test_count, total_pairs))
else:
# Slow
for i, p in enumerate(points):
for j in range(i+1, psize):
q = points[j]
if dist2(p, q) < masking_radius2:
#interaction_pairs.append((i, j))
pair_count += 1
if i % 100 == 0:
print('\r {:3} '.format(i), flush=True, end='')
print('\n', pair_count, 'pairs')
output with do_fast = 1
181937 / 499500 tests
13295 pairs
output with do_fast = 0
13295 pairs
Of course, if most of the point pairs are within masking_radius of each other, there won't be much benefit in using this technique. And sorting the points adds a little bit of time, but Python's TimSort is rather efficient, especially if the data is already partially sorted, so if the masking_radius is sufficiently small you should see a noticeable improvement in the speed.

I need some help to optimize a python code

I'm working on a KNN Classifier using Python but I have some problems.
The following piece of code takes 7.5s-9.0s to be completed and i'll have to run it for 60.000 times.
for fold in folds:
for dot2 in fold:
"""
distances[x][0] = Class of the dot2
distances[x][1] = distance between dot1 and dot2
"""
distances.append([dot2[0], calc_distance(dot1[1:], dot2[1:], method)])
The "folds" variable is a list with 10 folds that summed contain 60.000 inputs of images in the .csv format. The first value of each dot is the class it belongs to. All the values are in integer.
Is there a way to make this line run any faster ?
Here it is the calc_distance function
def calc_distancia(dot1, dot2, distance):
if distance == "manhanttan":
total = 0
#for each coord, take the absolute difference
for x in range(0, len(dot1)):
total = total + abs(dot1[x] - dot2[x])
return total
elif distance == "euclidiana":
total = 0
for x in range(0, len(dot1)):
total = total + (dot1[x] - dot2[x])**2
return math.sqrt(total)
elif distance == "supremum":
total = 0
for x in range(0, len(dot1)):
if abs(dot1[x] - dot2[x]) > total:
total = abs(dot1[x] - dot2[x])
return total
elif distance == "cosseno":
dist = 0
p1_p2_mul = 0
p1_sum = 0
p2_sum = 0
for x in range(0, len(dot1)):
p1_p2_mul = p1_p2_mul + dot1[x]*dot2[x]
p1_sum = p1_sum + dot1[x]**2
p2_sum = p2_sum + dot2[x]**2
p1_sum = math.sqrt(p1_sum)
p2_sum = math.sqrt(p2_sum)
quociente = p1_sum*p2_sum
dist = p1_p2_mul/quociente
return dist
EDIT:
Found a way to make it faster at least for the "manhanttan" method. Instead of:
if distance == "manhanttan":
total = 0
#for each coord, take the absolute difference
for x in range(0, len(dot1)):
total = total + abs(dot1[x] - dot2[x])
return total
i put
if distance == "manhanttan":
totalp1 = 0
totalp2 = 0
#for each coord, take the absolute difference
for x in range(0, len(dot1)):
totalp1 += dot1[x]
totalp2 += dot2[x]
return abs(totalp1-totalp2)
The abs() call is very heavy

There are many guides to "profiling python"; you should search for some, read them, and walk through the profiling process to ensure you know what parts of your work are taking the most time.
But if this is really the core of your work, it's a fair bet that that calc_distance is where the majority of the running time is being consumed.
Optimizing that deeply will probably require using NumPy accelerated math or a similar, lower-level approach.
As a quick and dirty approach requiring less invasive profiling and rewriting, try installing the PyPy implementation of Python and running under it. I have seen easy 2x or more accelerations compared to the standard (CPython) implementation.

I'm confused. Did you try the profiler?
python -m cProfile myscript.py
It will show you where the bulk of the time is being consumed and provide hard data to work with. eg. refactor to reduce the number of calls, restructure the input data, substitute this function for that, etc.
https://docs.python.org/3/library/profile.html

In the first place, you should avoid using a single calc_distance function that performs a linear search in a list of strings on every call. Define independent distance functions and call the right one. As Lee Daniel Crocker suggested, don't use the slicing, just start your loop ranges at 1.
For the cosine distance, I would recommend to normalize all the dot vectors once for all. This way the distance computation reduces to a dot product.
These micro-optimization can give you some speedup. But a better gain should be possible by switching to a better algorithm: the kNN classifier calls for a kD-tree, that will allow you to quickly remove a significant fraction of the points from consideration.
This is harder to implement (you'll have to slightly adapt for the different distances; the cosine distance will make it tricky.)

k-means clustering implementation in python, running out of memory

 Note: updates/solutions at the bottom of this question
As part of a product recommendation engine, I'm trying to segment my users based on their product preferences starting with using the k-means clustering algorithm.
My data is a dictionary of the form:
prefs = {
'user_id_1': { 1L: 3.0f, 2L: 1.0f, },
'user_id_2': { 4L: 1.0f, 8L: 1.5f, },
}
where the product ids are the longs, and ratings are floats. the data is sparse. I currently have about 60,000 users, most of whom have only rated a handful of products. The dictionary of values for each user is implemented using a defaultdict(float) to simplify the code.
My implementation of k-means clustering is as follows:
def kcluster(prefs,sim_func=pearson,k=100,max_iterations=100):
from collections import defaultdict
users = prefs.keys()
centroids = [prefs[random.choice(users)] for i in range(k)]
lastmatches = None
for t in range(max_iterations):
print 'Iteration %d' % t
bestmatches = [[] for i in range(k)]
# Find which centroid is closest for each row
for j in users:
row = prefs[j]
bestmatch=(0,0)
for i in range(k):
d = simple_pearson(row,centroids[i])
if d < bestmatch[1]: bestmatch = (i,d)
bestmatches[bestmatch[0]].append(j)
# If the results are the same as last time, this is complete
if bestmatches == lastmatches: break
lastmatches=bestmatches
centroids = [defaultdict(float) for i in range(k)]
# Move the centroids to the average of their members
for i in range(k):
len_best = len(bestmatches[i])
if len_best > 0:
items = set.union(*[set(prefs[u].keys()) for u in bestmatches[i]])
for user_id in bestmatches[i]:
row = prefs[user_id]
for m in items:
if row[m] > 0.0: centroids[i][m]+=(row[m]/len_best)
return bestmatches
As far as I can tell, the algorithm is handling the first part (assigning each user to its nearest centroid) fine.
The problem is when doing the next part, taking the average rating for each product in each cluster and using these average ratings as the centroids for the next pass.
Basically, before it's even managed to do the calculations for the first cluster (i=0), the algorithm bombs out with a MemoryError at this line:
if row[m] > 0.0: centroids[i][m]+=(row[m]/len_best)
Originally the division step was in a seperate loop, but fared no better.
This is the exception I get:
malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Any help would be greatly appreciated.
Update: Final algorithms
Thanks to the help recieved here, this is my fixed algorithm. If you spot anything blatantly wrong please add a comment.
First, the simple_pearson implementation
def simple_pearson(v1,v2):
si = [val for val in v1 if val in v2]
n = len(si)
if n==0: return 0.0
sum1 = 0.0
sum2 = 0.0
sum1_sq = 0.0
sum2_sq = 0.0
p_sum = 0.0
for v in si:
sum1+=v1[v]
sum2+=v2[v]
sum1_sq+=pow(v1[v],2)
sum2_sq+=pow(v2[v],2)
p_sum+=(v1[v]*v2[v])
# Calculate Pearson score
num = p_sum-(sum1*sum2/n)
temp = (sum1_sq-pow(sum1,2)/n) * (sum2_sq-pow(sum2,2)/n)
if temp < 0.0:
temp = -temp
den = sqrt(temp)
if den==0: return 1.0
r = num/den
return r
A simple method to turn simple_pearson into a distance value:
def distance(v1,v2):
return 1.0-simple_pearson(v1,v2)
And finally, the k-means clustering implementation:
def kcluster(prefs,k=21,max_iterations=50):
from collections import defaultdict
users = prefs.keys()
centroids = [prefs[u] for u in random.sample(users, k)]
lastmatches = None
for t in range(max_iterations):
print 'Iteration %d' % t
bestmatches = [[] for i in range(k)]
# Find which centroid is closest for each row
for j in users:
row = prefs[j]
bestmatch=(0,2.0)
for i in range(k):
d = distance(row,centroids[i])
if d <= bestmatch[1]: bestmatch = (i,d)
bestmatches[bestmatch[0]].append(j)
# If the results are the same as last time, this is complete
if bestmatches == lastmatches: break
lastmatches=bestmatches
centroids = [defaultdict(float) for i in range(k)]
# Move the centroids to the average of their members
for i in range(k):
len_best = len(bestmatches[i])
if len_best > 0:
for user_id in bestmatches[i]:
row = prefs[user_id]
for m in row:
centroids[i][m]+=row[m]
for key in centroids[i].keys():
centroids[i][key]/=len_best
# We may have made the centroids quite dense which significantly
# slows down subsequent iterations, so we delete values below a
# threshold to speed things up
if centroids[i][key] < 0.001:
del centroids[i][key]
return centroids, bestmatches

Not all these observations are directly relevant to your issues as expressed, but..:
a. why are the key in prefs, as shown, longs? unless you have billions of users, simple ints will be fine and save you a little memory.
b. your code:
centroids = [prefs[random.choice(users)] for i in range(k)]
can give you repeats (two identical centroids), which in turn would not make the K-means algorithm happy. Just use the faster and more solid
centroids = [prefs[u] for random.sample(users, k)]
c. in your code as posted you're calling a function simple_pearson which you never define anywhere; I assume you mean to call sim_func, but it's really hard to help on different issues while at the same time having to guess how the code you posted differs from any code that might actually be working
d. one more indication that this posted code may be different from anything that might actually work: you set bestmatch=(0,0) but then test with if d < bestmatch[1]: -- how is the test ever going to succeed? is the distance function returning negative values?
e. the point of a defaultdict is that just accessing row[m] magically adds an item to row at index m (with the value obtained by calling the defaultdict's factory, here 0.0). That item will then take up memory forevermore. You absolutely DON'T need this behavior, and therefore your code:
row = prefs[user_id]
for m in items:
if row[m] > 0.0: centroids[i][m]+=(row[m]/len_best)
is wasting huge amount of memory, making prefs into a dense matrix (mostly full of 0.0 values) from the sparse one it used to be. If you code instead
row = prefs[user_id]
for m in row:
centroids[i][m]+=(row[m]/len_best)
there will be no growth in row and therefore in prefs because you're looping over the keys that row already has.
There may be many other such issues, major like the last one or minor ones -- as an example of the latter,
f. don't divide a bazillion times by len_best: compute its inverse one outside the loop and multiply by that inverse -- also you don't need to do that multiplication inside the loop, you can do it at the end in a separate since it's the same value that's multiplying every item -- this saves no memory but avoids wantonly wasting CPU time;-). OK, these are two minor issues, I guess, not just one;-).
As I mentioned there may be many others, but with the density of issues already shown by these six (or seven), plus the separate suggestion already advanced by S.Lott (which I think would not fix your main out-of-memory problem, since his code still addressing the row defaultdict by too many keys it doesn't contain), I think it wouldn't be very productive to keep looking for even more -- maybe start by fixing these ones and if problems persist post a separate question about those...?

Your centroids does not need to be an actual list.
You never appear to reference anything other than centroids[i][m]. If you only want centroids[i], then perhaps it doesn't need to be a list; a simple dictionary would probably do.
centroids = defaultdict(float)
# Move the centroids to the average of their members
for i in range(k):
len_best = len(bestmatches[i])
if len_best > 0:
items = set.union(*[set(prefs[u].keys()) for u in bestmatches[i]])
for user_id in bestmatches[i]:
row = prefs[user_id]
for m in items:
if row[m] > 0.0: centroids[m]+=(row[m]/len_best)
May work better.