We Keep Coding

How to compute the distance between 3d points in a fast way

I have a 3d point cloud (x,y,z) in a txt file. I want to calculate the 3d distance between each point and all the other points in the point cloud, and save the number of points having distance less than a threshold. I have done it in python in the shown code but it takes too much time. I was asking for a faster one than the one I got.
from math import sqrt
import numpy as np
points_list = []
with open("D:/Point cloud data/projection_test_data3.txt") as chp:
for line in chp:
x, y, z = line.split()
points_list.append((float(x), float(y), float(z)))
j = 0
Final_density = 0
while j < len(points_list)-1:
i = 0
Density = 0
while i < len (points_list) - 1 :
if sqrt((points_list[i][0] - points_list[j][0])**2 + (points_list[i][1] - points_list[j][1])**2 + (points_list[i][2] - points_list[j][2])**2) < 0.15:
Density += 1
i += 1
Final_density = Density
with open("D:/Point cloud data/my_density.txt", 'a') as g:
g.write("{}\n".format(str(Final_density)))
j += 1

One (quick) option that might speed this up is to change the position of the file writing/opening so that you're not opening/closing the file as many times.
from math import sqrt
import numpy as np
points_list = []
with open("D:/Point cloud data/projection_test_data3.txt") as chp:
for line in chp:
x, y, z = line.split()
points_list.append((float(x), float(y), float(z)))
j = 0
Final_density = 0
with open("D:/Point cloud data/my_density.txt", 'a') as g:
while j < len(points_list)-1:
i = 0
Density = 0
while i < len (points_list) - 1 :
if sqrt((points_list[i][0] - points_list[j][0])**2 + (points_list[i][1] - points_list[j][1])**2 + (points_list[i][2] - points_list[j][2])**2) < 0.15:
Density += 1
i += 1
Final_density = Density
g.write("{}\n".format(str(Final_density)))
j += 1
Since it looks like you can use numpy, why not use it? You'll have to make sure the arrays are numpy arrays, but that should be simple.
if sqrt(...) < 0.15: to if np.linalg.norm(points_list[j] - points_list[i]) < 0.15:
This post (Finding 3d distances using an inbuilt function in python) has other ways to use a prebuilt function to get the 3d distance in python.
Edit thanks to #KellyBundy's comment:
You can also use np.linalg.norm(points_list - points_list[:, None], axis=1) to generate a matrix representing the distance between all points in the array. The diagonal will be 0 (corresponding to the distance between the given point and itself) and the matrix will be symmetric about the diagonal. You can use just the upper triangle to determine the distance between any given pair of points. Again, you'll have to modify your data structure to make all the points into a numpy array in the proper format (np.array([[point1x, point1y, point1z], [point2x, point2y, point2z]]), etc. (https://stackoverflow.com/a/46700369/2391458)
resulting matrix of the form:
[ [0 distance between points 0 and 1 distance between points 0 and 2......]
[distance between points 1 and 0 0 distance between points 2 and 0....]
.....

Quick x2 speed up – replace i = 0 with i = j + 1. That way you would check each pair only once, not twice.
More fundamental change – you can sort points by coordinates, and use sliding window algorithm. The idea is that if points are sorted by x coordinate, j-th point has x=1, and i-th point has x=1.01, then it might be near each other and you should check it. But if i-th point has x=2, then it cannot be near to j-th point, and since points are sorted, all points after i-th can be skipped (i.e. not checked in pair with j-th point).
If points are sparse, then it should significantly speed up function, and complexity would be O(n*log(n)) because of sorting.

In the if, instead of taking the sqrt and comparing it with 0.15, compare it with square of 0.15 which is 0.0225 directly. The result will be the same. sqrt is an expensive operation, it will save you time to not use it.
if (points_list[i][0] - points_list[j][0])**2 + (points_list[i][1] - points_list[j][1])**2 + (points_list[i][2] - points_list[j][2])**2 < 0.0225:

Particle-particle collision in box with periodic boundaries

The Task
For a class in molecular dynamics, I have to simulate 100 particles in a box with periodic boundaries. I have to take particle-particle collisions into account, since the walls are 'transparent' those interactions can be happen across the boundaries. Since the simulation should cover 50000 steps, and I'm expecting more and more additional tasks in the future, I want my code as efficient as possible (I have to use python, despite the long run time).
The Setting
The system consists of
100 Particles
Box with x = 10, y = 5
Mass = 2
Radius = 0.2
Velocity |v| = 0.5 per step
Simulation of 50000 steps
What I've done so fare
I have found this example for particles in a box with particle-particle collision. Since the author used a efficient implementation, I took his approach.
My relevant code parts are (in strong resemblance to the linked site):
class particlesInPeriodicBox:
# define the particle properties
def __init__(self,
initialState = [[0,0,0,0]], # state with form [x, y, vx, vy] for each particle
boundaries = [0, 10, 0, 5], # box boundaries with form [xmin, xmax, ymin, ymax] in nm
radius = 0.2, # particle radius in nm
mass = 2): # mass in g/mol. Here a parameter instead of a global variable
self.initialState = np.asarray(initialState, dtype=float)
self.state = self.initialState.copy()
self.radius = radius
self.time = 0 # keep count of time, if time, i want to implement a 'clock'
self.mass = mass # mass
self.boundaries = boundaries
def collision(self):
"""
now, one has to check for collisions. I do this by distance check and will solve this problems below.
To minimize the simulation time I then will only consider the particles that colided.
"""
dist = squareform(pdist(self.state[:, :2])) # direct distance
colPart1, colPart2 = np.where(dist < 2 * self.radius) # define collision partners 1 and 2 as those where the distance between the centeres are smaller than two times the radius
# resolve self-self collissions
unique = (colPart1 < colPart2)
colPart1 = colPart1[unique]
colPart2 = colPart2[unique]
"""
The following loop resolves the collisions. I zip the lists of collisionpartners to one aray,
where one entry contains both colisionpartners.
"""
for cp1, cp2 in zip(colPart1, colPart2): # cp1/cp2 are the two particles colliding in one collision.
# masses could be different in future...
m1 = self.mass[cp1]
m2 = self.mass[cp2]
# take the position (x,y) tuples for the two particles
r1 = self.state[cp1, :2]
r2 = self.state[cp2, :2]
# same with velocities
v1 = self.state[cp1, 2:]
v2 = self.state[cp2, 2:]
# get relative parameters
r = r1-r2
v = r2-r1
# center of mass velocity:
vcm = (m1 * v1 + m2 * v2) / (m1 + m2)
"""
This is the part with the elastic collision
"""
dotrr = np.dot(r, r) # the dot product of the relative position with itself
dotvr = np.dot(v, r) # the dot product of the relative velocity with the relative position
v = 2 * r * dotvr / dotrr - v # new relative velocity
"""
In this center of mass frame, the velocities 'reflect' on the center of mass in oposite directions
"""
self.state[cp1, 2:] = vcm + v * m2/(m1 + m2) # new velocity of particle 1 still considering possible different masses
self.state[cp2, 2:] = vcm - v * m1/(m1 + m2) # new velocity of particle 2
As I understand it, this technique of handling the operations to the whole arrays is more efficient than manually looping through it every time. Moving the particles 'trough' the wall is easy, I just subtract or add the dimension of the box, respectively. But:
The Problem
For now the algorithm only sees collision inside the box, but not across the boundaries. I thought about this problem a while now, and come up with the following Ideas:
I could make a total of 9 copies of this system in a 3x3 grid, and only considering the middle one, can so look into the neighboring cells for the nearest neighbor search. BUT i can't think of a effective way to implement this despite the fact, that this approach seams to be the standard way
Every other idea has some hand waving use of modulo, and im almost sure, that this is not the way to go.
If I had to boil it down, I guess my key questions are:
How do I take periodic boundaries into account when calculating
the distance between particles?
the actual particle-particle collision (elastic) and resulting directions?
For the first problem it might be possible to use techniques like in Calculation of Contact/Coordination number with Periodic Boundary Conditions, but Im not sure if that is the most efficient way.
Thank you!

Modulus is likely as quick an operation as you're going to get. In any self-respecting run-time system, this will attach directly to the on-chip floating-divide operations, which may well be faster than a tedious set of "if-subtract" pairs.
I think your 9-cell solution is overkill. Use 4 cells in a 2x2 matrix and check two regions: the original cell, and the same dimensions centered on the "four corners" point (middle of the 2x2). For any pair of points, the proper distance is the lesser of these two. Note that this method also gives you a frame in which you can easily calculate the momentum changes.
A third possible approach is to double the dimensions (ala the 2x2 above), but give each particle four sets of coordinates, one in each box. Alter your algorithms to consider all four when computing distance. If you have good vectorization packages and parallelism, this might be the preferred solution.

Fastest way to determine average of gridded values within a given radius of a point

I have a numpy 2D array of values. Each element in the array represents a grid point from a grid where each box is 13km on a side. I need to determine the average value of all points within 50 miles of a specific point on the grid.
My current solution determines a bounding box and then references items in the array within that box using their indices, which is slow with numpy. I'm trying to determine a faster solution.
Current solution:
num_x = 400 #horizontal dimension of the 2D array
num_y = 300 #vertical dimension of the 2D array
num_dx = 6 #maximum number of horizontal grid points that fit within 50 miles
num_dy = 6 #same as above but for vertical (square grid)
radius_m = 80467.2 #50 miles expressed in meters
values = [] # stores the extracted values
for ix in range(-num_dx,num_dx+1):
for jy in range(-num_dy,num_dy+1):
# Determine distance to this point
dist = ((ix*dx)**2+(jy*dy)**2)**0.5
if dist <= radius_m:
# Ensure this grid point actually exists within the grid
if (j+jy) < num_y and (i+ix) < num_x:
value = myarray[i+ix,j+jy]
if value is not masked and value >= 0:
values.append(float(value))
average = sum(values) / float(len(values))
This is slow (takes about 1.5 seconds) due to accessing myarray over 100 times to extract the value of a single element. Is there a vector method that would work better here? I can't seem to figure out a way to do this with a mask since the conditional is based on the location of the grid point relative to another, not the value of the element itself.

Your code isn't runable and seems to contain a bug for when i < num_dx or j < num_dy (then it wraps around to the other side of the array). But making some assumptions on your variable names, this is how I would do it:
# First make sure we stay in the grid
i1, i2 = max(i-num_dx, 0), min(i+num_dx+1, num_x)
j1, j2 = max(j-num_dy, 0), min(j+num_dy+1, num_y)
# Get the radius in blocks, grid should be homogeneous
radius_i = radius_m / 13000.0
# Calc distances per element by broadcasting
DX = np.arange(i1, i2) - i
DY = np.arange(j1, j2)[:, None] - j
mask = DX*DX + DY*DY <= radius_i*radius_i
# Get block of interest and apply mask
values = myarray[i1:i2, j1:j2][mask]

For interior points (where the radius doesn't extend outside your image), you can just compute a single mask that is used for any interior point. Start with an array of zeros:
mask = np.zeros((2 * num_dx + 1, 2 * num_dy + 1), dtype=np.int)
Assuming your point of interest is at the center of that array, set each element that falls within the radius to 1 (not shown here). Then,
indices = np.argwhere(mask.ravel() == 1)
Then for any interior element (i, j) in myarray, you would get the values within the radius like:
values = myarray[i-num_dx: i+num_dx+1, j-num_dy: j+num_dy+1].ravel()[indices]
For points near the border, you would make a copy of mask and set rows/cols outside the image to zero before setting indices.

Monte Carlo Simulation with Python: building a histogram on the fly

I have a conceptual question on building a histogram on the fly with Python. I am trying to figure out if there is a good algorithm or maybe an existing package.
I wrote a function, which runs a Monte Carlo simulation, gets called 1,000,000,000 times, and returns a 64 bit floating number at the end of each run. Below is the said function:
def MonteCarlo(df,head,span):
# Pick initial truck
rnd_truck = np.random.randint(0,len(df))
full_length = df['length'][rnd_truck]
full_weight = df['gvw'][rnd_truck]
# Loop using other random trucks until the bridge is full
while True:
rnd_truck = np.random.randint(0,len(df))
full_length += head + df['length'][rnd_truck]
if full_length > span:
break
else:
full_weight += df['gvw'][rnd_truck]
# Return average weight per feet on the bridge
return(full_weight/span)
df is a Pandas dataframe object having columns labeled as 'length' and 'gvw', which are truck lengths and weights, respectively. head is the distance between two consecutive trucks, span is the bridge length. The function randomly places trucks on the bridge as long as the total length of the truck train is less than the bridge length. Finally, calculates the average weight of the trucks existing on the bridge per foot (total weight existing on the bridge divided by the bridge length).
As a result I would like to build a tabular histogram showing the distribution of the returned values, which can be plotted later. I had some ideas in mind:
Keep collecting the returned values in a numpy vector, then use existing histogram functions once the MonteCarlo analysis is completed. This would not be feasable, since if my calculation is correct, I would need 7.5 GB of memory for that vector only (1,000,000,000 64 bit floats ~ 7.5 GB)
Initialize a numpy array with a given range and number of bins. Increase the number of items in the matching bin by one at the end of each run. The problem is, I do not know the range of values I would get. Setting up a histogram with a range and an appropriate bin size is an unknown. I also have to figure out how to assign values to the correct bins, but I think it is doable.
Do it somehow on the fly. Modify ranges and bin sizes each time the function returns a number. This would be too tricky to write from scratch I think.
Well, I bet there may be a better way to handle this problem. Any ideas would be welcome!
On a second note, I tested running the above function for 1,000,000,000 times only to get the largest value that is computed (the code snippet is below). And this takes around an hour when span = 200. The computation time would increase if I run it for longer spans (the while loop runs longer to fill the bridge with trucks). Is there a way to optimize this you think?
max_w = 0
i = 1
while i < 1000000000:
if max_w < MonteCarlo(df_basic, 15., 200.):
max_w = MonteCarlo(df_basic, 15., 200.)
i += 1
print max_w
Thanks!

Here is a possible solution, with fixed bin size, and bins of the form [k * size, (k + 1) * size[. The function finalizebins returns two lists: one with bin counts (a), and the other (b) with bin lower bounds (the upper bound is deduced by adding binsize).
import math, random
def updatebins(bins, binsize, x):
i = math.floor(x / binsize)
if i in bins:
bins[i] += 1
else:
bins[i] = 1
def finalizebins(bins, binsize):
imin = min(bins.keys())
imax = max(bins.keys())
a = [0] * (imax - imin + 1)
b = [binsize * k for k in range(imin, imax + 1)]
for i in range(imin, imax + 1):
if i in bins:
a[i - imin] = bins[i]
return a, b
# A test with a mixture of gaussian distributions
def check(n):
bins = {}
binsize = 5.0
for i in range(n):
if random.random() > 0.5:
x = random.gauss(100, 50)
else:
x = random.gauss(-200, 150)
updatebins(bins, binsize, x)
return finalizebins(bins, binsize)
a, b = check(10000)
# This must be 10000
sum(a)
# Plot the data
from matplotlib.pyplot import *
bar(b,a)
show()

Generate random points on a surface of the cylinder

I want to generate random points on the surface of cylinder such that distance between the points fall in a range of 230 and 250. I used the following code to generate random points on surface of cylinder:
import random,math
H=300
R=20
s=random.random()
#theta = random.random()*2*math.pi
for i in range(0,300):
theta = random.random()*2*math.pi
z = random.random()*H
r=math.sqrt(s)*R
x=r*math.cos(theta)
y=r*math.sin(theta)
z=z
print 'C' , x,y,z
How can I generate random points such that they fall with in the range(on the surfaceof cylinder)?

This is not a complete solution, but an insight that should help. If you "unroll" the surface of the cylinder into a rectangle of width w=2*pi*r and height h, the task of finding distance between points is simplified. You have not explained how to measure "distance along the surface" between points on the top of the cylinder and the side- this is a slightly tricky bit of geometry.
As for computing the distance along the surface when we created an artificial "seam", just use both (x1-x2) and (w -x1+x2) - whichever gives the shorter distance is the one you want.
I do think that #VincentNivoliers' suggestion to use Poisson disk sampling is very good, but with the constraints of h=300 and r=20 you will get terrible results no matter what.

The basic way of creating a set of random points with constraints in the positions between them, is to have a function that modulates the probability of points being placed at a certain location. this function starts out being a constant, and whenever a point is placed, forbidden areas surrounding the point are set to zero. That is difficult to do with continuous variables, but reasonably easy if you discretize your problem.
The other thing to be careful about is the being on a cylinder part. It may be easier to think of it as random points on a rectangular area that repeats periodically. This can be handled in two different ways:
the simplest is to take into consideration not only the rectangular tile where you are placing the points, but also its neighbouring ones. Whenever you place a point in your main tile, you also place one in the neighboring ones and compute their effect on the probability function inside your tile.
A more sophisticated approach considers the probability function then convolution of a kernel that encodes forbidden areas, with a sum of delta functions, corresponding to the points already placed. If this is computed using FFTs, the periodicity is anatural by product.
The first approach can be coded as follows:
from __future__ import division
import numpy as np
r, h = 20, 300
w = 2*np.pi*r
int_w = int(np.rint(w))
mult = 10
pdf = np.ones((h*mult, int_w*mult), np.bool)
points = []
min_d, max_d = 230, 250
available_locs = pdf.sum()
while available_locs:
new_idx = np.random.randint(available_locs)
new_idx = np.nonzero(pdf.ravel())[0][new_idx]
new_point = np.array(np.unravel_index(new_idx, pdf.shape))
points += [new_point]
min_mask = np.ones_like(pdf)
if max_d is not None:
max_mask = np.zeros_like(pdf)
else:
max_mask = True
for p in [new_point - [0, int_w*mult], new_point +[0, int_w*mult],
new_point]:
rows = ((np.arange(pdf.shape[0]) - p[0]) / mult)**2
cols = ((np.arange(pdf.shape[1]) - p[1]) * 2*np.pi*r/int_w/mult)**2
dist2 = rows[:, None] + cols[None, :]
min_mask &= dist2 > min_d*min_d
if max_d is not None:
max_mask |= dist2 < max_d*max_d
pdf &= min_mask & max_mask
available_locs = pdf.sum()
points = np.array(points) / [mult, mult*int_w/(2*np.pi*r)]
If you run it with your values, the output is usually just one or two points, as the large minimum distance forbids all others. but if you run it with more reasonable values, e.g.
min_d, max_d = 50, 200
Here's how the probability function looks after placing each of the first 5 points:
Note that the points are returned as pairs of coordinates, the first being the height, the second the distance along the cylinder's circumference.