We Keep Coding

Electric force between particles using numpy arrays

I am trying to simulate a particle flying at another particle while undergoing electrical repulsion (or attraction), called Rutherford-scattering. I have succeeded in simulating (a few) particles using for loops and python lists. However, now I want to use numpy arrays instead. The model will use the following steps:
For all particles:
Calculate radial distance with all other particles
Calculate the angle with all other particles
Calculate netto force in x-direction and y-direction
Create matrix with netto xForce and yForce for each particle
Create accelaration (also x and y component) matrix by a = F/mass
Update speed matrix
Update position matrix
My problem is that I do not know how I can use numpy arrays in calculating the force components.
Here follows my code which is not runnable.
import numpy as np
# I used this function to calculate the force while using for-loops.
def force(x1, y1, x2, x2):
angle = math.atan((y2 - y1)/(x2 - x1))
dr = ((x1-x2)**2 + (y1-y2)**2)**0.5
force = charge2 * charge2 / dr**2
xforce = math.cos(angle) * force
yforce = math.sin(angle) * force
# The direction of force depends on relative location
if x1 > x2 and y1<y2:
xforce = xforce
yforce = yforce
elif x1< x2 and y1< y2:
xforce = -1 * xforce
yforce = -1 * yforce
elif x1 > x2 and y1 > y2:
xforce = xforce
yforce = yforce
else:
xforce = -1 * xforce
yforce = -1* yforce
return xforce, yforce
def update(array):
# this for loop defeats the entire use of numpy arrays
for particle in range(len(array[0])):
# find distance of all particles pov from 1 particle
# find all x-forces and y-forces on that particle
xforce = # sum of all x-forces from all particles
yforce = # sum of all y-forces from all particles
force_arr[0, particle] = xforce
force_arr[1, particle] = yforce
return force
# begin parameters
t = 0
N = 3
masses = np.ones(N)
charges = np.ones(N)
loc_arr = np.random.rand(2, N)
speed_arr = np.random.rand(2, N)
acc_arr = np.random.rand(2, N)
force = np.random.rand(2, N)
while t < 0.5:
force_arr = update(loc_arry)
acc_arr = force_arr / masses
speed_arr += acc_array
loc_arr += speed_arr
t += dt
# plot animation

One approach to model this problem with arrays may be:
define the point coordinates as a Nx2 array. (This will help with extensibility if you advance to 3-D points later)
define the intermediate variables distance, angle, force as NxN arrays to represent the pairwise interactions
Numpy things to know about:
You can call most numeric functions on arrays if the arrays have the same shape (or conforming shapes, which is a nontrivial topic...)
meshgrid helps you generate the array indices necessary to shapeshift your Nx2 arrays to compute NxN results
and a tangential note (ha ha) arctan2() computes a signed angle, so you can bypass the complex "which quadrant" logic
For example you can do something like this. Note in get_dist and get_angle the arithmetic operations between points take place in the bottom-most dimension:
import numpy as np
# 2-D locations of particles
points = np.array([[1,0],[2,1],[2,2]])
N = len(points) # 3
def get_dist(p1, p2):
r = p2 - p1
return np.sqrt(np.sum(r*r, axis=2))
def get_angle(p1, p2):
r = p2 - p1
return np.arctan2(r[:,:,1], r[:,:,0])
ii = np.arange(N)
ix, iy = np.meshgrid(ii, ii)
dist = get_dist(points[ix], points[iy])
angle = get_angle(points[ix], points[iy])
# ... compute force
# ... apply the force, etc.
For the sample 3-point vector shown above:
In [246]: dist
Out[246]:
array([[0. , 1.41421356, 2.23606798],
[1.41421356, 0. , 1. ],
[2.23606798, 1. , 0. ]])
In [247]: angle / np.pi # divide by Pi to make the numbers recognizable
Out[247]:
array([[ 0. , -0.75 , -0.64758362],
[ 0.25 , 0. , -0.5 ],
[ 0.35241638, 0.5 , 0. ]])

Here is one go with only a loop for each time step, and it should work for any number of dimensions, I have tested with 3 too:
from matplotlib import pyplot as plt
import numpy as np
fig, ax = plt.subplots()
N = 4
ndim = 2
masses = np.ones(N)
charges = np.array([-1, 1, -1, 1]) * 2
# loc_arr = np.random.rand(N, ndim)
loc_arr = np.array(((-1,0), (1,0), (0,-1), (0,1)), dtype=float)
speed_arr = np.zeros((N, ndim))
# compute charge matrix, ie c1 * c2
charge_matrix = -1 * np.outer(charges, charges)
time = np.linspace(0, 0.5)
dt = np.ediff1d(time).mean()
for i, t in enumerate(time):
# get (dx, dy) for every point
delta = (loc_arr.T[..., np.newaxis] - loc_arr.T[:, np.newaxis]).T
# calculate Euclidean distance
distances = np.linalg.norm(delta, axis=-1)
# and normalised unit vector
unit_vector = (delta.T / distances).T
unit_vector[np.isnan(unit_vector)] = 0 # replace NaN values with 0
# calculate force
force = charge_matrix / distances**2 # norm gives length of delta vector
force[np.isinf(force)] = 0 # NaN forces are 0
# calculate acceleration in all dimensions
acc = (unit_vector.T * force / masses).T.sum(axis=1)
# v = a * dt
speed_arr += acc * dt
# increment position, xyz = v * dt
loc_arr += speed_arr * dt
# plotting
if not i:
color = 'k'
zorder = 3
ms = 3
for i, pt in enumerate(loc_arr):
ax.text(*pt + 0.1, s='{}q {}m'.format(charges[i], masses[i]))
elif i == len(time)-1:
color = 'b'
zroder = 3
ms = 3
else:
color = 'r'
zorder = 1
ms = 1
ax.plot(loc_arr[:,0], loc_arr[:,1], '.', color=color, ms=ms, zorder=zorder)
ax.set_aspect('equal')
The above example produces, where the black and blue points signify the start and end positions, respectively:
And when charges are equal charges = np.ones(N) * 2 the system symmetry is preserved and the charges repel:
And finally with some random initial velocities speed_arr = np.random.rand(N, 2):
EDIT
Made a small change to the code above to make sure it was correct. (I was missing -1 on the resultant force, ie. force between +/+ should be negative, and I was summing down the wrong axis, apologies for that. Now in the cases where masses[0] = 5, the system evolves correctly:

The classic approach is to calculate electric field for all particles in the system. Say you have 3 charged particles all with positive charge:
particles = np.array([[1,0,0],[2,1,0],[2,2,0]]) # location of each particle
q = np.array([1,1,1]) # charge of each particle
The easiest way to compute the electric field at each particle`s location is for loop:
def for_method(pos,q):
"""Computes electric field vectors for all particles using for-loop."""
Evect = np.zeros( (len(pos),len(pos[0])) ) # define output electric field vector
k = 1 / (4 * np.pi * const.epsilon_0) * np.ones((len(pos),len(pos[0]))) * 1.602e-19 # make this into matrix as matrix addition is faster
# alternatively you can get rid of np.ones and just define this as a number
for i, v0 in enumerate(pos): # s_p - selected particle | iterate over all particles | v0 reference particle
for v, qc in zip(pos,q): # loop over all particles and calculate electric force sum | v particle being calculated for
if all((v0 == v)): # do not compute for the same particle
continue
else:
r = v0 - v #
Evect[i] += r / np.linalg.norm(r) ** 3 * qc #! multiply by charge
return Evect * k
# to find electric field at each particle`s location call
for_method(particles, q)
This function returns array of vectors with the same shape as input particles array. To find force on each, you simply multiply this vector with q array of charges. From there on, you can easily find your acceleration and integrate the system using your favourite ODE solver.
Performance Optimization & Accuracy
For method is the slowest possible approach. The field can be computed using solely linear algebra granting significant speed boost. Following code is very efficient Numpy matrix "one-liner" (almost one-liner) to this problem:
def CPU_matrix_method(pos,q):
"""Classic vectorization of for Coulomb law using numpy arrays."""
k = 1 / (4 * np.pi * const.epsilon_0) * np.ones((len(pos),3)) * 1.602e-19 # define electric constant
dist = distance.cdist(pos,pos) # compute distances
return k * np.sum( (( np.tile(pos,len(pos)).reshape((len(pos),len(pos),3)) - np.tile(pos,(len(pos),1,1))) * q.reshape(len(q),1)).T * np.power(dist,-3, where = dist != 0),axis = 1).T
Note that this and following code also return electric field vector for each particle.
You can get even higher performance if you offload this onto the GPU using Cupy library. Following code is almost identical to the CPU_matrix_method, I only expanded the one-liner a little so that you could see better what is going on:
def GPU_matrix_method(pos,q):
"""GPU Coulomb law vectorization.
Takes in numpy arrays, performs computations and returns cupy array"""
# compute distance matrix between each particle
k_cp = 1 / (4 * cp.pi * const.epsilon_0) * cp.ones((len(pos),3)) * 1.602e-19 # define electric constant, runs faster if this is matrix
dist = cp.array(distance.cdist(pos,pos)) # could speed this up with cupy cdist function! use this: cupyx.scipy.spatial.distance.cdist
pos, q = cp.array(pos), cp.array(q) # load inputs to GPU memory
dist_mod = cp.power(dist,-3) # compute inverse cube of distance
dist_mod[dist_mod == cp.inf] = 0 # set all infinity entries to 0 (i.e. diagonal elements/ same particle-particle pairs)
# compute by magic
return k_cp * cp.sum((( cp.tile(pos,len(pos)).reshape((len(pos),len(pos),3)) - cp.tile(pos,(len(pos),1,1))) * q.reshape(len(q),1)).T * dist_mod, axis = 1).T
Regarding the accuracy of the mentioned algorithms, if you compute the 3 methods on the particles array you get identical results:
[[-6.37828367e-10 -7.66608512e-10 0.00000000e+00]
[ 5.09048221e-10 -9.30757576e-10 0.00000000e+00]
[ 1.28780145e-10 1.69736609e-09 0.00000000e+00]]
Regarding the performance, I computed each algorithm on systems ranging from 2 to 5000 charged particles. Additionally I also included Numba precompiled version of the for_method to make the for-loop approach competitive:
We see that for-loop performs terribly needing over 400 seconds to compute for system with 5000 particles. Zooming in to the bottom part:
This shows that matrix approach to this problem is orders of magnitude better. To be exact the 5000 particle evaluation took 18.5s for Numba for-loop, 4s for CPU matrix(5 times faster than Numba), and 0.8s for GPU matrix* (23 times faster than Numba). The significant difference shows for larger arrays.
* GPU used was Nvidia K100.

Solving heat equation with python (NumPy)

I solve the heat equation for a metal rod as one end is kept at 100 °C and the other at 0 °C as
import numpy as np
import matplotlib.pyplot as plt
dt = 0.0005
dy = 0.0005
k = 10**(-4)
y_max = 0.04
t_max = 1
T0 = 100
def FTCS(dt,dy,t_max,y_max,k,T0):
s = k*dt/dy**2
y = np.arange(0,y_max+dy,dy)
t = np.arange(0,t_max+dt,dt)
r = len(t)
c = len(y)
T = np.zeros([r,c])
T[:,0] = T0
for n in range(0,r-1):
for j in range(1,c-1):
T[n+1,j] = T[n,j] + s*(T[n,j-1] - 2*T[n,j] + T[n,j+1])
return y,T,r,s
y,T,r,s = FTCS(dt,dy,t_max,y_max,k,T0)
plot_times = np.arange(0.01,1.0,0.01)
for t in plot_times:
plt.plot(y,T[t/dt,:])
If changing the Neumann boundary condition as one end is insulated (not flux),
then, how the calculating term
T[n+1,j] = T[n,j] + s*(T[n,j-1] - 2*T[n,j] + T[n,j+1])
should be modified?

A typical approach to Neumann boundary condition is to imagine a "ghost point" one step beyond the domain, and calculate the value for it using the boundary condition; then proceed normally (using the PDE) for the points that are inside the grid, including the Neumann boundary.
The ghost point allows us to use the symmetric finite difference approximation to the derivative at the boundary, that is (T[n, j+1] - T[n, j-1]) / (2*dy) if y is the space variable. Non-symmetric approximation (T[n, j] - T[n, j-1]) / dy, which does not involve a ghost point, is much less accurate: the error it introduces is an order of magnitude worse than the error involved in the discretization of the PDE itself.
So, when j is the maximal possible index for T, the boundary condition says that "T[n, j+1]" should be understood as T[n, j-1] and this is what is done below.
for j in range(1, c-1):
T[n+1,j] = T[n,j] + s*(T[n,j-1] - 2*T[n,j] + T[n,j+1]) # as before
j = c-1
T[n+1, j] = T[n,j] + s*(T[n,j-1] - 2*T[n,j] + T[n,j-1]) # note the last term here

Wave on a string analysis

I am trying to analyse a wave on a string by solving the wave equation with Python. Here are my requirements for the solution.
1) I model reflective ends by using much larger masses on first and last point on the string -> Large inertia
2)No spring on edges. Then k[0] and k[-1] will be ZERO.
I have problem with indices. In my 2nd loop I get y[i,j-1], k[i-1], y[i-1,j]. In the first iteration the loop will then use y[0,-1], k[-1], y[-1,0]. These are my last points and not my first points. How can I avoid this problem?

What you need is initiating your mass array with one additional element. I mean
...
m = [mi]*N # mass array [!!!] instead of (N-1) [!!!]
...
Idem for your springs
...
k = [ki]*N
...
Consequently, you can keep k[0] equal to 10. since you model reflective ends. You may thus want to comment or drop this line
...
##k[0] = 0
...
For aesthetic considerations you may want to fill the gap at the end of the x-axis. In this case, simply do
N = 201 # Number of mass points
Your code thus becomes
from numpy import *
from matplotlib.pyplot import *
# Variables
N = 201 # Number of mass points
nT = 1200 # Number of time points
mi = 0.02 # mass in kg
m = [mi]*N # mass array
m[-1] = 100 # Large last mass reflective edges
m[0] = 100 # Large first mass reflective edges
ki = 10.#spring
k = [ki]*N
k[-1] = 0
dx = 0.2
kappa = ki*dx
my = mi/dx
c = sqrt(kappa/my) # velocity
dt = 0.04
# 3 vectors
x = arange( N )*dx # x points
t = arange( N )*dt # t points
y = zeros( [N, nT ] )# 2D array
# Loop over initial condition
for i in range(N-1):
y[i,0] = sin((7.*pi*i)/(N-1)) # Initial condition dependent on mass point
# Iterating over time and position to find next position of wave
for j in range(nT-1):
for i in range(N-1):
y[i,j+1] = 2*y[i,j] - y[i,j-1] + (dt**2/m[i])*(k[i-1]*y[i+1,j] -2*k[i-1]*y[i,j] + k[i]*y[i-1,j] )
#check values of edges
print y[:2,j+1],y[-2:,j+1]
# Creates an animation
cla()
ylabel("Amplitude")
xlabel("x")
ylim(-10,10)
plot(x,y[:,j-2])
pause(0.001)
close()
which produces
Following your comment, I think that if you want to see the wave traveling along the string before reflection, you should not initiate conditions everywhere (in space). I mean, e.g., doing
...
# Loop over initial condition
for i in range(N-1):
ci_i = sin(7.*pi*i/(N-1)) # Initial condition dependent on mass point
if np.sign(ci_i*y[i-1,0])<0:
break
else:
y[i,0] = ci_i
...
produces
New attempt after answers:
from numpy import *
from matplotlib.pyplot import *
N = 201
nT = 1200
mi = 0.02
m = [mi]*(N)
m[-1] = 1000
m[0] = 1000
ki = 10.
k = [ki]*N
dx = 0.2
kappa = ki*dx
my = mi/dx
c = sqrt(kappa/my)
dt = 0.04
x = arange( N )*dx
t = arange( N )*dt
y = zeros( [N, nT ] )
for i in range(N-1):
y[i,0] = sin((7.*pi*i)/(N-1))
for j in range(nT-1):
for i in range(N-1):
if j == 0: # if j = 0 then ... y[i,j-1]=y[i,j]
y[i,j+1] = 2*y[i,j] - y[i,j] + (dt**2/m[i])*(k[i-1]*y[i+1,j] -2*k[i-1]*y[i,j] + k[i]*y[i-1,j] )
else:
y[i,j+1] = 2*y[i,j] - y[i,j-1] + (dt**2/m[i])*( k[i-1]*y[i+1,j] -2*k[i-1]*y[i,j] + k[i]*y[i-1,j] )
cla()
ylim(-1,1)
plot(x,y[:,j-2])
pause(0.0001)
ylabel("Amplitude")
xlabel("x")
print len(x), len(t), N,nT
Here is a plot of the new attempt at solution with |amplitude| of anti node equal 1.0. Will this do anything with further solving the issue with indices?

How to create an array that can be accessed according to its indices in Numpy?

I am trying to solve the following problem via a Finite Difference Approximation in Python using NumPy:
$u_t = k \, u_{xx}$, on $0 < x < L$ and $t > 0$;
$u(0,t) = u(L,t) = 0$;
$u(x,0) = f(x)$.
I take $u(x,0) = f(x) = x^2$ for my problem.
Programming is not my forte so I need help with the implementation of my code. Here is my code (I'm sorry it is a bit messy, but not too bad I hope):
## This program is to implement a Finite Difference method approximation
## to solve the Heat Equation, u_t = k * u_xx,
## in 1D w/out sources & on a finite interval 0 < x < L. The PDE
## is subject to B.C: u(0,t) = u(L,t) = 0,
## and the I.C: u(x,0) = f(x).
import numpy as np
import matplotlib.pyplot as plt
# definition of initial condition function
def f(x):
return x^2
# parameters
L = 1
T = 10
N = 10
M = 100
s = 0.25
# uniform mesh
x_init = 0
x_end = L
dx = float(x_end - x_init) / N
#x = np.zeros(N+1)
x = np.arange(x_init, x_end, dx)
x[0] = x_init
# time discretization
t_init = 0
t_end = T
dt = float(t_end - t_init) / M
#t = np.zeros(M+1)
t = np.arange(t_init, t_end, dt)
t[0] = t_init
# Boundary Conditions
for m in xrange(0, M):
t[m] = m * dt
# Initial Conditions
for j in xrange(0, N):
x[j] = j * dx
# definition of solution to u_t = k * u_xx
u = np.zeros((N+1, M+1)) # NxM array to store values of the solution
# finite difference scheme
for j in xrange(0, N-1):
u[j][0] = x**2 #initial condition
for m in xrange(0, M):
for j in xrange(1, N-1):
if j == 1:
u[j-1][m] = 0 # Boundary condition
else:
u[j][m+1] = u[j][m] + s * ( u[j+1][m] - #FDM scheme
2 * u[j][m] + u[j-1][m] )
else:
if j == N-1:
u[j+1][m] = 0 # Boundary Condition
print u, t, x
#plt.plot(t, u)
#plt.show()
So the first issue I am having is I am trying to create an array/matrix to store values for the solution. I wanted it to be an NxM matrix, but in my code I made the matrix (N+1)x(M+1) because I kept getting an error that the index was going out of bounds. Anyways how can I make such a matrix using numpy.array so as not to needlessly take up memory by creating a (N+1)x(M+1) matrix filled with zeros?
Second, how can I "access" such an array? The real solution u(x,t) is approximated by u(x[j], t[m]) were j is the jth spatial value, and m is the mth time value. The finite difference scheme is given by:
u(x[j],t[m+1]) = u(x[j],t[m]) + s * ( u(x[j+1],t[m]) - 2 * u(x[j],t[m]) + u(x[j-1],t[m]) )
(See here for the formulation)
I want to be able to implement the Initial Condition u(x[j],t[0]) = x**2 for all values of j = 0,...,N-1. I also need to implement Boundary Conditions u(x[0],t[m]) = 0 = u(x[N],t[m]) for all values of t = 0,...,M. Is the nested loop I created the best way to do this? Originally I tried implementing the I.C. and B.C. under two different for loops which I used to calculate values of the matrices x and t (in my code I still have comments placed where I tried to do this)
I think I am just not using the right notation but I cannot find anywhere in the documentation for NumPy how to "call" such an array so at to iterate through each value in the proposed scheme. Can anyone shed some light on what I am doing wrong?
Any help is very greatly appreciated. This is not homework but rather to understand how to program FDM for Heat Equation because later I will use similar methods to solve the Black-Scholes PDE.
EDIT: So when I run my code on line 60 (the last "else" that I use) I get an error that says invalid syntax, and on line 51 (u[j][0] = x**2 #initial condition) I get an error that reads "setting an array element with a sequence." What does that mean?

Efficient method of calculating density of irregularly spaced points

I am attempting to generate map overlay images that would assist in identifying hot-spots, that is areas on the map that have high density of data points. None of the approaches that I've tried are fast enough for my needs.
Note: I forgot to mention that the algorithm should work well under both low and high zoom scenarios (or low and high data point density).
I looked through numpy, pyplot and scipy libraries, and the closest I could find was numpy.histogram2d. As you can see in the image below, the histogram2d output is rather crude. (Each image includes points overlaying the heatmap for better understanding)
My second attempt was to iterate over all the data points, and then calculate the hot-spot value as a function of distance. This produced a better looking image, however it is too slow to use in my application. Since it's O(n), it works ok with 100 points, but blows out when I use my actual dataset of 30000 points.
My final attempt was to store the data in an KDTree, and use the nearest 5 points to calculate the hot-spot value. This algorithm is O(1), so much faster with large dataset. It's still not fast enough, it takes about 20 seconds to generate a 256x256 bitmap, and I would like this to happen in around 1 second time.
Edit
The boxsum smoothing solution provided by 6502 works well at all zoom levels and is much faster than my original methods.
The gaussian filter solution suggested by Luke and Neil G is the fastest.
You can see all four approaches below, using 1000 data points in total, at 3x zoom there are around 60 points visible.
Complete code that generates my original 3 attempts, the boxsum smoothing solution provided by 6502 and gaussian filter suggested by Luke (improved to handle edges better and allow zooming in) is here:
import matplotlib
import numpy as np
from matplotlib.mlab import griddata
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import math
from scipy.spatial import KDTree
import time
import scipy.ndimage as ndi
def grid_density_kdtree(xl, yl, xi, yi, dfactor):
zz = np.empty([len(xi),len(yi)], dtype=np.uint8)
zipped = zip(xl, yl)
kdtree = KDTree(zipped)
for xci in range(0, len(xi)):
xc = xi[xci]
for yci in range(0, len(yi)):
yc = yi[yci]
density = 0.
retvalset = kdtree.query((xc,yc), k=5)
for dist in retvalset[0]:
density = density + math.exp(-dfactor * pow(dist, 2)) / 5
zz[yci][xci] = min(density, 1.0) * 255
return zz
def grid_density(xl, yl, xi, yi):
ximin, ximax = min(xi), max(xi)
yimin, yimax = min(yi), max(yi)
xxi,yyi = np.meshgrid(xi,yi)
#zz = np.empty_like(xxi)
zz = np.empty([len(xi),len(yi)])
for xci in range(0, len(xi)):
xc = xi[xci]
for yci in range(0, len(yi)):
yc = yi[yci]
density = 0.
for i in range(0,len(xl)):
xd = math.fabs(xl[i] - xc)
yd = math.fabs(yl[i] - yc)
if xd < 1 and yd < 1:
dist = math.sqrt(math.pow(xd, 2) + math.pow(yd, 2))
density = density + math.exp(-5.0 * pow(dist, 2))
zz[yci][xci] = density
return zz
def boxsum(img, w, h, r):
st = [0] * (w+1) * (h+1)
for x in xrange(w):
st[x+1] = st[x] + img[x]
for y in xrange(h):
st[(y+1)*(w+1)] = st[y*(w+1)] + img[y*w]
for x in xrange(w):
st[(y+1)*(w+1)+(x+1)] = st[(y+1)*(w+1)+x] + st[y*(w+1)+(x+1)] - st[y*(w+1)+x] + img[y*w+x]
for y in xrange(h):
y0 = max(0, y - r)
y1 = min(h, y + r + 1)
for x in xrange(w):
x0 = max(0, x - r)
x1 = min(w, x + r + 1)
img[y*w+x] = st[y0*(w+1)+x0] + st[y1*(w+1)+x1] - st[y1*(w+1)+x0] - st[y0*(w+1)+x1]
def grid_density_boxsum(x0, y0, x1, y1, w, h, data):
kx = (w - 1) / (x1 - x0)
ky = (h - 1) / (y1 - y0)
r = 15
border = r * 2
imgw = (w + 2 * border)
imgh = (h + 2 * border)
img = [0] * (imgw * imgh)
for x, y in data:
ix = int((x - x0) * kx) + border
iy = int((y - y0) * ky) + border
if 0 <= ix < imgw and 0 <= iy < imgh:
img[iy * imgw + ix] += 1
for p in xrange(4):
boxsum(img, imgw, imgh, r)
a = np.array(img).reshape(imgh,imgw)
b = a[border:(border+h),border:(border+w)]
return b
def grid_density_gaussian_filter(x0, y0, x1, y1, w, h, data):
kx = (w - 1) / (x1 - x0)
ky = (h - 1) / (y1 - y0)
r = 20
border = r
imgw = (w + 2 * border)
imgh = (h + 2 * border)
img = np.zeros((imgh,imgw))
for x, y in data:
ix = int((x - x0) * kx) + border
iy = int((y - y0) * ky) + border
if 0 <= ix < imgw and 0 <= iy < imgh:
img[iy][ix] += 1
return ndi.gaussian_filter(img, (r,r)) ## gaussian convolution
def generate_graph():
n = 1000
# data points range
data_ymin = -2.
data_ymax = 2.
data_xmin = -2.
data_xmax = 2.
# view area range
view_ymin = -.5
view_ymax = .5
view_xmin = -.5
view_xmax = .5
# generate data
xl = np.random.uniform(data_xmin, data_xmax, n)
yl = np.random.uniform(data_ymin, data_ymax, n)
zl = np.random.uniform(0, 1, n)
# get visible data points
xlvis = []
ylvis = []
for i in range(0,len(xl)):
if view_xmin < xl[i] < view_xmax and view_ymin < yl[i] < view_ymax:
xlvis.append(xl[i])
ylvis.append(yl[i])
fig = plt.figure()
# plot histogram
plt1 = fig.add_subplot(221)
plt1.set_axis_off()
t0 = time.clock()
zd, xe, ye = np.histogram2d(yl, xl, bins=10, range=[[view_ymin, view_ymax],[view_xmin, view_xmax]], normed=True)
plt.title('numpy.histogram2d - '+str(time.clock()-t0)+"sec")
plt.imshow(zd, origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
plt.scatter(xlvis, ylvis)
# plot density calculated with kdtree
plt2 = fig.add_subplot(222)
plt2.set_axis_off()
xi = np.linspace(view_xmin, view_xmax, 256)
yi = np.linspace(view_ymin, view_ymax, 256)
t0 = time.clock()
zd = grid_density_kdtree(xl, yl, xi, yi, 70)
plt.title('function of 5 nearest using kdtree\n'+str(time.clock()-t0)+"sec")
cmap=cm.jet
A = (cmap(zd/256.0)*255).astype(np.uint8)
#A[:,:,3] = zd
plt.imshow(A , origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
plt.scatter(xlvis, ylvis)
# gaussian filter
plt3 = fig.add_subplot(223)
plt3.set_axis_off()
t0 = time.clock()
zd = grid_density_gaussian_filter(view_xmin, view_ymin, view_xmax, view_ymax, 256, 256, zip(xl, yl))
plt.title('ndi.gaussian_filter - '+str(time.clock()-t0)+"sec")
plt.imshow(zd , origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
plt.scatter(xlvis, ylvis)
# boxsum smoothing
plt3 = fig.add_subplot(224)
plt3.set_axis_off()
t0 = time.clock()
zd = grid_density_boxsum(view_xmin, view_ymin, view_xmax, view_ymax, 256, 256, zip(xl, yl))
plt.title('boxsum smoothing - '+str(time.clock()-t0)+"sec")
plt.imshow(zd, origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
plt.scatter(xlvis, ylvis)
if __name__=='__main__':
generate_graph()
plt.show()

This approach is along the lines of some previous answers: increment a pixel for each spot, then smooth the image with a gaussian filter. A 256x256 image runs in about 350ms on my 6-year-old laptop.
import numpy as np
import scipy.ndimage as ndi
data = np.random.rand(30000,2) ## create random dataset
inds = (data * 255).astype('uint') ## convert to indices
img = np.zeros((256,256)) ## blank image
for i in xrange(data.shape[0]): ## draw pixels
img[inds[i,0], inds[i,1]] += 1
img = ndi.gaussian_filter(img, (10,10))

A very simple implementation that could be done (with C) in realtime and that only takes fractions of a second in pure python is to just compute the result in screen space.
The algorithm is
Allocate the final matrix (e.g. 256x256) with all zeros
For each point in the dataset increment the corresponding cell
Replace each cell in the matrix with the sum of the values of the matrix in an NxN box centered on the cell. Repeat this step a few times.
Scale result and output
The computation of the box sum can be made very fast and independent on N by using a sum table. Every computation just requires two scan of the matrix... total complexity is O(S + WHP) where S is the number of points; W, H are width and height of output and P is the number of smoothing passes.
Below is the code for a pure python implementation (also very un-optimized); with 30000 points and a 256x256 output grayscale image the computation is 0.5sec including linear scaling to 0..255 and saving of a .pgm file (N = 5, 4 passes).
def boxsum(img, w, h, r):
st = [0] * (w+1) * (h+1)
for x in xrange(w):
st[x+1] = st[x] + img[x]
for y in xrange(h):
st[(y+1)*(w+1)] = st[y*(w+1)] + img[y*w]
for x in xrange(w):
st[(y+1)*(w+1)+(x+1)] = st[(y+1)*(w+1)+x] + st[y*(w+1)+(x+1)] - st[y*(w+1)+x] + img[y*w+x]
for y in xrange(h):
y0 = max(0, y - r)
y1 = min(h, y + r + 1)
for x in xrange(w):
x0 = max(0, x - r)
x1 = min(w, x + r + 1)
img[y*w+x] = st[y0*(w+1)+x0] + st[y1*(w+1)+x1] - st[y1*(w+1)+x0] - st[y0*(w+1)+x1]
def saveGraph(w, h, data):
X = [x for x, y in data]
Y = [y for x, y in data]
x0, y0, x1, y1 = min(X), min(Y), max(X), max(Y)
kx = (w - 1) / (x1 - x0)
ky = (h - 1) / (y1 - y0)
img = [0] * (w * h)
for x, y in data:
ix = int((x - x0) * kx)
iy = int((y - y0) * ky)
img[iy * w + ix] += 1
for p in xrange(4):
boxsum(img, w, h, 2)
mx = max(img)
k = 255.0 / mx
out = open("result.pgm", "wb")
out.write("P5\n%i %i 255\n" % (w, h))
out.write("".join(map(chr, [int(v*k) for v in img])))
out.close()
import random
data = [(random.random(), random.random())
for i in xrange(30000)]
saveGraph(256, 256, data)
Edit
Of course the very definition of density in your case depends on a resolution radius, or is the density just +inf when you hit a point and zero when you don't?
The following is an animation built with the above program with just a few cosmetic changes:
used sqrt(average of squared values) instead of sum for the averaging pass
color-coded the results
stretching the result to always use the full color scale
drawn antialiased black dots where the data points are
made an animation by incrementing the radius from 2 to 40
The total computing time of the 39 frames of the following animation with this cosmetic version is 5.4 seconds with PyPy and 26 seconds with standard Python.

Histograms
The histogram way is not the fastest, and can't tell the difference between an arbitrarily small separation of points and 2 * sqrt(2) * b (where b is bin width).
Even if you construct the x bins and y bins separately (O(N)), you still have to perform some ab convolution (number of bins each way), which is close to N^2 for any dense system, and even bigger for a sparse one (well, ab >> N^2 in a sparse system.)
Looking at the code above, you seem to have a loop in grid_density() which runs over the number of bins in y inside a loop of the number of bins in x, which is why you're getting O(N^2) performance (although if you are already order N, which you should plot on different numbers of elements to see, then you're just going to have to run less code per cycle).
If you want an actual distance function then you need to start looking at contact detection algorithms.
Contact Detection
Naive contact detection algorithms come in at O(N^2) in either RAM or CPU time, but there is an algorithm, rightly or wrongly attributed to Munjiza at St. Mary's college London, which runs in linear time and RAM.
you can read about it and implement it yourself from his book, if you like.
I have written this code myself, in fact
I have written a python-wrapped C implementation of this in 2D, which is not really ready for production (it is still single threaded, etc) but it will run in as close to O(N) as your dataset will allow. You set the "element size", which acts as a bin size (the code will call interactions on everything within b of another point, and sometimes between b and 2 * sqrt(2) * b), give it an array (native python list) of objects with an x and y property and my C module will callback to a python function of your choice to run an interaction function for matched pairs of elements. it's designed for running contact force DEM simulations, but it will work fine on this problem too.
As I haven't released it yet, because the other bits of the library aren't ready yet, I'll have to give you a zip of my current source but the contact detection part is solid. The code is LGPL'd.
You'll need Cython and a c compiler to make it work, and it's only been tested and working under *nix environemnts, if you're on windows you'll need the mingw c compiler for Cython to work at all.
Once Cython's installed, building/installing pynet should be a case of running setup.py.
The function you are interested in is pynet.d2.run_contact_detection(py_elements, py_interaction_function, py_simulation_parameters) (and you should check out the classes Element and SimulationParameters at the same level if you want it to throw less errors - look in the file at archive-root/pynet/d2/__init__.py to see the class implementations, they're trivial data holders with useful constructors.)
(I will update this answer with a public mercurial repo when the code is ready for more general release...)

Your solution is okay, but one clear problem is that you're getting dark regions despite there being a point right in the middle of them.
I would instead center an n-dimensional Gaussian on each point and evaluate the sum over each point you want to display. To reduce it to linear time in the common case, use query_ball_point to consider only points within a couple standard deviations.
If you find that he KDTree is really slow, why not call query_ball_point once every five pixels with a slightly larger threshold? It doesn't hurt too much to evaluate a few too many Gaussians.

You can do this with a 2D, separable convolution (scipy.ndimage.convolve1d) of your original image with a gaussian shaped kernel. With an image size of MxM and a filter size of P, the complexity is O(PM^2) using separable filtering. The "Big-Oh" complexity is no doubt greater, but you can take advantage of numpy's efficient array operations which should greatly speed up your calculations.

Just a note, the histogram2d function should work fine for this. Did you play around with different bin sizes? Your initial histogram2d plot seems to just use the default bin sizes... but there's no reason to expect the default sizes to give you the representation you want. Having said that, many of the other solutions are impressive too.