I am creating a couple of multi dimensional arrays using NumPy and inititalising them based on the index as follows:
pos_data = []
# Some typical values
d = 2 # Could also be 3
vol_ext = (1000, 500) # If d = 3, this will have another entry
ratio = [5.0, 8.0] # Again, if d = 3, it will have another entry
for i in range(d):
pos_data.append(np.zeros(vol_ext))
if d == 2:
for y in range(vol_ext[1]):
for x in range(vol_ext[0]):
pos_data[0][x, y] = (x - 1.0) * ratio[0]
pos_data[1][x, y] = (y - 1.0) * ratio[1]
elif d == 3:
for z in range(vol_ext[2]):
for y in range(vol_ext[1]):
for x in range(vol_ext[0]):
pos_data[0][x, y, z] = (x - 1.0) * ratio[0]
pos_data[1][x, y, z] = (y - 1.0) * ratio[1]
pos_data[2][x, y, z] = (z - 1.0) * ratio[2]
The looping is a bit ugly and slow as well. Additionally, if I have a 3-dimensional object then I have to have another nested loop.
I was wondering if there is a Pythonic way to generate these values as they are just based on the x, y and z indices. I tried to use the combinatorics bit from itertools, but I could not make it work.
It's easy with np.meshgrid:
pos_data = np.meshgrid(*(r * (np.arange(s) - 1.0)
for s, r in zip(vol_ext, ratio)), indexing='ij')
I would generate a two or three dimensional numpy.meshgrid of data, then scale each entry by the ratio per slice.
For the 2D case:
(X, Y) = np.meshgrid(np.arange(vol_ext[1]), np.arange(vol_ext[0]))
pos_data = [(Y - 1) * ratio[0], (X - 1) * ratio[1]]
For the 3D case:
(X, Y, Z) = np.meshgrid(np.arange(vol_ext[2]), np.arange(vol_ext[1]), np.arange(vol_ext[0]))
pos_data = [(Z - 1) * ratio[0], (Y - 1) * ratio[1], (X - 1) * ratio[2]]
Example using your 2D data
pos_data has been generated by your code. I've created a new list pos_data2 that stores the equivalent list using the above solution:
In [40]: vol_ext = (1000, 500)
In [41]: (X, Y) = np.meshgrid(np.arange(vol_ext[1]), np.arange(vol_ext[0]))
In [42]: pos_data2 = [(Y - 1) * ratio[0], (X - 1) * ratio[1]]
In [43]: np.allclose(pos_data[0], pos_data2[0])
Out[43]: True
In [44]: np.allclose(pos_data[1], pos_data2[1])
Out[44]: True
Making this adaptive based on vol_ext
We can combine this with a list comprehension where we can take advantage of the fact that the output of numpy.meshgrid is a tuple:
pts = [np.arange(v) for v in reversed(vol_ext)]
pos_data = [(D - 1) * R for (D, R) in zip(reversed(np.meshgrid(*pts)), ratio)]
The first line of code generates the range of points per desired dimension as a list. We then use a list comprehension to compute the desired calculations per slice by iterating over each desired grid of points in the desired dimension combined with the correct ratio to apply.
Example Run
In [49]: pts = [np.arange(v) for v in reversed(vol_ext)]
In [50]: pos_data2 = [(D - 1) * R for (D, R) in zip(reversed(np.meshgrid(*pts)), ratio)]
In [51]: np.all([np.allclose(p, p2) for (p, p2) in zip(pos_data, pos_data2)])
Out[51]: True
The last line goes through each slice and ensures both lists align.
I think there are a couple of things to consider:
is there a reason that pos_data has to be a list?
don't have another variable (d) that you have to hard code, when it is always supposed to be the length of some other tuple.
With these in mind, you can solve your problem of variable numbers of for loops using itertools.product, which basically is just a shorthand for nested for loops. The positional args for product are the ranges of the loops.
My implementation is:
from itertools import product
vol_ext = (1000, 500) # If d = 3, this will have another entry
ratio = [5.0, 8.0] # Again, if d = 3, it will have another entry
pos_data_new = np.zeros((len(ratio), *vol_ext))
# now loop over each dimension in `vol_ext`. Since `product` expects
# positional arguments, we have to unpack a tuple of `range(vol)`s.
for inds in product(*(range(vol) for vol in vol_ext)):
# inds is now a tuple, and we have to combine it with a slice in
# in the first dimension, and use it as an array on the right hand
# side to do the computation.
pos_data_new[(slice(None),) + inds] = (np.array(inds) - 1) * ratio
I don't think this will be any faster, but it certainly looks nicer.
Note that pos_data_new is now an array, to get it as a list in the first dimension, as per the original example, is simple enough.
Related
I am trying to evaluate a function at discretized point and stored in column-major order like this:
import numpy as np;
N = 3 ##
n = N * N
h = 1 / (N + 1) # step size
h2 = h**2 #
deltaX = np.zeros(N)
deltaY = np.zeros(N);
def Function(x, y):
output = -20. * np.pi * np.sin(2 * np.pi * x) * sin(4 * np.pi * y)
return output
## Equally spaced delta:
for i in range(1, N + 1):
deltaX[i - 1] = i * h;
deltaY[i - 1] = i * h;
### Lexicographic Row order ###
### Evaluation of function at deltaX and deltaY
feval = np.zeros((n, 1))
How could I approach to evaluate the discretization for this function?
Good news: your function properly uses numpy operations, so is completely vectorized. That means that you can evaluate it at every element of the input arrays.
The shape of the inputs don't have to match exactly. They just have to broadcast together. That means that only non-singleton dimensions need to match.
So start by creating the appropriate input arrays. Numpy provides the tools to do this elegantly without looping:
N = 3
h = 1 / (N + 1)
delta_x = np.arange(1., N + 1.) * h
delta_y = np.linspace(h, N * h, N)[:, None]
I deliberately used two different ways to create the coordinate arrays, to serve as an example. In practice, you'd want to use one of the two methods.
The index [:, None] turns delta_y into a column vector. None introduces a new singleton axis. There are any number of Other ways to do the same thing, like `delta_y = ....reshape(-1, 1).
And read the docs I linked to, and for all the functions I used.
Now that you have a column in the y direction and a row in the x, you can call Function as just
val = Function(delta_x, delta_y)
The operation of arranging the 2D matrix val into a 1D array is called raveling. By default, it uses the default row-major order that numpy uses in memory. This order is also called "C" order. The alternative arrangement is to interpret the array in column major order, like Matlab does. This is called Fortran order. It will require a copy of the data since that's not how the elements are laid out in memory.
One way to ravel in Fortran order:
feval = val.ravel(order='F')
An alternative is to transpose and use C order:
feval = val.T.ravel()
The last two lines can be combined, so you end up with 3 lines:
delta_x = h * np.arange(1., N + 1.)
delta_y = h * np.arange(1., N + 1.)[:, None]
feval = Function(delta_x, delta_y).ravel(order='F')
You could make it into a one-liner, but that's pushing it.
I have a large field of 2D-position data, given as two arrays x and y, where len(x) == len(y). I would like to return the array of indices idx_masked at which (x[idx_masked], y[idx_masked]) is masked by an N x N int array called mask. That is, mask[x[idx_masked], y[idx_masked]] == 1. The mask array consists of 0s and 1s only.
I have come up with the following solution, but it (specifically, the last line below) is very slow, given that I have N x N = 5000 x 5000, repeated 1000s of times:
import numpy as np
import matplotlib.pyplot as plt
# example mask of one corner of a square
N = 100
mask = np.zeros((N, N))
mask[0:10, 0:10] = 1
# example x and y position arrays in arbitrary units
x = np.random.uniform(0, 1, 1000)
y = np.random.uniform(0, 1, 1000)
x_bins = np.linspace(np.min(x), np.max(x), N)
y_bins = np.linspace(np.min(y), np.max(y), N)
x_bin_idx = np.digitize(x, x_bins)
y_bin_idx = np.digitize(y, y_bins)
idx_masked = np.ravel(np.where(mask[y_bin_idx - 1, x_bin_idx - 1] == 1))
plt.imshow(mask[::-1, :])
plt.scatter(x, y, color='red')
plt.scatter(x[idx_masked], y[idx_masked], color='blue')
Is there a more efficient way of doing this?
Given that mask overlays your field with identically-sized bins, you do not need to define the bins explicitly. *_bin_idx can be determined at each location from a simple floor division, since you know that each bin is 1 / N in size. I would recommend using 1 - 0 for the total width (what you passed into np.random.uniform) instead of x.max() - x.min(), if of course you know the expected size of the range.
x0 = 0 # or x.min()
x1 = 1 # or x.max()
x_bin = (x1 - x0) / N
x_bin_idx = ((x - x0) // x_bin).astype(int)
# ditto for y
This will be faster and simpler than digitizing, and avoids the extra bin at the beginning.
For most purposes, you do not need np.where. 90% of the questions asking about it (including this one) should not be using where. If you want a fast way to access the necessary elements of x and y, just use a boolean mask. The mask is simply
selction = mask[x_bin_idx, y_bin_idx].astype(bool)
If mask is already a boolean (which it should be anyway), the expression mask[x_bin_idx, y_bin_idx] is sufficient. It results in an array of the same size as x_bin_idx and y_bin_idx (which are the same size as x and y) containing the mask value for each of your points. You can use the mask as
x[selection] # Elements of x in mask
y[selection] # Elements of y in mask
If you absolutely need the integer indices, where is sill not your best option.
indices = np.flatnonzero(selection)
OR
indices = selection.nonzero()[0]
If your goal is simply to extract values from x and y, I would recommend stacking them together into a single array:
coords = np.stack((x, y), axis=1)
This way, instead of having to apply indices twice, you can extract the values with just
coords[selection, :]
OR
coords[indices, :]
Depending on the relative densities of mask and x and y, either the boolean masking or linear indexing may be faster. You will have to time some relevant cases to get a better intuition.
I'm trying to sum a two dimensional function using the array method, somehow, using a for loop is not outputting the correct answer. I want to find (in latex) $$\sum_{i=1}^{M}\sum_{j=1}^{M_2}\cos(i)\cos(j)$$ where according to Mathematica the answer when M=5 is 1.52725. According to the for loop:
def f(N):
s1=0;
for p1 in range(N):
for p2 in range(N):
s1+=np.cos(p1+1)*np.cos(p2+1)
return s1
print(f(4))
is 0.291927.
I have thus been trying to use some code of the form:
def f1(N):
mat3=np.zeros((N,N),np.complex)
for i in range(0,len(mat3)):
for j in range(0,len(mat3)):
mat3[i][j]=np.cos(i+1)*np.cos(j+1)
return sum(mat3)
which again
print(f1(4))
outputs 0.291927. Looking at the array we should find for each value of i and j a matrix of the form
mat3=[[np.cos(1)*np.cos(1),np.cos(2)*np.cos(1),...],[np.cos(2)*np.cos(1),...]...[np.cos(N+1)*np.cos(N+1)]]
so for N=4 we should have
mat3=[[np.cos(1)*np.cos(1) np.cos(2)*np.cos(1) ...] [np.cos(2)*np.cos(1) ...]...[... np.cos(5)*np.cos(5)]]
but what I actually get is the following
mat3=[[0.29192658+0.j 0.+0.j 0.+0.j ... 0.+0.j] ... [... 0.+0.j]]
or a matrix of all zeros apart from the mat3[0][0] element.
Does anybody know a correct way to do this and get the correct answer? I chose this as an example because the problem I'm trying to solve involves plotting a function which has been summed over two indices and the function that python outputs is not the same as Mathematica (i.e., a function of the form $$f(E)=\sum_{i=1}^{M}\sum_{j=1}^{M_2}F(i,j,E)$$).
The return statement is not indented correctly in your sample code. It returns immediately in the first loop iteration. Indent it on the function body instead, so that both for loops finish:
def f(N):
s1=0;
for p1 in range(N):
for p2 in range(N):
s1+=np.cos(p1+1)*np.cos(p2+1)
return s1
>>> print(f(5))
1.527247272700347
I have moved your code to a more numpy-ish version:
import numpy as np
N = 5
x = np.arange(N) + 1
y = np.arange(N) + 1
x = x.reshape((-1, 1))
y = y.reshape((1, -1))
mat = np.cos(x) * np.cos(y)
print(mat.sum()) # 1.5272472727003474
The trick here is to reshape x to a column and y to a row vector. If you multiply them, they are matched up like in your loop.
This should be more performant, since cos() is only called 2*N times. And it avoids loops (bad in python).
UPDATE (regarding your comment):
This pattern can be extended in any dimension. Basically, you get something like a crossproduct. Where every instance of x is matched up with every instance of y, z, u, k, ... Along the corresponding dimensions.
It's a bit confusing to describe, so here is some more code:
import numpy as np
N = 5
x = np.arange(N) + 1
y = np.arange(N) + 1
z = np.arange(N) + 1
x = x.reshape((-1, 1, 1))
y = y.reshape((1, -1, 1))
z = z.reshape((1, 1, -1))
mat = z**2 * np.cos(x) * np.cos(y)
# x along first axis
# y along second, z along third
# mat[0, 0, 0] == 1**2 * np.cos(1) * np.cos(1)
# mat[0, 4, 2] == 3**2 * np.cos(1) * np.cos(5)
If you use this for many dimensions, and big values for N, you will run into memory problems, though.
I have the following code snippet (for Hough circle transform):
for r in range(1, 11):
for t in range(0, 360):
trad = np.deg2rad(t)
b = x - r * np.cos(trad)
a = y - r * np.sin(trad)
b = np.floor(b).astype('int')
a = np.floor(a).astype('int')
A[a, b, r-1] += 1
Where A is a 3D array of shape (height, width, 10), and
height and width represent the size of a given image.
My goal is to convert the snippet exclusively to numpy code.
My attempt is this:
arr_r = np.arange(1, 11)
arr_t = np.deg2rad(np.arange(0, 360))
arr_cos_t = np.cos(arr_t)
arr_sin_t = np.sin(arr_t)
arr_rcos = arr_r[..., np.newaxis] * arr_cos_t[np.newaxis, ...]
arr_rsin = arr_r[..., np.newaxis] * arr_sin_t[np.newaxis, ...]
arr_a = (y - arr_rsin).flatten().astype('int')
arr_b = (x - arr_rcos).flatten().astype('int')
Where x and y are two scalar values.
I am having trouble at converting the increment part: A[a,b,r] += 1. I thought of this: A[a,b,r] counts the number of occurrences of the pair (a,b,r), so a clue was to use a Cartesian product (but the arrays are too large).
Any tips or tricks I can use?
Thank you very much!
Edit: after filling A, I need (a,b,r) as argmax(A). The tuple (a,b,r) identifies a circle and its value in A represents the confidence value. So I want that tuple with the highest value in A. This is part of the voting algorithm from Hough circle transform: find circle parameter with unknown radius.
Method #1
Here's one way leveraging broadcasting to get the counts and update A (this assumes the a and b values computed in the intermediate steps are positive ones) -
d0,d1,d2 = A.shape
arr_r = np.arange(1, 11)
arr_t = np.deg2rad(np.arange(0, 360))
arr_b = np.floor(x - arr_r[:,None] * np.cos(arr_t)).astype('int')
arr_a = np.floor(y - arr_r[:,None] * np.sin(arr_t)).astype('int')
idx = (arr_a*d1*d2) + (arr_b * d2) + (arr_r-1)[:,None]
A.flat[:idx.max()+1] += np.bincount(idx.ravel())
# OR A.flat += np.bincount(idx.ravel(), minlength=A.size)
Method #2
Alternatively, we could avoid bincount to replace the last step in approach #1, like so -
idx.ravel().sort()
idx.shape = (-1)
grp_idx = np.flatnonzero(np.concatenate(([True], idx[1:]!=idx[:-1],[True])))
A.flat[idx[grp_idx[:-1]]] += np.diff(grp_idx)
Improvement with numexpr
We could also leverage numexpr module for faster sine, cosine computations, like so -
import numexpr as ne
arr_r2D = arr_r[:,None]
arr_b = ne.evaluate('floor(x - arr_r2D * cos(arr_t))').astype(int)
arr_a = ne.evaluate('floor(y - arr_r2D * sin(arr_t))').astype(int)
np.add(np.array ([arr_a, arr_b, 10]), 1)
I have a 2D array shaped (1002,1004). For this question it could be generated via:
a = numpy.arange( (1002 * 1004) ).reshape(1002, 1004)
What I do is generate two lists. The lists are generated via:
theta = (61/180.) * numpy.pi
x = numpy.arange(a.shape[0]) #(1002, )
y = numpy.arange(a.shape[1]) #(1004, )
max_y_for_angle = int(y[-1] - (x[-1] / numpy.tan(theta)))
The first list is given by:
x_list = numpy.linspace(0, x[-1], len(x))
Note that this list is identical to x. However, for illustration purposes and to give a clear picture I declared this 'list'.
What I now want to do is create a y_list which is as long as x_list. I want to use these lists to determine the elements in my 2D array. After I determine and store the sum of the elements, I want to shift my y_list by one and determine the sum of the elements again. I want to do this for max_y_for_angle iterations. The code I have is:
sum_list = numpy.zeros(max_y_for_angle)
for idx in range(max_y_for_angle):
y_list = numpy.linspace((len(x) / numpy.tan(theta)) + idx, y[0] + idx , len(x))
elements = 0
for i in range(len(x)):
elements += a[x_list[i]][y_list[i]]
sum_list[idx] = elements
This operation works. However, as one might imagine this takes a lot of time due to the for loop within a for loop. The number of iterations of the for loops do not help as well. How can I speed things up? The operation now takes about 1 s. I'm looking for something below 200 ms.
Is it maybe possible to return a list of the 2D array elements when the inputs are x_list and y_list? I tried the following but this does not work:
a[x_list][y_list]
Thank you very much!
It's possible to return an array of elements form a 2d array by doing a[x, y] where x and y are both integer arrays. This is called advanced indexing or sometimes fancy indexing. In your question you mention lists a lot but never actually use any lists in your code, x_list and y_list are both arrays. Also, numpy multidimensional arrays are generally indexed a[i, j] even when when i and j are integers values.
Using fancy indexing along with some clean up of you code produced this:
import numpy
def line_sums(a, thata):
xsize, ysize = a.shape
tan_theta = numpy.tan(theta)
max_y_for_angle = int(ysize - 1 - ((xsize - 1) / tan_theta))
x = numpy.arange(xsize)
y_base = numpy.linspace(xsize / tan_theta, 0, xsize)
y_base = y_base.astype(int)
sum_list = numpy.zeros(max_y_for_angle)
for idx in range(max_y_for_angle):
sum_list[idx] = a[x, y_base + idx].sum()
return sum_list
a = numpy.arange( (1002 * 1004) ).reshape(1002, 1004)
theta = (61/180.) * numpy.pi
sum_list = line_sums(a, theta)
Hope that helps.