I am working on a program in python that makes use of a function very similar to the addWeighted function in openCV. The difference is that it doesn't actually add the numpy arrays representing the images, instead, it takes whichever pixel is brighter at any particular coordinate and uses that value.
What I have been finding, however, is that despite the fact that these functions do very similar things, the addWeighted function is much faster. So my question is, how can I modify my current solution to be equally as fast? Is there a way I can use the multiprocessing module, or something similar?
Here is the code:
image = np.zeros(image_1.shape)
for row_index, row in enumerate(image_1):
for col_index, col in enumerate(row):
pixel_1 = image_1[row_index, col_index]
pixel_2 = image_2[row_index, col_index]
sum_1 = int(pixel_1[0]) + int(pixel_1[1]) + int(pixel_1[2])
sum_2 = int(pixel_2[0]) + int(pixel_2[1]) + int(pixel_2[2])
if sum_2 > sum_1:
image[row_index, col_index] = pixel_2
else:
image[row_index, col_index] = pixel_1
Where image_1 and image_2 are both numpy arrays representing images, both with the same shape (720, 1280, 3).
One vectorized approach would be -
mask = image_2.astype(int).sum(-1) > image_1.astype(int).sum(-1)
out = np.where(mask[...,None], image_2, image_1)
Steps :
Convert to int dtypes, sum along the last axis and perform element-wise comparisons. This would give us a mask.
Use np.where with this mask, extended to the same no. of dims as input arrays to do the choosing. This employs the concept of NumPy broadcasting to do the choosing in a vectorized manner. So, that's worth a good look.
Note: Alternatively, we can also use keepdims=True to keep the no. of dims while summing and thus avoid extending dims in the next step.
Related
I am trying to get rid of the for loop and instead do an array-matrix multiplication to decrease the processing time when the weights array is very large:
import numpy as np
sequence = [np.random.random(10), np.random.random(10), np.random.random(10)]
weights = np.array([[0.1,0.3,0.6],[0.5,0.2,0.3],[0.1,0.8,0.1]])
Cov_matrix = np.matrix(np.cov(sequence))
results = []
for w in weights:
result = np.matrix(w)*Cov_matrix*np.matrix(w).T
results.append(result.A)
Where:
Cov_matrix is a 3x3 matrix
weights is an array of n lenght with n 1x3 matrices in it.
Is there a way to multiply/map weights to Cov_matrix and bypass the for loop? I am not very familiar with all the numpy functions.
I'd like to reiterate what's already been said in another answer: the np.matrix class has much more disadvantages than advantages these days, and I suggest moving to the use of the np.array class alone. Matrix multiplication of arrays can be easily written using the # operator, so the notation is in most cases as elegant as for the matrix class (and arrays don't have several restrictions that matrices do).
With that out of the way, what you need can be done in terms of a call to np.einsum. We need to contract certain indices of three matrices while keeping one index alone in two matrices. That is, we want to perform w_{ij} * Cov_{jk} * w.T_{ki} with a summation over j, k, giving us an array with i indices. The following call to einsum will do:
res = np.einsum('ij,jk,ik->i', weights, Cov_matrix, weights)
Note that the above will give you a single 1d array, whereas you originally had a list of arrays with shape (1,1). I suspect the above result will even make more sense. Also, note that I omitted the transpose in the second weights argument, and this is why the corresponding summation indices appear as ik rather than ki. This should be marginally faster.
To prove that the above gives the same result:
In [8]: results # original
Out[8]: [array([[0.02803215]]), array([[0.02280609]]), array([[0.0318784]])]
In [9]: res # einsum
Out[9]: array([0.02803215, 0.02280609, 0.0318784 ])
The same can be achieved by working with the weights as a matrix and then looking at the diagonal elements of the result. Namely:
np.diag(weights.dot(Cov_matrix).dot(weights.transpose()))
which gives:
array([0.03553664, 0.02394509, 0.03765553])
This does more calculations than necessary (calculates off-diagonals) so maybe someone will suggest a more efficient method.
Note: I'd suggest slowly moving away from np.matrix and instead work with np.array. It takes a bit of getting used to not being able to do A*b but will pay dividends in the long run. Here is a related discussion.
I have an image I've read from file with shape (m,n,3) (i.e. it has 3 channels). I also have a matrix to convert the color space with dimensions (3,3). I've already arrived at a few different ways of applying this matrix to each vector in the image; for example,
np.einsum('ij,...j',transform,image)
appears to make for the same results as the following (far slower) implementation.
def convert(im: np.array, transform: np.array) -> np.array:
""" Convert an image array to another colorspace """
dimensions = len(im.shape)
axes = im.shape[:dimensions-1]
# Create a new array (respecting mutability)
new_ = np.empty(im.shape)
for coordinate in np.ndindex(axes):
pixel = im[coordinate]
pixel_prime = transform # pixel
new_[coordinate] = pixel_prime
return new_
However, I found that the following is even more efficient while testing on the example image with line_profiler.
np.moveaxis(np.tensordot(transform, X, axes=((-1),(-1))), 0, 2)
The problem I'm having here is using just a np.tensordot, i.e. removing the need for np.moveaxis. I've spent a few hours attempting to find a solution (I'm guessing it resides in choosing the correct axes), so I thought I'd ask others for help.
You can do it concisely with tensordot if you make image the first argument:
np.tensordot(image, transform, axes=(-1, 1))
You can get better performance from einsum by using the argument optimize=True (requires numpy 1.12 or later):
np.einsum('ij,...j', transform, image, optimize=True)
Or (as Paul Panzer pointed out in a comment), you can simply use matrix multiplication:
image # transform.T
They all take about the same time on my computer.
I'm with issues translating Matlab to Python code, specially when it involves matrices / arrays.
Here, I have a 2D numpy array called output and I am computing a vector of row-major indexes t_ind of the elements that are higher than a variable vmax:
t_ind = np.flatnonzero(output > vmax)
Now I'd like to use these indexes to create a matrix based on that. In MATLAB, I could do that directly:
output(t_ind) = 2*vmax - output(t_ind);
But in Python this does not work. Specifically, I get an IndexError stating that I'm out of bounds.
I tried to figure it out, but the most elegant solution that I could think involves using np.hstack() to transform the array into a vector, compare the indexes, collect them in another variable and come back.
Could you shed some light on this?
For a 1D array, the use of np.flatnonzero is correct. Specifically, the equivalent numpy syntax would be:
output[t_ind] = 2*vmax - output[t_ind]
Also, you can achieve the same thing using Boolean operators. MATLAB also has this supported, and so if you want to translate between the two, Boolean (or logical in the MATLAB universe) is the better way to go:
output[output > vmax] = 2*vmax - output[output > vmax]
For the 2D case, you don't use np.flatnonzero. Use np.where instead:
t_ind = np.where(output > v_max)
output[t_ind] = 2*vmax - output[t_ind]
t_ind will return a tuple of numpy arrays where the first element gives you the row locations and the second element gives you the column locations of those values that satisfied the Boolean condition that is placed into np.where.
As a small note, the case for Boolean indexing still applies to any dimensions of the matrix you desire. However, np.flatnonzero would compute row-major indices of those points that satisfy the input condition into np.flatnonzero. The reason why you're getting an error is because you are trying to use row-major indices to access a 2D array. Though linear indexing is supported in Python, this is not supported in numpy - you would have to access each dimension independently to do this indexing, which is what specifying t_ind as the input indexes into output would be doing.
Numpy supports both boolean indexing and multi-dimensional indexing so you don't need to jump through all those hoops, here are two ways to get what you want:
# The setup
import numpy as np
a = np.random.random((3, 4))
vmax = 1.2
output = np.zeros(a.shape, a.dtype)
# Method one, use a boolean array to index
mask = a > .5
output[mask] = 2 * vmax - a[mask]
# Method tow, use indices to index.
t_ind = np.nonzero(a > .5)
output[t_ind] = 2 * vmax - a[t_ind]
I have a np.ndarray with numbers that indicate spots of interest, I am interested in the spots which have values 1 and 9.
Right now they are being extracted as such:
maskindex.append(np.where(extract.variables['mask'][0] == 1) or np.where(megadatalist[0].variables['mask'][0] == 9))
xval = maskindex[0][1]
yval = maskindex[0][0]
I need to apply these x and y values to the arrays that I am operating on, to speed things up.
I have 140 arrays that are each 734 x 1468, I need the mean, max, min, std calculated for each field. And I was hoping there was an easy way for applying the masked array to speed up the operations, right now I am simply doing it on the entire arrays as such:
Average_List = np.mean([megadatalist[i].variables['analysed_sst'][0] for i in range(0,Numbers_of_datasets)], axis=0)
Average_Error_List = np.mean([megadatalist[i].variables['analysis_error'][0] for i in range(0,Numbers_of_datasets)], axis=0)
Std_List = np.std([megadatalist[i].variables['analysed_sst'][0] for i in range(0,Numbers_of_datasets)], axis=0)
Maximum_List = np.maximum.reduce([megadatalist[i].variables['analysed_sst'][0] for i in range(0,Numbers_of_datasets)])
Minimum_List = np.minimum.reduce([megadatalist[i].variables['analysed_sst'][0] for i in range(0,Numbers_of_datasets)])
Any ideas on how to speed things up would be highly appreciated
I may have solved it partially, depending on what you're aiming for. The following code reduces an array arr to a 1d array with only the relevant indicies. You can then do the needed calculations without considering the unwanted locations
arr = np.array([[0,9,9,0,0,9,9,1],[9,0,1,9,0,0,0,1]])
target = [1,9] # wanted values
index = np.where(np.in1d(arr.ravel(), target).reshape(arr.shape))
no_zeros = arr[index]
At this stage "all you need" is to reinsert the values "no_zeros" on an array of zeroes with appropriate shape, on the indices given in "index". One way is to flatten the index array and recalculate the indices, so that they match a flattened arr array. Then use numpy.insert(np.zeroes(arr.shape),new_index,no_zeroes) and then reshaping to the appropriate shape afterwards. Reshaping is constant time in numpy. Admittedly, I have not figured out a fast numpy way to create the new_index array.
Hope it helps.
In Numpy 1.4.1, what is the simplest or most efficient way of calculating the histogram of a masked array? numpy.histogram and pyplot.hist do count the masked elements, by default!
The only simple solution I can think of right now involves creating a new array with the non-masked value:
histogram(m_arr[~m_arr.mask])
This is not very efficient, though, as this unnecessarily creates a new array. I'd be happy to read about better ideas!
(Undeleting this as per discussion above...)
I'm not sure whether or not the numpy developers would consider this a bug or expected behavior. I asked on the mailing list, so I guess we'll see what they say.
Either way, it's an easy fix. Patching numpy/lib/function_base.py to use numpy.asanyarray rather than numpy.asarray on the inputs to the function will allow it to properly use masked arrays (or any other subclass of an ndarray) without creating a copy.
Edit: It seems like it is expected behavior. As discussed here:
If you want to ignore masked data it's
just on extra function call
histogram(m_arr.compressed())
I don't think the fact that this makes
an extra copy will be relevant,
because I guess full masked array
handling inside histogram will be a
lot more expensive.
Using asanyarray would also allow
matrices in and other subtypes that
might not be handled correctly by the
histogram calculations.
For anything else besides dropping
masked observations, it would be
necessary to figure out what the
masked array definition of a histogram
is, as Bruce pointed out.
Try hist(m_arr.compressed()).
This is a super old question, but these days I just use:
numpy.histogram(m_arr, bins=.., range=.., density=False, weights=m_arr_mask)
Where m_arr_mask is an array with the same shape as m_arr, consisting of 0 values for elements of m_arr to be excluded from the histogram and 1 values for elements that are to be included.
After running into casting issues by trying Erik's solution (see https://github.com/numpy/numpy/issues/16616), I decided to write a numba function to achieve this behavior.
Some of the code was inspired by https://numba.pydata.org/numba-examples/examples/density_estimation/histogram/results.html. I added the mask bit.
import numpy
import numba
#numba.jit(nopython=True)
def compute_bin(x, bin_edges):
# assuming uniform bins for now
n = bin_edges.shape[0] - 1
a_min = bin_edges[0]
a_max = bin_edges[-1]
# special case to mirror NumPy behavior for last bin
if x == a_max:
return n - 1 # a_max always in last bin
bin = int(n * (x - a_min) / (a_max - a_min))
if bin < 0 or bin >= n:
return None
else:
return bin
#numba.jit(nopython=True)
def masked_histogram(img, bin_edges, mask):
hist = numpy.zeros(len(bin_edges) - 1, dtype=numpy.intp)
for i, value in enumerate(img.flat):
if mask.flat[i]:
bin = compute_bin(value, bin_edges)
if bin is not None:
hist[int(bin)] += 1
return hist # , bin_edges
The speedup is significant. On a (1000, 1000) image: