I am currently using numpy to create an array. I would like to use vectorized implementations to more efficiently take the average of the elements in a position, (i, j). These arrays are coming from images in a file directory which have all been standardized to a fixed size.
However, when I try to add the image arrays, the sum of each element is returned in a form a (mod 256). How can I change the maximum value of the elements?
Your arrays are presumably of type numpy.uint8, so they wraparound when they hit 256.
If you want to get larger results, use astype to convert the first argument to a larger data type, e.g.:
a = np.array(..., dtype=np.uint8)
b = np.array(..., dtype=np.uint8)
c = a.astype(np.uint32) + b
and you'll get a result array of the larger data type too.
Per #Eric, to avoid the temporary, you can use the numpy add function (not method) to do the addition, passing a dtype so the result is of the new type even as the inputs are not converted, avoiding a temporary (at least at the Python level):
c = np.add(a, b, dtype=np.uint32)
You would be better off creating the output array first:
average = numpy.zeros(a.shape, numpy.float32)
image = numpy.zeros_like(average)
Then traversing the images and adding them up in-place:
for i in images:
image[:] = function_that_reads_images_as_uint8(i)
average += image
average /= len(images)
You might get away with int types if you didn't need the precision in the division step.
Related
I'm trying to use this 1000 dimension wikipedia word2vec model to analyze some documents.
Using introspection I found out that the vector representation of a word is a 1000 dimension numpy.ndarray, however whenever I try to create an ndarray to find the nearest words I get a value error:
ValueError: maximum supported dimension for an ndarray is 32, found 1000
and from what I can tell by looking around online 32 is indeed the maximum supported number of dimensions for an ndarray - so what gives? How is gensim able to output a 1000 dimension ndarray?
Here is some example code:
doc = [model[word] for word in text if word in model.vocab]
out = []
n = len(doc[0])
print(n)
print(len(model["hello"]))
print(type(doc[0]))
for i in range(n):
sum = 0
for d in doc:
sum += d[i]
out.append(sum/n)
out = np.ndarray(out)
which outputs:
1000
1000
<class 'numpy.ndarray'>
ValueError: maximum supported dimension for an ndarray is 32, found 1000
The goal here would be to compute the average vector of all words in the corpus in a format that can be used to find nearby words in the model so any alternative suggestions to that effect are welcome.
You're calling numpy's ndarray() constructor-function with a list that has 1000 numbers in it – your hand-calculated averages of each of the 1000 dimensions.
The ndarray() function expects its argument to be the shape of the matrix constructed, so it's trying to create a new matrix of shape (d[0], d[1], ..., d[999]) – and then every individual value inside that matrix would be addressed with a 1000-int set of coordinates. And, indeed numpy arrays can only have 32 independent dimensions.
But even if you reduced the list you're supplying to ndarray() to just 32 numbers, you'd still have a problem, because your 32 numbers are floating-point values, and ndarray() is expecting integral counts. (You'd get a TypeError.)
Along the approach you're trying to take – which isn't quite optimal as we'll get to below – you really want to create a single vector of 1000 floating-point dimensions. That is, 1000 cell-like values – not d[0] * d[1] * ... * d[999] separate cell-like values.
So a crude fix along the lines of your initial approach could be replacing your last line with either:
result = np.ndarray(len(d))
for i in range(len(d)):
result[i] = d[i]
But there are many ways to incrementally make this more efficient, compact, and idiomatic – a number of which I'll mention below, even though the best approach, at bottom, makes most of these interim steps unnecessary.
For one, instead of that assignment-loop in my code just above, you could use Python's bracket-indexing assignment option:
result = np.ndarray(len(d))
result[:] = d # same result as previous 3-lines w/ loop
But in fact, numpy's array() function can essentially create the necessary numpy-native ndarray from a given list, so instead of using ndarray() at all, you could just use array():
result = np.array(d) # same result as previous 2-lines
But further, numpy's many functions for natively working with arrays (and array-like lists) already include things to do averages-of-many-vectors in a single step (where even the looping is hidden inside very-efficient compiled code or CPU bulk-vector operations). For example, there's a mean() function that can average lists of numbers, or multi-dimensional arrays of numbers, or aligned sets of vectors, and so forth.
This allows faster, clearer, one-liner approaches that can replace your entire original code with something like:
# get a list of available word-vetors
doc = [model[word] for word in text if word in model.vocab]
# average all those vectors
out = np.mean(doc, axis=0)
(Without the axis argument, it'd average together all individual dimension-values , in all slots, into just one single final average number.)
I am processing a 1-dimensional array of data. I am looking for a greater than expected variance within a 7 item window, but would then need to correct values within a larger, 20 item window.
I'm using python and NumPy to accomplish the task. I started using numpy stride_tricks to create a moving window through the original array. Stride_tricks seemed the fastest way computationally to find the higher variance in the smaller windows. I get stuck when attempting to expand the window to correct the data.
Here's my current code:
with open('Sather-line-352-original.txt') as f:
array = np.array(map(int, f))
# shape defines the dimensions of the new temp array.
# strides define memory-based coordinates of original array items.
def pystride(array,frame_length,strided_items):
num_frames = 1 + ((len(array) - frame_length)/ strided_items)
row_stride = array.itemsize * strided_items
col_stride = array.itemsize
a_strided = np.lib.stride_tricks.as_strided(
array,
shape=(num_frames, frame_length),
strides=(row_stride, col_stride)
)
return a_strided
def find_max_min(array):
max_diff = 120
for sub in pystride(array,frame_length=7,strided_items=2):
max_val = max(sub)
min_val = min(sub)
if abs(max_val - min_val) >= max_diff:
# assign 'pointers' in original array indicating where large diffs are found.
sub[0] = int('{:<05}'.format(sub[0]))
find_max_min(array)
Particularly, is there a way to determine where the as_strided sub-array is in the original array? I've been modifying the data by appending 000 at the end of an integer value to act as a makeshift pointer, but that seems like a hack at best. Could I temporarily resize the sub-array for modification, and then resize back to the smaller window to continue scanning?
Here is a snippet of the array:
93,94,91,90,93,85,79,60,50,48,54,58,47,49,63,91,134,165,184,178,161,161,154,151,140,129,113,87,51,23,14,17,33,59,91,127,154,165,165,160,163
An example of data that requires correction occurs when the values fall from 140 to 14 within 7 values. Finding this means that everything from 23 to 33 will need to be bumped up between 51 and 59.
Any ideas would be appreciated.
Task
Given a numpy or pytorch matrix, find the indices of cells that have values that are larger than a given threshold.
My implementation
#abs_cosine is the matrix
#sim_vec is the wanted
sim_vec = []
for m in range(abs_cosine.shape[0]):
for n in range(abs_cosine.shape[1]):
# exclude diagonal cells
if m != n and abs_cosine[m][n] >= threshold:
sim_vec.append((m, n))
Concerns
Speed. All other computations are built on Pytorch, using numpy is already a compromise, because it has moved computations from GPU to CPU. Pure python for loops will make the whole process even worse (for small data set already 5 times slower). I was wondering if we can move the whole computation to Numpy (or pytorch) without invoking any for loops?
An improvement I can think of (but got stuck...)
bool_cosine = abs_cosine > threshold
which returns a boolean matrix of True and False. But I cannot find a way to quick retrieve the indices of the True cells.
The following is for PyTorch (fully on GPU)
# abs_cosine should be a Tensor of shape (m, m)
mask = torch.ones(abs_cosine.size()[0])
mask = 1 - mask.diag()
sim_vec = torch.nonzero((abs_cosine >= threshold)*mask)
# sim_vec is a tensor of shape (?, 2) where the first column is the row index and second is the column index
The following works in numpy
mask = 1 - np.diag(np.ones(abs_cosine.shape[0]))
sim_vec = np.nonzero((abs_cosine >= 0.2)*mask)
# sim_vec is a 2-array tuple where the first array is the row index and the second array is column index
This is about twice as fast than np.where
import numba as nb
#nb.njit(fastmath=True)
def get_threshold(abs_cosine,threshold):
idx=0
sim_vec=np.empty((abs_cosine.shape[0]*abs_cosine.shape[1],2),dtype=np.uint32)
for m in range(abs_cosine.shape[0]):
for n in range(abs_cosine.shape[1]):
# exclude diagonal cells
if m != n and abs_cosine[m,n] >= threshold:
sim_vec[idx,0]=m
sim_vec[idx,1]=n
idx+=1
return sim_vec[0:idx,:]
The first call takes about 0.2s longer (compilation overhead). If the array is on the GPU, there may be also a way to do the whole computation on the GPU.
Nevertheless I am not really satisfied with the performance, since a simple boolean operation is about 5 times faster than the solution shown above and 10 times faster than np.where. If the order of the indices doesn't matter this problem can also be parallelized.
I've been trying to find a more efficient way to iterate through an image and split their properties on a threshold. In searching online and discussing with some programming friends they introduced me to the concept of vectorizing (particularly using numpy) a function. After much searching and trial and error, I can't seem to get the hang of it. Can some one give me a link, or suggestion how to make the following code more efficient?
Im = plt.imread(img)
Imarray = np.array(Im)
for line in Imarray:
for pixel in line:
if pixel <= 20000:
dim_sum += pixel
dim_counter += 1
if pixel > 20000:
bright_sum += pixel
bright_counter += 1
bright_mean = bright_sum/bright_counter
dim_mean = dim_sum/dim_counter
Basically, each pixel holds a brightness amount between 0 and 30000 and I'm trying to average all pixels below 20000 and above 20000 respectively. The best way I know how to do this is using for loops (which are slow in python) and search through each pixel with if statements.
NumPy supports and encourages vectorization through its arrays and ufuncs. In your case, you have as input image a NumPy array. So, those comparisons could be done in one-go/ vectorized manner to give us boolean arrays of the same shape as the input array. Those boolean arrays when used for indexing into the input arrays would select the valid elements from it. This is called boolean-indexing and forms a key feature in such a vectorized selection.
Finally, we use NumPy ufunc ndarray.mean that again operates in a vectorized fashion to give us the mean values of the selected elements.
Thus, to put all those into code, we would have -
bright_mean, dim_mean = Im[Im > 20000].mean(), Im[Im <= 20000].mean()
For this particular problem, from code-efficiency point of view, it would make more sense to perform the comparison once. The comparison would give us a boolean array, which could be used twice later on, once as it is and second time being inverted. Thus, alternatively we would have -
mask = Im > 20000
bright_mean, dim_mean = Im[mask].mean(), Im[~mask].mean()
I am new to Tkinter (and Python) and I would like to find the most efficient way to format RGB values into a string so it can be used with the PhotoImage.put() function.
Let's say I have a Numpy rank 3 array in which the RGB values are stored, the 3rd dimension having a length of 3 for red, green and blue respectively. The most intuitive way to proceed would be:
for i in range(0, n_pixels_x):
for j in range(0, n_pixels_y):
hexcode = "#%02x%02x%02x" % (array[i,j,0], array[i,j,1], array[i,j,2])
img.put(hexcode, (j,i))
Unfortunately, this is way too slow for large images.
As described in the PhotoImage Wiki, it is possible to pass one large string to put() so the function is called only once. Then, I need to efficiently convert my array into such a string, which should be formatted like this (for a 4x2 image):
"{#ff0000 #ff0000 #ff0000 #ff0000} {#ff0000 #ff0000 #ff0000 #ff0000}"
Again, this could easily be done with nested for loops, but I would like to avoid them for efficiency reasons. Is there any way to use join() in order to do what I want?
If needed, I can store the content of my array differently, the only constraint being that I should be able to modify the color values easily.
Edit: After working on this a bit, I found a way to format my values approximately 10 times faster than by using nested loops. Here is the commented piece of code:
# 1. Create RGB array
array = np.zeros((n_pixels_x*n_pixels_y, 3))
array = np.asarray(array, dtype = "uint32")
array[1,:] = [0, 100, 255]
# 2. Create a format string
fmt_str = "{" + " ".join(["#%06x"]*n_pixels_x) + "}"
fmt_str = " ".join([fmt_str]*n_pixels_y)
# 3. Convert RGB values to hex
array_hex = (array[:,0]<<16) + (array[:,1]<<8) + array[:,2]
# 4. Format array
img_str = fmt_str % tuple(array_hex)
For a 640x480 array, steps 3 and 4 take ~0.1s to execute on my laptop (evaluated with timeit.default_timer()). Using nested loops, it takes between 0.9s and 1.0s.
I would still like to reduce the computation time, but I'm not sure if any improvement is still possible at this point.
I was able to find another way to format my array, and this really seems to be the quickest solution. The solution is to simply use Image and ImageTk to generate an image object directly from the array:
array = np.zeros((height, width, 3), 'uint8')
imageObject = Image.fromarray(array)
img = ImageTk.PhotoImage(image = imageObject, mode = 'RGB'))
This takes approximately 0.02s to run, which is good enough for my needs, and there is no need to use the put() function.
I actually found this answer from another question: How do I convert a numpy array to (and display) an image?