I am a beginner in Python and I usually program in C.
So, I have a numpy 2D array. I do the mean of the (i,j),(i+1,j),(i,j+1) and (i+1,j+1) values and I sum this mean if it is above a chosen value.
This is my python code :
Z=np.array([[1,2,3,4,5],[6,7,8,9,10],[11,12,13,14,15]])
sum=0.
value=7.
for i in range(np.shape(Z)[0]-1):
for j in range(np.shape(Z)[1]-1):
a = (Z[i,j] + Z[i+1,j] + Z[i,j+1] + Z[i+1,j+1]) / 4
if (a>=value):
sum+=a
print (sum)
I know It does not sound very pythonic. How can I write it in pythonic way to speed up this code on a large 2D numpy array ?
Thanks for answer
I'd do it this way:
quads = Z[:-1,:-1] + Z[1:,:-1] + Z[:-1,1:] + Z[1:,1:]
sum = quads[quads >= value * 4].sum() / 4
The first line computes the entire (x-1,y-1) array of sums of 2x2 elements:
array([[16, 20, 24, 28],
[36, 40, 44, 48]])
The second line compares each of those 8 elements with value * 4, rather than dividing quads / 4 which would create another array of the same size unnecessarily. This lets us do a single scalar multiply and a scalar divide at the end, instead of an array divide. But you could also write it this way if you don't care about the optimization:
quads /= 4
sum = quads[quads >= value].sum()
Related
I want to merge two arrays in python 2.7 using 'for loop' given:
from array import *
ary_1 = array ('i',[11,12,13])
ary_2 = array ('i',[14,15,16])
ary_3 = array ('i')
should give the output on ary_3 ,so ary_3 will display like this in specific order:
ary_3 = array ('i',[11,12,13,14,15,16])
Here's my code so far:
from array import *
ary_1 = array ('i',[11,12,13])
ary_2 = array ('i',[14,15,16])
ary_3 = array ('i')
ary_len = len (ary_1) + len (ary_2)
for i in range (0,ary_len):
ary_3.append (ary_1 [i])
ary_3.append (ary_2 [i])
if len (ary_3) == len (ary_1) + len (ary_2):
print ary_3,
break
Then the output was:
array('i',[11,14,12,15,13,16])
Not in order actually, and also if I add a new integer on either ary_1 or ary_2, it gives "index out of range" error so I found out that ary_1 and ary_2 should have an equal amount of integer/s to prevent this error.
If you want to combine the arrays, you can use the built-in method .extend:
ary_1.extend(ary_2)
print ary_1 #array('i', [11, 12, 13, 14, 15, 16])
As SethMMorton points out in the comments, if you do not want to override your first array:
ary_3 = ary_1 + ary_2
print ary_3 #array('i', [11, 12, 13, 14, 15, 16])
You should use one of the approaches above, but for learning purposes
in your original for loop you are (incorrectly) interleaving the two arrays by doing
ary_3.append (ary_1 [i])
ary_3.append (ary_2 [i])
If you wanted to keep the for loop, it should look something like:
ary_1_len = len(ary_1)
for i in range (0,ary_len):
if i < ary_1_len:
ary_3.append (ary_1 [i])
else:
ary_3.append (ary_2 [i-ary_1_len])
if len (ary_3) == len (ary_1) + len (ary_2):
print ary_3
break
Such that you populate the third array with the first array, and then the second array.
I am trying to create a 3D image mat1 from the data given to me by an object. But I am getting the error for the last line: mat1[x,y,z] = mat[x,y,z] + (R**2/U**2)**pf1[l,m,beta]:
IndexError: too many indices for array
What could possible be the problem here?
Following is my code :
mat1 = np.zeros((1024,1024,360),dtype=np.int32)
k = 498
gamma = 0.00774267
R = 0.37
g = np.zeros(1024)
g[0:512] = np.linspace(0,1,512)
g[513:] = np.linspace(1,0,511)
pf = np.zeros((1024,1024,360))
pf1 = np.zeros((1024,1024,360))
for b in range(0,1023) :
for beta in range(0,359) :
for a in range(0,1023) :
pf[a,b,beta] = (R/(((R**2)+(a**2)+(b**2))**0.5))*mat[a,b,beta]
pf1[:,b,beta] = np.convolve(pf[:,b,beta],g,'same')
for x in range(0,1023) :
for y in range(0,1023) :
for z in range(0,359) :
for beta in range(0,359) :
a = R*((-x*0.005)*(sin(beta)) + (y*0.005)*(cos(beta)))/(R+ (x*0.005)*(cos(beta))+(y*0.005)*(sin(beta)))
b = z*R/(R+(x*0.005)*(cos(beta))+(y*0.005)*(sin(beta)))
U = R+(x*0.005)*(cos(beta))+(y*0.005)*(sin(beta))
l = math.trunc(a)
m = math.trunc(b)
if (0<=l<1024 and 0<=m<1024) :
mat1[x,y,z] = mat[x,y,z] + (R**2/U**2)**pf1[l,m,beta]
The line where you do the convolution:
pf1 = np.convolve(pf[:,b,beta],g)
generates a 1-dimensional array, and not 3-dimensional as your call in the last line: pf1[l,m,beta]
To solve this you can use:
pf1[:,b,beta] = np.convolve(pf[:,b,beta],g,'same')
and you also need to predefine pf1:
pf1 = np.zeros((1024,1024,360))
Note that the convolution of f*g (np.convole(f,g)) returns normally a length of |f|+|g|-1. If you however use np.convolve with the parameter 'same' it returns an array which has the maximum length of f or g (i.e. max(|f|,|g|)).
Edit:
Furthermore you have to be sure that the dimensions of the matrices and the indices you use are correct, for example:
You define mat1 = np.zeros((100,100,100),dtype=np.int32), thus a 100x100x100 matrix, but in the last line you do mat1[x,y,z] where the variables x, y and z clearly get out of these dimensions. In this case they get to the range of the mat matrix. Probably you have to change the dimensions of mat1 also to those:
mat1 = np.zeros((1024,1024,360),dtype=np.int32)
Also be sure that the last variable indices you calculate (l and m) are within the dimensions of pf1.
Edit 2: The range(a,b) function returns an array from a to b, but not including b. So instead of range(0,1023) for example, you should write range(0,1024) (or shorter: range(1024)).
Edit 3: To check if l or m exceed the dimensions you could add an error as soon as they do:
l = math.trunc(a)
if l>=1024:
print 'l exceeded bounds: ',l
m = math.trunc(b)
if m>=1024:
print 'm exceeded bounds: ',m
Edit 4: note that your your code, especially your last for will take a long time! Your last nested for results in 1024*1024*360*360=135895449600 iterations. With a small time estimation I did (calculating the running time of the code in your for loop) your code might take about 5 days to run.
A small easy optimization you could do is instead of calculating the sin and cos several times, create a variable storing the value:
sinbeta = sin(beta)
cosbeta = cos(beta)
but it will probably still take several days. You might want to check how to optimize your calculations or calculate it with a C program for example.
My problem is about array manipulation in python. Want I want to do is to analyze a multiband raster with python and output another raster depending on the results.
Real case example: I have a raster with 3 bands and I want to see if the different band values for the same pixel are in increasing or decreasing order (from band 1 to band 3). If the values are in increasing order, a new value, lets say "1" will be assigned to that particular pixel in a new output raster (or can be also a new band in the same raster). If the values are in decreasing order, the value "2" will be assigned. I have defined the functions and read the array values but I m stuck at recursively build lists for every pixels, and assign new values for a new raster.
I will post the code that I have until now:
import arcpy
import numpy as np
arcpy.env.workspace = r"\\workspace"
def decrease(A):
return all(A[i] >= A[i+1] for i in range(len(A)-1)) # function to check if a list has decreasing values
def increase(A):
return all(A[i] <= A[i+1] for i in range(len(A)-1)) # function to check if a list has increasing values
def no0(A,val):
return[value for value in A if value !=val] # function to check if a list has 0 values, and if so, strip them
myArray = arcpy.RasterToNumpyArray(r"raster_3bands.img") # my 3 bands raster
print myArray.shape # to see how many dimensions the array has
for z in range(3): # where z is the number of bands, in my case 3
for y in range(6): #the number of columns, i.e. the vertical axis, in my case 6
for x in range(6): #the number of rows,i.e. the horizontal axis, in my case 6.
a = myArray[z,y,x] # get all the values for the 3 bands
print "The value for axes: ", z,y,x, " is: ",a
What I am missing:
1. the way to build for every pixel a list which will store the three correspondent band values, so that I can later run the Decrease and Increase functions on those lists
2. a way to assign new values, pixel by pixel and to store that array as a new raster.
Many thanks for your patience to read this,
Bogdan
So, here is the code for a 3 bands raster:
import arcpy, os
import numpy as np
myArray = arcpy.RasterToNumPyArray(r"\\nas2\home\bpalade\Downloads\test.img")
nbands = 3
print "Array dimensions are: ", myArray.shape
print "\n"
print "The array is: "
print "\n"
print myArray
increasing_pixels = np.product([(myArray[i] <= myArray[i+1]) for i in range(nbands-1)],axis=0)
decreasing_pixels = np.product([(myArray[i] >= myArray[i+1]) for i in range(nbands-1)],axis=0)
new_band = np.zeros_like(myArray[0])
new_band[increasing_pixels] = 1
new_band[decreasing_pixels] = 2
print "\n"
print "The new array is: "
print "\n"
print new_band
The result is attached as jpeg
I do not see the jpeg, so I copy/paste from my results window:
Array dimensions are: (3L, 4L, 4L)
The array is:
[[[60 62 62 60]
[64 64 63 60]
[62 62 58 55]
[59 57 54 50]]
[[53 55 55 55]
[57 57 56 55]
[55 55 51 50]
[52 50 47 45]]
[[35 37 37 36]
[39 39 38 36]
[37 37 33 31]
[34 32 29 26]]]
The new array is:
[[1 1 1 1]
[2 2 2 2]
[0 0 0 0]
[0 0 0 0]]
>>>
The first think to think about here is the structure of the code. When using numpy (and especially in a case like this) it is rare that you need to apply loops. You can simply apply vectorized operations along the axes.
In this case you can use array broadcasting to find which sets of pixels are increasing and decreasing.
increasing_pixels = (myArray[0] <= myArray[1]) * (myArray[1] <= myArray[2])
This will give you a boolean array where True pixels are those which increase across the three bands. The same can be done with decreasing pixels (this time True implies pixels that decrease across the band):
decreasing_pixels = (myArray[0] >= myArray[1]) * (myArray[1] >= myArray[2])
These arrays can act as a mask for your raster, allowing you to identify the pixels that you wish to change value. Below is an example of creating a new band based on the increasing or decreasing values of pixels across the original three bands.
new_band = np.zeros_like(myArray[0])
new_band[increasing_pixels==True] = 1
new_band[decreasing_pixels==True] = 2
EDIT:
For an arbitrary number of bands, the above comparisons can be replaced by:
increasing_pixels = np.product([(myArray[i] <= myArray[i+1]) for i in range(nbands-1)],axis=0)
decreasing_pixels = np.product([(myArray[i] >= myArray[i+1]) for i in range(nbands-1)],axis=0)
I am working on performing image processing using Numpy, specifically a running standard deviation stretch. This reads in X number of columns, finds the Std. and performs a percentage linear stretch. It then iterates to the next "group" of columns and performs the same operations. The input image is a 1GB, 32-bit, single band raster which is taking quite a long time to process (hours). Below is the code.
I realize that I have 3 nested for loops which is, presumably where the bottleneck is occurring. If I process the image in "boxes", that is to say loading an array that is [500,500] and iterating through the image processing time is quite short. Unfortunately, camera error requires that I iterate in extremely long strips (52,000 x 4) (y,x) to avoid banding.
Any suggestions on speeding this up would be appreciated:
def box(dataset, outdataset, sampleSize, n):
quiet = 0
sample = sampleSize
#iterate over all of the bands
for j in xrange(1, dataset.RasterCount + 1): #1 based counter
band = dataset.GetRasterBand(j)
NDV = band.GetNoDataValue()
print "Processing band: " + str(j)
#define the interval at which blocks are created
intervalY = int(band.YSize/1)
intervalX = int(band.XSize/2000) #to be changed to sampleSize when working
#iterate through the rows
scanBlockCounter = 0
for i in xrange(0,band.YSize,intervalY):
#If the next i is going to fail due to the edge of the image/array
if i + (intervalY*2) < band.YSize:
numberRows = intervalY
else:
numberRows = band.YSize - i
for h in xrange(0,band.XSize, intervalX):
if h + (intervalX*2) < band.XSize:
numberColumns = intervalX
else:
numberColumns = band.XSize - h
scanBlock = band.ReadAsArray(h,i,numberColumns, numberRows).astype(numpy.float)
standardDeviation = numpy.std(scanBlock)
mean = numpy.mean(scanBlock)
newMin = mean - (standardDeviation * n)
newMax = mean + (standardDeviation * n)
outputBlock = ((scanBlock - newMin)/(newMax-newMin))*255
outRaster = outdataset.GetRasterBand(j).WriteArray(outputBlock,h,i)#array, xOffset, yOffset
scanBlockCounter = scanBlockCounter + 1
#print str(scanBlockCounter) + ": " + str(scanBlock.shape) + str(h)+ ", " + str(intervalX)
if numberColumns == band.XSize - h:
break
#update progress line
if not quiet:
gdal.TermProgress_nocb( (float(h+1) / band.YSize) )
Here is an update:
Without using the profile module, as I did not want to start wrapping small sections of the code into functions I used a mix of print and exit statements to get a really rough idea about which lines were taking the most time. Luckily (and I do understand how lucky I was) one line was dragging everything down.
outRaster = outdataset.GetRasterBand(j).WriteArray(outputBlock,h,i)#array, xOffset, yOffset
It appears that GDAL is quite inefficient when opening the output file and writing out the array. With this in mind I decided to add my modified arrays "outBlock" to a python list, then write out chunks. Here is the segment that I changed:
The outputBlock was just modified ...
#Add the array to a list (tuple)
outputArrayList.append(outputBlock)
#Check the interval counter and if it is "time" write out the array
if len(outputArrayList) >= (intervalX * writeSize) or finisher == 1:
#Convert the tuple to a numpy array. Here we horizontally stack the tuple of arrays.
stacked = numpy.hstack(outputArrayList)
#Write out the array
outRaster = outdataset.GetRasterBand(j).WriteArray(stacked,xOffset,i)#array, xOffset, yOffset
xOffset = xOffset + (intervalX*(intervalX * writeSize))
#Cleanup to conserve memory
outputArrayList = list()
stacked = None
finisher=0
Finisher is simply a flag that handles the edges. It took a bit of time to figure out how to build an array from the list. In that, using numpy.array was creating a 3-d array (anyone care to explain why?) and write array requires a 2d array. Total processing time is now varying from just under 2 minutes to 5 minutes. Any idea why the range of times might exist?
Many thanks to everyone who posted! The next step is to really get into Numpy and learn about vectorization for additional optimization.
One way to speed up operations over numpy data is to use vectorize. Essentially, vectorize takes a function f and creates a new function g that maps f over an array a. g is then called like so: g(a).
>>> sqrt_vec = numpy.vectorize(lambda x: x ** 0.5)
>>> sqrt_vec(numpy.arange(10))
array([ 0. , 1. , 1.41421356, 1.73205081, 2. ,
2.23606798, 2.44948974, 2.64575131, 2.82842712, 3. ])
Without having the data you're working with available, I can't say for certain whether this will help, but perhaps you can rewrite the above as a set of functions that can be vectorized. Perhaps in this case you could vectorize over an array of indices into ReadAsArray(h,i,numberColumns, numberRows). Here's an example of the potential benefit:
>>> print setup1
import numpy
sqrt_vec = numpy.vectorize(lambda x: x ** 0.5)
>>> print setup2
import numpy
def sqrt_vec(a):
r = numpy.zeros(len(a))
for i in xrange(len(a)):
r[i] = a[i] ** 0.5
return r
>>> timeit.timeit(stmt='a = sqrt_vec(numpy.arange(1000000))', setup=setup1, number=1)
0.30318188667297363
>>> timeit.timeit(stmt='a = sqrt_vec(numpy.arange(1000000))', setup=setup2, number=1)
4.5400981903076172
A 15x speedup! Note also that numpy slicing handles the edges of ndarrays elegantly:
>>> a = numpy.arange(25).reshape((5, 5))
>>> a[3:7, 3:7]
array([[18, 19],
[23, 24]])
So if you could get your ReadAsArray data into an ndarray you wouldn't have to do any edge-checking shenanigans.
Regarding your question about reshaping -- reshaping doesn't fundamentally alter the data at all. It just changes the "strides" by which numpy indices the data. When you call the reshape method, the value returned is a new view into the data; the data isn't copied or altered at all, nor is the old view with the old stride information.
>>> a = numpy.arange(25)
>>> b = a.reshape((5, 5))
>>> a
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24])
>>> b
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
>>> a[5]
5
>>> b[1][0]
5
>>> a[5] = 4792
>>> b[1][0]
4792
>>> a.strides
(8,)
>>> b.strides
(40, 8)
Answered as requested.
If you are IO bound, you should chunk your reads/writes. Try dumping ~500 MB of data to an ndarray, process it all, write it out and then grab the next ~500 MB. Make sure to reuse the ndarray.
Without trying to completely understand exactly what you are doing, I notice that you aren't using any numpy slices or array broadcasting, both of which may speed up your code, or, at the very least, make it more readable. My apologies if these aren't germane to your problem.
I have two matrices. Both are filled with zeros and ones. One is a big one (3000 x 2000 elements), and the other is smaller ( 20 x 20 ) elements. I am doing something like:
newMatrix = (size of bigMatrix), filled with zeros
l = (a constant)
for y in xrange(0, len(bigMatrix[0])):
for x in xrange(0, len(bigMatrix)):
for b in xrange(0, len(smallMatrix[0])):
for a in xrange(0, len(smallMatrix)):
if (bigMatrix[x, y] == smallMatrix[x + a - l, y + b - l]):
newMatrix[x, y] = 1
Which is being painfully slow. Am I doing anything wrong? Is there a smart way to make this work faster?
edit: Basically I am, for each (x,y) in the big matrix, checking all the pixels of both big matrix and the small matrix around (x,y) to see if they are 1. If they are 1, then I set that value on newMatrix. I am doing a sort of collision detection.
I can think of a couple of optimisations there -
As you are using 4 nested python "for" statements, you are about as slow as you can be.
I can't figure out exactly what you are looking for -
but for one thing, if your big matrix "1"s density is low, you can certainly use python's "any" function on bigMtarix's slices to quickly check if there are any set elements there -- you could get a several-fold speed increase there:
step = len(smallMatrix[0])
for y in xrange(0, len(bigMatrix[0], step)):
for x in xrange(0, len(bigMatrix), step):
if not any(bigMatrix[x: x+step, y: y + step]):
continue
(...)
At this point, if still need to interact on each element, you do another pair of indexes to walk each position inside the step - but I think you got the idea.
Apart from using inner Numeric operations like this "any" usage, you could certainly add some control flow code to break-off the (b,a) loop when the first matching pixel is found.
(Like, inserting a "break" statement inside your last "if" and another if..break pair for the "b" loop.
I really can't figure out exactly what your intent is - so I can't give you more specifc code.
Your example code makes no sense, but the description of your problem sounds like you are trying to do a 2d convolution of a small bitarray over the big bitarray. There's a convolve2d function in scipy.signal package that does exactly this. Just do convolve2d(bigMatrix, smallMatrix) to get the result. Unfortunately the scipy implementation doesn't have a special case for boolean arrays so the full convolution is rather slow. Here's a function that takes advantage of the fact that the arrays contain only ones and zeroes:
import numpy as np
def sparse_convolve_of_bools(a, b):
if a.size < b.size:
a, b = b, a
offsets = zip(*np.nonzero(b))
n = len(offsets)
dtype = np.byte if n < 128 else np.short if n < 32768 else np.int
result = np.zeros(np.array(a.shape) + b.shape - (1,1), dtype=dtype)
for o in offsets:
result[o[0]:o[0] + a.shape[0], o[1]:o[1] + a.shape[1]] += a
return result
On my machine it runs in less than 9 seconds for a 3000x2000 by 20x20 convolution. The running time depends on the number of ones in the smaller array, being 20ms per each nonzero element.
If your bits are really packed 8 per byte / 32 per int,
and you can reduce your smallMatrix to 20x16,
then try the following, here for a single row.
(newMatrix[x, y] = 1 when any bit of the 20x16 around x,y is 1 ??
What are you really looking for ?)
python -m timeit -s '
""" slide 16-bit mask across 32-bit pairs bits[j], bits[j+1] """
import numpy as np
bits = np.zeros( 2000 // 16, np.uint16 ) # 2000 bits
bits[::8] = 1
mask = 32+16
nhit = 16 * [0]
def hit16( bits, mask, nhit ):
"""
slide 16-bit mask across 32-bit pairs bits[j], bits[j+1]
bits: long np.array( uint16 )
mask: 16 bits, int
out: nhit[j] += 1 where pair & mask != 0
"""
left = bits[0]
for b in bits[1:]:
pair = (left << 16) | b
if pair: # np idiom for non-0 words ?
m = mask
for j in range(16):
if pair & m:
nhit[j] += 1
# hitposition = jb*16 + j
m <<= 1
left = b
# if any(nhit): print "hit16:", nhit
' \
'
hit16( bits, mask, nhit )
'
# 15 msec per loop, bits[::4] = 1
# 11 msec per loop, bits[::8] = 1
# mac g4 ppc