How to optimize numpy mean window - python

I'm trying to get a new raster making looping through all array and using a search window 25x25 pixels. I wonder if you know a better way to do this because it takes too much time with my approach.
import sys
import os
import numpy as np
import math
from osgeo import gdal, osr, gdal_array, gdalnumeric
from osgeo.gdalnumeric import *
numpy.warnings.filterwarnings('ignore')
def mean_neighbors(M,x,y,w=1):
l = []
for i in range(max(0,x-w), x+(w+1)):
for j in range(max(0,y-w), y+(w+1)):
try:
t = M[i][j]
l.append(t)
except IndexError:
pass
return np.mean(l)
raster_file = gdal.Open('image.tif', gdal.GA_ReadOnly)
rst = gdalnumeric.BandReadAsArray(raster_file.GetRasterBand(1))
cob = np.zeros(rst.shape)
for i in range(rst.shape[0]):
for j in range(rst.shape[1]):
cob[i][j] = mean_neighbors(rst, i, j, 25) # want to optimize this function

Moving window mean is a very common function that you should not need to write yourself. Here are two fast implementations you can use:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.window.Rolling.mean.html
https://github.com/kwgoodman/bottleneck (move_mean)

Related

How can I generate in python an array like this?

I try to figure it out how can I implement a function in python to generate arrays like this (please ignore the fact that is the same number):
[[0101111010, 0101111010,0101111010, 0101111010,0101111010,0101111010,0101111010],
[0101111010,0101111010,0101111010,0101111010,0101111010,0101111010,0101111010]]
I write this code but I don't know if it's the best idea:
import numpy as np
import random
sol_per_pop = 20
num_genes = 10
pop_size = (sol_per_pop,num_genes)
population = np.random.randint(2, size=pop_size)
print(population)
I don't want to use string. I want to find the best solution. Thank you!
I don't really see what this would be useful for. But it might still be fun.
import random
int("{:b}".format(random.randint(0,1<<64)))
Or:
import random
r = random.randint(0,1<<64)
sum(10**i*((r>>i)&1) for i in range(64))
Or, if we must use numpy:
import numpy as np
significance = 10**np.cumsum(np.ones(64))
np.sum(significance*np.random.default_rng().integers(0, 2, 64))
Yet another idea:
import numpy as np
rng = np.random.default_rng()
significance = 10**np.cumsum(np.ones(64))
result = np.zeros(64)
result[nrng.choice(range(64), rng.integers(0, 64))] = 1
np.sum(significance*result)
As you can see, there are many approaches to solve the same problem.

Python: ValueError: setting an array element with a sequence. Error while solving simple ode

I have written a simple python code to simulate adaptive fuzzy sliding mode control. My problem is no matter what I do, I get this error. I tried converting list to arrays etc. But to no effect. Kindly help me iron out this issue. What I am trying to do, is, I have created a function that returns 53 elements and then the ode method of scipy integrates it.
Here is my code. PFA-
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import ode
from numpy import exp,sin,cos,concatenate,transpose,pi,tanh
from mpl_toolkits.mplot3d import Axes3D
c,eta,k,gamma1,gamma2,gamma3,gamma4,gamma5,gamma21,gamma22,gamma23,gamma24,gamma25=2,1.5,2,1,2,3,4,5,6,7,8,9,10
x=np.linspace(0,10,100)
xdot=np.linspace(0,10,100)
def member(x,z):
return exp(-((x+z)/(pi/24))**2)
m=[]
b=[pi/6,pi/12,0,-pi/12,-pi/6]
def f(t,Y):
x1,x2,x3,eta1,eta2,eta3,eta4,eta5,eta6,eta7,eta8,eta9,eta10,eta11,eta12,eta13,eta14,eta15,eta16,eta17,eta18,eta19,eta20,eta21,eta22,eta23,eta24,eta25,geta1,geta2,geta3,geta4,geta5,geta6,geta7,geta8,geta9,geta10,geta11,geta12,geta13,geta14,geta15,geta16,geta17,geta18,geta19,geta20,geta21,geta22,geta23,geta24,geta25=Y
eta=np.array([eta1,eta2,eta3,eta4,eta5,eta6,eta7,eta8,eta9,eta10,eta11,eta12,eta13,eta14,eta15,eta16,eta17,eta18,eta19,eta20,eta21,eta22,eta23,eta24,eta25])
geta=np.array([geta1,geta2,geta3,geta4,geta5,geta6,geta7,geta8,geta9,geta10,geta11,geta12,geta13,geta14,geta15,geta16,geta17,geta18,geta19,geta20,geta21,geta22,geta23,geta24,geta25])
m,z=[],[]
for i in range(5):
for j in range(5):
m.append(member(x1,b[i])*member(x2,b[j]))
for i in range(25):
z.append(m[i]/sum(m))
z=np.array(z)
e,de=x1-sin(t),x2-cos(t)
s=de+c**2*x3+2*c*e
f=np.dot(eta,z)
g=np.dot(geta,z)
u=(1./g)*(-f-c**2*e-2*c*de-sin(t)-eta*tanh(s)-k*s)
x1dot=x2
x2dot=f+g*u
x3dot=e
eta1dot,eta2dot,eta3dot,eta4dot,eta5dot,eta6dot,eta7dot,eta8dot,eta9dot,eta10dot,eta11dot,eta12dot,eta13dot,eta14dot,eta15dot,eta16dot,eta17dot,eta18dot,eta19dot,eta20dot,eta21dot,eta22dot,eta23dot,eta24dot,eta25dot,geta1dot,geta2dot,geta3dot,geta4dot,geta5dot,geta6dot,geta7dot,geta8dot,geta9dot,geta10dot,geta11dot,geta12dot,geta13dot,geta14dot,geta15dot,geta16dot,geta17dot,geta18dot,geta19dot,geta20dot,geta21dot,geta22dot,geta23dot,geta24dot,geta25dot=-2*s*z[0],-2*s*z[1],-2*s*z[2],-2*s*z[3],-2*s*z[4],-2*s*z[5],-2*s*z[6],-2*s*z[7],-2*s*z[8],-2*s*z[9],-2*s*z[10],-2*s*z[11],-2*s*z[12],-2*s*z[13],-2*s*z[14],-2*s*z[15],-2*s*z[16],-2*s*z[17],-2*s*z[18],-2*s*z[19],-2*s*z[20],-2*s*z[21],-2*s*z[22],-2*s*z[23],-2*s*z[24],-3*s*z[0]*u,-3*s*z[1]*u,-3*s*z[2]*u,-3*s*z[3]*u,-3*s*z[4]*u,-3*s*z[5]*u,-3*s*z[6]*u,-3*s*z[7]*u,-3*s*z[8]*u,-3*s*z[9]*u,-3*s*z[10]*u,-3*s*z[11]*u,-3*s*z[12]*u,-3*s*z[13]*u,-3*s*z[14]*u,-3*s*z[15]*u,-3*s*z[16]*u,-3*s*z[17]*u,-3*s*z[18]*u,-3*s*z[19]*u,-3*s*z[20]*u,-3*s*z[21]*u,-3*s*z[22]*u,-3*s*z[23]*u,-3*s*z[24]*u
return[x1dot,x2dot,x3dot,eta1dot,eta2dot,eta3dot,eta4dot,eta5dot,eta6dot,eta7dot,eta8dot,eta9dot,eta10dot,eta11dot,eta12dot,eta13dot,eta14dot,eta15dot,eta16dot,eta17dot,eta18dot,eta19dot,eta20dot,eta21dot,eta22dot,eta23dot,eta24dot,eta25dot,geta1dot,geta2dot,geta3dot,geta4dot,geta5dot,geta6dot,geta7dot,geta8dot,geta9dot,geta10dot,geta11dot,geta12dot,geta13dot,geta14dot,geta15dot,geta16dot,geta17dot,geta18dot,geta19dot,geta20dot,geta21dot,geta22dot,geta23dot,geta24dot,geta25dot]
Y,t=[],[]
for i in range(50):
Y.append(0.1)
#Y=np.array(Y)
y0=[0,0,0]
y0=y0+Y
t0=0
def int_amar():
t1,dt=10,.01
x=[]
for i in range(53):
x.append([])
#import pdb; pdb.set_trace()
odee=ode(f).set_integrator('dopri5',method='bdf',nsteps=1e6)
odee.set_initial_value(y0,t0)
while odee.successful() and odee.t<t1:
odee.integrate(odee.t+dt)
for i in range(53):
x[i].append(odee.y[i])
t.append(odee.t)
#for i in range(53):
# x[i]=np.array(x[i])
print(x1.shape)
return x
def main():
fun=int_amar()
Z,f,g,m=[[],[],[],[],[]],[],[],[]
for i in range(5):
for j in range(5):
m.append(member(fun[0],b[i])*member(fun[1],b[j]))
for i in range(25):
Z[i].append(m[i]/sum(m))
zetta1,zetta2,zetta3,zetta4,zetta5,zetta6,zetta7,zetta8,zetta9,zetta10,zetta11,zetta12,zetta13,zetta14,zetta15,zetta16,zetta17,zetta18,zetta19,zetta20,zetta21,zetta22,zetta23,zetta24,zetta25=Z[0],Z[1],Z[2],Z[3],Z[4],Z[5],Z[6],Z[7],Z[8],Z[9],Z[10],Z[11],Z[12],Z[13],Z[14],Z[15],Z[16],Z[17],Z[18],Z[19],Z[20],Z[21],Z[22],Z[23],Z[24],Z[25]
e=fun[0]-sin(t)
s=fun[1]+c**2*fun[2]+2*c*e
for i in range(len(fun[2])):
f.append(np.dot(np.array([[fun[3][i],fun[4][i],fun[5][i],fun[6][i],fun[7][i],fun[8][i],fun[9][i],fun[10][i],fun[11][i],fun[12][i],fun[13][i],fun[14][i],fun[15][i],fun[16][i],fun[17][i],fun[18][i],fun[19][i],fun[20][i],fun[21][i],fun[22][i],fun[23][i],fun[24][i],fun[25][i],fun[26][i],fun[27][i]]]),np.array([[zetta1[i]],[zetta2[i]],[zetta3[i]],[zetta4[i]],[zetta5[i]],[zetta6[i]],[zetta7[i]],[zetta8[i]],[zetta9[i]],[zetta10[i]],[zetta11[i]],[zetta12[i]],[zetta13[i]],[zetta14[i]],[zetta15[i]],[zetta16[i]],[zetta17[i]],[zetta18[i]],[zetta19[i]],[zetta20[i]],[zetta21[i]],[zetta22[i]],[zetta23[i]],[zetta24[i]],[zetta25[i]]])))
g.append(np.dot(np.array([[fun[28][i],fun[29][i],fun[30][i],fun[31][i],fun[32][i],fun[33][i],fun[34][i],fun[35][i],fun[36][i],fun[37][i],fun[38][i],fun[39][i],fun[40][i],fun[41][i],fun[42][i],fun[43][i],fun[44][i],fun[45][i],fun[46][i],fun[47][i],fun[48][i],fun[49][i],fun[50][i],fun[51][i],fun[52][i]]]),np.array([[zetta1[i]],[zetta2[i]],[zetta3[i]],[zetta4[i]],[zetta5[i]],[zetta6[i]],[zetta7[i]],[zetta8[i]],[zetta9[i]],[zetta10[i]],[zetta11[i]],[zetta12[i]],[zetta13[i]],[zetta14[i]],[zetta15[i]],[zetta16[i]],[zetta17[i]],[zetta18[i]],[zetta19[i]],[zetta20[i]],[zetta21[i]],[zetta22[i]],[zetta23[i]],[zetta24[i]],[zetta25[i]]])))
f,g=np.array(f),np.array(g)
u=(1./g)*(-f-c*e-sin(t)-eta*tanh(s)-k*s)
print(u.shape)
for i in range(12):
plt.subplot(3,4,i+1)
plt.plot(t,fun[i])
#plt.figure(2)
#plt.plot(t,u)
plt.show()
if __name__=='__main__':
main()
The issue is on line 30:
u=(1./g)*(-f-c**2*e-2*c*de-sin(t)-eta*tanh(s)-k*s)
eta is an array but the rest of the terms are scalars. This causes u to be an array and the resulting return list has a mix of scalars and arrays.
You probably mistyped the equation here.

customizing np.fft.fft function in python

I wish to perform a fourier transform of the function 'stress' from 0 to infinity and extract the real and imaginary parts. I have the following code that does it using a numerical integration technique:
import numpy as np
from scipy.integrate import trapz
import fileinput
import sys,string
window = 200000 # length of the array I wish to transform (number of data points)
time = np.linspace(1,window,window)
freq = np.logspace(-5,2,window)
output = [0]*len(freq)
for index,f in enumerate(freq):
visco = trapz(stress*np.exp(-1j*f*t),t)
soln = visco*(1j*f)
output[index] = soln
print 'f storage loss'
for i in range(len(freq)):
print freq[i],output[i].real,output[i].imag
This gives me a nice transformation of my input data.
Now I have an array of size 2x10^6, and using the above technique is not feasible(computation time scales as O(N^2)), so I have turned to the inbuilt fft function in numpy.
There aren't too many arguments that you can specify to change this function, and so I'm finding it difficult to customize it to my needs.
So far I have
import numpy as np
import fileinput
import sys, string
np.set_printoptions(threshold='nan')
N = len(stress)
fvi = np.fft.fft(stress,n=N)
gprime = fvi.real
gdoubleprime = fvi.imag
for i in range(len(stress)):
print gprime[i], gdoubleprime[i]
And it's not giving me accurate results.
The DFT in python is of the form A_k = summation(a_m * exp(-2*piimk/n)) where the summation is from m = 0 to m = n-1 (http://docs.scipy.org/doc/numpy-1.10.1/reference/routines.fft.html). How can I change it to the form that I have mentioned in my first code, i.e. exp(-1jfreq*t) (freq is the frequency and t is the time which have already been predefined)? Or is there a post processing of the data that I have to do?
Thanks in advance for all your help.

Scipy method matching python

I want to make this code faster, as it takes ~4 miliseconds for a 1000x1000 image with a window size of 10x10.
import numpy
import scipy.misc
from matplotlib import pyplot as plt
import time
def corr(a, b):
'''finds the correlation of 2 intensities'''
return (sum2(a,b)/(sum2(a,a)*sum2(b,b))**0.5)
def sum2(a,b):
s = 0
for x in range(len(a)):
s += a[x]*b[x]
return s
##the commented code displays the images
##plt.ion()
def find_same(img1,img2,startx,width,starty,hight):
'''inputs 2 filenames, startx, width of search window, and hight.'''
crop_img = img1[starty:(starty+hight),startx:(startx+width)]
plt.imshow(crop_img,interpolation='nearest')
plt.draw()
a = []
for x in numpy.nditer(crop_img): #converting image to array of intesities
a.append(float(x))
mcfinder = []
for x in range(img2.shape[1]-width):
finder = img2[starty:(starty+hight),x:(x+width)]
b = []
for y in numpy.nditer(finder):
b.append(float(y))
correl = corr(a,b) #find correlation
mcfinder.append(correl)
maxim = max(mcfinder)
place = mcfinder.index(maxim)
finder = img2[starty:(starty+hight),place:(place+width)]
## plt.imshow(finder,interpolation='nearest')
## plt.draw()
## time.sleep(1)
## plt.close()
return maxim,place
img1 = scipy.misc.imread('me1.bmp')
img2 = scipy.misc.imread('me2.bmp')
starttime = time.clock()
print find_same(img1,img2,210,40,200,40)
endtime = time.clock()
print endtime-starttime
Are there any ways to make this faster? Or am I doing this fundamentally wrong?
Please let me know. To run this you need matplotlib, scipy, and numpy.
[I lack the reputation to post this as a comment]
As mentioned in #cel's comment, you should vectorize your code by using only numpy operations instead of loops over lists.
It seems you are trying to do some template matching, did you have a look at the example for skimage.feature.match_template() from the scikit-image documentation? scikit-image also provides windowed views (skimage.util.view-as-windows()) of a numpy array which is very handy, when you are analyzing a numpy array block-by-block.
If you don't want to add another dependency you should use Scipy's special functions to compute the correlation for you, e.g. scipy.ndimage.filters.correlate() (also have a look at the other functions in scipy.ndimage.filter).

While loops for lists?

I'm pretty new to programming and I have a quick question. I am trying to make a Gaussian function for a range of stars. However i want the size of undercurve be at 100 for all the stars. I was thinking of doing a while loop saying that while the total length of undercurve be 100. However, I get an error and I'm guessing it has something to do with it being a list. I'm showing you guys my code to see if you can help me out here. Thanks!
I get a syntax error: can't assign to call function
import numpy
import random
import math
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
import scipy
from scipy import stats
from math import sqrt
from numpy import zeros
from numpy import numarray
variance = input("Input variance of the star:")
mean = input("Input mean of the star:")
space=numpy.linspace(-4,1,1000)
sigma = sqrt(variance)
Max = max(mlab.normpdf(space,mean,sigma))
normalized = (mlab.normpdf(space,mean,sigma))/Max
def random_y_pt():
return random.uniform(0,1)
def random_x_pt():
return random.uniform(-4,1)
import random
def undercurve(size):
result = []
for i in range(0,size):
y = random_y_pt()
x = random_x_pt()
if y < scipy.stats.norm(scale=variance,loc=mean).pdf(x):
result.append((x))
return result
size = 1
while len(undercurve(size)) < 100:
undercurve(size) = undercurve(1)+undercurve(size)
print undercurve(size)
plt.hist(undercurve(size),bins=20)
plt.show()
If your error is something like SyntaxError: can't assign to function call then that's because of your line
undercurve(size) = undercurve(1)+undercurve(size)
Which is trying to set the output of the right-hand side as the value of undercurve(size), which you cannot do.
It sounds like you actually want to see just the first 100 items in the list returned by undercurve(size). For that, use
undercurve(size)[:100]

Categories