I've set up numpy.seterr as follows:
np.seterr(invalid='raise', over ='raise', under='raise')
And I'm getting the following error:
c = beta[j,i] + oneminusbeta[j,i]
FloatingPointError: overflow encountered in double_scalars
I've checked what beta[j,i] and oneminusbeta[j,i] are at the point of crash, and these are their values:
beta: -131.340389182
oneminusbeta: 0.0
Please note, this line of addition (beta[j,i] + oneminusbeta[j,i]) has run for thousands of lines in a loop (that performs image classification) before crashing here at this point.
How can I deal with this? Is it necessary to change the type of the numpy arrays?
This is how I've initialized them:
beta = np.empty([m,n])
oneminusbeta = np.empty([m,n])
Is it possible to cast the individual values before adding them up? Rather than changing the entire array declarations? Or is this even a serious issue? Would it be safe to simply turn off the numpy.seterr configuration and let the calculations go ahead without raising the error?
Edit
Someone suggested below, and I suspected as well, that the values being added shouldn't cause an overflow. Then how can I find out where the overflow is really happening?
This is my code:
epthreshold = 709
enthreshold = -708
f.write("weights["+str(i)+", " + str(j)+"] = math.exp(beta: " +str(beta[j,i])+ " + oneminusbeta: " + str(oneminusbeta[j,i])+")\n" )
c = beta[j,i] + oneminusbeta[j,i]
weights[i,j] = math.exp(np.clip(c, enthreshold, epthreshold))
And when I check my log file, this is the line I get:
weights[5550, 13] = math.exp(beta: -131.340389182 + oneminusbeta: 0.0)
Edit 2
Here's the rest of my code, where variables n,m and H have already been initialized to integer values:
import numba
import numpy as np
import statsmodels.api as sm
weights = np.empty([n,m])
for curr_n in range(n):
for curr_m in range(m):
weights[curr_n,curr_m] = 1.0/(n)
beta = np.empty([m,n])
oneminusbeta = np.empty([m,n])
for curr_class in range(m):
for curr_sample in range(n):
beta[curr_class,curr_sample] = 1./m
epthreshold = 709 # positive exponential threshold
enthreshold = -708
for h in range(H):
print "Boosting round %d ... " % h
z = np.empty([n,m])
for j in range(m): # computing working responses and weights, Step 2(a)(i)
for i in range(no_samples):
i_class = y[i] #get the correct class for the current sample
if h == 0:
z[i,j] = (int(j==i_class) - beta[j,i])/((beta[j,i])*(1. - beta[j,i]))
weights[i,j] = beta[j,i]*(1. - beta[j,i])
else:
if j == i_class:
z[i,j] = math.exp(np.clip(-beta[j,i],enthreshold, epthreshold))
else:
z[i,j] = -math.exp(np.clip(oneminusbeta[j,i], enthreshold, epthreshold))
f.write("weights["+str(i)+", " + str(j)+"] = math.exp(beta: " +str(beta[j,i])+ " + oneminusbeta: " + str(oneminusbeta[j,i])+")\n" )
c = beta[j,i] + oneminusbeta[j,i]
weights[i,j] = math.exp(np.clip(c, enthreshold, epthreshold))
g_h = np.zeros([1,1])
j = 0
# Calculating regression coefficients per class
# building the parameters per j class
for y1_w in zip(z.T, weights.T):
y1, w = y1_w
temp_g = sm.WLS(y1, X, w).fit() # Step 2(a)(ii)
if np.allclose(g_h,0):
g_h = temp_g.params
else:
g_h = np.c_[g_h, temp_g.params]
j = j + 1
if np.allclose(g,0):
g = g_h
else:
g = g + g_h # Step(2)(a)(iii)
# now set g(x), function coefficients according to new formula, step (2)(b)
sum_g = g.sum(axis=1)
for j in range(m):
diff = (g[:,j] - ((1./m) * sum_g))
g[:,j] = ((m-1.)/m) * diff
g_per_round[h,:,j] = g[:,j]
#Now computing beta, Step 2(c)...."
Q = 0.
e = 0.
for j in range(m):
# Calculating beta and oneminusbeta for class j
aj = 0.0
for i in range(no_samples):
i_class = y[i]
X1 = X[i].reshape(1, no_features)
g1 = g[:,j].reshape(no_features, 1)
gc = g[:,i_class].reshape(no_features, 1)
dot = 1. + float(np.dot(X1, g1)) - float(np.dot(X1,gc))
aj = dot
sum_e = 0.
a_q = []
a_q.append(0.)
for j2 in range(m): # calculating sum of e's except for all j except where j=i_class
if j2 != i_class: # g based on j2, not necessarily g1?
g2 = g[:,j2].reshape(no_features, 1)
dot1 = 1. + float(np.dot(X1, g2)) - float(np.dot(X1,gc))
e2 = math.exp(np.clip(dot1,enthreshold, epthreshold))
sum_e = sum_e + e2
a_q.append(dot1)
if (int(j==i_class) == 1):
a_q_arr = np.array(a_q)
alpha = np.array(a_q_arr[1:])
Q = mylogsumexp(f,a_q_arr, 1, 0)
sumalpha = mylogsumexp(f,alpha, 1, 0)
beta[j,i] = -Q
oneminusbeta[j,i] = sumalpha - Q
else:
alpha = a_q
alpha = np.array(alpha[1:])
a_q_arr = np.array(a_q)
Q = mylogsumexp(f,a_q_arr, 0, aj)
sumalpha = log(math.exp(np.clip(Q, enthreshold, epthreshold)) - math.exp(np.clip(aj, enthreshold, epthreshold)))
beta[j,i] = aj - Q
oneminusbeta[j,i] = sumalpha - Q
and the function mylogsumexp is:
def mylogsumexp(f, a, is_class, maxaj, axis=None, b=None):
np.seterr(over="raise", under="raise", invalid="raise")
threshold = -sys.float_info.max
maxthreshold = sys.float_info.max
epthreshold = 709 # positive exponential threshold
enthreshold = -708
a = asarray(a)
if axis is None:
a = a.ravel()
else:
a = rollaxis(a, axis)
if is_class == 1:
a_max = a.max(axis=0)
else:
a_max = maxaj
#bnone = " none "
if b is not None:
a_max = maxaj
b = asarray(b)
if axis is None:
b = b.ravel()
else:
b = rollaxis(b, axis)
a = np.clip(a - a_max, enthreshold, epthreshold)
midout = np.sum(np.exp(a), axis=0)
midout = 1.0 + np.clip(midout - math.exp(a_max), threshold, maxthreshold)
out = np.log(midout)
else:
a = np.clip(a - a_max, enthreshold, epthreshold)
out = np.log(np.sum(np.exp(a)))
out += a_max
if out == float("inf"):
out = maxthreshold
if out == float("-inf"):
out = threshold
return out
Related
Good day to you.
From a previous model with cplex (MIP 1), I use M0 as a parameter.
Total_P = 8
M0 = np.array([[0,1,0,1,0,0,0,2]]).reshape(Total_P,1)
Then, I tried to develop a different model (MIP 2) with adjusted objectives and constraints from MIP2, however, the M0 now becomes a decision variable and use for several constraints (#constraint 38). So, I created a code as follows:
Total_P = 8
M0 = np.empty((Total_P,1), dtype = object)
for l in range(0,Total_P):
M0[l][0]= mdl.integer_var(name='M0'+ str(',') + str(l+1))
for l in range(0,Total_P):
if M0[l][0] is not None:
mdl.add_constraint((M0[1][0] + M0[5][0]) == 1)
mdl.add_constraint((M0[3][0] + M0[6][0]) == 1)
mdl.add_constraint((M0[0][0] + M0[2][0] + M0[4][0] + M0[7][0]) == 2)
After running the cplex with the same parameters input:
The result from MIP 1: obj_lambda=213
The result from MIP 2: obj_lamba= 205.0 and the M0 is [[0,1,1,0,0,0,1,1]]
The correct result of M0 from the MIP 2 should be [[0,1,0,1,0,0,0,2]] as in the M0 (parameter in MIP 1) and obj_lambda=213.
Here is the full code of MIP 1:
import cplex
from docplex.mp.model import Model
import numpy as np
mdl = Model(name='Scheduling MIP 1')
inf = cplex.infinity
bigM= 10000
Total_P = 8 # number of places
Total_T = 6 # number of transitions
r1 = 100 #processing time for PM1 (second)
r2 = 200 #processing time for PM2 (second)
v = 3 #robot moving time (second)
w = 5 #loading or unloading time (second)
M0 = np.array([[0,1,1,0,0,0,1,1]]).reshape(Total_P,1)
M0_temp = M0.reshape(1,Total_P)
M0_final = M0_temp.tolist()*Total_T
h = np.array([v+w,w+r1,v+w,w+r2,v+w,0,0,0]).reshape(Total_P,1)
#Parameter
AT =np.array([[1,-1,0,0,0,0],
[0,1,-1,0,0,0],
[0,0,1,-1,0,0],
[0,0,0,1,-1,0],
[0,0,0,0,1,-1],
[0,-1,1,0,0,0],
[0,0,0,-1,1,0],
[-1,1,-1,1,-1,1]])
AT_temp = AT.transpose()
places = np.array(['p1','p2','p3','p4','p5','p6','p7','p8'])
P_conflict = np.empty((1,Total_P), dtype = object)
P_zero = np.empty((1,Total_P), dtype = object)
#Define the place without conflict place
CP = np.count_nonzero(AT, axis=1, keepdims=True) #calculate the nonzero elements for each row
P_conflict = []
P_zero = []
for a in range(0, len(CP)):
if CP[a].item(0) > 2:
P_conflict.append(places[a])
else:
P_zero.append(places[a])
print('p_zero', np.shape(P_zero))
obj_lambda = [ ]
obj_lambda = mdl.continuous_var(lb = 0, ub=inf, name='obj_lambda')
x = np.array(mdl.continuous_var_list(Total_T, 0, inf, name='x')).reshape(Total_T,)
ind_x = np.where(AT[0] == 1)
print('ind_x', ind_x)
def get_index_value(input):
data = []
for l in range(len(P_zero)):
ind_x = np.where(AT[l] == input)
get_value = x[ind_x]
data.append(get_value)
return data
x_in = get_index_value(1)
x_out = get_index_value(-1)
#constraint 16
for l in range(0,len(P_zero)): #only for P non conflict
#if x_in is not None and x_out is not None and obj_lambda is not None:
mdl.add_constraint(x_out[l][0]-x_in[l][0] >= h[l][0] - M0[l][0]*obj_lambda)
#Decision Var
Z = np.empty((Total_T, Total_T), dtype = object)
for k in range(Total_T):
for i in range(Total_T):
Z[k][i] = mdl.binary_var(name='Z' + str(k+1) + str(',') + str(i+1))
storage_ZAT = []
for k in range(Total_T):
ZAT = np.matmul(Z[k].reshape(1,Total_T),AT_temp)
storage_ZAT.append(ZAT) #storage_ZAT = np.append(storage_ZAT, ZAT, axis=0)
ZAT_final = np.asarray(storage_ZAT).reshape(Total_T,Total_P)
M = np.empty((Total_T, Total_P), dtype = object)
for k in range(0,Total_T):
for l in range (0,Total_P):
if k == Total_T-1:
M[Total_T-1][l] = M0_final[0][l]
else:
M[k][l] = mdl.continuous_var(name='M' + str(k + 1) + str(',') + str(l + 1))
M_prev = np.empty((Total_T, Total_P), dtype = object)
if M is not None:
for k in range(0,Total_T):
for l in range (0,Total_P):
if k is not 0:
M_prev[k][l] = M[k-1][l]
else:
M_prev[0][l] = M0_final[0][l]
#Constraint 17
for k in range(Total_T):
for l in range(Total_P):
mdl.add_constraint(M[k][l] == M_prev[k][l] + ZAT_final[k][l])
#Constraint 18
mdl.add_constraints(mdl.sum(Z[k][i] for k in range(Total_T)) == 1 for i in range(Total_T))
# Constraint 19
mdl.add_constraints(mdl.sum(Z[k][i] for i in range(Total_T)) == 1 for k in range(Total_T))
# # Parameters
VW_temp = [[v + w]]
VW_temp = VW_temp*Total_T
VW = np.array(VW_temp) #eshape(Total_T,)
#Define S
S = np.array(mdl.continuous_var_list(Total_T, 0, inf, name='S')).reshape(Total_T,1)
#Constraint 20
for k in range(Total_T-1):
mdl.add_constraint(S[k][0] - S[k+1][0] <= -VW[k][0])
# # Constraint 21
mdl.add_constraint(S[Total_T-1][0] - (S[0][0] + obj_lambda) <=-VW[Total_T-1][0])
x_temp = x.reshape(Total_T,1)
print('x_temp',x_temp)
# Constraint 22
for k in range(Total_T):
for i in range(Total_T):
mdl.add_constraint(S[k][0] - x_temp[i][0] <= (1-Z[k][i])*bigM) #why x_temp? because it is the reshape of x
# Constraint 23
for k in range(Total_T):
for i in range(Total_T):
mdl.add_constraint(S[k][0] - x_temp[i][0] >= (Z[k][i]-1)*bigM)
mdl.minimize(obj_lambda)
mdl.print_information()
solver = mdl.solve() #(log_output=True)
if solver is not None:
mdl.print_solution()
miu = 1 / obj_lambda.solution_value
print('obj_lamba=', obj_lambda.solution_value)
print('miu =', miu)
else:
print("Solver is error")
Here is the code of MIP 2:
import cplex
from docplex.mp.model import Model
import numpy as np
mdl = Model(name='Scheduling MILP 2')
inf = cplex.infinity
bigM= 10000
Total_P = 8 # number of places
Total_T = 6 # number of transitions
r1 = 100 #processing time for PM1 (second)
r2 = 200 #processing time for PM2 (second)
v = 3 #robot moving time (second)
w = 5 #loading or unloading time (second)
h = np.array([v+w,w+r1,v+w,w+r2,v+w,0,0,0]).reshape(Total_P,1)
#Parameter
AT =np.array([[1,-1,0,0,0,0],
[0,1,-1,0,0,0],
[0,0,1,-1,0,0],
[0,0,0,1,-1,0],
[0,0,0,0,1,-1],
[0,-1,1,0,0,0],
[0,0,0,-1,1,0],
[-1,1,-1,1,-1,1]])
AT_temp = AT.transpose()
places = np.array(['p1','p2','p3','p4','p5','p6','p7','p8'])
P_conflict = np.empty((1,Total_P), dtype = object)
P_zero = np.empty((1,Total_P), dtype = object)
#Define the place without conflict place
CP = np.count_nonzero(AT, axis=1, keepdims=True) #calculate the nonzero elements for each row
P_conflict = []
P_zero = []
for a in range(0, len(CP)):
if CP[a].item(0) > 2:
P_conflict.append(places[a])
else:
P_zero.append(places[a])
print('p_zero', P_zero)
y = [ ] #miu
y = mdl.continuous_var(lb = 0, ub=inf, name='miu') #error: docplex.mp.utils.DOcplexException: Variable obj_lambda cannot be used as denominator of 1
x = np.array(mdl.continuous_var_list(Total_T, 0, inf, name='x')).reshape(Total_T,)
ind_x = np.where(AT[0] == 1)
def get_index_value(input):
data = []
for l in range(len(P_zero)):
ind_x = np.where(AT[l] == input)
get_value = x[ind_x]
data.append(get_value)
return data
x_in_hat = get_index_value(1)
x_out_hat = get_index_value(-1)
M0 = np.empty((Total_P,1), dtype = object)
for l in range(0,Total_P):
M0[l][0]= mdl.integer_var(name='M0'+ str(',') + str(l+1))
M0_temp= M0.reshape(1,Total_P)
M0_final = M0_temp.tolist()*Total_T
for k in range(Total_T):
for l in range(Total_P):
if M0_final[k][l] is not None:
mdl.add_constraint((M0_final[0][1] + M0_final[0][5]) == 1)
mdl.add_constraint((M0_final[0][3] + M0_final[0][6]) == 1)
mdl.add_constraint((M0_final[0][0] + M0_final[0][2] + M0_final[0][4] + M0_final[0][7]) == 2)
#constraint 30
for l in range(0,len(P_zero)): #only for P non conflict
mdl.add_constraint(x_out_hat[l][0]-x_in_hat[l][0] >= h[l][0]*y - M0[l][0])
#Decision Var
Z = np.empty((Total_T, Total_T), dtype = object)
for k in range(Total_T):
for i in range(Total_T):
# if k == 0 and i == 0:
# Z[0][0]=1
# else:
Z[k][i] = mdl.binary_var(name='Z' + str(k+1) + str(',') + str(i+1))
#
storage_ZAT = []
for k in range(Total_T):
ZAT = np.matmul(Z[k].reshape(1,Total_T),AT_temp)
storage_ZAT.append(ZAT) #storage_ZAT = np.append(storage_ZAT, ZAT, axis=0)
ZAT_final = np.asarray(storage_ZAT).reshape(Total_T,Total_P)
M = np.empty((Total_T, Total_P), dtype = object)
for k in range(0,Total_T):
for l in range (0,Total_P):
if k == Total_T-1:
M[Total_T-1][l] = M0_final[0][l]
else:
M[k][l] = mdl.continuous_var(name='M' + str(k + 1) + str(',') + str(l + 1))
M_prev = np.empty((Total_T, Total_P), dtype = object)
if M is not None:
for k in range(0,Total_T):
for l in range (0,Total_P):
if k is not 0:
M_prev[k][l] = M[k-1][l]
else:
M_prev[0][l] = M0_final[0][l]
#
#Constraint 31
for k in range(Total_T):
for l in range(Total_P):
mdl.add_constraint(M[k][l] == M_prev[k][l] + ZAT_final[k][l])
#
#Constraint 32
mdl.add_constraints(mdl.sum(Z[k][i] for k in range(Total_T)) == 1 for i in range(Total_T))
# Constraint 33
mdl.add_constraints(mdl.sum(Z[k][i] for i in range(Total_T)) == 1 for k in range(Total_T))
# #
# # # # Parameters
VW_temp = [[v + w]]
VW_temp = VW_temp*Total_T
VW = np.array(VW_temp) #eshape(Total_T,)
#
S_hat = np.array(mdl.continuous_var_list(Total_T, 0, inf, name='S_hat')).reshape(Total_T,1)
#Constraint 34
for k in range(0,Total_T-1):
if k < (Total_T-1):
mdl.add_constraint(S_hat[k][0] - S_hat[k+1][0] <= -VW[k][0]*y)
if k == (Total_T - 1):
mdl.add_constraint(S_hat[k][0] - (S_hat[0][0] + 1) <= -VW[Total_T - 1][0] * y)# Constraint 35
#
#
x_temp_hat = x.reshape(Total_T,1)
#
# Constraint 36
for k in range(Total_T):
for i in range(Total_T):
mdl.add_constraint((S_hat[k][0] - x_temp_hat[i][0]) <= (1-Z[k][i])*bigM) #why x_temp? because it is the reshape of x
# Constraint 37
for k in range(Total_T):
for i in range(Total_T):
mdl.add_constraint((S_hat[k][0] - x_temp_hat[i][0]) >= (Z[k][i]-1)*bigM)
mdl.maximize(y)
mdl.print_information()
solver = mdl.solve()
if solver is not None:
mdl.print_solution()
obj_lambda = 1/y.solution_value
print('miu =', y.solution_value)
print('obj_lamba=', obj_lambda)
print('M0 final=', M0_temp)
else:
print("Solver is error")
In this case, could anyone tell me: did I define the M0 as a decision variable and put them into the constraints correctly, please?
Thank you.
I am trying to implement a pso algorithm from Wikipedia https://en.wikipedia.org/wiki/Particle_swarm_optimization.
My problem is that when I am calling the cost function with a variable (Gbest), and then manually calling the cost function (with the Gbest data) I get a different output (cost) like the image bellow:
Code fault
I am new to python so thank you for any suggestions.
Here is the complete code:
import matplotlib.pyplot as plt
import numpy as np
from control.matlab import *
A = np.array([[0,0,1],[0,1,0],[1,2,-2]])
B = np.array( [[0],[1],[0]])
C = np.array([[0, 1,0]])
D = np.zeros([C.shape[0],B.shape[1]])
sys = ss(A,B,C,D)
sys_tf = tf(sys)
s = tf('s')
def cost(kp,ki):
global sys_tf, G, y, t, r
G = kp + ki/s
C = feedback(sys_tf*G, 1)
y, t = step(C, linspace(0,100))
r = np.ones(len(t))
return np.sum(y-r)**2
part = 100
ite = 10000
dim = 2
w = 0.001
wdamp = 0.99
phip = 0.9
phig = 0.1
blo, bup = -10,10
x = np.zeros([dim, part])
v = np.zeros([dim, part])
pbest = np.zeros([dim, part])
gbest = np.array([1000000,1000000])
for i in range(part):
for k in range(dim):
x[k][i] = pbest[k][i] = np.random.uniform(blo, bup)
v[k][i] = np.random.uniform(-np.abs(bup - blo), np.abs(bup - blo))
if cost(pbest[0][i], pbest[1][i]) < cost(gbest[0], gbest[1]):
gbest = np.array([pbest[0][i], pbest[1][i]])
for it in range(ite):
for i in range(part):
for k in range(dim):
rp = np.random.uniform(0,1)
rg = np.random.uniform(0,1)
v[k,:] = w*v[k,:] + phip*rp*(pbest[k,:] - x[k,:]) + phig*rg*(gbest[k] - x[k,:])
x[k,:] = x[k,:] + v[k,:]
w = w*wdamp
if cost(x[0][i], x[1][i]) < cost(pbest[0][i], pbest[1][i]):
pbest[:,i] = x[:,i]
if cost(pbest[0][i], pbest[1][i]) < cost(gbest[0], gbest[1]):
gbest = np.array([pbest[0][i], pbest[1][i]])
plt.plot(t, y, 'ro')
plt.plot(t, r, 'x')
plt.pause(0.005)
plt.title(gbest)
print([gbest, cost(gbest[0], gbest[1])])
I know this is not an ideal place for questions of this scope, but I'm not sure where else to ask this or how to break it down. I've been working on a function for the past couple weeks, that runs, but for it to be feasible for my purposes, I need to speed it up 200-300x.
I have an image array, where all pixels of similar color have been averaged and set to that average value. Then I have a 2D array of the same height and width, which labels each unique and non-contiguous feature of the image.
Using these I need to assess the size of each feature and its level of contrast to each of its neighbors. These values are used in an equation and if the output of that equation is below a certain threshold, that feature is merged with its most similar neighbor.
I've uploaded the image and the feature label array (printed with numpy.savetext()) to OneDrive and attached links
code:
def textureRemover(pix, labeledPix, ratio = 1.0):
numElements = numpy.amax(labeledPix)
maxSize = numpy.count_nonzero(labeledPix)
MAXIMUMCONTRAST = 443.405
for regionID in range(numElements):
start = time.clock()
regionID += 1
if regionID not in labeledPix:
continue
#print(regionID)
#print((regionID / numElements) * 100, '%')
neighborIDs = getNeighbors(labeledPix, regionID)
if 0 in neighborIDs:
neighborIDs.remove(0) #remove white value
regionMask = labeledPix == regionID
region = pix[regionMask]
size = numpy.count_nonzero(regionMask)
contrastMin = (ratio - (size / maxSize)) * MAXIMUMCONTRAST
regionMean = region.mean(axis = 0)
if len(neighborIDs) > 200:
contrast = numpy.zeros(labeledPix.shape)
contrast[labeledPix!=0] = numpy.sqrt(numpy.sum((regionMean - pix[labeledPix!=0])**2, axis = -1))
significantMask = (contrast < contrastMin)
significantContrasts = list(numpy.unique(contrast[significantMask]))
significantNeighbors = {}
for significantContrast in significantContrasts:
minContrast = min(significantContrasts)
if labeledPix[contrast == minContrast][0] in neighborIDs:
significantNeighbors[minContrast] = labeledPix[contrast == minContrast][0]
else:
significantContrasts.pop(significantContrasts.index(minContrast))
else:
significantNeighbors = {}
for neighborID in neighborIDs:
neighborMask = labeledPix == neighborID
neighbor = pix[neighborMask]
neighborMean = neighbor.mean(axis = 0)
contrast = numpy.sqrt(numpy.sum((regionMean - neighborMean)**2, axis = -1))
if contrast < contrastMin:
significantNeighbors[contrast] = neighborID
if significantNeighbors:
contrasts = significantNeighbors.keys()
minContrast = min(contrasts)
minNeighbor = significantNeighbors[minContrast]
neighborMask = labeledPix == minNeighbor
neighborSize = numpy.count_nonzero(neighborMask)
if neighborSize <= size:
labeledPix[neighborMask] = regionID
pix[neighborMask] = regionMean
else:
labeledPix[regionMask] = minNeighbor
pix[regionMask] = pix[neighborMask].mean(axis = 0)
print(time.clock() - start)
return pix
pix
labeledPix
I know I'm asking for a lot of help, but I've been stuck on this for a few weeks and am unsure what else I can do. Any help will be greatly appreciated!
Here is an optimized version of most of your logic (I underestimated the amount of work that would be...). I skipped the >200 branch and am using fake data because I couldn't access your link. When I switch off your >200 branch your and my code appear to give the same result but mine is quite a bit faster on the fake example.
Sample output:
original
26.056154000000003
optimized
0.763613000000003
equal
True
Code:
import numpy as np
from numpy.lib.stride_tricks import as_strided
def mockdata(m, n, k):
colors = np.random.random((m, n, 3))
i, j = np.ogrid[:m, :n]
labels = np.round(k*k * (np.sin(0.05 * i) + np.sin(0.05 * j)**2)).astype(int) % k
return colors, labels
DIAG_NEIGHBORS = True
MAXIMUMCONTRAST = 443.405
def textureRemover2(pix, labeledPix, ratio=1.0):
start = time.clock()
pix, labeledPix = pix.copy(), labeledPix.copy()
pixf, labeledPixf = pix.reshape(-1, 3), labeledPix.ravel()
m, n = labeledPix.shape
s, t = labeledPix.strides
# find all sizes in O(n)
sizes = np.bincount(labeledPixf)
n_ids = len(sizes)
# make index for quick access to labeled areas
lblidx = np.split(np.argsort(labeledPixf), np.cumsum(sizes[:-1]))
lblidx[0] = None
# find all mean colors in O(n)
regionMeans = np.transpose([np.bincount(labeledPix.ravel(), px)
/ np.maximum(sizes, 1)
for px in pix.reshape(-1, 3).T])
# find all neighbors in O(n)
horz = set(frozenset(p) for bl in as_strided(labeledPix, (m,n-1,2), (s,t,t))
for p in bl)
vert = set(frozenset(p) for bl in as_strided(labeledPix, (m-1,n,2), (s,t,s))
for p in bl)
nb = horz|vert
if DIAG_NEIGHBORS:
dwnrgt = set(frozenset(p) for bl in as_strided(
labeledPix, (m-1,n-1,2), (s,t,s+t)) for p in bl)
dwnlft = set(frozenset(p) for bl in as_strided(
labeledPix[::-1], (m-1,n-1,2), (-s,t,t-s)) for p in bl)
nb = nb|dwnrgt|dwnlft
nb = {p for p in nb if len(p) == 2 and not 0 in p}
nb_dict = {}
for a, b in nb:
nb_dict.setdefault(a, set()).add(b)
nb_dict.setdefault(b, set()).add(a)
maxSize = labeledPix.size - sizes[0]
for id_ in range(1, n_ids):
nbs = list(nb_dict.get(id_, set()))
if not nbs:
continue
d = regionMeans[id_] - regionMeans[nbs]
d = np.einsum('ij,ij->i', d, d)
mnd = np.argmin(d)
if d[mnd] < ((ratio - sizes[id_]/maxSize) * MAXIMUMCONTRAST)**2:
mn = nbs[mnd]
lrg, sml = (id_, mn) if sizes[id_] >= sizes[mn] else (mn, id_)
sizes[lrg], sizes[sml] = sizes[lrg] + sizes[sml], 0
for nb in nb_dict[sml]:
nb_dict[nb].remove(sml)
nb_dict[nb].add(lrg)
nb_dict[lrg].update(nb_dict[sml])
nb_dict[lrg].remove(lrg)
nb_dict[sml] = set()
pixf[lblidx[sml]] = regionMeans[lrg]
labeledPixf[lblidx[sml]] = lrg
lblidx[lrg], lblidx[sml] = np.r_[lblidx[lrg],lblidx[sml]], None
print(time.clock() - start)
return pix
from scipy.ndimage.morphology import binary_dilation
import time
STRUCTEL = np.ones((3,3), int) if DIAG_NEIGHBORS else np.array([[0,1,0],[1,1,1],[0,1,0]], int)
def getNeighbors(labeledPix, regionID):
nb = set(labeledPix[binary_dilation(labeledPix == regionID, structure=STRUCTEL)])
nb.remove(regionID)
return sorted(nb)
numpy = np
def textureRemover(pix, labeledPix, ratio = 1.0):
pix, labeledPix = pix.copy(), labeledPix.copy()
numElements = numpy.amax(labeledPix)
maxSize = numpy.count_nonzero(labeledPix)
MAXIMUMCONTRAST = 443.405
start = time.clock()
for regionID in range(numElements):
regionID += 1
if regionID not in labeledPix:
continue
#print(regionID)
#print((regionID / numElements) * 100, '%')
neighborIDs = getNeighbors(labeledPix, regionID)
if 0 in neighborIDs:
neighborIDs.remove(0) #remove white value
regionMask = labeledPix == regionID
region = pix[regionMask]
size = numpy.count_nonzero(regionMask)
contrastMin = (ratio - (size / maxSize)) * MAXIMUMCONTRAST
regionMean = region.mean(axis = 0)
if len(neighborIDs) > 20000:
contrast = numpy.zeros(labeledPix.shape)
contrast[labeledPix!=0] = numpy.sqrt(numpy.sum((regionMean - pix[labeledPix!=0])**2, axis = -1))
significantMask = (contrast < contrastMin)
significantContrasts = list(numpy.unique(contrast[significantMask]))
significantNeighbors = {}
for significantContrast in significantContrasts:
minContrast = min(significantContrasts)
if labeledPix[contrast == minContrast][0] in neighborIDs:
significantNeighbors[minContrast] = labeledPix[contrast == minContrast][0]
else:
significantContrasts.pop(significantContrasts.index(minContrast))
else:
significantNeighbors = {}
for neighborID in neighborIDs:
neighborMask = labeledPix == neighborID
neighbor = pix[neighborMask]
neighborMean = neighbor.mean(axis = 0)
contrast = numpy.sqrt(numpy.sum((regionMean - neighborMean)**2, axis = -1))
if contrast < contrastMin:
significantNeighbors[contrast] = neighborID
if significantNeighbors:
contrasts = significantNeighbors.keys()
minContrast = min(contrasts)
minNeighbor = significantNeighbors[minContrast]
neighborMask = labeledPix == minNeighbor
neighborSize = numpy.count_nonzero(neighborMask)
if neighborSize <= size:
labeledPix[neighborMask] = regionID
pix[neighborMask] = regionMean
else:
labeledPix[regionMask] = minNeighbor
pix[regionMask] = pix[neighborMask].mean(axis = 0)
print(time.clock() - start)
return pix
data = mockdata(200, 200, 1000)
print('original')
res0 = textureRemover(*data)
print('optimized')
res2 = textureRemover2(*data)
print('equal')
print(np.allclose(res0, res2))
I am using ode solver to solve stiff problem (since odeint function could not able to solve it). But by this way also I have some warnings and my plot get saturate at some point. Here is image What should I do? Here is the list of warnings:
DVODE-- Warning..internal T (=R1) and H (=R2) are
such that in the machine, T + H = T on the next step
(H = step size). solver will continue anyway
In above, R1 = 0.3667661010318D+00 R2 = 0.1426374862242D-16
DVODE-- Warning..internal T (=R1) and H (=R2) are
such that in the machine, T + H = T on the next step
(H = step size). solver will continue anyway
In above, R1 = 0.3667661010318D+00 R2 = 0.1426374862242D-16
DVODE-- Above warning has been issued I1 times.
it will not be issued again for this problem
In above message, I1 = 2
DVODE-- At current T (=R1), MXSTEP (=I1) steps
taken on this call before reaching TOUT
In above message, I1 = 500
In above message, R1 = 0.3667661010318D+00
My code:
import numpy as np
import matplotlib.pyplot as plt
import scipy.integrate as si
def func():
#arguments:::
w = 1./3.
xi = 2.86
phi1 = 1.645
phi2 = 2.* 1.202
gt = 10.**(-60)
Lt = (1.202*gt)/np.pi
Lin = 10.**-5
Lf = 0.49
dt = 0.0001
gin = gt*Lt/Lin
xin = (-np.log((3. - (xi**2)*Lin)/(3. - (xi**2)*Lt)) + np.log(Lin/Lt))/4.0
uin = -(np.log(Lin/Lt))/2.
state0 = [gin,xin,uin]
print state0
def eq(L, state):
g = state[0]
x = state[1]
u = state[2]
N = (-2.*g/(6.*np.pi + 5.*g))*(18./(1. - 2.*L) + 5.*np.log(1.- 2.*L) - phi1 + 6. )
B = (-(2. - N)*L) - ((g/np.pi)* (5.*np.log(1.-2.*L) - phi2 + (5.*N/40.)))
Eqs = np.zeros((3))
gdl = Eqs[0] = ((2.+N)*g)/B
xdl = Eqs[1] = -(2./(3.*(1.+w)))* (1./(1.-(xi**2)*L/3.))*(1./B)
udl = Eqs[2]= 1./B
return Eqs
ode = si.ode(eq)
# BDF method suited to stiff systems of ODEs
ode.set_integrator('vode',nsteps=500,method='bdf')
ode.set_initial_value(state0,Lin)
L = []
G = []
while ode.successful() and ode.t < Lf:
ode.integrate(ode.t + dt)
L.append(ode.t)
G.append(ode.y)
lam = np.vstack(L)
g,x,u = np.vstack(G).T
return g,x,u,lam
r= func()
L = r[3]
g = r[0]
lng = np.log10(g)
x = r[1]
u = r[2]
w = 1./3.
xi = 2.86
O_A = np.zeros(len(L))
q = np.zeros(len(L))
for i in np.arange(len(L)):
O_A[i] = xi**2*L[i]/3.
alpha = 2./ ((3.+3.*w) * (1.- (L[i]*xi**2)/3.) )
q[i] = 1./alpha - 1.
n = np.zeros(len(L)) #eta(n)
b = np.zeros(len(L))
for j in np.arange(len(L)):
n[j] =(-2.*g[j]/(6.*np.pi + 5.*g[j]))*(18./(1. - 2.*L[j]) + 5.*np.log(1.- 2.*L[j]) - 1.645 + 6. )
b[j]= (-(2. - n[j])*L[j]) - ((g[j]/np.pi)* (5.*np.log(1.-2.*L[j]) - 2.* 1.202 + ((5.*n[j])/4.)))
P = np.zeros(len(x))
for k in np.arange(len(x)):
C = (((3. - (xi**2)*L[k])/g[k])**(3./4.)) * (((2.*L[k] + (u[k]*b[k]))*xi**2) + (n[k] * (3.- L[k]*xi**2)) )
P[k] = (np.exp(3.*x[k])) * (np.exp(4.*u[k])) * C
plt.figure()
plt.plot(L,P)
plt.xlabel('Lambda ---->')
plt.ylabel('P ----->')
plt.title('lambda Vs P')
plt.show()
I wrote the next code. In 1-2 hours of execution time the RAM of my laptop (8gb) is filled and the sistem crash:
from scipy.stats import uniform
import numpy as np
cant_de_cadenas =[700,800,900]
cantidad_de_cadenas=np.array([])
for kkkkk in cant_de_cadenas:
cantidad_de_cadenas=np.append(cantidad_de_cadenas,kkkkk)
cantidad_de_cadenas=np.transpose(cantidad_de_cadenas)
b=10
h=b
Longitud=1
numero_experimentos=100
densidad_de_cadenas =cantidad_de_cadenas/(b**2)
prob_perc=np.array([])
tiempos=np.array([])
S_int=np.array([])
S_medio=np.array([])
desviacion_standard=np.array([])
desviacion_standard_nuevo=np.array([])
anisotropia_macroscopica_porcentual=np.array([])
componente_y=np.array([])
componente_x=np.array([])
import time
for N in cant_de_cadenas:
empieza=time.clock()
PERCOLACION=np.array([])
size_medio_intuitivo = np.array([])
size_medio_nuevo = np.array([])
std_dev_size_medio_intuitivo = np.array([])
std_dev_size_medio_nuevo = np.array([])
comp_y = np.array([])
comp_x = np.array([])
for u in xrange(numero_experimentos):
perco = False
array_x1=uniform.rvs(loc=-b/2, scale=b, size=N)
array_y1=uniform.rvs(loc=-h/2, scale=h, size=N)
array_angle=uniform.rvs(loc=-0.5*(np.pi), scale=np.pi, size=N)
array_pendiente_x=1./np.tan(array_angle)
random=uniform.rvs(loc=-1, scale=2, size=N)
lambda_sign=np.zeros([N])
for t in xrange(N):
if random[t]<0:
lambda_sign[t]=-1
else:
lambda_sign[t]=1
array_lambdas=(lambda_sign*Longitud)/np.sqrt(1+array_pendiente_x**2)
array_x2= array_x1 + array_lambdas*array_pendiente_x
array_y2= array_y1 + array_lambdas*1
array_x1 = np.append(array_x1, [-b/2, b/2, -b/2, -b/2])
array_y1 = np.append(array_y1, [-h/2, -h/2, -h/2, h/2])
array_x2 = np.append(array_x2, [-b/2, b/2, b/2, b/2])
array_y2 = np.append(array_y2, [h/2, h/2, -h/2, h/2])
M = np.zeros([N+4,N+4])
for j in xrange(N+4):
if j>0:
x_A1B1 = array_x2[j]-array_x1[j]
y_A1B1 = array_y2[j]-array_y1[j]
x_A1A2 = array_x1[0:j]-array_x1[j]
y_A1A2 = array_y1[0:j]-array_y1[j]
x_A2A1 = -1*x_A1A2
y_A2A1 = -1*y_A1A2
x_A2B2 = array_x2[0:j]-array_x1[0:j]
y_A2B2 = array_y2[0:j]-array_y1[0:j]
x_A1B2 = array_x2[0:j]-array_x1[j]
y_A1B2 = array_y2[0:j]-array_y1[j]
x_A2B1 = array_x2[j]-array_x1[0:j]
y_A2B1 = array_y2[j]-array_y1[0:j]
p1 = x_A1B1*y_A1A2 - y_A1B1*x_A1A2
p2 = x_A1B1*y_A1B2 - y_A1B1*x_A1B2
p3 = x_A2B2*y_A2B1 - y_A2B2*x_A2B1
p4 = x_A2B2*y_A2A1 - y_A2B2*x_A2A1
condicion_1=p1*p2
condicion_2=p3*p4
for k in xrange (j):
if condicion_1[k]<=0 and condicion_2[k]<=0:
M[j,k]=1
del condicion_1
del condicion_2
if j+1<N+4:
x_A1B1 = array_x2[j]-array_x1[j]
y_A1B1 = array_y2[j]-array_y1[j]
x_A1A2 = array_x1[j+1:]-array_x1[j]
y_A1A2 = array_y1[j+1:]-array_y1[j]
x_A2A1 = -1*x_A1A2
y_A2A1 = -1*y_A1A2
x_A2B2 = array_x2[j+1:]-array_x1[j+1:]
y_A2B2 = array_y2[j+1:]-array_y1[j+1:]
x_A1B2 = array_x2[j+1:]-array_x1[j]
y_A1B2 = array_y2[j+1:]-array_y1[j]
x_A2B1 = array_x2[j]-array_x1[j+1:]
y_A2B1 = array_y2[j]-array_y1[j+1:]
p1 = x_A1B1*y_A1A2 - y_A1B1*x_A1A2
p2 = x_A1B1*y_A1B2 - y_A1B1*x_A1B2
p3 = x_A2B2*y_A2B1 - y_A2B2*x_A2B1
p4 = x_A2B2*y_A2A1 - y_A2B2*x_A2A1
condicion_1=p1*p2
condicion_2=p3*p4
for k in xrange ((N+4)-j-1):
if condicion_1[k]<=0 and condicion_2[k]<=0:
M[j,k+j+1]=1
del condicion_1
del condicion_2
M[N,N+2]=0
M[N,N+3]=0
M[N+1,N+2]=0
M[N+1,N+3]=0
M[N+2,N]=0
M[N+2,N+1]=0
M[N+3,N]=0
M[N+3,N+1]=0
CD=np.array([])
POPOPO=[]
for g in xrange(N):
lala=0
r=False
while lala<=len(POPOPO)-1:
esta= g in POPOPO[lala]
if esta is True:
lala=len(POPOPO)
r=True
else:
lala=lala+1
if r is False:
L=np.array([g])
for s in xrange(N):
if M[g,s] != 0:
L=np.append(L,s)
x=0
while x<= N:
for l in xrange(N):
z= l in L
d=L[x]
if z is False and M[d,l] != 0:
L=np.append(L,l)
if x+1<len(L):
x+=1
else:
x=N+1.
q= len (L)
CD=np.append(CD, q)
POPOPO.append(L)
M_horizontal=M.copy()
M_horizontal[:,N+2] = np.zeros(N+4)
M_horizontal[:,N+3] = np.zeros(N+4)
M_horizontal[N+2] = np.zeros(N+4)
M_horizontal[N+3] = np.zeros(N+4)
L=np.array([N])
for s in xrange(N+4):
if M_horizontal[N,s] != 0:
L=np.append(L,s)
x=0
while x<= N+4:
for l in xrange(N+4):
z= l in L
d=L[x]
if z is False and M_horizontal[d,l] != 0:
L=np.append(L,l)
if x+1<len(L):
x+=1
else:
x=(N+4)+1.
LV1_in_L = N in L
LV2_in_L= (N+1) in L
if LV1_in_L is True and LV2_in_L is True:
perc_horiz=True
else:
perc_horiz=False
M_vertical=M.copy()
M_vertical[:,N] = np.zeros(N+4)
M_vertical[:,N+1] = np.zeros(N+4)
M_vertical[N] = np.zeros(N+4)
M_vertical[N+1] = np.zeros(N+4)
L=np.array([N+2])
for s in xrange(N+4):
if M_vertical[N+2,s] != 0:
L=np.append(L,s)
x=0
while x<= N+4:
for l in xrange(N+4):
z= l in L
d=L[x]
if z is False and M_vertical[d,l] != 0:
L=np.append(L,l)
if x+1<len(L):
x+=1
else:
x=(N+4)+1.
LH1_in_L = (N+2) in L
LH2_in_L= (N+3) in L
if LH1_in_L is True and LH2_in_L is True:
perc_ver = True
else:
perc_ver = False
if perc_ver is True or perc_horiz is True:
PERCOLACION=np.append(PERCOLACION,1)
perco=True
D = np.array([])
W = np.array([])
for c in xrange (int(min(CD)), int(max(CD)+1),1):
D=np.append(D,c)
frec = sum (CD == c)
W = np.append(W,frec)
if perco is True:
posicion=np.argmax(D)
D=np.delete(D,posicion)
W=np.delete(W,posicion)
if len(D) == 0 and len(W)==0:
S_medio_intuitivo_exp_u=0
S_medio_nuevo_exp_u = 0
std_dev_exp_u = 0
std_dev_nuevo_exp_u = 0
else:
S_medio_intuitivo_exp_u = np.average (D,weights=W)
peso_nuevo=D*W
S_medio_nuevo_exp_u = np.average (D,weights=peso_nuevo)
tipos=sum(W)
X=W*((D-S_medio_intuitivo_exp_u)**2)
S=sum(X)
std_dev_exp_u = np.sqrt(S/(tipos-1.))
tipos_nuevo=sum(peso_nuevo)
X_nuevo=peso_nuevo*((D-S_medio_nuevo_exp_u)**2)
S_nuevo=sum(X_nuevo)
std_dev_nuevo_exp_u = np.sqrt(S_nuevo/(tipos_nuevo-1.))
componente_longitudinal=Longitud*np.abs(np.cos(array_angle))
comp_y=np.append(comp_y, sum(componente_longitudinal)/N)
componente_transversal=Longitud*np.abs(np.sin(array_angle))
comp_x=np.append(comp_x, sum(componente_transversal)/N)
std_dev_size_medio_intuitivo=np.append(std_dev_size_medio_intuitivo, std_dev_exp_u)
std_dev_size_medio_nuevo=np.append(std_dev_size_medio_nuevo, std_dev_nuevo_exp_u)
size_medio_intuitivo=np.append(size_medio_intuitivo, S_medio_intuitivo_exp_u)
size_medio_nuevo=np.append(size_medio_nuevo, S_medio_nuevo_exp_u)
percolation_probability=sum(PERCOLACION)/numero_experimentos
prob_perc=np.append(prob_perc, percolation_probability)
S_int = np.append (S_int, sum(size_medio_intuitivo)/numero_experimentos)
S_medio=np.append (S_medio, sum(size_medio_nuevo)/numero_experimentos)
desviacion_standard = np.append (desviacion_standard, sum(std_dev_size_medio_intuitivo)/numero_experimentos)
desviacion_standard_nuevo=np.append (desviacion_standard_nuevo, sum(std_dev_size_medio_nuevo)/numero_experimentos)
tiempos=np.append(tiempos, time.clock()-empieza)
componente_y=np.append(componente_y, sum(comp_y)/numero_experimentos)
componente_x=np.append(componente_x, sum(comp_x)/numero_experimentos)
anisotropia_macroscopica_porcentual=100*(1-(componente_y/componente_x))
I tryed with gc and gc.collect() and 'del'command for deleting arrays after his use and nothing work!
What am I doing wrong? Why the memory becomes full while running (starts with 10% of RAM used and in 1-2hour is totally full used)?
Lets take an array M with size 300MB. When this is overwritten (for example, in each iteration), have we 300MB occupated in RAM memory or just 300MB? In the case that we have just 300MB, there should be no problem, so why have I this RAM issue? In the case of the RAM is acumulated, how can I do for free RAM memory occupated for the 'old' array?
Please help me, I'm totally stuck!
Thanks a lot!