I am starting to code up in Python and I come from a Matlab background. I have a problem with a for loop that I am trying to do.
So this is my for loop from Matlab,
ix = indoor(1);
idx = indoor(2)-indoor(1);
%Initialize X apply I.C
X = [ix;idx];
for k=(1:1:287)
X(:,k+1) = Abest*X(:,k) + Bbest*outdoor(k+1) + B1best* (cbest4/cbest1);
end
In this code Abest is a 2x2 matrix, Bbest is a 2x1 matrix, outdoor is a 288x1 vector, B1best is a 2x1 matrix. The matricies are found from a function using the matrix expodential command. c4 and c1 are terms defined before, constants.
In Python I have been able to get the matrix exponential command to work in my function but I can't get that for loop to work.
Xo = np.array([[ix],[idx]])
num1 = range(0,276)
for k in num1:
Xo[:,k+1] = Ae*Xo[:,k] + Be*outdoor[k+1] + Be1*(c4/c1)
Again Ae,Be,Be1 are matrices of the same size just like the Matlab ones. Same thing for the outdoor vector.
I have tried everything I can think of to make it work... The only thing that worked for me was,
Xo = np.zeros(())
#Initial COnditions
ix = np.array(indoor[0])
idx = np.array(indoor[1]-indoor[0])
Xo = np.array([[ix],[idx]])
#Range for the for loop
num1 = range(0,1)
for k in num1:
Xo = Ae*Xo[k] + Be*outdoor[k+1] + Be1*(c4/c1)
Now, this thing will work but only give me two points. If I change the range I get an error. I'm assuming this code works because my original Xo is just two states so k goes through those two states but that's not what I want.
If anyone could help me out that would be very helpful! If I'm making some code error, it's honestly because I'm not understanding the 'For loop' in python to well when it comes to data analysis and having it loop through the rows and increment the columns. Thank you for your time.
Upon Request here is my full code:
import scipy.io as sc
import math as m
import numpy as np
import matplotlib.pyplot as plt
import sys
from scipy.linalg import expm, sinm, cosm
import pandas as pd
df = pd.read_excel('datatemp.xlsx')
outdoor = np.array(df[['Outdoor']])
indoor = np.array(df[['Indoor']])
###########################. FUNCTION DEFINE. #################################################
#Progress bar
def progress(count, total, status=''):
percents = round(100.0 * count / float(total), 1)
sys.stdout.write(' %s%s ...%s\r' % ( percents, '%', status))
sys.stdout.flush()
#Define Matrix for Model
def Matrixbuild(c1,c2,c3):
A = np.array([[0,1],[-c3/c1,-c2/c1]])
B = np.array([[0],[1/c1]])
B1 = np.array([[1],[0]])
C = np.zeros((2,2))
D = np.zeros((2,2))
F = np.array([[0,1,0,1],[-c3/c1,-c2/c1,1/c1,0],[0,0,0,0],[0,0,0,0]])
R = np.array(expm(F))
Ae = np.array([[R.item(0),R.item(1)],[R.item(4),R.item(5)]])
Be = np.array([[R.item(2)],[R.item(6)]])
Be1 = np.array([[R.item(3)],[R.item(7)]])
return Ae,Be,Be1;
###########################. Data. #################################################
#USED FOR JUST TRYING WITHOUT ACTUAL DATA
# outdoor = np.array([5.8115,4.394,5.094,5.1123,5.1224])
# indoor = np.array([15.595,15.2429,15.0867,14.9982,14.8993])
###########################. Model Define. #################################################
Xo = np.zeros((2,288))
ix = np.array(indoor[0])
idx = np.array(indoor[1])
err_min = m.inf
c1spam = np.linspace(0.05,0.001,30)
c2spam = np.linspace(6.2,6.5,30)
c3spam = np.linspace(7.1,7.45,30)
totalspam = len(c1spam)*len(c2spam)*len(c3spam)
ind = 0
for c1 in c1spam:
for c2 in c2spam:
for c3 in c3spam:
c4 = 1.1
#MatrixBuild Function
result = Matrixbuild(c1,c2,c3)
Ae,Be,Be1 = result
Xo = np.array([ix,idx])
Datarange = range(0,len(outdoor)-1,1)
for k in Datarange:
Xo[:,k+1] = np.matmul(Ae,Xo[:,k]) + np.matmul(Be,outdoor[k+1]) + Be1*(c4/c1)
ind = ind + 1
print(Xo)
err = np.linalg.norm(Xo[0,range(0,287)]-indoor.T)
if err<err_min:
err_min = err
cbest = np.array([[c1],[c2],[c3],[c4]])
progress(ind,totalspam,status='Done')
# print(X)
# print(err)
# print(cbest)
###########################. Model with Cbest Values. #################################################
c1 = cbest[0]
c2 = cbest[1]
c3 = cbest[2]
result2 = Matrixbuild(c1,c2,c3)
AeBest,BeBest,Be1Best = result2
Xo = np.array([ix,idx])
Datarange = np.arange(0,len(outdoor)-1)
for k in Datarange:
Xo[:,k+1] = np.matmul(AeBestb,Xo[:,k]) + np.matmul(BeBest,outdoor[k+1]) + Be1Best*(c4/c1)
err = np.linalg.norm(Xo[0,range(0,287)]-indoor.T)
print(cbest)
print(err)
###########################. Plots. #################################################
plt.figure(0)
time = np.linspace(1,2,2)
plt.scatter(time,X[0],s=15,c="blue")
plt.scatter(time,indoor[0:2],s=15,c="red")
plt.show()
And again my error occurs in the line with the for loop of
for k in Datarange:
Xo[:,k+1] = np.matmul(Ae,Xo[k]) + np.matmul(Be,outdoor[k+1]) + Be1*(c4/c1)
I was trying to use np.matmul for matrix multiplication but even without it, it wasn't working.
If there are any other questions about my code please ask. Essentially I'm trying to find the best c1,c2,c3 coefficients that fit my data which is indoor temperature by using a basic second order constant coefficient model.
Have you tried with Xo[:,k+1] instead of Xo(:,k+1)? Python uses [] for slicing and indexing.
EDIT:
Xo = np.array([[ix],[idx]])
This creates a 1x1 array with 1 value: (ix, idx). I think you're looking for something like Xo = np.zeros((ix, idx)), which will give you an ixxidx array initialized to zeros. If you don't need the zeros you can use Xo = np.empty((ix, idx)).
See the docs on array creation.
So by reading into how python works a little more and allocation for arrays/matrices, I was able to find out how to do it. I needed to first allocate my 'Xo' value and then input the initial conditions in order for the For loop to work.
Xo = np.zeros((2,num2))
Xo = np.asmatrix(Xo)
Xo[0,0] = ix
Xo[1,0] = idx
Also for the 'for loop', I called the range some value like this,
num1 = range(0,4)
num2 = len(num1) + 1
This helped in order to calculate the total dimension of 'Xo', by calling it 'num2'. It was also defined like that because my 'For loop' went (k+1), this the dimension would grow larger, ex:
for k in num1:
Xo[:,k+1] = Ae*Xo[:,k] + Be*outdoor[k+1] + Be1*(c4/c1)
But there it is! I figured it by comparing Matlab printouts to Python printouts and just trying to debug one line at a time. Now I have the same exact value print out in both goods, so it is time to start using the python code!
Related
I am writing this simple function to use power iteration for the dominant eigenvalue. I want to put 2 stop conditionals. One for iterations and one for a precision threshold. But this error calculation does not work.
What a i doing wrong here in principle ?
#power ite. vanilla
A = np.random.uniform(low=-5.0, high=10.0, size=[3,3])
def power_iteration(A, maxiter, threshold):
b0 = np.random.rand(A.shape[1])
it = 0
error = 0
while True:
for i in range(maxiter):
b1 = np.dot(A, b0)
b1norm = np.linalg.norm(b1)
error = np.linalg.norm(b1-b0)
b0 = b1/b1norm
domeig = (b0#A#b0)/np.dot(b0, b0)
if error<threshold:
break
elif it>maxiter:
break
else:
error = 0
it = it + 1
return b0, domeig, it, error
result = power_iteration(A, 10, 0.1)
result
The output shows a very correct eigenvalue of ~9 and corresponding eigenvector ( i checked with numpy)
But the error is off. There is no way the length of the difference vector is 8. Considering the result is very close to the actual.
How i want to calculate error is the norm of the difference between the current eigenvector - the previous (b0). I start the error = 0 because the first iteration is guaranteed to give a big difference if b0 is chosen random
(array([ 0.06009408, 0.95411524, -0.2933476 ]),
9.001665234545708,
11,
8.001665234545815)
Tried to make a loop stop by 2 conditions. One gets ignored
Seems to work much better like this.
def power_it(matrix, iterations, threshold):
domeigenvector = np.random.rand(matrix.shape[1])
counter = np.random.rand(matrix.shape[1])
it = 0
error = 0
for i in range(iterations):
k1 = np.dot(A, domeigenvector)
k1norm = np.linalg.norm(k1)
domeigenvector = k1/k1norm
error = np.linalg.norm(domeigenvector-counter)
counter = domeigenvector
domeigenvalue = (domeigenvector#A#domeigenvector)/np.dot(domeigenvector, domeigenvector)
it = it + 1
if error < threshold:
break
return domeigenvalue, domeigenvector, it
I can now use Schur deflation to calculate the rest of the eigenpairs.
this is a follow up question to the one I posted earlier:
GEKKO - optimization in matrix form
I need to add one more constraint that tracks "inventory" ("Inv"), which tracks the sum(q[i,:] - q[:,i]). "Inv" will be a 4X1 column vector. I tried the following:
m = GEKKO(remote=False)
q = m.Array(m.Var,(4,4),lb=0,ub=10)
for i in range(4):
for j in range(4):
if j<=i:
q[i,j].upper=0 # set upper bound = 0
def profit(q):
profit = np.sum(q.flatten() * pmx.flatten())
return profit
Inv[0]=0
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
m.Equation(I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i]))) # New Line 1a inventory
Inv[i] = Inv[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])) # New Line 1b inventory. Keep either 1a or 1b
m.Equation(Inv[i] <= 15) # New Line 2 inventory constraint
m.Equation(Inv[4] = 0) # New Line 3 ending inventory should be equal to starting inventory
m.Maximize(profit(q))
m.solve()
print(q)
qr = np.array([[q[i,j].value[0] for j in range(4)] for i in range(4)])
Ir = np.array([Inv[i].value[0] for i in range(4)]) #New Line 4
Errors :
1a. Adding New Line 1a: "keyword can't be an expression"
1b. Replacing New Line 1a with 1b: no issues (but, I'm not sure if GEKKO will keep track of I or not.Also, I need to define "I", the way "q" was done...not sure how). Replacing = comment out 1a, then run the code with 1b.
New Line 2: Error = "object of type 'int' has no len()"; but type(I) shows as ndarray. (Kept New Lines 1a and 1b, and then added New Line 2)
New Line 3:Error = "keyword can't be an expression" (Kept Line 32and then added Line 3)
New Line 4: Error "'numpy.ndarray' object has no attribute 'value'" [removed Lines 3 and 4. This makes sense as if I can't capture "Inv" in the model, then it won't have the value attribute)
Questions:
1. Am I defining inventory correctly?
If yes, can it be done in the current model specification, or will it need an entirely different formulation? If so, could you please guide on what that would be?
Of the various videos posted on the GEKKO website, is there a specific one I should look at for more information? I was thinking the DO video, but I don't believe this is quite a dynamic optimization problem (as I'm not trying to optimize the best path)....
Thanks again for all your help,
----UPDATE 5/10
Also tried:
Inv = m.SV()
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
#m.Equation(I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])))
m.Equation(m.Inv.dt() == m.Inv + (np.sum(q[i,:]) - np.sum(q[:,i])))
#I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])
m.Equation(m.Inv <= 15)
#m.Equation(I[4] = 0)
m.Maximize(profit(q))
New Error: 'GEKKO' object has no attribute 'Inv'
One way to do this is to start with zero inventory with Inv[0]=0 and then track the inventory amount with gekko variables Inv[1:4]. A couple tips on building the model:
Use double equal signs for equality constraints
You can define a temporary variable such as I = Inv[2]+Inv[3] but it won't be a gekko variable
You may also want to look at Intermediate variables for those that are explicitly calculated. This can speed up the calculation.
I recommend this tutorial in the Dynamic Optimization course
import numpy as np
import scipy.optimize as opt
from gekko import GEKKO
p= np.array([4, 5, 6.65, 12]) #p = prices
pmx = np.triu(p - p[:, np.newaxis]) #pmx = price matrix, upper triangular
m = GEKKO(remote=False)
q = m.Array(m.Var,(4,4),lb=0,ub=10)
# only upper triangular can change
for i in range(4):
for j in range(4):
if j<=i:
q[i,j].upper=0 # set upper bound = 0
def profit(q):
profit = np.sum(q.flatten() * pmx.flatten())
return profit
Inv = m.Array(m.Var,5,lb=0,ub=15)
Inv[0].upper = 0 # start with 0 inventory
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
# track inventory
m.Equation(Inv[i+1]==Inv[i] + (m.sum(q[i,:])-m.sum(q[:,i])))
m.Equation(Inv[4] == 0) # Use double == sign, not needed in loop
m.Maximize(profit(q))
m.solve()
print(q)
# convert to matrix form
qr = np.array([[q[i,j].value[0] for j in range(4)] for i in range(4)])
for i in range(4):
rs = qr[i,:].sum()
print('Row sum ' + str(i) + ' = ' + str(rs))
cs = qr[:,i].sum()
print('Col sum ' + str(i) + ' = ' + str(cs))
Ir = np.array([Inv[i].value[0] for i in range(4)])
print(Ir)
So I have a challenge in my hand that I am trying to accomplish. I have a matlab code that works fine homever, I want to write the same code in python. Homever I don't get the same results.
I have tried using a different for loop than the one in matlab. Although these should give the same results I am fail at some point in the loop, although I couldn't figure out where the mistake was.
for ii = 1:100 #matlab code
healthy=2*randn(100,1000)+5;
patient=2*randn(100,1000)+7;
threshold=mu_healthy-sd_healthy:0.1:mu_patient+sd_patient;
for i=1:length(threshold)
TP(i)=sum(patient>=threshold(i));
FP(i)=sum(healthy>=threshold(i));
TN(i)=sum(healthy<threshold(i));
FN(i)=sum(patient<threshold(i));
end
FPR(ii,:)=FP/1000;
TPR(ii,:)=TP/1000;
def appending(): #python code
for n in range(0,50):
for x in range(0,1000):
for a in range(0,61):
if Apatient[x,n]>=newthreshold[a]:
TP[a].append(Apatient[x,n])
elif Ahealthy[x,n]>=newthreshold[a]:
FP.append(Ahealthy[x,n])
elif Apatient[x,n]<newthreshold[a]:
TN.append(Apatient[x,n])
elif Ahealthy[x,n]<newthreshold[a]:
FN.append(Ahealthy[x,n])
If you can run this in matlab, you will see FN,TN values with 61 values in each column. I want the same to happen in my loop as well,homever I get lots of elements if I run this code. Thanks
Just following MATLAB script, tried translation.
import numpy as np
mu_healthy = 5
sd_healthy = 2
mu_patient = 7
sd_patient = 2
threshold = np.arange(mu_healthy-sd_healthy, mu_patient+sd_patient+0.1, 0.1)
L = len(threshold)
TP = np.zeros([L,1])
FP = np.zeros([L,1])
TN = np.zeros([L,1])
FN = np.zeros([L,1])
FPR = np.zeros([100,L])
TPR = np.zeros([100,L])
for ii in range(0,100):
healthy = sd_healthy*np.random.normal(mu_healthy,1,[100,1000])
patient = sd_patient*np.random.normal(mu_patient,1,[100,1000])
for i in range(0, L):
TP[i] = np.sum(patient>=threshold[i])
FP[i] = np.sum(healthy>=threshold[i])
TN[i] = np.sum(healthy<threshold[i])
FN[i] = np.sum(patient<threshold[i])
FPR[ii,:] = FP[:,0]
TPR[ii,:] = TP[:,0]
FPR = FPR/1000
TPR = TPR/1000
I am fairly new to python and am trying to recreate the electric potential in a metal box using the laplace equation and the jacobi method. I have written a code that seems to work initially, however I am getting the error: IndexError: index 8 is out of bounds for axis 0 with size 7 and can not figure out why. any help would be awesome!
from visual import*
from visual.graph import*
import numpy as np
lenx = leny = 7
delta = 2
vtop = [-1,-.67,-.33,.00,.33,.67,1]
vbottom = [-1,-.67,-.33,.00,.33,.67,1]
vleft = -1
vright = 1
vguess= 0
x,y = np.meshgrid(np.arange(0,lenx), np.arange(0,leny))
v = np.empty((lenx,leny))
v.fill(vguess)
v[(leny-1):,:] = vtop
v [:1,:] = vbottom
v[:,(lenx-1):] = vright
v[:,:1] = vleft
maxit = 500
for iteration in range (0,maxit):
for i in range(1,lenx):
for j in range(1,leny-1):
v[i,j] = .25*(v[i+i][j] + v[i-1][j] + v[i][j+1] + v[i][j-1])
print v
Just from a quick glance at your code it seems as though the indexing error is happening at this part and can be changed accordingly:
# you had v[i+i][j] instead if v[i+1][j]
v[i,j] = .25*(v[i+1][j] + v[i-1][j] + v[i][j+1] + v[i][j-1])
You simply added and extra i to your indexing which would have definitely been out of range
This portion I was able to vectorize and get rid of a nested loop.
def EMalgofast(obsdata, beta, pjt):
n = np.shape(obsdata)[0]
g = np.shape(pjt)[0]
zijtpo = np.zeros(shape=(n,g))
for j in range(g):
zijtpo[:,j] = pjt[j]*stats.expon.pdf(obsdata,scale=beta[j])
zijdenom = np.sum(zijtpo, axis=1)
zijtpo = zijtpo/np.reshape(zijdenom, (n,1))
pjtpo = np.mean(zijtpo, axis=0)
I wasn't able to vectorize the portion below. I need to figure that out
betajtpo_1 = []
for j in range(g):
num = 0
denom = 0
for i in range(n):
num = num + zijtpo[i][j]*obsdata[i]
denom = denom + zijtpo[i][j]
betajtpo_1.append(num/denom)
betajtpo = np.asarray(betajtpo_1)
return(pjtpo,betajtpo)
I'm guessing Python is not your first programming language based on what I see. The reason I'm saying this is that in python, normally we don't have to deal with manipulating indexes. You act directly on the value or the key returned. Make sure not to take this as an offense, I do the same coming from C++ myself. It's a hard to remove habits ;).
If you're interested in performance, there is a good presentation by Raymond Hettinger on what to do in Python to be optimised and beautiful :
https://www.youtube.com/watch?v=OSGv2VnC0go
As for the code you need help with, is this helping for you? It's unfortunatly untested as I need to leave...
ref:
Iterating over a numpy array
http://docs.scipy.org/doc/numpy/reference/generated/numpy.true_divide.html
def EMalgofast(obsdata, beta, pjt):
n = np.shape(obsdata)[0]
g = np.shape(pjt)[0]
zijtpo = np.zeros(shape=(n,g))
for j in range(g):
zijtpo[:,j] = pjt[j]*stats.expon.pdf(obsdata,scale=beta[j])
zijdenom = np.sum(zijtpo, axis=1)
zijtpo = zijtpo/np.reshape(zijdenom, (n,1))
pjtpo = np.mean(zijtpo, axis=0)
betajtpo_1 = []
#manipulating an array of numerator and denominator instead of creating objects each iteration
num=np.zeros(shape=(g,1))
denom=np.zeros(shape=(g,1))
#generating the num and denom real value for the end result
for (x,y), value in numpy.ndenumerate(zijtpo):
num[x],denom[x] = num[x] + value *obsdata[y],denom[x] + value
#dividing all at once after instead of inside the loop
betajtpo_1= np.true_divide(num/denom)
betajtpo = np.asarray(betajtpo_1)
return(pjtpo,betajtpo)
Please leave me some feedback !
Regards,
Eric Lafontaine