We Keep Coding

Python 1 of 2 stop conditions gets ignored in a loop

I am writing this simple function to use power iteration for the dominant eigenvalue. I want to put 2 stop conditionals. One for iterations and one for a precision threshold. But this error calculation does not work.
What a i doing wrong here in principle ?
#power ite. vanilla
A = np.random.uniform(low=-5.0, high=10.0, size=[3,3])
def power_iteration(A, maxiter, threshold):
b0 = np.random.rand(A.shape[1])
it = 0
error = 0
while True:
for i in range(maxiter):
b1 = np.dot(A, b0)
b1norm = np.linalg.norm(b1)
error = np.linalg.norm(b1-b0)
b0 = b1/b1norm
domeig = (b0#A#b0)/np.dot(b0, b0)
if error<threshold:
break
elif it>maxiter:
break
else:
error = 0
it = it + 1
return b0, domeig, it, error
result = power_iteration(A, 10, 0.1)
result
The output shows a very correct eigenvalue of ~9 and corresponding eigenvector ( i checked with numpy)
But the error is off. There is no way the length of the difference vector is 8. Considering the result is very close to the actual.
How i want to calculate error is the norm of the difference between the current eigenvector - the previous (b0). I start the error = 0 because the first iteration is guaranteed to give a big difference if b0 is chosen random
(array([ 0.06009408, 0.95411524, -0.2933476 ]),
9.001665234545708,
11,
8.001665234545815)
Tried to make a loop stop by 2 conditions. One gets ignored

Seems to work much better like this.
def power_it(matrix, iterations, threshold):
domeigenvector = np.random.rand(matrix.shape[1])
counter = np.random.rand(matrix.shape[1])
it = 0
error = 0
for i in range(iterations):
k1 = np.dot(A, domeigenvector)
k1norm = np.linalg.norm(k1)
domeigenvector = k1/k1norm
error = np.linalg.norm(domeigenvector-counter)
counter = domeigenvector
domeigenvalue = (domeigenvector#A#domeigenvector)/np.dot(domeigenvector, domeigenvector)
it = it + 1
if error < threshold:
break
return domeigenvalue, domeigenvector, it
I can now use Schur deflation to calculate the rest of the eigenpairs.

How to structure python programs? Tried making it more structured, now runs 13 times slower

Im very new to programming, I wrote a simple program for a school project and wanted to make the code "prettier" by not just having the program be one giant function but instead be made up of multiple smaller functions with a singe purpose. I seemed to have messed up royally since the program now runs 13 times slower. How should I structured the program to make it run faster and just in general make programs easier to write, read and edit?
Here are the two programs:
First program (for reference values runs in ≈0:20):
import numpy as np
import matplotlib.pyplot as plt
def graf(a,b,H,p):
GM = 39.5216489684
x_0 = a + np.sqrt(a**2 - b**2)
v_0 = np.sqrt(GM*(2/x_0 - 1/a))
konstant_period = np.sqrt(a**3)*H
h = 1/H
'''starting position given by an elliptic orbit '''
stor_x_lista = [x_0]
stor_y_lista = [0]
hastighet_x = [0]
hastighet_y = [v_0]
liten_x_lista = []
liten_y_lista = []
''' a loop that approximates the points of the orbit'''
t = 0
tid_lista = []
n = 0
while n < konstant_period:
hastighet_x.append(hastighet_x[n] - h*GM* stor_x_lista[n]/(np.sqrt(stor_x_lista[n]**2 + stor_y_lista[n]**2))**3)
stor_x_lista.append(stor_x_lista[n] + h*hastighet_x[n])
hastighet_y.append(hastighet_y[n] - h*GM*stor_y_lista[n]/(np.sqrt(stor_x_lista[n]**2 + stor_y_lista[n]**2))**3)
stor_y_lista.append(stor_y_lista[n] + h*hastighet_y[n])
'''smaller list of points to run faster'''
if n % p == 0:
liten_x_lista.append(stor_x_lista[n])
liten_y_lista.append(stor_y_lista[n])
tid_lista.append(t)
n += 1
t += h
''' function that finds the angle'''
vinkel = []
siffra = 0
while siffra < len(liten_x_lista):
if liten_y_lista[siffra ] >= 0:
vinkel.append( np.arccos( liten_x_lista[siffra]/np.sqrt( liten_x_lista[siffra]**2 + liten_y_lista[siffra]**2)))
siffra += 1
elif liten_y_lista[siffra] < 0 :
vinkel.append( np.pi + np.arccos( -liten_x_lista[siffra]/np.sqrt( liten_x_lista[siffra]**2 + liten_y_lista[siffra]**2) ))
siffra += 1
'''get rid of line to find periodic function'''
mod_lista = []
modn = 0
while modn < len(vinkel):
mod_lista.append(vinkel[modn] - (2*np.pi*tid_lista[modn])/np.sqrt(a**3))
modn += 1
'''make all inputs have period 1'''
squeeze_tid = []
squeezen = 0
while squeezen < len(tid_lista):
squeeze_tid.append(tid_lista[squeezen]/np.sqrt(a**3))
squeezen += 1
del mod_lista[-1:]
del tid_lista[-1:]
del squeeze_tid[-1:]
plt.plot(squeeze_tid,mod_lista)
plt.title('p(t) där a = ' + str(a) + ' och b = ' + str(b))
plt.show
Second more split up program (for reference values runs in ≈4:20):
import numpy as np
import matplotlib.pyplot as plt
'''function that generates the points of the orbit'''
def punkt(a,b,H,p):
GM = 39.5216489684
x_0 = a + np.sqrt(a**2 - b**2)
v_0 = np.sqrt(GM*(2/x_0 - 1/a))
konstant_period = np.sqrt(a**3)*H
h = 1/H
'''starting position given by an elliptic orbit '''
stor_x_lista = [x_0]
stor_y_lista = [0]
hastighet_x = [0]
hastighet_y = [v_0]
liten_x_lista = []
liten_y_lista = []
''' a loop that approximates the points of the orbit'''
t = 0
tid_lista = []
n = 0
while n < konstant_period:
hastighet_x.append(hastighet_x[n] - h*GM* stor_x_lista[n]/(np.sqrt(stor_x_lista[n]**2 + stor_y_lista[n]**2))**3)
stor_x_lista.append(stor_x_lista[n] + h*hastighet_x[n])
hastighet_y.append(hastighet_y[n] - h*GM*stor_y_lista[n]/(np.sqrt(stor_x_lista[n]**2 + stor_y_lista[n]**2))**3)
stor_y_lista.append(stor_y_lista[n] + h*hastighet_y[n])
'''smaller list of points to run faster'''
if n % p == 0:
liten_x_lista.append(stor_x_lista[n])
liten_y_lista.append(stor_y_lista[n])
tid_lista.append(t)
n += 1
t += h
return (liten_x_lista,liten_y_lista,tid_lista)
''' function that finds the angle'''
def vinkel(a,b,H,p):
'''import lists'''
liten_x_lista = punkt(a,b,H,p)[0]
liten_y_lista = punkt(a,b,H,p)[1]
tid_lista = punkt(a,b,H,p)[2]
'''find the angle'''
vinkel_lista = []
siffra = 0
while siffra < len(liten_x_lista):
if liten_y_lista[siffra ] >= 0:
vinkel_lista.append( np.arccos( liten_x_lista[siffra]/np.sqrt( liten_x_lista[siffra]**2 + liten_y_lista[siffra]**2)))
siffra += 1
elif liten_y_lista[siffra] < 0 :
vinkel_lista.append( np.pi + np.arccos( -liten_x_lista[siffra]/np.sqrt( liten_x_lista[siffra]**2 + liten_y_lista[siffra]**2) ))
siffra += 1
return (vinkel_lista, tid_lista)
def periodisk(a,b,H,p):
'''import lists'''
tid_lista = vinkel(a,b,H,p)[1]
vinkel_lista = vinkel(a,b,H,p)[0]
'''get rid of linear line to find p(t)'''
mod_lista = []
modn = 0
while modn < len(vinkel_lista):
mod_lista.append((vinkel_lista[modn] - (2*np.pi*tid_lista[modn])/np.sqrt(a**3)))
modn += 1
'''make all inputs have period 1'''
squeeze_tid = []
squeezen = 0
while squeezen < len(tid_lista):
squeeze_tid.append(tid_lista[squeezen]/np.sqrt(a**3))
squeezen += 1
del mod_lista[-1:]
del tid_lista[-1:]
del squeeze_tid[-1:]
return (squeeze_tid,mod_lista)
'''fixa 3d-punkt av p(a,b) a är konstant b varierar??? '''
def hitta_amp(a):
x_b = []
y_b = []
n_b = 0.1
while n_b <= a:
x_b.append(n_b)
y_b.append(punkt(a,n_b,10**5,10**3))
return 0
def graf(a,b,H,p):
plt.plot(periodisk(a,b,H,p)[0],periodisk(a,b,H,p)[1])
plt.show
I would assume the thing that is going wrong is that the program is running the same, slow code multiple times instead of just running it once and then accessing the data. Is the problem that everything is done locally and nothing is stored globally or is it something else?
Just as a heads up, the only thing I know about programming is basic syntax, I have no clue how to actually write and run programs. I ran all the code in spyder if that affects anything.

plt.plot(periodisk(a,b,H,p)[0],periodisk(a,b,H,p)[1])
This code runs periodisk twice with the same arguments, thus at this point we know we run things at least 2 times slower.
You should do some_var = periodisk(a,b,H,p) and then some_var[0], some_var[1]. Or just use unpacking:
plt.plot(*periodisk(a,b,H,p))
tid_lista = vinkel(a,b,H,p)[1]
vinkel_lista = vinkel(a,b,H,p)[0]
Again doing the same thing twice (total: 4*time of (current) vinkel function). Again, smart assignment to fix this:
vinkel_lista, tid_lista = vinkel(a,b,H,p)
liten_x_lista = punkt(a,b,H,p)[0]
liten_y_lista = punkt(a,b,H,p)[1]
tid_lista = punkt(a,b,H,p)[2]
And now you repeat yourself thrice. (total: 12 * time of current punkt function)
liten_x_lista, liten_y_lista, tid_lista = punkt(a,b,H,p)
punkt function is like in original, so we arrived as total being 12 times slower - which quite matches your time estimations. :)

You are calling the functions once per returned list, you should only call them once.
When a method returns multiple variables, (e.g. punkt):
def punkt(a,b,H,p):
# Here is all your code
return (liten_x_lista,liten_y_lista,tid_lista)
You must be careful to only call the function once:
result = punkt(a,b,H,p)
liten_x_lista = result[0]
liten_y_lista = result[1]
tid_lista = result[2]
# As opposed to:
liten_x_lista = punkt(a,b,H,p)[0] # 1st call, ignoring results 2 and 3
liten_y_lista = punkt(a,b,H,p)[1] # 2nd call, ignoring results 1 and 3
tid_lista = punkt(a,b,H,p)[2] # 3rd call, ignoring results 1 and 2
Note: I would personally not return a list, but use python's unpacking:
def punkt(a,b,H,p):
# Here is all your code
return liten_x_lista, liten_y_lista, tid_lista
And you'd access it:
liten_x_lista, liten_y_lista, tid_lista = punkt(a,b,H,p)

How to parallelize a while loop?

I am attempting to perform a value function iteration (for an Aiyagari model). The loop I am hoping to optimize is here (EDITED TO INCLUDE MWE):
beta = 0.95
r =0.03
a_lb = -1.5
a_ub = 10
y_l = 0.9
y_h = 1.1
yGrid = [y_l, y_h]
aSz = 100
nstates = 2
V0 = np.zeros((nstates, aSz))
V1 = np.zeros((nstates, aSz))
aPol = np.zeros((nstates, aSz))
Tol = 0.0001
Iter_max = 300
err = 1.0
PI = np.matrix([[0.5, 0.5],[0.09, 0.91]])
aGrid = np.linspace(a_lb, a_ub, num = aSz)
#
#
#
Iter = 0
while (err> Tol) and (Iter < Iter_max):
V0l= intp.interp1d(aGrid, V0[0,:])
V0h= intp.interp1d(aGrid, V0[1,:])
for a_today in range(aSz):
for yix in range(nstates):
def objective(a_tomorrow):
c = yGrid[yix] + (1+r)*aGrid[a_today] - a_tomorrow
exp_cont_Val = PI[yix,0] * V0l(a_tomorrow) + PI[yix,1] * V0h(a_tomorrow)
return -(-1/c + beta* exp_cont_Val)
minima_val = opt.fminbound(objective , a_lb, min(a_ub, yGrid[yix] + (1+r) *aGrid[a_today] -0.00001))
aPol[yix, a_today] = minima_val
V1[yix, a_today] = -objective(aPol[yix, a_today])
err = (abs(V1-V0)).max()
Iter = Iter+1
V0=V1.copy()
print('Iteration ' + str( Iter) + ' with error ' + str( err))
This is a part of a much bigger loop that uses a bisection method to find a value for a variable r.
First I guess an arbitrary value for r and use it to fill the values in the c array. From my testing, this first part of the loop is very fast. For the second part (given above) I think the opt.fminbound has the most overhead. I tried to use jit, but I kept getting error messages, I would appreciate any insight.

Python Filtering a Point Cloud with PhotoScan Based on a Threshold Value - basic python help needed

I'm trying to implement a filter with Python to sort out the points on a point cloud generated by Agisoft PhotoScan. PhotoScan is a photogrammetry software developed to be user friendly but also allows to use Python commands through an API.
Bellow is my code so far and I'm pretty sure there is better way to write it as I'm missing something. The code runs inside PhotoScan.
Objective:
Selecting and removing 10% of points at a time with error within defined range of 50 to 10. Also removing any points within error range less than 10% of the total, when the initial steps of selecting and removing 10% at a time are done. Immediately after every point removal an optimization procedure should be done. It should stop when no points are selectable or when selectable points counts as less than 1% of the present total points and it is not worth removing them.
Draw it for better understanding:
Actual Code Under Construction (3 updates - see bellow for details):
import PhotoScan as PS
import math
doc = PS.app.document
chunk = doc.chunk
# using float with range and that by setting i = 1 it steps 0.1 at a time
def precrange(a, b, i):
if a < b:
p = 10**i
sr = a*p
er = (b*p) + 1
p = float(p)
for n in range(sr, er):
x = n/p
yield x
else:
p = 10**i
sr = b*p
er = (a*p) + 1
p = float(p)
for n in range(sr, er):
x = n/p
yield x
"""
Determine if x is close to y:
x relates to nselected variable
y to p10 variable
math.isclose() Return True if the values a and b are close to each other and
False otherwise
var is the tolerance here setted as a relative tolerance:
rel_tol is the relative tolerance – it is the maximum allowed difference
between a and b, relative to the larger absolute value of a or b. For example,
to set a tolerance of 5%, pass rel_tol=0.05. The default tolerance is 1e-09,
which assures that the two values are the same within about 9 decimal digits.
rel_tol must be greater than zero.
"""
def test_isclose(x, y, var):
if math.isclose(x, y, rel_tol=var): # if variables are close return True
return True
else:
False
# 1. define filter limits
f_ReconstUncert = precrange(50, 10, 1)
# 2. count initial point number
tiePoints_0 = len(chunk.point_cloud.points) # storing info for later
# 3. call Filter() and init it
f = PS.PointCloud.Filter()
f.init(chunk, criterion=PS.PointCloud.Filter.ReconstructionUncertainty)
a = 0
"""
Way to restart for loop!
should_restart = True
while should_restart:
should_restart = False
for i in xrange(10):
print i
if i == 5:
should_restart = True
break
"""
restartLoop = True
while restartLoop:
restartLoop = False
for count, i in enumerate(f_ReconstUncert): # for each threshold value
# count points for every i
tiePoints = len(chunk.point_cloud.points)
p10 = int(round((10 / 100) * tiePoints, 0)) # 10% of the total
f.selectPoints(i) # selects points
nselected = len([p for p in chunk.point_cloud.points if p.selected])
percent = round(nselected * 100 / tiePoints, 2)
if nselected == 0:
print("For threshold {} there´s no selectable points".format(i))
break
elif test_isclose(nselected, p10, 0.1):
a += 1
print("Threshold found in iteration: ", count)
print("----------------------------------------------")
print("# {} Removing points from cloud ".format(a))
print("----------------------------------------------")
print("# {}. Reconstruction Uncerntainty:"
" {:.2f}".format(a, i))
print("{} - {}"
" ({:.1f} %)\n".format(tiePoints,
nselected, percent))
f.removePoints(i) # removes points
# optimization procedure needed to refine cameras positions
print("--------------Optimizing cameras-------------\n")
chunk.optimizeCameras(fit_f=True, fit_cx=True,
fit_cy=True, fit_b1=False,
fit_b2=False, fit_k1=True,
fit_k2=True, fit_k3=True,
fit_k4=False, fit_p1=True,
fit_p2=True, fit_p3=False,
fit_p4=False, adaptive_fitting=False)
# count again number of points in point cloud
tiePoints = len(chunk.point_cloud.points)
print("= {} remaining points after"
" {} removal".format(tiePoints, a))
# reassigning variable to get new 10% of remaining points
p10 = int(round((10 / 100) * tiePoints, 0))
percent = round(nselected * 100 / tiePoints, 2)
print("----------------------------------------------\n\n")
# restart loop to investigate from range start
restartLoop = True
break
else:
f.resetSelection()
continue # continue to next i
else:
f.resetSelection()
print("for loop didnt work out")
print("{} iterations done!".format(count))
tiePoints = len(chunk.point_cloud.points)
print("Tiepoints 0: ", tiePoints_0)
print("Tiepoints 1: ", tiePoints)
Problems:
A. Currently I'm stuck on an endless processing because of a loop. I know it's about my bad coding. But how do I implement my objective and get away with the infinite loops? ANSWER: Got the code less confusing and updated above.
B. How do I start over (or restart) my search for valid threshold values in the range(50, 20) after finding one of them? ANSWER: Stack Exchange: how to restart a for loop
C. How do I turn the code more pythonic?
IMPORTANT UPDATE 1: altered above
Using a better range with float solution adapted from stackoverflow: how-to-use-a-decimal-range-step-value
# using float with range and that by setting i = 1 it steps 0.1 at a time
def precrange(a, b, i):
if a < b:
p = 10**i
sr = a*p
er = (b*p) + 1
p = float(p)
return map(lambda x: x/p, range(sr, er))
else:
p = 10**i
sr = b*p
er = (a*p) + 1
p = float(p)
return map(lambda x: x/p, range(sr, er))
# some code
f_ReconstUncert = precrange(50, 20, 1)
And also using math.isclose() to determine if selected points are close to the 10% selected points instead of using a manual solution through assigning new variables. This was implemented as follows:
"""
Determine if x is close to y:
x relates to nselected variable
y to p10 variable
math.isclose() Return True if the values a and b are close to each other and
False otherwise
var is the tolerance here setted as a relative tolerance:
rel_tol is the relative tolerance – it is the maximum allowed difference
between a and b, relative to the larger absolute value of a or b. For example,
to set a tolerance of 5%, pass rel_tol=0.05. The default tolerance is 1e-09,
which assures that the two values are the same within about 9 decimal digits.
rel_tol must be greater than zero.
"""
def test_threshold(x, y, var):
if math.isclose(x, y, rel_tol=var): # if variables are close return True
return True
else:
False
# some code
if test_threshold(nselected, p10, 0.1):
# if true then a valid threshold is found
# some code
UPDATE 2: altered on code under construction
Minor fixes and got to restart de for loop from beginning by following guidance from another Stack Exchange post on the subject. Have to improve the range now or alter the isclose() to get more values.
restartLoop = True
while restartLoop:
restartLoop = False
for i in range(0, 10):
if condition:
restartLoop = True
break
UPDATE 3: Code structure to achieve listed objectives:
threshold = range(0, 11, 1)
listx = []
for i in threshold:
listx.append(i)
restart = 0
restartLoop = True
while restartLoop:
restartLoop = False
for idx, i in enumerate(listx):
print("do something as printing i:", i)
if i > 5: # if this condition restart loop
print("found value for condition: ", i)
del listx[idx]
restartLoop = True
print("RESTARTING LOOP\n")
restart += 1
break # break inner while and restart for loop
else:
# continue if the inner loop wasn't broken
continue
else:
continue
print("restart - outer while", restart)

Parallel loop in python

Hi there I'm trying to run a big for loop, 239500 iterations, I have made some tests and I've found that 200 takes me 1 hour, resulting in 2 months of cpu time.
This is the loop:
for i in range(0, MonteCarlo):
print('Performing Monte Carlo ' + str(i) + '/' + str(MonteCarlo))
MCR = scramble(YearPos)
NewPos = reduce(operator.add, YearPos)
C = np.cov(VAR[NewPos, :], rowvar=0)
s, eof = eigs(C, k=neof, which='LR')
sc = (s.real / np.sum(s) * 100)**2
tcs = np.sum(sc)
MCH = sc/tcs
Hits[(MCH >= pcvar)] += 1
if (Hits >= CL).all():
print("Number of Hits is greater than 5 !!!")
break
Where np stands for numpy ans scramble stands for random.shuffle the calculations performed within the for loop are not dependent on each other.
Is there any way to do the loop in parallel, I have 12 cores and only 1 is running.... In Matlab I would make a parfor, is there any thing similar in python?
Thanks in advance