Confusing result with quadratic regression - python

So, I'm trying to fit some pairs of x,y data with a quadratic regression, a sample formula can be found at http://polynomialregression.drque.net/math.html.
Following is my code that does the regression using that explicit formula and using numpy inbuilt functions,
import numpy as np
x = [6.230825,6.248279,6.265732]
y = [0.312949,0.309886,0.306639472]
toCheck = x[2]
def evaluateValue(coeff,x):
c,b,a = coeff
val = np.around( a+b*x+c*x**2,9)
act = 0.306639472
error= np.abs(act-val)*100/act
print "Value = {:.9f} Error = {:.2f}%".format(val,error)
###### USing numpy######################
coeff = np.polyfit(x,y,2)
evaluateValue(coeff, toCheck)
################# Using explicit formula
def determinant(a,b,c,d,e,f,g,h,i):
# the matrix is [[a,b,c],[d,e,f],[g,h,i]]
return a*(e*i - f*h) - b*(d*i - g*f) + c*(d*h - e*g)
a = b = c = d = e = m = n = p = 0
a = len(x)
for i,j in zip(x,y):
b += i
c += i**2
d += i**3
e += i**4
m += j
n += j*i
p += j*i**2
det = determinant(a,b,c,b,c,d,c,d,e)
c0 = determinant(m,b,c,n,c,d,p,d,e)/det
c1 = determinant(a,m,c,b,n,d,c,p,e)/det
c2 = determinant(a,b,m,b,c,n,c,d,p)/det
evaluateValue([c2,c1,c0], toCheck)
######Using another explicit alternative
def determinantAlt(a,b,c,d,e,f,g,h,i):
return a*e*i - a*f*h - b*d*i +b*g*f + c*d*h - c*e*g # <- barckets removed
a = b = c = d = e = m = n = p = 0
a = len(x)
for i,j in zip(x,y):
b += i
c += i**2
d += i**3
e += i**4
m += j
n += j*i
p += j*i**2
det = determinantAlt(a,b,c,b,c,d,c,d,e)
c0 = determinantAlt(m,b,c,n,c,d,p,d,e)/det
c1 = determinantAlt(a,m,c,b,n,d,c,p,e)/det
c2 = determinantAlt(a,b,m,b,c,n,c,d,p)/det
evaluateValue([c2,c1,c0], toCheck)
This code gives this output
Value = 0.306639472 Error = 0.00%
Value = 0.308333580 Error = 0.55%
Value = 0.585786477 Error = 91.03%
As, you can see these are different from each other and third one is totally wrong. Now my questions are:
1. Why the explicit formula is giving slightly wrong result and how to improve that?
2. How numpy is giving so accurate result?
3. In the third case only by openning the parenthesis, how come the result changes so drastically?

So there are a few things that are going on here that are unfortunately plaguing the way you are doing things. Take a look at this code:
for i,j in zip(x,y):
b += i
c += i**2
d += i**3
e += i**4
m += j
n += j*i
p += j*i**2
You are building features such that the x values are not only squared, but cubed and fourth powered.
If you print out each of these values before you put them into the 3 x 3 matrix to solve:
In [35]: a = b = c = d = e = m = n = p = 0
...: a = len(x)
...: for i,j in zip(xx,y):
...: b += i
...: c += i**2
...: d += i**3
...: e += i**4
...: m += j
...: n += j*i
...: p += j*i**2
...: print(a, b, c, d, e, m, n, p)
...:
...:
3 18.744836 117.12356813829001 731.8283056811686 4572.738547313946 0.9294744720000001 5.807505391292503 36.28641270376207
When dealing with floating-point arithmetic and especially for small values, the order of operations does matter. What's happening here is that by fluke, the mix of both small values and large values that have been computed result in a value that is very small. Therefore, when you compute the determinant using the factored form and expanded form, notice how you get slightly different results but also look at the precision of the values:
In [36]: det = determinant(a,b,c,b,c,d,c,d,e)
In [37]: det
Out[37]: 1.0913403514223319e-10
In [38]: det = determinantAlt(a,b,c,b,c,d,c,d,e)
In [39]: det
Out[39]: 2.3283064365386963e-10
The determinant is on the order of 10-10! The reason why there's a discrepancy is because with floating-point arithmetic, theoretically both determinant methods should yield the same result but unfortunately in reality they are giving slightly different results and this is due to something called error propagation. Because there are a finite number of bits that can represent a floating-point number, the order of operations changes how the error propagates, so even though you are removing the parentheses and the formulas do essentially match, the order of operations to get to the result are now different. This article is an essential read for any software developer who deals with floating-point arithmetic regularly: What Every Computer Scientist Should Know About Floating-Point Arithmetic.
Therefore, when you're trying to solve the system with Cramer's Rule, inevitably when you divide by the main determinant in your code, even though the change is on the order of 10-10, the change is negligible between the two methods but you will get very different results because you're dividing by this number when solving for the coefficients.
The reason why NumPy doesn't have this problem is because they solve the system by least-squares and the pseudo-inverse and not using Cramer's Rule. I would not recommend using Cramer's Rule to find regression coefficients mostly due to experience and that there are more robust ways of doing it.
However to solve your particular problem, it's good to normalize the data so that the dynamic range is now centered at 0. Therefore, the features you use to construct your coefficient matrix are more sensible and thus the computational process has an easier time dealing with the data. In your case, something as simple as subtracting the data with the mean of the x values should work. As such, if you have new data points you want to predict, you must subtract by the mean of the x data first prior to doing the prediction.
Therefore at the beginning of your code, perform mean subtraction and regress on this data. I've showed you where I've modified the code given your source above:
import numpy as np
x = [6.230825,6.248279,6.265732]
y = [0.312949,0.309886,0.306639472]
# Calculate mean
me = sum(x) / len(x)
# Make new dataset that is mean subtracted
xx = [pt - me for pt in x]
#toCheck = x[2]
# Data point to check is now mean subtracted
toCheck = x[2] - me
def evaluateValue(coeff,x):
c,b,a = coeff
val = np.around( a+b*x+c*x**2,9)
act = 0.306639472
error= np.abs(act-val)*100/act
print("Value = {:.9f} Error = {:.2f}%".format(val,error))
###### USing numpy######################
coeff = np.polyfit(xx,y,2) # Change
evaluateValue(coeff, toCheck)
################# Using explicit formula
def determinant(a,b,c,d,e,f,g,h,i):
# the matrix is [[a,b,c],[d,e,f],[g,h,i]]
return a*(e*i - f*h) - b*(d*i - g*f) + c*(d*h - e*g)
a = b = c = d = e = m = n = p = 0
a = len(x)
for i,j in zip(xx,y): # Change
b += i
c += i**2
d += i**3
e += i**4
m += j
n += j*i
p += j*i**2
det = determinant(a,b,c,b,c,d,c,d,e)
c0 = determinant(m,b,c,n,c,d,p,d,e)/det
c1 = determinant(a,m,c,b,n,d,c,p,e)/det
c2 = determinant(a,b,m,b,c,n,c,d,p)/det
evaluateValue([c2,c1,c0], toCheck)
######Using another explicit alternative
def determinantAlt(a,b,c,d,e,f,g,h,i):
return a*e*i - a*f*h - b*d*i +b*g*f + c*d*h - c*e*g # <- barckets removed
a = b = c = d = e = m = n = p = 0
a = len(x)
for i,j in zip(xx,y): # Change
b += i
c += i**2
d += i**3
e += i**4
m += j
n += j*i
p += j*i**2
det = determinantAlt(a,b,c,b,c,d,c,d,e)
c0 = determinantAlt(m,b,c,n,c,d,p,d,e)/det
c1 = determinantAlt(a,m,c,b,n,d,c,p,e)/det
c2 = determinantAlt(a,b,m,b,c,n,c,d,p)/det
evaluateValue([c2,c1,c0], toCheck)
When I run this, we now get:
In [41]: run interp_test
Value = 0.306639472 Error = 0.00%
Value = 0.306639472 Error = 0.00%
Value = 0.306639472 Error = 0.00%
As some final reading for you, this is a similar problem that someone else encountered which I addressed in their question: Fitting a quadratic function in python without numpy polyfit. The summary is that I advised them not to use Cramer's Rule and to use least-squares through the pseudo-inverse. I showed them how to get exactly the same results without using numpy.polyfit. Also, using least-squares generalizes where if you have more than 3 points, you can still fit a quadratic through your points so that the model has the smallest error possible.

Related

minimal absolute value of the difference between A[i] and B[i] (array A is strictly increasing, array B is strictly decreasing)

Given two sequences A and B of the same length: one is strictly increasing, the other is strictly decreasing.
It is required to find an index i such that the absolute value of the difference between A[i] and B[i] is minimal. If there are several such indices, the answer is the smallest of them. The input sequences are standard Python arrays. It is guaranteed that they are of the same length. Efficiency requirements: Asymptotic complexity: no more than the power of the logarithm of the length of the input sequences.
I have implemented index lookup using the golden section method, but I am confused by the use of floating point arithmetic. Is it possible to somehow improve this algorithm so as not to use it, or can you come up with a more concise solution?
import random
import math
def peak(A,B):
def f(x):
return abs(A[x]-B[x])
phi_inv = 1 / ((math.sqrt(5) + 1) / 2)
def cal_x1(left,right):
return right - (round((right-left) * phi_inv))
def cal_x2(left,right):
return left + (round((right-left) * phi_inv))
left, right = 0, len(A)-1
x1, x2 = cal_x1(left, right), cal_x2(left,right)
while x1 < x2:
if f(x1) > f(x2):
left = x1
x1 = x2
x2 = cal_x1(x1,right)
else:
right = x2
x2 = x1
x1 = cal_x2(left,x2)
if x1 > 1 and f(x1-2) <= f(x1-1): return x1-2
if x1+2 < len(A) and f(x1+2) < f(x1+1): return x1+2
if x1 > 0 and f(x1-1) <= f(x1): return x1-1
if x1+1 < len(A) and f(x1+1) < f(x1): return x1+1
return x1
#value check
def make_arr(inv):
x = set()
while len(x) != 1000:
x.add(random.randint(-10000,10000))
x = sorted(list(x),reverse = inv)
return x
x = make_arr(0)
y = make_arr(1)
needle = 1000000
c = 0
for i in range(1000):
if abs(x[i]-y[i]) < needle:
c = i
needle = abs(x[i]-y[i])
print(c)
print(peak(x,y))
Approach
The poster asks about alternative, simpler solutions to posted code.
The problem is a variant of Leetcode Problem 852, where the goal is to find the peak index in a moutain array. We convert to a peak, rather than min, by computing the negative of the abolute difference. Our aproach is to modify this Python solution to the Leetcode problem.
Code
def binary_search(x, y):
''' Mod of https://walkccc.me/LeetCode/problems/0852/ to use function'''
def f(m):
' Absoute value of difference at index m of two arrays '
return -abs(x[m] - y[m]) # Make negative so we are looking for a peak
# peak using binary search
l = 0
r = len(arr) - 1
while l < r:
m = (l + r) // 2
if f(m) < f(m + 1): # check if increasing
l = m + 1
else:
r = m # was decreasing
return l
Test
def linear_search(A, B):
' Linear Search Method '
values = [abs(ai-bi) for ai, bi in zip(A, B)]
return values.index(min(values)) # linear search
def make_arr(inv):
random.seed(10) # added so we can repeat with the same data
x = set()
while len(x) != 1000:
x.add(random.randint(-10000,10000))
x = sorted(list(x),reverse = inv)
return x
# Create data
x = make_arr(0)
y = make_arr(1)
# Run search methods
print(f'Linear Search Solution {linear_search(x, y)}')
print(f'Golden Section Search Solution {peak(x, y)}') # posted code
print(f'Binary Search Solution {binary_search(x, y)}')
Output
Linear Search Solution 499
Golden Section Search Solution 499
Binary Search Solution 499

How do I write a Python code for partial fraction decomposition without using "apart"?

So I am very unexperienced with Python, I know basically nothing, and our teacher gave us the task to write a code that makes a partial fraction decomposition with this function:
I don't really know how to start or even how to define that function. I tried this at first: `
def function(x):
a = (x^4)-(3*x^2)+x+5
b = (x^11)-(3*x^10)-(x^9)+(7*x^8)-(9*x^7)+(23*x^6)-(11*x^5)-(3*x^4)-(4*x^3)-(32*x^2)-16
return a/b
But our maths script says that we need to split up the denominator and then make a system of equations out of it and solve it.
So I was thinking about defining each part of the function itself and then make a function somehow like a = 7*x and use it like f(x) = b/a^7 if this works but I don't really know. We are unfortunately not allowed to use "apart" which I think is a sympy-function?
Thank you so much in advance!
Sincerely, Phie
Addition: So after a few hours of trying I figured this. But I am very sure that this is not the way to do it. Also it tells me that variable l is not defined in z and I am sure that all the others aren't as well. I don't know what to do.
def function(x):
global a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v
a = (x^4)-(3*x^2)+x+5
b = 11
c = 10
d = 9
e = 8
f = 7
g = 6
h = 5
i = 4
j = 3
k = 2
l = x**b
m = 3*x**c
n = x**d
o = 7*x**e
p = 9*x**f
q = 23*x**g
r = 11*x**h
s = 3*x**i
t = 4*x**j
u = 32*x**k
v = 16
return a/(l-m-n+o-p+q-r-s-t-u-v)
print("We are starting the partial fraction decomposition with this
function: (x^4)-(3*x^2)+x+5 / (x^11)-(3*x^10)-(x^9)+(7*x^8)-(9*x^7)+
(23*x^6)-(11*x^5)-(3*x^4)-(4*x^3)-(32*x^2)-16")
z = l-m-n+o-p+q-r-s-t-u-v
while c >= 0:
c = c-1
z = z-l
while d >= 0:
d = d-1
z = z-m
while e >= 0:
e = e-1
z = z-n
while f >= 0:
f = f-1
z = z+o
while g >= 0:
g = g-1
z = z-p
while h >= 0:
h = h-1
z = z+q
while i >= 0:
i = i-1
z = z-r
while j >= 0:
j = j-1
z = z-s
while k >= 0:
k = k-1
z = z-t
print(z)
Since I just solved this myself, here's some input:
Let poly = function() for your function, although be careful to replace ^ with **. Include both from sympy import * and from sympy.abc import a, b, c, d, e, f, g, h, i, j, k, x.
Using factor(exp) you can find all the roots of your function, use these to define the 11 terms term_1 = a/(x-2), term_2 = b/(x2-)**2, ... , term_6 = (f*x + g)/(x**2 +1), ..., term_8 = (j*x + k)/(x**2 + 1) (you get the gist). Define your_sum = term_1 + ... + term_8, eq = Eq(your_sum, poly)
Define the variable your_sum = sum(term_1, ..., term_8), and use solve_undetermined_coeffs(eq, [a,b, ..., k], x))) to get the result.

How to find exponent of pow(a,b,c) in python

pow(a,x,c) operator in python returns (a**x)%c . If I have values of a, c, and the result of this operation, how can I find the value of x?
Additionally, this is all the information I have
pow(a,x,c) = pow(d,e,c)
Where I know the value of a,c,d, and e.
These numbers are very large (a = 814779647738427315424653119, d = 3, e = 40137673778629769409284441239, c = 1223334444555556666667777777) so I can not just compute these values directly.
I'm aware of the Carmichael's lambda function that can be used to solve for a, but I am not sure if and/or how this applies to solve for x.
Any help will be appreciated.
As #user2357112 says in the comments, this is the discrete logarithm problem, which is computationally very difficult for large c, and no fast general solution is known.
However, for small c there are still some things you can do. Given that a and c are coprime, there is an exponent k < c such that a^k = 1 mod c, after which the powers repeat. Let b = a^x. So, if you brute force it by calculating all powers of a until you get b, you'll have to loop at most c times:
def do_log(a, b, c):
x = 1
p = a
while p != b and p != 1:
x += 1
p *= a
p %= c
if p == b:
return x
else:
return None # no such x
If you run this calculation multiple times with the same a, you can do even better.
# a, c constant
p_to_x = {1: 0}
x = 1
p = a
while p != 1:
p_to_x[p] = x
x += 1
p *= a
p %= c
def do_log_a_c(b):
return p_to_x[b]
Here a cache is made in a loop running at most c times and the cache is accessed in the log function.

Better approach to find Nth term of the sequence

Given :
I : a positive integer
n : a positive integer
nth Term of sequence for input = I :
F(I,1) = (I * (I+1)) / 2
F(I,2) = F(I,1) + F(I-1,1) + F(I-2,1) + .... F(2,1) + F(1,1)
F(I,3) = F(I,2) + F(I-1,2) + F(I-2,2) + .... F(2,2) + F(2,1)
..
..
F(I,n) = F(I,n-1) + F(I-1,n-1) + F(I-2,n-1) + .... F(2,n-1) + F(1,n-1)
nth term --> F(I,n)
Approach 1 : Used recursion to find the above :
def recursive_sum(I, n):
if n == 1:
return (I * (I + 1)) // 2
else:
return sum(recursive_sum(j, n - 1) for j in range(I, 0, -1))
Approach 2 : Iteration to store reusable values in a dictionary. Used this dictionary to get the nth term.:
def non_recursive_sum_using_data(I, n):
global data
if n == 1:
return (I * (I + 1)) // 2
else:
return sum(data[j][n - 1] for j in range(I, 0, -1))
def iterate(I,n):
global data
data = {}
i = 1
j = 1
for i in range(n+1):
for j in range(I+1):
if j not in data:
data[j] = {}
data[j][i] = recursive_sum(j,i)
return data[I][n]
The recursion approach is obviously not efficient due to maximum recursion depth. Also the next approach's time and space complexity will be poor.
Is there better way to recurse ? or a different approach than recursion ?
I am curious if we can find a formula for nth term.
You could just cache your recursive results:
from functools import lru_cache
#lru_cache(maxsize=None)
def recursive_sum(I, n):
if n == 1:
return (I * (I + 1)) // 2
return sum(recursive_sum(j, n - 1) for j in range(I, 0, -1))
That way you can get the readability and brevity of the recursive approach without most of the performance issues since the function is only called once for each argument combination (I, n).
Using the usual binomial(n,k) = n!/(k!*(n-k)!), you have
F(I,n) = binomial(I+n, n+1).
Then you can choose the method you like most to compute binomial coefficients.
Here an example:
def binomial(n, k):
numerator = denominator = 1
t = max(k, n-k)
for low,high in enumerate(range(t+1, n+1), 1):
numerator *= high
denominator *= low
return numerator // denominator
def F(I,n): return binomial(I+n, n+1)
The formula for the nth term of the sequence is the one you have already mentioned.
Also rightly so you have identified that it will lead to an inefficient algorithm and stack overflow.
You can look into dynamic programming approach where u calculate F(I,N) just once and just reuse the value.
For example this is how the fibonacci seq is calculated.
[just-example] https://www.geeksforgeeks.org/program-for-nth-fibonacci-number/
You need to find the same pattern and cache the value
I have an example for this here in this small code written in golang
https://play.golang.org/p/vRi-QMj7z2v
the standard DP
One can do a (tiny) bit of math to rewrite your function:
F(i,n) = sum_{k=0}^{i-1} F(i-k, n-1) = sum_{k=1}^{i} F(k, n-1)
Now notice, that if you consider a matrix F_{ixn}, to compute F(i,n) we just need to add the elements of the previous column.
x----+---
| + |
|----+ |
|----+-F(i,n)
We conclude that we can build the first layer (aka column). Then the second one. And so forth until we get to the n-th layer.
Finally we take the last element of our final layer which is F(i,n)
The computation time is about O(I*n)
More math based but faster
An other way is to consider our layer as a vector variable X.
We can write the recurrence relation as
X_n = MX_{n-1}
where M is a triangular matrix with 1 in the lower part.
We then want to compute the general term of X_n so we want to compute M^n.
By following Yves Daoust
(I just copy from the link above)
Coefficients should be indiced _{n+1} and _n, but here it is _1 and '' for readability
Moreover the matrix is upper triangular but we can just take the transpose afterwards...
a_1 b_1 c_1 d_1 1 1 1 1 a b c d
a_1 b_1 c_1 = 0 1 1 1 * 0 a b c
a_1 b_1 0 0 1 1 0 0 a b
a_1 0 0 0 1 0 0 0 a
by going from last row to first:
a = 1
from b_1 = a+b = 1 + b = n, b = n
from c_1 = a+b+c = 1+n+c, c = n(n+1)/2
from d_1 = a+b+c+d = 1+n+n(n+1)/2 +d, d = n(n+1)(n+2)/6
I have not proved it but I hint that e_1 = n(n+1)(n+2)(n+3)/24 (so basically C_n^k)
(I think the proof lies more in the fact that F(i,n) = F(i,n-1) + F(i-1,n) )
More generally instead of taking variables a,b,c... but X_n(0), X_n(1)...
X_n(0) = 1
X_n(i) = n*...*(n+i-1) / i!
And by applying recusion for computing X:
X_n(0) = 1
X_n(i) = X_n(i-1)*(n+i-1)/i
Finally we deduce F(i,n) as the scalar product Y_{n-1} * X_1 where Y_n is the reversed vector of X_n and X_1(n) = n*(n+1)/2
from functools import lru_cache
#this is copypasted from schwobaseggl
#lru_cache(maxsize=None)
def recursive_sum(I, n):
if n == 1:
return (I * (I + 1)) // 2
return sum(recursive_sum(j, n - 1) for j in range(I, 0, -1))
def iterative_sum(I,n):
layer = [ i*(i+1)//2 for i in range(1,I+1)]
x = 2
while x <= n:
next_layer = [layer[0]]
for i in range(1,I):
#we don't need to compute all the sum everytime
#take the previous sum and add it the new number
next_layer.append( next_layer[i-1] + layer[i] )
layer = next_layer
x += 1
return layer[-1]
def brutus(I,n):
if n == 1:
return I*(I+1)//2
X_1 = [ i*(i+1)//2 for i in range(1, I+1)]
X_n = [1]
for i in range(1, I):
X_n.append(X_n[-1] * (n-1 + i-1) / i )
X_n.reverse()
s = 0
for i in range(0, I):
s += X_1[i]*X_n[i]
return s
def do(k,n):
print('rec', recursive_sum(k,n))
print('it ', iterative_sum(k,n))
print('bru', brutus(k,n))
print('---')
do(1,4)
do(2,1)
do(3,2)
do(4,7)
do(7,4)

Variable values not getting into inner for loop

My inner for loop is not using new values from the outer loop.
What's wrong, and how do I fix it?
import numpy as np
a = 0.0000001
b = 15.
d = 0.1
TOL = 1.0e-6
a1 = []
dd = 0.1
da1 = []
for i in range(0,10):
def f(v):
return np.cosh(d * v) - (1./v) * np.sinh(d * v) - 1.
FA = f(a)
FB = f(b)
for I in range(0,1000):
p = a + (b - a) / 2.0
FP = f(p)
if FA == 0 or (b - a)/2.0 < TOL:
break
I = I + 1
if FA * FP > 0:
a = p
FA = FP
if FA * FP < 0:
b = p
a1.append(p)
da1.append(d)
d = d + dd
print a1
print da1
Here is a second implementation. Variable d shows new values, but the inner loop keeps giving me the same result result, like it is not registering the new d value.
import numpy as np
a = 0.00001
a1 = []
dd = 0.1
da = 1.e-5
d = 0.1
yvs=[]
ds = []
EE = []
while d <= 1.:
dnew = d
print dnew
for i in range(0,1000000):
dnew = d
yv = np.cosh(dnew * a) - (1./a) * np.sinh(dnew * a) - 1.
yvs.append(yv)
a = a + da
a1.append(a)
i = i + 1
for ii in range(0,999999):
As = (a1[ii]+a1[ii+1])/2.
E = -1. * As**2
if yvs[ii]*yvs[ii+1] < 0:
EE.append(E)
print As, E
ii = ii + 1
d = dnew + dd
I deleted my earlier answer; it's not the main problem you're having.
You traced the wrong values: d and dnew do, indeed, change. However, they are not part of the data flow for the values you're worried about.
In the upper program, d depends exclusively on its starting value and increment value, both of them 0.1, and dd doesn't change. p depends exclusively on the values of a and b, which also don't change.
Yes, you do some nice work to compute FA, FB, FP -- but then you hit the bottom of the loop, you don't save them anywhere, and then you overwrite tehm on the next loop.
If the lower program, you have the same problem with As and E: you never change the parameters on which they depend (that's all in yvs, which you never print out), so the outputs are the same on every loop.
Since you are using one- and two-letter variables and haven't documented your code, I don't have a good idea of how to fix this: I have little idea what your program is supposed to do, although it appears to want to converge some computational series.

Categories