We Keep Coding

Error in newton raphson method finding root

I was trying to use the newton raphson method to compute the derivative of a function and I got the following error:
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
acc = 10**-4
x = sym.Symbol('x')
def p(x): #define the function
return 924*x**6 - 2772*x**5 + 3150*x**4 - 1680*x**3 + 420*x**2 - 42*x + 1
p_prime = sym.diff(p(x))
def newton(p,p_prime, acc, start_val):
x= start_val
delta = p(x)/p_prime(x)
while abs(delta) > acc:
delta = p(x)/p_prime(x)
print(abs(delta))
x =x- delta;
return round(x, acc);
a = newton(p, p_prime,-4, 0)
The error was:
delta = p(x)/p_prime(x)
TypeError: 'Add' object is not callable

There are a few mistakes in your code, correcting them as pointed out in the following modified code snippet, it works fine:
def p_prime(xval): # define as a function and substitute the value of x
return sym.diff(p(x)).subs(x, xval)
def newton(p, p_prime, prec, start_val): # rename the output precision variable
x = start_val # otherwise it's shadowing acc tolerance
delta = p(x) / p_prime(x) # call the function p_prime()
while abs(delta) > acc:
delta = p(x) / p_prime(x)
x = x - delta
return round(x, prec) # return when the while loop terminates
a = newton(p, p_prime,4, 0.1)
a
# 0.0338
The following animation shows how Newton-Raphson converges to a root of the polynomial p(x).

Your main problem is to call something which is not callable.
Functions are callable
Sympy expressions are not callable
import sympy as sym
def f(x):
return x**2
x = sym.Symbol("x")
g = 2*x
print(callable(f)) # True
print(callable(g)) # False
print(f(0)) # print(0)
print(g(0)) # error
So, in your case
def p(x):
return 924*x**6 - 2772*x**5 + 3150*x**4 - 1680*x**3 + 420*x**2 - 42*x + 1
print(p(0)) # p is callable, gives the result 1
print(p(1)) # p is callable, gives the result 1
print(p(2)) # p is callable, gives the result 8989
print(callable(p)) # True
And now, if you use a symbolic variable from sympy you get:
x = sym.Symbol("x")
myp = p(x)
print(callable(myp)) # False
print(myp(0)) # gives an error, because myp is an expression, which is not callable
So, if you use diff to get the derivative of the function, you will get a sympy expression. You must transform this expression to a callable function. To do it, you can use the lambda function:
def p(x):
return 924*x**6 - 2772*x**5 + 3150*x**4 - 1680*x**3 + 420*x**2 - 42*x + 1
x = sym.Symbol("x")
myp = p(x)
mydpdx = sym.diff(myp, x) # get the derivative, once myp is an expression
dpdx = lambda x0: mydpdx.subs(x, x0) # dpdx is callable
print(dpdx(0)) # gives '-42'
Another way to make a sympy expression callable is to use lambdify from sympy:
dpdx = sym.lambdify(x, mydpdx)

In sympy functions usually are represented as expressions. Therefore, sym.diff() returns an expression and not a callable function. It makes sense to also represent p as an expression involving x.
To "call" a function p on x, p.subs(x, value) is used. To get a numeric answer, use p.subs(x, value).evalf().
import sympy as sym
acc = 10 ** -4
x = sym.Symbol('x')
p = 924 * x ** 6 - 2772 * x ** 5 + 3150 * x ** 4 - 1680 * x ** 3 + 420 * x ** 2 - 42 * x + 1
p_prime = sym.diff(p)
def newton(p, p_prime, acc, start_val):
xi = start_val
delta = sym.oo
while abs(delta) > acc:
delta = (p / p_prime).subs(x, xi).evalf()
print(xi, delta)
xi -= delta
return xi
a = newton(p, p_prime, 10**-4, 0)
Intermediate values:
0 -0.0238095238095238
0.0238095238095238 -0.00876459050881560
0.0325741143183394 -0.00117130331903862
0.0337454176373780 -1.98196463560775e-5
0.0337652372837341
PS: To plot a function represented as an expression:
sym.plot(p, (x, -0.03, 1.03), ylim=(-0.5, 1.5))

Find maxima for a negative parabolic equation

I have the following negative quadratic equation
-0.03402645959398278x^{2}+156.003469x-178794.025
I want to know if there is a straight way (using numpy/scipy libraries or any other) to get the value of x when the slope of the derivative is zero (the maxima). I'm aware I could:
change the sign of the equation and apply the scipy.optimize.minima method or
using the derivative of the equation so I can get the value when the slope is zero
For instance:
from scipy.optimize import minimize
quad_eq = np.poly1d([-0.03402645959398278, 156.003469, -178794.025])
############SCIPY####################
neg_quad_eq = np.poly1d(np.negative(quad_eq))
fit = minimize(neg_quad_eq, x0=15)
slope_zero_neg = fit.x[0]
maxima = np.polyval(quad_eq, slope_zero_neg)
print(maxima)
##################numpy######################
import numpy as np
first_dev = np.polyder(quad_eq)
slope_zero = first_dev.r
maxima = np.polyval(quad_eq, slope_zero)
print(maxima)
Is there any straight way to get the same result?
print(maxima)

You don't need all that code... The first derivative of a x^2 + b x + c is 2a x + b, so solving 2a x + b = 0 for x yields x = -b / (2a) that is actually the maximum you are searching for
import numpy as np
import matplotlib.pyplot as plt
def func(x, a=-0.03402645959398278, b=156.003469, c=-178794.025):
result = a * x**2 + b * x + c
return result
def func_max(a=-0.03402645959398278, b=156.003469, c=-178794.025):
maximum_x = -b / (2 * a)
maximum_y = a * maximum_x**2 + b * maximum_x + c
return maximum_x, maximum_y
x = np.linspace(-50000, 50000, 100)
y = func(x)
mx, my = func_max()
print('maximum:', mx, my)
maximum: 2292.384674478263 15.955750522436574
and verify
plt.plot(x, y)
plt.axvline(mx, color='r')
plt.axhline(my, color='r')

How to use positional arguments appropriately for a complex problem

I am revisiting a school project, which I did not complete to my satisfaction. Namely, I wrote an algorithm that takes an ALMOST arbitrary size set of equations and solves them iteratively. The problem being the "almost" part. Essentially, it must have at least two equations, and will not solve for a single one. This is because, I believe, that I don't understand how to use positional arguments correctly.
Below, in the main method, I define two functions y_prime and z_prime. If I pass them both, I get a beautiful graph of my solutions. But, if I only pass y_prime along with its initial conditions and the solution vector to the rungekutta() function, things go haywire:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
def rungekutta(dt, y, t, *funcs):
"""
The following code was written in order to
reproduce the classic 4th order Runge-Kutta numerical
method of solving a system of differential equations.
The aim was to not only apply this to the budworm deforestation
model developed by Ludwig et al, but also to create an algorithm
that is generic enough to accept a wide range of ODEs and
systems of ODEs.
:param dt: time step "Delta t"
:param y: The solution vector at the last time step
:param t: The time at the last time step
:param funcs: the vector field dy/dt = f(t,y)
:return: The solution vector for the next time step
"""
k1 = [dt * f(*y, t) for f in funcs]
args = [y_n + 0.5 * k_1 for y_n, k_1 in zip((*y, t), (*k1, dt))]
k2 = [dt * f(*args) for f in funcs]
args = [y_n + 0.5 * k_2 for y_n, k_2 in zip((*y, t), (*k2, dt))]
k3 = [dt * f(*args) for f in funcs]
args = [y_n + k_3 for y_n, k_3 in zip((*y, t), (*k3, dt))]
k4 = [dt * f(*args) for f in funcs]
return [y_n + (k_1 + 2 * k_2 + 2 * k_3 + k_4) / 6 for y_n, k_1, k_2, k_3, k_4 in
zip(y, k1, k2, k3, k4)]
if __name__ == '__main__':
def y_prime(y, z, t):
return -t * y
def z_prime(y, z, t):
return z
t_0 = -10
t_n = 10
dt = .05
steps = int((t_n - t_0) / dt)
y_soln = [0] * steps
z_soln = [0] * steps
time = np.arange(t_0, t_n, dt)
y_soln[0] = 1.928749848e-22
z_soln[0] = .0000453999297625
for i in np.arange(1, steps):
y_soln[i] = rungekutta(dt, y_soln[i-1], time[i-1], y_prime)
The first error I received, when trying to pass a single equation was:
Traceback (most recent call last):
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 57, in <module>
y_soln[i] = rungekutta(dt, y_soln[i-1], time[i-1], y_prime, z_prime)
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 23, in rungekutta
k1 = [dt * f(*y, t) for f in funcs]
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 23, in <listcomp>
k1 = [dt * f(*y, t) for f in funcs]
TypeError: y_prime() argument after * must be an iterable, not float
This was because, I think, I have "y_soln" as a positional argument, but now there is only one and it is no longer iterable. So, I made it a tuple of 1 when I passed it in the main method:
for i in np.arange(1, steps):
y_soln[i] = rungekutta(dt, (y_soln[i-1],), time[i-1], y_prime)
That bit me in the butt, however, because now I am passing a tuple into my y_prime equation, when what it really needs is a float:
Traceback (most recent call last):
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 57, in <module>
y_soln[i] = rungekutta(dt, (y_soln[i-1],), time[i-1], y_prime)
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 23, in rungekutta
k1 = [dt * f(*y, t) for f in funcs]
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 23, in <listcomp>
k1 = [dt * f(*y, t) for f in funcs]
File "C:/Users/wesle/PycharmProjects/Budworms/RK4v2.py", line 38, in y_prime
return -t * y
TypeError: can't multiply sequence by non-int of type 'numpy.float64'
My only work-around so far has been to solve an extra, random equation like $y= y'$ in addition to whatever equation I'm interested in. This seems pretty inefficient though.
So, it seems like I'm damned if I do, or damned if I don't. Is there any remedy to this?
EDIT If you want to see the code actually work, replace this:
for i in np.arange(1, steps):
y_soln[i] = rungekutta(dt, (y_soln[i-1],), time[i-1], y_prime)
with the instance where I pass both equations and their solution vectors to the function:
for i in np.arange(1, steps):
y_soln[i], z_soln[i] = rungekutta(dt, (y_soln[i-1], z_soln[i-1]), time[i-1], y_prime, z_prime)

My solution ended up being to convert all lists to numpy arrays, which allowed me to take advantage of the built in element-wise scalar addition and multiplication. This made computing the "k" values much less cumbersome and convoluted:
def rk4(dt, t, field, y_0):
"""
:param dt: float - the timestep
:param t: array - the time mesh
:param field: method - the vector field y' = f(t, y)
:param y_0: array - contains initial conditions
:return: ndarray - solution
"""
# Initialize solution matrix. Each row is the solution to the system
# for a given time step. Each column is the full solution for a single
# equation.
y = np.asarray(len(t) * [y_0])
for i in np.arange(len(t) - 1):
k1 = dt * field(t[i], y[i])
k2 = dt * field(t[i] + 0.5 * dt, y[i] + 0.5 * k1)
k3 = dt * field(t[i] + 0.5 * dt, y[i] + 0.5 * k2)
k4 = dt * field(t[i] + dt, y[i] + k3)
y[i + 1] = y[i] + (k1 + 2 * k2 + 2 * k3 + k4) / 6
return y

Explicit Euler method doesn't behave how I expect

I have implemented the following explicit euler method in python:
def explicit_euler(df, x0, h, N):
"""Solves an ODE IVP using the Explicit Euler method.
Keyword arguments:
df - The derivative of the system you wish to solve.
x0 - The initial value of the system you wish to solve.
h - The step size.
N - The number off steps.
"""
x = np.zeros(N)
x[0] = x0
for i in range(0, N-1):
x[i+1] = x[i] + h * df(x[i])
return x
Following the article on wikipedia I can plot the function and verify that I get the same plot: . I believe that here the method I have written is working correctly.
Next I tried to use it to solve the last system given on this page and instead of the plot shown there I obtain this:
I am not sure why my plot doesn't match the one shown on the webpage. The explicit euler method seems to work fine when I use it to solve systems where the slope doesn't change, but for an oscillating function it never seems to mimic it at all. Not even showing the expected error gain as indicated on the linked webpage. I am not sure what is wrong with the method I have implemented.
Here is the code used for plotting and the derivative:
def g(t):
return -0.5 * np.exp(t * 0.5) * np.sin(5 * t) + 5 * np.exp(t * 0.5)
* np.cos(5 * t)
h = 0.001
x0 = 0
tn = 4
N = int(tn / h)
x = ee.explicit_euler(f, x0, h, N)
t = np.arange(0, tn, h)
fig = plt.figure()
plt.plot(t, x, label="Explicit Euler")
plt.plot(t, (np.exp(0.5 * t) * np.sin(5 * t)), label="Analytical
solution")
#plt.plot(t, np.exp(0.5 * t), label="Analytical solution")
plt.xlabel('Timesteps t')
plt.ylabel('x(t)=e^(0.5*t) * sin(5*t)')
plt.legend()
plt.grid()
plt.show()
Edit:
As requested here is the current equation I am applying the method to:
y'-y=-0.5*e^(t/2)*sin(5t)+5e^(t/2)*cos(5t)
Where y(0)=0.
I would like to make clear however that this behaviour doesn't occur just for this equation but all equations where the slope has a change in sign, or oscillating behaviour.
Edit 2:
Ok thanks. Yes the code below does indeed work. But I have one further question. In the simple example I had for the exponential function, I had defined a method:
def f(x):
return x
for the system f'(x)=x. This gave the output of my first graph which looks correct. I then defined another function:
def k(x):
return cos(x)
for the system f'(x)=cos(x), this does not give expected output. But when I change the function definition to
def k(t, x):
return cos(t)
I get the expected output. If I change my function
def f(t, x):
return t
I get an incorrect output. Am I always actually evaluating the function at a time step and is it just by chance for the system x'=x that at each time step the value is just the value of x?
I had understood that the Euler method used the value of the previously calculated value in order to get the next value. But if I run code for my function k(x)=cos(x), I get output pictured below, which must be incorrect. This now uses the updated code you provided.
def k(t, x):
return np.cos(x)
h = 0.1 # Step size
x0 = (0, 0) # Initial point of iteration
tn = 10 # Time step to iterate to
N = int(tn / h) # Number of steps
x = ee.explicit_euler(k, x0, h, N)
t = np.arange(0, tn, h)

The problem is that you have incorrectly raised the function g, you want to solve the equation:
From where we observe that:
y' = y -0.5*e^(t/2)*sin(5t)+5e^(t/2)*cos(5t)
Then we define the function f(t, y) = y -0.5*e^(t/2)*sin(5t)+5e^(t/2)*cos(5t) as:
def f(t, y):
return y -0.5 * np.exp(t * 0.5) * np.sin(5 * t) + 5 * np.exp(t * 0.5) * np.cos(5 * t)
The initial point of iteration is f0=(t(0), y(0)):
f0 = (0, 0)
Then from Euler's equations:
def explicit_euler(df, x0, h, N):
"""Solves an ODE IVP using the Explicit Euler method.
Keyword arguments:
df - The derivative of the system you wish to solve.
x0 - The initial value of the system you wish to solve.
h - The step size.
N - The number off steps.
"""
x = np.zeros(N)
t, x[0] = x0
for i in range(0, N-1):
x[i+1] = x[i] + h * df(t ,x[i])
t += h
return x
Complete Code:
def explicit_euler(df, x0, h, N):
"""Solves an ODE IVP using the Explicit Euler method.
Keyword arguments:
df - The derivative of the system you wish to solve.
x0 - The initial value of the system you wish to solve.
h - The step size.
N - The number off steps.
"""
x = np.zeros(N)
t, x[0] = x0
for i in range(0, N-1):
x[i+1] = x[i] + h * df(t ,x[i])
t += h
return x
def df(t, y):
return -0.5 * np.exp(t * 0.5) * np.sin(5 * t) + 5 * np.exp(t * 0.5) * np.cos(5 * t) + y
h = 0.001
f0 = (0, 0)
tn = 4
N = int(tn / h)
x = explicit_euler(df, f0, h, N)
t = np.arange(0, tn, h)
fig = plt.figure()
plt.plot(t, x, label="Explicit Euler")
plt.plot(t, (np.exp(0.5 * t) * np.sin(5 * t)), label="Analytical solution")
#plt.plot(t, np.exp(0.5 * t), label="Analytical solution")
plt.xlabel('Timesteps t')
plt.ylabel('x(t)=e^(0.5*t) * sin(5*t)')
plt.legend()
plt.grid()
plt.show()
Screenshot:
Dump y' and what is on the right side is what you should place in the df function.
We will modify the variables to maintain the same standard for the variables, and will y be the dependent variable, and t the independent variable.
Equation 2: In this case the equation f'(x)=cos(x) will be rewritten to:
y'=cos(t)
Then:
def df(t, y):
return np.cos(t)
In conclusion, if we have an equation of the following form:
y' = f(t, y)
Then:
def df(t, y):
return f(t, y)

how can I do a maximum likelihood regression using scipy.optimize.minimize

How can I do a maximum likelihood regression using scipy.optimize.minimize? I specifically want to use the minimize function here, because I have a complex model and need to add some constraints. I am currently trying a simple example using the following:
from scipy.optimize import minimize
def lik(parameters):
m = parameters[0]
b = parameters[1]
sigma = parameters[2]
for i in np.arange(0, len(x)):
y_exp = m * x + b
L = sum(np.log(sigma) + 0.5 * np.log(2 * np.pi) + (y - y_exp) ** 2 / (2 * sigma ** 2))
return L
x = [1,2,3,4,5]
y = [2,3,4,5,6]
lik_model = minimize(lik, np.array([1,1,1]), method='L-BFGS-B', options={'disp': True})
When I run this, convergence fails. Does anyone know what is wrong with my code?
The message I get running this is 'ABNORMAL_TERMINATION_IN_LNSRCH'. I am using the same algorithm that I have working using optim in R.

Thank you Aleksander. You were correct that my likelihood function was wrong, not the code. Using a formula I found on wikipedia I adjusted the code to:
import numpy as np
from scipy.optimize import minimize
def lik(parameters):
m = parameters[0]
b = parameters[1]
sigma = parameters[2]
for i in np.arange(0, len(x)):
y_exp = m * x + b
L = (len(x)/2 * np.log(2 * np.pi) + len(x)/2 * np.log(sigma ** 2) + 1 /
(2 * sigma ** 2) * sum((y - y_exp) ** 2))
return L
x = np.array([1,2,3,4,5])
y = np.array([2,5,8,11,14])
lik_model = minimize(lik, np.array([1,1,1]), method='L-BFGS-B')
plt.scatter(x,y)
plt.plot(x, lik_model['x'][0] * x + lik_model['x'][1])
plt.show()
Now it seems to be working.
Thanks for the help!