Consider a simple problem:
max log(x)
subject to x >= 1e-4
To solve the problem with scipy.optimize.minimize:
import numpy as np
from scipy.optimize import minimize
from math import log
def func(x):
return log(x[0])
def func_deriv(x):
return np.array([1 / x[0]])
cons = ({'type': 'ineq',
'fun' : lambda x: x[0] - 1e-4,
'jac' : lambda x: np.array([1])})
minimize(func, [1.0], jac=func_deriv, constraints=cons, method='SLSQP')
The script encounters ValueError because log(x) is evaluated with negative x. It seems that the function value is evaluated even if the constraint is not satisfied.
I understand that using bounds in minimize() could avoid the problem, but this is just a simplification of my original problem. In my original problem, the constraint x >= 1e-4 cannot be represented easily as bounds of x, but rather of the form g(x) >= C, so bounds wouldn't help.
If we only care about the function value with x > ε, it is possible to define a safe function extending the domain.
Take the log function as an example. It is possible to extend log with another cubic function, while making the bridge point ε smooth:
safe_log(x) = log(x) if x > ε else a * (x - b)**3
To calculate a and b, we have to satisfy:
log(ε) = a * (ε - b)**3
1 / ε = 3 * a * (ε - b)**2
Hence the safe_log function:
eps = 1e-3
def safe_log(x):
if x > eps:
return log(x)
logeps = log(eps)
a = 1 / (3 * eps * (3 * logeps * eps)**2)
b = eps * (1 - 3 * logeps)
return a * (x - b)**3
And it looks like this:
Related
So i have this newton optimation problem where i must found the value f'(x) and f''(x) where x = 2.5 and the f = 2 * sin(x) - ((x)**2/10) for calculating, but i tried using sympy and np.diff for the First and Second Derivative but no clue, cause it keep getting error so i go back using manual derivate, Any clue how to derivative the function f with help of other library, Here's the code
def Newton(x0):
x = x0
f = lambda x : 2 * np.sin (x) - ((x)**2/10)
f_x0 = f(x0)
#First Derivative
f1 = lambda x : 2 * np.cos (x) - ((x)/5)
f_x1 = f1(x0)
#Second Derivative
f2 = lambda x : -2 * np.sin (x) - (1/5)
f_x2 = f2(x0)
x1 = x0 - (f_x1/f_x2)
x0 = x1
return x,f_x0,f_x1,f_x2,x0
finding first and second derivative without the manual way.
In your case, the derivates can be calculated using the scipy library as follows:
from scipy.misc import derivative
def f(x):
return 2 * sin(x) - ((x)**2/10)
print("First derivative:" , derivative(f, 2.5, dx=1e-9))
print("Second derivative", derivative(f, 2.5, n=2, dx=0.02))
Here the first and second derivative is calculated for your function at x=2.5.
The same can be done with the sympy library and some may find this easier than the above method.
from sympy import *
x = Symbol('x')
y = 2 * sin(x) - ((x)**2/10) #function
yprime = y.diff(x) #first derivative function
ydoubleprime = y.diff(x,2) #second derivative function
f_first_derivative = lambdify(x, yprime)
f_second_derivative = lambdify(x, ydoubleprime)
print("First derivative:" , f_first_derivative(2.5))
print("Second derivative",f_second_derivative(2.5))
I have the following function to_minimize which should be equal to the log-likelihood of a dataset for a Weibull distribution, truncated from the left at d.
import numpy as np
from scipy.optimize import minimize
def to_minimize(args, data, d=1):
theta, tau = args
n = len(data)
if tau <= 0 or theta <= 0:
pass
term1 = n * (np.log(tau) - tau * np.log(theta) - (-d / theta) ** tau)
term2 = 0
for x in data:
term2 += (tau - 1) * np.log(x) + (-x / theta) ** tau
return term1 + term2
data = numpy.random.rand(100)
weibull = minimize(lambda args: -to_minimize(args, data),
x0=np.array((1., 1.)), bounds=np.array([(1e-15, 10), (1e-15, 10)]))
As far as I can tell, the only thing that should cause an error of the form
RuntimeWarning: invalid value encountered in double_scalars
should be if tau or theta are 0. But the bounds on those parameters is specifically above 0 so why does my optimization routine crash?
After calling np.seterr(all='raise') and debugging some more, I noticed that I had an error in my calculations. The - in the exponential function has to be applied after the power. Otherwise it will try to take the root of a negative number which won't work for obvious reasons.
I am running scipy.optimize.minimize trying to maximize the likelihood for left-truncated data on a Gompertz distribution. Since the data is left-truncated at 1, I get this likelihood:
# for a single point x_i, the left-truncated log-likelihood is:
# ln(tau) + tau*(ln(theta) - ln(x_i)) - (theta / x_i) ** tau - ln(x_i) - ln(1 - exp(-(theta / d) ** tau))
def to_minimize(args, data, d=1):
theta, tau = args
if tau <= 0 or theta <= 0 or theta / d < 0 or np.exp(-(theta / d) ** tau) >= 1:
print('ERROR')
term1 = len(data) * (np.log(tau) + tau * np.log(theta) - np.log(1 - np.exp(-(theta / d) ** tau)))
term2 = 0
for x in data:
term2 += (-(tau + 1) * np.log(x)) - (theta / x) ** tau
return term1 + term2
This will fail in all instances where the if statement is true. In other words, tau and theta have to be strictly positive, and theta ** tau must be sufficiently far away from 0 so that np.exp(-theta ** tau) is "far enough away" from 1, since otherwise the logarithm will be undefined.
These are the constraints which I thus defined. I used the notation with a dict instead of a NonlinearConstraints object since it seems that this methods accepts strict inequality (np.exp(-x[0] ** x[1]) must be strictly less than 1). Maybe I have misunderstood the documentation on this.
def constraints(x):
return [1 - np.exp(-(x[0]) ** x[1])]
To maximize the likelihood, I minimize the negative likelihood.
opt = minimize(lambda args: -to_minimize(args, data),
x0=np.array((1, 1)),
constraints={'type': 'ineq', 'fun': constraints},
bounds=np.array([(1e-15, 10), (1e-15, 10)]))
As I take it, the two arguments should then never be chosen in a way such that my code fails. Yet, the algorithm tries to move theta very close to its lower bound and tau very close to its upper bound so that the logarithm becomes undefined.
What makes my code fail?
Both forms of constraints, i.e. NonlinearConstraint and dict constraints don't support strict inequalities. Typically, one therefore uses g(x) >= c + Ɛ to model the strict inequality g(x) > c, where Ɛ is a sufficiently small number.
Note also that it is not guaranteed that each iteration lies inside the feasible region. Internally, most of the methods try to bring it back into the feasible region by a simple clipping of the bounds. In cases where this doesn't work, you can try NonlinearConstraints keep_feasible option and then use the trust-constr method:
import numpy as np
from scipy.optimize import NonlinearConstraint, minimize
def con_fun(x):
return 1 - np.exp(-(x[0]) ** x[1])
# 1.0e-8 <= con_fun <= np.inf
con = NonlinearConstraint(con_fun, 1.0e-8, np.inf, keep_feasible=True)
x0 = np.array((1., 1.))
bounds = np.array([(1e-5, 10), (1e-5, 10)])
opt = minimize(lambda args: -to_minimize(args, data),
x0=x0, constraints=(con,),
bounds=bounds, method="trust-constr")
I have the same problem as in this question but don't want to add only one but several constraints to the optimization problem.
So e.g. I want to maximize x1 + 5 * x2 with the constraints that the sum of x1 and x2 is smaller than 5 and x2 is smaller than 3 (needless to say that the actual problem is far more complicated and cannot just thrown into scipy.optimize.minimize as this one; it just serves to illustrate the problem...).
I can to an ugly hack like this:
from scipy.optimize import differential_evolution
import numpy as np
def simple_test(x, more_constraints):
# check wether all constraints evaluate to True
if all(map(eval, more_constraints)):
return -1 * (x[0] + 5 * x[1])
# if not all constraints evaluate to True, return a positive number
return 10
bounds = [(0., 5.), (0., 5.)]
additional_constraints = ['x[0] + x[1] <= 5.', 'x[1] <= 3']
result = differential_evolution(simple_test, bounds, args=(additional_constraints, ), tol=1e-6)
print(result.x, result.fun, sum(result.x))
This will print
[ 1.99999986 3. ] -16.9999998396 4.99999985882
as one would expect.
Is there a better/ more straightforward way to add several constraints than using the rather 'dangerous' eval?
An example is something like this::
additional_constraints = [lambda(x): x[0] + x[1] <= 5., lambda(x):x[1] <= 3]
def simple_test(x, more_constraints):
# check wether all constraints evaluate to True
if all(constraint(x) for constraint in more_constraints):
return -1 * (x[0] + 5 * x[1])
# if not all constraints evaluate to True, return a positive number
return 10
There is a proper solution to the problem described in the question, to enforce multiple nonlinear constraints with scipy.optimize.differential_evolution.
The proper way is by using the scipy.optimize.NonlinearConstraint function.
Here below I give a non-trivial example of optimizing the classic Rosenbrock function inside a region defined by the intersection of two circles.
import numpy as np
from scipy import optimize
# Rosenbrock function
def fun(x):
return 100*(x[1] - x[0]**2)**2 + (1 - x[0])**2
# Function defining the nonlinear constraints:
# 1) x^2 + (y - 3)^2 < 4
# 2) (x - 1)^2 + (y + 1)^2 < 13
def constr_fun(x):
r1 = x[0]**2 + (x[1] - 3)**2
r2 = (x[0] - 1)**2 + (x[1] + 1)**2
return r1, r2
# No lower limit on constr_fun
lb = [-np.inf, -np.inf]
# Upper limit on constr_fun
ub = [4, 13]
# Bounds are irrelevant for this problem, but are needed
# for differential_evolution to compute the starting points
bounds = [[-2.2, 1.5], [-0.5, 2.2]]
nlc = optimize.NonlinearConstraint(constr_fun, lb, ub)
sol = optimize.differential_evolution(fun, bounds, constraints=nlc)
# Accurate solution by Mathematica
true = [1.174907377273171, 1.381484428610871]
print(f"nfev = {sol.nfev}")
print(f"x = {sol.x}")
print(f"err = {sol.x - true}\n")
This prints the following with default parameters:
nfev = 636
x = [1.17490808 1.38148613]
err = [7.06260962e-07 1.70116282e-06]
Here is a visualization of the function (contours) and the feasible region defined by the nonlinear constraints (shading inside the green line). The constrained global minimum is indicated by the yellow dot, while the magenta one shows the unconstrained global minimum.
This constrained problem has an obvious local minimum at (x, y) ~ (-1.2, 1.4) on the boundary of the feasible region which will make local optimizers fail to converge to the global minimum for many starting locations. However, differential_evolution consistently finds the global minimum as expected.
I have encountered the following problem with scipy.fsolve, but I don't what to do:
U = 0.00043
ThC =1.19
Dist = 7
IncT = 0.2
pcw = 1180000
k = 1.19
B = U * pcw / (2 * k)
fugato = fsolve((((Ql/(2*math.pi* k))*math.exp(B * x)*special.kv(0, B * x))-IncT),0.01)
print fugato
I get the error TypeError: 'numpy.float64' object is not callable in fsolve.
How do I fix this problem?
The argument to fsolve must be a function.
I presume that you want to solve your equation for x? If so, writing:
fugato = fsolve(lambda x: Ql/(2*math.pi* k)*math.exp(B * x)*special.kv(0, B * x)-IncT,
0.01)
works.
To explain what's going on here, the construct lambda x: 2*x is a function definition. It is similar to writing:
def f(x):
return 2*x
The lambda construction is commonly used to define functions that you only need once. This is often the case when registering callbacks, or to represent a mathematical expression. For instance, if you wanted to integrate f(x) = 2*x, you could write:
from scipy.integrate import quad
integral = quad(lambda x: 2*x, 0., 3.)
Similarly, if you want to solve 2*x = 1, you can write:
from scipy.optimize import fsolve
fsolve(lambda x: 2*x-1, 0.)