I am trying to solve the set of linear equations:
min || Ax - B ||^2
for t in [0,1]
such that the coefficients x in this equation satisfy the linear equation:
C x = D
This system attempts to fit a set of Polynomials to approximate a function F(t) over the range of t.
A is a matrix, representing the map of the set of polynomials over the range of t values
x is a vector of coefficients (what I want) corresponding to a weight applied to each polynomial in A
B is a vector representing the F(t) values,
C is a matrix and D a vector, which together represent the boundary conditions on the coefficients of this system
This is a case of solving linear equations using the constraint of ordinary least squares.
While there are known closed form solutions e.g. Karush-Kuhn-Tucker I'm looking for a routing in scipy / numpy that can be used to solve this.
Research has shown the scipy.optimize module, which includes functions such as:
scipy.optimize.least_squares .
scipy.optimize.nnls .
scipy.optimize.lsq_linear .
The above is suggested both from this question and this question.
But these do not have conditions that work for a constraint of some other linear equation. What can I use in scipy and numpy to do this?
For this you may use scipy.minimize(method='SLSQP'). The documentation is here.
You can define the equality constraint as a callable function with the signature:
def cons1(x):
return sum(D - C*x)
SLSQP essential uses the constraint then to drive your optimisation problem. Note that this is a gradient based decent, so you will most likely find a local minima to high dimensional problems.
Another option is scipy.minimize(method=’trust-constr’), the documentation is here. These methods are natively implemented in python so the source code and modifications are accessible through.
If you have a smooth monotonic or context function, in my experience SLSQP should suffice.
Related
I work with Lie group SO(3). I know there exists an analytical expression for tangent application
from which the derivative of matrix exponential can be simply computed as
What I would like to do is to compute T(w).
The problem with simply implementing function T myself is that it is numerically imprecise for small norms of vector omega. I know there are exponential scipy.linalg.expm and logarithmic maps scipy.linalg.logm already in SciPy. Since this is similar I wonder if there is also the tangent map or the derivative. I noticed the scipy.linalg.expm_frechet, however, I didn't know how to connect it to T.
This does not directly answer your question but for
it is numerically imprecise for small norms of vector omega
you can just use Taylor series expansion for small |\omega| ie use something like
(1-cos(x))/x^2 = 1/2 - x^2/24, if |x| < eps, normal expression otherwise
for suitably small eps (say 1e-4)
and the same idea for (x-sin(x))/x^3
For numeric evaluation of the derivatives you can translate your code to use pytorch, and then autograd will do the work for you a.
For symbolic differentiation you can use sympy b
I've been looking around for a nonlinear constrained optimization package for Python (to deal with problems that are NOT necessarily convex) that can directly handle matrix variables. More specifically, I'm dealing with optimization problems where the optimization variables are matrices, and where there are equality constraints with both sides of the equations being matrices. An example of such an optimization problem is the Orthogonal Procrustes problem (https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem).
In my search I have come across SciPy, pyOpt, Ipopt and GEKKO, but neither of them seem to directly support matrix variables (from the documentation I was able to find). I have considered doing some maneuvering to convert the matrices into vectors when necessary and vice versa (through numpy.reshape or something similar), but I would like to avoid that option as much as possible. The reason for this is that my problems are fairly large, and constantly reshaping arrays would significantly harm the efficiency of the optimization procedure.
Any suggestions?
Here is a problem with matrices:
min(sum(sum(B))
s.t. AX=B
sum(sum(A))=5
sum(sum(X))=2
It is configured using the m.Array method in Python GEKKO with A, X, and B as 2D matrices but they could be higher dimensional as well.
from gekko import GEKKO
import numpy as np
m = GEKKO(remote=False)
ni = 3; nj = 2; nk = 4
# solve AX=B
A = m.Array(m.Var,(ni,nj),lb=0)
X = m.Array(m.Var,(nj,nk),lb=0)
AX = np.dot(A,X)
B = m.Array(m.Var,(ni,nk),lb=0)
# equality constraints
m.Equations([AX[i,j]==B[i,j] for i in range(ni) \
for j in range(nk)])
m.Equation(5==m.sum([m.sum([A[i][j] for i in range(ni)]) \
for j in range(nj)]))
m.Equation(2==m.sum([m.sum([X[i][j] for i in range(nj)]) \
for j in range(nk)]))
# objective function
m.Minimize(m.sum([m.sum([B[i][j] for i in range(ni)]) \
for j in range(nk)]))
m.solve()
print(A)
print(X)
print(B)
You mentioned non-convexity so you may need to use a multi-start or other method to find a global solution in Gekko or else use a global optimizer in a different Python package. The axb or qobj object are valuable if some of your constraints are linear equation or quadratic objectives with constant matrices. You can use those for large-scale and sparse systems. The APMonitor and Gekko papers also review some of the other Python packages for optimization.
I have a polynomial function for which I would like to find all local extrema. I can evaluate the polynomial via P(x) and to its derivative via d_P(x).
My first thought was to use minimize_scalar, however this does not seem to be able to take advantage of the fact that I can evaluate the derivative. Alternatively, I can use the more general minimize function and provide the gradient.
Is there a rule of thumb about which method will work better, or is this something where I should test out both methods and see what works better. Since the function I am optimizing is a polynomial (well behaved) I wonder if it really matters so much which I use, but if someone has a more background that would be great.
In particular, P(x) is the (unique) polynomial of degree n which alternatively attains a value of 1 or -1 on a set of n-1 points.
Here is a sample of the P(x) scaled so that P(0)=1. Note that the y axis is plotted on a symlog scale.
Since you have a continuous scalar function, the documentation of minimize_scalar suggests a more discrete optimization approach. Since it doesn't use gradient information you won't have trouble with noise/discontinuities/discreteness in your objective. However, if you use minimize in conjunction with a gradient based method then you will have trouble with convergence for noise/discontinuities/discreteness.
If the objective function is fist order continuous then both minimize and minimize_scalar should yield the same solution for a given bound.
I'm trying to run an optimization with scipy.optimize.differential_evolution. The code calls for bounds for each variable in x. But I want to a solution where parts of x must be integers, while others can range freely as floats. The relevant part of my code looks like
bounds = [(0,3),(0,3),(0,3),???,???]
result = differential_evolution(func, bounds)
What do I replace the ???'s with to force those variables to be ints in a given range?
As noted in the comments there isn't direct support for a "integer constraint".
You could however minimize a modified objective function, e.g.:
def func1(x):
return func(x) + K * (x[3] - round(x[3]))**2
and this will force x[3] towards an integer value (unfortunately you have to tune the K parameter).
An alternative is to round (some of) the real-valued parameters before evaluating the objective function:
def func1(x):
z = x;
z[3] = round(z[3])
return func(z)
Both are common techniques to approach a discrete optimization problem using Differential Evolution and they work quite well.
Differential evolution can support integer constraint but the current scipy implementation would need to be changed.
From the scipy source code it appears that their DE is based Storn, R and Price, K, Differential Evolution - a Simple and Efficient Heuristic for Global Optimization over Continuous Spaces, Journal of Global Optimization, 1997
However there has been progress in this field as pointed out by this review paper Recent advances in differential evolution – An updated survey
There are a few papers which introduce changes to the algorithm so that it can handle ints. I have not had time to look at all the options but perhaps this paper could help.
An Improved Differential Evolution Algorithm for Mixed Integer Programming Problems
What do I replace the ???'s with to force those variables to be ints in a given range?
wrapdisc is a package that is a thin wrapper which will let you optimize bounded integer variables alongside floats with various scipy.optimize optimizers. There is a usage example in its readme. With it, you don't have to adapt your objective function at all. It internally uses rounding in order to support integers, although this detail is hidden from the user.
Using scipy's interpolate.splprep function get a parametric spline on parameter u, but the domain of u is not the line integral of the spline, it is a piecewise linear connection of the input coordinates. I've tried integrate.splint, but that just gives the individual integrals over u. Obviously, I can numerically integrate a bunch of Cartesian differential distances, but I was wondering if there was closed-form method for getting the length of a spline or spline segment (using scipy or numpy) that I was overlooking.
Edit: I am looking for a closed-form solution or a very fast way to converge to a machine-precision answer. I have all but given up on the numerical root-finding methods and am now primarily after a closed-form answer. If anyone has any experience integrating elliptical functions or can point me to a good resource (other than Wolfram), That would be great.
I'm going to try Maxima to try to get the indefinite integral of what I believe is the function for one segment of the spline: I cross-posted this on MathOverflow
Because both x & y are cubic parametric functions, there isn't a closed solution in terms of simple functions. Numerical integration is the way to go. Either integrating the arc length expression or simply adding line segment lengths - depends on the accuracy you are after and how much effort you want to exert.
An accurate and fast "Adding length of line segments" method:
Using recurvise subdivision (a form of de Casteljeau's algorithm) to generate points, can give you a highly accurate representation with minimal number of points.
Only subdivide subdivisions if they fail to meet a criteria. Usually the criteria is based on the length joining the control points (the hull or cage).
For cubic, usually comparing closeness of P0P1+P1P2+P2P3 to P0P3, where P0, P1, P2 & P3 are the control points that define your bezier.
You can find some Delphi code here:
link text
It should be relatively easy to convert to Python.
It will generate the points. The code already calculates the length of the segments in order to test the criteria. You can simply accumulate those length values along the way.
You can integrate the function sqrt(x'(u)**2+y'(u)**2) over u, where you calculate the derivatives x' and y' of your coordinates with scipy.interpolate.splev. The integration can be done with one of the routines from scipy.integrate (quad is precise [Clenshaw-Curtis], romberg is generally faster). This should be more precise, and probably faster than adding up lots of small distances (which is equivalent to integrating with the rectangle rule).