Is there a good library to numericly solve an LCP in python ?
Edit: I need a working python code example because most libraries seem to only solve quadratic problems and i have problems converting an LCP into a QP.
For quadratic programming with Python, I use the qp-solver from cvxopt (source). Using this, it is straightforward to translate the LCP problem into a QP problem (see Wikipedia). Example:
from cvxopt import matrix, spmatrix
from cvxopt.blas import gemv
from cvxopt.solvers import qp
def append_matrix_at_bottom(A, B):
l = []
for x in xrange(A.size[1]):
for i in xrange(A.size[0]):
l.append(A[i+x*A.size[0]])
for i in xrange(B.size[0]):
l.append(B[i+x*B.size[0]])
return matrix(l,(A.size[0]+B.size[0],A.size[1]))
M = matrix([[ 4.0, 6, -4, 1.0 ],
[ 6, 1, 1.0 2.0 ],
[-4, 1.0, 2.5, -2.0 ],
[ 1.0, 2.0, -2.0, 1.0 ]])
q = matrix([12, -10, -7.0, 3])
I = spmatrix(1.0, range(M.size[0]), range(M.size[1]))
G = append_matrix_at_bottom(-M, -I) # inequality constraint G z <= h
h = matrix([x for x in q] + [0.0 for _x in range(M.size[0])])
sol = qp(2.0 * M, q, G, h) # find z, w, so that w = M z + q
if sol['status'] == 'optimal':
z = sol['x']
w = matrix(q)
gemv(M, z, w, alpha=1.0, beta=1.0) # w = M z + q
print(z)
print(w)
else:
print('failed')
Please note:
the code is totally untested, please check carefully;
there surely are better solution techniques than transforming LCP into QP.
Take a look at the scikit OpenOpt. It has an example of doing quadratic programming and I believe that it goes beyond SciPy's optimization routines. NumPy is required to use OpenOpt. I believe that the wikipedia page that you pointed us to for LCP describes how to solve a LCP by QP.
The best algorithm for solving MCPs (mixed nonlinear complementarity problems, more general than LCP) is the PATH solver: http://pages.cs.wisc.edu/~ferris/path.html
The PATH solver is available in matlab and GAMS. Both are coming with a python API. I have chosen to use GAMS because there is a free version. So here is a step by step solution to solve an LCP with the python API of GAMS. I used python 3.6:
Download and install GAMS: https://www.gams.com/download/
Install the API to python like here: https://www.gams.com/latest/docs/API_PY_TUTORIAL.html
I used conda, changed the directory to were the apifiles of python 3.6 were and entered
python setup.py install
Create a .gms-file (GAMS file) lcp_for_py.gms containing:
sets i;
alias(i,j);
parameters m(i,i),b(i);
$gdxin lcp_input
$load i m b
$gdxin
positive variables z(i);
equations Linear(i);
Linear(i).. sum(j,m(i,j)*z(j)) + b(i) =g= 0;
model lcp linear complementarity problem/Linear.z/;
options mcp = path;
solve lcp using mcp;
display z.L;
Your python code is like this:
import gams
#Set working directory, GamsWorkspace and the Database
worDir = "<THE PATH WHERE YOU STORED YOUR .GMS-FILE>" #"C:\documents\gams\"
ws=gams.GamsWorkspace(working_directory=worDir)
db=ws.add_database(database_name="lcp_input")
#Set the matrix and the vector of the LCP as lists
matrix = [[1,1],[2,1]]
vector = [0,-2]
#Create the Gams Set
index = []
for k in range(0,len(matrix)):
index.append("i"+str(k+1))
i = db.add_set("i",1,"number of decision variables")
for k in index:
i.add_record(k)
#Create a Gams Parameter named m and add records
m = db.add_parameter_dc("m", [i,i], "matrix of the lcp")
for k in range(0,len(matrix)):
for l in range(0,len(matrix[0])):
m.add_record([index[k],index[l]]).value = matrix[k][l]
#Create a Gams Parameter named b and add records
b = db.add_parameter_dc("b",[i],"bias of quadratics")
for k in range(0, len(vector)):
b.add_record(index[k]).value = vector[k]
#run the GamsJob using the Gams File and the database
lcp = ws.add_job_from_file("lcp_for_py.gms")
lcp.run(databases = db)
#Save the solution as a list an print it
z = []
for rec in lcp.out_db["z"]:
z.append(rec.level)
print(z)
OpenOpt has a free LCP solver written in Python + NumPy see http://openopt.org/LCP
Related
I have finally managed to wrap some Fortran into Python using numpy's f2py.
I am going to need to write a small library of subroutines that I can then use for a larger project, but I am already having trouble with the example subroutine I just compiled.
This is a simple algorithm to solve linear systems of equations in Fortran 90:
SUBROUTINE solve_lin_sys(A, c, x, n)
! =====================================================
! Uses gauss elimination and backwards substitution
! to reduce a linear system and solve it.
! Problem (0-div) can arise if there are 0s on main diag.
! =====================================================
IMPLICIT NONE
INTEGER:: i, j, k
REAL*8::fakt, summ
INTEGER, INTENT(in):: n
REAL*8, INTENT(inout):: A(n,n)
REAL*8, INTENT(inout):: c(n)
REAL*8, INTENT(out):: x(n)
DO i = 1, n-1 ! pick the var to eliminate
DO j = i+1, n ! pick the row where to eliminate
fakt = A(j,i) / A(i,i) ! elimination factor
DO k = 1, n ! eliminate
A(j,k) = A(j,k) - A(i,k)*fakt
END DO
c(j)=c(j)-c(i)*fakt ! iterate on known terms
END DO
END DO
! Actual solving:
x(n) = c(n) / A(n,n) ! last variable being solved
DO i = n-1, 1, -1
summ = 0.d0
DO j = i+1, n
summ = summ + A(i,j)*x(j)
END DO
x(i) = (c(i) - summ) / A(i,i)
END DO
END SUBROUTINE solve_lin_sys
I successfully made it into a Python module and I can import it, but I can't figure out what kind of arguments I should pass it into python.
This is what I've tried:
import numpy as np
import fortran_operations as ops
sys = [[3, -0.1, -0.2],
[0.1, 7, -0.3],
[0.3, -0.2, 10]]
known = [7.85, -19.3, 71.4]
A = np.array(sys)
c = np.array(known)
x = ops.solve_lin_sys(A, c, 3)
I tried using both the normal list and the numpy array as an argument for the Fortran function, but in all cases I get the error:
x = ops.solve_lin_sys(A, c, 3) ------ ValueError: failed in converting 1st
argument `a' of fortran_operations.solve_lin_sys to C/Fortran array
I thought you were supposed to simply use numpy arrays. Printing solve_lin_sys.doc also states that I should use rank 2 arrays for A and rank 1 for C. Isn't that already the shape of the arrays when I defined them like that?
Sorry if this has a stupid solution, I'm a beginner with all of this.
Thank you in advance
Arrays of dimension 2 and higher must be continuous with 'Fortran' order of elements.
import ops
sys = [[3, -0.1, -0.2],
[0.1, 7, -0.3],
[0.3, -0.2, 10]]
known = [7.85, -19.3, 71.4]
A = np.array(sys, dtype='d', order='F')
c = np.array(known, dtype='d')
x = ops.solve_lin_sys(A, c, 3)
So I am new to gurobi and I decided to start working with it on a well known problem as regression. I found this official notebook, where an L0 penalized regression model was solved and I took just the part of the regression model out of it. However, when I solve this problem in gurobi, I get a really strange solution, totally different from the actual correct regression solution.
The code I am running is:
import gurobipy as gp
from gurobipy import GRB
import numpy as np
from sklearn.datasets import load_boston
from itertools import product
boston = load_boston()
x = boston.data
x = x[:, [0, 2, 4, 5, 6, 7, 10, 11, 12]] # select non-categorical variables
response = boston.target
samples, dim = x.shape
regressor = gp.Model()
# Append a column of ones to the feature matrix to account for the y-intercept
x = np.concatenate([x, np.ones((samples, 1))], axis=1)
# Decision variables
beta = regressor.addVars(dim + 1, name="beta") # Beta
# Objective Function (OF): minimize 1/2 * RSS using the fact that
# if x* is a minimizer of f(x), it is also a minimizer of k*f(x) iff k > 0
Quad = np.dot(x.T, x)
lin = np.dot(response.T, x)
obj = sum(0.5 * Quad[i, j] * beta[i] * beta[j] for i, j in product(range(dim + 1), repeat=2))
obj -= sum(lin[i] * beta[i] for i in range(dim + 1))
obj += 0.5 * np.dot(response, response)
regressor.setObjective(obj, GRB.MINIMIZE)
regressor.optimize()
beta_sol_gurobi = np.array([beta[i].X for i in range(dim+1)])
The solution provided by this code is
array([1.22933632e-14, 2.40073891e-15, 1.10109084e-13, 2.93142174e+00,
6.14486489e-16, 3.93021623e-01, 5.52707727e-15, 8.61271603e-03,
1.55963041e-15, 3.19117429e-13])
While the true linear regression solution should be
from sklearn import linear_model
lr = linear_model.LinearRegression()
lr.fit(x, response)
lr.coef_
lr.intercept_
That yields,
array([-5.23730841e-02, -3.35655253e-02, -1.39501039e+01, 4.40955833e+00,
-7.33680982e-03, -1.24312668e+00, -9.59615262e-01, 8.60275557e-03,
-5.17452533e-01])
29.531492975441015
So gurobi solution is completely different. Any guess / suggestion on whats happening? Am I doing anything wrong here?
PD: I know that this problem can be solved using other packages, or even other optimization frameworks, but I am specially interested in solving it in gurobi python, since I want to start using gurobi in some more complex problems.
The wrong result is due to your decision variables. Since Gurobi assumes the lower bound 0 for all variables by default, you need to explicitly set the lower bound:
beta = regressor.addVars(dim + 1, lb = -GRB.INFINITY, name="beta") # Beta
I am in the process of translating some MATLAB code into Python. There is one line that is giving me a bit of trouble:
[q,f_dummy,exitflag, output] = quadprog(H,f,-A,zeros(p*N,1),E,qm,[],[],q0,options);
I looked up the documentation in MATLAB to find that the quadprog function is used for optimization (particularly minimization).
I attempted to find a similar function in Python (using numpy) and there does not seem to be any.
Is there a better way to translate this line of code into Python? Or are there other packages that can be used? Do I need to make a new function that accomplishes the same task?
Thanks for your time and help!
There is a library called CVXOPT that has quadratic programming in it.
def quadprog_solve_qp(P, q, G=None, h=None, A=None, b=None):
qp_G = .5 * (P + P.T) # make sure P is symmetric
qp_a = -q
if A is not None:
qp_C = -numpy.vstack([A, G]).T
qp_b = -numpy.hstack([b, h])
meq = A.shape[0]
else: # no equality constraint
qp_C = -G.T
qp_b = -h
meq = 0
return quadprog.solve_qp(qp_G, qp_a, qp_C, qp_b, meq)[0]
I will start by mentioning that quadratic programming problems are a subset of convex optimization problems which are a subset of optimization problems.
There are multiple python packages which solve quadratic programming problems, notably
cvxopt -- which solves all kinds of convex optimization problems (including quadratic programming problems). This is a python version of the previous cvx MATLAB package.
quadprog -- this is exclusively for quadratic programming problems but doesn't seem to have much documentation.
scipy.optimize.minimize -- this is a very general minimizer which can solve quadratic programming problems, as well as other optimization problems (convex and non-convex).
You might also benefit from looking at the answers to this stackoverflow post which has more details and references.
Note: The code snippet in user1911226' answer appears to come from this blog post:
https://scaron.info/blog/quadratic-programming-in-python.html
which compares some of these quadratic programming packages. I can't comment on their answer, but they claim to be mentioning the cvxopt solution, but the code is actually for the quadprog solution.
OSQP is a specialized free QP solver based on ADMM. I have adapted the OSQP documentation demo and the OSQP call in the qpsolvers repository for your problem.
Note that matrices H and G are supposed to be sparse in CSC format. Here is the script
import numpy as np
import scipy.sparse as spa
import osqp
def quadprog(P, q, G=None, h=None, A=None, b=None,
initvals=None, verbose=True):
l = -np.inf * np.ones(len(h))
if A is not None:
qp_A = spa.vstack([G, A]).tocsc()
qp_l = np.hstack([l, b])
qp_u = np.hstack([h, b])
else: # no equality constraint
qp_A = G
qp_l = l
qp_u = h
model = osqp.OSQP()
model.setup(P=P, q=q,
A=qp_A, l=qp_l, u=qp_u, verbose=verbose)
if initvals is not None:
model.warm_start(x=initvals)
results = model.solve()
return results.x, results.info.status
# Generate problem data
n = 2 # Variables
H = spa.csc_matrix([[4, 1], [1, 2]])
f = np.array([1, 1])
G = spa.csc_matrix([[1, 0], [0, 1]])
h = np.array([0.7, 0.7])
A = spa.csc_matrix([[1, 1]])
b = np.array([1.])
# Initial point
q0 = np.ones(n)
x, status = quadprog(H, f, G, h, A, b, initvals=q0, verbose=True)
You could use the solve_qp function from qpsolvers. It can be installed, along with a starter kit of open source solvers, by pip install qpsolvers[open_source_solvers]. Then you can replace your line with:
from qpsolvers import solve_qp
solver = "proxqp" # or "osqp", "quadprog", "cvxopt", ...
x = solve_qp(H, f, G, h, A, b, initvals=q_0, solver=solver, **options)
There are many solvers available in Python, each coming with their pros and cons. Make sure you try different values for the solver keyword argument to find the one that fits your problem best.
Here is a standalone example based on your question and the other comments:
import numpy as np
from qpsolvers import solve_qp
H = np.array([[4.0, 1.0], [1.0, 2.0]])
f = np.array([1.0, 1])
G = np.array([[1.0, 0.0], [0.0, 1.0]])
h = np.array([0.7, 0.7])
A = np.array([[1.0, 1.0]])
b = np.array([1.0])
q_0 = np.array([1.0, 1.0])
solver = "cvxopt" # or "osqp", "proxqp", "quadprog", ...
options = {"verbose": True}
x = solve_qp(H, f, G, h, A, b, initvals=q_0, solver=solver, **options)
I have m = 10, n = 5, A=randn(m,n);[U,S,V]=svd(A); This returns a full 10x5 S matrix in MATLAB whereas Python only returns S as a 5x1 array. How do I recover the complete S matrix in Python? I have tried looking up several StackOverflow posts online but surprisingly doesn't shed light on this.
Also, how much does a Python IDE matter? I use Spyder but have been told that Vim is perhaps the most common.
Thanks a lot.
To recover the Complete matrix you can do as follow :
import numpy as np
m = 10
n = 5
A=np.random.randn(m,n)
U,S,V =np.linalg.svd(A)
It's right that S.shape = (5,).
You want something similar to https://www.mathworks.com/help/matlab/ref/svd.html with A = 4x2 where final S = 4×2 too.
To do that you define a matrix B = np.zeros(A.shape). And you fill its diagonal with the element of S. By diagonal I mean where i==j as follow :
B = np.zeros(A.shape)
for i in range(m) :
for j in range(n) :
if i == j : B[i,j] = S[j]
Now B.shape = (10,5) as expected
Or in a more compact form :
C = np.array([[S[j] if i==j else 0 for j in range(n)] for i in range(m)])
I hope it helps
For the second question, I use gedit (standard text editor) running the code in ipython shell.
You can have a look to jupyter too
The SVD of a matrix can be written as
A = U S V^H
Where the ^H signifies the conjugate transpose. Matlab's svd command returns U, S and V, while numpy.linalg.svd returns U, the diagonal of S, and V^H. Thus, to get the same S and V as in Matlab you need to reconstruct the S and also get the V:
import numpy
m = 10
n = 5
A = numpy.random.randn(m, n)
U, sdiag, VH = numpy.linalg.svd(A)
S = numpy.zeros((m, n))
numpy.fill_diagonal(S, sdiag)
V = VH.T.conj() # if you know you have real values only you can leave out the .conj()
I'm new to signal processing (and numpy, scipy, and matlab for that matter). I'm trying to estimate vowel formants with LPC in Python by adapting this matlab code:
http://www.mathworks.com/help/signal/ug/formant-estimation-with-lpc-coefficients.html
Here is my code so far:
#!/usr/bin/env python
import sys
import numpy
import wave
import math
from scipy.signal import lfilter, hamming
from scikits.talkbox import lpc
"""
Estimate formants using LPC.
"""
def get_formants(file_path):
# Read from file.
spf = wave.open(file_path, 'r') # http://www.linguistics.ucla.edu/people/hayes/103/Charts/VChart/ae.wav
# Get file as numpy array.
x = spf.readframes(-1)
x = numpy.fromstring(x, 'Int16')
# Get Hamming window.
N = len(x)
w = numpy.hamming(N)
# Apply window and high pass filter.
x1 = x * w
x1 = lfilter([1., -0.63], 1, x1)
# Get LPC.
A, e, k = lpc(x1, 8)
# Get roots.
rts = numpy.roots(A)
rts = [r for r in rts if numpy.imag(r) >= 0]
# Get angles.
angz = numpy.arctan2(numpy.imag(rts), numpy.real(rts))
# Get frequencies.
Fs = spf.getframerate()
frqs = sorted(angz * (Fs / (2 * math.pi)))
return frqs
print get_formants(sys.argv[1])
Using this file as input, my script returns this list:
[682.18960189917243, 1886.3054773107765, 3518.8326108511073, 6524.8112723782951]
I didn't even get to the last steps where they filter the frequencies by bandwidth because the frequencies in the list aren't right. According to Praat, I should get something like this (this is the formant listing for the middle of the vowel):
Time_s F1_Hz F2_Hz F3_Hz F4_Hz
0.164969 731.914588 1737.980346 2115.510104 3191.775838
What am I doing wrong?
Thanks very much
UPDATE:
I changed this
x1 = lfilter([1., -0.63], 1, x1)
to
x1 = lfilter([1], [1., 0.63], x1)
as per Warren Weckesser's suggestion and am now getting
[631.44354635609318, 1815.8629524985781, 3421.8288991389031, 6667.5030877036006]
I feel like I'm missing something since F3 is very off.
UPDATE 2:
I realized that the order being passed to scikits.talkbox.lpc was off due to a difference in sampling frequency. Changed it to:
Fs = spf.getframerate()
ncoeff = 2 + Fs / 1000
A, e, k = lpc(x1, ncoeff)
Now I'm getting:
[257.86573127888488, 774.59006835496086, 1769.4624576002402, 2386.7093679399809, 3282.387975973973, 4413.0428174593926, 6060.8150432549655, 6503.3090645887842, 7266.5069407315023]
Much closer to Praat's estimation!
The problem had to do with the order being passed to the lpc function. 2 + fs / 1000 where fs is the sampling frequency is the rule of thumb according to:
http://www.phon.ucl.ac.uk/courses/spsci/matlab/lect10.html
I have not been able to get the results you expect, but I do notice two things which might cause some differences:
Your code uses [1, -0.63] where the MATLAB code from the link you provided has [1 0.63].
Your processing is being applied to the entire x vector at once instead of smaller segments of it (see where the MATLAB code does this: x = mtlb(I0:Iend); ).
Hope that helps.
There are at least two problems:
According to the link, the "pre-emphasis filter is a highpass all-pole (AR(1)) filter". The signs of the coefficients given there are correct: [1, 0.63]. If you use [1, -0.63], you get a lowpass filter.
You have the first two arguments to scipy.signal.lfilter reversed.
So, try changing this:
x1 = lfilter([1., -0.63], 1, x1)
to this:
x1 = lfilter([1.], [1., 0.63], x1)
I haven't tried running your code yet, so I don't know if those are the only problems.