We Keep Coding

Get different results from Pulp and Linprog

I am new to linear programming and trying both Pulp and (SciPy) Linprog. Each gives me different results.
I think it might be because Linprog is using interior-point method whereas Pulp is probably using simplex? If so, is there a way to get Pulp produce the same result is Linprog?
import pulp
from pulp import *
from scipy.optimize import linprog
# Pulp
# Upper bounds
r = {1: 11, 2: 11, 3: 7, 4: 11, 5: 7}
# Create the model
model = LpProblem(name="small-problem", sense=LpMaximize)
# Define the decision variables
x = {i: LpVariable(name=f"x{i}", lowBound=0, upBound=r[i]) for i in range(1, 6)}
# Add constraints
model += (lpSum(x.values()) <= 35, "headroom")
# Set the objective
model += lpSum([7 * x[1], 7 * x[2], 11 * x[3], 7 * x[4], 11 * x[5]])
# Solve the optimization problem
status = model.solve()
# Get the results
print(f"status: {model.status}, {LpStatus[model.status]}")
print(f"objective: {model.objective.value()}")
for var in x.values():
print(f"{var.name}: {var.value()}")
for name, constraint in model.constraints.items():
print(f"{name}: {constraint.value()}")
# linprog
c = [-7, -7, -11, -7, -11]
bounds = [(0, 11), (0, 11), (0, 7), (0, 11), (0, 7)]
A_ub = [[1, 1, 1, 1, 1]]
B_ub = [[35]]
res = linprog(c, A_ub=A_ub, b_ub=B_ub, bounds=bounds)
print(res)
Output from code above:
status: 1, Optimal
objective: 301.0
x1: 10.0
x2: 0.0
x3: 7.0
x4: 11.0
x5: 7.0
headroom: 0.0
con: array([], dtype=float64)
fun: -300.9999999581466
message: 'Optimization terminated successfully.'
nit: 4
slack: array([4.60956784e-09])
status: 0
success: True
x: array([7., 7., 7., 7., 7.])
Bonus question: How would I formulate a problem where I want to maximum values for x[i]'s given some constraints? Above I am trying to maximise sum of x[i]'s but wondering if there is a better way.

As #Erwin Kalvelagen has already pointed out in the comments not all LPs have a unique solution. In your case you have two groups of variables {x1, x2, x4} and {x3, x5} that have the same coefficients in all occurrences.
In your case it is optimal to use the maximal possible value for x3, x5 and what ever is still available towards 35 in your constraint is distributed between x1, x2, x4 arbitrarily (as it makes no difference for the objective).
Note that your pulp solution is a basic solution while your scipy solution is not. And yes, this likely is because the two use different algorithms to solve the problem.

Constrained Optimization Problem : Python

I am sure , there must be a simple solution that keeps evading me.
I have a function
f=ax+by+c*z
and a constraint
lx+my+n*z=B
Need to find the (x,y,z), that maximizes f subject to the constraint.
I also need
x,y,z>=0
I remember having seen a solution like this.
This example uses
a,b,c=2,4,10 and l,m,n=1,2,4 and B=5
Ideally, this should give me x=1,y=0 , z=1, such that f=12
import numpy as np
from scipy.optimize import minimize
def objective(x, sign=-1.0):
x1 = x[0]
x2 = x[1]
x3 = x[2]
return sign*((2*x1) + (4*x2)+(10*x3))
def constraint1(x, sign=1.0):
return sign*(1*x[0] +2*x[1]+4*x[2]- 5)
x0=[0,0,0]
b1 = (0,None)
b2 = (0,None)
b3=(0,None)
bnds= (b1,b2,b3)
con1 = {'type': 'ineq', 'fun': constraint1}
cons = [con1]
sol = minimize (objective,x0,method='SLSQP',bounds=bnds,constraints=cons)
print(sol)
This is generating bizarre solution. What am I missing ?

The problem as you stated originally without integer constraints can be solved simply and efficiently by linprog:
import scipy.optimize
c = [-2, -4, -10]
A_eq = [[1, 2, 4]]
b_eq = 5
# bounds are for non-negative values by default
scipy.optimize.linprog(c, A_eq=A_eq, b_eq=b_eq)
I would recommend against using more general purpose solvers to solve narrow problems like this as you will often encounter worse performance and sometimes unexpected results.

You need to change your constraint to an 'equality constraint'. Also, your problem didn't specify that integer answers were required, so there is a better non-integer answer to this knapsack problem. (I don't have much experience with scipy.optimize and I'm not sure if it can work integer LP problems.)
In [13]: con1 = {'type': 'eq', 'fun': constraint1}
In [14]: cons = [con1,]
In [15]: sol = minimize (objective,x0,method='SLSQP',bounds=bnds,constraints=cons)
In [16]: print(sol)
fun: -12.5
jac: array([ -2., -4., -10.])
message: 'Optimization terminated successfully.'
nfev: 10
nit: 2
njev: 2
status: 0
success: True
x: array([0. , 0. , 1.25])

Like Jeff said, scipy.optimize only works with linear programming problems.
You can try using PuLP instead for Integer Optimization problems:
from pulp import *
prob = LpProblem("F Problem", LpMaximize)
# a,b,c=2,4,10 and l,m,n=1,2,4 and B=5
a,b,c=2,4,10
l,m,n=1,2,4
B=5
# x,y,z>=0
x = LpVariable("x",0,None,LpInteger)
y = LpVariable("y",0,None,LpInteger)
z = LpVariable("z",0,None,LpInteger)
# f=ax+by+c*z
prob += a*x + b*y + c*z, "Objective Function f"
# lx+my+n*z=B
prob += l*x + m*y + n*z == B, "Constraint B"
# solve
prob.solve()
print("Status:", LpStatus[prob.status])
for v in prob.variables():
print(v.name, "=", v.varValue)
Documentation is here: enter link description here

scipy.integrate.solve_ivp vectorized

Trying to use the vectorized option for solve_ivp and strangely it throws an error that y0 must be 1 dimensional.
MWE :
from scipy.integrate import solve_ivp
import numpy as np
import math
def f(t, y):
theta = math.pi/4
ham = np.array([[1,0],[1,np.exp(-1j*theta*t)]])
return-1j * np.dot(ham,y)
def main():
y0 = np.eye(2,dtype= np.complex128)
t0 = 0
tmax = 10**(-6)
sol=solve_ivp( lambda t,y :f(t,y),(t0,tmax),y0,method='RK45',vectorized=True)
print(sol.y)
if __name__ == '__main__':
main()
The calling signature is fun(t, y). Here t is a scalar, and there are two options for the ndarray y: It can either have shape (n,); then fun must return array_like with shape (n,). Alternatively it can have shape (n, k); then fun must return an array_like with shape (n, k), i.e. each column corresponds to a single column in y. The choice between the two options is determined by vectorized argument (see below). The vectorized implementation allows a faster approximation of the Jacobian by finite differences (required for stiff solvers).
Error :
ValueError: y0 must be 1-dimensional.
Python 3.6.8
scipy.version
'1.2.1'

The meaning of vectorize here is a bit confusing. It doesn't mean that y0 can be 2d, but rather that y as passed to your function can be 2d. In other words that func may be evaluated at multiple points at once, if the solver so desires. How many points is up to the solver, not you.
Change the f to show the shape a y at each call:
def f(t, y):
print(y.shape)
theta = math.pi/4
ham = np.array([[1,0],[1,np.exp(-1j*theta*t)]])
return-1j * np.dot(ham,y)
A sample call:
In [47]: integrate.solve_ivp(f,(t0,tmax),[1j,0],method='RK45',vectorized=False)
(2,)
(2,)
(2,)
(2,)
(2,)
(2,)
(2,)
(2,)
Out[47]:
message: 'The solver successfully reached the end of the integration interval.'
nfev: 8
njev: 0
nlu: 0
sol: None
status: 0
success: True
t: array([0.e+00, 1.e-06])
t_events: None
y: array([[0.e+00+1.e+00j, 1.e-06+1.e+00j],
[0.e+00+0.e+00j, 1.e-06-1.e-12j]])
Same call, but with vectorize=True:
In [48]: integrate.solve_ivp(f,(t0,tmax),[1j,0],method='RK45',vectorized=True)
(2, 1)
(2, 1)
(2, 1)
(2, 1)
(2, 1)
(2, 1)
(2, 1)
(2, 1)
Out[48]:
message: 'The solver successfully reached the end of the integration interval.'
nfev: 8
njev: 0
nlu: 0
sol: None
status: 0
success: True
t: array([0.e+00, 1.e-06])
t_events: None
y: array([[0.e+00+1.e+00j, 1.e-06+1.e+00j],
[0.e+00+0.e+00j, 1.e-06-1.e-12j]])
With False, the y passed to f is (2,), 1d; with True it is (2,1). I'm guessing it could be (2,2) or even (2,3) if the solver method so desires. That could speed up the execution, with fewer calls to f. In this case, it doesn't matter.
quadrature has a similar vec_func boolean parameter:
Numerical Quadrature of scalar valued function with vector input using scipy
A related bug/issue discussion:
https://github.com/scipy/scipy/issues/8922

Python Numpy Logistic Regression

I'm trying to implement vectorized logistic regression in python using
numpy. My Cost function (CF) seems to work OK. However there is a
problem with gradient calculation. It returns 3x100 array whereas it
should return 3x1. I think there is a problem with the (hypo-y) part.
def sigmoid(a):
return 1/(1+np.exp(-a))
def CF(theta,X,y):
m=len(y)
hypo=sigmoid(np.matmul(X,theta))
J=(-1./m)*((np.matmul(y.T,np.log(hypo)))+(np.matmul((1-y).T,np.log(1-hypo))))
return(J)
def gr(theta,X,y):
m=len(y)
hypo=sigmoid(np.matmul(X,theta))
grad=(1/m)*(np.matmul(X.T,(hypo-y)))
return(grad)
X is a 100x3 arrray, y is 100x1, and theta is a 3x1 arrray. It seems both functions are working individually, however this optimization function gives an error:
optim = minimize(CF, theta, method='BFGS', jac=gr, args=(X,y))
The error: "ValueError: shapes (3,100) and (3,100) not aligned: 100 (dim 1) != 3 (dim 0)"

I think there is a problem with the (hypo-y) part.
Spot on!
hypo is of shape (100,) and y is of shape (100, 1). In the element-wise - operation, hypo is broadcasted to shape (1, 100) according to numpy's broadcasting rules. This results in a (100, 100) array, which causes the matrix multiplication to result in a (3, 100) array.
Fix this by bringing hypo into the same shape as y:
hypo = sigmoid(np.matmul(X, theta)).reshape(-1, 1) # -1 means automatic size on first dimension
There is one more issue: scipy.optimize.minimize (which I assume you are using) expects the gradient to be an array of shape (k,) but the function gr returns a vector of shape (k, 1). This is easy to fix:
return grad.reshape(-1)
The final function becomes
def gr(theta,X,y):
m=len(y)
hypo=sigmoid(np.matmul(X,theta)).reshape(-1, 1)
grad=(1/m)*(np.matmul(X.T,(hypo-y)))
return grad.reshape(-1)
and running it with toy data works (I have not checked the math or the plausibility of the results):
theta = np.reshape([1, 2, 3], 3, 1)
X = np.random.randn(100, 3)
y = np.round(np.random.rand(100, 1))
optim = minimize(CF, theta, method='BFGS', jac=gr, args=(X,y))
print(optim)
# fun: 0.6830931976615066
# hess_inv: array([[ 4.51307367, -0.13048255, 0.9400538 ],
# [-0.13048255, 3.53320257, 0.32364498],
# [ 0.9400538 , 0.32364498, 5.08740428]])
# jac: array([ -9.20709950e-07, 3.34459058e-08, 2.21354905e-07])
# message: 'Optimization terminated successfully.'
# nfev: 15
# nit: 13
# njev: 15
# status: 0
# success: True
# x: array([-0.07794477, 0.14840167, 0.24572182])

Trying to understand how scipy.optimize can be used?

I am trying to use stats.optimize.minimize function. First, I am trying something very simple.
I define:
lik1 = lambda n,k,p: math.log(stats.binom.pmf(k,n,p))
I am trying to see if minimize will give me the correct MLE, which is, k/n == p.
Then I try:
optimize.minimize(lik1, 0.5, args=(10,2))
where I am assuming n == 10 and k == 2 and my guess for p (the argument x0) is 0.5. I get the following error:
fun: nan
hess_inv: array([[1]])
jac: array([ nan])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 3
nit: 0
njev: 1
status: 2
success: False
x: array([ 0.5])
What am I doing wrong?

A few changes:
Select a more appropriate minimization method for this problem. The minimize function defaults to the BFGS method when no constraints or bounds are provided which is a method for unconstrained optimization. It fails because it tries to evaluate the function for values of p > 1. You could provide some reasonable bounds, or I've found here that using the TNC method works in this instance.
The order of the function arguments should be (p, n, k)
You want to maximize the log, or equivalently minimize the negative of the log.
Code:
import scipy as sp
import scipy.stats
import scipy.optimize
lik1 = lambda p, n, k: -sp.log(sp.stats.binom.pmf(k, n, p))
res = sp.optimize.minimize(lik1, 0.5, args=(10, 2), method='TNC')
print(res)
Output:
fun: array([ 1.19736175])
jac: array([ 1.22124533e-05])
message: 'Converged (|f_n-f_(n-1)| ~= 0)'
nfev: 10
nit: 4
status: 1
success: True
x: array([ 0.20000019])