We Keep Coding

Computationally efficient way of maximizing a complicated multivariable function

I've been trying to find an efficient way of maximizing the following monster function in four variables but the program is taking ages to run and I'm not even sure if the results are correct. Can anyone help me code it better in Python?
Here's the function:
where
a=[p,q,r,s].
Y is the measured data sampled at 30 points.
Here's my code.
import numpy as np
import math
Y=Y_t #Y_t is a predefined column vector with 30 entries.
tstep=0.05 #in s
N=30
cov=np.zeros([30,30])
def R(p,q,r,t):
om_D=p*np.sqrt(1-q**2)
return np.pi*r*(np.exp(-q*p*abs(t)))*(np.cos(om_D*t)+(q/(np.sqrt(1-q**2)))*(np.sin(om_D*abs(t))))/(2*q*(p**3))
def I(m,p):
if m==p:
return 1
else:
return 0
def func(a):
a1=a[0] #natural angular frequency bounds=[3,20]
a2=a[1] #damping ratio bounds=[0,1]
a3=a[2] #psd of forcing signal bounds=[300,600]
a4=a[3] #variance of noise bounds=[0,0.0001] in m
#assuming uniform prior for a, we only have to maximise the likelihood function
for i in range(30):
for j in range(30):
cov[i,j]+=R(a1,a2,a3,(j-i)*tstep)+a4*I(i,j)
P=((2*np.pi)**(-N/2)) * ((np.linalg.det(cov))**(-0.5)) * np.exp((-0.5) *np.linalg.multi_dot([np.transpose(Y),np.linalg.inv(cov),Y]))
return (-1)*P[0]
a_start=[5,0.05,100,0.00001]
bnds=((5,20),(0,1),(300,600),(0,0.0001))
result=spo.differential_evolution(func,bounds=bnds)
print(result.x) ```

There is an issue in cov initialization that is why it does not converge. Also an issue on bound for damping ratio, was (0, 1) now (0.0001, 0.999) the ratio should not be 0 or 1 because if it is there will be division by zero error in R(). Code is fixed now see also the output.
Code
import time
import numpy as np
from scipy.optimize import differential_evolution
Y = [[-0.00445551], [-0.01164452], [-0.02171495], [-0.03475491], [-0.00770873], [ 0.0492236 ],
[ 0.07264838], [ 0.03066707], [-0.02457141], [-0.04065968], [-0.01135125], [ 0.02677074], [ 0.06517749],
[ 0.09611112], [ 0.12300657], [ 0.0923581 ], [ 0.03982604], [-0.01473844], [-0.09024497], [-0.14304097],
[-0.17447606], [-0.16926952], [-0.12006193], [-0.00120763], [ 0.11006087], [ 0.19978283], [ 0.24388584],
[ 0.18768875], [ 0.12844553], [ 0.03099409]] #Y_t is a predefined column vector with 30 entries.
tstep = 0.05 #in s
N = 30
def R(p,q,r,t):
om_D = p*np.sqrt(1-q**2)
return np.pi*r*(np.exp(-q*p*abs(t)))*(np.cos(om_D*t)+(q/(np.sqrt(1-q**2)))*(np.sin(om_D*abs(t))))/(2*q*(p**3))
def I(m,p):
if m==p:
return 1
else:
return 0
def func(a):
cov=np.zeros([N,N])
a1=a[0] #natural angular frequency bounds=[3,20]
a2=a[1] #damping ratio bounds=[0,1]
a3=a[2] #psd of forcing signal bounds=[300,600]
a4=a[3] #variance of noise bounds=[0,0.0001] in m
#assuming uniform prior for a, we only have to maximise the likelihood function
for i in range(N):
for j in range(N):
cov[i,j]+=R(a1,a2,a3,(j-i)*tstep)+a4*I(i,j)
P=((2*np.pi)**(-N/2)) * ((np.linalg.det(cov))**(-0.5)) * np.exp((-0.5) *np.linalg.multi_dot([np.transpose(Y),np.linalg.inv(cov),Y]))
return (-1)*P[0]
if __name__ == '__main__':
t0 = time.perf_counter()
a_start = [5, 0.05, 350, 0.00001]
bnds = ((5, 20), (0.0001, 0.999), (300, 600), (0, 0.0001))
result=differential_evolution(func, x0=a_start, bounds=bnds, maxiter=1000)
print(result)
print(f'elapse: {time.perf_counter() - t0:0.0f}s')
Output
fun: array([-2.76736878e+11])
jac: array([-2.91459845e+11, -4.55652161e+12, 1.27377279e+10, 3.34234132e+14])
message: 'Optimization terminated successfully.'
nfev: 3430
nit: 56
success: True
x: array([ 20. , 0.999, 300. , 0. ])
elapse: 55s
Scipy minimize is very fast
Changes:
from scipy.optimize import minimize
result = minimize(func, x0=a_start, bounds=bnds, options={'maxiter': 100, 'disp': True})
Output:
fun: array([-2.76736878e+11])
hess_inv: <4x4 LbfgsInvHessProduct with dtype=float64>
jac: array([-2.91459845e+11, -4.55652161e+12, 1.27377279e+10, 3.34234132e+14])
message: 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
nfev: 30
nit: 4
njev: 6
status: 0
success: True
x: array([ 20. , 0.999, 300. , 0. ])
elapse: 0.5s
Optuna
Optuna after 1000 trials is right there too.
value is positive here because I use maximize direction. In both scipy's DE and minimize values have to be negated.
best param: {'a1': 20, 'a2': 0.9989999999999999, 'a3': 300, 'a4': 0.0}
best value: 276736878140.3103
best trial num: 73
elapse: 22s

Constrained Optimization Problem : Python

I am sure , there must be a simple solution that keeps evading me.
I have a function
f=ax+by+c*z
and a constraint
lx+my+n*z=B
Need to find the (x,y,z), that maximizes f subject to the constraint.
I also need
x,y,z>=0
I remember having seen a solution like this.
This example uses
a,b,c=2,4,10 and l,m,n=1,2,4 and B=5
Ideally, this should give me x=1,y=0 , z=1, such that f=12
import numpy as np
from scipy.optimize import minimize
def objective(x, sign=-1.0):
x1 = x[0]
x2 = x[1]
x3 = x[2]
return sign*((2*x1) + (4*x2)+(10*x3))
def constraint1(x, sign=1.0):
return sign*(1*x[0] +2*x[1]+4*x[2]- 5)
x0=[0,0,0]
b1 = (0,None)
b2 = (0,None)
b3=(0,None)
bnds= (b1,b2,b3)
con1 = {'type': 'ineq', 'fun': constraint1}
cons = [con1]
sol = minimize (objective,x0,method='SLSQP',bounds=bnds,constraints=cons)
print(sol)
This is generating bizarre solution. What am I missing ?

The problem as you stated originally without integer constraints can be solved simply and efficiently by linprog:
import scipy.optimize
c = [-2, -4, -10]
A_eq = [[1, 2, 4]]
b_eq = 5
# bounds are for non-negative values by default
scipy.optimize.linprog(c, A_eq=A_eq, b_eq=b_eq)
I would recommend against using more general purpose solvers to solve narrow problems like this as you will often encounter worse performance and sometimes unexpected results.

You need to change your constraint to an 'equality constraint'. Also, your problem didn't specify that integer answers were required, so there is a better non-integer answer to this knapsack problem. (I don't have much experience with scipy.optimize and I'm not sure if it can work integer LP problems.)
In [13]: con1 = {'type': 'eq', 'fun': constraint1}
In [14]: cons = [con1,]
In [15]: sol = minimize (objective,x0,method='SLSQP',bounds=bnds,constraints=cons)
In [16]: print(sol)
fun: -12.5
jac: array([ -2., -4., -10.])
message: 'Optimization terminated successfully.'
nfev: 10
nit: 2
njev: 2
status: 0
success: True
x: array([0. , 0. , 1.25])

Like Jeff said, scipy.optimize only works with linear programming problems.
You can try using PuLP instead for Integer Optimization problems:
from pulp import *
prob = LpProblem("F Problem", LpMaximize)
# a,b,c=2,4,10 and l,m,n=1,2,4 and B=5
a,b,c=2,4,10
l,m,n=1,2,4
B=5
# x,y,z>=0
x = LpVariable("x",0,None,LpInteger)
y = LpVariable("y",0,None,LpInteger)
z = LpVariable("z",0,None,LpInteger)
# f=ax+by+c*z
prob += a*x + b*y + c*z, "Objective Function f"
# lx+my+n*z=B
prob += l*x + m*y + n*z == B, "Constraint B"
# solve
prob.solve()
print("Status:", LpStatus[prob.status])
for v in prob.variables():
print(v.name, "=", v.varValue)
Documentation is here: enter link description here

scipy minimize SLSQP - 'Singular matrix C in LSQ subproblem'

I'm trying to solve a pretty basic optimization problem using SciPy. The problem is constrained and with variable bounds and I'm pretty sure it's linear.
When I run the following code the execution fails with the error message 'Singular matrix C in LSQ subproblem'. Does anyone know what the problem might be? Thanks in advance.
Edit: I'll add a short description of what the code should do here.
I define a 'demand' vector at the beginning of the code. This vector describes the demand of a certain product indexed over some period of time. What I want to figure out is how to place a set of orders so as to fill this demand under some constraints. These constraints are;
We must have the items in stock if there is a demand at a certain point in time (index in demand)
We can not place an additional order until 4 'time units' after an order has been placed
We can not place an order in the last 4 time units
This is my code;
from scipy.optimize import minimize
import numpy as np
demand = np.array([5, 10, 10, 7, 3, 7, 1, 0, 0, 0, 8])
orders = np.array([0.] * len(demand))
def objective(orders):
return np.sum(orders)
def items_in_stock(orders):
stock = 0
for i in range(len(orders)):
stock += orders[i]
stock -= demand[i]
if stock < 0.:
return -1.
return 0.
def four_weeks_order_distance(orders):
for i in range(len(orders)):
if orders[i] != 0.:
num_orders = (orders[i+1:i+5] != 0.).any()
if num_orders:
return -1.
return 0.
def four_weeks_from_end(orders):
if orders[-4:].any():
return -1.
else:
return 0.
con1 = {'type': 'eq', 'fun': items_in_stock}
con2 = {'type': 'eq', 'fun': four_weeks_order_distance}
con3 = {'type': 'eq', 'fun': four_weeks_from_end}
cons = [con1, con2, con3]
b = [(0, 100)]
bnds = b * len(orders)
x0 = orders
x0[0] = 10.
minimize(objective, x0, method='SLSQP', bounds=bnds, constraints=cons)

Though I am not an Operational Researcher, I believe it is because of the fact that the constraints you implemented are not continuous. I made little changes so that the constraints are now continuous in nature.
from scipy.optimize import minimize
import numpy as np
demand = np.array([5, 10, 10, 7, 3, 7, 1, 0, 0, 0, 8])
orders = np.array([0.] * len(demand))
def objective(orders):
return np.sum(orders)
def items_in_stock(orders):
"""In-equality Constraint: Idea is to keep the balance of stock and demand.
Cumulated stock should be greater than demand. Also, demand should never cross the stock.
"""
stock = 0
stock_penalty = 0
for i in range(len(orders)):
stock += orders[i]
stock -= demand[i]
if stock < 0:
stock_penalty -= abs(stock)
return stock_penalty
def four_weeks_order_distance(orders):
"""Equality Constraint: An order can't be placed until four weeks after any other order.
"""
violation_count = 0
for i in range(len(orders) - 6):
if orders[i] != 0.:
num_orders = orders[i + 1: i + 5].sum()
violation_count -= num_orders
return violation_count
def four_weeks_from_end(orders):
"""Equality Constraint: No orders in the last 4 weeks
"""
return orders[-4:].sum()
con1 = {'type': 'ineq', 'fun': items_in_stock} # Forces value to be greater than zero.
con2 = {'type': 'eq', 'fun': four_weeks_order_distance} # Forces value to be zero.
con3 = {'type': 'eq', 'fun': four_weeks_from_end} # Forces value to be zero.
cons = [con1, con2, con3]
b = [(0, 100)]
bnds = b * len(orders)
x0 = orders
x0[0] = 10.
res = minimize(objective, x0, method='SLSQP', bounds=bnds, constraints=cons,
options={'eps': 1})
Results
status: 0
success: True
njev: 22
nfev: 370
fun: 51.000002688311334
x: array([ 5.10000027e+01, 1.81989405e-15, -6.66999371e-16,
1.70908182e-18, 2.03187432e-16, 1.19349893e-16,
1.25059614e-16, 4.55582386e-17, 6.60988392e-18,
3.37907550e-17, -5.72760251e-18])
message: 'Optimization terminated successfully.'
jac: array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0.])
nit: 23
[ round(l, 2) for l in res.x ]
[51.0, 0.0, -0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.0]
So, the solution suggests to make all the orders in the first week.
It avoids the out of stock situation
Single purchase(order) respects the no order in the next four week after an order.
No last 4 week purchase

Minimization of an equation using Python

I have four vectors.
x = [0.4, -0.3, 0.9]
y1 = [0.3, 1, 0]
y2 = [1, -0.9, 0.5]
y3 =[0.6, 0.01, 0.8]
I need to minimize following equation:
where 0 <= a,b,g <= 1. I have tried to use scipy.minimize but I could not understand how that can be used for this equation. Is there any library for optimization that I can use or is there any easier way in Python to do it?
My ultimate goal is to find values of a,b,g between 0-1 that give me minimum value given these four vectors as input.

Edit 0: I fixed the problem by using a Bounds instance. The array x should be what you are looking for. Here is the answer.
fun: 0.34189582276366093
hess_inv: <3x3 LbfgsInvHessProduct with dtype=float64>
jac: array([ 6.91014296e-01, 3.49720253e-07, -2.88657986e-07])
message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
nfev: 40
nit: 8
status: 0
success: True
x: array([0. , 0.15928136, 0.79907217])
I worked on that a little bit. I get stuck with an error, but I feel like I am on the correct way to solve it. Here is the code.
import numpy as np
from scipy.optimize import Bounds,minimize
def cost_function(ini):
x = np.array([0.4, -0.3, 0.9])
y1 = np.array([0.3, 1, 0])
y2 = np.array([1, -0.9, 0.5])
y3 =np.array([0.6, 0.01, 0.8])
L = np.linalg.norm(np.transpose(x) - np.dot(ini[0],y1) - np.dot(ini[1],y2) - np.dot(ini[2],y3))
return L
ini = np.random.rand(3)
min_b= np.zeros(3)
max_b= np.ones(3)
bnds=Bounds(min_b,max_b)
print(minimize(cost_function,x0 =ini,bounds=bnds))
However, I am getting the error ValueError: length of x0 != length of bounds, although the lengths are equal. I could not find a solution, maybe you do. Good luck! Let me know if you find a solution and if it works!

scipy.optimize Constrained Minimization SLSQP - Cannot Match the target 100%

I am trying to get the minimized weight that's closer to average weight combined. My current problem is using the SLSQP solver I cannot find the right weight that meet the target 100%. Is there another solver I could use to solve my problem? Or any math suggestions. Please help.
My math right now is
**min⁡(∑|x-mean(x)|)**
**s.t.** Aw-b=0, w>=0
**bound** 0.2'<'x<0.5
Data
nRow# Variable1 Variable2 Variable3
1 3582.00 233445193.00 559090945.00
2 3394.00 217344811.00 496500751.00
3 3356.00 237746918.00 493639029.00
4 3256.00 219204892.00 461547877.00
5 3415.00 225272825.00 501057960.00
6 3505.00 242819442.00 505073223.00
7 3442.00 215258725.00 490458632.00
8 3381.00 227681178.00 503102998.00
9 3392.00 215189377.00 487026744.00
w1 w2 w3
Target 8531.00 429386951.00 1079115532.00
Question: Find the minimized weight that are closer to average
Python Code:
A = np.array([[3582.000000, 3394.000000, 3356.000000, 3256.000000, 3415.000000,
3505.000000, 3442.000000, 3381.000000, 3392.000000],
[233445193.000000, 217344811.000000, 237746918.000000,
219204892.000000, 225272825.000000, 242819442.000000,
215258725.000000, 227681178.000000, 215189377.000000],
[559090945.000000, 496500751.000000, 493639029.000000,
461547877.000000, 501057960.000000, 505073223.000000,
490458632.000000, 503102998.000000, 487026744.000000]])
b = np.array([8531, 1079115532, 429386951])
n=9
def fsolveMin(A,b,n):
# The constraints: Ax = b
def cons(x):
return A.dot(x)-b
cons = ({'type':'eq','fun':cons},
{'type':'ineq','fun':lambda x:x[0]})
# The minimizing constraints: the total absolute difference between the coefficients
def fn(x):
return np.sum(np.abs(x-np.mean(x)))
# Initialize the coefficients randomly
z0 = abs(np.random.randn(len(A[1,:])))
# Set up bound
# bnds = [(0, None)]*n
# Solve the problem
sol = minimize(fn, x0 = z0, constraints = cons, method = 'SLSQP', options={'disp': True})
#expected 35%
print(sol.x)
print(A.dot(sol.x))
#print(fn(sol.x))
print(str(fsolveMin(A,b,n))+"\n\n")

To give you an idea how bloated this will get with low-level tools like scipy's linprog as we have to mimic their standard-form:
The basic idea is to:
add an auxiliary-variable for the mean, like Erwin proposed
add n-auxiliary-variables to handle the absolute values like documented in lpsolve's docs
(and i'm using n-extra-aux-vars as temp-vars for x-mean)
Now this example works.
For your data you have to be careful in regards to the bounds. Make sure, the problem stays feasible. The problem itself is pretty unstable as hard-constraints are used for Ax=b; usually you would minimize some norm / least-squares here (no LP anymore; QP/SOCP) and add this error to the objective)!
It might be needed to switch the solver from method='simplex' to method='interior-point' at some point (only available since scipy 1.0).
Alternative:
Using cvxpy, formulation is much easier (both variants mentioned) and you get quite a good solver (ECOS) for free.
Code:
import numpy as np
import scipy.optimize as spo
np.set_printoptions(linewidth=120)
np.random.seed(1)
""" Create random data """
M, N = 2, 3
A = np.random.random(size=(M,N))
x_hidden = np.random.random(size=N)
b = A.dot(x_hidden) # target
n = N
print('Original A')
print(A)
print('hidden x: ', x_hidden)
print('target: ', b)
""" Optimize as LP """
def solve(A, b, n):
print('Reformulation')
am, an = A.shape
# Introduce aux-vars
# 1: y = mean
# n: z = x - mean
# n: abs(z)
n_plus_aux_vars = 3*n + 1
# Equality constraint: y = mean
eq_mean_A = np.zeros(n_plus_aux_vars)
eq_mean_A[:n] = 1. / n
eq_mean_A[n] = -1.
eq_mean_b = np.array([0])
print('y = mean A:')
print(eq_mean_A)
print('y = mean b:')
print(eq_mean_b)
# Equality constraints: Ax = b
eq_A = np.zeros((am, n_plus_aux_vars))
eq_A[:, :n] = A[:, :n]
eq_b = np.copy(b)
print('Ax=b A:')
print(eq_A)
print('Ax=b b:')
print(eq_b)
# Equality constraints: z = x - mean
eq_mean_A_z = np.hstack([-np.eye(n), np.ones((n, 1)), + np.eye(n), np.zeros((n, n))])
eq_mean_b_z = np.zeros(n)
print('z = x - mean A:')
print(eq_mean_A_z)
print('z = x - mean b:')
print(eq_mean_b_z)
# Inequality constraints: absolute values -> x <= x' ; -x <= x'
ineq_abs_0_A = np.hstack([np.zeros((n, n)), np.zeros((n, 1)), np.eye(n), -np.eye(n)])
ineq_abs_0_b = np.zeros(n)
ineq_abs_1_A = np.hstack([np.zeros((n, n)), np.zeros((n, 1)), -np.eye(n), -np.eye(n)])
ineq_abs_1_b = np.zeros(n)
# Bounds
# REMARK: Do not touch anything besides the first bounds-row!
bounds = [(0., 1.) for i in range(n)] + \
[(None, None)] + \
[(None, None) for i in range(n)] + \
[(0, None) for i in range(n)]
# Objective
c = np.zeros(n_plus_aux_vars)
c[-n:] = 1
A_eq = np.vstack((eq_mean_A, eq_A, eq_mean_A_z))
b_eq = np.hstack([eq_mean_b, eq_b, eq_mean_b_z])
A_ineq = np.vstack((ineq_abs_0_A, ineq_abs_1_A))
b_ineq = np.hstack([ineq_abs_0_b, ineq_abs_1_b])
print('solve...')
result = spo.linprog(c, A_ineq, b_ineq, A_eq, b_eq, bounds=bounds, method='simplex')
print(result)
x = result.x[:n]
print('x: ', x)
print('residual Ax-b: ', A.dot(x) - b)
print('mean: ', result.x[n])
print('x - mean: ', x - result.x[n])
print('l1-norm(x - mean) / objective: ', np.linalg.norm(x - result.x[n], 1))
solve(A, b, n)
Output:
Original A
[[ 4.17022005e-01 7.20324493e-01 1.14374817e-04]
[ 3.02332573e-01 1.46755891e-01 9.23385948e-02]]
hidden x: [ 0.18626021 0.34556073 0.39676747]
target: [ 0.32663584 0.14366255]
Reformulation
y = mean A:
[ 0.33333333 0.33333333 0.33333333 -1. 0. 0. 0. 0. 0. 0. ]
y = mean b:
[0]
Ax=b A:
[[ 4.17022005e-01 7.20324493e-01 1.14374817e-04 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00]
[ 3.02332573e-01 1.46755891e-01 9.23385948e-02 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00]]
Ax=b b:
[ 0.32663584 0.14366255]
z = x - mean A:
[[-1. -0. -0. 1. 1. 0. 0. 0. 0. 0.]
[-0. -1. -0. 1. 0. 1. 0. 0. 0. 0.]
[-0. -0. -1. 1. 0. 0. 1. 0. 0. 0.]]
z = x - mean b:
[ 0. 0. 0.]
solve...
fun: 0.078779576294411263
message: 'Optimization terminated successfully.'
nit: 10
slack: array([ 0.07877958, 0. , 0. , 0. , 0.07877958, 0. , 0.76273076, 0.68395118,
0.72334097])
status: 0
success: True
x: array([ 0.23726924, 0.31604882, 0.27665903, 0.27665903, -0.03938979, 0.03938979, 0. , 0.03938979,
0.03938979, 0. ])
x: [ 0.23726924 0.31604882 0.27665903]
residual Ax-b: [ 5.55111512e-17 0.00000000e+00]
mean: 0.276659030053
x - mean: [-0.03938979 0.03938979 0. ]
l1-norm(x - mean) / objective: 0.0787795762944
Now for your original data, things get tough!
You need to:
scale your data as the magnitude of your variables will hurt the solver
depending on your task you might need to analyze the effects of scaling / invert it
use the interior-point solver
be careful with the bounds
Example:
A = np.array([[3582.000000, 3394.000000, 3356.000000, 3256.000000, 3415.000000,
3505.000000, 3442.000000, 3381.000000, 3392.000000],
[233445193.000000, 217344811.000000, 237746918.000000,
219204892.000000, 225272825.000000, 242819442.000000,
215258725.000000, 227681178.000000, 215189377.000000],
[559090945.000000, 496500751.000000, 493639029.000000,
461547877.000000, 501057960.000000, 505073223.000000,
490458632.000000, 503102998.000000, 487026744.000000]])
b = np.array([8531., 1079115532., 429386951.])
A /= 10000. # scaling
b /= 10000. # scaling
bounds = [(-50., 50.) for i in range(n)] + \
...
result = spo.linprog(c, A_ineq, b_ineq, A_eq, b_eq, bounds=bounds, method='interior-point')
Output:
solve...
con: array([ 3.98410760e-09, 1.18067724e-08, 8.12879938e-04, 1.75969041e-03, -3.93853838e-09, -3.96305566e-09,
-4.10043555e-09, -3.94957667e-09, -3.88362764e-09, -3.89452381e-09, -3.95134592e-09, -3.92182287e-09,
-3.85762178e-09])
fun: 52.742900697626389
message: 'Optimization terminated successfully.'
nit: 8
slack: array([ 5.13245265e+01, 1.89309145e-08, 1.83429094e-09, 4.28687782e-09, 1.03726911e-08, 2.77000474e-09,
1.41837413e+00, 6.75769654e-09, 8.65285462e-10, 2.78501844e-09, 3.09591539e-09, 5.27429006e+01,
1.30944103e-08, 5.32994799e-09, 3.15369669e-08, 2.51943821e-09, 7.54848797e-09, 3.22510447e-09])
status: 0
success: True
x: array([ -2.51938304e+01, 4.68432810e-01, 2.68398831e+01, 4.68432822e-01, 4.68432815e-01, 4.68432832e-01,
-2.40754247e-01, 4.68432818e-01, 4.68432819e-01, 4.68432822e-01, -2.56622633e+01, -7.91749954e-09,
2.63714503e+01, 4.40376624e-09, -2.52137156e-09, 1.43834811e-08, -7.09187065e-01, 3.95395716e-10,
1.17990950e-09, 2.56622633e+01, 1.10134149e-08, 2.63714503e+01, 8.69064406e-09, 7.85131955e-09,
1.71534858e-08, 7.09187068e-01, 7.15309226e-09, 2.04519496e-09])
x: [-25.19383044 0.46843281 26.83988313 0.46843282 0.46843282 0.46843283 -0.24075425 0.46843282 0.46843282]
residual Ax-b: [ -1.18067724e-08 -8.12879938e-04 -1.75969041e-03]
mean: 0.468432821891
x - mean: [ -2.56622633e+01 -1.18805552e-08 2.63714503e+01 4.54189575e-10 -6.40499920e-09 1.04889573e-08 -7.09187069e-01
-3.52642715e-09 -2.67771227e-09]
l1-norm(x - mean) / objective: 52.7429006758
Edit
Here a SOCP-based least-squares (soft-constrained) approach, which i would recommend in terms of numerical-stability! This approach can and should be tuned for whatever you need. It's implemented using the already mentioned cvxpy modelling-tool using the ECOS solver.
The basic idea:
instead of: min(l1-norm(x - mean(x)) st. Ax=b
solve: min(l2-norm(Ax-b) + c * l1-norm(x - mean(x)))
where c is the nonnegative trade-off parameter
small c: Ax=b is more important
big c: x - mean(x) is more important
Example code & output for your data and bounds of -50, 50 and c=1e-3:
import numpy as np
import cvxpy as cvx
""" DATA """
A = np.array([[3582.000000, 3394.000000, 3356.000000, 3256.000000, 3415.000000,
3505.000000, 3442.000000, 3381.000000, 3392.000000],
[233445193.000000, 217344811.000000, 237746918.000000,
219204892.000000, 225272825.000000, 242819442.000000,
215258725.000000, 227681178.000000, 215189377.000000],
[559090945.000000, 496500751.000000, 493639029.000000,
461547877.000000, 501057960.000000, 505073223.000000,
490458632.000000, 503102998.000000, 487026744.000000]])
b = np.array([8531., 1079115532., 429386951.])
n = 9
# A /= 10000. scaling would be a good idea
# b /= 10000. """
""" SOCP-based least-squares approach """
def solve(A, b, n, c=1e-1):
x = cvx.Variable(n)
y = cvx.Variable(1) # mean
lower_bounds = np.zeros(n) - 50 # -50
upper_bounds = np.zeros(n) + 50 # 50
constraints = []
constraints.append(x >= lower_bounds)
constraints.append(x <= upper_bounds)
constraints.append(y == cvx.sum_entries(x) / n)
objective = cvx.Minimize(cvx.norm(A*x-b, 2) + c * cvx.norm(x - y, 1))
problem = cvx.Problem(objective, constraints)
problem.solve(solver=cvx.ECOS, verbose=True)
print('Objective: ', problem.value)
print('x: ', x.T.value)
print('mean: ', y.value)
print('Ax-b: ')
print((A*x - b).value)
print('x - mean: ', (x - y).T.value)
solve(A, b, n)
Output:
ECOS 2.0.4 - (C) embotech GmbH, Zurich Switzerland, 2012-15. Web: www.embotech.com/ECOS
It pcost dcost gap pres dres k/t mu step sigma IR | BT
0 +2.637e-17 -1.550e+06 +7e+08 1e-01 2e-04 1e+00 2e+07 --- --- 1 1 - | - -
1 -8.613e+04 -1.014e+05 +8e+06 1e-03 2e-06 2e+03 2e+05 0.9890 1e-04 2 1 1 | 0 0
2 -1.287e+03 -1.464e+03 +1e+05 1e-05 9e-08 4e+01 3e+03 0.9872 1e-04 3 1 1 | 0 0
3 +1.794e+02 +1.900e+02 +2e+03 2e-07 1e-07 1e+01 5e+01 0.9890 5e-03 5 3 4 | 0 0
4 -1.388e+00 -6.826e-01 +1e+02 1e-08 7e-08 9e-01 3e+00 0.9458 4e-03 7 6 6 | 0 0
5 +5.491e+00 +5.683e+00 +1e+01 1e-09 8e-09 2e-01 3e-01 0.9617 6e-02 1 1 1 | 0 0
6 +6.480e+00 +6.505e+00 +1e+00 2e-10 5e-10 3e-02 4e-02 0.8928 2e-02 1 1 1 | 0 0
7 +6.746e+00 +6.746e+00 +2e-02 3e-12 5e-10 5e-04 6e-04 0.9890 5e-03 1 0 0 | 0 0
8 +6.759e+00 +6.759e+00 +3e-04 2e-12 2e-10 6e-06 7e-06 0.9890 1e-04 1 0 0 | 0 0
9 +6.759e+00 +6.759e+00 +3e-06 2e-13 2e-10 6e-08 8e-08 0.9890 1e-04 2 0 0 | 0 0
10 +6.758e+00 +6.758e+00 +3e-08 5e-14 2e-10 7e-10 9e-10 0.9890 1e-04 1 0 0 | 0 0
OPTIMAL (within feastol=2.0e-10, reltol=4.7e-09, abstol=3.2e-08).
Runtime: 0.002901 seconds.
Objective: 6.757722879805085
x: [[-18.09169736 -5.55768047 11.12130645 11.48355878 -1.13982006
12.4290884 -3.00165819 -1.05158589 -2.4468432 ]]
mean: 0.416074272576
Ax-b:
[[ 2.17051777e-03]
[ 1.90734863e-06]
[ -5.72204590e-06]]
x - mean: [[-18.50777164 -5.97375474 10.70523218 11.0674845 -1.55589434
12.01301413 -3.41773246 -1.46766016 -2.86291747]]
This approach will always output a feasible solution (for our task) and you can then decide on the observed residual if it works for you.
As you observed, a lower-bound of 0 is deadly, in all formulations (look at the magnitude difference in your data!).
Here a lower-bound of 0 will get you a solution with some high residual-error.
E.g.:
c=1e-7
bounds = 0 / 15
Output:
Objective: 785913288.2410747
x: [[ -5.57966858e-08 -4.74997454e-08 1.56066068e+00 1.68021234e-07
-3.55602958e-08 1.75340641e-06 -4.69609562e-08 -3.10216680e-08
-4.39482554e-08]]
mean: 0.173406926909
Ax-b:
[[ -3.29341696e+03]
[ -7.08072860e+08]
[ 3.41016903e+08]]
x - mean: [[-0.17340698 -0.17340697 1.38725375 -0.17340676 -0.17340696 -0.17340517
-0.17340697 -0.17340696 -0.17340697]]

First introduce a free variable mu with the constraint:
mu = sum(i, x(i))/n
Then introduce non-negative variables y(i) with:
-y(i) <= x(i) - mu <= y(i)
Now you can minimize
min sum(i,y(i))
This is now a straight LP (linear objective and linear constraints) and can be solved with any LP solver.