We Keep Coding

PuLP: Minimizing the standard deviation of decision variables

In an optimization problem developed in PuLP i use the following objective function:
objective = p.lpSum(vec[r] for r in range(0,len(vec)))
All variables are non-negative integers, hence the sum over the vector gives the total number of units for my problem.
Now i am struggling with the fact, that PuLP only gives one of many solutions and i would like to narrow down the solution space to results that favors the solution set with the smallest standard deviation of the decision variables.
E.g. say vec is a vector with elements 6 and 12. Then 7/11, 8/10, 9/9 are equally feasible solutions and i would like PuLP to arrive at 9/9.
Then the objective
objective = p.lpSum(vec[r]*vec[r] for r in range(0,len(vec)))
would obviously create a cost function, that would help the case, but alas, it is non-linear and PuLP throws an error.
Anyone who can point me to a potential solution?

Instead of minimizing the standard deviation (which is inherently non-linear), you could minimize the range or bandwidth. Along the lines of:
minimize maxv-minv
maxv >= vec[r] for all r
minv <= vec[r] for all r

scipy bvp solver for four differential equation systems

I am trying to solve this systems but I get error.
I have to definition y3d=0 because y3'=0 in the equation systems. but when I did this, program cant solve. if I say y3d=y[3] then program run,
equation system that ı have to solve is like this:
dy1/dx=y2
dy2/dx=-y3*y1
dy3/dx=0
dy4/d=y1**2+y2**2 and boundary condition y1(0)=y1(1)=0 and y4(0)= 0 y4(1)=1
can scipy handle this?
import numpy as np
from scipy.integrate import solve_bvp
import matplotlib.pyplot as plt
def eqts(x,y):
y1d=y[1]
y2d=-y[2]*y[0]
y3d=0
y4d=y[0]**2+y[1]**2
return np.vstack((y1d,y2d,y3d,y4d))
def bc(ya,yb):
return np.array([ya[0],yb[3],ya[0],yb[3]-1])
x = np.linspace(0,1,10)
y= np.zeros((4,x.size))
y[2,:]=1
sol=solve_bvp(eqts,bc,x,y)
Unfortunately I get the following error message ;
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 10 and the array at index 2 has size 1

Well, first of all, in your script, your boundary conditions are overdetermined. Nowhere it is said that y3(0) = 0 or y3(1) = 0. Actually, it is not: y3(t) is a constant but it is not zero. If you impose such condition y3(t) = 0, things will not work at all. On top of that, this system looks non-linear (quadratic) but actually is a linear system. You can solve it explicitly without python. If I am not mistaken, the only way you can have a solution is when y3 > 0, which gives you
y1(t) = B * sin(k*pi*t)
y2(t) = k*pi*B * cos(k*pi*t)
y3(t) = k^2*pi^2
y4(t) = t + (k^2pi^2 - 1) B^2 * sin(2*k*pi*t) / (4*k*pi)
where B = sqrt( 2*pi*k / (k^2*pi^2 + 1) )
and k is an arbitrary non-zero integer
or at least something along those lines.

Determinant
The main issue is here
y1d=y[1]
y2d=-y[2]*y[0]
y3d=0
y4d=y[0]**2+y[1]**2
In your implementation all y1d, y2d, and y4d are vector, but y3d is scalar!
You may use y3d = np..zeros_like(y1d)
solve_bvp requires to return a rhs with same size of y see https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_bvp.html?highlight=solve_bvp#scipy.integrate.solve_bvp

In scipy's solve_bvp there is the possibility to treat constant components as parameters that are to be fitted. Thus one can define the system in dimension 3 as
def eqn(t,x,p): return x[1], -p[0]*x[0],x[0]**2+x[1]**2
def bc(x0,x1,p): return x0[0], x1[0], x0[2], x1[2]-1
t_init = np.linspace(0,1,6)
x_init = np.zeros([3,len(t_init)])
x_init[0,1]=1; # break the symmetry that gives a singular Jacobian
Another variant to get a general initial guess would be to fill it with random noise.
To get different solutions the initial conditions need to be different, one point of influence is the frequency square. Setting it close to the expected values gives indeed different solutions.
for w0 in np.arange(1,10)*3:
res = solve_bvp(eqn,bc,t_init,x_init,p=[w0**2])
print(res.message,res.p[0])
This gives as output
The algorithm converged to the desired accuracy. 9.869869348810965
The algorithm converged to the desired accuracy. 39.47947747757308
The algorithm converged to the desired accuracy. 88.82805487260174
The algorithm converged to the desired accuracy. 88.8280548726001
The algorithm converged to the desired accuracy. 157.91618155379464
The algorithm converged to the desired accuracy. 157.91618155379678
The algorithm converged to the desired accuracy. 355.31352482772894
The algorithm converged to the desired accuracy. 355.3135248277307
The algorithm converged to the desired accuracy. 157.91618163528014
As one can see, in the higher frequencies the given initial frequency gets balanced against the lower frequency behavior of the initial functions. This is a general problem, if not forced to stay orthogonal to the lower frequency solutions, the solver tends towards the smoother lower frequencies.
Adding plot commands plt.plot(res.x, res.y[0]) etc.
shows the expected sinusoidal solutions.

Python constrained non-linear optimization with scipy.optimize fails to find optimal solutions

I have a problem I am trying to find a solution for where I have 5 single variable polynomials that have a single peak in the range i'm concerned with. My goal is to find some values of the variable for each polynomial (under certain min/max and sum of all variables constraints) that maximizes the value of these curves multiplied by a constant for each curve.
I've set up some code using the scipy.optimize package and numpy. It seems to be able to reach solution, but the solution it reaches does not appear to be anywhere close to optimal. For example, the trivial case is for an input of 488 MW. This particular input value has a solution where each x0-x4 variable is it at the peak of it's function, which is as follows:
x0=90
x1=100
x2=93
x3=93
x4=112
The result it provides we with is:
x0=80
x1=97
x2=105
x3=80
x4=126
This does satisfy my constraint, but it does not appear to minimize the objective function.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
U1_Max=103
U1_Min=80
U2_Max=102
U2_Min=80
U3_Max=105
U3_Min=80
U4_Max=100
U4_Min=80
U5_Max=126
U5_Min=90
# Our whole goal here is to maximze the sum of several efficiency efficiency curves times the
# output MW of each unit. "The most efficiency where it matters the most"
# Assuming all units are available for assignment his would look something like:
#Where have the following efficiency curves:
#U1: Efficiency=-0.0231*(MW)^2+4.189*MW-102.39
#U2: Efficiency= -0.01*(MW)^2+1.978*MW-8.7451
#U3: Efficiency= -0.025*MW^2+4.5017*MW-115.37
#U4: Efficiency= -0.01*(MW)^2+1.978*MW-8.7451
#U5: Efficiency= -0.0005*(MW)^2+0.1395*(MW)^2-13.327*MW+503.41
#So I think we want to
#Maximize U1(x0)*U1_MAX+U2(x1)*U2_MAX+U3(x2)*U3_MAX+U4(x3)*U4_MAX+U5(x4)*U5_MAX
#I think this can also be stated as:
#Minimize (U1(x0)-100)*U1_MAX+(U2(x1)-100)*U2_MAX)+(U3(x2)-100)*U3_MAX)+(U4(x3)-100)*U4_MAX)+(U5(x4)-100)*U5_MAX)
#Which means 'minimize the sum of the products of the difference between 100% efficient and actual and the unit nameplates'
#By Choosing {x1, x2, x3, x4, x5}
#Such that x1+x2+x3+x4+x5=MW_Total
#Such that U1_Min<x1<U1Max
#Such that U2_Min<x2<U2Max
#Such that U3_Min<x3<U3Max
#Such that U4_Min<x4<U4Max
#Such that U5_Min<x5<U5Max
##so let's type that out....
#stack overflow says the optimizer does best if the function being optimized is around 1-5ish so we will get it there-ish.
def objective(x):
return (
(
((100-0.0231*x[0]**2+4.189*x[0]-102.39))*U1_Max
+((100-0.01*x[1]**2+1.978*x[1]-8.7451))*U2_Max
+((100-0.025*x[2]**2+4.5017*x[2]-115.37))*U3_Max
+((100-0.01*x[3]**2+1.978*x[3]-8.7451))*U4_Max
+((100-0.0005*x[4]**3+0.1395*x[4]**2-13.327*x[4]+503.41))*U5_Max
)
)
x=np.zeros(5)
print(
(
((100-0.0231*x[0]**2+4.189*x[0]-102.39))*U1_Max
+((100-0.01*x[1]**2+1.978*x[1]-8.7451))*U2_Max
+((100-0.025*x[2]**2+4.5017*x[2]-115.37))*U3_Max
+((100-0.01*x[3]**2+1.978*x[3]-8.7451))*U4_Max
+((100-0.0005*x[4]**3+0.1395*x[4]**2-13.327*x[4]+503.41))*U5_Max
)
)
#Now, let's formally define our constraints
#Note that this must be of a form that satisfies 'constraint equal to zero'
#First, the sum of all MW commands should be qual to the total MW commanded
def constraint1(x):
return -x[0]-x[1]-x[2]-x[3]-x[4]+MW_Total
#Since this is a numeric process let's give it some starting 'guess' conditions.
n=5
x0=np.zeros(n)
x0[0]=90
x0[1]=100
x0[2]=93
x0[3]=93
x0[4]=112
# show initial starting uess
print('Start by guessing: ')
print(x0)
print('Which gives a scaled algorithim value of: ')
print(
(
((100-0.0231*x0[0]**2+4.189*x0[0]-102.39))*U1_Max
+((100-0.01*x0[1]**2+1.978*x0[1]-8.7451))*U2_Max
+((100-0.025*x0[2]**2+4.5017*x0[2]-115.37))*U3_Max
+((100-0.01*x0[3]**2+1.978*x0[3]-8.7451))*U4_Max
+((100-0.0005*x0[4]**3+0.1395*x0[4]**2-13.327*x0[4]+503.41))*U5_Max
)
)
print('Which gives actual MW total of: ')
print(x0[0]+x0[1]+x0[2]+x0[3]+x0[4])
#Next, Let's give it some bounds to operate in
U1_Bnds=(U1_Min, U1_Max)
U2_Bnds=(U2_Min, U2_Max)
U3_Bnds=(U3_Min, U3_Max)
U4_Bnds=(U4_Min, U4_Max)
U5_Bnds=(U5_Min, U5_Max)
Bnds=(U1_Bnds, U2_Bnds, U3_Bnds, U4_Bnds, U5_Bnds)
con1 = {'type': 'eq', 'fun': constraint1}
print('MW Generated is: ')
for i in range (410,536):
MW_Total=i
solution = minimize(objective,x0,method='SLSQP',bounds=Bnds,constraints=con1,options={'maxiter': 10000000, 'eps':1.4901161193847656e-10})
x = solution.x
print(solution.x[0],solution.x[1],solution.x[2],solution.x[3],solution.x[4])
I would expect that for my trivial case of 488 MW it would give me the optimal answer. What am I doing wrong?

By looking at your objective and constraint definition, it looks like you are in the case of a quadratic objective function with a linear constraint.
The theory for this is well known and provides convergence guarantees, you can refer to the wikipedia page.
I don't know that well the scipy SLSQP interface but it looks like you are using less information than what you could do. Try to cast your problem in the form of Quadratic objective function with linear constraint. Also cast your constraint in a scipy.optimize.LinearConstraint object.
And please use functions calls such as print(objective(x)) and print(solution.x) in your code, this would enhance readability.

Derivatives blow up in python

I am trying to find higher order derivatives of a dataset (x,y). x and y are 1D arrays of length N.
Let's say I generate them as :
xder0=np.linspace(0,10,1000)
yder0=np.sin(xder0)
I define the derivative function which takes in 2 array (x,y) and returns (x1, y1) where y1 is the derivative calculated at each index as : (y[i+1]-y[i])/(x[i+1]-x[i]). x1 is just the mean of x[i+1] and x[i]
Here is the function that does it:
def deriv(x,y):
delx =np.zeros((len(x)-1), dtype=np.longdouble)
ydiff=np.zeros((len(x)-1), dtype=np.longdouble)
for i in range(len(x)-1):
delx[i] =(x[i+1]+x[i])/2.0
ydiff[i] =(y[i+1]-y[i])/(x[i+1]-x[i])
return delx, ydiff
Now to calculate the first derivative, I call this function as:
xder1, yder1 = deriv(xder0, yder0)
Similarly for second derivative, I call this function giving first derivatives as input:
xder2, yder2 = deriv(xder1, yder1)
And it goes on:
xder3, yder3 = deriv(xder2, yder2)
xder4, yder4 = deriv(xder3, yder3)
xder5, yder5 = deriv(xder4, yder4)
xder6, yder6 = deriv(xder5, yder5)
xder7, yder7 = deriv(xder6, yder6)
xder8, yder8 = deriv(xder7, yder7)
xder9, yder9 = deriv(xder8, yder8)
Something peculiar happens after I reach order 7. The 7th order becomes very noisy! Earlier derivatives are all either sine or cos functions as expected. However 7th order is a noisy sine. And hence all derivatives after that blow up.
Any idea what is going on?

This is a well known stability issue with numerical interpolation using equally-spaced points. Read the answers at http://math.stackexchange.com.
To overcome this problem you have to use non-equally-spaced points, like the roots of Lagendre polynomial. The instability occurs due to the unavailability of information at the boundaries, thus more concentration of points at the boundaries is required, as per the roots of say Lagendre polynomials or others with similar properties such as Chebyshev polynomial.

On ordinary differential equations (ODE) and optimization, in Python

I want to solve this kind of problem:
dy/dt = 0.01*y*(1-y), find t when y = 0.8 (0<t<3000)
I've tried the ode function in Python, but it can only calculate y when t is given.
So are there any simple ways to solve this problem in Python?
PS: This function is just a simple example. My real problem is so complex that can't be solve analytically. So I want to know how to solve it numerically. And I think this problem is more like an optimization problem:
Objective function y(t) = 0.8, Subject to dy/dt = 0.01*y*(1-y), and 0<t<3000
PPS: My real problem is:
objective function: F(t) = 0.85,
subject to: F(t) = sqrt(x(t)^2+y(t)^2+z(t)^2),
x''(t) = (1/F(t)-1)*250*x(t),
y''(t) = (1/F(t)-1)*250*y(t),
z''(t) = (1/F(t)-1)*250*z(t)-10,
x(0) = 0, y(0) = 0, z(0) = 0.7,
x'(0) = 0.1, y'(0) = 1.5, z'(0) = 0,
0<t<5

This differential equation can be solved analytically quite easily:
dy/dt = 0.01 * y * (1-y)
rearrange to gather y and t terms on opposite sides
100 dt = 1/(y * (1-y)) dy
The lhs integrates trivially to 100 * t, rhs is slightly more complicated. We can always write a product of two quotients as a sum of the two quotients * some constants:
1/(y * (1-y)) = A/y + B/(1-y)
The values for A and B can be worked out by putting the rhs on the same denominator and comparing constant and first order y terms on both sides. In this case it is simple, A=B=1. Thus we have to integrate
1/y + 1/(1-y) dy
The first term integrates to ln(y), the second term can be integrated with a change of variables u = 1-y to -ln(1-y). Our integrated equation therefor looks like:
100 * t + C = ln(y) - ln(1-y)
not forgetting the constant of integration (it is convenient to write it on the lhs here). We can combine the two logarithm terms:
100 * t + C = ln( y / (1-y) )
In order to solve t for an exact value of y, we first need to work out the value of C. We do this using the initial conditions. It is clear that if y starts at 1, dy/dt = 0 and the value of y never changes. Thus plug in the values for y and t at the beginning
100 * 0 + C = ln( y(0) / (1 - y(0) )
This will give a value for C (assuming y is not 0 or 1) and then use y=0.8 to get a value for t. Note that because of the logarithm and the factor 100 multiplying t y will reach 0.8 within a relatively short range of t values, unless the initial value of y is incredibly small. It is of course also straightforward to rearrange the equation above to express y in terms of t, then you can plot the function as well.
Edit: Numerical integration
For a more complexed ODE which cannot be solved analytically, you will have to try numerically. Initially we only know the value of the function at zero time y(0) (we have to know at least that in order to uniquely define the trajectory of the function), and how to evaluate the gradient. The idea of numerical integration is that we can use our knowledge of the gradient (which tells us how the function is changing) to work out what the value of the function will be in the vicinity of our starting point. The simplest way to do this is Euler integration:
y(dt) = y(0) + dy/dt * dt
Euler integration assumes that the gradient is constant between t=0 and t=dt. Once y(dt) is known, the gradient can be calculated there also and in turn used to calculate y(2 * dt) and so on, gradually building up the complete trajectory of the function. If you are looking for a particular target value, just wait until the trajectory goes past that value, then interpolate between the last two positions to get the precise t.
The problem with Euler integration (and with all other numerical integration methods) is that its results are only accurate when its assumptions are valid. Because the gradient is not constant between pairs of time points, a certain amount of error will arise for each integration step, which over time will build up until the answer is completely inaccurate. In order to improve the quality of the integration, it is necessary to use more sophisticated approximations to the gradient. Check out for example the Runge-Kutta methods, which are a family of integrators which remove progressive orders of error term at the cost of increased computation time. If your function is differentiable, knowing the second or even third derivatives can also be used to reduce the integration error.
Fortunately of course, somebody else has done the hard work here, and you don't have to worry too much about solving problems like numerical stability or have an in depth understanding of all the details (although understanding roughly what is going on helps a lot). Check out http://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html#scipy.integrate.ode for an example of an integrator class which you should be able to use straightaway. For instance
from scipy.integrate import ode
def deriv(t, y):
return 0.01 * y * (1 - y)
my_integrator = ode(deriv)
my_integrator.set_initial_value(0.5)
t = 0.1 # start with a small value of time
while t < 3000:
y = my_integrator.integrate(t)
if y > 0.8:
print "y(%f) = %f" % (t, y)
break
t += 0.1
This code will print out the first t value when y passes 0.8 (or nothing if it never reaches 0.8). If you want a more accurate value of t, keep the y of the previous t as well and interpolate between them.

As an addition to Krastanov`s answer:
Aside of PyDSTool there are other packages, like Pysundials and Assimulo which provide bindings to the solver IDA from Sundials. This solver has root finding capabilites.

Use scipy.integrate.odeint to handle your integration, and analyse the results afterward.
import numpy as np
from scipy.integrate import odeint
ts = np.arange(0,3000,1) # time series - start, stop, step
def rhs(y,t):
return 0.01*y*(1-y)
y0 = np.array([1]) # initial value
ys = odeint(rhs,y0,ts)
Then analyse the numpy array ys to find your answer (dimensions of array ts matches ys). (This may not work first time because I am constructing from memory).
This might involve using the scipy interpolate function for the ys array, such that you get a result at time t.
EDIT: I see that you wish to solve a spring in 3D. This should be fine with the above method; Odeint on the scipy website has examples for systems such as coupled springs that can be solved for, and these could be extended.

What you are asking for is a ODE integrator with root finding capabilities. They exist and the low-level code for such integrators is supplied with scipy, but they have not yet been wrapped in python bindings.
For more information see this mailing list post that provides a few alternatives: http://mail.scipy.org/pipermail/scipy-user/2010-March/024890.html
You can use the following example implementation which uses backtracking (hence it is not optimal as it is a bolt-on addition to an integrator that does not have root finding on its own): https://github.com/scipy/scipy/pull/4904/files