I often have to solve nonlinear problems in which the number of variables exceeds the number of constraints (or sometimes the other way around). Usually some of the constraints or variables are redundant in a complicated way. Is there any way to solve such problems?
Most of the scipy solvers seem to assume that the number of constraints equals the number of variables, and that the Jacobian is nonsingular. leastsq works sometimes but it doesn't even try when the constraints are fewer than the number of variables. I realize that I could just run fmin on linalg.norm(F), but this is much less efficient than any method which makes use of the Jacobian.
Here is an example of a problem which demonstrates what I am talking about. It obviously has a solution, but leastsq gives an error. Of course, this example is easy to solve by hand, I just put it here to demonstrate the issue.
import numpy as np
import scipy.optimize
mat = np.random.randn(5, 7)
def F(x):
y = np.dot(mat, x)
return np.array([ y[0]**2 + y[1]**3 + 12, y[2] + 17 ])
x0 = np.random.randn(7)
scipy.optimize.leastsq(F, x0)
The error message I get is:
Traceback (most recent call last):
File "question.py", line 13, in <module>
scipy.optimize.leastsq(F, x0)
File "/home/dstahlke/apps/scipy/lib64/python2.7/site-packages/scipy/optimize/minpack.py", line 278, in leastsq
raise TypeError('Improper input: N=%s must not exceed M=%s' % (n,m))
TypeError: Improper input: N=7 must not exceed M=2
I have scoured the net for an answer and have even asked on the SciPy mailing list, and got no response. For now I hacked the SciPy source so that the newton_krylov solver uses pinv(), but I don't think this is an optimal solution.
How about resize the return array from F() to the number of variables:
import numpy as np
import scipy.optimize
mat = np.random.randn(5, 7)
def F(x):
y = np.dot(mat, x)
return np.resize(np.array([ y[0]**2 + y[1]**3 + 12, y[2] + 17]), 7)
while True:
x0 = np.random.randn(7)
r = scipy.optimize.leastsq(F, x0)
err = F(r[0])
norm = np.dot(err, err)
if norm < 1e-6:
break
print err
Related
I was trying to solve the following optimization problem:
$\begin{array}{l}
\min \sum\limits_{i = 1}^N {\left{ {\log [1 + \exp ({\theta ^T}{u^i})] + [1 - {v^i}{\theta ^T}{u^i}]} \right}} \
U \in {R^{3X10}},V \in {R^{10}},\theta \in {R^4}
\end{array}$
My constraints are 0 <= U(i,1)*x <= 0, i=1 to N.
I have written the code as follows:
import cvxpy as cp
import numpy as np
import math as mp
N=10
U_size=3
V_size=1
n=U_size+V_size
u_i=np.ones((N,1))
Gen_U_1_3= np.random.randint(low=0,high=6,size=(N, n-1))
Gen_V_4= np.random.randint(low=0,high=2,size=(N, 1))
Gen_U_V= np.concatenate((Gen_U_1_3,Gen_V_4),axis=1)
U=np.concatenate((u_i,Gen_U_1_3),axis=1)
V=Gen_V_4
x = cp.Variable(n)
#objective function
sum=0
for i in range(N):
sum +=mp.log(1+mp.exp(x.T#U[i,:].T))+(1-(V[i,:]#x.T#U[i,:].T))
const =[(U[i,:]#x<=1, U[i,:]#x>=0) for i in range(N) ]
# Construct the problem.
prob = cp.Problem(cp.Minimize(sum),
const )
prob.solve()
result = prob.solve()
When I run this I get the following error:
Traceback (most recent call last):
File "c:\Users\Admin\demo_optimization_1.py", line 19, in <module>
sum +=mp.log(1+mp.exp(x.T#U[i,:].T))+(1-(V[i,:]#x.T#U[i,:].T))
TypeError: must be real number, not MulExpression
I am new to python. I tried to solve this error but could do this so far. Can anyone suggest what is wrong with the expression of the Objective function?
I'm trying to exponentiate a complex matrix in Python and am running into some trouble. I'm using the scipy.linalg.expm function, and am having a rather strange error message when I try the following code:
import numpy as np
from scipy import linalg
hamiltonian = np.mat('[1,0,0,0;0,-1,0,0;0,0,-1,0;0,0,0,1]')
# This works
t_list = np.linspace(0,1,10)
unitary = [linalg.expm(-(1j)*t*hamiltonian) for t in t_list]
# This doesn't
t_list = np.linspace(0,10,100)
unitary = [linalg.expm(-(1j)*t*hamiltonian) for t in t_list]
The error when the second experiment is run is:
This works!
Traceback (most recent call last):
File "matrix_exp.py", line 11, in <module>
unitary_t = [linalg.expm(-1*t*(1j)*hamiltonian) for t in t_list]
File "/usr/lib/python2.7/dist-packages/scipy/linalg/matfuncs.py", line 105, in expm
return scipy.sparse.linalg.expm(A)
File "/usr/lib/python2.7/dist- packages/scipy/sparse/linalg/matfuncs.py", line 344, in expm
X = _fragment_2_1(X, A, s)
File "/usr/lib/python2.7/dist- packages/scipy/sparse/linalg/matfuncs.py", line 462, in _fragment_2_1
X[k, k] = exp_diag[k]
TypeError: only length-1 arrays can be converted to Python scalars
This seems really strange since all I changed was the range of t I was using. Is it because the Hamiltonian is diagonal? In general, the Hamiltonians won't be, but I also want it to work for diagonal ones. I don't really know the mechanics of expm, so any help would be greatly appreciated.
That is interesting. One thing I can say is that the problem is specific to the np.matrix subclass. For example, the following works fine:
h = np.array(hamiltonian)
unitary = [linalg.expm(-(1j)*t*h) for t in t_list]
Digging a little deeper into the traceback, the exception is being raised in _fragment_2_1 in scipy.sparse.linalg.matfuncs.py, specifically these lines:
n = X.shape[0]
diag_T = T.diagonal().copy()
# Replace diag(X) by exp(2^-s diag(T)).
scale = 2 ** -s
exp_diag = np.exp(scale * diag_T)
for k in range(n):
X[k, k] = exp_diag[k]
The error message
X[k, k] = exp_diag[k]
TypeError: only length-1 arrays can be converted to Python scalars
suggests to me that exp_diag[k] ought to be a scalar, but is instead returning a vector (and you can't assign a vector to X[k, k], which is a scalar).
Setting a breakpoint and examining the shapes of these variables confirms this:
ipdb> l
751 # Replace diag(X) by exp(2^-s diag(T)).
752 scale = 2 ** -s
753 exp_diag = np.exp(scale * diag_T)
754 for k in range(n):
755 import ipdb; ipdb.set_trace() # breakpoint e86ebbd4 //
--> 756 X[k, k] = exp_diag[k]
757
758 for i in range(s-1, -1, -1):
759 X = X.dot(X)
760
761 # Replace diag(X) by exp(2^-i diag(T)).
ipdb> exp_diag.shape
(1, 4)
ipdb> exp_diag[k].shape
(1, 4)
ipdb> X[k, k].shape
()
The underlying problem is that exp_diag is assumed to be either 1D or a column vector, but the diagonal of an np.matrix object is a row vector. This highlights a more general point that np.matrix is generally less well-supported than np.ndarray, so in most cases it's better to use the latter.
One possible solution would be to use np.ravel() to flatten diag_T into a 1D np.ndarray:
diag_T = np.ravel(T.diagonal().copy())
This seems to fix the problem you're encountering, although there may be other issues relating to np.matrix that I haven't spotted yet.
I've opened a pull request here.
This is my code
import os
import sys
import numpy as np
import scipy
from scipy.optimize import leastsq
def peval (inp_mat,p):
m0,m1,m2,m3,m4,m5,m6,m7 = p
out_mat = np.array(np.zeros(inp_mat.shape,dtype=np.float32))
mid = inp_mat.shape[0]/2
for xy in range(0,inp_mat.shape[0]):
if (xy<(inp_mat.shape[0]/2)):
out_mat[xy] = ( ( (inp_mat[xy+mid]*m0)+(inp_mat[xy]*m1)+ m2 ) /( (inp_mat[xy+mid]*m6)+(inp_mat[xy]*m7)+1 ) )
else:
out_mat[xy] = ( ( (inp_mat[xy]*m3)+(inp_mat[xy-mid]*m4)+ m5 ) /( (inp_mat[xy]*m6)+(inp_mat[xy-mid]*m7)+1 ) )
return np.array(out_mat)
def residuals(p, out_mat, inp_mat):
m0,m1,m2,m3,m4,m5,m6,m7 = p
err=np.array(np.zeros(inp_mat.shape,dtype=np.float32))
if (out_mat.shape == inp_mat.shape):
for xy in range(0,inp_mat.shape[0]):
err[xy] = err[xy]+ (out_mat[xy] -inp_mat[xy])
return np.array(err)
f = open('/media/anilil/Data/Datasets/repo/txt_op/vid.txt','r')
x = np.loadtxt(f,dtype=np.int16,comments='#',delimiter='\t')
nof = x.shape[0]/72 # Find the number of frames
x1 = x.reshape(-1,60,40)
x1_1= x1[0,:,:].flatten()
x1_2= x1[1,:,:].flatten()
x= []
y= []
for xy in range(1,50,1):
y.append(x1[xy,:,:].flatten())
x.append(x1[xy-1,:,:].flatten())
x=np.array(x,dtype=np.float32)
y=np.array(y,dtype=np.float32)
length = x1_1.shape#initail guess
p0 = np.array([1,1,1,1,1,1,1,1],dtype=np.float32)
abc=leastsq(residuals, p0,args=(y,x))
print ('Size of first matrix is '+str(x1_1.shape))
print ('Size of first matrix is '+str(x1_2.shape))
print ("Done with program")
I have tried adding np.array in most places with no use.
Could someone please help me ?
Another question here is do I give the output of the residuals() as a single value by adding all errorsnp.sum(err,axis=1). or leave it the way it is ?
When I return np.sum(err,axis=1) in the function residuals(). There is no change in the initial guess. It just remains the same.
I.E error is for each item in the input output mapping. or a combined error overall ?
Example data.
Output
ValueError: object too deep for desired array
Traceback (most recent call last):
File "/media/anilil/Data/charm/mv_clean/.idea/nose_reduction_mpeg.py", line 49, in <module>
abc=leastsq(residuals, p0,args=(y,x))
File "/usr/lib/python2.7/dist-packages/scipy/optimize/minpack.py", line 378, in leastsq
gtol, maxfev, epsfcn, factor, diag)
minpack.error: Result from function call is not a proper array of floats.
leastsq requires a 1D array to be returned from your residuals function.
Currently you calculate the residuals for the whole image and return that as a 2D array.
The simple fix would be to flatten the array of residuals (turning your 2D array into a 1D one).
So instead of returning
return np.array(err)
Do this instead
return err.flatten()
Note that err is already a numpy array so doesn't need to be cast before the return (I guess that slipped in when you were trying to debug it!)
from __future__ import division
import functools
import warnings
import numpy as np
import scipy as sp
from scipy import integrate
from numpy import exp, pi
import matplotlib.pyplot as plt
warnings.simplefilter("ignore", np.ComplexWarning)
def legendrePoly(a,n):
def integrand(t):
return ( ((a + exp(2j*pi*t))**2 - 1)/(2*exp(2j*pi*t)) )**n
return sp.integrate.quad(integrand,0,1)[0]
basisDim = 6
legendreBasis = [functools.partial(legendrePoly, n=i) for i in range(basisDim)]
integrand = [lambda x,i=i: exp(x) * legendreBasis[i](x) for i in range(basisDim)]
normalizingConst = [lambda x,i=i: legendreBasis[i](x)**2 for i in range(basisDim)]
basisCoeff = [sp.integrate.quad(integrand[i],-1,1)[0]
/sp.integrate.quad(normalizingConst[i],-1,1)[0] for i in range(basisDim)]
approxPoly = lambda x: sum(basisCoeff[i]*legendreBasis[i](x) for i in range(basisDim))
t = np.arange(-1, 1, 1e-3)
plt.plot(t,exp(t),'b')
plt.plot(t,approxPoly(t),'r')
plt.show()
I'm using Legendre polynomials as a basis for a polynomial approximation to the exponential function. I'm also using Cauchy's Integral Formula to evaluate them, rather than importing them directly from numpy.
Everything runs fine up to and including defining approxPoly, and approxPoly returns the expected value for any input I enter. But for some reason when I try to plot approxPoly(t), it returns the error: Supplied function does not return a valid float.
This error seems to suggest that when my functions in legendreBasis call to scipy.integrate.quad, that something goes wrong there, but if that was the case then approxPoly wouldn't work, yet if you evaluate it at 2000 points between -1 and 1 manually, and plot these points, everything works fine, but is that not exactly what plt.plot doing when it tries to graph my function?
Traceback:
Traceback (most recent call last):
File "/private/var/folders/mb/yyp8v3_95l538z3g7jsttq540000gn/T/Cleanup At Startup/Exercise-413072597.643.py", line 34, in <module>
plt.plot(t,approxPoly(t),'r')
File "/private/var/folders/mb/yyp8v3_95l538z3g7jsttq540000gn/T/Cleanup At Startup/Exercise-413072597.643.py", line 29, in <lambda>
approxPoly = lambda x: sum(basisCoeff[i]*legendreBasis[i](x) for i in range(basisDim))
File "/private/var/folders/mb/yyp8v3_95l538z3g7jsttq540000gn/T/Cleanup At Startup/Exercise-413072597.643.py", line 29, in <genexpr>
approxPoly = lambda x: sum(basisCoeff[i]*legendreBasis[i](x) for i in range(basisDim))
File "/private/var/folders/mb/yyp8v3_95l538z3g7jsttq540000gn/T/Cleanup At Startup/Exercise-413072597.643.py", line 18, in legendrePoly
return sp.integrate.quad(integrand,0,1)[0]
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/integrate/quadpack.py", line 247, in quad
retval = _quad(func,a,b,args,full_output,epsabs,epsrel,limit,points)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/integrate/quadpack.py", line 312, in _quad
return _quadpack._qagse(func,a,b,args,full_output,epsabs,epsrel,limit)
quadpack.error: Supplied function does not return a valid float.
logout
scipy.integrate.quad doesn't integrate functions that return arrays. That means when you try to call approxPoly(t), t gets passed around until it ends up in this function:
def legendrePoly(a,n):
def integrand(t):
return ( ((a + exp(2j*pi*t))**2 - 1)/(2*exp(2j*pi*t)) )**n
return sp.integrate.quad(integrand,0,1)[0]
integrand returns an array, and sp.integrate.quad chokes. The same problem occurs a number of times in your code. Everything seems to be written assuming scalar arguments.
You can probably fix this by calling vectorize on approxPoly:
plt.plot(t,np.vectorize(approxPoly)(t),'r')
NumPy will then call approxPoly on each element of t separately.
I seem to be getting an error when I use the root-finder in scipy. I was wondering if anyone could point out what I'm doing wrong.
The function I'm finding the root of is just an easy example, and not particularly important.
If I run this code with scipy 0.9.0:
import numpy as np
from scipy.optimize import fsolve
tmpFunc = lambda xIn: (xIn[0]-4)**2 + (xIn[1]-5)**2 + (xIn[2]-7)**3
x0 = [3,4,5]
xFinal = fsolve(tmpFunc, x0 )
print xFinal
I get the following error message:
Traceback (most recent call last):
File "tmpStack.py", line 7, in <module>
xFinal = fsolve(tmpFunc, x0 )
File "/usr/lib/python2.7/dist-packages/scipy/optimize/minpack.py", line 115, in fsolve
_check_func('fsolve', 'func', func, x0, args, n, (n,))
File "/usr/lib/python2.7/dist-packages/scipy/optimize/minpack.py", line 26, in _check_func
raise TypeError(msg)
TypeError: fsolve: there is a mismatch between the input and output shape of the 'func' argument '<lambda>'.
Well it looks like I was trying to use this routine incorrectly. This routine requires the same number of equations and variables vs. the one equation with three variables I gave it. So if the input to the function to be minimized is a 3-D array the output should be a 3-D array. This code works:
import numpy as np
from scipy.optimize import fsolve
tmpFunc = lambda xIn: np.array( [(xIn[0]-4)**2 + xIn[1], (xIn[1]-5)**2 - xIn[2]) \
, (xIn[2]-7)**3 + xIn[0] ] )
x0 = [3,4,5]
xFinal = fsolve(tmpFunc, x0 )
print xFinal
Which represents solving three equations simultaneously.