I am trying out lmfit and using as an example problem below. In this example, I am simply solving for x in a system Ax = y. Here A is a 3*2 array, y is a 3*1 array. I have declared all of them as arrays.
import numpy as np
from lmfit import minimize, Parameters
A = np.array([1,2,-1,3,-2,5])
A = A.reshape(3,2)
y = np.array([12, 13, 21])
def residual(params, A, y, eps_y=1):
x = params['x'].value
y_hat = np.dot(A, x)
return (y - y_hat)/eps_y
x = np.array([0,0])
params = Parameters()
params.add('x', x)
out = minimize(residual, params, args=(A,y))
print out.value
When running this I get an error: "ValueError: object too deep for desired array".
I have found instances of similar problems researching here and on web. In general, most often reason cited is that A, x and y should be arrays and not matrix. Also in some solutions, x and y are asked to be a kept as a vector with shape (len(v),). Above is already in compliance with these suggestions but I am still getting "ValueError: object too deep for desired array".
I have wasted quite a bit of time trying to solve this problem and am stumped now. Any help on this will be very welcome.
The documentation for Parameter is here:
http://newville.github.io/lmfit-py/parameters.html#Parameter
It specifically states that the value of a parameter must be a numerical value, and not an array of any kind. So instead of doing:
x = np.array([0,0])
params.add('x', x)
do:
params.add('x0', 0)
params.add('x1', 0)
and then change the residuals function to:
def residual(params, A, y, eps_y=1):
x0 = params['x0'].value
x1 = params['x1'].value
y_hat = np.dot(A, [x0, x1])
return (y - y_hat)/eps_y
Related
I'm struggling with using scipy to deal with lasso regression.
I wrote a code that could theoretically compute the problem:
where p = 1.
However it returns such kind of error and I'm wondering where did I go wrong. The dimension of x is (50,3) and y is (50, 1):
ValueError: The user-provided objective function must return a scalar value.
My code goes like this:
def norm_one_regression(w, X, y, lambda_):
a = y - X # w
a = a.T # a
return a + lambda_ * np.linalg.norm(w, 1)
# using scipy
w0 = np.ones((3, 1))
X0 = data['x']
y0 = data['y']
lambda_0 = 1
minimum = sop.fmin(norm_one_regression, w0, args=(X0, y0, lambda_0))
print(minimum)
Thanks for your consideration and help.
I am trying to implement a gradient descent function in Python from scratch which I have implemented and work in GNU Octave. Unfortunately I am stuck. I fiddled with it for a while and checked the NumPy documentation but so far no luck.
I am aware of libraries such as scikit-learn, however my purpose is to learn to code such a function from scratch. Perhaps I am going about it the wrong way.
Below you will find all the code necessary to reproduce the error.
Thanks in advance for your help.
Actual result: test fails with error -> "ValueError: matmul: Input operand 0 does not have enough dimensions (has 0, gufunc core with signature (n?,k),(k,m?)->(n?,m?) requires 1)"
Expected result: and array with values [5.2148, -0.5733]
Function gradientDescent() in Octave:
function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);
for iter = 1:num_iters
theta = theta - (alpha/m)*X'*(X*theta-y);
J_history(iter) = computeCost(X, y, theta);
end
Function gradient_descent() in python:
from numpy import zeros
def compute_cost(X, y, theta):
m = len(y)
ans = (X.T # theta).T - y
J = (ans # ans.T) / (2 * m)
return J[0, 0]
def gradient_descent(X, y, theta, alpha, num_iters):
m = len(y)
J_history = zeros((num_iters, 1), dtype=int)
for iter in range(num_iters):
theta = theta - (alpha / m) # X.T # (X # theta - y)
J_history[iter] = compute_cost(X, y, theta)
return theta
the test file: test_ml_utils.py
import unittest
import numpy as np
from ml.ml_utils import compute_cost, gradient_descent
class TestGradientDescent(unittest.TestCase):
# TODO: implement tests for Gradient Descent function
# [theta J_hist] = gradientDescent([1 5; 1 2; 1 4; 1 5],[1 6 4 2]',[0 0]',0.01,1000);
def test_gradient_descent_00(self):
X = np.array([[1, 5], [1, 2], [1, 4], [1, 5]])
y = np.array([1, 6, 4, 2])
theta = np.zeros(2)
alpha = 0.01
num_iter = 1000
r_theta = np.array([5.2148, -0.5733])
result = gradient_descent(X, y, theta, alpha, num_iter)
self.assertEqual((round(result, 4), r_theta), 'Result is wrong!')
if __name__ == '__main__':
unittest.main()
The __matmul__ operator # in Python binds more tightly than -. That means you're trying to do matrix multiplication with the operands (alpha / m), which is a scalar, and X.T, which is actually a matrix. See operator precedence.
In the Octave code, (alpha - m) * X' is doing scalar multiplication, not matrix, so if you want that same behavior in Python, use * rather than #. This seems to be because Octave overloads the * operator to perform scalar multiplication if one operand is a scalar, but matrix multiplication if both operands are matrices.
Adding to Adam's answer (which is correct in regards to the error you're getting).
However I wanted to add more generally, this code is meaningless to a reader without some sort of hint (whether programmatically or in the form of a comment) what dimensions the different variables take.
E.g., there is a hint in the code that y is likely to be 2-dimensional, and you're using len to get its size. Just as an example of how this might fail silently, consider this:
>>> y = numpy.array([[1,2,3,4,5]])
>>> len( y )
1
whereas presumably you want
>>> numpy.shape( y )
(1, 5)
or
>>> numpy.size( y )
5
I note in your unit test that you're passing a rank 1 vector instead of rank 2, so it turns out y is 1D instead of 2D, but operates with X which is 2D due to broadcasting. Your code therefore works despite the logic implied, but in the absence of making such things explicit, this is a runtime error waiting to happen.
I started out my ML journey, taking Andrew Ng's Machine learning course on Coursera, and tried to implement Multivariate Linear Regression using Python. However, I'm facing a lot of trouble parsing the data and converting it into a proper working Numpy array. There seems to be so many Numpy functions that perform the same type of processing on the data, that it's hard to figure which function does what. The major problem cause of this is that it affects the algorithm that I'm running.
When I implement the code using np.matrix() to convert the data into a Numpy matrix I get a Cost function of 2064911681.6185248. I get the same Cost function when I use np.values instead of np.matrix(). However, every python solution for this problem online, gets a cost function of 2105448288.629247 using np.newaxis for X and Y. Whenever, I try to use np.newaxis I get a Type Error saying Key is Invalid.
My question is, why does parsing the data in different ways give different cost functions, even though the data shape is how I want it to be? I've provided my Code below. Is there a single efficient and correct way to convert the data into Numpy arrays?
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
#Load the data
data = pd.read_csv("ex1data2.txt", header=None, names=["Size", "No. of Bedrooms","Price"])
print(data.head(),"\n")
#Initialize columns and size of dataset
cols = data.shape[1]
m = data.shape[0]
theta = np.matrix(np.zeros(cols))
ones = np.ones((m,1))
#Initializing Parameters
alpha = 0.01
iters = 400
cost_list = np.zeros(iters)
#Setting X and Y
X = np.matrix(data.iloc[:,:cols-1])
Y = np.matrix(data.iloc[:,cols-1:])
#Feature Scaling and Adding Ones to X
X = (X - np.mean(X)) / np.std(X)
X = np.hstack((ones, X))
print(f"X Shape: {X.shape} \nY Shape: {Y.shape} \nTheta Shape: {theta.shape} \n")
#Defining Algorithms
def hypothesis(X, theta):
h = np.dot(X, theta.T)
return h
def cost_function(X, Y, theta, m):
squared_error = np.power((hypothesis(X, theta) - Y) , 2)
J = np.sum(squared_error) / (2*m)
return J
def gradient_descent(X, Y, theta, m, alpha, iters):
for i in range(iters):
error = hypothesis(X, theta) - Y
temp = np.dot(error.T, X)
theta = theta - ((alpha/m) * temp)
cost_list[i] = cost_function(X, Y, theta, m)
return theta, cost_list
#Printing Initial and Final Values
print(f"Inital Theta = {theta}")
print(f"Inital Cost = {cost_function(X, Y, theta, m)}")
new_theta, cost_list = gradient_descent(X, Y, theta, m, alpha, iters)
print(f"Final Theta = {new_theta}")
print(f"Final Cost = {cost_function(X, Y, new_theta, m)}")
#Plotting Cost vs Iterations
plt.plot(cost_list, color="Red", label="Cost")
plt.xlabel("Iterations")
plt.ylabel("Cost")
plt.title("Cost vs Iterations")
plt.legend()
plt.show()
I tried to plot pairs (x,y) which are solutions to the equation y^2 + y = x^3 + y(x^2) using the following code:
import numpy as np
y = np.arange(0,12,.01)
x = np.arange(0,18,.01)
for i in enumerate(y):
for j in enumerate(x):
if (i**2)+i == j**3 + i*(j**2):
plot(i,j)
However, I got the following error:
"TypeError: unsupported operand type(s) for ** or pow(): 'tuple' and 'int'"
Anybody can help me to figure out how can I plot solutions?
I have also another question: Is it possible to store each pair of solution in a variable so we can use specific pairs of solutions in the future?
enumerate yields (index, value) tuples, you can use unpacking to get each one separately in the loop header:
for i_ind, i in enumerate(y):
for j_ind, j in enumerate(x):
I suggest you modify your formulation and plot the square difference between LHS and RHS instead:
x = np.linspace(-1, 1, 100)
y = np.linspace(-1, 1, 100)
X, Y = np.meshgrid(x, y)
def F(p):
x, y = p
v = y**2 + y - x**3 - y * (x**2)
return v**2
Z = F((X, Y))
plt.contourf(X, Y, Z)
plt.colorbar()
idx = np.where(np.abs(Z) < 1e-4)
plt.scatter(X[idx], Y[idx], marker='+', color='w')
You can also use scipy.optimize.fmin(F, [1, 1]) to get the exact solution of F=0, where [1, 1] is your initial guess. In this case, different initial guess will result in different solution.
while the other answer describes the mechanical transform needed to stop that exception being thrown, it's not going to get any points in your plot. Numpy is a numeric library, while you want/need to solve the equation.
numpy.arange(0, 12, 0.01) generates a finite series of numbers starting at 0 and incrementing by 0.01 until it hits 12. you give an equation above which is mostly translated into Python code, and evaluate it on these finite series of numbers and ask it to plot values where both expressions evaluate to the same number. floats in Python are 64-bit floats so this equality is checked to approx 15 decimal digits, which will basically not happen for your numbers.
instead of doing what you're attempting to do you want to use something like Sympy which would actually help you. to get started we do:
from sympy import symbols, solveset, simplify, lambdify
x, y = symbols('x y')
eq1 = y**2 + y
eq2 = x**3 + x*y**2
ss = solveset(eq1 - eq2, y)
which names a pair of symbols, puts them into some equations and solves for when eq1 == eq2 (equivalently when their difference is zero).
next you want to plot these solutions, which can be done in this case:
import numpy as np
import matplotlib.pyplot as plt
xx = np.linspace(-5, 4, 101)
for eq in ss:
fn = lambdify(x, eq)
yy = fn(xx)
# hide complex solutions
yy[~np.isreal(yy)] = np.nan
plt.plot(xx, np.real(yy), label=simplify(eq))
plt.legend()
this starts by pulling in numpy and matplotlib, then sampling points on the x-axis at which we want to evaluate the functions, then loop through solutions turning them into Python functions that we can evaluate to get numeric answers, next we evaluate this on our sampled points, discard complex solutions, and plot the results.
this gives:
the gap around x = -4.5 is due to sampling at this point being sparse. we can check by evaluating our equations at x=-4.5, y=10, and I get ~110 for both equations
I woul like to solve an n-dimensional optimisation problem using iminuit.
So my approach is the following.
I am trying to figure out how to extend this:
def f(x,y,z):
return (x-1.)**2 + (y-2*x)**2 + (z-3.*x)**2 -1.
to a variable "x" that is a numpy.array.
I would like to do something like this:
x = [1,2,3,4,5]
y = [2,4,6,8,10]# y=2x
class StraightLineChi2:
def __init__(self,x,y):
self.x = x
self.y = y
def __call__(self,m,c): #lets try to find slope and intercept
chi2 = sum((y - m*x+c)**2 for x,y in zip(self.x,self.y))
return chi2
but in my case x is my unknown, and it is an array. Like in many optimization/minimization problems, the function is a f=f(x1,...,xn) where n can be big. x1,...,xn are the unknowns of the problem.
(These examples are taken from here)
Something similar is achieved "hacking" pyminuit2, like described here
For your example I recommend you using iminuit and probfit. Having an argument as a list of parameter is not exactly what you want to do since you will get confused which parameter is what very soon.
Here is an example taken straight from probfit tutorial. Also see the documentation
import iminuit
import probfit
x = np.linspace(0, 10, 20)
y = 3 * x + 15 + np.random.randn(len(x))
err = np.ones(len(x))
def line(x, m, c): # define it to be parabolic or whatever you like
return m * x + c
chi2 = probfit.Chi2Regression(line, x, y, err)
minuit = iminuit.Minuit(chi2)
minuit.migrad();
print(minuit.values) #{'c': 16.137947520534624, 'm': 2.8862774144823855}