We Keep Coding

Numpy.dot dot product function for statsmodels

I am learning statsmodels.api module to use python for regression analysis. So I started from the simple OLS model.
In econometrics, the function is like: y = Xb + e
where X is NxK dimension, b is Kx1, e is Nx1, so adding together y is Nx1. This is perfectly fine from linear algebra point of view.
But I followed the tutorial from Statsmodels as the following:
import numpy as np
nsample = 100 # total obs is 100
x = np.linspace(0, 10, 100) # using np.linspace(start, stop, number)
X = np.column_stack((x, x**2))
beta = np.array([1, 0.1, 10])
e = np.random.normal(size = nsample) # draw numbers from normal distribution
default at mu = 0, and std.dev = 1, size = set by user
# e is n x 1
# Now, we add the constant/intercept term to X
X = sm.add_constant(X)
# Now, we compute the y
y = np.dot(X, beta) + e
So this generates the correct answer. But I have a question about the generation of beta = np.array([1,0.1,10]). This beta, if we use:
beta.shape
(3,)
It has a dimension of (3,), the same goes with y and e except X:
X.shape
(100,3)
e.shape
(100,)
y.shape
(100,)
So I guess initiating array using the following three ways
o = array([1,2,3])
o1 = array([[1],[2],[3]])
o2 = array([[1,2,3]])
print(o.shape)
print(o1.shape)
print(o2.shape)
----------------
(3,)
(3, 1)
(1, 3)
If I use beta = array([[1],[2],[3]]), which is a (3,1), and np.dot(X, beta) gets me a wrong answer, although the dimension seems to work.
If I use array([[1,2,3]]), which is a row vector, the dimension doesn't match for dot product in numpy, neither in linear algebra.
So, I am wondering why for a NxK dot Kx1 numpy dot product, we have to use a (N,K) dot (K,) instead of (N,K) dot (K,1) matrices. What operation makes only np.array([1, 0.1, 10]) works for numpy.dot() while np.array([[1], [0.1], [10]]) doesn't.
Thank you very much.
Some update
Sorry about the confusion, the codes in Statsmodels are randomly generated so I tried to fix the X and get the following input:
f = array([[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15]])
o = array([1,2,3])
o1 = array([[1],[2],[3]])
o2 = array([[1,2,3]])
print(o.shape)
print(o1.shape)
print(o2.shape)
print("---------")
print(np.dot(f,o))
print(np.dot(f,o1))
r1 = np.dot(f,o)
r2 = np.dot(f,o1)
type1 = type(np.dot(f,o))
type2 = type(np.dot(f,o1))
tf = type1 is type2
tf2 = type1 == type2
print(type1)
print(type2)
print(tf)
print(tf2)
-------------------------
(3,)
(3, 1)
(1, 3)
---------
[14 32 50 68 86]
[[14]
[32]
[50]
[68]
[86]]
<class 'numpy.ndarray'>
<class 'numpy.ndarray'>
True
True
Sorry again for the confusion and inconvenience, they worked fine.

python/numpy is not a matrix-based language as it is Matlab or Octave or Scilab. These follow the rules of matrix multplication strictly. So
np.dot(f,o) ---------> f*o in Matlab/Octave/Scilab
np.dot(f,o1) ---------> f*o1 does not work in Matlab/Octave/Scilab
python/numpy has the 'broadcasting' which are the rules how the different data types and operations give together a result. It's not obvious why np.dot(f,o1) even should work, but the broadcasting defines some usefull results. You will have to consult the docs for that.
In python/numpy the * is not a matrix operator. You can find out what the broadcasting gives for
print(f*o)
print(f*o1)
print(f*o2)
Rather recently python/numpy has introduced the matrix operator #. You might find out what happens with
print(f#o)
print(f#o1)
print(f#o2)
Does this give some impressions ?

Difference in matrix multiplication tensorflow vs numpy

I have a case where matrix multiplication of two matrices with certain dimensions work in numpy, but doesn't work in tensorflow.
x = np.ndarray(shape=(10,20,30), dtype = float)
y = np.ndarray(shape=(30,40), dtype = float)
z = np.matmul(x,y)
print("np shapes: %s x %s = %s" % (np.shape(x), np.shape(y), np.shape(z)))
This works as expected and prints:
np shapes: (10, 20, 30) x (30, 40) = (10, 20, 40)
However in tensorflow when I try to multiply placeholder and variable of the same shapes as the numpy arrays above I get an error
x = tf.placeholder(tf.float32, shape=(10,20,30))
y = tf.Variable(tf.truncated_normal([30,40], name='w'))
print("tf shapes: %s x %s" % (x.get_shape(), y.get_shape()))
tf.matmul(x,y)
Results in
tf shapes: (10, 20, 30) x (30, 40)
InvalidArgumentError:
Shape must be rank 2 but is rank 3 for 'MatMul_12'
(op: 'MatMul') with input shapes: [10,20,30], [30,40].
Why does this operation fail?

Don't know why tf.matmul does not support this kind of multiplication (may be one of the core developers could provide a meaningful answer).
But if you just want to be able to multiply tensors in this way, take a look at tf.einsum function. It could operate with tensors of arbitrary rank.

As suggested by Dmytro tf.einsum can be used to multiply these two arrays.
x = np.ndarray(shape=(10,20,30), dtype = float)
y = np.ndarray(shape=(30,40), dtype = float)
These two operations produce exactly the same result:
np.einsum('ijk,kl->ijl', x, y)
np.matmul(x,y)
And corresponding tensorflow operation also works
tf.einsum('ijk,kl->ijl', tf_x,tf_y)

People already told you that you can use tf.einsum() to get the result you want.
import tensorflow as tf
x = tf.random_normal([10, 20, 30])
y = tf.random_normal([30, 40])
z = tf.einsum('ijk,kl->ijl', x, y)
The reason why tf.matmul() does not work the way you expected is written in the documentation.
The inputs must be matrices (or tensors of rank > 2, representing
batches of matrices), with matching inner dimensions, possibly after
transposition.
In your case you have a matrix y and a tensor x (rank 3 > 2). In your case inner dimensions do not match. If you want, them to match, you will need to have something like this:
import tensorflow as tf
a, b, c = 12, 50, 20
x = tf.random_normal([a, b, c])
y = tf.random_normal([a, c, b])
z = tf.matmul(x, y)
But clearly it calculates not the stuff you want.

Python SciPy linprog optimization fails with status 3

Trying to minimize a simple linear function with linprog. The coefficients are the elements of arr2 multiplied by -1. There are only inequality constraints for each variable, such as -1 <= x1 <= 1, -2 <= x2 <= 2 and so on.
If a choose not to specify bounds in linprog:
from scipy.optimize import linprog
import numpy as np
import pandas as pd
numdim = 28
arr1 = np.ones(numdim)
arr1 = - arr1
arr2 = np.array([
19.53,
128.97,
3538,
931.8,
0.1825,
150.88,
10315,
0.8109,
3.9475,
3022,
31.77,
10323,
110.93,
220,
2219.5,
119.2,
703.6,
616,
338,
84.67,
151.13,
111.28,
29.515,
29.67,
158800,
167.15,
0.06802,
1179
])
constr_a = []
for i in range(numdim):
constr_default = np.zeros(numdim)
constr_default[i] = 1
constr_a.append(constr_default)
for i in range(numdim):
constr_default = np.zeros(numdim)
constr_default[i] = -1
constr_a.append(constr_default)
constr_a = np.asarray(constr_a)
constr_b = np.arange(1, 2*numdim + 1, 1)
constr_b[numdim:] = constr_b[:numdim]
print linprog(np.transpose(arr1 * arr2), constr_a, constr_b, bounds=(None, None))
I get the following result:
fun: -4327476.2887400016
message: 'Optimization failed. The problem appears to be unbounded.'
status: 3
I've tried changing the last row to:
print linprog(np.transpose(arr1 * arr2), constr_a, constr_b, bounds=(-1000, 1000))
The numbers specified as bounds are random. The output is:
fun: -4327476.2887400296
message: 'Optimization terminated successfully.'
status: 0
which gives us a slightly different result and the desired status.
My question is, do I misuse the library and in which way? Which answer is correct? This code was expected to work without specifying the 'bounds' parameter. I cannot use this parameter because these simple constraints are unique for each variable.
I use python 2.7 and scipy 0.17.1. Big thanks in advance.
Upd
constr_a should be a matrix according to the documentation (https://docs.scipy.org/doc/scipy/reference/optimize.linprog-simplex.html) and actually is in the code. To be sure the syntax is correct, we can cut the number of dimensions to 2:
from scipy.optimize import linprog
import numpy as np
import pandas as pd
numdim = 2
arr1 = np.ones(numdim)
arr1 = - arr1
arr2 = np.array([
19.53,
128.97
])
constr_a = []
for i in range(numdim):
constr_default = np.zeros(numdim)
constr_default[i] = 1
constr_a.append(constr_default)
for i in range(numdim):
constr_default = np.zeros(numdim)
constr_default[i] = -1
constr_a.append(constr_default)
constr_a = np.asarray(constr_a)
constr_b = np.arange(1, 2*numdim + 1, 1)
constr_b[numdim:] = constr_b[:numdim]
print constr_a
print constr_b
print linprog(np.transpose(arr1 * arr2), constr_a, constr_b, bounds=(None, None))
and this will work.

the constr_a list is not properly formed. It is an array of array's instead of being an array of scalar. This might be leading to a improper lower bound causing the optimization to fail.
Perhaps
constr_a.append(constr_default)
should be
constr_a.append(constr_default[i])
inspect both the bound arrays to make sure they have proper form and values.

Python:ValueError: shapes (3,) and (118,1) not aligned: 3 (dim 0) != 118 (dim 0)

I am trying to do logistic regression using fmin but there is an error showing up due to different shapes of array. Here is the code.
import numpy as np
import scipy.optimize as sp
data= #an array of dim (188,3)
X=data[:,0:2]
y=data[:,2]
m,n=np.shape(X)
y=y.reshape(m,1)
x=np.c_[np.ones((m,1)),X]
theta=np.zeros((n+1,1))
def hypo(x,theta):
return np.dot(x,theta)
def sigmoid(z):
return 1/(1+np.exp(-z))
def gradient(x,y,theta):#calculating Gradient
m=np.shape(x)[0]
t=hypo(x,theta)
hx=sigmoid(t)
J=-(np.dot(np.transpose(np.log(hx)),y)+np.dot(np.transpose(np.log(1-hx)),(1-y)))/m
grad=np.dot(np.transpose(x),(hx-y))/m
J= J.flatten()
grad=grad.flatten()
return J,grad
def costFunc(x,y,theta):
return gradient(x,y,theta)[0]
def Grad():
return gradient(x,y,theta)[1]
sp.fmin( costFunc, x0=theta, args=(x, y), maxiter=500, full_output=True)
error that is showing
File "<ipython-input-3-31a0d7ca38c8>", line 35, in costFunc
return gradient(x,y,theta)[0]
File "<ipython-input-3-31a0d7ca38c8>", line 25, in gradient
t=hypo(x,theta)
File "<ipython-input-3-31a0d7ca38c8>", line 16, in hypo
return np.dot(x,theta)
ValueError: shapes (3,) and (118,1) not aligned: 3 (dim 0) != 118 (dim 0)
Any kind of help will be appreciated

data= #an array of dim (188,3)
X=data[:,0:2]
y=data[:,2]
m,n=np.shape(X)
y=y.reshape(m,1)
x=np.c_[np.ones((m,1)),X]
theta=np.zeros((n+1,1))
so after this
In [14]: y.shape
Out[14]: (188, 1) # is this (118,1)?
In [15]: x.shape
Out[15]: (188, 3)
In [16]: theta.shape
Out[16]: (3, 1)
This x and theta can dotted - np.dot(x,theta), and (188,3) with (3,1) - matching the 3's.
But that's not what your costFunc is getting. Tracing back from the error message it looks like x is (3,), and theta is (118,1). which obviously cannot be dotted.
You need to review how fmin calls your function. Do you have the parameters in the right order? For example, maybe costFunc(theta, x, y) is the correct order (assuming the x and y in costFunc are meant to match with the args=(x,y).
The docs for fmin include:
func : callable func(x,*args)
The objective function to be minimized.
x0 : ndarray
Initial guess.
args : tuple, optional
Extra arguments passed to func, i.e. ``f(x,*args)``.
It looks like fmin is feeding your costFunc 3 arguments, corresponding in size to your (theta, x, y), i.e. (3,), (118,3), (118,1). The numbers don't quite match, but I think you get the idea. The first argument to consFunc is the one that the fmin will vary, the rest you provide in args.

fsolve - mismatch between input and output

I'm trying to solve an overdetmined system of equations with three unknowns. I'm able to get solution with fsolve and lsqnonlin in MATLAB by calling the system of equations through a for loop.
But in python using scipy, I'm getting the following error message:
fsolve: there is a mismatch between the input and output shape of the 'func' argument 'fnz'
The code is given below:
from xlrd import open_workbook
import numpy as np
from scipy import optimize
g = [0.5,1,1.5]
wb = open_workbook('EThetaValuesA.xlsx')
sheet=wb.sheet_by_index(0)
y=sheet.col_values(0,1)
t1=sheet.col_values(1,1)
t2=sheet.col_values(2,1)
t3=sheet.col_values(3,1)
def fnz(g):
i=0
sol=[0 for i in range(len(t1))]
x1 = g[0]
x2 = g[1]
x3 = g[2]
print len(t1)
for i in range(len(t1)):
# various set of t1,t2 and t3 gives the various eqns
print i
sol[i]=x1+t1[i]/(x2*t2[i]+x3*t3[i])-y[i]
return sol
Anz = optimize.fsolve(fnz,g)
print Anz
Could anyone please suggest where I'm wrong? Thank you in advance.

The exception means that the result from fnz() function call does not has the same dimension as the input g, which is a list of 3 elements, or can be seen as an array of shape (3,).
To illustrate the problem, if we define:
def fnz(g):
return [2,3,5]
Anz = optimize.fsolve(fnz,g)
There will not be such an exception. But this will:
def fnz(g):
return [2,3,4,5]
Anz = optimize.fsolve(fnz,g)
The result from fnz() should have the same length as t1, which I am sure is longer than 3 elements.