I'm having a problem where I'm getting different random numbers across different computers despite
scipy.__version__ == '1.2.1' on all computers
numpy.__version__ == '1.15.4' on all computers
random_state seed is fixed to the same number (42) in every function call that generates random numbers for reproducible results
The code is a bit to complex to post in full here, but I noticed results start to diverge specifically when sampling from a multivariate normal:
import numpy as np
from scipy import stats
seed = 42
n_sim = 1000000
d = corr_mat.shape[0] # corr_mat is a 15x15 correlation matrix, numpy.ndarray
# results diverge from here across different hardware
z = stats.multivariate_normal(mean=np.zeros(d), cov=corr_mat).rvs(n_sim, random_state=seed)
corr_mat is a correlation matrix (see Appendix below) and is the same across all computers.
The two different computers we are testing on are
Computer 1
OS: Windows 7
Processor: Intel(R) Xeon(R) CPU E5-2623 v4 # 2.60Ghz 2.60 Ghz (2 processors)
RAM: 64 GB
System type: 64-bit
Computer 2
OS: Windows 7
Processor: Intel(R) Xeon(R) CPU E5-2660 v3 # 2.10Ghz 2.10 Ghz (2 processors)
RAM: 64 GB
System type: 64-bit
Appendix
corr_mat
>>> array([[1. , 0.15, 0.25, 0.25, 0.25, 0.25, 0.1 , 0.1 , 0.1 , 0.25, 0.25,
0.25, 0.1 , 0.1 , 0.1 ],
[0.15, 1. , 0. , 0. , 0. , 0. , 0.15, 0.05, 0.15, 0.15, 0.15,
0. , 0.15, 0.15, 0.15],
[0.25, 0. , 1. , 0.25, 0.25, 0.25, 0.2 , 0. , 0.2 , 0.2 , 0.2 ,
0.25, 0.2 , 0.2 , 0.2 ],
[0.25, 0. , 0.25, 1. , 0.25, 0.25, 0.2 , 0. , 0.2 , 0.2 , 0.2 ,
0.25, 0.2 , 0.2 , 0.2 ],
[0.25, 0. , 0.25, 0.25, 1. , 0.25, 0.2 , 0. , 0.2 , 0.2 , 0.2 ,
0.25, 0.2 , 0.2 , 0.2 ],
[0.25, 0. , 0.25, 0.25, 0.25, 1. , 0.2 , 0. , 0.2 , 0.2 , 0.2 ,
0.25, 0.2 , 0.2 , 0.2 ],
[0.1 , 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 1. , 0.15, 0.25, 0.25, 0.25,
0.2 , 0.25, 0.25, 0.25],
[0.1 , 0.05, 0. , 0. , 0. , 0. , 0.15, 1. , 0.15, 0.15, 0.15,
0. , 0.15, 0.15, 0.15],
[0.1 , 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 0.25, 0.15, 1. , 0.25, 0.25,
0.2 , 0.25, 0.25, 0.25],
[0.25, 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 0.25, 0.15, 0.25, 1. , 0.25,
0.2 , 0.25, 0.25, 0.25],
[0.25, 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 0.25, 0.15, 0.25, 0.25, 1. ,
0.2 , 0.25, 0.25, 0.25],
[0.25, 0. , 0.25, 0.25, 0.25, 0.25, 0.2 , 0. , 0.2 , 0.2 , 0.2 ,
1. , 0.2 , 0.2 , 0.2 ],
[0.1 , 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 0.25, 0.15, 0.25, 0.25, 0.25,
0.2 , 1. , 0.25, 0.25],
[0.1 , 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 0.25, 0.15, 0.25, 0.25, 0.25,
0.2 , 0.25, 1. , 0.25],
[0.1 , 0.15, 0.2 , 0.2 , 0.2 , 0.2 , 0.25, 0.15, 0.25, 0.25, 0.25,
0.2 , 0.25, 0.25, 1. ]])
The following is an educated guess which I cannot validate since I don't have multiple machines.
Sampling from a correlated multinormal is typically done by sampling from an uncorrelated standard normal and then multiplying with a "square root" of the covariance matrix. I get a fairly similar sample to the one scipy produces with seed set at 42 and your covariance matrix if I use instead identity(15) for the covariance and then multiply with l*sqrt(d) where l,d,r = np.linalg.svd(covariance)
SVD is I suppose complex enough to explain small differences between platforms.
How can this snowball into something significant?
I think your choice of covariance matrix is to blame, since it has nonunique eigenvalues. As a consequence SVD is not unique, since eigenspaces to a given multiple eigenvalue can be rotated. This has the potential to hugely amplify a small numerical difference.
It would be interesting to see whether the differences you see persist if you test with a different covariance matrix with unique eigenvalues.
Edit:
For reference, here is what i tried for your smaller (6D) example:
>>> cm6 = np.array([[1,.5,.15,.15,0,0], [.5,1,.15,.15,0,0],[.15,.15,1,.25,0,0],[.15,.15,.25,1,0,0],[0,0,0,0,1,.1],[0,0,0,0,.1,1]])
>>> ls6,ds6,rs6 = np.linalg.svd(cm6)
>>> np.random.seed(42)
>>> cs6 = stats.multivariate_normal(cov=cm6).rvs()
>>> np.random.seed(42)
>>> is6 = stats.multivariate_normal(cov=np.identity(6)).rvs()
>>> LS6 = ls6*np.sqrt(ds6)
>>> np.allclose(cs6, LS6#is6)
True
As you report that the problem persists with unique eigenvalues here is one more possibility. Above I have used svd to compute eigen vectors / values which is ok since cov is symmetric. What happens if we use eigh instead?
>>> de6,le6 = np.linalg.eigh(cm6)
>>> LE6 = le6*np.sqrt(de6)
>>> cs6
array([-0.00364915, -0.23778611, -0.50111166, -0.7878898 , -0.91913994,
1.12421904])
>>> LE6#is6
array([ 0.54338614, 1.04010029, -0.71379193, -0.88313042, -0.60813547,
0.26082989])
These are different. Why? First, eigh orders the eigenspaces the other way round:
>>> ds6
array([1.7 , 1.1 , 1.05, 0.9 , 0.75, 0.5 ])
>>> de6
array([0.5 , 0.75, 0.9 , 1.05, 1.1 , 1.7 ])
Does that fix it? Almost.
>>> LE6[:, ::-1]#is6
array([-0.00364915, -0.23778611, -0.50111166, -0.7878898 , -1.12421904,
0.91913994])
We see that the last two samples are swapped and their signs flipped. Turns out this is due to the sign of one eigen vector being inverted.
So even for unique eigen values we can get large differences because of ambiguities in (1) the order of eigen spaces and (2) the sign of eigen vectors.
Related
I have a 3D NumPy array arr. Here is an example:
>>> arr
array([[[0.05, 0.05, 0.9 ],
[0.4 , 0.5 , 0.1 ],
[0.7 , 0.2 , 0.1 ],
[0.1 , 0.2 , 0.7 ]],
[[0.98, 0.01, 0.01],
[0.2 , 0.3 , 0.95],
[0.33, 0.33, 0.34],
[0.33, 0.33, 0.34]]])
For each layer of the cube (i.e., for each matrix), I want to find the index of the column containing the largest number in the matrix. For example, let's take the first layer:
>>> arr[0]
array([[0.05, 0.05, 0.9 ],
[0.4 , 0.5 , 0.1 ],
[0.7 , 0.2 , 0.1 ],
[0.1 , 0.2 , 0.7 ]])
Here, the largest element is 0.9, and it can be found on the third column (i.e. index 2). In the second layer, instead, the max can be found on the first column (the largest number is 0.98, the column index is 0).
The expected result from the previous example is:
array([2, 0])
Here's what I have done so far:
tmp = arr.max(axis=-1)
argtmp = arr.argmax(axis=-1)
indices = np.take_along_axis(
argtmp,
tmp.argmax(axis=-1).reshape((arr.shape[0], -1)),
1,
).reshape(-1)
The code above works, but I'm wondering if it can be further simplified as it seems too much complicated from my point of view.
Find the maximum in each column before applying argmax:
arr.max(-2).argmax(-1)
Reducing the column to a single maximum value will not change which column has the largest value. Since you don't care about the row index, this saves you a lot of trouble.
I have a 1D numpy array, for example the following:
import numpy as np
arr = np.array([0.33, 0.2, 0.8, 0.9])
Now I would like to change the array so that also one minus the value is included. That means the array should look like:
[[0.77, 0.33],
[0.8, 0.2],
[0.2, 0.8],
[0.1, 0.9]]
How can this be done?
>>> np.vstack((1 - arr, arr)).T
array([[0.67, 0.33],
[0.8 , 0.2 ],
[0.2 , 0.8 ],
[0.1 , 0.9 ]])
Alternatively, you can create an empty array and fill in entries:
>>> np.empty((*arr.shape, 2))
>>> x[..., 0] = 1 - arr
>>> x[..., 1] = arr
>>> x
array([[0.67, 0.33],
[0.8 , 0.2 ],
[0.2 , 0.8 ],
[0.1 , 0.9 ]])
Try column_stack
np.column_stack([1 - arr, arr])
Out[33]:
array([[0.67, 0.33],
[0.8 , 0.2 ],
[0.2 , 0.8 ],
[0.1 , 0.9 ]])
Use:
arr=np.insert(1-arr,np.arange(len(arr)),arr).reshape(-1,2)
arr
Output:
array([[0.33, 0.67],
[0.2 , 0.8 ],
[0.8 , 0.2 ],
[0.9 , 0.1 ]])
Generation of a list of many lists each with different ranges
Isc_act = [0.1, 0.2, 0.3]
I_cel = []
a = []
for i in range(0,len(Isc_act)):
a = np.arange(0, Isc_act[i], 0.1*Isc_act[i])
I_cel[i].append(a)
print(I_cel)
Output is:
IndexError: list index out of range
My code is giving error. But, I want to get I_cel = [[0,0.01,..,0.1],[0,0.02,0.04,...,0.2],[0, 0.03, 0.06,...,0.3]]. Hence, the 'nested list' I_cel has three lists and each list has 10 values.
The simplest fix to your code, probably what you were intending to do:
Isc_act = [0.1, 0.2, 0.3]
I_cel = []
for i in range(0,len(Isc_act)):
a = np.arange(0, Isc_act[i], 0.1*Isc_act[i])
I_cel.append(a)
print(I_cel)
Note that the endpoint will be one step less than you wanted! For example row zero, you have to pick two of the below:
Steps of size 0.01
Start point 0.0 and end point 0.1
10 elements total
You can not have all three.
More numpythonic approach:
>>> Isc_act = [0.1, 0.2, 0.3]
>>> (np.linspace(0, 1, 11).reshape(11,1) # [Isc_act]).T
array([[0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 ],
[0. , 0.02, 0.04, 0.06, 0.08, 0.1 , 0.12, 0.14, 0.16, 0.18, 0.2 ],
[0. , 0.03, 0.06, 0.09, 0.12, 0.15, 0.18, 0.21, 0.24, 0.27, 0.3 ]])
linspace gives better control of the end point when dealing with floats:
In [84]: [np.linspace(0,x,11) for x in [.1,.2,.3]]
Out[84]:
[array([0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 ]),
array([0. , 0.02, 0.04, 0.06, 0.08, 0.1 , 0.12, 0.14, 0.16, 0.18, 0.2 ]),
array([0. , 0.03, 0.06, 0.09, 0.12, 0.15, 0.18, 0.21, 0.24, 0.27, 0.3 ])]
Or we could scale just one array (arange with integers is predictable):
In [86]: np.array([.1,.2,.3])[:,None]*np.arange(0,11)
Out[86]:
array([[0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ],
[0. , 0.2, 0.4, 0.6, 0.8, 1. , 1.2, 1.4, 1.6, 1.8, 2. ],
[0. , 0.3, 0.6, 0.9, 1.2, 1.5, 1.8, 2.1, 2.4, 2.7, 3. ]])
Case 1 (solved): Array A has shape (say) (300,50). Array B is an indices array with the shape (300,5), such that B[i,j] indicate for the row i the index of another row to "concate" next to the row i. The end result is an array C with the shape (300,5,50), such that C[i,j,:] = A[B[i,j],:]. This can be done by calling A[B,:].
Here is small script example for case 1:
import numpy as np
## A is the data array
A = np.arange(20).reshape((5,4))
## B indicate for each row which rows to pull together
B = np.array([[0,2],[1,2],[2,0],[3,4],[4,1]])
A[B,:] #The desired result
Case 2 (unsolved): Same problem, only now A is shaped (100,300,50). If B is the indicies matrix shaped (100,300,5), the end result would be an array C with the shape (100,300,5,50) such that C[i,j,k,:] = A[i,B[i,j,k],:]. A[B,:] doesn't work anymore, because it result with a shape (100,300,5,300,50), due to broadcasting.
How should I approach this with indexing?
One approach would be reshaping to 2D keeping the number of columns intact and then indexing into the first axis with the flattened B indices and finally reshaping back to the desired one.
Thus, the implementation would be -
A.reshape(-1,A.shape[-1])[B.ravel()].reshape(100,300,5,50)
Those reshaping being merely views into the arrays, should be quite efficient.
This solves both cases. Here's a sample run for the case #1 -
1) Inputs :
In [667]: A = np.random.rand(3,4)
...: B = np.random.randint(0,3,(3,5))
...:
2) Original method :
In [668]: A[B,:]
Out[668]:
array([[[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.1 , 0.91, 0.1 , 0.98]],
[[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02]],
[[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.45, 0.16, 0.02, 0.02]]])
3) Proposed method :
In [669]: A.reshape(-1,A.shape[-1])[B.ravel()].reshape(3,5,4)
Out[669]:
array([[[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.1 , 0.91, 0.1 , 0.98]],
[[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.1 , 0.91, 0.1 , 0.98],
[ 0.45, 0.16, 0.02, 0.02]],
[[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.48, 0.6 , 0.96, 0.21],
[ 0.45, 0.16, 0.02, 0.02],
[ 0.45, 0.16, 0.02, 0.02]]])
I'm taking a Computational Neuroscience class on Coursera. So far it's been going great! However, I'm getting a little stuck on one of the quiz problems.
I am not taking this class for a certificate or anything. Solely for fun. I already took the quiz and after awhile, I guessed the answer, so this is not even going to be answering the quiz.
The question is framed as the following:
Suppose that we had a linear recurrent network of 5 input nodes and 5 output nodes. Let us say that our network's weight matrix W is:
W = [0.6 0.1 0.1 0.1 0.1]
[0.1 0.6 0.1 0.1 0.1]
[0.1 0.1 0.6 0.1 0.1]
[0.1 0.1 0.1 0.6 0.1]
[0.1 0.1 0.1 0.1 0.6]
(Essentially, all 0.1, besides 0.6 on the diagonals.)
Suppose that we have a static input vector u:
u = [0.6]
[0.5]
[0.6]
[0.2]
[0.1]
Finally, suppose that we have a recurrent weight matrix M:
M = [-0.25, 0, 0.25, 0.25, 0]
[0, -0.25, 0, 0.25, 0.25]
[0.25, 0, -0.25, 0, 0.25]
[0.25, 0.25, 0, -0.25, 0]
[0, 0.25, 0.25, 0, -0.25]
Which of the following is the steady state output v_ss of the network?
(Hint: See the lecture on recurrent networks, and consider writing some Octave or Matlab code to handle the eigenvectors/values (you may use the "eig" function))'
The notes for the class can be found here. Specifically, the equation for the steady state formula can be found on slides 5 and 6.
I have the following code.
import numpy as np
# Construct W, the network weight matrix
W = np.ones((5,5))
W = W / 10.
np.fill_diagonal(W, 0.6)
# Construct u, the static input vector
u = np.zeros(5)
u[0] = 0.6
u[1] = 0.5
u[2] = 0.6
u[3] = 0.2
u[4] = 0.1
# Connstruct M, the recurrent weight matrix
M = np.zeros((5,5))
np.fill_diagonal(M, -0.25)
for i in range(3):
M[2+i][i] = 0.25
M[i][2+i] = 0.25
for i in range(2):
M[3+i][i] = 0.25
M[i][3+i] = 0.25
# We need to matrix multiply W and u together to get h
# NOTE: cannot use W * u, that's going to do a scalar multiply
# it's element wise otherwise
h = W.dot(u)
print 'This is h'
print h
# Ok then the big deal is:
# h dot e_i
# v_ss = sum_(over all eigens) ------------ e_i
# 1 - lambda_i
eigs = np.linalg.eig(M)
eigenvalues = eigs[0]
eigenvectors = eigs[1]
v_ss = np.zeros(5)
for i in range(5):
v_ss += (np.dot(h,eigenvectors[:, i]))/((1.0-eigenvalues[i])) * eigenvectors[:,i]
print 'This is our steady state v_ss'
print v_ss
The correct answer is:
[0.616, 0.540, 0.609, 0.471, 0.430]
This is what I am getting:
This is our steady state v_ss
[ 0.64362264 0.5606784 0.56007018 0.50057043 0.40172501]
Can anyone spot my bug? Thank you so much! I greatly appreciate it and apologize for the long blog post. Essentially, all you need to look at, is slide 5 and 6 on that top link.
I tryied your solution with my matrices:
W = np.array([[0.6 , 0.1 , 0.1 , 0.1 , 0.1],
[0.1 , 0.6 , 0.1 , 0.1 , 0.1],
[0.1 , 0.1 , 0.6 , 0.1 , 0.1],
[0.1 , 0.1 , 0.1 , 0.6 , 0.1],
[0.1 , 0.1 , 0.1 , 0.1 , 0.6]])
u = np.array([.6, .5, .6, .2, .1])
M = np.array([[-0.75 , 0 , 0.75 , 0.75 , 0],
[0 , -0.75 , 0 , 0.75 , 0.75],
[0.75 , 0 , -0.75 , 0 , 0.75],
[0.75 , 0.75 , 0.0 , -0.75 , 0],
[0 , 0.75 , 0.75 , 0 , -0.75]])
and your code generated the right solution:
This is h
[ 0.5 0.45 0.5 0.3 0.25]
This is our steady state v_ss
[ 1.663354 1.5762684 1.66344153 1.56488258 1.53205348]
Maybe the problem is with the Test on coursera. Have you tryed to contact them on the forum?