I need to solve an equation in the form Ax=By for x. I know I shouldn't solve it by inverting B but I couldn't solve B^-1Ax=y with scipy.gmres or linalg.solve since it fails when I try to invert B with linalg.inv. It returns the error message "Singular matrix".
Is there any other way to invert a matrix? Efficiency is not important since I need to do it just once. I dont want to solve the equation twice like first for T=Ax and then x.
I'd join the suggestion of user hilberts-drinking-problem and not invert B but follow a two step approach: First multiply B and y to yield a vector By and then solve the system A·x=By. The following illustrates this approach using small arrays as test data:
from numpy import array;
from scipy.sparse import coo_matrix
from scipy.sparse.linalg import gmres
A = coo_matrix((3, 3), dtype=float)
A.setdiag( [ 2, 4, -1 ] )
A.setdiag( [ 2, -0.5 ], 1 )
print( "A", A )
B = coo_matrix((3, 2), dtype=float)
B.setdiag( [ 1, -2 ] )
B.setdiag( [ -0.5, 4 ], -1 )
print( "B", B )
y = array( [ 13, 5 ] )
print( "y", y )
By = B * y
print( "By", By )
x = gmres( A, By )
print( "x", x )
Related
I have two arrays named x and y. The goal is to iterate them as the input for pandas calculation.
Here's an example.
Iterating each x and y and appending the calculation result to the res list is slow.
The calculation is to get the exponential of each column modified by a and then sum together, multiply with b. Anyway, this calculation can be replaced by any other calculations.
import numpy as np
import pandas as pd
np.random.seed(0)
df = pd.DataFrame(np.random.randint(0,5,size=(5, 1)),columns=['data'])
x = np.linspace(1, 24, 4)
y = np.linspace(10, 1500, 5)
res = []
for a in x:
for b in y:
res.append(np.exp(-df/a).sum().values[0]*b)
res = np.array(res).reshape(4, 5)
expected output:
array([[ 11.67676844, 446.63639283, 881.59601721, 1316.5556416 ,
1751.51526599],
[ 37.52524129, 1435.34047927, 2833.15571725, 4230.97095523,
5628.78619321],
[ 42.79406912, 1636.87314392, 3230.95221871, 4825.0312935 ,
6419.1103683 ],
[ 44.93972433, 1718.94445549, 3392.94918665, 5066.95391781,
6740.95864897]])
You can use numpy broadcasting:
res = np.array(res).reshape(4, 5)
print (res)
[[ 11.67676844 446.63639283 881.59601721 1316.5556416 1751.51526599]
[ 37.52524129 1435.34047927 2833.15571725 4230.97095523 5628.78619321]
[ 42.79406912 1636.87314392 3230.95221871 4825.0312935 6419.1103683 ]
[ 44.93972433 1718.94445549 3392.94918665 5066.95391781 6740.95864897]]
res = np.exp(-df.to_numpy()/x).sum(axis=0)[:, None] * y
print (res)
[[ 11.67676844 446.63639283 881.59601721 1316.5556416 1751.51526599]
[ 37.52524129 1435.34047927 2833.15571725 4230.97095523 5628.78619321]
[ 42.79406912 1636.87314392 3230.95221871 4825.0312935 6419.1103683 ]
[ 44.93972433 1718.94445549 3392.94918665 5066.95391781 6740.95864897]]
I think what you want is:
z = -df['data'].to_numpy()
res = np.exp(z/x[:, None]).sum(axis=1)[:, None]*y
output:
array([[ 11.67676844, 446.63639283, 881.59601721, 1316.5556416 ,
1751.51526599],
[ 37.52524129, 1435.34047927, 2833.15571725, 4230.97095523,
5628.78619321],
[ 42.79406912, 1636.87314392, 3230.95221871, 4825.0312935 ,
6419.1103683 ],
[ 44.93972433, 1718.94445549, 3392.94918665, 5066.95391781,
6740.95864897]])
Say I have a matrix Y of random float numbers from 0 to 10 with shape (10, 3):
import numpy as np
np.random.seed(99)
Y = np.random.uniform(0, 10, (10, 3))
print(Y)
Output:
[[6.72278559 4.88078399 8.25495174]
[0.31446388 8.08049963 5.6561742 ]
[2.97622499 0.46695721 9.90627399]
[0.06825733 7.69793028 7.46767101]
[3.77438936 4.94147452 9.28948392]
[3.95454044 9.73956297 5.24414715]
[0.93613093 8.13308413 2.11686786]
[5.54345785 2.92269116 8.1614236 ]
[8.28042566 2.21577372 6.44834702]
[0.95181622 4.11663239 0.96865261]]
I am now given a matrix X with same shape that can be seen as obtained by adding small noises to Y and then shuffling the rows:
X = np.random.normal(Y, scale=0.1)
np.random.shuffle(X)
print(X)
Output:
[[ 4.04067271 9.90959141 5.19126867]
[ 5.59873104 2.84109306 8.11175891]
[ 0.10743952 7.74620162 7.51100441]
[ 3.60396019 4.91708372 9.07551354]
[ 0.9400948 4.15448712 1.04187208]
[ 2.91884302 0.47222752 10.12700505]
[ 0.30995155 8.09263241 5.74876947]
[ 1.11247872 8.02092335 1.99767444]
[ 6.68543696 4.8345869 8.17330513]
[ 8.38904822 2.11830619 6.42013343]]
Now I want to sort the matrix X based on Y by row. I already know each pair of column values in each matching pair of rows are not different from each other more than a tolerance of 0.5. I managed to write the following code and it is working fine.
def sort_X_by_Y(X, Y, tol):
idxs = [next(i for i in range(len(X)) if all(abs(X[i] - row) <= tol)) for row in Y]
return X[idxs]
print(sort_X_by_Y(X, Y, tol=0.5))
Output:
[[ 6.68543696 4.8345869 8.17330513]
[ 0.30995155 8.09263241 5.74876947]
[ 2.91884302 0.47222752 10.12700505]
[ 0.10743952 7.74620162 7.51100441]
[ 3.60396019 4.91708372 9.07551354]
[ 4.04067271 9.90959141 5.19126867]
[ 1.11247872 8.02092335 1.99767444]
[ 5.59873104 2.84109306 8.11175891]
[ 8.38904822 2.11830619 6.42013343]
[ 0.9400948 4.15448712 1.04187208]]
However, in reality I am sorting (1000, 3) matrices and my code is way too slow. I feel like there should be more numpyic way to code this. Any suggestions?
This is a vectorized version of your algorithm. It runs ~26.5x faster than your implementation for 1000 samples. But an additional boolean array with shape (1000,1000,3) is created. There is a chance that rows will have similar values within the tolerance and a wrong row is selected.
tol = .5
X[(np.abs(Y[:, np.newaxis] - X) <= tol).all(2).argmax(1)]
Output
array([[ 6.68543696, 4.8345869 , 8.17330513],
[ 0.30995155, 8.09263241, 5.74876947],
[ 2.91884302, 0.47222752, 10.12700505],
[ 0.10743952, 7.74620162, 7.51100441],
[ 3.60396019, 4.91708372, 9.07551354],
[ 4.04067271, 9.90959141, 5.19126867],
[ 1.11247872, 8.02092335, 1.99767444],
[ 5.59873104, 2.84109306, 8.11175891],
[ 8.38904822, 2.11830619, 6.42013343],
[ 0.9400948 , 4.15448712, 1.04187208]])
More robust solutions with L1-norm
X[np.abs(Y[:, np.newaxis] - X).sum(2).argmin(1)]
Or L2-norm
X[((Y[:, np.newaxis] - X)**2).sum(2).argmin(1)]
I would like to interpolate between two lists in which 1st one contains numbers and second one contains arrays.
I tried using interp1d from scipy, but it did not work
from scipy import interpolate
r = [2,3,4]
t = [5,6,7]
f = [r,t]
q = [10,20]
c = interpolate.interp1d(q, f)
I would like to get an array, for example at value 15, which should be interpolated values between r and t arrays
Error message:
ValueError: x and y arrays must be equal in length along interpolation axis.
In the simple example of the OP it does not make a difference whether one takes 1D or 2D interpolation. If more vectors come into play, however, it makes a difference. Here both options, using numpy and taking care of floating point.
from scipy.interpolate import interp1d
from scipy.interpolate import interp2d
import numpy as np
r = np.array( [ 1, 1, 2], np.float )
s = np.array( [ 2, 3, 4], np.float )
t = np.array( [ 5, 6, 12], np.float ) # length of r,s,t,etc must be equal
f = np.array( [ r, s, t ] )
q = np.array( [ 0, 10, 20 ], np.float ) # length of q is length of f
def interpolate_my_array1D( x, xData, myArray ):
out = myArray[0].copy()
n = len( out )
for i in range(n):
vec = myArray[ : , i ]
func = interp1d( xData, vec )
out[ i ] = func( x )
return out
def interpolate_my_array2D( x, xData, myArray ):
out = myArray[0].copy()
n = len( out )
xDataLoc = np.concatenate( [ [xx] * n for xx in xData ] )
yDataLoc = np.array( range( n ) * len( xData ), np.float )
zDataLoc = np.concatenate( myArray )
func = interp2d( xDataLoc, yDataLoc, zDataLoc )
out = np.fromiter( ( func( x, yy ) for yy in range(n) ), np.float )
return out
print interpolate_my_array1D( 15., q, f )
print interpolate_my_array2D( 15., q, f )
giving
>> [3.5 4.5 5.5]
>> [2.85135135 4.17567568 6.05405405]
Following is the link to the interp1d function in scipy documentation interpolate SciPy.
From the docs you can see that the function does not take a list of list as an input. the inputs need to be either numpy arrays or list of primitive values.
I have a distributional analysis algorithm for words. It generates observational vectors for each target word and from this table, I use stats.spearmanr() to calculate the distances (rescaled from [-1,1] to [0,1]), generating a distance matrix (Y). Then I use hierarchy.average() to obtain the clustering (Z). Finally, a dendrogram is generated and plotted.
The problem I have is this: the dendrogram scale varies with the number of target words. I was assuming that its distance axis varied along the [0,1] range (as obtained (and rescaled) with spearmanr()), as presented above. But it is [0, 0.5] for, say, 50 words, but [0, 1] for 150, and [0, 2] for 1000.
Why is that so (that the distance scale have values bigger than the ones in Y)?
I'd appreciate any ideas on this issue, because I can't seem to find any hint in the documentation and over the web (which makes me worried about making the wrong questions...). And I'd need a fixed scale or at least a way to know which one the dendrogram is using, for cut level specification purposes. Thanks in advance for any help.
The simplified code:
# coding: utf-8
# Estatísticas e visualização
import numpy as np
import scipy, random
import scipy.stats
# Clusterização e visualização do dendrograma
import scipy.cluster.hierarchy as hac
import matplotlib.pyplot as plt
def remap(x, in_min, in_max, out_min, out_max):
return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min
random.seed('7622')
sizes = [50, 250, 500, 1000, 2000]
for n in sizes:
# Generate observation matrix
X = []
for i in range(n):
vet = []
for j in range(300):
# Generate random observations
vet.append(random.randint(0, 50))
X.append(vet)
# X is a matrix where lines are variables (target words) and columns are observations (contexts of occurrence)
Y = scipy.stats.spearmanr(X, axis=1)
# Y rescaling
for i in range(len(Y[0])):
Y[0][i] = [ remap(v, -1, 1, 0, 1) for v in Y[0][i] ]
print 'Y [', np.matrix(Y[0]).min(), ',', np.matrix(Y[0]).max(), ']'
# Clustering
Z = hac.average(Y[0])
print 'n=', n, \
'Z [', min([ el[2] for el in Z ]), ',', max([ el[2] for el in Z ]), ']'
[UPDATE] Results of the above code:
Y [ 0.401120498124 , 1.0 ]
n= 50 Z [ 0.634408300876 , 0.77633631869 ]
Y [ 0.379375733574 , 1.0 ]
n= 250 Z [ 0.775241869849 , 0.969704246048 ]
Y [ 0.37559031365 , 1.0 ]
n= 500 Z [ 0.935671154717 , 1.16505319575 ]
Y [ 0.370600337649 , 1.0 ]
n= 1000 Z [ 1.19646327361 , 1.47897594053 ]
Y [ 0.359010408057 , 1.0 ]
n= 2000 Z [ 1.56890165007 , 1.96898566034 ]
I get TypeError: Failed to convert object. Is there some way to do tf.not_equal() or equivalent on a sparse tensor? It must stay sparse; conversion to dense not permitted.
Supposing you want to compare two sparse tensors, and you have numbers in them, I think it's easiest to subtract one from the other and keep resulting non-zero values as "True" with tf.sparse_retain(). DomJack's answer only works if you want to compare a sparse tensor to a constant, but that's much easier with tf.sparse_retain() like the function sparse_not_equal_to_constant() below. (Please note this is not an accurate not_equal operation because it only tests existing values for inequality. Since the non-listed elements of a sparse tensor are zero, if the constant we're comparing to is not itself zero, then the rest of the matrix should also be marked as not equal. That's best done when converting back to dense, with the default_value parameter, but considering where the matrix had values to start with.) Tested code for comparing two sparse tensors, including function to compare it to constant:
import tensorflow as tf
import numpy as np
def sparse_not_equal_to_constant( s, c ):
a = tf.sparse_retain( s, tf.not_equal( c, s.values ) )
return tf.SparseTensor( a.indices, tf.ones_like( a.values, dtype = tf.bool ), dense_shape = s.dense_shape )
def sparse_not_equal( a, b ):
neg_b = tf.SparseTensor( b.indices, -b.values, dense_shape = b.dense_shape )
difference = tf.sparse_add( a, neg_b )
return sparse_not_equal_to_constant( difference, 0.0 )
# test data
a = tf.SparseTensor( [ [ 0, 0 ], [ 1, 4 ], [ 2, 3 ] ], [ 5.0, 6, 7 ], dense_shape = ( 5, 5 ) )
b = tf.SparseTensor( [ [ 0, 0 ], [ 0, 2 ], [ 2, 3 ] ], [ 5.0, 6, 2 ], dense_shape = ( 5, 5 ) )
e = sparse_not_equal( a, b )
f = tf.sparse_tensor_to_dense( e, default_value = False )
with tf.Session() as sess:
print( sess.run( f ) )
Outputs:
[[False False True False False]
[False False False False True]
[False False False True False]
[False False False False False]
[False False False False False]]
as expected.
I think you'll have to operate on the indices/values independently.
import numpy as np
import tensorflow as tf
def sparse_not_equal(sparse_tensor, value):
indices = sparse_tensor.indices
values = sparse_tensor.values
condition = tf.squeeze(tf.where(tf.not_equal(values, value)), axis=-1)
indices = tf.gather(indices, condition)
values = tf.ones(shape=(tf.shape(indices)[0],), dtype=tf.bool)
return tf.SparseTensor(
indices,
values,
sparse_tensor.dense_shape)
def get_sparse():
vals = tf.constant([2, 3, 4, 2])
indices = tf.constant(np.array([[1], [4], [5], [10]]))
dense_shape = [16]
return tf.SparseTensor(indices, vals, dense_shape)
sparse_tensor = get_sparse()
sparse_filtered = sparse_not_equal(sparse_tensor, 2)
with tf.Session() as sess:
s = sess.run(sparse_filtered)
print(s)