Consider the following code snippet:
import theano.tensor as T
import theano.tensor
import numpy as np
batch_shape = (50, 40, 30, 30)
batch_size = batch_shape[0]
ncols = batch_shape[1]*batch_shape[2]*batch_shape[3]
minibatch = theano.tensor.tensor4(name='minibatch',
dtype=theano.config.floatX)
xflat = minibatch.reshape((batch_size,ncols))
partition = np.array([1, 2, 3])
xsub1 = xflat[:,partition]
partition = np.array([1])
xsub2 = xflat[:,partition]
print "xsub1.type: ", xsub1.type
print "xsub2.type: ", xsub2.type
If you run it, you get the following output:
xsub1.type: TensorType(float64, matrix)
xsub2.type: TensorType(float64, col)
Apparently indexing with an array of length 1 turns xsub2 into a col instead of a matrix. How can i make xsub2 be a matrix?
A col or "column vector" is the name Theano uses for a symbolic matrix that it knows contains only one column. It should be possible to use it just like a matrix.
Theano often doesn't know the shape of a particular symbolic tensor, only its dimensionality. However, in some circumstances, such as that given in the question, Theano is able to infer that a tensor has a particular special case of shape and can sometimes use this information to optimize the computation. This is why col (and row) exist as special cases of matrix.
If you think about the shape more than the type then you'll see that Theano is behaving just the same as numpy:
import theano
import theano.tensor
import numpy as np
def compute(minibatch):
xflat = minibatch.reshape((minibatch.shape[0], -1))
partition = np.array([1, 2, 3])
xsub1 = xflat[:, partition]
partition = np.array([1])
xsub2 = xflat[:, partition]
return xsub1, xsub2
def compile_theano_version():
minibatch = theano.tensor.tensor4(name='minibatch', dtype=theano.config.floatX)
xsub1, xsub2 = compute(minibatch)
print xsub1.type, xsub2.type
return theano.function([minibatch], [xsub1, xsub2])
def numpy_version(minibatch):
return compute(minibatch)
def main():
batch_shape = (50, 40, 30, 30)
minibatch = np.random.standard_normal(size=batch_shape).astype(theano.config.floatX)
xsub1, xsub2 = numpy_version(minibatch)
print xsub1.shape, xsub2.shape
theano_version = compile_theano_version()
xsub1, xsub2 = theano_version(minibatch)
print xsub1.shape, xsub2.shape
main()
This prints
(50L, 3L) (50L, 1L)
TensorType(float64, matrix) TensorType(float64, col)
(50L, 3L) (50L, 1L)
So a col is indeed a matrix with one column and not a vector.
Related
I'm trying to create an output tensor with dimensionality 32 x 576 x 2 from an operation between matrices M and X, with the following shapes:
M.shape: (576, 2, 2048)
X.shape: (32, 2048)
The operation I'm defining is an element-wise cosine similarity, from the following equation:
which represents the cosine similarity between the feature vector 𝑥 and the vector M_j,k.
This is how I've implemented it in code (incorrectly), where BATCH_SIZE=32, C=576, V=2:
#tf.function
def call(self, X):
M = self.kernel
norm_M = tf.norm(M, ord=2, axis=2)
norm_X = tf.norm(X, ord=2, axis=1)
l_r = (some scalar value, separate to this question)
# Compute cosine similarity between X and M
# as a matrix with dimensionality:
# BATCH_SIZE x C x V
feature_batch_size = tf.shape(X)[0]
c = tf.shape(M)[0]
v = tf.shape(M)[1]
output_matrix = tf.zeros([feature_batch_size, c, v])
output_matrix = tf.Variable(output_matrix, trainable=False)
for row in tf.range(feature_batch_size):
for column in tf.range(c):
for channel in tf.range(v):
a = tf.tensordot(M[column][channel], X[row], 1)
b = norm_M[column][channel] * norm_X[row]
output_matrix[row][column][channel] = a / b
return [output_matrix, l_r]
This fails on the line output_matrix[row][column][channel] = a / b because it's unhappy with an assignment to an individual row:column:channel of a tf.Variable.
Is there a better way to do this operation over these two matrices to create the desired output matrix so that it can be done without these three nested for loops and maintain compatibility with the tf.Function graph functionality?
If not, what can I do to assign variables to individual elements on a tf.Variable as I'm unsuccessfully attempting to do here?
Extra information:
norm_M.shape: (576, 2)
norm_X.shape: (32,)
You can replace these loops completely by using vectorized operations in the place of for loops.
num = tf.einsum('ij,klj->ikl',X,M)
denom = tf.einsum('i,jk->ijk',norm_X, norm_M)
output_matrix = num/denom
So I need a ND convolutional layer that also supports complex numbers. So I decided to code it myself.
I tested this code on numpy alone and it worked. Tested with several channels, 2D and 1D and complex. However, I have problems when I do it on TF.
This is my code so far:
def call(self, inputs):
with tf.name_scope("ComplexConvolution_" + str(self.layer_number)) as scope:
inputs = self._verify_inputs(inputs) # Check inputs are of expected shape and format
inputs = self.apply_padding(inputs) # Add zeros if needed
output_np = np.zeros( # I use np because tf does not support the assigment
(inputs.shape[0],) + # Per each image
self.output_size, # Image out size
dtype=self.input_dtype # To support complex numbers
)
img_index = 0
for image in inputs:
for filter_index in range(self.filters):
for i in range(int(np.prod(self.output_size[:-1]))): # for each element in the output
index = np.unravel_index(i, self.output_size[:-1])
start_index = tuple([a * b for a, b in zip(index, self.stride_shape)])
end_index = tuple([a+b for a, b in zip(start_index, self.kernel_shape)])
# set_trace()
sector_slice = tuple(
[slice(start_index[ind], end_index[ind]) for ind in range(len(start_index))]
)
sector = image[sector_slice]
new_value = tf.reduce_sum(sector * self.kernels[filter_index]) + self.bias[filter_index]
# I use Tied Bias https://datascience.stackexchange.com/a/37748/75968
output_np[img_index][index][filter_index] = new_value # The complicated line
img_index += 1
output = apply_activation(self.activation, output_np)
return output
input_size is a tuple of shape (dim1, dim2, ..., dim3, channels). An 2D rgb conv for example will be (32, 32, 3) and inputs will have shape (None, 32, 32, 3).
The output size is calculated from an equation I found in this paper: A guide to convolution arithmetic for deep learning
out_list = []
for i in range(len(self.input_size) - 1): # -1 because the number of input channels is irrelevant
out_list.append(int(np.floor((self.input_size[i] + 2 * self.padding_shape[i] - self.kernel_shape[i]) / self.stride_shape[i]) + 1))
out_list.append(self.filters)
Basically, I use np.zeros because if I use tf.zeros I cannot assign the new_value and I get:
TypeError: 'Tensor' object does not support item assignment
However, in this current state I am getting:
NotImplementedError: Cannot convert a symbolic Tensor (placeholder_1:0) to a numpy array.
On that same assignment. I don't see an easy fix, I think I should change the strategy of the code completely.
In the end, I did it in a very inefficient way based in this comment, also commented here but at least it works:
new_value = tf.reduce_sum(sector * self.kernels[filter_index]) + self.bias[filter_index]
indices = (img_index,) + index + (filter_index,)
mask = tf.Variable(tf.fill(output_np.shape, 1))
mask = mask[indices].assign(0)
mask = tf.cast(mask, dtype=self.input_dtype)
output_np = array * mask + (1 - mask) * new_value
I say inefficient because I create a whole new array for each assignment. My code is taking ages to compute for the moment so I will keep looking for improvements and post here if I get something better.
I have successfully performed 2d interpolation in python using the RectBivariateSpline method from scipy.interpolate. However, it is performed on numpy arrays. I want to perform it on tensors solely using tensorflow.
This is what I have right now: It works if all are numpy arrays. However, I am having a hard time to rewrite it in tensorflow.
x_old = np.arange(0,256)
y_old = np.arange(0,256)
#x = tensor of shape [256,256]
#y = tensor of shape [256,256]
#in_im = tensor of shape [256,256,3]
#out_im = tensor of shape [256,256,3]
for d in range(0,3):
interpf = RectBivariateSpline( x_old, y_old, in_im[:,:,d])
out_im[:,:,d] = interpf.ev(x[:,:], y[:,:])
The resize operators in tf.image might be what you are looking for, e.g. tf.image.resize_bicubic (https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/image/resize_bicubic)
To convert tensors into numpy array is the solution.
This question about conversion might be helpful.
In short, Any tensor returned by Session.run or eval is a NumPy array.
Example code is below.
import tensorflow as tf
import numpy as np
from scipy.interpolate import RectBivariateSpline
x = tf.constant([1,2,3,4])
y = tf.constant([1,2,3,4,5])
vals = tf.constant([
[4,1,4,4,2],
[4,2,3,2,6],
[3,7,4,3,5],
[2,4,5,3,4]
])
sess = tf.Session()
x, y, vals = sess.run([x, y, vals]) # x, y vals are now ndarray
rect_B_spline = RectBivariateSpline(x, y, vals)
a = tf.constant([3.2, 3.8, 2.2])
b = tf.constant([2.4, 4.3, 3.3])
a = sess.run([a, b])
print(rect_B_spline.ev(a, b))
So assuming I have this:
TensorShape([Dimension(None), Dimension(32)])
And I use tf.split on this tensor _X with the dimension above:
_X = tf.split(_X, 128, 0)
What is the shape of this new tensor? The output is a list so its hard to know the shape of this new tensor.
tf.split() returns the list of tensor objects. You could know shape of each tensor object as follows
import tensorflow as tf
X = tf.random_uniform([256, 32]);
Y = tf.split(X,128,0)
Y_shape = tf.shape(Y[1])
sess = tf.Session()
X_v,Y_v,Y_shape_v = sess.run([X,Y,Y_shape])
# numpy style
print X_v.shape
print len(Y_v)
print Y_v[100].shape
# TF style
print len(Y)
print Y_shape_v
Output :
(256, 32)
128
(2, 32)
128
[ 2 32]
I hope this helps !
tf.split(X, row = n, column = m) is used to split the data set of the variable into n number of pieces row wise and m numbers of pieces column wise.
For example, we have data_set x of size (10,10),
then tf.split(x, 2, 0) will break the data_set of x in 2 set of size (5, 10)
but if we take tf.split(x, 2, 2),
then we will get 4 sets of data of size (5, 5).
The new version of tensorflow defines split function as follows:
tf.split(
value,
num_or_size_splits,
axis=0,
num=None,
name='split'
)
however, when I try to run it in R:
X = tf$random_uniform(minval=0,
maxval=10,shape(256, 32),name = "X");
Y = tf$split(X,num_or_size_splits = 2,axis = 0)
it reports error message:
Error in py_call_impl(callable, dots$args, dots$keywords) :
ValueError: Rank-0 tensors are not supported as the num_or_size_splits argument to split. Argument provided: 2.0
I have a case where matrix multiplication of two matrices with certain dimensions work in numpy, but doesn't work in tensorflow.
x = np.ndarray(shape=(10,20,30), dtype = float)
y = np.ndarray(shape=(30,40), dtype = float)
z = np.matmul(x,y)
print("np shapes: %s x %s = %s" % (np.shape(x), np.shape(y), np.shape(z)))
This works as expected and prints:
np shapes: (10, 20, 30) x (30, 40) = (10, 20, 40)
However in tensorflow when I try to multiply placeholder and variable of the same shapes as the numpy arrays above I get an error
x = tf.placeholder(tf.float32, shape=(10,20,30))
y = tf.Variable(tf.truncated_normal([30,40], name='w'))
print("tf shapes: %s x %s" % (x.get_shape(), y.get_shape()))
tf.matmul(x,y)
Results in
tf shapes: (10, 20, 30) x (30, 40)
InvalidArgumentError:
Shape must be rank 2 but is rank 3 for 'MatMul_12'
(op: 'MatMul') with input shapes: [10,20,30], [30,40].
Why does this operation fail?
Don't know why tf.matmul does not support this kind of multiplication (may be one of the core developers could provide a meaningful answer).
But if you just want to be able to multiply tensors in this way, take a look at tf.einsum function. It could operate with tensors of arbitrary rank.
As suggested by Dmytro tf.einsum can be used to multiply these two arrays.
x = np.ndarray(shape=(10,20,30), dtype = float)
y = np.ndarray(shape=(30,40), dtype = float)
These two operations produce exactly the same result:
np.einsum('ijk,kl->ijl', x, y)
np.matmul(x,y)
And corresponding tensorflow operation also works
tf.einsum('ijk,kl->ijl', tf_x,tf_y)
People already told you that you can use tf.einsum() to get the result you want.
import tensorflow as tf
x = tf.random_normal([10, 20, 30])
y = tf.random_normal([30, 40])
z = tf.einsum('ijk,kl->ijl', x, y)
The reason why tf.matmul() does not work the way you expected is written in the documentation.
The inputs must be matrices (or tensors of rank > 2, representing
batches of matrices), with matching inner dimensions, possibly after
transposition.
In your case you have a matrix y and a tensor x (rank 3 > 2). In your case inner dimensions do not match. If you want, them to match, you will need to have something like this:
import tensorflow as tf
a, b, c = 12, 50, 20
x = tf.random_normal([a, b, c])
y = tf.random_normal([a, c, b])
z = tf.matmul(x, y)
But clearly it calculates not the stuff you want.