TLDR
My for lambda layers to get tensor slices only get the last column of data.
I have a (Batch_size, R) shape tensor that I will be running through an embedding layer for each of the R features seperately. I wrote the following code to split the input (Batch_size, R) shaped tensor into R (None,) slices.
R=2
inp = tf.keras.Input(shape = (R,), dtype=tf.int32)
SLICES = []
for i in range(R):
slice_ = tf.keras.layers.Lambda(lambda a: a[:,i], name=f"slice_{i}", dtype=tf.int32)(inp)
SLICES.append(slice_)
model = tf.keras.Model(inputs= inp, outputs = SLICES)
Running tf.keras.utils.plot_model(model, show_shapes=True, show_dtype=True) makes it appear that the code works. Running data into the model shows that there is a problem: the model takes the last feature and copies it for all layers.
input_ = np.array([[1,2],[3,4],[5,6]])
model.predict(input_)
[array([2, 4, 6], dtype=int32), array([2, 4, 6], dtype=int32)]
Approach 1
I "fixed" the problem in the R=2 case by getting rid of the for loop and writing each layer by hand.
slice1 = tf.keras.layers.Lambda(lambda a: a[:,0], name=f"first_slice", dtype=tf.int32)(inp)
slice2 = tf.keras.layers.Lambda(lambda a: a[:,1], name=f"second_slice", dtype=tf.int32)(inp)
model = tf.keras.Model(inputs= inp, outputs = [slice1, slice2])
input_ = np.array([[1,2],[3,4],[5,6]])
model.predict(input_)
[array([1, 3, 5], dtype=int32), array([2, 4, 6], dtype=int32)]
This is clearly undesirable for any number of reasons.
Approach 2
Another approach is to do the embedding on the raw features. Unfortunately, I have a CutMix like layer in front of the embedding operation, preventing me from embedding the raw features.
How can I get the for loop to correctly copy each slice of the tensor?
The reason why your first block of codes not working is you need to write the lambda function like this instead: lambda a,k=i: a[:,k]
R=2
inp = tf.keras.Input(shape = (R,), dtype=tf.int32)
SLICES = []
for i in range(R):
slice_ = tf.keras.layers.Lambda(lambda a,k=i: a[:,k], name=f"slice_{i}", dtype=tf.int32)(inp)
SLICES.append(slice_)
model = tf.keras.Model(inp, SLICES)
input_ = np.array([[1,2],[3,4],[5,6]])
print(model.predict(input_))
Outputs:
[array([1, 3, 5], dtype=int32), array([2, 4, 6], dtype=int32)]
Related
In my Tensorflow 2 model, I want my batch size to be parametric, such that I can build tensors which have appropriate batch size dynamically. I have the following code:
batch_size_param = 128
tf_batch_size = tf.keras.Input(shape=(), name="tf_batch_size", dtype=tf.int32)
batch_indices = tf.range(0, tf_batch_size, 1)
md = tf.keras.Model(inputs={"tf_batch_size": tf_batch_size}, outputs=[batch_indices])
res = md(inputs={"tf_batch_size": batch_size_param})
The code throws an error in tf.range:
ValueError: Shape must be rank 0 but is rank 1
for 'limit' for '{{node Range}} = Range[Tidx=DT_INT32](Range/start, tf_batch_size, Range/delta)' with input shapes: [], [?], []
I think the problem is with the fact that tf.keras.Input automatically tries to expand the input array at the first dimension, since it expects the partial shape of the input without the batch size and will attach the batch size according to the shape of the input array, which in my case a scalar. I can just feed the scalar value as a constant integer into tf.range but this time, I won't be able to change it after the model graph has been compiled.
Interestingly, I failed to find a proper way to input only a scalar into a TF-2 model even though I checked the documentation, too. So, what would be the best way to handle such a case?
Don't use tf.keras.Input and just define the model by subclassing.
import tensorflow as tf
class ScalarModel(tf.keras.Model):
def __init__(self):
super().__init__()
def call(self, x):
return tf.range(0, x, 1)
print(ScalarModel()(10))
# tf.Tensor([0 1 2 3 4 5 6 7 8 9], shape=(10,), dtype=int32)
I'm not sure if this is actually a good idea, but you could use tf.squeeze like
inp = keras.Input(shape=(), dtype=tf.int32)
batch_indices = tf.range(tf.squeeze(inp))
model = keras.Model(inputs=inp, outputs=batch_indices)
so that
model(6)
gives
<tf.Tensor: shape=(6,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5])>
Edit:
Depending on what you want to achieve, it might also be worth looking into ragged tensors:
inp = keras.Input(shape=(), dtype=tf.int32)
batch_indices = tf.ragged.range(inp)
model = keras.Model(inputs=inp, outputs=batch_indices)
would make
model(np.array([6,7]))
return
<tf.RaggedTensor [[0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4, 5, 6]]>
I'm trying to use Perceptron to reduce a tensor of size: [1, 24, 768] to another tensor with size of [1, 1, 768]. The only way I could use was to first reshape the input tensor to [1, 1, 24*768] and then pass it through linear layers. I'm wondering if there's a more elegant way of this transformation --other than using RNNs cause I do not want to use that. Are there other methods generally for the transformation that I want to make? Here is my code for doing the above operation:
lin = nn.Linear(24*768, 768)
# x is in shape of [1, 24, 768]
# out is in shape of [1, 1, 768]
x = x.view(1,1,-1)
out = lin(x)
If the broadcasting is what's bothering you, you could use a nn.Flatten to do it:
>>> m = nn.Sequential(
... nn.Flatten(),
... nn.Linear(24*768, 768))
>>> x = torch.rand(1, 24, 768)
>>> m(x).shape
torch.Size([1, 768])
If you really want the extra dimension you can unsqueeze the tensor on axis=1:
>>> m(x).unsqueeze(1).shape
torch.Size([1, 1, 768])
Say I have two rank 1 tensors of different (important) length:
import tensorflow as tf
x = tf.constant([1, 2, 3])
y = tf.constant([4, 5])
Now I want to append y to the end of x to give me the tensor:
<tf.Tensor: shape=(5,), dtype=int32, numpy=array([1, 2, 3, 4, 5], dtype=int32)>
But I can't seem to figure out how.
I will be doing this inside a function that I will decorate with tf.function, and it is my understanding that everything needs to be tensorflow operations for the tf.function decorator to work. That is, converting x and y to numpy arrays and back to a tensor will cause problems.
Thanks!
EDIT:
The solution is to use tf.concat() as pointed out by #Andrey:
tf.concat([x, y], axis=0)
It turns out that the problem originated when trying to append a single number to the end of a rank 1 tensor as follows:
x = tf.constant([1, 2, 3])
y = tf.constant(5)
tf.concat([x, y], axis=0)
which fails since here y is a rank 0 tensor of shape (). This can be solved by writing:
x = tf.constant([1, 2, 3])
y = tf.constant([5])
tf.concat([x, y], axis=0)
since y will then be a rank 1 tensor of shape (1,).
Use tf.concat():
import tensorflow as tf
t1 = tf.constant([1, 2, 3])
t2 = tf.constant([4, 5])
output = tf.concat([t1, t2], 0)
I want to apply "tf.nn.max_pool()" on a single image but I get a result with dimension that is totally different than the input:
import tensorflow as tf
import numpy as np
ifmaps_1 = tf.Variable(tf.random_uniform( shape=[ 7, 7, 3], minval=0, maxval=3, dtype=tf.int32))
ifmaps=tf.dtypes.cast(ifmaps_1, dtype=tf.float64)
ofmaps_tf = tf.nn.max_pool([ifmaps], ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding="SAME")[0] # no padding
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
print("ifmaps_tf = ")
print(ifmaps.eval())
print("ofmaps_tf = ")
result = sess.run(ofmaps_tf)
print(result)
I think this is related to trying to apply pooling to single example not on a batch. I need to do the pooling on a single example.
Any help is appreciated.
Your input is (7,7,3), kernel size is (3,3) and stride is (2,2). So if you do not want any paddings, (state in your comment), you should use padding="VALID", that will return a (3,3) tensor as output. If you use padding="SAME", it will return (4,4) tensor.
Usually, the formula of calculating output size for SAME pad is:
out_size = ceil(in_sizei/stride)
For VALID pad is:
out_size = ceil(in_size-filter_size+1/stride)
I want to modify my input by adding several different suffixes to the input vectors. For example, if the (single) input is [1, 5, 9, 3] I want to create three vectors (stored as matrix) like this:
[[1, 5, 9, 3, 1, 0, 0],
[1, 5, 9, 3, 0, 1, 0],
[1, 5, 9, 3, 0, 0, 1]]
Of course, this is just one observation so the input to the model is (None, 4) in this case. The simple way is to prepare the input data somewhere else (numpy most probably) and adjust the shape of input accordingly. That I can do but I would prefer doing it inside TensorFlow/Keras.
I have isolated the problem into this code:
import keras.backend as K
from keras import Input, Model
from keras.layers import Lambda
def build_model(dim_input: int, dim_eye: int):
input = Input((dim_input,))
concat = Lambda(lambda x: concat_eye(x, dim_input, dim_eye))(input)
return Model(inputs=[input], outputs=[concat])
def concat_eye(x, dim_input, dim_eye):
x = K.reshape(x, (-1, 1, dim_input))
x = K.repeat_elements(x, dim_eye, axis=1)
eye = K.expand_dims(K.eye(dim_eye), axis=0)
eye = K.tile(eye, (-1, 1, 1))
out = K.concatenate([x, eye], axis=2)
return out
def main():
import numpy as np
n = 100
dim_input = 20
dim_eye = 3
model = build_model(dim_input, dim_eye)
model.compile(optimizer='sgd', loss='mean_squared_error')
x_train = np.zeros((n, dim_input))
y_train = np.zeros((n, dim_eye, dim_eye + dim_input))
model.fit(x_train, y_train)
if __name__ == '__main__':
main()
The problem seems to be in the -1 in shape argument in tile function. I tried to replace it with 1 and None. Each has its own error:
-1: error during model.fit
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected multiples[0] >= 0, but got -1
1: error duting model.fit
tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [32,3,20] vs. shape[1] = [1,3,3]
None: error during build_model:
Failed to convert object of type <class 'tuple'> to Tensor. Contents: (None, 1, 1). Consider casting elements to a supported type.
You need to use K.shape() instead to get the symbolic shape of input tensor. That's because the batch size is None and therefore passing K.int_shape(x)[0] or None or -1 as a part of the second argument of K.tile() would not work:
eye = K.tile(eye, (K.shape(x)[0], 1, 1))