We Keep Coding

How do you add a transformer layer to a neural net?

So, I have an Siamese-LSTM implementation where the net itself is pretty simple and looks like this:
x = Sequential()
x.add(Embedding(len(embeddings), embedding_dim,
weights=[embeddings], input_shape=(max_seq_length,), trainable=False))
x.add(LSTM(n_hidden))
shared_model = x
I want to be able compare it to the transformer equivalent. To be able to achieve that I need to replace "LSTM" with a transformer layer. Is there a way to make it work?
EDIT: LSTM comes from:
from tensorflow.python.keras.layers import LSTM

Implementing LSTM in Keras

I would appreciate your help with creating an LSTM model in Keras. My input is a 2D numpy array with rows that are numeric time-series of equal length.
I've written the following lines to create an LSTM model that operates on the raw data without an embedding layer:
lstm_clf = Sequential()
lstm_clf.add(LSTM(100, input_shape=(X_train[0], X_train[1]))
lstm_clf.add(Dense(7, activation='softmax'))
lstm_clf.compile(loss='sparse_categorical_crossentropy', optimizer='Adam', metrics=['accuracy'])
When I reach the fitting phase, I get the following error: "ValueError: Error when checking input: expected lstm_11_input to have 3 dimensions, but got array with shape (100, 289)".
Could anyone please explain to me what I'm doing wrong and how to fix the code. It must be related to the input shape, but I've no idea what it is.
Thanks a lot for your help,
Alexander.

I was able to find the answer. Here's the correct code:
lstm_clf = Sequential()
lstm_clf.add(LSTM(100, input_shape=(1, max_seq_len)))
lstm_clf.add(Dense(7, activation='softmax'))
lstm_clf.compile(loss='sparse_categorical_crossentropy', optimizer='Adam', metrics=['accuracy'])
My data also had to be reshaped, as following:
X_seqs_train = reshape(X_seqs_train, (X_seqs_train.shape[0], 1, X_seqs_train.shape[1]))
X_seqs_test = reshape(X_seqs_test, (X_seqs_test.shape[0], 1, X_seqs_test.shape1))
These changes solved the problem. A clue that led to the answer was found here.

Keras model predicting experience from hours

I am very new to Keras, neural networks and machine learning having just started to learn yesterday. I decided to try predicting the experience over an hour (0 to 23) (for a game and my own generated data-set) that a user would earn. Currently running what I have the predictions seem to be very low and very poor. I have tried a relu activation, which produced predictions all to be zero and from a bit of research, LeakyReLU.
This is the code I have for the prediction model so far:
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LeakyReLU
import numpy
numpy.random.seed(7)
dataset = numpy.loadtxt("experience.csv", delimiter=",")
X = dataset[: ,0]
Y = dataset[: ,1]
model = Sequential()
model.add(Dense(12, input_dim = 1, activation=LeakyReLU(0.3)))
model.add(Dense(8, activation=LeakyReLU(0.3)))
model.add(Dense(1, activation=LeakyReLU(0.3)))
model.compile(loss = 'mean_absolute_error', optimizer='adam', metrics = ['accuracy'])
model.fit(X, Y, epochs=120, batch_size=10, verbose = 0)
predictions = model.predict(X)
rounded = [round(x[0]) for x in predictions]
print(rounded)
I have also tried playing around with the hidden levels of the network, but honestly have no idea how many there should be or a good way to justify an amount.
If it helps here is the data-set I have been using:
https://raw.githubusercontent.com/NightShadeII/xpPredictor/master/experience.csv
Thankyou for any help

Looking at your data it does not seem like a classification problem.
You have two options:
-> Look at the second column and bucket them depending on the ranges and make classes that can be predicted, for instance: 0, 1, 2 etc. Now it tries to train but does not have enough examples for millions of classes that it thinks you are trying to predict.
-> If you want real valued output and not classes, try using linear regression.

How to train a model with only an Embedding layer in Keras and no labels

I have some text without any labels. Just a bunch of text files. And I want to train an Embedding layer to map the words to embedding vectors. Most of the examples I've seen so far are like this:
from keras.models import Sequential
from keras.layers import Embedding, Flatten, Dense
model = Sequential()
model.add(Embedding(max_words, embedding_dim, input_length=maxlen))
model.add(Flatten())
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.summary()
model.compile(optimizer='rmsprop',
loss='binary_crossentropy',
metrics=['acc'])
model.fit(x_train, y_train,
epochs=10,
batch_size=32,
validation_data=(x_val, y_val))
They all assume that the Embedding layer is part of a bigger model which tries to predict a label. But in my case, I have no label. I'm not trying to classify anything. I just want to train the mapping from words (more precisely integers) to embedding vectors. But the fit method of the model, asks for x_train and y_train (as the example given above).
How can I train a model only with an Embedding layer and no labels?
[UPDATE]
Based on the answer I've got from #Daniel Möller, Embedding layer in Keras is implementing a supervised algorithm and thus cannot be trained without labels. Initially, I was thinking that it is a variation of Word2Vec and thus does not need labels to be trained. Apparently, that's not the case. Personally, I ended up using the FastText which has nothing to do with Keras or Python.

Does it make sense to do that without a label/target?
How will your model decide which values in the vectors are good for anything if there is no objective?
All embeddings are "trained" for a purpose. If there is no purpose, there is no target, if there is no target, there is no training.
If you really want to transform words in vectors without any purpose/target, you've got two options:
Make one-hot encoded vectors. You may use the Keras to_categorical function for that.
Use a pretrained embedding. There are some available, such as glove, embeddings from Google, etc. (All of they were trained at some point for some purpose).
A very naive approach based on our chat, considering word distance
Warning: I don't really know anything about Word2Vec, but I'll try to show how to add the rules for your embedding using some naive kind of word distance and how to use dummy "labels" just to satisfy Keras' way of training.
from keras.layers import Input, Embedding, Subtract, Lambda
import keras.backend as K
from keras.models import Model
input1 = Input((1,)) #word1
input2 = Input((1,)) #word2
embeddingLayer = Embedding(...params...)
word1 = embeddingLayer(input1)
word2 = embeddingLayer(input2)
#naive distance rule, subtract, expect zero difference
word_distance = Subtract()([word1,word2])
#reduce all dimensions to a single dimension
word_distance = Lambda(lambda x: K.mean(x, axis=-1))(word_distance)
model = Model([input1,input2], word_distance)
Now that our model outputs directly a word distance, our labels will be "zero", they're not really labels for a supervised training, but they're the expected result of the model, something necessary for Keras to work.
We can have as loss function the mae (mean absolute error) or mse (mean squared error), for instance.
model.compile(optimizer='adam', loss='mse')
And training with word2 being the word after word1:
xTrain = entireText
xTrain1 = entireText[:-1]
xTrain2 = entireText[1:]
yTrain = np.zeros((len(xTrain1),))
model.fit([xTrain1,xTrain2], yTrain, .... more params.... )
Although this may be completely wrong regarding what Word2Vec really does, it shows the main points that are:
Embedding layers don't have special properties, they're just trainable lookup tables
Rules for creating an embedding should be defined by the model and expected outputs
A Keras model will need "targets", even if those targets are not "labels" but a mathematical trick for an expected result.

Output tensors to a Model must be Keras tensors

I was trying to make a model learned from difference between two model output. So I made code like below. But it occurred error read:
TypeError: Output tensors to a Model must be Keras tensors. Found:
Tensor("sub:0", shape=(?, 10), dtype=float32)
I have found related answer including lambda, but I couldn't solve this issue.
Does anyone know this issue?
It might be seen converting tensor to keras's tensor.
Thx in advance.
from keras.layers import Dense
from keras.models import Model
from keras.models import Sequential
left_branch = Sequential()
left_branch.add(Dense(10, input_dim=784))
right_branch = Sequential()
right_branch.add(Dense(10, input_dim=784))
diff = left_branch.output - right_branch.output
model = Model(inputs=[left_branch.input, right_branch.input], outputs=[diff])
model.compile(optimizer='rmsprop', loss='binary_crossentropy', loss_weights=[1.])
model.summary(line_length=150)

It's better to keep all operations done by a layer, do not subtract outputs like that (I wouldn't risk hidden errors for doing things differently from what the documentation expects):
from keras.layers import *
def negativeActivation(x):
return -x
left_branch = Sequential()
left_branch.add(Dense(10, input_dim=784))
right_branch = Sequential()
right_branch.add(Dense(10, input_dim=784))
negativeRight = Activation(negativeActivation)(right_branch.output)
diff = Add()([left_branch.output,negativeRight])
model = Model(inputs=[left_branch.input, right_branch.input], outputs=diff)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', loss_weights=[1.])
When joining models like that, I do prefer using the Model way of doing it, with layers, instead of using Sequential:
def negativeActivation(x):
return -x
leftInput = Input((784,))
rightInput = Input((784,))
left_branch = Dense(10)(leftInput) #Dense(10) creates a layer
right_branch = Dense(10)(rightInput) #passing the input creates the output
negativeRight = Activation(negativeActivation)(right_branch)
diff = Add()([left_branch,negativeRight])
model = Model(inputs=[leftInput, rightInput], outputs=diff)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', loss_weights=[1.])
With this, you can create other models with the same layers, they will share the same weights:
leftModel = Model(leftInput,left_branch)
rightModel = Model(rightInput,right_branch)
fullModel = Model([leftInput,rightInput],diff)
Training one of them will affect the others if they share the same layer.
You can train just the right part in the full model by making left_branch.trainable = False before compiling (or compile again for training), for instance.

I think I solve this, but it might be exact solution.
I added some code like below :
diff = left_branch.output - right_branch.output
setattr(diff, '_keras_history', getattr(right_branch.output, '_keras_history'))
setattr(diff, '_keras_shape', getattr(right_branch.output, '_keras_shape'))
setattr(diff, '_uses_learning_phase', getattr(right_branch.output, '_uses_learning_phase'))
The reason why the error occur is diff tensor doesn't have attr named _keras_history so on. So adding intentionally them to diff tensor could prevent above error. I checked original code ran and being possible to learn.