I know, there are a lot of related questions, but they are outdated, mostly they are even dealing with TensorFlow 1.
I have 1 GPU (GeForce 960) which is recognized by TensorFlow, so the installation was successful.
I'm not sure if this is the right way to do it, but this is how I train a Keras-model:
def create_model():
model = Sequential()
model.add(Conv2D(128, (3,3), padding="valid"))
model.add(layers.BatchNormalization())
model.add(layers.Activation(activations.relu))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(layers.Dense(10, activation="softmax"))
return model
strategy = tf.distribute.MirroredStrategy(devices=["/gpu:0"])
with strategy.scope():
model = create_model()
model.compile(loss="categorical_crossentropy",
optimizer="adam",
metrics=["acc"])
train_dataset, test_dataset = get_dataset()
model.fit(train_dataset,
epochs=20,
verbose=1,
validation_data=test_dataset)
But I get a lot of problems:
The exact same code is equally fast when I turn off the Strategy part
I always get this "warning": BaseCollectiveExecutor::StartAbort Out of range: End of sequence
I found out, that when I run this code, with the strategy part turned off, in a different Anaconda environment which does not have GPU support (CUDA etc), then it is way slowlier. So, is the GPU automatically used when you are in a GPU supporting environment (because, as stated in 1., it is equally fast without the strategy part)?
Is this the right way to use my GPU? If not, what is the right way?
Related
I have a very simple LSTM model which I've built in tensorflow and it works on CPU. However, I want to use this model on GPU. For the pytorch, I've defined the device and etc, however for tensorflow, I don't have any idea why it can not work. Do you have any suggestion for me? Thanks
model = Sequential()
model.add(LSTM(64, activation='relu', input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True))
model.add(LSTM(32, activation='relu', return_sequences=False))
model.add(Dropout(0.1))
model.add(Dense(Y_train.shape[1], kernel_regularizer='l2'))
callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=50)
opt = keras.optimizers.Adam(learning_rate=0.0008)
model.compile(optimizer=opt, loss='mse')
#model.summary()
history = model.fit(X_train, Y_train, epochs=2, batch_size=100, validation_data=(X_val, Y_val), callbacks=[callback],verbose=1, device).to(device)
For tensorflow, the models run on GPU for computations by default. It is given on their official documentation.
Is there some kind of error that shows up when you run your model? Because this should work just fine when running on GPU as well instead of a CPU.
So this question might stem from a lack of knowledge about tensorflow. But I am trying to build a multilayer perceptron with tensorflow 2.0, but without Keras.
The reason being that it is a requirement for my machine learning course that we do not use keras. Why you might ask? I am not sure.
I already have implemented our model in tensorflow 2.0 with Keras ease, and now I want to do the exact same thing without keras.
model = Sequential()
model.add(Dense(64, activation='relu', input_dim=784))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(5, activation='softmax'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
X_train = X[:7000]
y_train = tf.keras.utils.to_categorical(y[:7000], num_classes=5)
X_dev = X[7000:]
y_dev = tf.keras.utils.to_categorical(y[7000:], num_classes=5)
model.fit(X_train, y_train,
epochs=100,
batch_size=128)
score = model.evaluate(X_dev, y_dev, batch_size=128)
print(score)
Here is my problem. Whenever I look up the documentation on Tensorflow 2.0, then even the guides on custom training are using Keras.
As placeholders and sessions are a thing of the past in tensorflow 2.0, as I understand it, then I am a bit unsure of how to structure it.
I can make tensor objects. I have the impression that I need to use eager execution and use gradient tape. But I still am unsure of how to put these things together.
Now my question is. Where should I look to get a better understanding? Which direction has the greatest descent?
Please do tell me if I am doing this stack overflow post wrong. It is my first time here.
As #Daniel Möller stated, there are these tutorials for custom training and custom layers on the official TensorFlow page. As stated on the custom training page:
This tutorial used tf.Variable to build and train a simple linear model.
There is also this blog that creates custom layers and training without Keras API. You can check this code on Google Colab, which uses Cifar-10 with custom layers and training in the same manner.
I've created keras model to recognize human activity, based on data from mobile accelerometer:
model = Sequential()
model.add(Reshape((const.PERIOD, const.N_FEATURES), input_shape=(240,)))
model.add(Conv1D(100, 10, activation='relu', input_shape=(const.PERIOD, const.N_FEATURES)))
model.add(Conv1D(100, 10, activation='relu'))
model.add(MaxPooling1D(const.N_FEATURES))
model.add(Conv1D(160, 10, activation='relu'))
model.add(Conv1D(160, 10, activation='relu'))
model.add(Flatten())
model.add(Dropout(0.5))
model.add(Dense(7, activation='softmax'))
model.summary()
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
I've tested model, and the accuracy after ten epochs is like 85-90%. I don't know, but when I converse my model to TF Lite and I run interpreter in my android app, there's horrible predictions. What can be reason of that bad results? No compatibility on keras -> tensorflow -> tensorflow lite line? Should I run it with another way, using something like servlet + keras model?
A few suggestions:
Try to visualize your tflite graph with
https://lutzroeder.github.io/netron/. See if there's anything
unexpected.
Try to debug with tensorflow lite's python API first. Feed the same
input to the keras model and tflite model and compare the output
tensor.
I am going through tutorial for handwritten text recognition. And to do hand written digit recognition the author has constructed a Keras model as follows:
# # Creating CNN model
input_shape = (28,28,1)
number_of_classes = 10
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(number_of_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),metrics=['accuracy'])
model.summary()
history = model.fit(X_train, y_train,epochs=5, shuffle=True,
batch_size = 200,validation_data= (X_test, y_test))
model.save('digit_classifier2.h5')
Source (here)
I am very confused that on how has the author choose these layers. I know how Conv2D works by applying filters to an image, I know what is activation function. In short I have a rough understanding of what each term means.
What I am finding it difficult is how do I know what is happening in each step of this code?
For example lets take this python code:
values_List=[11,34,43]
for index, num in enumerate(values_List):
print(index,num)
I know that line 1 initializes a list named values_List
Line 2 iterates through this list
Line 3 prints output as (index of a number , number)
This python code is easy to understand and debug. But I am confused that if there is any error inside the keras layers. How do I proceed to debug this Keras code ? How do I see output on each step inside the Keras code ?
In short, you can't easily debug in Keras cause it is a high-level API made for the faster and easier implementations of Neural network architecture using pre-defined layers and functions there is less chance of error inside these layers or function cause it is well tested.
If you want to more fine-grained control on you you need to implement in Low-level API like Tensorflow v1 or use tf.GradientTape with tf-keras in TensorFlow v2 to see gradients at each step.
You can also try Tensorwatch by Microsoft for a deeper understanding of your model -
https://github.com/microsoft/tensorwatch
I am trying to develop a CNN for signature recognition to identify which person a given signature belongs to. There are 3 different classes(persons) and 23 signatures for each of them. Having this little amount of samples, I decided to use the Keras ImageDataGenerator to create additional images.
However, testing the CNN on different machines (Windows 10 and Mac OS) gives different accuracy scores when evaluating the model on the test data. 100% on windows and 93% on mac OS. Both machines run python 3.7.3 64-bit.
The data is split using train_test_split from sklearn.model_selection with 0.8 for training and 0.2 for testing, random_state is 1. Data and labels are properly normalised and adjusted to fit into the CNN. Various numbers for steps_per_epoch, batch_size and epoch have been tried.
I have tried using both np.random.seed and tensorflow.set_random_seed, generating 100% accuracy on the test data using seed(1) on PC, however the same seed on the other machine still yields a different accuracy score.
Here is the CNN architecture along with the method call for creating additional images. The following code yields an accuracy of 100% on one machine and 93.33% on the other.
seed(185)
set_random_seed(185)
X_train, X_test, y_train, y_test = train_test_split(data, labels, train_size=0.8, test_size=0.2, random_state=1)
datagen = ImageDataGenerator()
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit_generator(datagen.flow(X_train, y_train, batch_size=64), validation_data=(X_test,y_test),steps_per_epoch= 30, epochs=10)
model.evaluate(X_test, y_test)
EDIT
So after more research I've discovered that using different hardware, specifically different graphics cards, will result in varying accuracies.
Saving the trained model and using that to evalute data on different machines is the ideal solution.
Providing the solution here (Answer Section), even though it is present in the Question Section, for the benefit of the community.
Using different hardware, specifically different graphics cards will result in varying accuracies though we use same seeds, code and dataset. Saving the trained model and using that to evaluate data on different machines is the ideal solution.