We Keep Coding

An FFNN in pyhton for recognition of nanocrystals

I tried with this one but i think its not really good working so any idea ?
the code:
import tensorflow as tf
import numpy as np
(x_train, y_train), (x_test, y_test) = load_data_MEB()
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))
test_loss, test_acc = model.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', test_acc)
The project is the recognition of nanocrystals (present in cement) on SEM.
We have a database of 1000 SEM pictures where we can see the crystals, we need a program in python or any other programming language (CNN but ideally FFNN) to recognize these crystals and eventually their shape (square and triangular).

Is Tensorflow Dataset + FeatureColumn API slower than plain numpy?

I'm playing with TF Dataset API and experiencing really slow times (10x slower) for training compared with plain Numpy array data.
Here is a simple example on MNIST, someone find something wrong?
import tensorflow as tf
from tensorflow import keras
from tensorflow import feature_column
import pandas as pd
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train/255.0, x_test/255.0
x_train = pd.DataFrame(x_train.reshape(x_train.shape[0], x_train.shape[1]*x_train.shape[2]), columns=['f_'+str(i) for i in range(x_train.shape[1]*x_train.shape[2])])
train_ds = tf.data.Dataset.from_tensor_slices((dict(x_train), y_train))
train_ds = train_ds.batch(1024)
x_test= pd.DataFrame(x_test.reshape(x_test.shape[0], x_test.shape[1]*x_test.shape[2]), columns=['f_'+str(i) for i in range(x_test.shape[1]*x_test.shape[2])])
test_ds = tf.data.Dataset.from_tensor_slices((dict(x_test), y_test))
test_ds = test_ds.batch(1024)
feature_columns1 = []
for i in list(x_test.columns.values):
feature_columns1.append(feature_column.numeric_column(i))
model = tf.keras.models.Sequential([
tf.keras.layers.DenseFeatures(feature_columns1),
tf.keras.layers.Dense(128,activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(
loss='sparse_categorical_crossentropy',
optimizer=tf.keras.optimizers.Adam(0.001),
metrics=['accuracy'],
)
model.fit(train_ds, epochs=6)
model.evaluate(test_ds)

traning MNIST model with proper hyper params in Python

here is my code, i don't know why it gives me 0.3% accuracy
can anyone tell me what is the problem with this code?
def train_mnist():
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
history = model.fit(x_train, y_train, epochs=5)
return history.epoch, history.history['acc'][-1]
train_mnist()
thanks in Adavnce

this will work! try this
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer='adam',
loss=loss_fn,
metrics=['accuracy'])

The problem seem to be your loss function
Try this:
Method 1
You could use categorical_crossentropy as loss but the last layer should be
tf.keras.layers.Dense(10,activation='softmax')
and then
model.compile(optimizer = 'adam',
loss"categorical_crossentropy",
metrics=["accuracy"])
Method 2
In your case, the sparse_categorical_crossentropy loss need to define
tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True,name='sparse_categorical_crossentropy')
To understand the difference b\w these two see this

How to save split training and testing data to file?

I'm using Python and Keras to predict a continuous value from given data. I've already build my Neural Network model and got the results that I wanted. Here's what my code looks like:
X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(X_scale, Y, test_size=0.3)
X_val, X_test, Y_val, Y_test = train_test_split(X_val_and_test, Y_val_and_test, test_size=0.5)
print(X_train.shape, X_val.shape, X_test.shape, Y_train.shape, Y_val.shape, Y_test.shape)
# Output: (693, 3) (148, 3) (149, 3) (693,) (148,) (149,)
#Building out model
model = Sequential([
Dense(9, activation='relu', input_shape=(3,)),
Dense(3, activation='relu'),
Dense(1, activation='linear'),
])
#Compiling model
model.compile(loss='mean_absolute_percentage_error',
metrics=['mse'],
optimizer='RMSprop')
#fit the model
hist = model.fit(X_train, Y_train,
batch_size=100, epochs=20, verbose=1,
validation_data=(X_val, Y_val))
# evaluate model
model.evaluate(X_test, Y_test)
# Output: 149/149 [==============================] - 0s 42us/step
# [93.14884595422937, 171.0550879152029]
Now, after building the model, I would like to split the data and get accuracies for training and testing data separately. How can I do that? Or extract the test and train data as a .csv file

See all correctly and incorrectly identified images when training on the mnist dataset

I'm trying to find a way to visualize which numbers in the mnist dataset a model was able to correctly identify and which ones it wasn't.
What I can't seem to find is if such a visualization is possible in tensorboard or if I would need to use/create something else to achieve it.
I'm currently working from the basic tutorial provided for tensorflow 2.0 with tensorboard added.
import datetime
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
model.fit(x_train,
y_train,
epochs=5,
validation_data=(x_test, y_test),
callbacks=[tensorboard_callback])
model.evaluate(x_test, y_test)

It appears the what-if tool is what I was looking for, it allows you to visually sort testing data depending on whether it was correctly or incorrectly identified by the model.
If you want to test it out here is their demo that I used to get the above image and they have multiple other demos on the tools site.