We Keep Coding

Could not find class for TF Ops: TensorListFromTensor when I'm trying to import a trained model with Tensorflow in DeepLearning4j

I'm new to Tensorflow and I'm trying to import a frozen graph (.pb file) that was trained in Python into a Java project using Deeplearning4j.
It seems that the model was saved successfully and it is working in Python, but when I try to import it with DL4J I'm getting the following issue and I don't know why:
Exception in thread "main" java.lang.IllegalStateException: Could not find class for TF Ops: TensorListFromTensor
at org.nd4j.common.base.Preconditions.throwStateEx(Preconditions.java:639)
at org.nd4j.common.base.Preconditions.checkState(Preconditions.java:301)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:283)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:141)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:87)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:73)
at MLModel.loadModel(MLModel.java:30)
This is my model in Python:
def RNN():
inputs = tf.keras.layers.Input(name='inputs',shape=[max_len])
layer = tf.keras.layers.Embedding(max_words,50,input_length=max_len)(inputs)
layer = tf.keras.layers.LSTM(64)(layer)
layer = tf.keras.layers.Dense(256,name='FC1')(layer)
layer = tf.keras.layers.Activation('relu')(layer)
layer = tf.keras.layers.Dropout(0.5)(layer)
layer = tf.keras.layers.Dense(12,name='out_layer')(layer)
layer = tf.keras.layers.Activation('softmax')(layer)
model = tf.keras.models.Model(inputs=inputs,outputs=layer)
return model
Actually I based on this blog how to export the model: Save, Load and Inference From TensorFlow 2.x Frozen Graph
And this is how I'm trying to import the model in Java with DeepLearning4J:
public static void loadModel(String filepath) throws Exception{
File file = new File(filepath);
if (!file.exists()){
file = new File(filepath);
}
sd = TFGraphMapper.importGraph(file);
if (sd == null) {
throw new Exception("Error loading model : " + file);
}
}
I'm getting the exception in sd = TFGraphMapper.importGraph(file);
Does anyone know if I'm missing something?

That is the old model import. Please use the new one. The old one is not and will not be supported. You can find that here:
https://deeplearning4j.konduit.ai/samediff/explanation/model-import-framework
Both tensorflow and onnx work similarly. For tensorflow use:
//create the framework importer
TensorflowFrameworkImporter tensorflowFrameworkImporter = new TensorflowFrameworkImporter();
File pathToPbFile = ...;
SameDiff graph = tensorflowFrameworkImporter.runImport(pathToPbFile.getAbsolutePath(),Collections.emptyMap());
File an issue on the github repo: https://github.com/deeplearning4j/deeplearning4j/issues/new if something doesn't work for you.
Also note that if you use the tf keras api you can also import it using the keras hdf5 format (the old one).
For many graphs, you may also need to save the model and freeze it. You can use that here:
def convert_saved_model(saved_model_dir) -> GraphDef:
"""
Convert the saved model (expanded as a directory)
to a frozen graph def
:param saved_model_dir: the input model directory
:return: the loaded graph def with all parameters in the model
"""
saved_model = tf.saved_model.load(saved_model_dir)
graph_def = saved_model.signatures['serving_default']
frozen = convert_variables_to_constants_v2(graph_def)
return frozen.graph.as_graph_def()
We publish more code and utilities for that kind of thing here:
https://github.com/deeplearning4j/deeplearning4j/tree/master/contrib/omnihub/src/omnihub/frameworks

How to re-download tokenizer for huggingface?

I have the exact same problem as https://github.com/huggingface/transformers/issues/11243, except it only does not work in Jupyter lab. It does work in python in my shell. EDIT: It is now not working in shell either after I closed and reopened the shell.
I downloaded the cardiffnlp/twitter-roberta-base-emotion model using:
model_name = "cardiffnlp/twitter-roberta-base-emotion"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
I saved the model with model.save_pretrained(model_name) and now I can't load the tokenizer. If I run:
tokenizer = AutoTokenizer.from_pretrained(model_name)
it gives the error:
OSError: Can't load tokenizer for 'cardiffnlp/twitter-roberta-base-emotion'. Make sure that:
- 'cardiffnlp/twitter-roberta-base-emotion' is a correct model identifier listed on 'https://huggingface.co/models'
(make sure 'cardiffnlp/twitter-roberta-base-emotion' is not a path to a local directory with something else, in that case)
- or 'cardiffnlp/twitter-roberta-base-emotion' is the correct path to a directory containing relevant tokenizer files
Because I saved the model and not the tokenizer yesterday, I can't load the tokenizer anymore. What can I do to fix this? I don't understand how to save the tokenizer if I can't load the tokenizer.

The model and tokenizer are two different things yet do share the same location to which you download them. You need to save both the tokenizer and the model. I wrote a simple utility to help.
import typing as t
from loguru import logger
from pathlib import Path
import torch
from transformers import PreTrainedModel
from transformers import PreTrainedTokenizer
class ModelLoader:
"""ModelLoader
Downloading and Loading Hugging FaceModels
Download occurs only when model is not located in the local model directory
If model exists in local directory, load.
"""
def __init__(
self,
model_name: str,
model_directory: str,
tokenizer_loader: PreTrainedTokenizer,
model_loader: PreTrainedModel,
):
self.model_name = Path(model_name)
self.model_directory = Path(model_directory)
self.model_loader = model_loader
self.tokenizer_loader = tokenizer_loader
self.save_path = self.model_directory / self.model_name
if not self.save_path.exists():
logger.debug(f"[+] {self.save_path} does not exit!")
self.save_path.mkdir(parents=True, exist_ok=True)
self.__download_model()
self.tokenizer, self.model = self.__load_model()
def __repr__(self):
return f"{self.__class__.__name__}(model={self.save_path})"
# Download model from HuggingFace
def __download_model(self) -> None:
logger.debug(f"[+] Downloading {self.model_name}")
tokenizer = self.tokenizer_loader.from_pretrained(f"{self.model_name}")
model = self.model_loader.from_pretrained(f"{self.model_name}")
logger.debug(f"[+] Saving {self.model_name} to {self.save_path}")
tokenizer.save_pretrained(f"{self.save_path}")
model.save_pretrained(f"{self.save_path}")
logger.debug("[+] Process completed")
# Load model
def __load_model(self) -> t.Tuple:
logger.debug(f"[+] Loading model from {self.save_path}")
tokenizer = self.tokenizer_loader.from_pretrained(f"{self.save_path}")
# Check if GPU is available
device = "cuda" if torch.cuda.is_available() else "cpu"
logger.info(f"[+] Model loaded in {device} complete")
model = self.model_loader.from_pretrained(f"{self.save_path}").to(device)
logger.debug("[+] Loading completed")
return tokenizer, model
def retrieve(self) -> t.Tuple:
"""Retriver
Returns:
Tuple: tokenizer, model
"""
return self.tokenizer, self.model
You can use it as
…
model_name = "cardiffnlp/twitter-roberta-base-emotion"
model_directory = "/tmp" # or where you want to store models
tokenizer_loader = AutoTokenizer
model_loader = AutoModelForSequenceClassification
get_model = ModelLoader(model_name=model_name, model_directory=model_directory, tokenizer_loader=tokenizer_loader, model_loader=model_loader)
model, tokenizer = get_model.retrieve()

Problem building tensorflow model from huggingface weights

I need to work with the pretrained BERT model ('dbmdz/bert-base-italian-xxl-cased') from Huggingface with Tensorflow (at this link).
After reading this on the website,
Currently only PyTorch-Transformers compatible weights are available. If you need access to TensorFlow checkpoints, please raise an issue!
I raised the issue and promptly a download link to an archive containing the following files was given to me. The files are the following ones:
$ ls bert-base-italian-xxl-cased/
config.json model.ckpt.index vocab.txt
model.ckpt.data-00000-of-00001 model.ckpt.meta
I'm now trying to load the model and work with it but everything I tried failed.
I tried following this suggestion from an Huggingface discussion site:
bert_folder = str(Config.MODELS_CONFIG.BERT_CHECKPOINT_DIR) # folder in which I have the files extracted from the archive
from transformers import BertConfig, TFBertModel
config = BertConfig.from_pretrained(bert_folder) # this gets loaded correctly
After this point I tried several combinations in order to load the model but always unsuccessfully.
eg:
model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config)
model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config, from_pt=True)
model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config, from_pt=True)
model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased", config=config, local_files_only=True)
Always results in this error:
404 Client Error: Not Found for url: https://huggingface.co/models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index/resolve/main/tf_model.h5
...
...
OSError: Can't load weights for '../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index'. Make sure that:
- '../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index' is a correct model identifier listed on 'https://huggingface.co/models'
- or '../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.
So my question is: How can I load this pre-trained BERT model from those files and use it in tensorflow?

You can try the following snippet to load dbmdz/bert-base-italian-xxl-cased in tensorflow.
from transformers import AutoTokenizer, TFBertModel
model_name = "dbmdz/bert-base-italian-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = TFBertModel.from_pretrained(model_name)
If you want to load from the given tensorflow checkpoint, you could try like this:
model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config, from_tf=True)

Set up tensorboard on Matterport - Mask RCNN

I am following this tutorial for image detection using Matterport repo.
I tried following this guide and edited the code to
How can I edit the following code to visualize the tensorboard ?
import tensorflow as tf
import datetime
%load_ext tensorboard
sess = tf.Session()
file_writer = tf.summary.FileWriter('/path/to/logs', sess.graph)
And then in the model area
# prepare config
config = KangarooConfig()
config.display()
# define the model
model = MaskRCNN(mode='training', model_dir='./', config=config)
model.keras_model.metrics_tensors = []
# Tensorflow board
logdir = os.path.join(
"logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
# load weights (mscoco) and exclude the output layers
model.load_weights('mask_rcnn_coco.h5',
by_name=True,
exclude=[
"mrcnn_class_logits", "mrcnn_bbox_fc", "mrcnn_bbox",
"mrcnn_mask"
])
# train weights (output layers or 'heads')
model.train(train_set,
test_set,
learning_rate=config.LEARNING_RATE,
epochs=5,
layers='heads')
I am not sure where to callbacks=[tensorboard_callback] ?

In your model.train, if you look closely in the source code documentation, there is parameter called custom_callbacks, which defaults to None.
It is there where you need to write your code, so to train with a custom callback, you will need to add this line of code:
model.train(train_set,
test_set,
learning_rate=config.LEARNING_RATE,
custom_callbacks = [tensorboard_callback],
epochs=5,
layers='heads')

You only have to open Anaconda Prompt and write tensorboard --logdir= yourlogdirectory, where yourlogdirectory is the directory containing the model checkpoint.
It should look something like this: logs\xxxxxx20200528T1755, where xxxx stands for the name you give to your configuration.
This command will generate a web address, copy it in our browser of preference.

NameError:name 'create_model' is not defined ....i have tried importing model from keras but it hasnt solved it .how to solve?

I tried creating a model using tensorflow. When I tried executing it shows me
the other files are in this link------- github.com/llSourcell/tensorflow_chatbot
def train():
enc_train, dec_train=data_utils.prepare_custom_data(
gConfig['working_directory'])
train_set = read_data(enc_train,dec_train)
def seq2seq_f(encoder_inputs,decoder_inputs,do_decode):
return tf.nn.seq2seq.embedding_attention_seq2seq(
encoder_inputs,decoder_inputs, cell,
num_encoder_symbols=source_vocab_size,
num_decoder_symbols=target_vocab_size,
embedding_size=size,
output_projection=output_projection,
feed_previous=do_decode)
with tf.Session(config=config) as sess:
model = create_model(sess,False)
while True:
sess.run(model)
checkpoint_path = os.path.join(gConfig['working_directory'],'seq2seq.ckpt')
model.saver.save(sess, checkpoint_path, global_step=model.global_step)
other than this the other python files ive used are in the github link specified in the comments section below
this is the code defining create_model in the execute.py file
def create_model(session, forward_only):
"""Create model and initialize or load parameters"""
model = seq2seq_model.Seq2SeqModel( gConfig['enc_vocab_size'], gConfig['dec_vocab_size'], _buckets, gConfig['layer_size'], gConfig['num_layers'], gConfig['max_gradient_norm'], gConfig['batch_size'], gConfig['learning_rate'], gConfig['learning_rate_decay_factor'], forward_only=forward_only)
if 'pretrained_model' in gConfig:
model.saver.restore(session,gConfig['pretrained_model'])
return model
ckpt = tf.train.get_checkpoint_state(gConfig['working_directory'])
# the checkpoint filename has changed in recent versions of tensorflow
checkpoint_suffix = ""
if tf.__version__ > "0.12":
checkpoint_suffix = ".index"
if ckpt and tf.gfile.Exists(ckpt.model_checkpoint_path + checkpoint_suffix):
print("Reading model parameters from %s" % ckpt.model_checkpoint_path)
model.saver.restore(session, ckpt.model_checkpoint_path)
else:
print("Created model with fresh parameters.")
session.run(tf.initialize_all_variables())
return model

Okay, it seems like you have copied code but you did not structure it. If create_model() is defined in another file then you have to import it. Have you done that? (i.e. from file_with_methods import create_model). You should consider editing your post and adding more of your code, if you want us to help.
Alternative: You could also clone the github repository(that you shared in your comment) and just change whatever you want to change in the execution.py file. This way you can keep the "hierarchy" that the owner uses and you could add your own code where needed.