We Keep Coding

Could not find class for TF Ops: TensorListFromTensor when I'm trying to import a trained model with Tensorflow in DeepLearning4j

I'm new to Tensorflow and I'm trying to import a frozen graph (.pb file) that was trained in Python into a Java project using Deeplearning4j.
It seems that the model was saved successfully and it is working in Python, but when I try to import it with DL4J I'm getting the following issue and I don't know why:
Exception in thread "main" java.lang.IllegalStateException: Could not find class for TF Ops: TensorListFromTensor
at org.nd4j.common.base.Preconditions.throwStateEx(Preconditions.java:639)
at org.nd4j.common.base.Preconditions.checkState(Preconditions.java:301)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:283)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:141)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:87)
at org.nd4j.imports.graphmapper.tf.TFGraphMapper.importGraph(TFGraphMapper.java:73)
at MLModel.loadModel(MLModel.java:30)
This is my model in Python:
def RNN():
inputs = tf.keras.layers.Input(name='inputs',shape=[max_len])
layer = tf.keras.layers.Embedding(max_words,50,input_length=max_len)(inputs)
layer = tf.keras.layers.LSTM(64)(layer)
layer = tf.keras.layers.Dense(256,name='FC1')(layer)
layer = tf.keras.layers.Activation('relu')(layer)
layer = tf.keras.layers.Dropout(0.5)(layer)
layer = tf.keras.layers.Dense(12,name='out_layer')(layer)
layer = tf.keras.layers.Activation('softmax')(layer)
model = tf.keras.models.Model(inputs=inputs,outputs=layer)
return model
Actually I based on this blog how to export the model: Save, Load and Inference From TensorFlow 2.x Frozen Graph
And this is how I'm trying to import the model in Java with DeepLearning4J:
public static void loadModel(String filepath) throws Exception{
File file = new File(filepath);
if (!file.exists()){
file = new File(filepath);
}
sd = TFGraphMapper.importGraph(file);
if (sd == null) {
throw new Exception("Error loading model : " + file);
}
}
I'm getting the exception in sd = TFGraphMapper.importGraph(file);
Does anyone know if I'm missing something?

That is the old model import. Please use the new one. The old one is not and will not be supported. You can find that here:
https://deeplearning4j.konduit.ai/samediff/explanation/model-import-framework
Both tensorflow and onnx work similarly. For tensorflow use:
//create the framework importer
TensorflowFrameworkImporter tensorflowFrameworkImporter = new TensorflowFrameworkImporter();
File pathToPbFile = ...;
SameDiff graph = tensorflowFrameworkImporter.runImport(pathToPbFile.getAbsolutePath(),Collections.emptyMap());
File an issue on the github repo: https://github.com/deeplearning4j/deeplearning4j/issues/new if something doesn't work for you.
Also note that if you use the tf keras api you can also import it using the keras hdf5 format (the old one).
For many graphs, you may also need to save the model and freeze it. You can use that here:
def convert_saved_model(saved_model_dir) -> GraphDef:
"""
Convert the saved model (expanded as a directory)
to a frozen graph def
:param saved_model_dir: the input model directory
:return: the loaded graph def with all parameters in the model
"""
saved_model = tf.saved_model.load(saved_model_dir)
graph_def = saved_model.signatures['serving_default']
frozen = convert_variables_to_constants_v2(graph_def)
return frozen.graph.as_graph_def()
We publish more code and utilities for that kind of thing here:
https://github.com/deeplearning4j/deeplearning4j/tree/master/contrib/omnihub/src/omnihub/frameworks

How to re-download tokenizer for huggingface?

I have the exact same problem as https://github.com/huggingface/transformers/issues/11243, except it only does not work in Jupyter lab. It does work in python in my shell. EDIT: It is now not working in shell either after I closed and reopened the shell.
I downloaded the cardiffnlp/twitter-roberta-base-emotion model using:
model_name = "cardiffnlp/twitter-roberta-base-emotion"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
I saved the model with model.save_pretrained(model_name) and now I can't load the tokenizer. If I run:
tokenizer = AutoTokenizer.from_pretrained(model_name)
it gives the error:
OSError: Can't load tokenizer for 'cardiffnlp/twitter-roberta-base-emotion'. Make sure that:
- 'cardiffnlp/twitter-roberta-base-emotion' is a correct model identifier listed on 'https://huggingface.co/models'
(make sure 'cardiffnlp/twitter-roberta-base-emotion' is not a path to a local directory with something else, in that case)
- or 'cardiffnlp/twitter-roberta-base-emotion' is the correct path to a directory containing relevant tokenizer files
Because I saved the model and not the tokenizer yesterday, I can't load the tokenizer anymore. What can I do to fix this? I don't understand how to save the tokenizer if I can't load the tokenizer.

The model and tokenizer are two different things yet do share the same location to which you download them. You need to save both the tokenizer and the model. I wrote a simple utility to help.
import typing as t
from loguru import logger
from pathlib import Path
import torch
from transformers import PreTrainedModel
from transformers import PreTrainedTokenizer
class ModelLoader:
"""ModelLoader
Downloading and Loading Hugging FaceModels
Download occurs only when model is not located in the local model directory
If model exists in local directory, load.
"""
def __init__(
self,
model_name: str,
model_directory: str,
tokenizer_loader: PreTrainedTokenizer,
model_loader: PreTrainedModel,
):
self.model_name = Path(model_name)
self.model_directory = Path(model_directory)
self.model_loader = model_loader
self.tokenizer_loader = tokenizer_loader
self.save_path = self.model_directory / self.model_name
if not self.save_path.exists():
logger.debug(f"[+] {self.save_path} does not exit!")
self.save_path.mkdir(parents=True, exist_ok=True)
self.__download_model()
self.tokenizer, self.model = self.__load_model()
def __repr__(self):
return f"{self.__class__.__name__}(model={self.save_path})"
# Download model from HuggingFace
def __download_model(self) -> None:
logger.debug(f"[+] Downloading {self.model_name}")
tokenizer = self.tokenizer_loader.from_pretrained(f"{self.model_name}")
model = self.model_loader.from_pretrained(f"{self.model_name}")
logger.debug(f"[+] Saving {self.model_name} to {self.save_path}")
tokenizer.save_pretrained(f"{self.save_path}")
model.save_pretrained(f"{self.save_path}")
logger.debug("[+] Process completed")
# Load model
def __load_model(self) -> t.Tuple:
logger.debug(f"[+] Loading model from {self.save_path}")
tokenizer = self.tokenizer_loader.from_pretrained(f"{self.save_path}")
# Check if GPU is available
device = "cuda" if torch.cuda.is_available() else "cpu"
logger.info(f"[+] Model loaded in {device} complete")
model = self.model_loader.from_pretrained(f"{self.save_path}").to(device)
logger.debug("[+] Loading completed")
return tokenizer, model
def retrieve(self) -> t.Tuple:
"""Retriver
Returns:
Tuple: tokenizer, model
"""
return self.tokenizer, self.model
You can use it as
…
model_name = "cardiffnlp/twitter-roberta-base-emotion"
model_directory = "/tmp" # or where you want to store models
tokenizer_loader = AutoTokenizer
model_loader = AutoModelForSequenceClassification
get_model = ModelLoader(model_name=model_name, model_directory=model_directory, tokenizer_loader=tokenizer_loader, model_loader=model_loader)
model, tokenizer = get_model.retrieve()

How do I load the model artifact from AWS Sagemaker built-in container?

I'm using the linear-learner container from Sagemaker to train a model. The training has completed and the model artifact is saved in S3. I download it which is a .tar.gz file and there is the actual model file stored in it called model-algo-1 without format extension. I'm trying to load this model and inspect the model coefficients but not sure how to do so.
I tried pickle and joblib but they didn't work. Does anyone know how to load the model file trained from Sagemaker built-in container? Or is there any other way I can check the model coefficients? It's a logistic regression model.
Thanks

I managed to get this working.
From the SageMaker documentation, there are two classes that can be used to load an already deployed Linear Learner
Option 1: Use sagemaker.LinearLearnerModel to build predictor from S3 artificat
According to the documentation, this class:
"Reference LinearLearner s3 model data"
You can use it as following:
Step 1: Build LinearLearner Model using the artificats hosted on S3
from sagemaker import LinearLearnerModel, get_execution_role, Session
import boto3
sess = boto3.Session(region_name=region_name)
sagemaker_session = Session(boto_session=sess)
role = get_execution_role(sagemaker_session)
model = LinearLearnerModel(model_data, role, sagemaker_session=sagemaker_session)
Step 2: Deploy the model to an endpoint (in this example it is a serverless endpoint)
my_serverless_inference_config = ServerlessInferenceConfig(memory_size_in_mb=2048, max_concurrency=1)
linear_predictor = model .deploy(endpoint_name=my_endpoint_name,serverless_inference_config=my_serverless_inference_config,serializer=CSVSerializer(), deserializer=JSONDeserializer())
from sagemaker.predictor import csv_serializer, json_deserializer
linear_regressor.serializer = csv_serializer
linear_regressor.deserializer = json_deserializer
Step 3: use the sagemaker.predictor object returned by the deploy function
result = linear_predictor.predict(X_test)
#Iterate the result JSON to get an NP array of all the predictions so we can compare to Y test
predictions = np.array([res['score'] for res in result['predictions']])
Option 2: Use LinearLearnerPredictor to build a predictor from an already endpoint
You can follow this option if your model has an already deployed endpoint
from sagemaker import LinearLearnerPredictor, get_execution_role, Session
import boto3
sess = boto3.Session(region_name=region_name)
sagemaker_session = Session(boto_session=sess)
role = get_execution_role(sagemaker_session)
predictor = LinearLearnerPredictor(endpoint_name, sagemaker_session=sagemaker_session)

Set up tensorboard on Matterport - Mask RCNN

I am following this tutorial for image detection using Matterport repo.
I tried following this guide and edited the code to
How can I edit the following code to visualize the tensorboard ?
import tensorflow as tf
import datetime
%load_ext tensorboard
sess = tf.Session()
file_writer = tf.summary.FileWriter('/path/to/logs', sess.graph)
And then in the model area
# prepare config
config = KangarooConfig()
config.display()
# define the model
model = MaskRCNN(mode='training', model_dir='./', config=config)
model.keras_model.metrics_tensors = []
# Tensorflow board
logdir = os.path.join(
"logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
# load weights (mscoco) and exclude the output layers
model.load_weights('mask_rcnn_coco.h5',
by_name=True,
exclude=[
"mrcnn_class_logits", "mrcnn_bbox_fc", "mrcnn_bbox",
"mrcnn_mask"
])
# train weights (output layers or 'heads')
model.train(train_set,
test_set,
learning_rate=config.LEARNING_RATE,
epochs=5,
layers='heads')
I am not sure where to callbacks=[tensorboard_callback] ?

In your model.train, if you look closely in the source code documentation, there is parameter called custom_callbacks, which defaults to None.
It is there where you need to write your code, so to train with a custom callback, you will need to add this line of code:
model.train(train_set,
test_set,
learning_rate=config.LEARNING_RATE,
custom_callbacks = [tensorboard_callback],
epochs=5,
layers='heads')

You only have to open Anaconda Prompt and write tensorboard --logdir= yourlogdirectory, where yourlogdirectory is the directory containing the model checkpoint.
It should look something like this: logs\xxxxxx20200528T1755, where xxxx stands for the name you give to your configuration.
This command will generate a web address, copy it in our browser of preference.

How to use multiple inputs for custom Tensorflow model hosted by AWS Sagemaker

I have a trained Tensorflow model that uses two inputs to make predictions. I have successfully set up and deployed the model on AWS Sagemaker.
from sagemaker.tensorflow.model import TensorFlowModel
sagemaker_model = TensorFlowModel(model_data='s3://' + sagemaker_session.default_bucket()
+ '/R2-model/R2-model.tar.gz',
role = role,
framework_version = '1.12',
py_version='py2',
entry_point='train.py')
predictor = sagemaker_model.deploy(initial_instance_count=1,
instance_type='ml.m4.xlarge')
predictor.predict([data_scaled_1.to_csv(),
data_scaled_2.to_csv()]
)
I always receive an error. I could use an AWS Lambda function, but I don't see any documentation on specifying multiple inputs to deployed models. Does anyone know how to do this?

You need to actually build a correct signature when deploying the model first.
Also, you need to deploy with tensorflow serving.
At inference, you need to also give a proper input format when requesting: basically sagemaker docker server takes the request input and passes it by to tensorflow serving. So, the input needs to match TF serving inputs.
Here is a simple example of deploying a Keras multi-input multi-output model in Tensorflow serving using Sagemaker and how to make inference afterwards:
import tarfile
from tensorflow.python.saved_model import builder
from tensorflow.python.saved_model.signature_def_utils import predict_signature_def
from tensorflow.python.saved_model import tag_constants
from keras import backend as K
import sagemaker
#nano ~/.aws/config
#get_ipython().system('nano ~/.aws/config')
from sagemaker import get_execution_role
from sagemaker.tensorflow.serving import Model
def serialize_to_tf_and_dump(model, export_path):
"""
serialize a Keras model to TF model
:param model: compiled Keras model
:param export_path: str, The export path contains the name and the version of the model
:return:
"""
# Build the Protocol Buffer SavedModel at 'export_path'
save_model_builder = builder.SavedModelBuilder(export_path)
# Create prediction signature to be used by TensorFlow Serving Predict API
signature = predict_signature_def(
inputs={
"input_type_1": model.input[0],
"input_type_2": model.input[1],
},
outputs={
"decision_output_1": model.output[0],
"decision_output_2": model.output[1],
"decision_output_3": model.output[2]
}
)
with K.get_session() as sess:
# Save the meta graph and variables
save_model_builder.add_meta_graph_and_variables(
sess=sess, tags=[tag_constants.SERVING], signature_def_map={"serving_default": signature})
save_model_builder.save()
# instanciate model
model = ....
# convert to tf model
serialize_to_tf_and_dump(model, 'model_folder/1')
# tar tf model
with tarfile.open('model.tar.gz', mode='w:gz') as archive:
archive.add('model_folder', recursive=True)
# upload it to s3
sagemaker_session = sagemaker.Session()
inputs = sagemaker_session.upload_data(path='model.tar.gz')
# convert to sagemaker model
role = get_execution_role()
sagemaker_model = Model(model_data = inputs,
name='DummyModel',
role = role,
framework_version = '1.12')
predictor = sagemaker_model.deploy(initial_instance_count=1,
instance_type='ml.t2.medium', endpoint_name='MultiInputMultiOutputModel')
At inference, here is how to request for predictions:
import json
import boto3
x_inputs = ... # list with 2 np arrays of size (batch_size, ...)
data={
'inputs':{
"input_type_1": x[0].tolist(),
"input_type_2": x[1].tolist()
}
}
endpoint_name = 'MultiInputMultiOutputModel'
client = boto3.client('runtime.sagemaker')
response = client.invoke_endpoint(EndpointName=endpoint_name, Body=json.dumps(data), ContentType='application/json')
predictions = json.loads(response['Body'].read())

You likely need to customize the inference functions loaded in the endpoints. In the SageMaker TF SDK doc here you can find that there are two options for SageMaker TensorFlow deployment:
Python Endpoint, that is the default, check if modifying the
input_fn can accomodate your inference scheme
TF Serving
endpoint
You can diagnose error in Cloudwatch (accessible through the sagemaker endpoint UI), choose the most appropriate serving architecture among the above-mentioned two and customize the inference functions if need be

Only the TF serving endpoint supports multiple inputs in one inference request. You can follow the documentation here to deploy a TFS endpoint -
https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst