I'm trying to do post-training full 8-bit quantization of a Keras model to compile and deploy to EdgeTPU.
I have a trained Keras model saved as .h5 file, and am trying to go through the steps as specified here: https://coral.withgoogle.com/docs/edgetpu/models-intro/, for deployment to the Coral Dev Board.
I'm following these instructions for quantization: https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization_of_weights_and_activations)
I’m trying to use the following code:
import tensorflow as tf
num_calibration_steps = 100
def representative_dataset_gen():
for _ in range(num_calibration_steps):
# Get sample input data as a numpy array in a method of your choosing.
yield [X_train_quant_conv]
converter = tf.compat.v1.lite.TFLiteConverter.from_keras_model_file('/tmp/classNN_simple.h5')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.representative_dataset = representative_dataset_gen
tflite_full_integer_quant_model = converter.convert()
where X_train_quant_conv is a subset of my training data converted to np.array and of type np.float32
When running this piece of code, I get the following error:
ValueError: Cannot set tensor: Dimension mismatch
I’ve tried changing the function representative_dataset_gen() in different ways, but every time I get a new error. I’m not sure how this function should be. I’m also in doubt of what value num_calibration_steps should have.
Any suggestions or working examples are very appreciated.
This question is very similar to this answered question: Convert Keras model to quantized Tensorflow Lite model that can be used on Edge TPU
You might want to look at my demo script for quantization, on github.
It's just a guess since I can't see what X_train_quant_conv really is, but in my working demo, I yield one image at a time (random data created on the fly, in my case) in representative_dataset_gen(). The image is stored as batch of size 1 (e.g., tensor shape is (1, 56, 56, 32) for my 52x52x32 image). There are 32 channels, though there would typically just be 3, for a color image. I think representative_dataset_gen() has to yield a list containing a tensor (or more than one?) for which the first dimension is of length 1.
image_shape = (56, 56, 32)
def representative_dataset_gen():
num_calibration_images = 10
for i in range(num_calibration_images):
image = tf.random.normal([1] + list(image_shape))
yield [image]
Related
I want to be able to predict the Class of a Single Image from the Learner and i always get an Index Out of Bound Exception .
Here is the Code
data = ImageDataLoader.from_folder(path, train="Train", valid ="Valid",
ds_tfms=get_transforms(), size=(256,256), bs=32, num_workers=4)
//Model is a Sequential One
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)// The Model
learn.fit_one_cycle(100, lr_max=3e-3)
Img = //PIL Image Path
learn.predict(img)
The Model is able to Predict on ImageDataLoader but not on a Single Image .If anyone has any clue it would be much appreciated
Here is a Link to FastAi but didnt solve the issue
https://forums.fast.ai/t/how-to-use-learner-predict-list-index-out-of-range/81998/7
EDIT NOTE : I have tried to convert the Image to a tensor Flow but another error is given .Photo of the Error
I think that the problem is that data is a rank 4 tensor whereas Img is rank 3. In other words, it is missing the #points or batch dimension up front. In TF one can fix that with tf.expand_dims like so
img = tf.expand_dims(img, axis=0)
or fixing it when passing to the model
learn.predict(tf.expand_dims(img, axis=0))
You can also look at tf.newaxis (see the second code example here).
I have read the Coreml guide which shows how to convert a pb model to mlmodel by using coremltools. However, I get the error below when trying to follow the guide. Which means the input shape must be specific.
ValueError: "ResizeBilinear" op: the second input, which is the output size, must be known statically
So, have anyone know how to convert the flexible input shape mlmodel?
Here is my code:
import coremltools as ct
def mlmodel_image(pb):
input_shape = ct.Shape(shape=(1, ct.RangeDim(1, 720), ct.RangeDim(1, 1280), 3))
model_input = ct.ImageType(shape=input_shape)
mlmodel = ct.convert(pb, inputs=[model_input], source='TensorFlow')
mlmodel.save(pb.replace(".pb", "_img.mlmodel"))
print('------save to ', pb.replace(".pb", "_img.mlmodel"))
please try my sample:
https://github.com/dhrebeniuk/RealTimeFastStyleTransfer
And look my article with attached Google Colab Notebook in PyTorch.
There is instructions how run Style Transfer on iOS with maximum performance.
I am trying to predict on a savedmodel using the following code
features = np.ones((20, 40, 3), dtype=np.float32)
features = tf.convert_to_tensor(value, dtype=tf.float32)
imported_model = tf.saved_model.load(export_dir=os.path.join(os.path.join(model_path, directory)))
import_fn = imported_model.signatures["serving_default"]
import_fn(features)
I get the following error when running it using Tensorflow 2. The model prediction works fine when I use the saved_model_cli.
tensorflow.python.framework.errors_impl.InvalidArgumentError: In[0] is not a matrix. Instead it has shape [20,40,3]
[[node dense/BiasAdd (defined at model_manager.py:54) ]] [Op:__inference_pruned_318590]
The saved cli command is as follows
saved_model_cli run --dir ./model_dir --tag_set serve --signature_def serving_default --input_exprs 'input=np.ones((20, 40, 3), dtype=np.float32)'
InvalidArgumentError is typically caused by Data Type mismatch in the input.
Based on your error " In[0] is not a matrix. Instead, it has shape [20,40,3] ". You could try to manipulate your input data to properly match the input type and shape that the model is originally trained. You could also inspect how the model treats your input when you used the saved_model_cli compared to Python IDE. As you might be missing some preprocessing steps when you used the Python IDE that is being done when using saved_model_cli.
You could read about using Saved_Model format usage more in this link.
PS. Please dont point me to converting Keras model directly to tflite as my .h5 file would fail to convert to .tflite directly. I somehow managed to convert my .h5 file to .pb
I have followed this Jupyter notebook for face recognition using Keras. I then saved my model to a model.h5 file, then converted it to a frozen graph, model.pb using this.
Now I want to use my tensorflow file in Android. For this I will need to have Tensorflow Lite, which requires me to convert my model into a .tflite format.
For this, I'm trying to follow the official guidelines for it here. As you can see there, it requires input_array and output_array arrays. How do I obtain details of these things from my model.pb file?
input arrays and output arrays are the arrays which store input and output tensors respectively.
They intend to inform the TFLiteConverter about the input and output tensors which will be used at the time of inference.
For a Keras model,
The input tensor is the placeholder tensor of the first layer.
input_tensor = model.layers[0].input
The output tensor may relate with a activation function.
output_tensor = model.layers[ LAST_LAYER_INDEX ].output
For a Frozen Graph,
import tensorflow as tf
gf = tf.GraphDef()
m_file = open('model.pb','rb')
gf.ParseFromString(m_file.read())
We get the names of the nodes,
for n in gf.node:
print( n.name )
To get the tensor,
tensor = n.op
The input tensor may be a placeholder tensor. Output tensor is the tensor which you run using session.run()
For conversion, we get,
input_array =[ input_tensor ]
output_array = [ output_tensor ]
Im using TFCoreml in python to convert my Tensorflow model into CoreML for development on an iOS device using the CoreML Libs.
Im using the following python code to try and convert the model to CoreML.
import tfcoreml as tf_converter
tf_converter.convert(tf_model_path = 'frozen_inference_graph.pb',
mlmodel_path = 'ml_model.mlmodel',
output_feature_names = ['SemanticPredictions:0'],
input_name_shape_dict = {'ImageTensor:0' : [1, 512, 512, 3]})
This gives the following error:
Shape Translator missing for OP of type Slice.
I read the docs a bit further for TFCoreml and it states that Slice isn't fully supported, and some custom conversion code is required for this to work. In the TFCoreml documentation it suggests breaking the frozen graph into sub graphs and converting them individually then merging them back together post conversion.
I updated my code to use custom layers, but I don't really understand how the custom conversion functions work.
Just need a few pointers on where to look to begin understanding how to write these custom conversion methods so I can solve my problem with converting the Tensorflow model to CoreML.
[EDIT]
I did some more reading into the TFCoreml examples and documentation and have adapted my solution to this.
import tfcoreml as tf_converter
def _convert_slice(**kwargs):
tf_op = kwargs["op"]
coreml_nn_builder = kwargs["nn_builder"]
constant_inputs = kwargs["constant_inputs"]
params = NeuralNetwork_pb2.CustomLayerParams()
params.className = 'Slice'
params.description = "Custom layer that corresponds to the slice TF op"
# get the value of begin
begin = constant_inputs.get(tf_op.inputs[1].name, [0,0,0,0])
size = constant_inputs.get(tf_op.inputs[2].name, [0,0,0,0])
# add begin and size as two repeated weight fields
begin_as_weights = params.weights.add()
begin_as_weights.floatValue.extend(map(float, begin))
size_as_weights = params.weights.add()
size_as_weights.floatValue.extend(map(float, size))
coreml_nn_builder.add_custom(name=tf_op.name,
input_names=[tf_op.inputs[0].name],
output_names=[tf_op.outputs[0].name],
custom_proto_spec=params)
coreml_model = tfcoreml.convert(
tf_model_path='frozen_inference_graph.pb',
mlmodel_path='my_model.mlmodel',
input_name_shape_dict={'ImageTensor:0':[1, 512, 512, 3]},
output_feature_names=['SemanticPredictions:0'],
add_custom_layers=True,
custom_conversion_functions={'Slice': _convert_slice}) # dictionary has op name as the key
print("\n \n ML Model layers info: \n")
# inspect the CoreML model: this should be same as the one we got above
spec = coreml_model.get_spec()
_print_coreml_nn_layer_info(spec)
I'm still getting the same error as before
Shape Translator missing for OP of type Slice.
But i did notice I'm also getting this error/warning
custom_conversion_functions={'Slice': _convert_slice}) # dictionary has op name as the key
Any assistance would be appreciated