We Keep Coding

How to convert a style transfer tensorflow model to mlmodel with flexible input shape?

I have read the Coreml guide which shows how to convert a pb model to mlmodel by using coremltools. However, I get the error below when trying to follow the guide. Which means the input shape must be specific.
ValueError: "ResizeBilinear" op: the second input, which is the output size, must be known statically
So, have anyone know how to convert the flexible input shape mlmodel?
Here is my code:
import coremltools as ct
def mlmodel_image(pb):
input_shape = ct.Shape(shape=(1, ct.RangeDim(1, 720), ct.RangeDim(1, 1280), 3))
model_input = ct.ImageType(shape=input_shape)
mlmodel = ct.convert(pb, inputs=[model_input], source='TensorFlow')
mlmodel.save(pb.replace(".pb", "_img.mlmodel"))
print('------save to ', pb.replace(".pb", "_img.mlmodel"))

please try my sample:
https://github.com/dhrebeniuk/RealTimeFastStyleTransfer
And look my article with attached Google Colab Notebook in PyTorch.
There is instructions how run Style Transfer on iOS with maximum performance.

How to write image data to tensorboard without opening new categories

I wrote a Keras/Tensorflow callback that writes a confusion matrix to the Images tab in Tensorboard. This worked fine in TF 2.1. Unfortunately I had to convert it to TF 1.14, as other packages depend on this version.
Everything works fine (more or less), except for the Tensorboard Reports.
The Problem
As you can see in the screenshot below, there are many categories(? tags? channels? I am not sure about the terminology) instead of only one.
screenshot: multiple categories for images
Seemingly correlated to this issue, the training scalars like "val_loss" only plot the first datapoint and nothing afterwards. See second screenshot
screenshot: scalars showing one data point
Also, Tensorboard is printing the following error:
File <path/to/event-file> updated even though the current file is <path/to/new-event-file>
So I assume somehow my TF FileWriters are disagreeing on where to write.
Regarding the Confusion Matrix Callback: The writing function looks like this:
def _figure_to_summary(self, fig, step):
# attach a new canvas if not exists
if fig.canvas is None:
matplotlib.backends.backend_agg.FigureCanvasAgg(fig)
fig.canvas.draw()
w, h = fig.canvas.get_width_height()
img = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
img = img.reshape((1, h, w, 3))
with K.get_session().as_default():
tensor = tf.constant(img)
image = tf.summary.image(self.title, tensor).eval()
self._summary_writer.add_summary(image, global_step=step)
self._summary_writer.flush()
With fig being the matplot figure of the Confusion Matrix and step as an int, which should enable Tensorboard to add the little slider over the image to show the history of the matrix.
The model is trained as follows:
run_id = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
tb_path = os.path.join("tensor/", run_id)
tb_reporter = TensorBoard(tb_path)
summ_wr = FileWriter(tb_path)
conf_matr = ConfusionMatrix(va, TARGET_CLASSES.keys(), summ_wr, normalize=True)
cb_list = [tb_reporter, conf_matr]
model.fit(tr, epochs=500, validation_data=va, callbacks=cb_list)
Where summ_wr becomes self._summary_writer of the callback.
What I tried
I only tried changing the way the summary is written. Trying tf.merge_all(), opening, closing and reopening the FileWriter in various combinations, but nothing changed. When I deactivate the custom callback, the Tensorboard callback works as expected.
My Questions
How can I write Image Data into the same category every time?
Will the image get the scrollbar?
How can I fix the problem, that the Keras Tensorboard callback does not show its data?
I assume all the problems are related and solving one results in solving all of them, but I am completely stumped on how to do it.
I am thankful for any suggestions :)
Edit:
I just found this question: Tensorboard Image Summaries
This seems to solve the problem, unfortunately the answer does not have a lot of context, so I don't how to integrate the solution into the code.

The solution I found consists of two parts.
First: I stole the FileWriter from the Tensorboard Callback from keras. Tensorboard.writer contains the FileWriter after set_model has been called.
Second: The Summary must be created via the constructor and be passed the tag-keyword. This forces the images into the same group and the writers do not interfere with each other.
I found the second part of the solution here: TensorBoard: How to write images to get a steps slider?

Post-training full integer quantization of Keras model

I'm trying to do post-training full 8-bit quantization of a Keras model to compile and deploy to EdgeTPU.
I have a trained Keras model saved as .h5 file, and am trying to go through the steps as specified here: https://coral.withgoogle.com/docs/edgetpu/models-intro/, for deployment to the Coral Dev Board.
I'm following these instructions for quantization: https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization_of_weights_and_activations)
I’m trying to use the following code:
import tensorflow as tf
num_calibration_steps = 100
def representative_dataset_gen():
for _ in range(num_calibration_steps):
# Get sample input data as a numpy array in a method of your choosing.
yield [X_train_quant_conv]
converter = tf.compat.v1.lite.TFLiteConverter.from_keras_model_file('/tmp/classNN_simple.h5')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.representative_dataset = representative_dataset_gen
tflite_full_integer_quant_model = converter.convert()
where X_train_quant_conv is a subset of my training data converted to np.array and of type np.float32
When running this piece of code, I get the following error:
ValueError: Cannot set tensor: Dimension mismatch
I’ve tried changing the function representative_dataset_gen() in different ways, but every time I get a new error. I’m not sure how this function should be. I’m also in doubt of what value num_calibration_steps should have.
Any suggestions or working examples are very appreciated.
This question is very similar to this answered question: Convert Keras model to quantized Tensorflow Lite model that can be used on Edge TPU

You might want to look at my demo script for quantization, on github.
It's just a guess since I can't see what X_train_quant_conv really is, but in my working demo, I yield one image at a time (random data created on the fly, in my case) in representative_dataset_gen(). The image is stored as batch of size 1 (e.g., tensor shape is (1, 56, 56, 32) for my 52x52x32 image). There are 32 channels, though there would typically just be 3, for a color image. I think representative_dataset_gen() has to yield a list containing a tensor (or more than one?) for which the first dimension is of length 1.
image_shape = (56, 56, 32)
def representative_dataset_gen():
num_calibration_images = 10
for i in range(num_calibration_images):
image = tf.random.normal([1] + list(image_shape))
yield [image]

TensorFlow: How to perform image categorisation on multiple images

Hoping that somebody can help out with a TensorFlow query. It's not a difficult one, I'm sure. I am just somewhat lacking in knowledge relating to TensorFlow and NumPy.
Without any prior experience of TensorFlow I have implemented Python code from a tutorial for doing image classification. This works. Once trained, it can tell the difference between a cat and a dog.
This is currently hard-wired for a single image. I would like to be able to classify multiple images (i.e. the contents of a folder), and do this efficiently. What I have done so far in an effort to achieve this is to simply add a loop around everything, so it runs all the code for each image. However, timing the operation shows that classification of each successive image takes longer than the previous one. Therefore there is some kind of incremental overhead. Some operation is taking more time with every loop. I cannot immediately see what this is.
There are two options to improve this. Either:
(1) Leave the loop largely as it is and prevent this slowdown, or
(2) (Preferable IMHO, if it is possible) Pass a list of images to TensorFlow for classification, and get back a list of results. This seems more efficient.
This is the code:
import tensorflow as tf
import numpy as np
import os,glob,cv2
import sys,argparse
import time
try:
inputdir = [redacted - insert input dir here]
for f in os.listdir(inputdir):
start_time = time.time()
filename = os.path.join(inputdir,f)
image_size=128
num_channels=3
images = []
image = cv2.imread(filename) # read image using OpenCV
# Resize image to desired size and preprocess exactly as done during training...
image = cv2.resize(image, (image_size, image_size),0,0, cv2.INTER_LINEAR)
images.append(image)
images = np.array(images, dtype=np.uint8)
images = images.astype('float32')
images = np.multiply(images, 1.0/255.0)
# The input to the network is of shape [None image_size image_size num_channels]. Hence we reshape.
x_batch = images.reshape(1, image_size,image_size,num_channels)
sess = tf.Session() # restore the saved model
saver = tf.train.import_meta_graph('dogs-cats-model.meta') # Step 1: Recreate the network graph. At this step only graph is created
saver.restore(sess, tf.train.latest_checkpoint('./')) # Step 2: Load the weights saved using the restore method
graph = tf.get_default_graph() # access the default graph which we have restored
# Now get hold of the op that we can be processed to get the output.
# In the original network y_pred is the tensor that is the prediction of the network
y_pred = graph.get_tensor_by_name("y_pred:0")
## Feed the images to the input placeholders...
x= graph.get_tensor_by_name("x:0")
y_true = graph.get_tensor_by_name("y_true:0")
y_test_images = np.zeros((1, 2))
# Create the feed_dict that is required to be fed to calculate y_pred...
feed_dict_testing = {x: x_batch, y_true: y_test_images}
result=sess.run(y_pred, feed_dict=feed_dict_testing)
# Note: result is a numpy.ndarray
print(f + '\t' + str(result) + ' ' + '%.2f' % (time.time()-start_time) + ' seconds')
# next image
except:
import traceback
tb = traceback.format_exc()
print(tb)
finally:
input() # keep window open until key is pressed
What I tried to do to modify the above was to create a list of filenames using...
images.append(image)
...and then taking the rest of the code out of the loop. However, this didn't work. It resulted in the following error:
ValueError: cannot reshape array of size 294912 into shape
(1,128,128,3)
At this line:
x_batch = images.reshape(1, image_size,image_size,num_channels)
Apparently this Reshape method doesn't work (as implemented, at least) on a list of images.
So my questions are:
What would causing the steady increase in image classification time that I have seen as images are iterated?
Can I perform classification on multiple images in one go, rather than one-by-one in a loop?
Thanks in advance.

Your issues:
1 a) The main reason why it is so slow is: You are re-creating the graph for every image.
1 b) The incremental overhead is coming from creating a new session every time without destroying the old session. The with syntax helps with that. e.g.:
with tf.Session(graph=tf.Graph()) as session:
# do something with the session
But that won't be a noticable issue after addressing a).
When thinking about the problem, one might realise which parts of your code depend on the image and which don't. The TensorFlow related part that is different per image is the call to session.run, feeding in the image. Everything else can be moved out of the loop.
2) You can also classify multiple images in one go. The first dimension of x_batch is the batch size. You are specifying one. But you may exhaust your memory resources trying to do that for a very large number of images.

How to preprocess image before feeding into LeNet CNN in Caffe?

I am having issue in feeding my own image into LeNet using Caffe library. I have deployed and initialised the weight obtained through training with no difficulties. As the net is trained using input size of 28x28, I tried resizing the input image to 28x28 and feed into the deployed LeNet, but it gave me "unhashable numpy array" error.
Not only that, I also tried to transpose it with img = img.transpose(img, (2,0,1)) after resizing but it gave me "TypeError: only length-1 arrays can be converted to Python scalars"
Below are the python codes I tried so far in pre-processing my image :
img = caffe.io.load_image('number5.png')
img = caffe.io.resize_image(img, (28,28), interp_order=3)
img = img.transpose(img, (2,0,1))
I am a beginner in using Caffe and still in the process of learning. I hope someone can give me some example or insights in how to pre-process an image before feeding into the net.
Thank You.
Best regards.

Just use this instead:
img = img.transpose((2,0,1))

You can use caffe.io.Transformer
This is used to preprocess caffe 'data' blob.
Define it as
transformer = caffe.io.Transformer({'data':net.blobs['data'].data.shape})
transformer.set_transpose('data',(2,0,1))
then you can,
img = caffe.io.load_image('number5.png')
img = caffe.io.resize_image(img, (28,28), interp_order=3)
img_transposed=transformer.preprocess('data',img)