We Keep Coding

TorchServe: How to convert bytes output to tensors

I have a model that is served using TorchServe. I'm communicating with the TorchServe server using gRPC. The final postprocess method of the custom handler defined returns a list which is converted into bytes for transfer over the network.
The post process method
def postprocess(self, data):
# data type - torch.Tensor
# data shape - [1, 17, 80, 64] and data dtype - torch.float32
return data.tolist()
The main issue is at the client where converting the received bytes from TorchServe to a torch Tensor is inefficiently done via ast.literal_eval
# This takes 0.3 seconds
response = self.inference_stub.Predictions(
inference_pb2.PredictionsRequest(model_name=model_name, input=input_data))
# This takes 0.84 seconds
predictions = torch.as_tensor(literal_eval(
response.prediction.decode('utf-8')))
Using numpy.frombuffer or torch.frombuffer return the following error.
import numpy as np
np.frombuffer(response.prediction)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: buffer size must be a multiple of element size
np.frombuffer(response.prediction, dtype=np.float32)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: buffer size must be a multiple of element size
Using torch
import torch
torch.frombuffer(response.prediction, dtype = torch.float32)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: buffer length (2601542 bytes) after offset (0 bytes) must be a multiple of element size (4)
Is there an alternative, more efficient solution of converting the received bytes into torch.Tensor?

One hack I've found that has significantly increased the performance while sending large tensors is to return a list of json.
In your handler's postprocess function:
def postprocess(self, data):
output_data = {}
output_data['data'] = data.tolist()
return [output_data]
At the clients side when you receive the grpc response, decode it using json.loads
response = self.inference_stub.Predictions(
inference_pb2.PredictionsRequest(model_name=model_name, input=input_data))
decoded_output = response.prediction.decode('utf-8')
preds = torch.as_tensor(json.loads(decoded_output))
preds should have the output tensor
Update:
There's an even faster method and should completely solve the bottleneck. Use tf.io.serialize_tensor from tensorflow to serialize your tensor inside postprocess
def postprocess(self, data):
return [tf.io.serialize_tensor(data.cpu()).numpy()]
Decode it using tf.io.parse_tensor
response = self.inference_stub.Predictions(
inference_pb2.PredictionsRequest(model_name=model_name, input=input_data))
prediction = response.prediction
torch.as_tensor(tf.io.parse_tensor(prediction, out_type=tf.float32).numpy())

Why is Pytorch giving me a datatype error: Float vs Double?

I am working on migrating some working Pytorch code I found online (which is a 2D image classification example using the MNIST data; apologies that I lost track of the original source and am unable to find it) to what I need, which is converting a 1D collection of values into a numerical score. I created my own Dataset class. When I call model(), I get an error: RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm. My first level of confusion is that I can't find any reference to Python even having a Double datatype. And my second is why I get the error--when I put in debug code to show the datatype of mat1 and its elements, I am told that it is a Tensor which claims to be float64. I also wonder why it is expecting a scalar for mat1, which the documentation describes as a matrix/tensor.
The full error dump is
Traceback (most recent call last):
File "mlalan.py", line 174, in <module>
outputs = model(images)
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "mlalan.py", line 80, in forward
x = activate(self.fc1(x))
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm
Some of the key code from my Dataset class is
class RandomDataset(Dataset):
def __init__(self, csv_file, transform=None):
self.data_frame = pd.read_csv(csv_file, dtype=float)
def __getitem__(self, idx):
raw = self.data_frame.values[idx]
sample = raw[0:6], raw[6:8]
return sample
The full source code is at http://8wheels.org/mlalan.py.

By default, in Python, float means float32. However, in Pandas and Numpy, float means float64. I was able to resolve the problem by adding a call to astype as below. The "32" is required for it to work.
raw = self.data_frame.values[idx].astype(np.float32)
Thanks!
Now I can move on to the next crash :-)

Uber Ludwig: Issue Making Predictions

I decided to mess with Uber Ludwig again. I wanted to make a simple demo using the python API that learns to add 1 to the input number. I have successfully produced a model, but the issue arises when predicting. I am running on the newest release from github on PopOS 19.10 on CPU TensorFlow.
Thank you for any help.
Edit: I have reproduced the issue on windows as well.
The error is as follows
Traceback (most recent call last):
File "predict.py", line 3, in <module>
x = model.predict({"numberIn":[1]}, return_type='dict')
File "/home/user/.local/lib/python3.7/site-packages/ludwig/api.py", line 914, in predict
gpu_fraction=gpu_fraction,
File "/home/user/.local/lib/python3.7/site-packages/ludwig/api.py", line 772, in _predict
self.model_definition['preprocessing']
File "/home/user/.local/lib/python3.7/site-packages/ludwig/data/preprocessing.py", line 159, in build_data
preprocessing_parameters
File "/home/user/.local/lib/python3.7/site-packages/ludwig/data/preprocessing.py", line 180, in handle_missing_values
dataset_df[feature['name']] = dataset_df[feature['name']].fillna(
AttributeError: 'list' object has no attribute 'fillna'
Here is my prediction script
from ludwig.api import LudwigModel
model = LudwigModel.load("/home/user/Documents/ludwig-test/plus1/results/api_experiment_run_0/model")
x = model.predict({"numberIn":[1]}, return_type='dict')
#x = model.predict({"numberIn":[1]}, return_type=<class 'dict'>) I tried this with no success
print(x)
Here is the contents of my training script.
mydata = {"numberIn":[], "value":[]}
for x in range(10000):
mydata["numberIn"].append(x)
mydata["value"].append(x + 1)
from ludwig.api import LudwigModel
print("Imported Ludwig")
modelobject = LudwigModel(model_definition_file="modeldef.yaml")
stats = modelobject.train(data_dict=mydata)
modelobject.close()
modeldef.yaml
input_features:
-
name: numberIn
type: numerical
output_features:
-
name: value
type: numerical

Solution: Input argument of predict function is not positional and data_dict needs to be specified in this case.
x = modelobject.predict(data_dict=mydictionary)

Theano scan function and argument number lstm

I am new to Neural Networks and I am trying to modify this code RNN-Classifier and instead of using the GRU_step, I would rather use an LSTM.
I added one extra parameter c_prev
def lstm_step(x, h_prev, c_prev, W_xz, W_hz, W_xm, W_hm):
and after applying all the LSTM equations I am returning them both (h and c)
My hidden vector looks like that:
hidden_vector, _ = theano.scan(
lstm_step,
sequences=input_vectors,
outputs_info=initial_hidden_vector,
non_sequences=[W_xz, W_hz, W_xm, W_hm]
)
hidden_vector = hidden_vector[-1]
I get an exception like this and don't understand why it does not see the c_prev as an existant parameter (or how/where can I feed it with some values, so that it's not empty?)
python rnnclassifier.py data/sentiment.train.txt data/sentiment.test.txt
Traceback (most recent call last):
File "rnnclassifier.py", line 167, in <module>
rnn_classifier = RnnClassifier(word2id_len, n_classes)
File "rnnclassifier.py", line 110, in __init__
non_sequences=[W_xz, W_hz, W_xm, W_hm]
File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan.py",
line 773, in scan condition, outputs, updates =
scan_utils.get_updates_and_outputs(fn(*args))
TypeError: lstm_step() takes exactly 7 arguments (6 given)
I am new to this topic and would appreciate any help or advice! Thank you.

TypeError: 'float' object cannot be interpreted as an integer ( python 3.4 version)

I am trying to run a video file and getting error as below.
$ /usr/bin/python3.4 /home/ramakrishna/PycharmProjects/Lanedect/driving-lane-departure-warning-master/main.py
Traceback (most recent call last):
File "/home/ramakrishna/PycharmProjects/Lanedect/driving-lane-departure-warning-master/main.py", line 19, in <module>
img_aug = process_frame(img)
File "/home/ramakrishna/PycharmProjects/Lanedect/driving-lane-departure-warning-master/lane.py", line 615, in process_frame
output = create_output_frame(offcenter, pts, img_undist_, fps, curvature, curve_direction, binary_sub)
File "/home/ramakrishna/PycharmProjects/Lanedect/driving-lane-departure-warning-master/lane.py", line 467, in create_output_frame
whole_frame = np.zeros((h*2.5,w*2.34, 3), dtype=np.uint8)
TypeError: 'float' object cannot be interpreted as an integer

Below line is reason for error.
np.zeros((h*2.5,w*2.34, 3), dtype=np.uint8)
np.zeros expects dimensions as integers, while h*2.5 and w*2.34 evaluates as float. If you wish you can cast arguments to integer using int().

I finally got the solution to it..I initially tried replacing floating values to 3 and 2 for 3.5 and 3.24 respectively.But got error as these values reduce the total frame dimension.Then changed it to
np.zeros((h*3,w*3,3), dtype=np.uint8) and it works..!!