I have a model that is served using TorchServe. I'm communicating with the TorchServe server using gRPC. The final postprocess method of the custom handler defined returns a list which is converted into bytes for transfer over the network.
The post process method
def postprocess(self, data):
# data type - torch.Tensor
# data shape - [1, 17, 80, 64] and data dtype - torch.float32
return data.tolist()
The main issue is at the client where converting the received bytes from TorchServe to a torch Tensor is inefficiently done via ast.literal_eval
# This takes 0.3 seconds
response = self.inference_stub.Predictions(
inference_pb2.PredictionsRequest(model_name=model_name, input=input_data))
# This takes 0.84 seconds
predictions = torch.as_tensor(literal_eval(
response.prediction.decode('utf-8')))
Using numpy.frombuffer or torch.frombuffer return the following error.
import numpy as np
np.frombuffer(response.prediction)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: buffer size must be a multiple of element size
np.frombuffer(response.prediction, dtype=np.float32)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: buffer size must be a multiple of element size
Using torch
import torch
torch.frombuffer(response.prediction, dtype = torch.float32)
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: buffer length (2601542 bytes) after offset (0 bytes) must be a multiple of element size (4)
Is there an alternative, more efficient solution of converting the received bytes into torch.Tensor?
One hack I've found that has significantly increased the performance while sending large tensors is to return a list of json.
In your handler's postprocess function:
def postprocess(self, data):
output_data = {}
output_data['data'] = data.tolist()
return [output_data]
At the clients side when you receive the grpc response, decode it using json.loads
response = self.inference_stub.Predictions(
inference_pb2.PredictionsRequest(model_name=model_name, input=input_data))
decoded_output = response.prediction.decode('utf-8')
preds = torch.as_tensor(json.loads(decoded_output))
preds should have the output tensor
Update:
There's an even faster method and should completely solve the bottleneck. Use tf.io.serialize_tensor from tensorflow to serialize your tensor inside postprocess
def postprocess(self, data):
return [tf.io.serialize_tensor(data.cpu()).numpy()]
Decode it using tf.io.parse_tensor
response = self.inference_stub.Predictions(
inference_pb2.PredictionsRequest(model_name=model_name, input=input_data))
prediction = response.prediction
torch.as_tensor(tf.io.parse_tensor(prediction, out_type=tf.float32).numpy())
Related
I'm trying to create a tensorflow dataset from 6500 .npy files of shape [256,256].
My previous method (for less files) is to load them and stack them into an np.array, and the use tf.data.Dataset.from_tensor_slices((stacked_data)).
With the current number of files I get ValueError: Cannot create a tensor proto whose content is larger than 2GB.
I'm now trying the following:
def data_generator():
processed = []
for i in range(len(onlyfiles)):
processed.append(tf.convert_to_tensor(np.load(onlyfiles[i], mmap_mode='r')))
yield iter(tf.concat(processed, 0))
_dataset = tf.data.Dataset.from_generator(generator=data_generator,output_types=tf.float32)
onlyfiles is the list of the filenames
I get multiple errors, one of which is the following:
2022-10-01 11:25:44.602505: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: TypeError: `generator` yielded an element that could not be converted to the expected type. The expected type was float32, but the yielded element was <generator object Tensor.__iter__ at 0x7fe6d7d506d0>.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 653, in generator_py_func
ret_arrays.append(script_ops.FuncRegistry._convert( # pylint: disable=protected-access
File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/ops/script_ops.py", line 195, in _convert
result = np.asarray(value, dtype=dtype, order="C")
TypeError: float() argument must be a string or a number, not 'generator'
What should I change? Is there another method to do it?
Because I created the dataset, is there a better way to prepare it for the Tensorflow implementation?
After a few days, I found this solution. I don't know how good it it, but I'll post it just in case someone finds it useful:
#tf.function
def input_fn():
tf.compat.v1.enable_eager_execution()
mypath = 'tensorflow_datasets/Dataset_1/'
list_of_file_names = [join(mypath, f) for f in listdir(mypath) if isfile(join(mypath, f))]
def gen():
for i in itertools.count(1):
data1 = np.load(list_of_file_names[i%len(list_of_file_names)])
data2 = np.where(data1 > 1, data1, 1)
yield tf.convert_to_tensor(np.where(data2>0, 20*np.log10(data2), 0))
dataset = tf.data.Dataset.from_generator(gen, (tf.float32))
return dataset.make_one_shot_iterator().get_next()
I usually do such things as follows
dataset = tf.data.Dataset.from_tensor_slices(list_of_file_names)
# Optional
dataset = dataset.repeat().shuffle(...)
def read_file(file_name):
full_path_to_image_file = ... # build full path
buffer = tf.io.read_file(full_path_to_image_file)
tensor = ... # converte from buffer to tensor
return tensor
dataset = dataset.map(read_file, num_parallel_calls=...)
As an option you can read file with np.load inside py_function (use decode ("utf-8") to convert byte string to ordinary python string) like
def read_file(file_path):
tensor = tf.py_function(
func=lambda path: np.load(path.numpy().decode("utf-8")),
inp=[file_path],
Tout=tf.float32
)
tensor.set_shape(img_shape)
return tensor
I am working on migrating some working Pytorch code I found online (which is a 2D image classification example using the MNIST data; apologies that I lost track of the original source and am unable to find it) to what I need, which is converting a 1D collection of values into a numerical score. I created my own Dataset class. When I call model(), I get an error: RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm. My first level of confusion is that I can't find any reference to Python even having a Double datatype. And my second is why I get the error--when I put in debug code to show the datatype of mat1 and its elements, I am told that it is a Tensor which claims to be float64. I also wonder why it is expecting a scalar for mat1, which the documentation describes as a matrix/tensor.
The full error dump is
Traceback (most recent call last):
File "mlalan.py", line 174, in <module>
outputs = model(images)
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "mlalan.py", line 80, in forward
x = activate(self.fc1(x))
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm
Some of the key code from my Dataset class is
class RandomDataset(Dataset):
def __init__(self, csv_file, transform=None):
self.data_frame = pd.read_csv(csv_file, dtype=float)
def __getitem__(self, idx):
raw = self.data_frame.values[idx]
sample = raw[0:6], raw[6:8]
return sample
The full source code is at http://8wheels.org/mlalan.py.
By default, in Python, float means float32. However, in Pandas and Numpy, float means float64. I was able to resolve the problem by adding a call to astype as below. The "32" is required for it to work.
raw = self.data_frame.values[idx].astype(np.float32)
Thanks!
Now I can move on to the next crash :-)
I got TypeError: expected torch.LongTensor (got torch.cuda.FloatTensor).
How do I convert torch.cuda.FloatTensor to torch.LongTensor?
Traceback (most recent call last):
File "train_v2.py", line 110, in <module>
main()
File "train_v2.py", line 81, in main
model.update(batch)
File "/home/Desktop/squad_vteam/src/model.py", line 131, in update
loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
File "/home/Desktop/squad_vteam/src/model.py", line 94, in adversarial_loss
adv_embedding = torch.LongTensor(adv_embedding)
TypeError: expected torch.LongTensor (got torch.cuda.FloatTensor)
You have a float tensor f and want to convert it to long, you do long_tensor = f.long()
You have cuda tensor i.e data is on gpu and want to move it to cpu you can do cuda_tensor.cpu().
So to convert a torch.cuda.Float tensor A to torch.long do A.long().cpu()
Best practice for Pytorch 0.4.0 is to write device agnostic code: That is, instead of using .cuda() or .cpu() you can simply use .to(torch.device("cpu"))
A = A.to(dtype=torch.long, device=torch.device("cpu"))
Note that .to() is not an "in-place" operation (see, e.g., this answer), thus you need to assign A.to(...) back into A.
If you have a tensor t.
t = t.cpu()
would be the old way.
t = t.to("cpu")
would be the new API.
Reading in a binary file and XORing bytes to find out if there is an AES 128 key present.
with open("FakeMemDump1.bin","rb") as memorydump:
mem = memorydump.read(memorybytes)
for i in mem:
w = mem[i]^mem[i+15]
x = mem[i+4]^mem[i+19]
y = mem[i+8]^mem[i+23]
z = mem[i+12]^mem[i+27]
if w==mem[i+16] and x==mem[i+20] and y==mem[i+24] and z==mem[i+28]:
keySched = mem[i-1:i+175]
print ("Key Schedule : "%keySched)
Traceback (most recent call last):
File "AES128Keyfind.py", line 8, in <module>
w = mem[i]^mem[i+15]
TypeError: string indices must be integers, not str
Above is the error I am receiving, is the byte being called passed in as a string even though the file is read as a binary file?
I've got a scipy sparse matrix (csr:Compressed Sparse Row matrix). I'd like to use Orange's feature selection methods (Orange.feature.scoring.score_all (InfoGain/MDL)). However, from my understanding I'll have to create a Table which only accepts a numpy array as an arguments. Therefore, whenever I tried to convert the csr matrix to an array, using (.toarray()), I get the following error (because the size of the matrix):
Traceback (most recent call last):
File "C:\Users\NMS\Desktop\PyExp\experiments_acl2013.py", line 249, in <module>
print(X_train.toarray())
File "C:\Python27\lib\site-packages\scipy\sparse\compressed.py", line 561, in toarray
return self.tocoo(copy=False).toarray(order=order, out=out)
File "C:\Python27\lib\site-packages\scipy\sparse\coo.py", line 238, in toarray
B = self._process_toarray_args(order, out)
File "C:\Python27\lib\site-packages\scipy\sparse\base.py", line 635, in _process_toarray_args
return np.zeros(self.shape, dtype=self.dtype, order=order)
ValueError: array is too big.
Is there another approach that can allow me to pass a sparse matrix to create a table?
OR
Is there a way to apply InfoGain or MDL, in Orange, without creating a table using my sparse matrix directly?
when passing memmap to Table I get the following error:
>>> t2 = Table(d2, mm)
Traceback (most recent call last):
File "<pyshell#125>", line 1, in <module>
t2 = Table(d2, mm)
TypeError: invalid arguments
When passing the memmap with out the domain I get the following:
>>> mm
memmap([[0, 1, 2, 4],
[9, 8, 6, 3]])
>>> t2 = Table(mm)
Traceback (most recent call last):
File "<pyshell#128>", line 1, in <module>
t2 = Table(mm)
TypeError: invalid arguments for constructor (domain or examples or both expected)
Here it goes a workaround. For a given coo_matrix called m (obtained with m.tocoo()):
1) create a numpy.memmap array for writing:
mm = np.memmap('test.memmap', mode='w+', dtype=m.dtype, shape=m.shape)
2) copy the data to the memmap array, which should work:
for i,j,v in zip(m.row, m.col, m.data):
mm[i,j] = v
3) You can access the memmap as detailed in the documentation...