I am trying to implement deep dreaming for a sound processing neural network and keep running into issues related to the handling of symbolic tensors, which I cannot seem to circumvent.
I narrowed the issue down to the fact that when functions are decorated with #tf.function, tensors passed to them are handled as symbolic tensors, which do not seem to share the same attributes as "regular" tensors and cannot be casted into other classes.
Below is a minimum reproducible example of the aforementioned issue:
import numpy as np
import tensorflow as tf
print(np.__version__)
print(tf.__version__)
tf.compat.v1.enable_eager_execution()
x = np.random.rand(35000,1)
x_tensor = tf.convert_to_tensor(x)
#tf.function
def some_function(some_tensor):
batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
temp = some_tensor.numpy() # Error 2
temp = np.array(some_tensor) # Error 3
return(temp)
batched_x = tf.keras.preprocessing.timeseries_dataset_from_array(data=x_tensor, targets=None, sequence_length=256, sequence_stride=64)
temp = x_tensor.numpy()
temp = np.array(x_tensor)
temp = some_function(x_tensor)
Executing the code above throws the following error (Error 1):
1.19.5
2.4.1
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-11-61d46064abc7> in <module>()
22 temp = x_tensor.numpy()
23 temp = np.array(x_tensor)
---> 24 temp = some_function(x_tensor)
8 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
975 except Exception as e: # pylint:disable=broad-except
976 if hasattr(e, "ag_error_metadata"):
--> 977 raise e.ag_error_metadata.to_exception(e)
978 else:
979 raise
TypeError: in user code:
<ipython-input-11-61d46064abc7>:15 some_function *
batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/preprocessing/timeseries.py:141 timeseries_dataset_from_array **
if sampling_rate <= 0 or sampling_rate >= len(data):
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:860 __len__
"shape information.".format(self.name))
TypeError: len is not well defined for symbolic Tensors. (some_tensor:0) Please call `x.shape` rather than `len(x)` for shape information.
Furthermore, commenting out the code inside the function one by one yields following issues:
Error 2:
AttributeError: 'Tensor' object has no attribute 'numpy'
Error 3:
NotImplementedError: Cannot convert a symbolic Tensor (some_tensor:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
As you can see, the operations outside the function are perfectly executable but fail when performed on the symbolic tensor inside the function. I have seen this issue (or similar issues) being reported in different forums before but nowhere a satisfying solution was provided.
To compute the gradient ascend, I am dependent on batching the data using the tf.keras.preprocessing.timeseries_dataset_from_array before feeding it to the network to retrieve the activations. I am not looking to modify the network's architecture nor is passing the data preprocessed to the function an option for me. I feel like there should be a straight-forward way to manipulate symbolic tensors with tensorflow's built-in functions. Also, tf.make_ndarray() does not work.
Any help with either debugging the code or guiding me towards helpful references will be highly appreciated.
Thanks in advance!
I recommend you take a look at some articles that explain eager vs graph execution mode such as:
https://towardsdatascience.com/eager-execution-vs-graph-execution-which-is-better-38162ea4dbf6
Tensorflow has eager tensors which can be converted to numpy values and symbolic tensors which represent nodes in an execution graph.
When you decorate a function with #tf.function you are marking this function as a graph function. tensorflow will execute this function in python once in order to build a graph; this graph is then executed each time the model node is called; this execution is done via the tensorflow core library and can happen on CPU, GPU, etc... As such it is not meaningful to call .numpy() on a symbolic tensor as the graph executor does not have access to a python runtime.
You can however build models that mix eager and graph nodes... be aware that any eager nodes will be rather slow.
Eager tensors have their place when one is attempting to debug the math of a node; but in general, since execution speed is important, it is more useful to think in terms of graphs and symbolic tensors.
Related
I'm trying to train a model in mixed precision. However, I want a few of the layers to be in full precision for stability reasons. How do I force an individual layer to be float32 when using torch.autocast? In particular, I'd like for this to be onnx compileable.
Is it something like:
with torch.cuda.amp.autocast(enabled=False, dtype=torch.float32):
out = my_unstable_layer(inputs.float())
Edit:
Looks like this is indeed the official method. See the torch docs.
I think the motivation of torch.autocast is to automate the reduction of precision (not the increase).
If you have functions that need a particular dtype, you should consider using, custom_fwd
import torch
#torch.cuda.amp.custom_fwd(cast_inputs=torch.complex128)
def get_custom(x):
print(' Decorated function received', x.dtype)
def regular_func(x):
print(' Regular function received', x.dtype)
get_custom(x)
x = torch.tensor(0.0, dtype=torch.half, device='cuda')
with torch.cuda.amp.autocast(False):
print('autocast disabled')
regular_func(x)
with torch.cuda.amp.autocast(True):
print('autocast enabled')
regular_func(x)
autocast disabled
Regular function received torch.float16
Decorated function received torch.float16
autocast enabled
Regular function received torch.float16
Decorated function received torch.complex128
Edit: Using torchscript
I am not sure how much you can rely on this, due to a comment in the documentation. However the comment is apparently outdated.
Here is an example where I trace the model with autocast enabled, feeze it and then I use it and the value is indeed cast to the specified type
class Cast(torch.nn.Module):
#torch.cuda.amp.custom_fwd(cast_inputs=torch.float64)
def forward(self, x):
return x
with torch.cuda.amp.autocast(True):
model = torch.jit.trace(Cast().eval(), x)
model = torch.jit.freeze(model)
x = torch.tensor(0.0, dtype=torch.half, device='cuda')
print(model(x).dtype)
torch.float64
But I suggest you to validate this approach before using it for a serious application.
I can use the tf.make_tensor_proto in tensorflow2 to convert the tensor into ndarray like this way
action=tf.Variable([20.,10.,6.,9.],dtype=tf.float64)
bound=tf.Variable([10,20,5,10],dtype=tf.float64)
mask=tf.greater(bound, action )
proto_tensor=tf.make_tensor_proto(mask)
ndarry=tf.make_ndarray(proto_tensor)
print(ndarry)
and this can outpt[False True False True]with ndarry type
However, I want to do this in tensorflow1 with session and I try this way
#using sess.run must disable eager execution in tensorflow 2
tf.compat.v1.disable_eager_execution()
action=tf.Variable([20.,10.,6.,9.],dtype=tf.float64)
bound=tf.Variable([10,20,5,10],dtype=tf.float64)
sess = tf.compat.v1.Session()
mask = tf.greater(bound,action)
proto_tensor=tf.compat.v1.make_tensor_proto(mask)
ndarry=tf.compat.v1.make_ndarray(proto_tensor)
print(sess.run(ndarry))
it will have this error
Traceback (most recent call last):
File "C:\Users\stitch\Desktop\CG_metting0\gym-BSS-master\DDPG-With-EnvBSS\tensor_test_v2.py", line 407, in <module>
proto_tensor=tf.compat.v1.make_tensor_proto(mask)
File "C:\Users\stitch\anaconda\envs\CG_tensor2\lib\site-packages\tensorflow_core\python\framework\tensor_util.py", line 451, in make_tensor_proto
_AssertCompatible(values, dtype)
File "C:\Users\stitch\anaconda\envs\CG_tensor2\lib\site-packages\tensorflow_core\python\framework\tensor_util.py", line 328, in _AssertCompatible
raise TypeError("List of Tensors when single Tensor expected")
TypeError: List of Tensors when single Tensor expected
And I try case by case see that , if do not using session in v1(therefore we don't need to disable the eager execution), but in my study case, I need to have session in my code, is there anyway that I can change it to the ndarray under this function(make_tesensor_proto and make_ndarry in tensorflow) without error?
What was wrong: You cannot convert a tf.variable to a numpy array using graph execution and need to use a tf.constant instead. Also using tf.make_tensor_proto is pointless since sess.run converts the tensor into a numpy array for you.
Explanation: Convert your actions to TF constants instead of Variables or use Constants instead (not possible to convert variables). Then apply the same logic to your bound. Next, create the mask as you did previously. Finally, create a session and evaluate the mask in the session.
Here is the code to convert your mask into a numpy array using graph execution:
import tensorflow as tf
import numpy as np
graph = tf.Graph()
with graph.as_default():
action=tf.constant([20.,10.,6.,9.],dtype=tf.float64)
bound=tf.constant([10,20,5,10],dtype=tf.float64)
mask = tf.greater(bound,action)
with tf.compat.v1.Session() as sess:
result = sess.run(mask)
print(result)
Based on your situation I think this is the best way of doing it. But you could also use tf.py_func to create a function that would convert the mask into a numpy array.
While running a sentdex tutorial script of a cryptocurrency RNN, link here
YouTube Tutorial: Cryptocurrency-predicting RNN Model,
but have encountered an error when attempting to train the model. My tensorflow version is 2.0.0 and I'm running python 3.6. When attempting to train the model I receive the following error:
File "C:\python36-64\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 734, in fit
use_multiprocessing=use_multiprocessing)
File "C:\python36-64\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 224, in fit
distribution_strategy=strategy)
File "C:\python36-64\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 497, in _process_training_inputs
adapter_cls = data_adapter.select_data_adapter(x, y)
File "C:\python36-64\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py", line 628, in select_data_adapter
_type_name(x), _type_name(y)))
ValueError: Failed to find data adapter that can handle input: <class 'numpy.ndarray'>, (<class 'list'> containing values of types {"<class 'numpy.float64'>"})
Any advice would be greatly appreciated!
Have you checked whether your training/testing data and training/testing labels are all numpy arrays? It might be that you're mixing numpy arrays with lists.
You can avoid this error by converting your labels to arrays before calling model.fit():
train_x = np.asarray(train_x)
train_y = np.asarray(train_y)
validation_x = np.asarray(validation_x)
validation_y = np.asarray(validation_y)
If you encounter this problem while dealing with a custom generator inheriting from the keras.utils.Sequence class, you might have to make sure that you do not mix a Keras or a tensorflow - Keras-import.
This might especially happen when you have to switch to a previous tensorflow version for compatibility (like with cuDNN).
If you for example use this with a tensorflow-version > 2...
from keras.utils import Sequence
class generatorClass(Sequence):
def __init__(self, x_set, y_set, batch_size):
...
def __len__(self):
...
def __getitem__(self, idx):
return ...
... but you actually try to fit this generator in a tensorflow-version < 2, you have to make sure to import the Sequence-class from this version like:
keras = tf.compat.v1.keras
Sequence = keras.utils.Sequence
class generatorClass(Sequence):
...
I had a similar problem. In my case it was a problem that I was using a tf.keras.Sequential model but a keras generator.
Wrong:
from keras.preprocessing.sequence import TimeseriesGenerator
gen = TimeseriesGenerator(...)
Correct:
gen = tf.keras.preprocessing.sequence.TimeseriesGenerator(...)
This error occured when I updated tensorflow from 1.x to 2.x
It was solved after changing my import from
import keras
to
import tensorflow.keras as keras
For some reason I also experienced this problem when I passed my custom generator function directly to model.fit(), rather than creating an instance of it first.
I.e, given:
def batch_generator(...):
...
yield(...)
I called model.fit(batch_generator,...), rather than:
generator_instance = batch_generator(...)
model.fit(generator_instance, ...)
may be it will help someone.
First check your data type if it is numpy array & possibly ur algo required a DF.
print(X.shape, X.dtype)
print(y.shape, y.dtype)
convert your numpy array into Pandas DF
train_x = pd.DataFrame(train_x)
train_y = pd.DataFrame(train_y)
I got a very short question, which has probably a very simple answer but I just can't figure it out, although I tried for hours now.
I'm using Tensorflow Estimator and I want to access the global step within my model_fn. I've tried tf.train.get_global_step, which returns me a Tensor. I need the global_step as an integer though (or as a string)!
So I've tried to eval() (= tf.get_default_session().run(t)), but it doesn't work..
Cheers!
You can use tf.cast to cast the Tensor to int or string.
For example,
tf.cast(tf.train.get_global_step(), dtype=tf.int)
See the reference here.
One way would be to parse it from the latest checkpoint file in the model_dir.
So assuming you can pass the model_dir into the model_fn (either through the params argument of tf.estimator.Estimator(..., params={'model_dir': 'path/to/model_dir'}) or through tf.flags.FLAGS, you can then use this utility function:
import tensorflow as tf
def get_global_step_from_model_dir(model_dir):
latest_checkpoint_file = tf.train.latest_checkpoint(model_dir)
if latest_checkpoint_file is None:
return 0
else:
return int(os.path.basename(latest_checkpoint_file).split('-')[-1])
Is there a nice way to distinguish programmatically between tensors, variables, and ops in TensorFlow? This can come up, for example, when reloading a model and tf.local_variables() can have both tensors and variables in it. If you try to initialize a tensor, you get an error.
Below is some code for my current hack to get around this, but is there a better way? Part of the issue is that the type of variables, tensors, etc. is, e.g., tensorflow.python.ops.variables.Variable but it seems that tensorflow.python isn't accessible anymore (I think it was in some earlier releases?). The example only shows variables vs tensors, but I've also needed to distinguish ops from tensors before and had to use similar hacks.
import tensorflow as tf
vars_list = [tf.Variable(0), tf.constant(0)]
# init = tf.variables_initializer(vars_list) # -> AttributeError: 'Tensor' object has no attribute 'initializer'
var_type = type(tf.Variable(0))
init = tf.variables_initializer([v for v in vars_list if type(v) == var_type])
Normally, in Python, one would use
isinstance(x, tf.Variable)
or
isinstance(x, (tf.Variable, tf.Tensor))
etc.