Is there a nice way to distinguish programmatically between tensors, variables, and ops in TensorFlow? This can come up, for example, when reloading a model and tf.local_variables() can have both tensors and variables in it. If you try to initialize a tensor, you get an error.
Below is some code for my current hack to get around this, but is there a better way? Part of the issue is that the type of variables, tensors, etc. is, e.g., tensorflow.python.ops.variables.Variable but it seems that tensorflow.python isn't accessible anymore (I think it was in some earlier releases?). The example only shows variables vs tensors, but I've also needed to distinguish ops from tensors before and had to use similar hacks.
import tensorflow as tf
vars_list = [tf.Variable(0), tf.constant(0)]
# init = tf.variables_initializer(vars_list) # -> AttributeError: 'Tensor' object has no attribute 'initializer'
var_type = type(tf.Variable(0))
init = tf.variables_initializer([v for v in vars_list if type(v) == var_type])
Normally, in Python, one would use
isinstance(x, tf.Variable)
or
isinstance(x, (tf.Variable, tf.Tensor))
etc.
Related
I am trying to implement deep dreaming for a sound processing neural network and keep running into issues related to the handling of symbolic tensors, which I cannot seem to circumvent.
I narrowed the issue down to the fact that when functions are decorated with #tf.function, tensors passed to them are handled as symbolic tensors, which do not seem to share the same attributes as "regular" tensors and cannot be casted into other classes.
Below is a minimum reproducible example of the aforementioned issue:
import numpy as np
import tensorflow as tf
print(np.__version__)
print(tf.__version__)
tf.compat.v1.enable_eager_execution()
x = np.random.rand(35000,1)
x_tensor = tf.convert_to_tensor(x)
#tf.function
def some_function(some_tensor):
batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
temp = some_tensor.numpy() # Error 2
temp = np.array(some_tensor) # Error 3
return(temp)
batched_x = tf.keras.preprocessing.timeseries_dataset_from_array(data=x_tensor, targets=None, sequence_length=256, sequence_stride=64)
temp = x_tensor.numpy()
temp = np.array(x_tensor)
temp = some_function(x_tensor)
Executing the code above throws the following error (Error 1):
1.19.5
2.4.1
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-11-61d46064abc7> in <module>()
22 temp = x_tensor.numpy()
23 temp = np.array(x_tensor)
---> 24 temp = some_function(x_tensor)
8 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
975 except Exception as e: # pylint:disable=broad-except
976 if hasattr(e, "ag_error_metadata"):
--> 977 raise e.ag_error_metadata.to_exception(e)
978 else:
979 raise
TypeError: in user code:
<ipython-input-11-61d46064abc7>:15 some_function *
batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/preprocessing/timeseries.py:141 timeseries_dataset_from_array **
if sampling_rate <= 0 or sampling_rate >= len(data):
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:860 __len__
"shape information.".format(self.name))
TypeError: len is not well defined for symbolic Tensors. (some_tensor:0) Please call `x.shape` rather than `len(x)` for shape information.
Furthermore, commenting out the code inside the function one by one yields following issues:
Error 2:
AttributeError: 'Tensor' object has no attribute 'numpy'
Error 3:
NotImplementedError: Cannot convert a symbolic Tensor (some_tensor:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
As you can see, the operations outside the function are perfectly executable but fail when performed on the symbolic tensor inside the function. I have seen this issue (or similar issues) being reported in different forums before but nowhere a satisfying solution was provided.
To compute the gradient ascend, I am dependent on batching the data using the tf.keras.preprocessing.timeseries_dataset_from_array before feeding it to the network to retrieve the activations. I am not looking to modify the network's architecture nor is passing the data preprocessed to the function an option for me. I feel like there should be a straight-forward way to manipulate symbolic tensors with tensorflow's built-in functions. Also, tf.make_ndarray() does not work.
Any help with either debugging the code or guiding me towards helpful references will be highly appreciated.
Thanks in advance!
I recommend you take a look at some articles that explain eager vs graph execution mode such as:
https://towardsdatascience.com/eager-execution-vs-graph-execution-which-is-better-38162ea4dbf6
Tensorflow has eager tensors which can be converted to numpy values and symbolic tensors which represent nodes in an execution graph.
When you decorate a function with #tf.function you are marking this function as a graph function. tensorflow will execute this function in python once in order to build a graph; this graph is then executed each time the model node is called; this execution is done via the tensorflow core library and can happen on CPU, GPU, etc... As such it is not meaningful to call .numpy() on a symbolic tensor as the graph executor does not have access to a python runtime.
You can however build models that mix eager and graph nodes... be aware that any eager nodes will be rather slow.
Eager tensors have their place when one is attempting to debug the math of a node; but in general, since execution speed is important, it is more useful to think in terms of graphs and symbolic tensors.
I would like to have a PyTorch sub-class of Module that keeps sub-modules in a list (because there may be a variable number of sub-modules depending on the constructor's arguments). I set this list in the following way:
self.hidden_layers = [torch.nn.Linear(i, o) for i, o in pairwise(self.layer_sizes)]
According to this and this question, a submodule is only registered by __setattr__, when a Module object is assigned to an attribute of self. Because hidden_layers is not assigned an object of type Module, the submodules in the list are not registered as submodules, and as a result self.parameters() does not iterate over the submodules' parameters.
I suppose I could explicitly call __subattr__ for each element of the list but that would be quite ugly. Is there a more correct way to register a submodule that is not a direct attribute of Module?
Use nn.ModuleList.
self.hidden_layers = nn.ModuleList([torch.nn.Linear(i, o) for i, o in pairwise(self.layer_sizes)])
As answered nn.ModuleList is what you want.
What you can also use is nn.Sequential. You can create a list of layers and then combine them via nn.Sequential, which will just act as a wrapper and combines all layers to essential one layer/module. This has the advantage that you only need one call to forward it through all the layers, which is nice if you have a dynamic count of modules, so you don't have to write the loops on your own.
One example would be in the pytorch ResNet code: https://github.com/pytorch/vision/blob/497744b9d510ff2df756f479ee5a19fce0d579b6/torchvision/models/resnet.py#L177
I have a Variable defined somewhere in the code, and I don't have control over its creation parameters. I want to change its initializer after it has been instanciated. However the straightforward approach doesn't work:
import tensorflow as tf
foo = tf.get_variable('foo', shape=(), initializer=tf.initializers.constant(0))
foo.initializer = tf.initializers.constant(1)
# AttributeError: can't set attribute
Why is that, and what is the work-around?
I use TF 1.14.
It turns our that you can change the initializer via the protected Variable._initializer_op member.
However the initializer op is the concrete operator that assigns a value to the Variable — it is not an tf.keras.initializer.Initializer or one of its children that one provides as the initializer argument of Variable:
import tensorflow as tf
foo = tf.get_variable('foo', shape=(), initializer=tf.initializers.constant(0))
foo._initializer_op = foo.assign(tf.ones_like(foo))
with tf.Session() as sess:
tf.global_variables_initializer().run()
print(foo.eval())
# 1.0
This inconsistency in the API could be one of the reasons why the _initializer_op is a protected member rather than a public one.
I'm looking for a way to change floatx in keras directly in python.
floatx is the default float type (float16, float32 . . .)
The config is stored in a json file at:
$HOME/.keras/keras.json
But I'm looking for a way to change the config inside my python programm without changing the config file itself.
There is a similiar question, in which somebody ask the same for changing the backend, which is also stored in keras.json.
The accepted answer involves setting the environment variable KERAS_BACKEND and reload the keras module, but I didn't find a similar environment variable for floatx.
Turns out keras.backend has function for setting and retrieving the floatx value (scroll down in the link):
keras.backend.floatx()
>>> 'float32'
keras.backend.set_floatx('float16')
keras.backend.floatx()
>>> 'float16'
Also you are not allowed to reload the keras module after using set_floatx like when changing backend, because then keras will simply reread the config file and return to its previous value:
keras.backend.floatx()
>>> 'float32'
keras.backend.set_floatx('float16')
keras.backend.floatx()
>>> 'float16'
importlib.reload(keras.backend)
keras.backend.floatx()
>>> 'float32'
Well, the floatx var should certainly be used in keras.json, as described in documentation.
The least buggy way to do it is using the file indeed and reloading the module.
Using K.set_floatx, at least for me, left parts of the models unchanged (even if sef_floatx was the very first thing I did after loading the keras model in a new python kernel)
Even though, I faced yet another bug when setting precision to float16: all my loss functions very quickly became nan. Unfortunately I had to go back to float32 (the default) to have the possibility of training.
I got a very short question, which has probably a very simple answer but I just can't figure it out, although I tried for hours now.
I'm using Tensorflow Estimator and I want to access the global step within my model_fn. I've tried tf.train.get_global_step, which returns me a Tensor. I need the global_step as an integer though (or as a string)!
So I've tried to eval() (= tf.get_default_session().run(t)), but it doesn't work..
Cheers!
You can use tf.cast to cast the Tensor to int or string.
For example,
tf.cast(tf.train.get_global_step(), dtype=tf.int)
See the reference here.
One way would be to parse it from the latest checkpoint file in the model_dir.
So assuming you can pass the model_dir into the model_fn (either through the params argument of tf.estimator.Estimator(..., params={'model_dir': 'path/to/model_dir'}) or through tf.flags.FLAGS, you can then use this utility function:
import tensorflow as tf
def get_global_step_from_model_dir(model_dir):
latest_checkpoint_file = tf.train.latest_checkpoint(model_dir)
if latest_checkpoint_file is None:
return 0
else:
return int(os.path.basename(latest_checkpoint_file).split('-')[-1])