I'm using the tensorflow.data.Dataset api by tensorflow. However I need to create datasets on the fly filtering out elements other dataset. While training goes well and I can iterate over the training set and the dev set, when I reinitialize the iterator with a new dataset that I just created with a filter, i receive the following exception:
tensorflow.python.framework.errors_impl.NotFoundError: Function tf_predicate_5HKZIzWZBv8 is not defined.
I'm using the following function to create an initialiser out of a dataset:
self.iterator.make_initializer(dataset)
where self.iterator is defined as follow:
self.iterator = tf.data.Iterator.from_structure(ds_types, ds_shapes)
Do you guys have any idea about why this is happening? Note that it happens if I call make_initializer after I have created a session, run a dataset, and then create a new initializer. If after the creation I also recreate the Session everything works (except the fact that all the variables have to be reinitialized)
I found the solution and I'm sharing in case somebody will run into this problem. The thing is that, as I'm defining a new dataset after the session has been initialised it, it doesn't have the new operation I'm adding for the new dataset (In this case I'm using a new filter everytime I create a new dataset) and that's why the session can't find the operation. To overcome the problem I defined all the datasets I needed to use before the session is initialised and I used a filter that takes as input a placeholder so that I always use the same filter feeded everytime at iterator init time with the right value.
Related
I'm trying to modify a LSTM model implemented with Sonnet and Tensorflow. The idea is that the model should update itself whenever new results are received.
The problem is that Python classes are used and added automatically during its definition. This means that I don't have explicit access to them once the algorithm is running, and I cannot change them using methods like tf.variable or tf.method.
I've tried changing the values of the Python object directly, but it only affects the original instance of the class and not the one used in the model.
I understand that in order to modify a model during execution I need to create a new method, but I don't know where to create it or how to make it work for Sonnet.
Could anyone give some tip on how to solve this problem, or point me towards the right direction to find an answer?
I am currently trying to use keras to predict ... stuffs. I am using a HDF5 file as input. The file contains 195 objects, each one is a list of matrices with one attribute. I would like keras to learn on the list of matrices and predict the attribute. But here is the issue, so far I have seen that one object can only be assigned to one variable. That would be meaningless in my case.
I would like to know whether or not it was possible to load all of these objects at once, say under one variable, in keras to predict the attribute ? For instance here are some objects,
['10gs', '1a30', '1bcu',..., '4tmn']
I know I can assign one variable to one object,
dataset=infile['1a30']
However I am not sure how to assign several objects to one variable? Do I need to create a list of objects ? Here's what I am trying to get,
dataset=infile['all of my objects'].
In fine, I will be using it in keras but I am not too sure whether it is necessary as it seems to me it is a HDF file issue (misunderstanding).
In TensorFlow I have a graph that has in it a string_input_producer that is used in it's input pipeline.
I am loading the graph from a checkpoint file, so I do not have access to the original object when it was created. Nonetheless, I need to run the enqueue method of this object.
I have tried getting the FIFOQueue object using get_operation_by_name and get_tensor_by_name, which obviously did not work because the queue is neither an operation nor a tensor. Is there any function like the mentioned that would do what I want? (fake e.g. get_queue_by_name) How can I solve my problem otherwise?
I think that as of today that is not possible. However, you can call methods of the queue object by getting them using get_operation_by_name. For example get_operation_by_name("queue_name/enqueue"). You can also make your own Queue object which internally simply calls the corresponding graph operations.
I have a model where I need to assign to the weights (trainable variables) new external values every N iterations.
I can think of a few solutions:
Save and restore
Not good as I would need to serialization, go through a file system calls, etc. (even if I use something like tmpfs)
Using placeholders and assign operations
I would create a placeholder and assign op for each trainable variable. Everytime I want to assign something to the weights, I ran the assign ops.
However, I understand that this means I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph.
In addition I would be using much more memory than necessary..
Use a feed_dict for trainable variable and trigger ops that assign each variable to itself?
Does this work? Is there any drawback?
Before coding something I thought it was a good idea to ask?
What is the recommended way to assign new external values to variables efficiently (memory/timewise)?
Your 3-rd option sounds like the best one.
You can feed values to tensors that aren’t placeholders.
TensorFlow's feed mechanism lets you inject data into any Tensor in a
computation graph. A python computation can thus feed data directly
into the graph.
Any tensors that are feedable can be fed. To check if a tensor is feedable or not, use: tf.Graph.is_feedable(tensor).
In recent versions of Tensorflow Variable class has load method. It does exactly what you want.
https://www.tensorflow.org/api_docs/python/tf/Variable#load
You can use the assign operations with placeholders.
I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph
In addition I would be using much more memory than necessary..
No. You would only need to feed values to the placeholders when you run the assign operations. Don't make the assign operation part of your training graph and only run them when you want to assign new values.
If the assigning turns out to be a bottleneck (for small N it might slow down your program) you can consider other methods of getting data into TensorFlow.
I have realized that there is some funky stuff going on with the way Tensorflow seems to be managing graphs.
Since building (and rebuilding) models is so tedious, I decided to wrap my custom model in a class so I could easily re-instantiate it elsewhere.
When I was training and testing the code (in the original place) it would work fine, however in the code where I loaded the graph's variables I would get all sorts of weird errors - variable redefinitions and everything else. This (from my last question about a similar thing) was the hint that everything was being called twice.
After doing a TON of tracing, it came down to the way I was using the loaded code. It was being used from within a class that had a structure like so
class MyModelUser(object):
def forecast(self):
# .. build the model in the same way as in the training code
# load the model checkpoint
# call the "predict" function on the model
# manipulate the prediction and return it
And then in some code that uses MyModelUserI had
def test_the_model(self):
model_user = MyModelUser()
print(model_user.forecast()) # 1
print(model_user.forecast()) # 2
and I (obviously) expected to see two forecasts when this was called. Instead, the first forecast was called and worked as expected, but the second call threw a TON of variable reuse ValueError an example of one of these was:
ValueError: Variable weight_def/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
I managed to quell the errors by adding a series of try/except blocks that used get_variable to create the variable, and then on exception, called reuse_variables on the scope and then get_variable without anything but the name. This brought on a new set of nasty errors, one of which was:
tensorflow.python.framework.errors.NotFoundError: Tensor name "weight_def/weights/Adam_1" not found in checkpoint files
On a whim I said "what if I move the modeling building code to __init__ so its only built once?"
My new model user:
class MyModelUser(object):
def __init__(self):
# ... build the model in the same way as in the training code
# load the model checkpoint
def forecast(self):
# call the "predict" function on the model
# manipulate the prediction and return it
and now:
def test_the_model(self):
model_user = MyModelUser()
print(model_user.forecast()) # 1
print(model_user.forecast()) # 2
Works as expected, printing two forecasts with no errors. This leads me to believe I can also get rid of the variable reuse stuff.
My question is this:
Why did this fix it? In theory, the graph should be reinstanced every single time in the original predict method, so it shouldn't be creating more than one graph. Does Tensorflow persist the graph even after the function completes? Is this why moving the creation code to __init__ worked? This has left me hopelessly confused.
By default, TensorFlow uses a single global tf.Graph instance that is created when you first call a TensorFlow API. If you do not create a tf.Graph explicitly, all operations, tensors, and variables will be created in that default instance. This means that each call in your code to model_user.forecast() will be adding operations to the same global graph, which is somewhat wasteful.
There are (at least) two possible courses of action here:
The ideal action would be to restructure your code so that MyModelUser.__init__() constructs an entire tf.Graph with all of the operations needed to perform forecasting, and MyModelUser.forecast() simply performs sess.run() calls on the existing graph. Ideally, you would only create a single tf.Session as well, because TensorFlow caches information about the graph in the session, and the execution would be more efficient.
The less invasive—but probably less efficient—change would be to create a new tf.Graph for every call to MyModelUser.forecast(). It's unclear from the question how much state is created in the MyModelUser.__init__() method, but you could do something like the following to put the two calls in different graphs:
def test_the_model(self):
with tf.Graph(): # Create a local graph
model_user_1 = MyModelUser()
print(model_user_1.forecast())
with tf.Graph(): # Create another local graph
model_user_2 = MyModelUser()
print(model_user_2.forecast())
TF has a default graph that new operations etc get added to. When you call your function twice, you will add the same things twice to the same graph. So, either build the graph once and evaluate it multiple times (as you have done, which is also the "normal" approach), or, if you want to change things, you can use reset_default_graph https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#reset_default_graph to reset the graph in order to have a fresh state.