I have realized that there is some funky stuff going on with the way Tensorflow seems to be managing graphs.
Since building (and rebuilding) models is so tedious, I decided to wrap my custom model in a class so I could easily re-instantiate it elsewhere.
When I was training and testing the code (in the original place) it would work fine, however in the code where I loaded the graph's variables I would get all sorts of weird errors - variable redefinitions and everything else. This (from my last question about a similar thing) was the hint that everything was being called twice.
After doing a TON of tracing, it came down to the way I was using the loaded code. It was being used from within a class that had a structure like so
class MyModelUser(object):
def forecast(self):
# .. build the model in the same way as in the training code
# load the model checkpoint
# call the "predict" function on the model
# manipulate the prediction and return it
And then in some code that uses MyModelUserI had
def test_the_model(self):
model_user = MyModelUser()
print(model_user.forecast()) # 1
print(model_user.forecast()) # 2
and I (obviously) expected to see two forecasts when this was called. Instead, the first forecast was called and worked as expected, but the second call threw a TON of variable reuse ValueError an example of one of these was:
ValueError: Variable weight_def/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
I managed to quell the errors by adding a series of try/except blocks that used get_variable to create the variable, and then on exception, called reuse_variables on the scope and then get_variable without anything but the name. This brought on a new set of nasty errors, one of which was:
tensorflow.python.framework.errors.NotFoundError: Tensor name "weight_def/weights/Adam_1" not found in checkpoint files
On a whim I said "what if I move the modeling building code to __init__ so its only built once?"
My new model user:
class MyModelUser(object):
def __init__(self):
# ... build the model in the same way as in the training code
# load the model checkpoint
def forecast(self):
# call the "predict" function on the model
# manipulate the prediction and return it
and now:
def test_the_model(self):
model_user = MyModelUser()
print(model_user.forecast()) # 1
print(model_user.forecast()) # 2
Works as expected, printing two forecasts with no errors. This leads me to believe I can also get rid of the variable reuse stuff.
My question is this:
Why did this fix it? In theory, the graph should be reinstanced every single time in the original predict method, so it shouldn't be creating more than one graph. Does Tensorflow persist the graph even after the function completes? Is this why moving the creation code to __init__ worked? This has left me hopelessly confused.
By default, TensorFlow uses a single global tf.Graph instance that is created when you first call a TensorFlow API. If you do not create a tf.Graph explicitly, all operations, tensors, and variables will be created in that default instance. This means that each call in your code to model_user.forecast() will be adding operations to the same global graph, which is somewhat wasteful.
There are (at least) two possible courses of action here:
The ideal action would be to restructure your code so that MyModelUser.__init__() constructs an entire tf.Graph with all of the operations needed to perform forecasting, and MyModelUser.forecast() simply performs sess.run() calls on the existing graph. Ideally, you would only create a single tf.Session as well, because TensorFlow caches information about the graph in the session, and the execution would be more efficient.
The less invasive—but probably less efficient—change would be to create a new tf.Graph for every call to MyModelUser.forecast(). It's unclear from the question how much state is created in the MyModelUser.__init__() method, but you could do something like the following to put the two calls in different graphs:
def test_the_model(self):
with tf.Graph(): # Create a local graph
model_user_1 = MyModelUser()
print(model_user_1.forecast())
with tf.Graph(): # Create another local graph
model_user_2 = MyModelUser()
print(model_user_2.forecast())
TF has a default graph that new operations etc get added to. When you call your function twice, you will add the same things twice to the same graph. So, either build the graph once and evaluate it multiple times (as you have done, which is also the "normal" approach), or, if you want to change things, you can use reset_default_graph https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#reset_default_graph to reset the graph in order to have a fresh state.
Related
I'm trying to modify a LSTM model implemented with Sonnet and Tensorflow. The idea is that the model should update itself whenever new results are received.
The problem is that Python classes are used and added automatically during its definition. This means that I don't have explicit access to them once the algorithm is running, and I cannot change them using methods like tf.variable or tf.method.
I've tried changing the values of the Python object directly, but it only affects the original instance of the class and not the one used in the model.
I understand that in order to modify a model during execution I need to create a new method, but I don't know where to create it or how to make it work for Sonnet.
Could anyone give some tip on how to solve this problem, or point me towards the right direction to find an answer?
I'm trying to implement a dive-and-fix algorithm in Gurobi. What I want to build is a function into which you put an optimized model last_model, make a deepcopy of this model called new_model and add certain constraints to new_model accoording to certain optimized values of last_model.
I found the function .copy() that would make a deepcopy for me. But I’m still having an awful time adding constraints to my copied new_model as I can’t in any way alter my constraints. (And yes, i am using last_model.update() before copying)
If I don’t do anything to my variables after new_model = last_model.copy() and tried to add a constant on z, it would tell me that Variable not in model.
I’ve tried .getVarByName(‘z’), which would tell me that z was a NoneType. (I found this on stackexchange)
I’ve tried new_model._data = last_model._data, which just returns that the function _data does not exist. (I found this on the gurobi support site)
I’ve tried .getVars which would just create a list and does not allow me to add any constraints on the actual variables.
You are on the right track with getVars() - you really do need to get the variables of the copied model again to be able to add new constraints. The variables of the original model are only going to refer to the original model, even if they may have the same name. Think of the variables as handles for the actual variables in the model - they can only refer to one model.
I'm using the tensorflow.data.Dataset api by tensorflow. However I need to create datasets on the fly filtering out elements other dataset. While training goes well and I can iterate over the training set and the dev set, when I reinitialize the iterator with a new dataset that I just created with a filter, i receive the following exception:
tensorflow.python.framework.errors_impl.NotFoundError: Function tf_predicate_5HKZIzWZBv8 is not defined.
I'm using the following function to create an initialiser out of a dataset:
self.iterator.make_initializer(dataset)
where self.iterator is defined as follow:
self.iterator = tf.data.Iterator.from_structure(ds_types, ds_shapes)
Do you guys have any idea about why this is happening? Note that it happens if I call make_initializer after I have created a session, run a dataset, and then create a new initializer. If after the creation I also recreate the Session everything works (except the fact that all the variables have to be reinitialized)
I found the solution and I'm sharing in case somebody will run into this problem. The thing is that, as I'm defining a new dataset after the session has been initialised it, it doesn't have the new operation I'm adding for the new dataset (In this case I'm using a new filter everytime I create a new dataset) and that's why the session can't find the operation. To overcome the problem I defined all the datasets I needed to use before the session is initialised and I used a filter that takes as input a placeholder so that I always use the same filter feeded everytime at iterator init time with the right value.
Let's say we have some method foo we call during graph construction time that returns some tf.Tensors or a nested structure of them every time is called, and multiple other methods that make use of foo's result. For efficiency and to avoid spamming the TF graph with unnecessary repeated operations, it might be tempting to make foo cache its result (to reuse the subgraph it produces) the first time is called. However, that will fail if foo is ever used in the context of a control flow, like tf.cond, tf.map_fn or tf.while_loop.
My questions are:
When is it safe to cache tf.Tensor objects in such a way that does not cause problems with control flows? Perhaps is there some way to retrieve the control flow under which a tf.Tensor was created (if any), store it and compare it later to see if a cached result can be reused?
How would the answer to the question above apply to tf.Operations?
(Question text updated to make clearer that foo creates a new set of tensors every time is called)
TL;DR: TF already caches what it needs to, don't bother with it yourself.
Every time you call sess.run([some_tensors]) TF's engine find the minimum subgraph needed to compute all tensors in [some_tensors] and runs it from top to bottom (possibly on new data, if you're not feeding it the same data).
That means, caching of results in-between sess.run calls is useless towards saving computation, because they will be recomputed anyway.
If, instead, you're concerned with having multiple tensors using the same data as input in one call of sess.run, don't worry, TF is smart enough. if you have input A and B = 2*A, C = A + 1, as long as you do one sess.run call as sess.run([B,C]) A will be evaluated only once (and then implicitly cached by the TF engine).
I am using object detection in tensorflow api. In my previous work I used to check the current step and save my model every n steps, something like the approach mentioned here.
In this case though the authors use TensorFlow-Slim to perform the training. So, they use a tf.train.Saver which is passed to the actual function which performs the training: slim.learning.train(). While this function has some parameters regarding the interval for writing down the training model using the parameter save_interval_secs it is time dependent and not step dependent.
So, since tf.train.Saver is a "passive" utility as mentioned here and just saves a model with the provided parameters, meaning is ignorant of any time or step notions, and also in object detection code the control is passed in TensorFlow-Slim, by passing the saver as a parameter, in this case how can I achieve to save my model step-ward (every n steps instead of every x seconds)?
The only solution is to dig into slim code and edit it (with all risks coming from this)? Or is there another option I am not familiar with?
P.S.1
I found out there is a astonishingly similar question about this option here but unfortunately it did not have any answers. So, since my problem persists I will leave this question intact to raise some interest in the issue.
P.S.2
Looking into slim.learning code I found out that in train() after passing the parameters it just passes the control over to supervisor.Supervisor which refers to tf.train.Supervisor which is a bit odd since this class is considered deprecated. The use of supervisor is also mentioned in the docstrings of slim.learning.