We Keep Coding

Changing model parameters in TensorFlow and Sonnet

I'm trying to modify a LSTM model implemented with Sonnet and Tensorflow. The idea is that the model should update itself whenever new results are received.
The problem is that Python classes are used and added automatically during its definition. This means that I don't have explicit access to them once the algorithm is running, and I cannot change them using methods like tf.variable or tf.method.
I've tried changing the values of the Python object directly, but it only affects the original instance of the class and not the one used in the model.
I understand that in order to modify a model during execution I need to create a new method, but I don't know where to create it or how to make it work for Sonnet.
Could anyone give some tip on how to solve this problem, or point me towards the right direction to find an answer?

How to make a copy of a Gurobi model and add constraints to the copy in Python?

I'm trying to implement a dive-and-fix algorithm in Gurobi. What I want to build is a function into which you put an optimized model last_model, make a deepcopy of this model called new_model and add certain constraints to new_model accoording to certain optimized values of last_model.
I found the function .copy() that would make a deepcopy for me. But I’m still having an awful time adding constraints to my copied new_model as I can’t in any way alter my constraints. (And yes, i am using last_model.update() before copying)
If I don’t do anything to my variables after new_model = last_model.copy() and tried to add a constant on z, it would tell me that Variable not in model.
I’ve tried .getVarByName(‘z’), which would tell me that z was a NoneType. (I found this on stackexchange)
I’ve tried new_model._data = last_model._data, which just returns that the function _data does not exist. (I found this on the gurobi support site)
I’ve tried .getVars which would just create a list and does not allow me to add any constraints on the actual variables.

You are on the right track with getVars() - you really do need to get the variables of the copied model again to be able to add new constraints. The variables of the original model are only going to refer to the original model, even if they may have the same name. Think of the variables as handles for the actual variables in the model - they can only refer to one model.

Sporadically assigning values to variables (placeholders vs feed_dict)

I have a model where I need to assign to the weights (trainable variables) new external values every N iterations.
I can think of a few solutions:
Save and restore
Not good as I would need to serialization, go through a file system calls, etc. (even if I use something like tmpfs)
Using placeholders and assign operations
I would create a placeholder and assign op for each trainable variable. Everytime I want to assign something to the weights, I ran the assign ops.
However, I understand that this means I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph.
In addition I would be using much more memory than necessary..
Use a feed_dict for trainable variable and trigger ops that assign each variable to itself?
Does this work? Is there any drawback?
Before coding something I thought it was a good idea to ask?
What is the recommended way to assign new external values to variables efficiently (memory/timewise)?

Your 3-rd option sounds like the best one.
You can feed values to tensors that aren’t placeholders.
TensorFlow's feed mechanism lets you inject data into any Tensor in a
computation graph. A python computation can thus feed data directly
into the graph.
Any tensors that are feedable can be fed. To check if a tensor is feedable or not, use: tf.Graph.is_feedable(tensor).

In recent versions of Tensorflow Variable class has load method. It does exactly what you want.
https://www.tensorflow.org/api_docs/python/tf/Variable#load

You can use the assign operations with placeholders.
I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph
In addition I would be using much more memory than necessary..
No. You would only need to feed values to the placeholders when you run the assign operations. Don't make the assign operation part of your training graph and only run them when you want to assign new values.
If the assigning turns out to be a bottleneck (for small N it might slow down your program) you can consider other methods of getting data into TensorFlow.

Dynamically modify layer's parameters in Caffe

I use the follwoing code to load the net and set up it, the parameters of layers are stored in deploy.prototxt.
net = caffe.Net(deploy.prototxt, caffemodel, caffe.TEST)
However, what I want to do is to modify the parameters (e.g. kernel_size, or pad, etc.) of layers dynamically instead of modifying the prototxt file and reload it.
Is there any way to do so?

You can write your own get/set methods and expose them to python.
In layer.hpp:
virtual float GetParameter(const std::string param_name) {return -1;}
virtual void SetParameter(const std::string param_name, float val) {}
Then redefine these methods in layers where you would like to get/set parameters dynamically.
The last step is to expose the methods to python. In _caffe.cpp add this for bp::class_<Layer...:
.def("get_parameter", &Layer<Dtype>::GetParameter)
.def("set_parameter", &Layer<Dtype>::SetParameter)

I would suggest changing the way you think of this problem. What does it depend on for what you mentioned "dynamically modified parameter"? A most commonly used variable (which I was facing) is the current iteration times. For example I want to reduce the parameter value in every 10000 times. Based on that, in the layer when you are using that parameter, apply the function to modify it. That's the same as modifying the prototxt file.
To get the iteration times in specific layer, I just put one other's solution here. It is quite straightforward and could probably reduce your work load significantly compared to modifying prototxt file. Hopefully you could get inspired from this solution and apply it in your case.
https://stackoverflow.com/a/38386603/6591990

[Solution at the end of the post]
I needed to finetune a model and hence wanted to change the lr_mult parameters of individual layers programmatically. My search for help started from the title of this thread and thankfully ended in the below mentioned link titled 'How to modify a prototxt programmatically?'. https://github.com/BVLC/caffe/issues/4878
The parameters can be accessed and modified after loading the model definition prototxt file in google/protobuf in text_format. The modified protobuf can be written as a file.

How does Tensorflow manage graphs?

I have realized that there is some funky stuff going on with the way Tensorflow seems to be managing graphs.
Since building (and rebuilding) models is so tedious, I decided to wrap my custom model in a class so I could easily re-instantiate it elsewhere.
When I was training and testing the code (in the original place) it would work fine, however in the code where I loaded the graph's variables I would get all sorts of weird errors - variable redefinitions and everything else. This (from my last question about a similar thing) was the hint that everything was being called twice.
After doing a TON of tracing, it came down to the way I was using the loaded code. It was being used from within a class that had a structure like so
class MyModelUser(object):
def forecast(self):
# .. build the model in the same way as in the training code
# load the model checkpoint
# call the "predict" function on the model
# manipulate the prediction and return it
And then in some code that uses MyModelUserI had
def test_the_model(self):
model_user = MyModelUser()
print(model_user.forecast()) # 1
print(model_user.forecast()) # 2
and I (obviously) expected to see two forecasts when this was called. Instead, the first forecast was called and worked as expected, but the second call threw a TON of variable reuse ValueError an example of one of these was:
ValueError: Variable weight_def/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
I managed to quell the errors by adding a series of try/except blocks that used get_variable to create the variable, and then on exception, called reuse_variables on the scope and then get_variable without anything but the name. This brought on a new set of nasty errors, one of which was:
tensorflow.python.framework.errors.NotFoundError: Tensor name "weight_def/weights/Adam_1" not found in checkpoint files
On a whim I said "what if I move the modeling building code to __init__ so its only built once?"
My new model user:
class MyModelUser(object):
def __init__(self):
# ... build the model in the same way as in the training code
# load the model checkpoint
def forecast(self):
# call the "predict" function on the model
# manipulate the prediction and return it
and now:
def test_the_model(self):
model_user = MyModelUser()
print(model_user.forecast()) # 1
print(model_user.forecast()) # 2
Works as expected, printing two forecasts with no errors. This leads me to believe I can also get rid of the variable reuse stuff.
My question is this:
Why did this fix it? In theory, the graph should be reinstanced every single time in the original predict method, so it shouldn't be creating more than one graph. Does Tensorflow persist the graph even after the function completes? Is this why moving the creation code to __init__ worked? This has left me hopelessly confused.

By default, TensorFlow uses a single global tf.Graph instance that is created when you first call a TensorFlow API. If you do not create a tf.Graph explicitly, all operations, tensors, and variables will be created in that default instance. This means that each call in your code to model_user.forecast() will be adding operations to the same global graph, which is somewhat wasteful.
There are (at least) two possible courses of action here:
The ideal action would be to restructure your code so that MyModelUser.__init__() constructs an entire tf.Graph with all of the operations needed to perform forecasting, and MyModelUser.forecast() simply performs sess.run() calls on the existing graph. Ideally, you would only create a single tf.Session as well, because TensorFlow caches information about the graph in the session, and the execution would be more efficient.
The less invasive—but probably less efficient—change would be to create a new tf.Graph for every call to MyModelUser.forecast(). It's unclear from the question how much state is created in the MyModelUser.__init__() method, but you could do something like the following to put the two calls in different graphs:
def test_the_model(self):
with tf.Graph(): # Create a local graph
model_user_1 = MyModelUser()
print(model_user_1.forecast())
with tf.Graph(): # Create another local graph
model_user_2 = MyModelUser()
print(model_user_2.forecast())

TF has a default graph that new operations etc get added to. When you call your function twice, you will add the same things twice to the same graph. So, either build the graph once and evaluate it multiple times (as you have done, which is also the "normal" approach), or, if you want to change things, you can use reset_default_graph https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#reset_default_graph to reset the graph in order to have a fresh state.