We Keep Coding

How to reorder different sets of parameters in dm-haiku

In dm-haiku, parameters of neural networks are defined in dictionaries where keys are module (and submodule) names. If you would like to traverse through the values, there are multiple ways of doing so as shown in this dm-haiku issue. However, the dictionary doesn't respect the ordering of the modules and makes it hard to parse submodules. For example, if I have 2 linear layers, each followed by a mlp layer, then using hk.data_structures.traverse(params) will (roughly) return:
['linear', 'linear_2', 'mlp/~/1', 'mlp/~/2'].
whereas I would like it to return:
['linear', 'mlp/~/1', 'linear_2', 'mlp/~/2'].
My reason for wanting this form is if creating an invertible neural network and wanting to reverse the order the params are called, isolating substituent parts for other purposes (e.g. transfer learning), or, in general, wanting more control of how and where to (re)use trained parameters.
To deal with this, I've resorted to regex the names and put them in the order that I want, then using hk.data_structures.filter(predicate, params) to filter by the sorted module names. Although, this is quite tedious if I have to remake a regex every time I want to do this.
I'm wondering if there is a way to convert a dm-haiku dictionary of params to something like a pytree with a hierarchy and ordering that makes this easier? I believe equinox handles parameters in this manner (and I'm going to look more into how that is done soon), but wanted to check to see if I'm overlooking a simple method to allow grouping, reversing, and other permutations of the params's dictionary?

According to source code https://github.com/deepmind/dm-haiku/blob/main/haiku/_src/filtering.py#L42#L46 haiku use the sorted function of dict (haiku parameters are vanilla dict since 0.0.6) for hk.data_structures.traverse. Therefore you can't get the result you want without modifying the function itself. By the way, I don't get precisely what do you mean by "to reverse the order the params are called". All parameters are passed together in input and then the only thing that determines the order of use is the architecture of the function itself so you should manually invert the forward pass but you don't need to change something in params.

How to plot Histogram summary for delta weight in Federated Tensorflow?

I am analyzing a method that I have implemented in Tensorflow Federated with FedAvg. I need to have a histogram for every clients' delta weights that are communicated to the server. Each client separately called in simulation/federated_avaraging.py, but the thing is I can not call the following API in there. tf.summary.histogram(). any help would be appreciated.

In TFF, TensorFlow represents "local computation"; so if you need a way to inspect something across clients, you will need to first aggregate the values you want via TFF, or inspect the returned values in native python.
If you want to use TF ops, I would recommend using the tff.federated_collect intrinsic, to "gather" all the values you want on the server, then federated_map a TF function which takes these values and produces your desired visualization.
If you would rather work at the Python level, there is an easy option here (this is the approach I would take): simply return the results of training at the clients from your tff.federated_computation; when you invoke this computation, this will materialize a Python list of these results, and you can visualize it however you want. This would be roughly along the lines of something like:
#tff.federated_computation(...)
def train_one_round(...):
...
trained_clients = run_training(...)
new_model = update_global_model(trained_clients,...)
return new_model, trained_clients
In this example, this function will return a tuple, the second element of which is a Python list representing the results of training at all clients.

How to load multiple objects (list of list) in HDF5?

I am currently trying to use keras to predict ... stuffs. I am using a HDF5 file as input. The file contains 195 objects, each one is a list of matrices with one attribute. I would like keras to learn on the list of matrices and predict the attribute. But here is the issue, so far I have seen that one object can only be assigned to one variable. That would be meaningless in my case.
I would like to know whether or not it was possible to load all of these objects at once, say under one variable, in keras to predict the attribute ? For instance here are some objects,
['10gs', '1a30', '1bcu',..., '4tmn']
I know I can assign one variable to one object,
dataset=infile['1a30']
However I am not sure how to assign several objects to one variable? Do I need to create a list of objects ? Here's what I am trying to get,
dataset=infile['all of my objects'].
In fine, I will be using it in keras but I am not too sure whether it is necessary as it seems to me it is a HDF file issue (misunderstanding).

Sporadically assigning values to variables (placeholders vs feed_dict)

I have a model where I need to assign to the weights (trainable variables) new external values every N iterations.
I can think of a few solutions:
Save and restore
Not good as I would need to serialization, go through a file system calls, etc. (even if I use something like tmpfs)
Using placeholders and assign operations
I would create a placeholder and assign op for each trainable variable. Everytime I want to assign something to the weights, I ran the assign ops.
However, I understand that this means I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph.
In addition I would be using much more memory than necessary..
Use a feed_dict for trainable variable and trigger ops that assign each variable to itself?
Does this work? Is there any drawback?
Before coding something I thought it was a good idea to ask?
What is the recommended way to assign new external values to variables efficiently (memory/timewise)?

Your 3-rd option sounds like the best one.
You can feed values to tensors that aren’t placeholders.
TensorFlow's feed mechanism lets you inject data into any Tensor in a
computation graph. A python computation can thus feed data directly
into the graph.
Any tensors that are feedable can be fed. To check if a tensor is feedable or not, use: tf.Graph.is_feedable(tensor).

In recent versions of Tensorflow Variable class has load method. It does exactly what you want.
https://www.tensorflow.org/api_docs/python/tf/Variable#load

You can use the assign operations with placeholders.
I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph
In addition I would be using much more memory than necessary..
No. You would only need to feed values to the placeholders when you run the assign operations. Don't make the assign operation part of your training graph and only run them when you want to assign new values.
If the assigning turns out to be a bottleneck (for small N it might slow down your program) you can consider other methods of getting data into TensorFlow.

How to get a tensorflow op by name?

You can get a tensor by name with tf.get_default_graph().get_tensor_by_name("tensor_name:0")
But can you get an operation, such as Optimizer.minimize, or an enqueue operation on a queue?
In my first model I returned all tensors and ops I would need from a build_model function. But the list of tensors got ugly. In later models I tossed all tensors and ops in a dictionary for easier access. This time around I thought I'd just look up tensors by name as I needed them, but I don't know how to do that with ops.
Or is there a better way to do this? I find various tensors and ops are needed all over the place. Training, inference code, test cases, hence the desire for a nice standard way of accessing the various parts of the graph without passing variables all over the place.

You can use the tf.Graph.get_operation_by_name() method to get a tf.Operation by name. For example, to get an operation called "enqueue" from the default graph:
op = tf.get_default_graph().get_operation_by_name("enqueue")