I have thought that adding a new module which will do the center pooling.
I was looking in the tensorflow code and there is a file named gen_nn_ops.py which internally calls function from another file by passing "Maxpool", "AvgPool", etc. parameters to do the required computation.
I want to do centre pooling which selects the center element in the window. I have code ready for the matlab and c++ versions but need to know how to add a new module in TensorFlow for such computation. Also where to set the backpropogation code.
A custom pooling layer would probably be implemented in C++. To see what you'd need to do, let's see where the implementation of tf.nn.max_pool_with_argmax() lives:
The Python wrapper function (tf.nn.max_pool_with_argmax() itself) is automatically generated, in gen_nn_ops.py. This is ultimately imported into nn.py, so that it appears under tf.nn when you import tensorflow as tf.
In C++, there is an op registration in ops/nn_ops.cc, and a kernel registration in kernels/maxpooling_op.cc.
The kernel itself is defined in kernels/maxpooling_op.cc.
The gradient is defined as a separate op—"MaxPoolWithArgmaxGrad"—in the same places.
There's a fair amount of work to do to add a new op (see this tutorial for a more complete guide), but hopefully these pointers can help!
Related
I'm trying to modify a LSTM model implemented with Sonnet and Tensorflow. The idea is that the model should update itself whenever new results are received.
The problem is that Python classes are used and added automatically during its definition. This means that I don't have explicit access to them once the algorithm is running, and I cannot change them using methods like tf.variable or tf.method.
I've tried changing the values of the Python object directly, but it only affects the original instance of the class and not the one used in the model.
I understand that in order to modify a model during execution I need to create a new method, but I don't know where to create it or how to make it work for Sonnet.
Could anyone give some tip on how to solve this problem, or point me towards the right direction to find an answer?
I am working on a project and I have a .so (shared object) which contains a bunch of functions inside which deal with matrix-matrix addition and matrix-matrix multiplication.I "LD_PRELOAD" this .so file while running a python script that uses TensorFlow.
In a nutshell, instead of using the matrix multiplication functions built into TensorFlow's I am using this .so file which has matrix multiply functions (TensorFlow-mkl-enabled uses libmklml_intel.so file for matrix multiply functions). This is the format in which mkl takes matrix multiplication.
Now, the problem at hand is that I need to monitor three things:
What are the dimensions of matrices that are multiplied?
How many times is this function called?
What is the % if time run time the script spends in this function?
I can think of two approaches
Trace where these function calls are being made from the TensorFlow framework and then do changes to get the required information. But I feel that this approach involves modification at multiple places as there's not just one single function which calls cblas. Additionally I'll have to scrape through the TensorFlow Source code which is a huge task. And it may also call for building TF from source again
But, if we are somehow able to moniter the interaction of the .so file with TF and maybe somehow register all the data sent while calling, maybe with a clever shell script or with a tool that I am unaware of? What are my best options? Would Perf be of any help here
Thanks in Advance
Newbie here
I have read and studied the TFF guide and APIs pages precisely. But I am confused in some detail parts.
For example, when I want to wrap/decorate a TF/python function, use these two below APIs:
1. tff.tf_computation()
2. tff.federated_computation()
I can not find what are differences between them and when I am allowed to use them. Especially, in case I want to use other algorithms except for FedAvg or FedSgd. I wonder if you know:
How they could be used to manipulate inputs? do they work on #CLIENT or #SERVER?
How I could use them in another usage except for the output of tff.federated_mean or tff.federated_sum that the value will be in the server?
How I am able to have access to the detail of data and metrics in #CLIENT and #SERVER?
Why we should invoke the tff.tf_computation() from tff.federated_computation()? In this link, there was not any explanation about them.
Do these APIs (e.g. tff.federated_mean or tff.federated_sum) modify the output elements of each #CLIENT and bring them to the #SERVER?
Could anyone help me to understand intuitive behind the concept?
A possible rule of thumb about the different function decorators:
tff.tf_computation is for wrapping TF logic. Think "tensors in, tensors out": this should be very similar to the usage of tf.function, where the parameters and return values are tensors, or nested structures of tensors. TFF intrinsics (e.g. tff.federated_mean) cannot be used inside a tff.tf_computation, and tff.tf_computations cannot call tff.federated_computations. The type signature is always on unplaced.
tff.federated_computation should be used to wrap TFF programming abstractions. Think "tensors here, tensors there": Inside this context, a tff.tf_computation can be applied to tff.Values and tff.Values can be communicated to other placements using the intrinsics. The type signature can accept federated types (i.e. types with placements).
For your list of questions:
Both can work on values placed at CLIENTS or SERVER. For example, tff.tf_computation called my_comp can be applied to a value v with type int32#CLIENTS with tff.federated_map(my_comp, v), which will run my_comp on each client.
tff.federated_map() supports applying a computation pointwise (across clients) to data not on the server. You can manipulate the metrics on each client using tff.federated_map. TFF isn't intended for separate options on different clients; the abstractions do not support addressing individuals. You may be able to simulate this in Python, see Operations performed on the communications between the server and clients.
The values of placed data can be inspected in simulation simply by returning them from a tff.Computation, and invoking that computation. The values should be available in the Python environment.
tff.tf_computations should be invokable from anywhere, if there is documentation that says otherwise please point to it. I believe what was intended to highlight is that a tff.federated_computation may invoke a tff.tf_computation, but not vice versa.
The tutorials (Federated Learning for Image Classification and Federated Learning for Text Generation) show examples of printing out the metrics in simulation. You may also be interested in the answer to how to print local outputs in tensorflow federated?
tff.tf_computations should be executed directly if desired. This will avoid any of the federated part of TFF, and simply delegate to TensorFlow. To apply the computation to federated values and use in combination with federated intrinsics, they must be called inside a tff.federated_computation.
I implemented a model in TensorFlow (Python) that I previously programmed in C++ using Eigen, where it worked as expected. But the model is not working as expected in Python, and it's probably because I am defining tensors incorrectly or I am mixing up dimensions.
I am trying to get a feel for the problems by using Visual Studio's (2017) debugger (if a different IDE is better for this then I'm all ears, but I would prefer to stick with VS), but tensors do not evaluate to anything - and I can understand this because the tensor defines an operation and not a data object (well it only produces a data object after calling a session.run).
However, constant and variable tensors - and any other tensors built solely on top of such tensors - come with predefined data. So hey, why not be able to inspect the value through the debugging UI?
So my question: is there a way to inpect the data with some extension?
For example, if I was working in C++ and with Eigen, I can use Eigen.natvis as described here. Anything similar for TensorFlow? It's not just a matter of seeing the evaluated value, either. It would be nice to see things like the shape, etc... while debugging.
I would also be open to other debugging techniques of TensorFlow code, if anyone has a good suggestion.
TensorFlow includes tfdbg, a debugger for TensorFlow models, where you can step through each execution step, check values, stop on NaN, etc. See the programmer's guide TensorFlow Debugger and The Debugger Dashboard for more information.
tfdbg can be a bit cumbersome to setup and use though. A quick alternative to check intermediate values is to use tf.Print operations. TensorFlow includes a few other debugging operations that you may find useful to check for some basic things.
EDIT: Another tool that can be useful is eager execution. This allows you to use TensorFlow operations as if they were regular Python operations (they return the result of the operation instead of the graph object), so it is a good way to check if some particular code does what you expect.
I am using fann2 (the Python binding for the FANN library) to train neural networks. I have trouble getting the weights of the network and the bias terms. I can see that the neural_net object has the following 2 methods:
get_connection_array
get_bias_array
Both methods require arguments, and I can't figure out what they should be (I can't find any documentation for these methods in Python). Any ideas?
Thanks for any help!
The python module is just a thin wrapper around th C++ code. Looks like this:
void get_bias_array(helper_array<unsigned int>* ARGOUT)
The argument is an array that the data gets copied into. I'm not sure what this is in python; I think you can pass in a new helper array and this call will allocate storage and copy the data.