I am using fann2 (the Python binding for the FANN library) to train neural networks. I have trouble getting the weights of the network and the bias terms. I can see that the neural_net object has the following 2 methods:
get_connection_array
get_bias_array
Both methods require arguments, and I can't figure out what they should be (I can't find any documentation for these methods in Python). Any ideas?
Thanks for any help!
The python module is just a thin wrapper around th C++ code. Looks like this:
void get_bias_array(helper_array<unsigned int>* ARGOUT)
The argument is an array that the data gets copied into. I'm not sure what this is in python; I think you can pass in a new helper array and this call will allocate storage and copy the data.
Related
I am working on a project and I have a .so (shared object) which contains a bunch of functions inside which deal with matrix-matrix addition and matrix-matrix multiplication.I "LD_PRELOAD" this .so file while running a python script that uses TensorFlow.
In a nutshell, instead of using the matrix multiplication functions built into TensorFlow's I am using this .so file which has matrix multiply functions (TensorFlow-mkl-enabled uses libmklml_intel.so file for matrix multiply functions). This is the format in which mkl takes matrix multiplication.
Now, the problem at hand is that I need to monitor three things:
What are the dimensions of matrices that are multiplied?
How many times is this function called?
What is the % if time run time the script spends in this function?
I can think of two approaches
Trace where these function calls are being made from the TensorFlow framework and then do changes to get the required information. But I feel that this approach involves modification at multiple places as there's not just one single function which calls cblas. Additionally I'll have to scrape through the TensorFlow Source code which is a huge task. And it may also call for building TF from source again
But, if we are somehow able to moniter the interaction of the .so file with TF and maybe somehow register all the data sent while calling, maybe with a clever shell script or with a tool that I am unaware of? What are my best options? Would Perf be of any help here
Thanks in Advance
Newbie here
I have started using TensorFlow 2.0 and have a little uncertainty with regard to one aspect.
Suppose I have this use case: while ingesting data with the tf.data.Dataset I want to apply some specific augmentation operations upon some images. However, the external libraries that I am using require that the image is a numpy array, not a tensor.
When using tf.data.Dataset.from_tensor_slices(), the flowing data needs to be of type Tensor. Concrete example:
def my_function(tensor_image):
print(tensor_image.numpy()
return
data = tf.data.Dataset.from_tensor_slices(tensor_images).map(my_function)
The code above does not work yielding an
'Tensor' object has no attribute 'numpy' error.
I have read the documentation on TensorFlow 2.0 stating that if one wants to use an arbitrary python logic, one should use tf.py_function or only TensorFlow primitives according to:
How to convert "tensor" to "numpy" array in tensorflow?
My question is the following: Is there another way to use arbitrary python code in a function with a custom decorator/an easier way than to use tf.py_function?
To me honestly it seems that there must be a more elegant way than passing to a tf.py_function, transforming to a numpy array, perform operations A,B,C,D and then retransform to a tensor and yield the result.
There is no other way of doing it, because tf.data.Datasets are still (and they will always be, I suppose, for performance reasons) executed in graph mode and, thus, you cannot use anything outside of the tf.* methods, that can be easily converted by TensorFlow to its graph representation.
Using tf.py_function is the only way to mix Python execution (and thus, you can use any Python library) and graph execution when using a tf.data.Dataset object (on the contrary of what happens when using TensorFlow 2.0, that being eager by default allow this mixed execution naturally).
I have read and studied the TFF guide and APIs pages precisely. But I am confused in some detail parts.
For example, when I want to wrap/decorate a TF/python function, use these two below APIs:
1. tff.tf_computation()
2. tff.federated_computation()
I can not find what are differences between them and when I am allowed to use them. Especially, in case I want to use other algorithms except for FedAvg or FedSgd. I wonder if you know:
How they could be used to manipulate inputs? do they work on #CLIENT or #SERVER?
How I could use them in another usage except for the output of tff.federated_mean or tff.federated_sum that the value will be in the server?
How I am able to have access to the detail of data and metrics in #CLIENT and #SERVER?
Why we should invoke the tff.tf_computation() from tff.federated_computation()? In this link, there was not any explanation about them.
Do these APIs (e.g. tff.federated_mean or tff.federated_sum) modify the output elements of each #CLIENT and bring them to the #SERVER?
Could anyone help me to understand intuitive behind the concept?
A possible rule of thumb about the different function decorators:
tff.tf_computation is for wrapping TF logic. Think "tensors in, tensors out": this should be very similar to the usage of tf.function, where the parameters and return values are tensors, or nested structures of tensors. TFF intrinsics (e.g. tff.federated_mean) cannot be used inside a tff.tf_computation, and tff.tf_computations cannot call tff.federated_computations. The type signature is always on unplaced.
tff.federated_computation should be used to wrap TFF programming abstractions. Think "tensors here, tensors there": Inside this context, a tff.tf_computation can be applied to tff.Values and tff.Values can be communicated to other placements using the intrinsics. The type signature can accept federated types (i.e. types with placements).
For your list of questions:
Both can work on values placed at CLIENTS or SERVER. For example, tff.tf_computation called my_comp can be applied to a value v with type int32#CLIENTS with tff.federated_map(my_comp, v), which will run my_comp on each client.
tff.federated_map() supports applying a computation pointwise (across clients) to data not on the server. You can manipulate the metrics on each client using tff.federated_map. TFF isn't intended for separate options on different clients; the abstractions do not support addressing individuals. You may be able to simulate this in Python, see Operations performed on the communications between the server and clients.
The values of placed data can be inspected in simulation simply by returning them from a tff.Computation, and invoking that computation. The values should be available in the Python environment.
tff.tf_computations should be invokable from anywhere, if there is documentation that says otherwise please point to it. I believe what was intended to highlight is that a tff.federated_computation may invoke a tff.tf_computation, but not vice versa.
The tutorials (Federated Learning for Image Classification and Federated Learning for Text Generation) show examples of printing out the metrics in simulation. You may also be interested in the answer to how to print local outputs in tensorflow federated?
tff.tf_computations should be executed directly if desired. This will avoid any of the federated part of TFF, and simply delegate to TensorFlow. To apply the computation to federated values and use in combination with federated intrinsics, they must be called inside a tff.federated_computation.
I have thought that adding a new module which will do the center pooling.
I was looking in the tensorflow code and there is a file named gen_nn_ops.py which internally calls function from another file by passing "Maxpool", "AvgPool", etc. parameters to do the required computation.
I want to do centre pooling which selects the center element in the window. I have code ready for the matlab and c++ versions but need to know how to add a new module in TensorFlow for such computation. Also where to set the backpropogation code.
A custom pooling layer would probably be implemented in C++. To see what you'd need to do, let's see where the implementation of tf.nn.max_pool_with_argmax() lives:
The Python wrapper function (tf.nn.max_pool_with_argmax() itself) is automatically generated, in gen_nn_ops.py. This is ultimately imported into nn.py, so that it appears under tf.nn when you import tensorflow as tf.
In C++, there is an op registration in ops/nn_ops.cc, and a kernel registration in kernels/maxpooling_op.cc.
The kernel itself is defined in kernels/maxpooling_op.cc.
The gradient is defined as a separate op—"MaxPoolWithArgmaxGrad"—in the same places.
There's a fair amount of work to do to add a new op (see this tutorial for a more complete guide), but hopefully these pointers can help!
I use the follwoing code to load the net and set up it, the parameters of layers are stored in deploy.prototxt.
net = caffe.Net(deploy.prototxt, caffemodel, caffe.TEST)
However, what I want to do is to modify the parameters (e.g. kernel_size, or pad, etc.) of layers dynamically instead of modifying the prototxt file and reload it.
Is there any way to do so?
You can write your own get/set methods and expose them to python.
In layer.hpp:
virtual float GetParameter(const std::string param_name) {return -1;}
virtual void SetParameter(const std::string param_name, float val) {}
Then redefine these methods in layers where you would like to get/set parameters dynamically.
The last step is to expose the methods to python. In _caffe.cpp add this for bp::class_<Layer...:
.def("get_parameter", &Layer<Dtype>::GetParameter)
.def("set_parameter", &Layer<Dtype>::SetParameter)
I would suggest changing the way you think of this problem. What does it depend on for what you mentioned "dynamically modified parameter"? A most commonly used variable (which I was facing) is the current iteration times. For example I want to reduce the parameter value in every 10000 times. Based on that, in the layer when you are using that parameter, apply the function to modify it. That's the same as modifying the prototxt file.
To get the iteration times in specific layer, I just put one other's solution here. It is quite straightforward and could probably reduce your work load significantly compared to modifying prototxt file. Hopefully you could get inspired from this solution and apply it in your case.
https://stackoverflow.com/a/38386603/6591990
[Solution at the end of the post]
I needed to finetune a model and hence wanted to change the lr_mult parameters of individual layers programmatically. My search for help started from the title of this thread and thankfully ended in the below mentioned link titled 'How to modify a prototxt programmatically?'. https://github.com/BVLC/caffe/issues/4878
The parameters can be accessed and modified after loading the model definition prototxt file in google/protobuf in text_format. The modified protobuf can be written as a file.