In TensorFlow, we can use tf.nn.l2_loss() for doing L2 regularization. Let's say I want to define my own regularization operator for L1 regularization (call it tf.nn.l1_loss()). How would I go about it? I am having a hard time locating operator definitions in the TensorFlow source code.
As the comment suggested, there is a how-to guide for adding an op to TensorFlow. This guide covers adding a new op that is implemented in C++. In general, you should do this in the following situations:
The op cannot be implemented using existing TensorFlow ops (for example, l1_loss could be implemented using the existing element-wise and reduction operators as a Python function).
A C++ implementation is necessary for performance (or memory consumption) reasons.
The op could be implemented as a composition of ops, but it has a gradient that can be computed more efficiently (or with better numerical stability) than computing the gradients op-by-op. (This is why tf.nn.l2_loss is implemented as a fused op in C++.)
Related
I am attempting code a neural network from scratch. I am using performance by tensorflow (keras) as a ceiling (for now). Comparing an implementation of 'Forward Propagation' using the same dataset, weights, and biases (and other hyperparameters), I am noticing a very small difference in loss where ideally they should be the same.
I believe this difference aggregates as multiple passes are made. I would appreciate an insight on this matter.
The programming language I am using is Python 3.5 and data structures are implemented through the package numpy with much of data being of type numpy.float64.
I would like to implement the code for model.predict (https://keras.io/models/model/) in C++. But I am unable to find the exact logic (equations, formula) used in prediction?
For C++, I implemented the source code here: https://github.com/Dobiasd/frugally-deep
but unfortunately could not find the equation behind the predict function. (Frugally deep exports the model as a .json file and does the prediction using the predict function).
Would there be any resources that I could refer to find the equations for model.predict?
model.predict implements a forward pass of the model, so there is no direct equation, the computation is inferred from the computation graph of the model.
So in order to implement the same behavior, you have to do a forward pass through the layers of the model, where each layer implements its own computation, so its not a simple recommendation of use equation X, because its a large set of computational formulas that you have to implement, one for each kind of layer.
Looking at the repo, it appears you're looking for this.
I read that adding a new function is not so straightforward in this answer, so that is not a viable option. Also it was mentioned here that it is possible to implement Gaussian using the tools made available by tensorflow. Can someone please give a detailed answer on how to exactly implement a Gaussian activation function in tf such that it behaves like a normal non-linear function and can be trained by back-prop ?
In general, if you want to implement a complex function in tensorflow, you can use all the basic mathematic functions that tensorflow provides and many of the common functions will be implemented.
Those functions provided by tensorflow's api are usually implemented with gradient descent compatibility in mind. So if you implement your complex function using derivable tensorflow operations, your function will be compatible with gradient descent.
So when you have a new function to implement, search for its mathematical formula, most of the time you will find corresponding mathematical operations in tensorflow.
(To provide a specific, ready to use answer for your particular problem is not in the best interest of you or SO), but check the formula of a gaussian function and it should be easy to implement.
I want to implement Direct Feedback Alignemnt in Tensorflow.
Reference paper: https://arxiv.org/pdf/1609.01596v5.pdf, Nøkland (2016)
I implemented a simple network that does DFA in pure Python, having explicitly the backprop, I just switched the backward pass of BP with the one of DFA.
where B1 and B2 are two random matrices with appropriate dimensions.
Tensorflow training obviously does backprop by default. Where and what should I modify in Tensorflow code in order to get this training behavior?
I want to evolve a neural network using a genetic algorithm in order to approximate mathematical functions (linear, cubic, sine, tanh, etc). The requirement is that the NN should be evolved in terms of topology, weights and activation function of the neurons. The evaluation function is one that will calculate the error between the output of the NN and the function that should be approximated (mean squared error).
My main concern is that I want to be able to have absolute control over the representation/encoding/genome as I want to evolve layers of neurons, their weights and their activation functions at the same time, so I have to come up with a chromosome that incorporates all of these things.
What tools would you recommend? I'm checking DEAP for evolving the neural network and I was thinking of PyLearn 2 for implementing the actual best configuration the GA will result in.
The state-of-the-art for neural nets is Torch7 that is written in Lua. This is what Facebook AI and Google DeepMind use (as well as my lab). But I guess lua is not the best language to implement the sampling algorithms.
The alternative state-of-the-art framework is written in python it is called Theano, but I have a strong preference towards Torch. However, both of them might be too much for your task.
An easy alternative could also be to use Autograd, an automatic numpy differentiation library https://github.com/HIPS/autograd to easily train your neural net and have absolute control over it. There are several examples in their documentation.
Finally, genetic algorithms is another name for Monte Carlo, and there are many resources around implementing such things python (ex. https://codereview.stackexchange.com/questions/41004/organising-code-for-a-genetic-algorithm-in-python or github).
DEAP sounds a good and popular choice, and PyEvolve is outdated.
Edit: I forgot to mention Keras (http://keras.io/) which actually might be another good alternative and is Theano based.
You might also look at Blocks. It's a library built on theano that manages the computation graph via annotations of variables. It allows richer (ie, more complex, more recurrancy, more of those weird neural-stack-like structures you daydreamed) models than libraries like keras, pylearn2, and lasgne. Although those libraries might be better if you only want to 'evolve' the variations of the classical MLP architecture.