I read that adding a new function is not so straightforward in this answer, so that is not a viable option. Also it was mentioned here that it is possible to implement Gaussian using the tools made available by tensorflow. Can someone please give a detailed answer on how to exactly implement a Gaussian activation function in tf such that it behaves like a normal non-linear function and can be trained by back-prop ?
In general, if you want to implement a complex function in tensorflow, you can use all the basic mathematic functions that tensorflow provides and many of the common functions will be implemented.
Those functions provided by tensorflow's api are usually implemented with gradient descent compatibility in mind. So if you implement your complex function using derivable tensorflow operations, your function will be compatible with gradient descent.
So when you have a new function to implement, search for its mathematical formula, most of the time you will find corresponding mathematical operations in tensorflow.
(To provide a specific, ready to use answer for your particular problem is not in the best interest of you or SO), but check the formula of a gaussian function and it should be easy to implement.
Related
I would like to implement a cost-sensitive loss function in PyTorch. My two-class training dataset is heavily imbalanced, where 75% of the data are label '0' and only 25% of the data are label '1'.
I am new to PyTorch but my supervisor is adamant that I use it (they have more experience with it).
I found some implementations in Keras, but I am not that strong in coding to be able to port it over to PyTorch.
I have read around to find some resources to create a cost-sensitive loss function.
This paper uses something which I think might work (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9417097), but I do not understand how the code is implemented despite having access to it here (https://github.com/emadeldeen24/AttnSleep/blob/f993511426900f9fca20594a738bf8bee1116381/utils/util.py).
This website describes the math very detailedly but I do not understand it: https://medium.com/rv-data/how-to-do-cost-sensitive-learning-61848bf4f5e7
Here is an implementation in Keras which I have trouble with converting to PyTorch: https://towardsdatascience.com/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9
I also found this implementation in PyTorch, but have trouble with understanding it: https://discuss.pytorch.org/t/dealing-with-imbalanced-datasets-in-pytorch/22596/21
Could you please help me to understand the last link's implementation of the cost-sensitive loss function?
Thank you.
I would like to implement the code for model.predict (https://keras.io/models/model/) in C++. But I am unable to find the exact logic (equations, formula) used in prediction?
For C++, I implemented the source code here: https://github.com/Dobiasd/frugally-deep
but unfortunately could not find the equation behind the predict function. (Frugally deep exports the model as a .json file and does the prediction using the predict function).
Would there be any resources that I could refer to find the equations for model.predict?
model.predict implements a forward pass of the model, so there is no direct equation, the computation is inferred from the computation graph of the model.
So in order to implement the same behavior, you have to do a forward pass through the layers of the model, where each layer implements its own computation, so its not a simple recommendation of use equation X, because its a large set of computational formulas that you have to implement, one for each kind of layer.
Looking at the repo, it appears you're looking for this.
What mathematical model does the Linear Regression function use in scikit learn? The Ordinary Least Squares model has more than one way to minimize the cost function. I've found the form of the function it solves here, but I'm also interested which method it uses exactly. Can anyone elaborate? Thank you!
You can basically hunt around the source code enough, and you'll find it.
In base.py, you can find it uses linal.lstsq.
In linalg.py, you can see that lstsq uses the dglelsd family.
This family is SVD-related. It uses SVD to solve the original problem.
In TensorFlow, we can use tf.nn.l2_loss() for doing L2 regularization. Let's say I want to define my own regularization operator for L1 regularization (call it tf.nn.l1_loss()). How would I go about it? I am having a hard time locating operator definitions in the TensorFlow source code.
As the comment suggested, there is a how-to guide for adding an op to TensorFlow. This guide covers adding a new op that is implemented in C++. In general, you should do this in the following situations:
The op cannot be implemented using existing TensorFlow ops (for example, l1_loss could be implemented using the existing element-wise and reduction operators as a Python function).
A C++ implementation is necessary for performance (or memory consumption) reasons.
The op could be implemented as a composition of ops, but it has a gradient that can be computed more efficiently (or with better numerical stability) than computing the gradients op-by-op. (This is why tf.nn.l2_loss is implemented as a fused op in C++.)
I want to evolve a neural network using a genetic algorithm in order to approximate mathematical functions (linear, cubic, sine, tanh, etc). The requirement is that the NN should be evolved in terms of topology, weights and activation function of the neurons. The evaluation function is one that will calculate the error between the output of the NN and the function that should be approximated (mean squared error).
My main concern is that I want to be able to have absolute control over the representation/encoding/genome as I want to evolve layers of neurons, their weights and their activation functions at the same time, so I have to come up with a chromosome that incorporates all of these things.
What tools would you recommend? I'm checking DEAP for evolving the neural network and I was thinking of PyLearn 2 for implementing the actual best configuration the GA will result in.
The state-of-the-art for neural nets is Torch7 that is written in Lua. This is what Facebook AI and Google DeepMind use (as well as my lab). But I guess lua is not the best language to implement the sampling algorithms.
The alternative state-of-the-art framework is written in python it is called Theano, but I have a strong preference towards Torch. However, both of them might be too much for your task.
An easy alternative could also be to use Autograd, an automatic numpy differentiation library https://github.com/HIPS/autograd to easily train your neural net and have absolute control over it. There are several examples in their documentation.
Finally, genetic algorithms is another name for Monte Carlo, and there are many resources around implementing such things python (ex. https://codereview.stackexchange.com/questions/41004/organising-code-for-a-genetic-algorithm-in-python or github).
DEAP sounds a good and popular choice, and PyEvolve is outdated.
Edit: I forgot to mention Keras (http://keras.io/) which actually might be another good alternative and is Theano based.
You might also look at Blocks. It's a library built on theano that manages the computation graph via annotations of variables. It allows richer (ie, more complex, more recurrancy, more of those weird neural-stack-like structures you daydreamed) models than libraries like keras, pylearn2, and lasgne. Although those libraries might be better if you only want to 'evolve' the variations of the classical MLP architecture.