I want to perform fitting using following Fit_Curve function but I'm getting the runtime error which may arise when the value of x is too small. But I don't want to change the value of x in my code. I read an answer related to this question that I need to perform sum simplification of fitting function. Is there any other way or easiest way to get rid of this error. Is it just a warning or is it an error? Does this warning effect my final result means fitting?
def Fit_Curve(x,a,beta,b,gamma):
return (a*(x**beta)*np.exp(-b*(x**gamma)))
matrix_check.py:105: RuntimeWarning: divide by zero encountered in
power return (a*(x**beta)*np.exp(-b*(x**gamma)))
Related
import numpy as np
import math as m
def distance(list1_features, list2_features):
distance = (list1_features - list2_features)
"""The Euclidean distance between two arrays of feature values."""
return m.sqrt(np.sum((distance)**2))`enter code here`
distances_btw_song=[]
for index, row in lyrics.iterrows():
point1= np.array(lyrics.iloc[0:1,3:])
point2=np.array(lyrics.iloc[0:1,3])
distances=distance(point1,point2)
distances_btw_song.append(distances)
lyrics["Distance"]= distances_btw_song
lyrics.head()
I got this code from my partner and we are both using the same code however when the code reaches the distance(point1,point2) it gives me float object not callable but when my partner ran it, he got the table.
DataBase:
Traceback
The reuse of the variable distance is likely the culprit, though you should include more data about the error, like a stack trace. You somehow have the value of distance set to a float instead of a function. Depending on how you're running your code, there's a few ways this might accidentally happen. Again, you need to include more context around how the error occurred, not just the code and an error.
I figured out what was the problem I just changed
def distance(list1_features, list2_features):
to this
def euclidean_distance(list1_features, list2_features):
I wish to use scipy's optimize.fmin function to find the minimum of a function, which is a function of both variables I wish to minimize over and parameters which do not change (are not optimized over).
I am able to do this when optimizing over a single variable here:
from scipy import optimize
c1=4
c2=-1
def f(x,c1,c2):
return x**2+c1+c2
guess_f=1
minimum = optimize.fmin(f,guess_f,args=(c1,c2),maxfun=400,maxiter=400,ftol=1e-2,xtol=1e-4)
However, I cannot get this to work when I add another variable to minimize over:
def g(x,y,c1,c2):
return x*y+c1+c2
guess_g=[1,1]
minimum2= optimize.fmin(g,guess_g,args=(c1,c2),maxfun=400,maxiter=400,ftol=1e-2,xtol=1e-4)
I get the following error message:
TypeError: g() missing 1 required positional argument: 'c2'
I did find Multiple variables in SciPy's optimize.minimize, and a solution is presented here in which the variables to be optimized over need to be grouped together as their own array. I try something like this below:
def g(params,c1,c2):
x,y=params
# print(params)
return x*y+c1*x+c2
guess_g=[1,1]
minimum2= optimize.fmin(g,guess_g,args=(c1,c2),maxfun=4000,maxiter=4000,ftol=1e-2,xtol=1e-4)
I do not receive a TypeError, but what I do get is the "Warning: Maximum number of function evaluations has been exceeded." message along with a RuntimeWarning: overflow encountered in double_scalars after removing the cwd from sys.path. (additionally, I tried using the optimize.minimize command to do the same thing, but was unable to get it to work when adding the extra arguments, but I do not post that code here as the question is already getting long).
So this does not seem to be the correct way to do this.
How do I go about optimizing with optimize.fmin function over multiple variables, while also giving my function additional arguments?
I am trying to train a neural network in Python 3.7. For this, I am using Numpy to perform calculations and do my matrix multiplications. I find this error
RuntimeWarning: overflow encountered in multiply (when I am multiplying matrices)
This, in turn, results in nan values, which raises errors like
RuntimeWarning: invalid value encountered in multiply
RuntimeWarning: invalid value encountered in sign
Now, I have seen many answers related to this question, all explaining why this happens. But I want to know, "How do I solve this problem?". I have tried using the default math module, but that still doesn't work and raises errors like
TypeError: only size-1 arrays can be converted to Python scalars
I know I can use for loops to do the multiplications, but that is computationally very expensive, and also lengthens and complicates the code a lot. Is there any solution to this problem? Like doing something with Numpy (I am aware that there are ways to handle exceptions, but not solve them), and if not, then perhaps alternative to Numpy, which doesn't require me to change my code much?
I don't really mind if the precision of my data is compromised a bit. (If it helps the dtype for the matrices is float64)
EDIT:
Here is a dummy version of my code:
import numpy as np
network = np.array([np.ones(10), np.ones(5)])
for i in range(100000):
for lindex, layer in enumerate(network):
network[lindex] *= abs(np.random.random(len(layer)))*200000
I think the overflow error occurs when I am adding large values to the network.
This is a problem I too have faced with my neural network while using ReLu activators because of the infinite range on the positive side. There are two solutions to this problem:
A) Use another activation function: atan,tanh,sigmoid or any other one with limited range
However if you do not find those suitable:
B)Dampen the ReLu activations. This can be done by scaling down all values of the ReLu and ReLu prime function. Here's the difference in code:
##Normal Code
def ReLu(x,derivative=False):
if derivative:
return 0 if x<0 else 1
return 0 if x<0 else x
##Adjusted Code
def ReLu(x,derivative=False):
scaling_factor = 0.001
if derivative:
return 0 if x<0 else scaling_factor
return 0 if x<0 else scaling_factor*x
Since you are willing to compromise on the precision this is a perfect solution for you! In the ending, you can multiply by the inverse of the scaling_factor to get the approximate solution- approximate because of rounding discrepancies.
I have a function I want to minimize with scipy.optimize.fmin. Note that I force a print when my function is evaluated.
My problem is, when I start the minimization, the value printed decreases untill it reaches a certain point (the value 46700222.800). There it continues to decrease by very small bites, e.g., 46700222.797,46700222.765,46700222.745,46700222.699,46700222.688,46700222.678
So intuitively, I feel I have reached the minimum, since the length of each step are minus then 1. But the algorithm keeps running untill I get a "Maximum number of function evaluations has been exceeded" error.
My question is: how can I force my algorithm to accept the value of the parameter when the function evaluation reaches a value from where it does not really evolve anymore (let say, I don't gain more than 1 after an iteration). I read that the options ftol could be used but it has absolutely no effect on my code. In fact, I don't even know what value to put for ftol. I tried everything from 0.00001 to 10000 and there is still no convergence.
There is actually no need to see your code to explain what is happening. I will answer point by point quoting you.
My problem is, when I start the minimization, the value printed decreases
untill it reaches a certain point (the value 46700222.800). There it
continues to decrease by very small bites, e.g.,
46700222.797,46700222.765,46700222.745,46700222.699,46700222.688,46700222.678
Notice that the difference between the last 2 values is -0.009999997913837433, i.e. about 1e-2. In the convention of minimization algorithm, what you call values is usually labelled x. The algorithm stops if these 2 conditions are respected AT THE SAME TIME at the n-th iteration:
convergence on x: the absolute value of the difference between x[n] and the next iteration x[n+1] is smaller than xtol
convergence on f(x): the absolute value of the difference between f[n] and f[n+1] is smaller than ftol.
Moreover, the algorithm stops also if the maximum number of iterations is reached.
Now notice that xtol defaults to a value of 1e-4, about 100 times smaller than the value 1e-2 that appears for your case. The algorithm then does not stop, because the first condition on xtol is not respected, until it reaches the maximum number of iterations.
I read that the options ftol could be used but it has absolutely no
effect on my code. In fact, I don't even know what value to put for
ftol. I tried everything from 0.00001 to 10000 and there is still no
convergence.
This helped you respecting the second condition on ftol, but again the first condition was never reached.
To reach your aim, increase also xtol.
The following methods will also help you more in general when debugging the convergence of an optimization routine.
inside the function you want to minimize, print the value of x and the value of f(x) before returning it. Then run the optimization routine. From these prints you can decide sensible values for xtol and ftol.
consider nondimensionalizing the problem. There is a reason if ftol and xtol default both to 1e-4. They expect you to formulate the problem so that x and f(x) are of order O(1) or O(10), say numbers between -100 and +100. If you carry out the nondimensionalization you handle a simpler problem, in the way that you often know what values to expect and what tolerances you are after.
if you are interested just in a rough calculation and can't estimate typical values for xtol and ftol, and you know (or you hope) that your problem is well behaved, i.e. that it will converge, you can run fmin in a try block, pass to fmin only maxiter=20 (say), and catch the error regarding the Maximum number of function evaluations has been exceeded.
I just spent three hours digging into the source code of scipy.minimize. In it, the "while" loop in function "_minimize_neldermead" deals with the convergence rule:
if (numpy.max(numpy.ravel(numpy.abs(sim[1:] - sim[0]))) <= xtol and
numpy.max(numpy.abs(fsim[0] - fsim[1:])) <= ftol):
break"
"fsim" is the variable that stores results from functional evaluation. However, I found that fsim[0] = f(x0) which is the function evaluation of the initial value, and it never changes during the "while" loop. fsim[1:] updates itself all the time. The second condition of the while loop was never satisfied. It might be a bug. But my knowledge of mathematical optimization is far from enough to judge it.
My current solution: design your own system to control the convergence. Add this in your function:
global x_old, Q_old
if (np.absolute(x_old-x).sum() <= 1e-4) and (np.absolute(Q_old-Q).sum() <= 1e-4):
return None
x_old = x; Q_old = Q
Here Q=f(x). Don't forget to give them an initial value.
Update 01/30/15:
I got it! This should be the correct code for the second line of the if function (i.e. remove numpy.absolute):
numpy.max(fsim[0] - fsim[1:]) <= ftol)
btw, this is my first debugging of a open source software. I just created an issue on GitHub.
Update 01/31/15 - 1:
I don't think my previous update is correct. Nevertheless, this is the a screenshot of the iterations of a function using the original code.
It prints the values of sim and fsim variable for each iteration. As you can see, the changes of each iteration is less than both of xtol and ftol values, but it just kept going without stopping. The original code compares the difference between fsim[0] and the rest of fsim values, i.e. the value here is always 87.63228689 - 87.61312213 = .01916476, which is greater than ftol=1e-2.
Update 01/31/15 - 2:
Here is the data and code that I used to reproduce the previous results. It includes two data files and one iPython Notebook file.
From the documentation it looks like you DO want to change the ftol arg.
Post your code so we can look at your progress.
edit: Try increasing xtol as well.
Your question is a bit ambiguous. Are you printing the value of your function, or the point where it is evaluated?
My understanding of xtol and ftol is as follows. The iteration stops
when the change in the value of the function between iterations is less than ftol
AND
when the change in x between successive iterations is less than xtol
When you say "...accept the value of the parameter...", this suggests you should change xtol.
Fairly simple question: How should I use #pm.stochastic? I have read some blog posts that claim #pm.stochasticexpects a negative log value:
#pm.stochastic(observed=True)
def loglike(value=data):
# some calculations that generate a numeric result
return -np.log(result)
I tried this recently but found really bad results. Since I also noticed that some people used np.log instead of -np.log, I give it a try and worked much better. What is really expecting #pm.stochastic? I'm guessing there was a small confusion on the sign required due to a very popular example using something like np.log(1/(1+t_1-t_0)) which was written as -np.log(1+t_1-t_0)
Another question: What is this decorator doing with the value argument? As I understand it, we start with some proposed value for the priors that need to enter in the likelihood and the idea of #pm.stochastic is basically produce some number to compare this likelihood to the number generated by the previous iteration in the sampling process. The likelihood should receive the value argument and some values for the priors, but I'm not sure if this is all value is doing because that's the only required argument and yet I can write:
#pm.stochastic(observed=True)
def loglike(value=[1]):
data = [3,5,1] # some data
# some calculations that generate a numeric result
return np.log(result)
And as far as I can tell, that produces the same result as before. Maybe, it works in this way because I added observed=True to the decorator. If I would have tried this in a stochastic variable with observed=False by default, value would be changed in each iteration trying to obtain a better likelihood.
#pm.stochastic is a decorator, so it is expecting a function. The simplest way to use it is to give it a function that includes value as one of its arguments, and returns a log-likelihood.
You should use the #pm.stochastic decorator to define a custom prior for a parameter in your model. You should use the #pm.observed decorator to define a custom likelihood for data. Both of these decorators will create a pm.Stochastic object, which takes its name from the function it decorates, and has all the familiar methods and attributes (here is a nice article on Python decorators).
Examples:
A parameter a that has a triangular distribution a priori:
#pm.stochastic
def a(value=.5):
if 0 <= value < 1:
return np.log(1.-value)
else:
return -np.inf
Here value=.5 is used as the initial value of the parameter, and changing it to value=1 raises an exception, because it is outside of the support of the distribution.
A likelihood b that has is normal distribution centered at a, with a fixed precision:
#pm.observed
def b(value=[.2,.3], mu=a):
return pm.normal_like(value, mu, 100.)
Here value=[.2,.3] is used to represent the observed data.
I've put this together in a notebook that shows it all in action here.
Yes confusion is easy since the #stochastic returns a likelihood which is the opposite of the error essentially. So you take the negative log of your custom error function and return THAT as your log-likelihood.