I want to create a customized distribution based on a Levy truncated law, which reads
p(r) = (r + r0)**(-beta)*exp(-r/k).
So I defined it in the following way:
import numpy as np
import scipy.stats as st
class LevyPDF(st.rv_continuous):
def _pdf(self,r):
r0 = 100
k = 1500
beta = 1.6
return (r + r0)**(-beta)*np.exp(-r/k)
Suppose that I want to find the distribution of distances between r = 0 and r = 50km. Then:
nmin = 0
nmax = 50
my_cv = LevyPDF(a=nmin, b=nmax, name='LevyPDF')
x = np.linspace(nmin, nmax, (nmax-nmin)*2)
I do not understand why:
sum(my_cv.cdf(x)) = 2.22
instead of 1.
Then how can I define an histogram of N = 2000000 random distances based on the distribution that I defined?
Using your minimal example (slightly adapted):
import scipy.stats as st
import numpy as np
import matplotlib.pyplot as plt
class LevyPDF(st.rv_continuous):
def _pdf(self,r):
r0 = 100
k = 1500
beta = 1.6
return (r + r0)**(-beta)*np.exp(-r/k)
nmin = 0
nmax = 50
my_cv = LevyPDF(a=nmin, b=nmax, name='LevyPDF')
To sample from your random variable, use rvs() method from rv_continuous class:
N = 50000
X = my_cv.rvs(size=N, random_state=1)
Will return an array of size (N,) with random variates sampled from your distribution. Use random_state option to freeze your example and make your script repeatable (it defines random seed for your sampling).
Note as N softly increases, computation time drastically increases.
To plot histogram, use matplotlib library, see hist:
fig, axe = plt.subplots()
n, bins, patches = axe.hist(X, 50, normed=1, facecolor='green', alpha=0.75)
plt.show(axe)
Bellow a example of sampling from Chi Square with 40 Degrees of Freedom:
from scipy import stats
import numpy as np
import matplotlib.pyplot as plt
rv = stats.chi2(40)
N = 200000
X = rv.rvs(size=N, random_state=1)
fig, axe = plt.subplots()
n, bins, patches = axe.hist(X, 50, normed=1, facecolor='green', alpha=0.75)
plt.show(axe)
It leads to:
Related
So let's say we have a random variable M and it's continuous. Its PDF function is (4x^3)/81 for 0≤x≤3 and 0 for outside of this interval. And let's say we also have another continuous random variable N that depends on M like N = M^2 + 3. How can I plot the joint CDF of M and N?
Stackoverflow won't let me type this joint CDF derivation in because it thinks it is improperly formatted code, but here is an image of it.
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
m = np.linspace(0,3,100)
n = np.linspace(3,12,100)
m_mesh,n_mesh = np.meshgrid(m,n)
joint_cdf = np.minimum(m_mesh,np.sqrt(n_mesh-3))**4/81
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(m_mesh, n_mesh, joint_cdf, cmap=cm.coolwarm,linewidth=0, antialiased=False)
I have used seaborn's kdeplot on some data.
import seaborn as sns
import numpy as np
sns.kdeplot(np.random.rand(100))
Is it possible to return the fwhm from the curve created?
And if not, is there another way to calculate it?
You can extract the generated kde curve from the ax. Then get the maximum y value and search the x positions nearest to the half max:
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
ax = sns.kdeplot(np.random.rand(100))
kde_curve = ax.lines[0]
x = kde_curve.get_xdata()
y = kde_curve.get_ydata()
halfmax = y.max() / 2
maxpos = y.argmax()
leftpos = (np.abs(y[:maxpos] - halfmax)).argmin()
rightpos = (np.abs(y[maxpos:] - halfmax)).argmin() + maxpos
fullwidthathalfmax = x[rightpos] - x[leftpos]
ax.hlines(halfmax, x[leftpos], x[rightpos], color='crimson', ls=':')
ax.text(x[maxpos], halfmax, f'{fullwidthathalfmax:.3f}\n', color='crimson', ha='center', va='center')
ax.set_ylim(ymin=0)
plt.show()
Note that you can also calculate a kde curve from scipy.stats.gaussian_kde if you don't need the plotted version. In that case, the code could look like:
import numpy as np
from scipy.stats import gaussian_kde
data = np.random.rand(100)
kde = gaussian_kde(data)
x = np.linspace(data.min(), data.max(), 1000)
y = kde(x)
halfmax = y.max() / 2
maxpos = y.argmax()
leftpos = (np.abs(y[:maxpos] - halfmax)).argmin()
rightpos = (np.abs(y[maxpos:] - halfmax)).argmin() + maxpos
fullwidthathalfmax = x[rightpos] - x[leftpos]
print(fullwidthathalfmax)
I don't believe there's a way to return the fwhm from the random dataplot without writing the code to calculate it.
Take into account some example data:
import numpy as np
arr_x = np.linspace(norm.ppf(0.00001), norm.ppf(0.99999), 10000)
arr_y = norm.pdf(arr_x)
Find the minimum and maximum points and calculate difference.
difference = max(arr_y) - min(arr_y)
Find the half max (in this case it is half min)
HM = difference / 2
Find the nearest data point to HM:
nearest = (np.abs(arr_y - HM)).argmin()
Calculate the distance between nearest and min to get the HWHM, then mult by 2 to get the FWHM.
I want to know how scipy.stats uses its methods fit and pdf. According to the documentation, fit(data, a, loc = 0, scale = 1) estimates parameters for data while pdf(x, a, loc=0, scale=1) computes probability density function . But I couldn't find how fit and pdf are actually performed, statistically and mathematically.
I am using the sm.datasets.elnino data, using the code from tmthydvnprt
import warnings
import numpy as np
import pandas as pd
import scipy.stats as st
import statsmodels as sm
import matplotlib
import matplotlib.pyplot as plt
data = pd.Series(sm.datasets.elnino.load_pandas().data.set_index('YEAR').values.ravel())
y, x = np.histogram(data, bins = 50, density = True)
x = (x + np.roll(x, -1))[:-1] / 2.0
distribution = st.gennorm
params = distribution.fit(data)
arg = params[:-2]
loc = params[-2]
scale = params[-1]
pdf = distribution.pdf(x, loc = loc, scale = scale, *arg)
sse = np.sum(np.power(y - pdf, 2.0))
Using data, arg = 4.3836, loc = 23.2991, scale = 3.8499.
I want to know what arg, loc, and scale represent and how they are calculated.
Thank you.
I have a set of integer values, and I want to set them to Weibull distribution and get the best fit parameters. Then I draw the histogram of data together with the pdf of Weibull distribution, using the best fit parameters. This is the code I used.
from jtlHandler import *
import warnings
import numpy as np
import pandas as pd
import scipy.stats as st
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
def get_pdf(latencies):
a = np.array(latencies)
ag = st.gaussian_kde(a)
ak = np.linspace(np.min(a), np.max(a), len(a))
agv = ag(ak)
plt.plot(ak,agv)
plt.show()
return (ak,agv)
def fit_to_distribution(distribution, data):
params = distribution.fit(data)
# Return MLEs for shape (if applicable), location, and scale parameters from data.
#
# MLE stands for Maximum Likelihood Estimate. Starting estimates for the fit are given by input arguments; for any arguments not provided with starting estimates, self._fitstart(data) is called to generate such.
return params
def make_distribution_pdf(dist, params, end):
arg = params[:-2]
loc = params[-2]
scale = params[-1]
# Build PDF and turn into pandas Series
x = np.linspace(0, end, end)
y = dist.pdf(x, loc=loc, scale=scale, *arg)
pdf = pd.Series(y, x)
return pdf
latencies = getLatencyList("filename")
latencies = latencies[int(9*(len(latencies)/10)):len(latencies)]
data = pd.Series(latencies)
params = fit_to_distribution(st.weibull_max, data)
print("Parameters for the fit: "+str(params))
# Make PDF
pdf = make_distribution_pdf(st.weibull_max, params, max(latencies))
# Display
plt.figure()
ax = pdf.plot(lw=2, label='PDF', legend=True)
data.plot(kind='hist', bins=200, normed=True, alpha=0.5, label='Data',
legend=True, ax=ax)
ax.set_title('Weibull distribution')
ax.set_xlabel('Latnecy')
ax.set_ylabel('Frequency')
plt.savefig("image.png")
This is the resulting figure.
As it is seen, the Weibull approximation is not simmilar to the original distribution of data.
How can I get the best Weibull approximation to my data?
You can fit a data set (set of numbers) to any distribution using the following two methods.
import os
import matplotlib.pyplot as plt
import sys
import math
import numpy as np
import scipy.stats as st
from scipy.stats._continuous_distns import _distn_names
from scipy.optimize import curve_fit
def fit_to_distribution(distribution, latency_values):
distribution = getattr(st, distribution)
params = distribution.fit(latency_values)
return params
def make_distribution_pdf(distribution, latency_list):
distribution = getattr(st, distribution)
params = distribution.fit(latency_list)
arg = params[:-2]
loc = params[-2]
scale = params[-1]
x = np.linspace(min(latency_list), max(latency_list), 10000)
y = distribution.pdf(x, loc=loc, scale=scale, *arg)
return x, y
I've plotted a 3-d mesh in Matlab by below little m-file:
[x,n] = meshgrid(0:0.1:20, 1:1:100);
mu = 0;
sigma = sqrt(2)./n;
f = normcdf(x,mu,sigma);
mesh(x,n,f);
I am going to acquire the same result by utilization of Python and its corresponding modules, by below code snippet:
import numpy as np
from scipy.integrate import quad
import matplotlib.pyplot as plt
sigma = 1
def integrand(x, n):
return (n/(2*sigma*np.sqrt(np.pi)))*np.exp(-(n**2*x**2)/(4*sigma**2))
tt = np.linspace(0, 20, 2000)
nn = np.linspace(1, 100, 100)
T = np.zeros([len(tt), len(nn)])
for i,t in enumerate(tt):
for j,n in enumerate(nn):
T[i, j], _ = quad(integrand, -np.inf, t, args=(n,))
x, y = np.mgrid[0:20:0.01, 1:101:1]
plt.pcolormesh(x, y, T)
plt.show()
But the output of the Python is is considerably different with the Matlab one, and as a matter of fact is unacceptable.
I am afraid of wrong utilization of the functions just like linespace, enumerate or mgrid...
Does somebody have any idea about?!...
PS. Unfortunately, I couldn't insert the output plots within this thread...!
Best
..............................
Edit: I changed the linespace and mgrid intervals and replaced plot_surface method... The output is 3d now with the suitable accuracy and smoothness...
From what I see the equivalent solution would be:
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
x, n = np.mgrid[0:20:0.01, 1:100:1]
mu = 0
sigma = np.sqrt(2)/n
f = norm.cdf(x, mu, sigma)
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(x, n, f, rstride=x.shape[0]//20, cstride=x.shape[1]//20, alpha=0.3)
plt.show()
Unfortunately 3D plotting with matplotlib is not as straight forward as with matlab.
Here is the plot from this code:
Your Matlab code generate 201 points through x:
[x,n] = meshgrid(0:0.1:20, 1:1:100);
While your Python code generate only 20 points:
tt = np.linspace(0, 19, 20)
Maybe it's causing accuracy problems?
Try this code:
tt = np.linspace(0, 20, 201)
The seminal points to resolve the problem was:
1- Necessity of the equivalence regarding the provided dimensions of the linespace and mgrid functions...
2- Utilization of a mesh with more density to make a bee line into a high degree of smoothness...
3- Application of a 3d plotter function, like plot_surf...
The current code is totally valid...