Is there a way to calculate cumulative distribution function without using scipy? - python

Is there a way to calculate the cumulative distribution function (using the probability density function)? I've seen many scipy and numpy ideas out, but is there any other way?

Related

A simple way to compute cumulative distribution function in Python

I'm trying to compute the distribution function of any of the usual distributions in Python... However, all the methods I've seen involve first drawing N samples from said distribution, and then order them somehow, and then do a cumulative sum.
In Mathematica, I can just do CDF[ChiSquaredDistribution[df],quantile]. If I want another distribution, I just substitute ChiSquaredDistribution for the name of that other distribution.
Is there a simple way, like in Mathematica, to compute a cumulative distribution function in Python?

Find underlaying normal distribution of random vectors

I am trying to solve a statistics-related real world problem with Python and am looking for inputs on my ideas: I have N random vectors from a m-dimensional normal distribution. I have no information about the means and the covariance matrix of the underlying distribution, in fact also that it is a normal distribution is only an assumption, a very plausible one though. I want to compute an approximation of the mean vector and covariance matrix of the distribution. The number of random vectors is in the order of magnitude of 100 to 300, the dimensionality of the normal distribution is somewhere between 2 and 5. The time for the calculation should ideally not exceed 1 minute on a standard home computer.
I am currently thinking about three approaches and am happy about all suggestions for other approaches or preferences between those three:
Fitting: Make a multi dimensional histogram of all random vectors and fit a multi dimensional normal distribution to the histogram. Problem about that approach: The covariance matrix has many entries, this could possibly be a problem for the fitting process?
Invert cumulative distribution function: Make a multi dimensional histogram as approximation of the density function of the random vectors. Then integrate this to get a multi dimensional cumulative distribution function. For one dimension, this is invertible and one could use the cum-dist function to distribute random numbers like in the original distribution. Problem: For the multi-dimensional case the cum-dist function is not invertible(?) and I don't know if this approach still works then?
Bayesian: Use Bayesian Statistics with some normal distribution as prior and update for every observation. The result should always be again a normal distribution. Problem: I think this is computationally expensive? Also, I don't want the later updates have more impact on the resulting distribution than the earlier ones.
Also, maybe there is some library which has this task already implemented? I did not find exactly this in Numpy or Scipy, maybe someone has an idea where else to look?
If the simple estimates described in the section Parameter estimation of the wikipedia article on the multivariate normal distribution are sufficient for your needs, you can use numpy.mean to compute the mean and numpy.cov to compute the sample covariance matrix.

Python - calculate normal distribution

I'm quite new to python world. Also, I'm not a statistician. I'm in the need to implementing mathematical models developed by mathematicians in a computer science programming language. I've chosen python after some research. I'm comfortable with programming as such (PHP/HTML/javascript).
I have a column of values that I've extracted from a MySQL database & in need to calculate the below:
Normal distribution of it. (I don't have the sigma & mu values. These need to be calculated too apparently).
Mixture of normal distribution
Estimate density of normal distribution
Calculate 'Z' score
The array of values looks similar to the one below ( I've populated sample data)-
data = [3,3,3,3,3,3,3,9,12,6,3,3,3,3,9,21,3,12,3,6,3,30,12,6,3,3,24,30,3,3,3,12,3,3,3,3,3,3,3,6,9,3,3,3,3,3,3,3,3,3,3,3,3,33,3,3,3,6,3,3,6,6,15,3,3,3,3,6,3,3,3,3,3,3,3,3,12,12,3,3,3,3,3,3,78,9,12,3,6,3,15,6,3,3,3,30,3,6,78,3,9,9,3,78,3,3,3,3,3,12,15,3,3,78,3,3,33,78,15,9,3,3,21,6,3,6,30,6,6,3,3,3,3,12,3,3,3,3,3,12,3,3,3,3,3,3,3,3,3,3,3,3,12,6,3,3,9,3,3,12,3,3,3,3,6,3,3,6,3,3,18,6,3,3,3,3,3,6,3,3,3,3,3,3,3,3,9,21,3,9,3,3,12,12,3,3,15,30,3,12,3,3,6,3,3,3,9,9,6,6,3,3,27,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,12,6,3,3,3,3,30,3,3,3,3,6,18,24,6,3,3,42,3,3,6,3,15,3,3,3,3,9,3,60,81,54,3,9,3,3,6,3,6,3,3,3,3,6,3,3,3,33,24,3,3,3,3,3,3,3,3,3,3,3,3,3,93,3,3,21,3,3,3,3,6,6,30,3,3,3,3,6,3,9,3,3,6,3,6,3,3,3,39,9,30,6,45,3,3,3,3,3,24,12,3,6,3,78,3,3,3,3,3,3,3,3,3,3,3,9,6,3,3,3,6,15,3,78,3,3,30,3,3,3,33,24,3,3,6,3,3,3,6,3,3,3,12,15,3,3,3,21,3,3,3,3,9,6,3,6,3,3,3,3,6,6,3,15,6,9,3,3,18,3,3,3,3,3,3,3,3,21,3,3,6,3,3,3,3,3,3,12,3,3,3,3,3,3,6,21,12,3,6,9,3,3,3,3,9,15,3,6,78,6,6,3,9,3,9,3,6,3,3,3,24,3,3,6,3,3,27,3,6,3,3,3,3,3,3,3,3,3,3,3,3,21,3,9,6,6,9,27,30,3,3,9,12,6,3,3,12,9,3,21,3,6,9,9,3,3,3,3,9,6,3,3,6,3,3,3,3,3,6,3,6,3,3,3,24,6,3,3,3,3,3,3,3,3,3,3,18,3,3,3,3,3,9,6,3,3,3,18,3,9,3,3,15,9,12,3,18,3,6,3,3,3,6,3,3,3,3,3,3,3,21,9,15,3,3,3,21,3,3,3,3,3,6,9,3,3,21,6,3,3,15,3,18,3,3,21,3,21,3,9,3,6,21,3,9,15,3,69,21,3,3,3,9,3,3,3,12,3,3,9,3,3,27,3,3,9,3,9,3,3,3,3,3,30,3,12,21,18,27,3,3,12,3,6,3,30,3,21,9,15,6,3,3,3,15,9,12,12,33,3,3,30,3,6,6,21,3,3,12,3,3,6,51,3,3,3,3,12,3,6,3,9,78,21,3,3,21,18,6,12,3,3,3,21,9,6,3,3,3,3,3,3,6,3,6,27,3,3,3,3,3,3,12,3,3,3,3,6,3,18,3,3,15,3,3,18,9,6,3,3,24,3,6,12,30,3,12,24,3,3,3,9,3,12,27,3,3,6,3,9,3,9,3,15,3,6,3,3,9,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,6,3,3,3,9,15,3,3,3,3,9,3,6,3,3,3,3,27,3,6,3,3,3,3,3,3,3,3,3,3,9,3,3,3,12,3,3,3,27,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,9,3,3,3,3,3,3,15,3,3,3,3,3,3,12,3,6,6,3,3,3,3,6,3,3,6,3,3,3,3,3,6,3,3,3,3,6,12,6,3,3,3,3,6,3,3,3,3,3,3,3,3,3,6,3,6,3,3,6,3,3,6,3,3,3,6,6,6,3,3,27,3,3,3,3,3,3,3,27,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,6,3,3,3,6,3,54,75,3,57,3,6,27,18,3,3,3,3,27,3,3,3,3,3,9,3,27,3,3,6,6,30,3,3,6,3,3,3,6,15,3,6,3,3,6,3,3,3,3,6,3,3,27,9,3,18,3,3,6,6,3,9,3,3,3,6,3,3,3,3,3,3,3,3,6,3,3,3,6,3,3,6,3,3,3,3,6,6,3,3,3,6,6,3,3,3,3,3,3,3,6,3,3,6,3,3,3,3,3,6,3,18,3,3,6,3,6,3,3,3,3,3,3,3,3,6,15,3,6,15,6,3,3,3,3,3,3,3,3,3,3,3,3,6,3,6,3,3,6,12,3,3,6,3,3,6,3,3,3,3,3,27,3,3,3,3,9,3,27,3,3,27,3,3,3,3,3,3,9,6,3,9,3,6,3,3,6,3,6,3,3,3,6,3,3,6,3,18,3,3,3,9,6,3,3,3,3,3,6,3,6,6,3,18,27,3,3,3,6,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,21,3,3,3,3,6,9,3,3,3,3,3,3,6,3,6,3,3,3,3,3,6,3,6,3,3,3,3,3,18,3,3,18,3,3,3,3,6,3,3,3,18,6,3,3,3,3,3,3,3,6,3,3,3,6,3,3,3,3,3,3,6,3,3,3,3,3,3,6,3,3,6,3,6,3,3,3,6,3,3,6,3,3,3,3,6,3,3,3,6,3,3,3,3,3,3,3,6,6,3,3,3,3,3,6,3,6,3,54,3,6,3,6,6,6,3,3,3,3,3,3,6,3,3,6,3,3,6,3,3,9,12,3,6,3,3,3,3,3,6,6,3,3,3,3,6,3,6,3,3,3,3,3,3,3,3,6,3,3,3,3,3,6,3,3,3,3,3,12,3,3,6,9,27,21,3,3,3,3,3,21,6,3,3,3,3,3,3,3,3,3,3,3,6,3,3,12,3,3,3,3,3,3,3,3,3,3,3,6,3,3,6,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,9,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,6,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,6,3,3,3,3,6,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,6,3,3,3,3,3,3,6,6,3,3,3,3,3,3,6,3,3,6,3,3,3,6,3,3,3,3,6,6,3,6,3,6,6,3,9,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,6,3,3,3,9,9,3,3,3,3,3,6,3,3,3,3,6,3,3,3,3,6,3,3,3,3,3,6,3,6,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,6,3,3,6,3,3,3,3,3,3,3,6,3,3,3,135,3,9,3,3,6,9,3,3,3,6,3,3,3,3,6,3,3,6,6,3,3,3,3,3,3,3,3,3,3,3,3,6,6,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,6,3,3,3,135,3,3,3,6,3,3,3,3,6,6,3,3,69,87,57,9,3,3,3,12,3,6,3,3,3,6,3,3,3,3,3,3,3,3,3,3,6,9,12,3,3,3,3,3,3,3,3,6,3,3,9,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,9,3,3,3,3,12,3,3,33,3,6,3,3,3,3,3,3,6,3,6,3,3,6,3,3,3,6,3,6,3,3,6,3,3,3,6,3,3,6,3,3,3,6,3,3,3,3,9,3,3,6,6,3,3,3,6,6,3,3,3,3,3,3,6,3,3,3,3,6,3,3,3,6,3,18,3,6,3,3,3,3,9,3,3,3,3,3,3,6,3,3,6,3,3,3,3,3,135,3,9,3,3,3,3,3,3,3,3,6,6,3,6,6,3,3,6,3,3,3,6,6,3,3,3,3,6,9,3,3,3,3,3,3,6,6,3,3,3,3,3,3,135,3,3,3,6,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,6,6,6,3,3,3,6,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,9,6,3,3,3,9,3,3,3,3,9,3,3,3,3,3,3,3,3,3,9,3,6,6,3,6,3,3,6,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,9,3,24,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,6,3,3,3,3,3,3,6,3,135,3,3,3,3,3,3,6,6,3,3,3,3,3,3,3,3,6,3,3,3,3,3,9,6,3,3,3,9,3,3,3,3,3,3,6,3,3,6,3,9,3,3,3,6,3,3,3,6,6,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,9,3,3,3,3,3,9,6,3,9,3,6,3,3,21,9,3,3,3,6,3,3,3,3,6,3,3,3,3,9,3,3,3,3,3,3,3,135,3,6,6,6,3,6,3,3,9,6,6,3,3,3,3,3,3,9,3,6,3,3,3,3,3,3,3,6,9,6,3,3,6,3,6,6,3,3,3,3,6,3,6,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,6,3,6,3,12,3,24,3,3,3,3,3,3,21,3,3,3,3,3,3,3,6,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,15,3,3,3,3,3,3,3,6,3,3,6,6,3,3,9,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,9,3,3,3,6,3,3,3,6,3,6,3,3,3,3,3,3,3,3,3,12,3,3,3,3,3,3,6,3,6,6,3,3,3,6,3,3,6,3,3,3,3,9,6,3,3,3,6,9,3,3,3,6,9,3,6,3,3,3,3,3,3,6,3,3,3,3,6,6,3,3,3,3,3,3,3,3,3,3,9,15,3,3,3,6,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,12,3,3,3,6,6,6,3,3,3,6,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,12,12,6,3,3,3,3,3,3,3,3,3,9,6,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,6,3,3,3,3,6,3,3,3,6,3,3,3,3,3,3,3,6,3,3,3,6,3,3,6,3,3,12,3,3,3,6,3,3,3,3,564,84,3,60,6,15,3,3,3,3,3,6,3,3,3,3,3,3,3,9,3,3,3,3,3,3,3,3,3,3,3,6,9,3,3,3,3,3,9,3,3,3,3,3,12,6,3,3,3,3,3,3,3,3,6,3,3,3,3,9,57,3,6,3,6,3,3,6,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,9,3,3,3,3,6,3,3,3,6,12,3,6,3,3,3,3,3,3,3,3,6,3,6,3,3,3,6,3,3,6,3,3,36,3,3,6,6,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,12,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,6,3,3,6,3,6,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,3,3,3,3,12,6,3,3,3,3,3,3,3,12,3,3,3,6,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,9,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,12,3,3,3,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,9,3,3,3,3,3,3,3,9,3,3,3,3,3,3,3,3,3,6,3,3,3,3,3,3,3,3,3,3,6,3,3,3,27,3,3,6,3,3,3,3,3,6,3,3,3,3,6,3,3,9,3,3,3,12,3,3,3,3,3,6,9,3,6,3,3]
I've looked around & found quite a bit about cumulative distribution as here (These have the mu & sigma values ready anyway which isn't the case in my scenario). I'm not too sure if cumulative normal distribution & normal distribution are the same. Could I please get some pointers on how to get started with this please?
I'd very much appreciate any help here please.
A distribution and the cumulative distribution are not the same - the latter is the integral of the former. If the normal distribution looks like a "bell", the cumulative normal distribution looks like a gentle "step" function.
E.g., for the following "bells"
you'd get the following "steps"
If you have an array data, the following will fit it to a normal distribution using scipy.stats.norm:
import numpy as np
from scipy.stats import norm
mu, std = norm.fit(data)
This will return the mean and standard deviation, the combination of which define a normal distribution.
Normal and cumulative distributions are not the same. I'll leave that bit of research to you.
The formula for normal distribution is easy if you have the mean and standard deviation:
The thing that you may look at is the normal distribution not the cumulative normal distribution. You can calculate the frequency of each element that occurs in the array and plot it to visualize the distribution.
Then you can use numpy to calculate mean = numpy.mean(array) and standard deviation as std = numpy.std(array).
Hope this helps.

Studentized range statistic (q*) in Python Scipy

I am wondering if it is possible to find the Studentized range statistic (q*) in Python Scipy lib as an input into Tukey's HSD calculation, similar to interpolating a table such as this (http://cse.niaes.affrc.go.jp/miwa/probcalc/s-range/srng_tbl.html#fivepercent) or pulling from a continuous distribution.
I have found some guidance here (http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.tukeylambda.html#scipy.stats.tukeylambda), but lost on how to input the df (degrees of freedom) or k (# of samples groups).
I am looking for something like the critical F or critical t statistic, which can be obtained via
scipy.stats.f.isf(alpha, df-between, df-within)
or
scipy.stats.t.isf(alpha, df).
from statsmodels.stats.libqsturng import psturng, qsturng
provides cdf (or tail probabilities) and quantile function (inverse of cdf or of survival function, I don't remember)
It was written by Roger Lew as a package for interpolating the distribution of the studentized range statistic and was incorporated in statsmodels for use in tukeyhsd.
Until now it has only be used internally in statsmodels, and you would have to check the limitations and explanation in libqsturng.
As reference, statsmodels has a tukeyhsd function and a MultipleComparison class.
http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.multicomp.pairwise_tukeyhsd.html

How to calculate doubly-non central cumulative F distribution in Python

This is an alternative question to a previous one I asked here:
How to calculate inverse of cumulative doubly non-central f distribution in python
I found a way to avoid the inverse problem, so I just need to calculate the cumulative doubly non-central f distribution (DNCF) in python. I don't know much about statistics but wolfram indicates the DNCF is just a ratio of non-central chi squared distributions http://mathworld.wolfram.com/NoncentralF-Distribution.html.
Given my DNCF distribution with numerator degrees of freedom and non-centrality parameter (Dfn,Ncn) and denominator degrees of freedom/ non-centrality parameter (Dfd,Ncd) would it be accurate to simply do something like this:
def DNCF(x,Dfn,Dfd,Ncn,Ncd):
out=scipy.stats.ncx2.cdf(x,Dfn,Ncn)/scipy.stats.ncx2.cdf(x,Dfd,Ncd)
return out
This seems like it would work to me, but I would appreciate some verification from someone who knows more about statistics.
Thanks

Categories