We Keep Coding

How to make a biased random number out of a large set of numbers

I want to make Python 3.7.1 pick a number between 0 and 100. However I want a lower number to be much more likely than a higher number, in a reverse exponential-smooth-graduation-curve kind of way (doesn't have to be exact).
I guess I could start with
myrandomnumber = random.randint(0, 100)
And then link that to an array of some sort to determine differing percentages for each number. I've seen other people do that with random die rolls, but the thing is, that's quite neat for only for six possibilities, I want to do this for a hundred (or more) and don't want to sit there making a huge array with a hundred entries just for that. Of course I could do it this way I suppose, but I feel like Python probably has a really easy way to do this that I'm missing.
Thanks, folks!

What you probably want is a gamma distributed random number.
For example with a k=1 and θ=2.0:
There are algorithms for using the evenly-distributed random function to generate normal, exponential, or gamma distributed values.
But since you're in python, you could probably jump straight to using numpy's random.gamma function:
#the (1,2) shape ends basically at 20. Multiply by 5 to get my 0..100 scale
numpy.random.gamma(1, 2.0) * 5

I'm going with the assumption that you want to generate integer values over a bounded range, and that you mean non-uniformly distributed when you talk about "bias". Since you don't have a particular parametric distribution in mind, one approach is to start with a continuous distribution and take the "floor" of the outcomes using int(). You'll want to increase the upper bound by 1 so that rounding down gives values inclusive of that bound.
One easy choice is a triangular distribution. Python provides random.triangular() function, which takes 3 arguments—the lower bound, upper bound, and the mode. Here's a discretized version:
import random as rnd
import math
import sys
def triangle(upper_bound):
return int(rnd.triangular(0.0, float(upper_bound + 1) - sys.float_info.epsilon, 0.0))
I've subtracted float's epsilon from the upper bound to prevent the (extremely unlikely) chance of getting an outcome of 101 when your upper bound is 100. Another bounded distribution choice might be the beta distribution, which you could then scale and truncate.
If you want the distribution shifted even further down the scale towards 0, you could use distributions such as the exponential, or more generally the gamma, with truncation and rounding. Both of those have infinite support, so there are a couple of ways to truncate. The simpler way is to use acceptance/rejection—keep generating values until you get one in range:
def expo_w_rejection(upper_bound, scale_param = 0.4):
upper_bound += 1
while True:
candidate = rnd.expovariate(1.0 / (upper_bound * scale_param))
if candidate < upper_bound:
return int(candidate)
As before, bump the upper limit up by 1 to get outcomes that include the upper limit after truncating. I've also included an optional scale_param which should be a value strictly between 0 and 1, i.e., not inclusive of either limit. Values closer to 0 will cause the results to bunch more to the left, values closer to 1 yield less bunching.
The other method would be to use the inverse transform technique for generating, and to restrict the range of the uniform to not exceed the upper bound based on evaluating the cumulative distribution function at the target upper bound:
def trunc_exp(upper_bound, scale_param = 0.4):
upper_bound = float(upper_bound) + 1.0 - sys.float_info.epsilon
trunc = 1.0 - math.exp(-1.0 / scale_param)
return int((-upper_bound * scale_param) * math.log(1.0 - trunc * rnd.random()))
Both approaches yield distributionally simular results, as can be seen in the following screenshot. "Column 1" was generated with truncated inversion, while "Column 2" was generated with acceptance/rejection.

Two different value for the Wavelet Transform (Daubechies D4)

I am testing this code
protected final double sqrt_3 = Math.sqrt( 3 );
protected final double denom = 4 * Math.sqrt( 2 );
//
// forward transform scaling (smoothing) coefficients
//
protected final double h0 = (1 + sqrt_3)/denom;
protected final double h1 = (3 + sqrt_3)/denom;
protected final double h2 = (3 - sqrt_3)/denom;
protected final double h3 = (1 - sqrt_3)/denom;
//
// forward transform wavelet coefficients
//
protected final double g0 = h3;
protected final double g1 = -h2;
protected final double g2 = h1;
protected final double g3 = -h0;
protected void transform( double a[], int n )
{
if (n >= 4) {
int i, j;
int half = n >> 1;
double tmp[] = new double[n];
i = 0;
for (j = 0; j < n-3; j = j + 2) {
tmp[i] = a[j]*h0 + a[j+1]*h1 + a[j+2]*h2 + a[j+3]*h3;
tmp[i+half] = a[j]*g0 + a[j+1]*g1 + a[j+2]*g2 + a[j+3]*g3;
i++;
}
tmp[i] = a[n-2]*h0 + a[n-1]*h1 + a[0]*h2 + a[1]*h3;
tmp[i+half] = a[n-2]*g0 + a[n-1]*g1 + a[0]*g2 + a[1]*g3;
for (i = 0; i < n; i++) {
a[i] = tmp[i];
}
}
} // transform
to perform a Daubechies D4 wavelet transform on this discrete array:
[1,2,0,4,5,6,8,10]
the result is
- 0 : 1.638357430415108
- 1 : 3.6903274198537357
- 2 : -2.6439375651698196
- 3 : 79.01146993331695
- 4 : 7.399237211089009
- 5 : 0.3882285676537802
- 6 : -39.6029588778518
- 7 : -19.794010741818195
- 8 : -2.1213203435596424
- 9 : 0.0
but when I use python pywt.dwt on the same array, I get this:
import pywt
[cA, cD] = pywt.dwt([1,2,0,4,5,6,8,10], 'db4')
>>> >>> cA
array([ 7.14848277, 1.98754736, 1.9747116 , 0.95510018, 4.90207373,
8.72887094, 14.23995582])
>>> cD
array([-0.5373913 , -2.00492859, 0.01927609, 0.1615668 , -0.0823509 ,
-0.32289939, 0.92816281])
Beyond the different values, one has 10 items and the other 7.
what am I missing?

I have never used any of these codes, also, not really sure about your question! But, maybe, this information might help you to get closer to an answer to your question:
Daubechies 4 Wiki
Daubechies Coefficients Wiki
Before that, I think your input vector (signal) maybe too small to make Wavelet calculations come up right? Not sure though! Maybe, try something in 1x128 size.
Maybe, Java code is Fast Wavelet Transform. Guessing based on the following methods:
Code
/**
Forward Daubechies D4 transform
*/
public void daubTrans( double s[] )
{
final int N = s.length;
int n;
for (n = N; n >= 4; n >>= 1) {
transform( s, n );
}
}
/**
Inverse Daubechies D4 transform
*/
public void invDaubTrans( double coef[])
{
final int N = coef.length;
int n;
for (n = 4; n <= N; n <<= 1) {
invTransform( coef, n );
}
}
Based on above methods, it seems this would be "Fast Wavelet Transform", which I'm also not so sure about their calculations, you might look into this link.
There are so many so-called, similar "terms" on Wavelet transforms that it might be best to go through their math to see things, and find out what the exact method might be (e.g., Discrete Wavelet Transform, Continuous Wavelet Transform, Discrete with Packet Decomposition). Every library has some terminologies and assumptions and make different calculations. You might print to see, if you would get anything close to D4 Wavelet = {−0.1830127, −0.3169873, 1.1830127, −0.6830127}; for DB4, first. Or, you may do other testing to see, if the calculations are correct.
Methods of Decomposition in Wavelets
It looks like cA and cD are coefficients of "Approximated" and "Details" signals decomposed by a discrete Wavelet transform. However, I'm not so sure, to how many layers you might have been decomposed your input vector.
There are two well-known ways of decomposing a signal in Wavelet, one is "packet" (which decomposes both "approximations" and "details" signals, so you would get 2^4=16 sub-signals for decomposing your original signal to 4 layers).
The other decomposition method decomposes the low-frequency part of signals. So, you might need to find out about your level of decomposition that your vector is being decomposed.
Also, if you write your own code, you can decompose it however you wish.
Simple Keys to Understand Wavelet
Shifting (Time) vs Scale (Frequency)
There is one simple thing that if you understand, then Wavelet becomes much easier. First, as you may know, Wavelet is a time-frequency method. However, instead of plotting time vs frequency, you do time vs scale, where scale is "inverse" of frequency.
Children of a Wavelet Function such as DB4
Wavelet transform maps a Wavelet function - such as DB4 - throughout your original signal, and that's how it would compute those numbers that you have printed out, perhaps. One thing to consider is to find a base function, DB4, that would "look like" you original signal. How do you do that?
Basically, you pick a base function, DB4, then Wavelet transform creates multiple forms of that base function (e.g., imagine you name them DB4-0, DB4-2, DB4-3, DB4-4, ...,DB4-15). These children are created based on:
(a) Shifting (in a for loop by incrementing time, sliding a child function, then calculating coefficients), shifting is in relation with time, obviously.
(b) Scaling (means "stretching" a Wavelet function, vertically or horizontally, which would changes the frequency nature of the base function, then sliding it again through time), which is converse relation with frequency, meaning that higher scale, lower frequencies, and vice versa.
Therefore, this depends on how many children functions you may need, based on the decompositions (sub-signals). If you have 16 sub-signals (4 level of decomposition with a packet method), then you will have 16 of those "children" functions mapping throughout your signal. And that's how coefficients vectors being calculated. Then, you may toss those unnecessary sub-signals, and keep focusing on those sub-signals (frequencies) that you might be interested in. The thing is Wavelet reserves (maintain) the time information, as opposed to Fourier.
Normal Decomposition
Also, since you are a good programmer, I'm pretty sure, you can quickly crack the code and I don't think you might be missing anything here. You can just go through their methods and read a few pages of wikipedia, and you would be probably there, if you wish so.
If you might have really exciting details questions, you may try DSP SE. So many signals experts are in there. Sorry about that! Wrote this too fast, also not a good writer/explainer, later hopefully others would edit and provide the right answer. Not really expert.
In short, you are not missing on anything, good method, good luck and best wishes!

Integer optimization/maximization in numpy

I need to estimate the size of a population, by finding the value of n which maximises scipy.misc.comb(n, a)/n**b where a and b are constants. n, a and b are all integers.
Obviously, I could just have a loop in range(SOME_HUGE_NUMBER), calculate the value for each n and break out of the loop once I reach an inflexion in the curve. But I wondered if there was an elegant way of doing this with (say) numpy/scipy, or is there some other elegant way of doing this just in pure Python (e.g. like an integer equivalent of Newton's method?)

As long as your number n is reasonably small (smaller than approx. 1500), my guess for the fastest way to do this is to actually try all possible values. You can do this quickly by using numpy:
import numpy as np
import scipy.misc as misc
nMax = 1000
a = 77
b = 100
n = np.arange(1, nMax+1, dtype=np.float64)
val = misc.comb(n, a)/n**b
print("Maximized for n={:d}".format(int(n[val.argmax()]+0.5)))
# Maximized for n=181
This is not especially elegant but rather fast for that range of n. Problem is that for n>1484 the numerator can already get too large to be stored in a float. This method will then fail, as you will run into overflows. But this is not only a problem of numpy.ndarray not working with python integers. Even with them, you would not be able to compute:
misc.comb(10000, 1000, exact=True)/10000**1001
as you want to have a float result in your division of two numbers larger than the maximum a float in python can hold (max_10_exp = 1024 on my system. See sys.float_info().). You couldn't use your range in that case, as well. If you really want to do something like that, you will have to take more care numerically.

You essentially have a nicely smooth function of n that you want to maximise. n is required to be integral but we can consider the function instead to be a function of the reals. In this case, the maximising integral value of n must be close to (next to) the maximising real value.
We could convert comb to a real function by using the gamma function and use numerical optimisation techniques to find the maximum. Another approach is to replace the factorials with Stirling's approximation. This gives a moderately complicated but tractable algebraic expression. This expression is not hard to differentiate and set to zero to find the extrema.
I did this and obtained
n * (b + (n-a) * log((n-a)/n) ) = a * b - a/2
This is not straightforward to solve algebraically but easy enough numerically (e.g. using Newton's method, as you suggest).
I may have made a mistake in the algebra, but I typed the a = 77, b = 100 example into Wolfram Alpha and got 180.58 so the approach seems to work.

python 3.1 - Creating normal distribution

I have scipy and numpy, Python v3.1
I need to create a 1D array of length 3million, using random numbers between (and including) 100-60,000. It has to fit a normal distribution.
Using 'a = numpy.random.standard_normal(3000000)', I get a normal distribution for that required length; not sure how to achieve the required range.

A standard normal distribution has mean 0 and standard deviation 1. What I understand from your requirements is that you need a ((60000-100)/2, (60000-100)/2) one. Take each value from standard_normal() result, multiply it by the new variance, and add the new mean.
I haven't used NumPy, but a quick search of the docs says that you can achieve what you want directly bu using numpy.random.normal()
One last tidbit: normal distributions are not bounded. That means there isn't a value with probability zero. Your requirements should be in terms of means and variances (or standard deviations), and not of limits.

If you want a truly random normal distribution, you can't guarentee how far the numbers will spread. You can reduce the probability of outliers, however, by specifying the standard deviation
>>> n = 3000000
>>> sigma5 = 1.0 / 1744278
>>> n * sigma5
1.7199093263803131 # Expect one values in 3 mil outside range at 5 stdev.
>>> sigma6 = 1.0 / 1 / 506800000
>>> sigma6 = 1.0 / 506800000
>>> n * sigma6
0.0059194948697711127 # Expect 0.005 values in 3 mil outside range at 6 stdev.
>>> sigma7 = 1.0 / 390600000000
>>> n * sigma7
7.6804915514592934e-06
Therefore, in this case, ensuring that the standard deviation is only 1/6 or 1/7 of half the range will give you reasonable confidence that your data will not exceed the range.
>>> range = 60000 - 100
>>> spread = (range / 2) / 6 # Anything outside of the range will be six std. dev. from the mean
>>> mean = (60000 + 100) / 2
>>> a = numpy.random.normal(loc = mean, scale = spread, size = n)
>>> min(a)
6320.0238199673404
>>> max(a)
55044.015566089176
Of course, you can still can values that fall outside the range here

try this nice little method:
You'll want a method that just makes one random number.
import random
list = [random.randint(min,max) for i in range(numitems)]
This will give you a list with numitems random numbers between min and max.
Of course, 3000000 is a lot of items to have in memory. Consider making the random numbers as they are needed by the program.

Range of python's random.random() from the standard library

Does python's random.random() ever return 1.0 or does it only return up until 0.9999..?

>>> help(random.random)
Help on built-in function random:
random(...)
random() -> x in the interval [0, 1).
That means 1 is excluded.

Docs are here: http://docs.python.org/library/random.html
...random(), which
generates a random float uniformly in
the semi-open range [0.0, 1.0).
So, the return value will be greater than or equal to 0, and less than 1.0.

The other answers already clarified that 1 is not included in the range, but out of curiosity, I decided to look at the source to see precisely how it is calculated.
The CPython source can be found here
/* random_random is the function named genrand_res53 in the original code;
* generates a random number on [0,1) with 53-bit resolution; note that
* 9007199254740992 == 2**53; I assume they're spelling "/2**53" as
* multiply-by-reciprocal in the (likely vain) hope that the compiler will
* optimize the division away at compile-time. 67108864 is 2**26. In
* effect, a contains 27 random bits shifted left 26, and b fills in the
* lower 26 bits of the 53-bit numerator.
* The orginal code credited Isaku Wada for this algorithm, 2002/01/09.
*/
static PyObject *
random_random(RandomObject *self)
{
unsigned long a=genrand_int32(self)>>5, b=genrand_int32(self)>>6;
return PyFloat_FromDouble((a*67108864.0+b)*(1.0/9007199254740992.0));
}
So the function effectively generates m/2^53 where 0 <= m < 2^53 is an integer. Since floats have 53 bits of precision normally, this means that on the range [1/2, 1), every possible float is generated. For values closer to 0, it skips some possible float values for efficiency but the generated numbers are uniformly distributed within the range. The largest possible number generated by random.random is precisely
0.99999999999999988897769753748434595763683319091796875

Python's random.random function returns numbers that are less than, but not equal to, 1.
However, it can return 0.

From the code in Antimony's answers it is easy to see that random.random() never returns exactly 1.0 on platforms that have at least 53 bit mantissa for calculations involving constants not annotated with 'f' in C. That's the precision IEEE 754 prescribes and is standard today.
However, on platforms with lower precision, for example if Python is compiled with -fsingle-precision-constant for use on an embedded platform, adding b to a*67108864.0 can result in rounding up to 2^53 if b is close enough to 2^26 and this would mean that 1.0 is returned. Note that this happens regardless of what precision Python's PyFloat_FromDouble function uses.
One way to test for this would be to check a few hundred random numbers whether the 53rd bit is ever 1. If it is 1 at least once this proofs sufficient precision and you are fine. If not, rounding is the most likely explanation meaning that random.random() can return 1.0. It's of course possible that you were just unlucky. You can push the certainty as high as you want by testing more numbers.