We Keep Coding

Finding the value of machine epsilon using python

I wrote a simple code in python that gives me the same value of machine epsilon using the numpy command:np.finfo(float).eps
The code is:
eps=1
while eps+1 != 1:
eps /= 2
print(eps)
But I didn't want to stop here ! I used smaller and smaller numbers to divide eps, for example:
eps=1
while eps+1 != 1:
eps /= 1.1
print (eps)
With this, I got a value of 1.158287085355336e-16 for epsilon. I noticed that epsilon was converging to a number, my last attempt at 1.0000001 gave me the value of 1.1102231190697707e-16.
Is this value closer to the real value of epsilon for my Pc? I think I'm not considering something important and my line of thinking is wrong.
Thank you in advance for the help !

The term “machine epsilon” is not consistently defined, so it's better to avoid that word and instead to say what specifically you're talking about:
ulp(1), the magnitude of the least significant digit, or Unit in the Last Place, of 1.
This is the distance from 1 to the next larger floating-point number.
More generally, ulp(𝑥) is the distance from 𝑥 to the next larger floating-point number in magnitude.
In binary64 floating-point, with 53 bits of precision, ulp(1) is 2⁻⁵² ≈ 2.220446049250313 × 10⁻¹⁶.
In decimal64 floating-point, with 16 digits of precision, ulp(1) is 10⁻¹⁵.
In general, for floating-point in radix 𝛽 with 𝑝 digits of precision (including the implicit 1 digit), ulp(1) = 𝛽1 − 𝑝.
The relative error bound, sometimes also called unit roundoff or u.
A floating-point operation may round the outcome of a mathematical function such as 𝑥 + 𝑦, giving fl(𝑥 + 𝑦) = (𝑥 + 𝑦)⋅(1 + 𝛿) for some relative error 𝛿.
For basic arithmetic operations in IEEE 754 (+, −, *, /, sqrt), the result of computing the floating-point operation is guaranteed to be correctly rounded, which in the default rounding mode means it yields the nearest floating-point number, or one of the two nearest such numbers if 𝑥 + 𝑦 lies exactly halfway between them.
In binary64 floating-point, with 53 bits of precision, the relative error of an operation correctly rounded to nearest is at most 2⁻⁵³ ≈ 1.1102230246251565 × 10⁻¹⁶.
In decimal64 floating-point, with 16 digits of precision, the relative error of an operation correctly rounded to nearest is at most 5 × 10⁻¹⁶.
In general, when floating-point arithmetic in radix 𝛽 with 𝑝 digits of precision is correctly rounded to nearest, 𝛿 is bounded in magnitude by the relative error bound (𝛽/2) 𝛽−𝑝.
What the Python iteration while 1 + eps != 1: eps /= 2 computes, starting with eps = 1., is the relative error bound in binary64 floating-point, since that's the floating-point that essentially all Python implementations use.
If you had a version of Python that worked in a different radix, say b, you would instead want to use while 1 + eps != 1: eps /= b.
If you do eps /= 1.1 or eps /= 1.0001, you will get an approximation to the relative error bound erring on the larger side, with no particular significance to the result.
Note that sys.float_info.epsilon is ulp(1), rather than the relative error bound.
They are always related: ulp(1)/2 is the relative error bound in every floating-point format.

If you want the actual machine epsilon for a Python float on your PC, you can get it from the epsilon attribute of sys.float_info. By the way, on my x86-64 machine, numpy.finfo(float) gives me 2.220446049250313e-16, which is the expected machine epsilon for a 64-bit float.
Your intuition for trying to find the value eps such that 1 + eps != 1 is True is good, but machine epsilon is an upper bound on the relative error due to rounding in floating-point arithmetic. Not to mention, the inexact nature of floating-point arithmetic can be mystifying sometimes: note that 0.1 + 0.1 + 0.1 != 0.3 evaluates to True. Also, if I modify your code to
eps = 1
while eps + 9 != 9:
eps = eps / 1.0000001
print(eps)
I get, after around maybe half a minute,
8.881783669690459e-16
The relative error in this case is 8.881783669690459e-16 / 9 = 9.868648521878288e-17.

Your program is almost fine.
IT should be
eps=1
while eps+1 != 1:
eps /= 2
print(2*eps)
The result is
2.220446049250313e-16
Which is the epsilon of the machine. You can check this with this piece of
code:
import sys
sys.float_info.epsilon
The reason we should multiply by 2 in the last line is because you went 1 division too far inside the while loop.

Python Numerical Differentiation and the minimum value for h

I calculate the first derivative using the following code:
def f(x):
f = np.exp(x)
return f
def dfdx(x):
Df = (f(x+h)-f(x-h)) / (2*h)
return Df
For example, for x == 10 this works fine. But when I set h to around 10E-14 or below, Df starts
to get values that are really far away from the expected value f(10) and the relative error between the expected value and Df becomes huge.
Why is that? What is happening here?

The evaluation of f(x) has, at best, a rounding error of |f(x)|*mu where mu is the machine constant of the floating point type. The total error of the central difference formula is thus approximately
2*|f(x)|*mu/(2*h) + |f'''(x)|/6 * h^2
In the present case, the exponential function is equal to all of its derivatives, so that the error is proportional to
mu/h + h^2/6
which has a minimum at h = (3*mu)^(1/3), which for the double format with mu=1e-16 is around h=1e-5.
The precision is increased if instead of 2*h the actual difference (x+h)-(x-h) between the evaluation points is used in the denominator. This can be seen in the following loglog plot of the distance to the exact derivative.

You are probably encountering some numerical instability, as for x = 10 and h =~ 1E-13, the argument for np.exp is very close to 10 whether h is added or subtracted, so small approximation errors in the value of np.exp are scaled significantly by the division with the very small 2 * h.

In addition to the answer by #LutzL I will add some info from a great book Numerical Recipes 3rd Edition: The Art of Scientific Computing from chapter 5.7 about Numerical Derivatives, especially about the choice of optimal h value for given x:
Always choose h so that h and x differ by an exactly representable number. Funny stuff like 1/3 should be avoided, except when x is equal to something along the lines of 14.3333333.
Round-off error is approximately epsilon * |f(x) * h|, where epsilon is floating point accuracy, Python represents floating point numbers with double precision so it's 1e-16. It may differ for more complicated functions (where precision errors arise further), though it's not your case.
Choice of optimal h: Not getting into details it would be sqrt(epsilon) * x for simple forward case, except when your x is near zero (you will find more information in the book), which is your case. You may want to use higher x values in such cases, complementary answer is already provided. In the case of f(x+h) - f(x-h) as in your example it would amount to epsilon ** 1/3 * x, so approximately 5e-6 times x, which choice might be a little difficult in case of small values like yours. Quite close (if one can say so bearing in mind floating point arithmetic...) to practical results posted by #LutzL though.
You may use other derivative formulas, except the symmetric one you are using. You may want to use the forward or backward evaluation(if the function is costly to evaluate and you have calculated f(x) beforehand. If your function is cheap to evaluate, you may want to evaluate it multiple times using higher order methods to make the precision error smaller (see five-point stencil on wikipedia as provided in the comment to your question).

This Python tutorial explains the reason behind the limited precision. In summary, decimals are ultimately represented in binary and the precision is about 17 significant digits. So, you are right that it gets fuzzy beyond 10E-14.

Python: When finding the product of a large array, how best to reduce floating point error?

Suppose I have a large array with a bunch of floats in it, and I need to find the product while losing as little precision as possible to floating point errors:
import numpy as np
randoms = np.random.uniform(0.5, 1.61, 10000)
print(randoms[0:10])
array([ 1.01422339, 0.65581167, 0.8154046 , 1.49519379, 0.96114304,
1.20167417, 0.93667198, 0.66899907, 1.26731008, 1.59689486])
A presumably bad approach is to loop through the array and iteratively multiply. This will obviously have an error that compounds with each multiplication, so should be avoided if possible:
product_1 = 1
for i in randoms:
product_1 = product_1 * i
print(product_1)
64355009.758539267
The next method is to use numpy's built-in prod function, however this returns the exact same value as above, indicating that this is how prod is actually computing it:
product_2 = np.prod(randoms)
print(product_2)
64355009.758539267
print(product_1 == product_2)
True
A third way is to compute the logarithm of every term, sum them, and exponentiate at the end. Each logarithm is computed separately so there isn't the same compounding of the error, but the logarithm process and the exponentiation process both introduce some error themselves. In any case it produces a different answer:
product_3 = np.exp(np.sum(np.log(randoms)))
print(product_3)
64355009.758538999
print(product_3 == product_1)
False
I know that I'm not losing that much precision in this example, but for what I actually need to do, the compounding errors do end up causing troubles, enough that I'm considering using a package that can do symbolic / arbitrary precision computation. So, which method is the best here? Are there other ways I haven't considered?

I tried some experiments. The code is below but first some comments.
It is possible to compute the result exactly by converting the values into exact rational numbers, computing the product exactly, and then performing a final conversion to a float. It can be done with the fractions module included with Python but it will eventually get very slow. I used the gmpy2 module for faster rational arithmetic.
There are some subtleties with the formatting of binary floating-point values for display. Recent versions of Python return the shortest possible decimal string that will result in the original value. numpy floats have a different formatting. And so does the gmpy2.mpfr type. And Decimal obviously used a different formatting rule. So I always convert the result calculated to a Python float.
In addition to the user-definable decimal precision of the Decimal type, I also used gmpy2.mpfr since it supports user-definable binary precision.
The program outputs several values:
Sequential multiplication using 53-bit precision (IEEE 64-bit format).
Exact value using rational arithmetic.
Using Decimal with 28 decimal digits of precision.
Using Decimal with a user-specified precision.
Using mpfr with a user-specified precision.
Using a recursive multiplication method to minimize the number of multiplications.
Here is the code. You can modify the Decimal and mpfr precision and test the accuracy.
import numpy as np
from gmpy2 import mpq, mpfr, get_context, round2
from decimal import Decimal, getcontext
randoms = np.random.uniform(0.5, 1.61, 10000)
# Sequential multiplication using 53-bit binary precision.
product_1 = 1
for i in randoms:
product_1 = product_1 * i
print("53-bit binary: ", float(product_1))
# Exact value by converting all floats to fractions and then a final
# conversion to float. Uses gmpy2 for speed.
product_2 = 1
for i in randoms:
product_2 = product_2 * mpq(i)
print("exact using mpq: ", float(mpfr(product_2, precision=53)))
# Decimal math with 28 decimal digits (~93 bits of precision.)
product_3 = 1
for i in randoms:
product_3 = product_3 * Decimal(i)
print("Decimal(prec=28): ", float(product_3))
# Choose your own decimal precision.
getcontext().prec=18
product_4 = 1
for i in randoms:
product_4 = product_4 * Decimal(i)
print("Decimal(prec=%s): %s" % (getcontext().prec, float(product_4)))
# Choose your own binary precision.
get_context().precision = 60
product_5 = 1
for i in randoms:
product_5 = product_5 * mpfr(i)
print("mpfr(precision=%s): %s" % (get_context().precision, float(product_5)))
# Recursively multiply pairs of numbers together.
def rmult(d):
if len(d) == 1:
return d[0]
# If the length is odd, extend with 1.
if len(d) & 1:
d.append(1)
temp = []
for i in range(len(d)//2):
temp.append(d[2*i] * d[2*i+1])
return rmult(temp)
print("recursive 53-bit: ", float(rmult(list(randoms))))
As a rough guideline, as the number of multiplications increase, the intermediate precision will need to increase. Rational arithmetic will effectively give you infinite intermediate precision.
How critical is it that the result is 100% accurate?

Rounding cmath complex numbers?

I have instances in my code where two complex numbers (using the cmath module) that should be exactly the same, do not cancel out due to the floating point precision of the base 2 system causing the numbers to deviate from each other by a small difference in value at some nth decimal place.
If they were floating numbers of sufficient size, it would be a simple matter of just rounding them to a decimal place where the value difference no longer exists.
How could I do the same for the real and imaginary parts of complex numbers represented using the cmath module?
e.g. The following two complex numbers should be exactly the same, how could I implement some code to ensure that the real and imaginary components of some complex number are rounded to the nearest ith decimal place of my choice?
(0.6538461538461539-0.2692307692307693j)
(0.6538461538461539-0.26923076923076916j)
One possible solution, recommended by jonrsharpe:
if abs(a - b) < threshold:
a = b

"round" does not work directly on complex numbers, but it does work separately on the real resp. imaginary part of the number, e.g. rounding on 4 digits:
x = 0.6538461538461539-0.2692307692307693j
x_real = round(x.real, 4)
x_imag = round(x.imag, 4)
x = x_real + x_imag * 1j

Numpy is defacto for numerical computing in python. Try np.round():
import numpy as np
x = 0.6538461538461539-0.2692307692307693j
print(np.round(x,decimals=3))
print(np.round(x,decimals=3)==np.round(x,decimals=3)) # True
print(np.round(x,decimals=3)==np.round(x,decimals=4)) # False

Rounding errors in quadratic solver

I'm pretty new to python and am trying to write some code to solve a given quadratic function. I'm having some trouble with rounding errors in floats, I think because I am dividing two numbers that are very large with a very small difference. (Also I'm assuming all inputs have real solutions for now.) I've put two different versions of the quadratic equation to show my problem. It works fine for most inputs, but when I try a = .001, b = 1000, c = .001 I get two answers that have a significant difference. Here is my code:
from math import sqrt
a = float(input("Enter a: "))
b = float(input("Enter b: "))
c = float(input("Enter c: "))
xp = (-b+sqrt(b**2-4*a*c))/(2*a)
xn = (-b-sqrt(b**2-4*a*c))/(2*a)
print("The solutions are: x = ",xn,", ",xp,sep = '')
xp = (2*c)/(-b-sqrt(b**2-4*a*c))
xn = (2*c)/(-b+sqrt(b**2-4*a*c))
print("The solutions are: x = ",xn,", ",xp,sep = '')

I'm no expert in the maths field but I believe you should use numpy (a py module for maths), due to internal number representation on computers your calculus will not match real math. (floating point arithmetics)
http://docs.python.org/2/tutorial/floatingpoint.html
Check this is almost exaclty what you want.
http://www.annigeri.in/2012/02/python-class-for-quadratic-equations.html

To get more precise results with floating point, be careful not to subtract similar quantities. For the quadratic x^2 + a x + b = 0 you know that the roots x1 and x2 make
b = x1 * x2
Compute the one with larger absolute value, and get the other one from this relation.

Solutions:
Numpy as suggested by user dhunter is usually the best solution for math in python. The numpy libraries are capable of doing quick and accurate math in a number of different fields.
Decimal data types were added in python 2.4 If you do not want to download an external library and do not anticipate doing many long or complex equations, decimal datatypes may fit the bill.
Simply add:
from decimal import *
to the top of your code and then replace all instances of the word float with the word Decimal (note the uppercase "D".)
Ex: Decimal(1.1047262519) as opposed to float(1.1047262519)
Theory:
Float arithmetic is based off of binary math and is therefore not always exactly what a user would expect. An excelent description of the float Vs. decimal types is located Here

The previously-mentioned numpy module is not particularly relevant to the rounding error mentioned in the question. On the other hand, the decimal module can be used in a brute-force manner to get accurate computations. The following snippet from an ipython interpreter session illustrates its use (with default 28-digit accuracy), and also shows that the corresponding floating-point calculation only has 5 decimal places of accuracy.
In [180]: from decimal import Decimal
In [181]: a=Decimal('0.001'); b=Decimal('1000'); c=Decimal('0.001')
In [182]: (b*b - 4*a*c).sqrt()
Out[182]: Decimal('999.9999999979999999999980000')
In [183]: b-(b*b - 4*a*c).sqrt()
Out[183]: Decimal('2.0000000000020000E-9')
In [184]: a = .001; b = 1000; c = .001
In [185]: math.sqrt(b*b - 4*a*c)
Out[185]: 999.999999998
In [186]: b-math.sqrt(b*b - 4*a*c)
Out[186]: 1.999978849198669e-09
In [187]: 2*a*c/b
Out[187]: 1.9999999999999997e-09
Taylor series for the square root offers an alternative method to use when 4ac is tiny compared to b**2. In this case, √(b*b-4*a*c) ≈ b - 4*a*c/(2*b), whence b - √(b*b-4*a*c) ≈ 2*a*c/b. As can be seen in the line [187] entries above, Taylor series computation gives a 12-digits-accurate result while using floating point instead of Decimal. Using another Taylor series term might add a couple more digits of accuracy.

There are special cases that you should deal with:
a == 0 means a linear equation and one root: x = -c/b
b == 0 means two roots of the form x1, x2 = ±sqrt(-c/a)
c == 0 means two roots, but one of them is zero: x*(ax+b) = 0
If the discriminant is negative, you have two complex conjugate roots.
I'd recommend calculating the discriminant this way:
discriminant = b*sqrt(1.0-4.0*a*c/b)
I'd also recommend reading this:
https://math.stackexchange.com/questions/187242/quadratic-equation-error