We Keep Coding

Finding the value of machine epsilon using python

I wrote a simple code in python that gives me the same value of machine epsilon using the numpy command:np.finfo(float).eps
The code is:
eps=1
while eps+1 != 1:
eps /= 2
print(eps)
But I didn't want to stop here ! I used smaller and smaller numbers to divide eps, for example:
eps=1
while eps+1 != 1:
eps /= 1.1
print (eps)
With this, I got a value of 1.158287085355336e-16 for epsilon. I noticed that epsilon was converging to a number, my last attempt at 1.0000001 gave me the value of 1.1102231190697707e-16.
Is this value closer to the real value of epsilon for my Pc? I think I'm not considering something important and my line of thinking is wrong.
Thank you in advance for the help !

The term “machine epsilon” is not consistently defined, so it's better to avoid that word and instead to say what specifically you're talking about:
ulp(1), the magnitude of the least significant digit, or Unit in the Last Place, of 1.
This is the distance from 1 to the next larger floating-point number.
More generally, ulp(𝑥) is the distance from 𝑥 to the next larger floating-point number in magnitude.
In binary64 floating-point, with 53 bits of precision, ulp(1) is 2⁻⁵² ≈ 2.220446049250313 × 10⁻¹⁶.
In decimal64 floating-point, with 16 digits of precision, ulp(1) is 10⁻¹⁵.
In general, for floating-point in radix 𝛽 with 𝑝 digits of precision (including the implicit 1 digit), ulp(1) = 𝛽1 − 𝑝.
The relative error bound, sometimes also called unit roundoff or u.
A floating-point operation may round the outcome of a mathematical function such as 𝑥 + 𝑦, giving fl(𝑥 + 𝑦) = (𝑥 + 𝑦)⋅(1 + 𝛿) for some relative error 𝛿.
For basic arithmetic operations in IEEE 754 (+, −, *, /, sqrt), the result of computing the floating-point operation is guaranteed to be correctly rounded, which in the default rounding mode means it yields the nearest floating-point number, or one of the two nearest such numbers if 𝑥 + 𝑦 lies exactly halfway between them.
In binary64 floating-point, with 53 bits of precision, the relative error of an operation correctly rounded to nearest is at most 2⁻⁵³ ≈ 1.1102230246251565 × 10⁻¹⁶.
In decimal64 floating-point, with 16 digits of precision, the relative error of an operation correctly rounded to nearest is at most 5 × 10⁻¹⁶.
In general, when floating-point arithmetic in radix 𝛽 with 𝑝 digits of precision is correctly rounded to nearest, 𝛿 is bounded in magnitude by the relative error bound (𝛽/2) 𝛽−𝑝.
What the Python iteration while 1 + eps != 1: eps /= 2 computes, starting with eps = 1., is the relative error bound in binary64 floating-point, since that's the floating-point that essentially all Python implementations use.
If you had a version of Python that worked in a different radix, say b, you would instead want to use while 1 + eps != 1: eps /= b.
If you do eps /= 1.1 or eps /= 1.0001, you will get an approximation to the relative error bound erring on the larger side, with no particular significance to the result.
Note that sys.float_info.epsilon is ulp(1), rather than the relative error bound.
They are always related: ulp(1)/2 is the relative error bound in every floating-point format.

If you want the actual machine epsilon for a Python float on your PC, you can get it from the epsilon attribute of sys.float_info. By the way, on my x86-64 machine, numpy.finfo(float) gives me 2.220446049250313e-16, which is the expected machine epsilon for a 64-bit float.
Your intuition for trying to find the value eps such that 1 + eps != 1 is True is good, but machine epsilon is an upper bound on the relative error due to rounding in floating-point arithmetic. Not to mention, the inexact nature of floating-point arithmetic can be mystifying sometimes: note that 0.1 + 0.1 + 0.1 != 0.3 evaluates to True. Also, if I modify your code to
eps = 1
while eps + 9 != 9:
eps = eps / 1.0000001
print(eps)
I get, after around maybe half a minute,
8.881783669690459e-16
The relative error in this case is 8.881783669690459e-16 / 9 = 9.868648521878288e-17.

Your program is almost fine.
IT should be
eps=1
while eps+1 != 1:
eps /= 2
print(2*eps)
The result is
2.220446049250313e-16
Which is the epsilon of the machine. You can check this with this piece of
code:
import sys
sys.float_info.epsilon
The reason we should multiply by 2 in the last line is because you went 1 division too far inside the while loop.

Slight difference in results between Python and Matlab [duplicate]

I am writing a program where I need to delete duplicate points stored in a matrix. The problem is that when it comes to check whether those points are in the matrix, MATLAB can't recognize them in the matrix although they exist.
In the following code, intersections function gets the intersection points:
[points(:,1), points(:,2)] = intersections(...
obj.modifiedVGVertices(1,:), obj.modifiedVGVertices(2,:), ...
[vertex1(1) vertex2(1)], [vertex1(2) vertex2(2)]);
The result:
>> points
points =
12.0000 15.0000
33.0000 24.0000
33.0000 24.0000
>> vertex1
vertex1 =
12
15
>> vertex2
vertex2 =
33
24
Two points (vertex1 and vertex2) should be eliminated from the result. It should be done by the below commands:
points = points((points(:,1) ~= vertex1(1)) | (points(:,2) ~= vertex1(2)), :);
points = points((points(:,1) ~= vertex2(1)) | (points(:,2) ~= vertex2(2)), :);
After doing that, we have this unexpected outcome:
>> points
points =
33.0000 24.0000
The outcome should be an empty matrix. As you can see, the first (or second?) pair of [33.0000 24.0000] has been eliminated, but not the second one.
Then I checked these two expressions:
>> points(1) ~= vertex2(1)
ans =
0
>> points(2) ~= vertex2(2)
ans =
1 % <-- It means 24.0000 is not equal to 24.0000?
What is the problem?
More surprisingly, I made a new script that has only these commands:
points = [12.0000 15.0000
33.0000 24.0000
33.0000 24.0000];
vertex1 = [12 ; 15];
vertex2 = [33 ; 24];
points = points((points(:,1) ~= vertex1(1)) | (points(:,2) ~= vertex1(2)), :);
points = points((points(:,1) ~= vertex2(1)) | (points(:,2) ~= vertex2(2)), :);
The result as expected:
>> points
points =
Empty matrix: 0-by-2

The problem you're having relates to how floating-point numbers are represented on a computer. A more detailed discussion of floating-point representations appears towards the end of my answer (The "Floating-point representation" section). The TL;DR version: because computers have finite amounts of memory, numbers can only be represented with finite precision. Thus, the accuracy of floating-point numbers is limited to a certain number of decimal places (about 16 significant digits for double-precision values, the default used in MATLAB).
Actual vs. displayed precision
Now to address the specific example in the question... while 24.0000 and 24.0000 are displayed in the same manner, it turns out that they actually differ by very small decimal amounts in this case. You don't see it because MATLAB only displays 4 significant digits by default, keeping the overall display neat and tidy. If you want to see the full precision, you should either issue the format long command or view a hexadecimal representation of the number:
>> pi
ans =
3.1416
>> format long
>> pi
ans =
3.141592653589793
>> num2hex(pi)
ans =
400921fb54442d18
Initialized values vs. computed values
Since there are only a finite number of values that can be represented for a floating-point number, it's possible for a computation to result in a value that falls between two of these representations. In such a case, the result has to be rounded off to one of them. This introduces a small machine-precision error. This also means that initializing a value directly or by some computation can give slightly different results. For example, the value 0.1 doesn't have an exact floating-point representation (i.e. it gets slightly rounded off), and so you end up with counter-intuitive results like this due to the way round-off errors accumulate:
>> a=sum([0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]); % Sum 10 0.1s
>> b=1; % Initialize to 1
>> a == b
ans =
logical
0 % They are unequal!
>> num2hex(a) % Let's check their hex representation to confirm
ans =
3fefffffffffffff
>> num2hex(b)
ans =
3ff0000000000000
How to correctly handle floating-point comparisons
Since floating-point values can differ by very small amounts, any comparisons should be done by checking that the values are within some range (i.e. tolerance) of one another, as opposed to exactly equal to each other. For example:
a = 24;
b = 24.000001;
tolerance = 0.001;
if abs(a-b) < tolerance, disp('Equal!'); end
will display "Equal!".
You could then change your code to something like:
points = points((abs(points(:,1)-vertex1(1)) > tolerance) | ...
(abs(points(:,2)-vertex1(2)) > tolerance),:)
Floating-point representation
A good overview of floating-point numbers (and specifically the IEEE 754 standard for floating-point arithmetic) is What Every Computer Scientist Should Know About Floating-Point Arithmetic by David Goldberg.
A binary floating-point number is actually represented by three integers: a sign bit s, a significand (or coefficient/fraction) b, and an exponent e. For double-precision floating-point format, each number is represented by 64 bits laid out in memory as follows:
The real value can then be found with the following formula:
This format allows for number representations in the range 10^-308 to 10^308. For MATLAB you can get these limits from realmin and realmax:
>> realmin
ans =
2.225073858507201e-308
>> realmax
ans =
1.797693134862316e+308
Since there are a finite number of bits used to represent a floating-point number, there are only so many finite numbers that can be represented within the above given range. Computations will often result in a value that doesn't exactly match one of these finite representations, so the values must be rounded off. These machine-precision errors make themselves evident in different ways, as discussed in the above examples.
In order to better understand these round-off errors it's useful to look at the relative floating-point accuracy provided by the function eps, which quantifies the distance from a given number to the next largest floating-point representation:
>> eps(1)
ans =
2.220446049250313e-16
>> eps(1000)
ans =
1.136868377216160e-13
Notice that the precision is relative to the size of a given number being represented; larger numbers will have larger distances between floating-point representations, and will thus have fewer digits of precision following the decimal point. This can be an important consideration with some calculations. Consider the following example:
>> format long % Display full precision
>> x = rand(1, 10); % Get 10 random values between 0 and 1
>> a = mean(x) % Take the mean
a =
0.587307428244141
>> b = mean(x+10000)-10000 % Take the mean at a different scale, then shift back
b =
0.587307428244458
Note that when we shift the values of x from the range [0 1] to the range [10000 10001], compute a mean, then subtract the mean offset for comparison, we get a value that differs for the last 3 significant digits. This illustrates how an offset or scaling of data can change the accuracy of calculations performed on it, which is something that has to be accounted for with certain problems.

Look at this article: The Perils of Floating Point. Though its examples are in FORTRAN it has sense for virtually any modern programming language, including MATLAB. Your problem (and solution for it) is described in "Safe Comparisons" section.

type
format long g
This command will show the FULL value of the number. It's likely to be something like 24.00000021321 != 24.00000123124

Try writing
0.1 + 0.1 + 0.1 == 0.3.
Warning: You might be surprised about the result!

Maybe the two numbers are really 24.0 and 24.000000001 but you're not seeing all the decimal places.

Check out the Matlab EPS function.
Matlab uses floating point math up to 16 digits of precision (only 5 are displayed).

Python behaviour when approaching 0.0

I have written a Python program that needs to run until the initial value gets to 0 or until another case is met. I wanted to say anything useful about the amount of loops the program would go through until it would reach 0 as this is necessary to find the solution to another program. But for some reason I get the following results, depending on the input variables and searching the Internet thus far has not helped solve my problem. I wrote the following code to simulate my problem.
temp = 1
cool = 0.4999 # This is the important variable
count = 0
while temp != 0.0:
temp *= cool
count += 1
print(count, temp)
print(count)
So, at some point I'd expect the script to stop and print the amount of loops necessary to get to 0.0. And with the code above that is indeed the case. After 1075 loops the program stops and returns 1075. However, if I change the value of cool to something above 0.5 (for example 0.5001) the program seems to run indefinitely. Why is this the case?

The default behavior in many implementations of floating-point arithmetic is to round the real-number arithmetic result to the nearest number representable in floating-point arithmetic.
In any floating-point number format, there is some smallest positive representable value, say s. Consider how s * .25 would be evaluated. The real number result is .25•s. This is not a representable value, so it must be rounded to the nearest representable number. It is between 0 and s, so those are the two closest representable values. Clearly, it is closer to 0 than it is to s, so it is rounded to 0, and that is the floating-point result of the operation.
Similarly, when s is multiplied by any number between 0 and ½, the result is rounded to zero.
In contrast, consider s * .75. The real number result is .75•s. This is closer to s than it is to 0, so it is rounded to s, and that is the floating-point result of the operation.
Similarly, when s is multiplied by any number between ½ and 1, the result is rounded to s.
Thus, if you start with a positive number and continually multiply by some fraction between 0 and 1, the number will get smaller and smaller until it reaches s, and then it will either jump to 0 or remain at s depending on whether the fraction is less than or greater than ½.
If the fraction is exactly ½, .5•s is the same distance from 0 and s. The usual rule for breaking ties favors the number with the even low bit in its fraction portion, so the result is rounded to 0.
Note that Python does not fully specific floating-point semantics, so the behaviors may vary from implementation to implementation. Most commonly, IEEE-754 binary64 is used, in which the smallest representable positive number is 2−1074.

Python: Why does have 2^-n work for n>52 and not 1+2^-n-1?

I'm pretty new to python, and I've made a table which calculates T=1+2^-n-1 and C=2^n, which both give the same values from n=40 to n=52, but for n=52 to n=61 I get 0.0 for T, whereas C gives me progressively smaller decimals each time - why is this?
I think I understand why T becomes 0.0, because of python using binary floating point and because of the machine epsilon value - but I'm slightly confused as to why C doesn't also become 0.0.
import numpy as np
import math
t=np.zeros(21)
c=np.zeros(21)
for n in range(40,61):
m=n-40
t[m]=1+2**(-n)-1
c[m]=2**(-n)
print (n,t[m],c[m])

The "floating" in floating point means that values are represented by storing a fixed number of leading digits and a scale factor, rather than assuming a fixed scale (which would be fixed point).
2**-53 only takes one (binary) digit to represent (not including the scale), but 1+2**-53 would take 54 to represent exactly. Python floats only have 53 binary digits of precision; 2**-53 can be represented exactly, but 1+2**-53 gets rounded to exactly 1, and subtracting 1 from that gives exactly 0. Thus, we have
>>> 2**-53
1.1102230246251565e-16
>>> 1+(2**-53)-1
0.0
Postscript: you might wonder why 2**-53 displays as a value not equal to the exact mathematical value when I said it was exact. That's due to the float->string conversion logic, which only keeps enough decimal digits to reconstruct the original float (instead of printing a bunch of digits at the end that are usually just noise).

The difference between both is indeed due to floating-point representation. Indeed, if you perform 1 + X where X is a very very small number, then the floating-point representation sets its exponent value to 0 and the precision is ensured by the mantissa, which is 52-bit on a 64-bit computer. Therefore, 1 + 2^(-X) if X > 52 is equal to 1. However, even 2^-100 can be represented in double-precision floating-point, so you can see C decrease for a larger number of samples.

Why float value is rounded in python?

I have read that minimal float value that Python support is something in power -308.
Exact number doesn't matter because:
>>> -1.42108547152e-14 + 360.0 == 360.0
True
How is that? I have CPython 2.7.3 on Windows.
It cause me errors. My problem will be fixed if I will compare my value -1.42108547152e-14 (computed somehow) to some "delta" and do this:
if v < delta:
v = 0
What delta should I choose? In other words, with values smaller than what this effect will occur?
Please note that NumPy is not available.

An (over)simplified explanation is: A (normal) double-precision floating point number holds (what is equivalent to) approximately 16 decimal digits. Let's try to do your addition by hand:
360.0000000000000000000000000
- 0.0000000000000142108547152
______________________________
359.9999999999999857891452848
If you round this to 16 figures (3 before the point and 13 after), you get 360.
Now, in reality this is done in binary. The "16 decimal digits" is therefore not a precise rule. In reality, the precision here (between 256.0 and 512.0) is 44 binary digits for the fractional part of the number. So the number closest to 360 which can be represented is 360 minus {2 to the -44th power}, which gives:
359.9999999999999431565811391 (truncated)
But since our result before was closer to 360.0 than to this number, 360.0 is what you get.

Most processors use IEEE 754 binary floating-point arithmetic. In this format, numbers are represented as a sign s, a fraction f, and an exponent e. The fraction is also called a significand.
The sign s is a bit 0 or 1 representing + or –, respectively.
In double precision, the significand f is a 53-bit binary numeral with a radix point after the first bit, such as 1.10100001001001100111000110110011001010100000001000112.
In double precision, the exponent e is an integer from –1022 to +1023.
The combined value represented by the sign, significand, and exponent is (-1)s•2e•f.
When you add two numbers, the processor figures out what exponent to use for the result. Then, given that exponent, it figures out what fraction to use for the result. When you add a large number and a small number, the entire result will not fit into the significand. So, the processor must round the mathematical result to something that will fit into the significand.
In the cases you ask about, the second added number is so small that the rounding produces the same value as the first number. In order to change the first number, you must add a value that is at least half the value of the lowest bit in the significand of the first number. (When rounding, if part that does not fit in the significand is more than half the lowest bit of the significand, it is rounded up. If it is exactly half, it is rounded up if that makes the lowest bit zero or down if that makes the lowest bit zero.)
There are additional issues in floating-point, such as subnormal numbers, infinities, how the exponent is stored, and so on, but the above explains the behavior you asked about.
In the specific case you ask about, adding to 360, adding any value greater than 2-45 will produce a sum greater than 360. Adding any positive value less than or equal to 2-45 will produce exactly 360. This is because the highest bit in 360 is 28, so the lowest bit in its significand is 2-44.