I have no idea how or where to start, I need to have some vocabulary or terms to get going and researching so let's ask the community.
Problem: I have a value X that is the final answer of the equation. I have a range of values, let's say 10 (A-J), that will be used in the equation. Using simple calculations (+, -, *, /, (), ^) to form an equation that uses some or all of the values A-J to solve the equation to the value X.
Example: A^2+B*C+(D+E+F+G)*J+30 = X
Input: final value X, the values used in the equation, max number of values to be used in the equation meaning I want to use at least Z number of given values in the equation (in the example Z = 8)
Output: the equation that solves it with the given values
Can this be turned into a python script for example? Is it possible at all to calculate this way? What are the terms that describe this kind of calculations?
If I understand the question correctly:
The algorithm you are looking for outputs a mathematical function, along with the specific values that, when applied to the function, give you the input value x, for an arbitrary value x.
In general I believe this is not possible. It may be possible from a non-deterministic point of view, where you try and guess values but that usually isn't feasible from an algorithmic or computational standpoint.
Let's first limit the function to one variable. Finding a function that gives you a value x, for some value a, i.e. f(a) = x, is the same as asking for f(a) - x = 0.
Limiting the operations to +,-,*, we see that f(a) is a polynomial.
Limiting the problem in this way relates the solution to algebraic numbers and constructible numbers. The more general mathematical theory that explain properties of these numbers and functions is called Galois Theory.
It is possible to find the polynomial of a given the input value x IF and only IF the input value x is algebraic. You can produce a simple algorithm that takes powers of x until that power of x is an integer (or a rational number).
You can produce a simple algorithm that takes powers of the irrational part of x until that power of x is an integer (or a rational number). Note that even in this algorithm you would need to take into account some sort of measurement error, because sqrt(2) = 1.41421356237... for an infinite number of decimal places, and the computer can only keep track of some finite amount of decimal places. For example:
def sqrt(x):
return x**(1/2)
if __name__ == "__main__":
num = sqrt(2)
print(num)
print(num**2)
Will output:
1.4142135623730951
2.0000000000000004
So the output of such a simple algorithm will never be 100% correct. It is possible for a human to do, but not a computer. You may want to look into the field of symbolic computation, but the algorithm for solving even part of your problem will not be easy to turn into a script.
If you are okay with not solving this problem 100% of the time, you may want to look into linear approximations and non-linear approximations.
The reason why even without measurement errror, I believe this is not possible in general is that adding the operations (^,/) may not result in a polynomial ring. Which is the basis of solving problems relating to algebraic numbers.
Introducing extra variables b,c,....,n,.. to f, such that f(a,b,c,...,n,...) = x would also restrict what functions would satisfy the properties of a ring.
Regardless, this problem is difficult, even when only considering polynomials of one variable. Googling the terms in bold may help provide additional insight into your problem. Sorry if some of this was already obvious for you and in any case I hope this helps!
Why do comparisons of NaN values behave differently from all other values?
That is, all comparisons with the operators ==, <=, >=, <, > where one or both values is NaN returns false, contrary to the behaviour of all other values.
I suppose this simplifies numerical computations in some way, but I couldn't find an explicitly stated reason, not even in the Lecture Notes on the Status of IEEE 754 by Kahan which discusses other design decisions in detail.
This deviant behavior is causing trouble when doing simple data processing. For example, when sorting a list of records w.r.t. some real-valued field in a C program I need to write extra code to handle NaN as the maximal element, otherwise the sort algorithm could become confused.
Edit:
The answers so far all argue that it is meaningless to compare NaNs.
I agree, but that doesn't mean that the correct answer is false,
rather it would be a Not-a-Boolean (NaB), which fortunately doesn't exist.
So the choice of returning true or false for comparisons is in my view arbitrary,
and for general data processing it would be advantageous if it obeyed the usual laws
(reflexivity of ==, trichotomy of <, ==, >),
lest data structures which rely on these laws become confused.
So I'm asking for some concrete advantage of breaking these laws, not just philosophical reasoning.
Edit 2:
I think I understand now why making NaN maximal would be a bad idea, it would mess up the computation of upper limits.
NaN != NaN might be desirable to avoid detecting convergence in a loop such as
while (x != oldX) {
oldX = x;
x = better_approximation(x);
}
which however should better be written by comparing the absolute difference with a small limit.
So IMHO this is a relatively weak argument for breaking reflexivity at NaN.
I was a member of the IEEE-754 committee, I'll try to help clarify things a bit.
First off, floating-point numbers are not real numbers, and floating-point arithmetic does not satisfy the axioms of real arithmetic. Trichotomy is not the only property of real arithmetic that does not hold for floats, nor even the most important. For example:
Addition is not associative.
The distributive law does not hold.
There are floating-point numbers without inverses.
I could go on. It is not possible to specify a fixed-size arithmetic type that satisfies all of the properties of real arithmetic that we know and love. The 754 committee has to decide to bend or break some of them. This is guided by some pretty simple principles:
When we can, we match the behavior of real arithmetic.
When we can't, we try to make the violations as predictable and as easy to diagnose as possible.
Regarding your comment "that doesn't mean that the correct answer is false", this is wrong. The predicate (y < x) asks whether y is less than x. If y is NaN, then it is not less than any floating-point value x, so the answer is necessarily false.
I mentioned that trichotomy does not hold for floating-point values. However, there is a similar property that does hold. Clause 5.11, paragraph 2 of the 754-2008 standard:
Four mutually exclusive relations are possible: less than, equal, greater than, and unordered. The last case arises when at least one operand is NaN. Every NaN shall compare unordered with everything, including itself.
As far as writing extra code to handle NaNs goes, it is usually possible (though not always easy) to structure your code in such a way that NaNs fall through properly, but this is not always the case. When it isn't, some extra code may be necessary, but that's a small price to pay for the convenience that algebraic closure brought to floating-point arithmetic.
Addendum:
Many commenters have argued that it would be more useful to preserve reflexivity of equality and trichotomy on the grounds that adopting NaN != NaN doesn’t seem to preserve any familiar axiom. I confess to having some sympathy for this viewpoint, so I thought I would revisit this answer and provide a bit more context.
My understanding from talking to Kahan is that NaN != NaN originated out of two pragmatic considerations:
That x == y should be equivalent to x - y == 0 whenever possible (beyond being a theorem of real arithmetic, this makes hardware implementation of comparison more space-efficient, which was of utmost importance at the time the standard was developed — note, however, that this is violated for x = y = infinity, so it’s not a great reason on its own; it could have reasonably been bent to (x - y == 0) or (x and y are both NaN)).
More importantly, there was no isnan( ) predicate at the time that NaN was formalized in the 8087 arithmetic; it was necessary to provide programmers with a convenient and efficient means of detecting NaN values that didn’t depend on programming languages providing something like isnan( ) which could take many years. I’ll quote Kahan’s own writing on the subject:
Were there no way to get rid of NaNs, they would be as useless as Indefinites on CRAYs; as soon as one were encountered, computation would be best stopped rather than continued for an indefinite time to an Indefinite conclusion. That is why some operations upon NaNs must deliver non-NaN results. Which operations? … The exceptions are C predicates “ x == x ” and “ x != x ”, which are respectively 1 and 0 for every infinite or finite number x but reverse if x is Not a Number ( NaN ); these provide the only simple unexceptional distinction between NaNs and numbers in languages that lack a word for NaN and a predicate IsNaN(x).
Note that this is also the logic that rules out returning something like a “Not-A-Boolean”. Maybe this pragmatism was misplaced, and the standard should have required isnan( ), but that would have made NaN nearly impossible to use efficiently and conveniently for several years while the world waited for programming language adoption. I’m not convinced that would have been a reasonable tradeoff.
To be blunt: the result of NaN == NaN isn’t going to change now. Better to learn to live with it than to complain on the internet. If you want to argue that an order relation suitable for containers should also exist, I would recommend advocating that your favorite programming language implement the totalOrder predicate standardized in IEEE-754 (2008). The fact that it hasn’t already speaks to the validity of Kahan’s concern that motivated the current state of affairs.
NaN can be thought of as an undefined state/number. similar to the concept of 0/0 being undefined or sqrt(-3) (in the real number system where the floating point lives).
NaN is used as a sort of placeholder for this undefined state. Mathematically speaking, undefined is not equal to undefined. Neither can you say an undefined value is greater or less than another undefined value. Therefore all comparisons return false.
This behaviour is also advantageous in the cases where you compare sqrt(-3) to sqrt(-2). They would both return NaN but they are not equivalent even though they return the same value. Therefore having equality always returning false when dealing with NaN is the desired behaviour.
To throw in yet another analogy. If I hand you two boxes, and tell you that neither of them contains an apple, would you tell me that the boxes contain the same thing?
NaN contains no information about what something is, just what it isn't. Therefore these elements can never definitely be said to be equal.
From the wikipedia article on NaN, the following practices may cause NaNs:
All mathematical operations> with a NaN as at least one operand
The divisions 0/0, ∞/∞, ∞/-∞, -∞/∞, and -∞/-∞
The multiplications 0×∞ and 0×-∞
The additions ∞ + (-∞), (-∞) + ∞ and equivalent subtractions.
Applying a function to arguments outside its domain, including taking the square root of a negative number, taking the logarithm of a negative number, taking the tangent of an odd multiple of 90 degrees (or π/2 radians), or taking the inverse sine or cosine of a number which is less than -1 or greater than +1.
Since there is no way to know which of these operations created the NaN, there is no way to compare them that makes sense.
I don't know the design rationale, but here's an excerpt from the IEEE 754-1985 standard:
"It shall be possible to compare floating-point numbers in all supported formats, even if the operands' formats differ. Comparisons are exact and never overflow nor underflow. Four mutually exclusive relations are possible: less than, equal, greater than, and unordered. The last case arises when at least one operand is NaN. Every NaN shall compare unordered with everything, including itself."
It only looks peculiar because most programming environments that allow NaNs do not also allow 3-valued logic. If you throw 3-valued logic into the mix, it becomes consistent:
(2.7 == 2.7) = true
(2.7 == 2.6) = false
(2.7 == NaN) = unknown
(NaN == NaN) = unknown
Even .NET does not provide a bool? operator==(double v1, double v2) operator, so you are still stuck with the silly (NaN == NaN) = false result.
I'm guessing that NaN (Not A Number) means exactly that: This is not a number and thus comparing it does not really make sense.
It's a bit like arithmetic in SQL with null operands: They all result in null.
The comparisons for floating point numbers compare numeric values. Thus, they can't be used for non numeric values. NaN therefore cannot be compared in a numeric sense.
MeToo came here to understand the reasoning, why NaN == NaN equals false.
After reading (nearly) all I still was puzzled, why a == NaN cannot replace a function like isNaN(), because it seems to be so obvious.
But things are not that simple.
Nobody has mentioned vector geometry yet. But many computations take place in the 2nd or 3rd dimension, so in vector space.
After thinking about this a bit, I immediately realized, why it is a good thing to have NaN not to compare to itself. Following hopefully is easy enough to understand for others, too.
Vectors
Bear with me, it takes a while until NaN shows up.
First let me explain a bit for people who are not deep inside math
In vector geometry we usually use something like complex numbers.
A complex number is made of two floats a + bi (where i denotes the imaginary value with i * i == -1) which allow us to address all points on the 2 dimensional plane. With floating point we cannot express each value, so we have to approximate a bit. So if we round the values to some value we can express, we can still try to create numerically stable algorithms, which give us some good approximation of what we want to archive.
Enter infinity
No NaN here yet. Please be patient. I'll get to the point later down below.
If we want to specify some point far far away, we might leave the range of numbers we can express, which results in infinity. In IEEE floats we luckily have +inf (I write it as inf) or -inf for this (written as -inf).
This is good:
a + inf i makes sense, right? It is the vector to some point on the x-axes at location a and on the y-axes at location "positive infinity". But wait a bit, we are talking vectors here!
Vectors have an origin and a point they point to. Normalized vectors are those, which start at location (0,0).
Now think of a vector with origin of (0,0) which points to (a,inf).
Still makes sense? Not quite. As we look a bit closer, we will see, that the normalized vector (0,inf) is the same vector! As the vector is so long, the derivation of a in the infinty can no more be seen. Or said otherwise:
For infinitively long vectors in the cartesian coordinate system, the finite axis can be expressed as 0, because we are allowed to approximate (if we are not allowed to approximate, we cannot use floating point!).
So the replacement-vector (0,inf) is still suitable. In fact, any (x,inf) is a suitable replacement for a finite x. So why not use 0 from our origin of our normalized vector.
Hence what do we get here? Well, with allowing inf in our vectors, we actually get 8 possible infinite vectors, each 45 degrees rotated (degrees in parentheses):
(inf,0) (0), (inf,inf) (45), (0,inf) (90), (-inf,inf) (135), (-inf,0) (180), (-inf,-inf) (225), (0,-inf) (270) and (inf,-inf) (315)
All this does not cause any trouble. In fact, it is good to be able to express more than just finite vectors. This way we have a natural extension of our model.
Polar coordinates
Still no NaN here, but we are getting closer
Above we used complex numbers as cartesian coordinates. But complex numbers also have a 2nd option how we can write them. That is polar coordinates.
Polar coordinates are made up of a length and an angle like [angle,length]. So if we transform our complex number into polar coordinates, we will see, that we can express a bit more than just 8 angles in [angle,inf].
Hence, if you want to create a mathematical model which allows infinitely long vectors in some multidimensional space, you definitively want to use polar coordinates in your calculation as much as you can.
All you have to do for this is to convert the cartesian coordinates into the polar ones and vice versa.
How to do this is left as exercise for the reader.
Enter NaN
Now, what do we have?
We have a mathematical model which calculates with polar coordinates.
And we have some output device, which uses cartesian coordinates, probably.
What we now want to do is to be able to convert between those two. What do we need for this?
We need floating point, of course!
And as we perhaps need to calculate with some few terabillion coordinates, (perhaps we render some weather forecast or have some collision data from the large hadron collider) we do not want to include slow and error prone error processing (WTF? Error prone error processing? You bet!) in all those complex mathematical (hopefully numerically stable) steps.
How do we propagate errors then?
Well, as said by IEEE: We use NaN for error propagation
So what we have up to here?
Some calculation in the polar coordinate space
Some conversion into cartesian space
NaN as rescue if something fails
And this then leads to ..
.. why NaN == NaN must be false
To explain this, let's reduce this complex stuff above all to a simple result of 2 vectors in cartesian coordinates:
(a,b) and (c,d)
And we want to compare those two. This is how this comparison looks like:
a == c && b == d
Everything correct so far?
Yes. But only until we observe following two polar vectors which might be the source of our two cartesian vectors:
[NaN,inf] and [0,NaN]
Certainly those two vectors are not equal in the polar coordinate space. But after conversion into cartesian space, both come out as:
(NaN,NaN) and (NaN,NaN)
Well, should they suddenly compare equal?
Surely not!
Thanks to IEEE defining that NaN == NaN must return false, our very primitive vector comparison still gives us the expected result!
And I think, that exactly is the motivation behind, why IEEE defined it as it is.
Now we have to live with this mess. But is it a mess, indeed? I'm undecided. But, at least, I now can understand the (probable) reasoning.
Hopefully I did not miss something.
Some last words
The primitive way of comparing things usually is not fully appropriate when it comes to floating point numbers.
In floating point, you usually do not use ==, you rather use something like abs(a-b) < eps with eps being some very small value. This is because already something like 1/3 + 1/3 * 2.0 == 1.0 might not be true, depending on which hardware you run.
1/3 + 1/3 * 2.0 == 1/3 + 1/3 + 1/3 should be true on all reasonable hardware. So even == can be used. Only carefully. But is not ruled out.
However this does not render above reasoning void. Because above is not a mathematically proof for that the IEEE is right. It is just an example, which should allow to understand the source of the reasoning behind, and why it is probably better to have it defined the way it is.
Even that it is a PITA for all programming people like me.
The over-simplified answer is that a NaN has no numeric value, so there is nothing in it to compare to anything else.
You might consider testing for and replacing your NaNs with +INF if you want them to act like +INF.
While I agree that comparisons of NaN with any real number should be unordered, I think there is just cause for comparing NaN with itself. How, for example does one discover the difference between signaling NaNs and quiet NaNs? If we think of the signals as a set of Boolean values (i.e. a bit-vector) one might well ask whether the bit-vectors are the same or different and order the sets accordingly. For example, on decoding a maximum biased exponent, if the significand were left shifted so as to align the most significant bit of the significand on the most significant bit of the binary format, a negative value would be a quiet NaN and any positive value would be a signaling NaN. Zero of course is reserved for infinity and the comparison would be unordered. MSB alignment would allow for the direct comparison of signals even from different binary formats. Two NaNs with the same set of signals would therefore be equivalent and give meaning to equality.
Because mathematics is the field where numbers "just exist".
In computing you must initialize those numbers and keep their state according to your needs.
At those old days memory initialization worked in the ways you could never rely on. You never could allow yourself to think about this "oh, that would be initialized with 0xCD all the time, my algo will not broke".
So you need proper non-mixing solvent which is sticky enough to not not letting your algorithm getting sucked into and broken.
Good algorithms involving numbers are mostly going to work with relations, and those if() relations will be omitted.
This is just grease which you can put into new variable at creation, instead of programming random hell from computer memory. And your algorithm whatever it is, will not break.
Next, when you still suddenly finding out that your algorithm is producing NaNs, it is possible to clean it out, looking into every branch one at a time. Again, "always false" rule is helping a lot in this.
Very short answer:
Because the following:
nan / nan = 1
must NOT hold. Otherwise inf/inf would be 1.
(Therefore nan can not be equal to nan. As for > or <, if nan would respect any order relation in a set satisfying the Archimedean property, we would have again nan / nan = 1 at the limit).
I have a big matrix with values that vary greatly in orders of magnitude. To calculate the sum as accurate as possible, my approach would be to reshape the ndarray into a 1-dimensional array, sort it and then add it up, starting with the smallest entries. Is there a better / more efficient way to do this?
I think that, given floating point precision problems, the best known algorithm for your task is Kahan summation. For practical purposes, Kahan summation has an error bound that is independent of the number of summands, while naive summation has an error bound that grows linearly with the number of summands.
NumPy does not use Kahan summation, and there is no easy way of implementing it without a big performance tradeoff. But it uses the next best thing, pairwise summation, where error grows, under some reasonable assumptions, like the square root of the logarithm of the number of summands.
So it is very likely that Numpy is on its own already able to provide sufficiently good precision for your problem. To validate this, I would actually run a few sample cases through Kahan summation (the pseudocode in the Wikipedia link above can be trivially converted to Python), and take this as the golden, best possible result, and compare it against:
Calling np.sum on your matrix as is.
Calling np.sum on your matrix after reshaping to 1D, which may give better results if your matrix is not contiguous in memory.
Calling np.sum on a sorted version of the 1D array.
For most cases these last three options should behave similarly, but the only way to know is to actually test it.
I'm trying to write an implementation of Wilson's spectral density factorization algorithm [1] for Python. The algorithm iteratively factorizes a [QxQ] matrix function into its square root (it's sort of an extension of the Newton-Raphson square-root finder for spectral density matrices).
The problem is that my implementation only converges for matrices of size 45x45 and smaller. So after 20 iterations, the summed squared difference between matrices is about 2.45e-13. However, if I make an input of size 46x46, it does not converge until the 100th or so iteration. For 47x47 or larger, the matrices never converge; the error fluctuates between 100 and 1000 for about 100 iterations, and then starts to grow very quickly.
How would you go about trying to debug something like this? There doesn't appear to be any specific point at which it goes crazy, and the matrices are too large for me to actually attempt to do the calculation by hand. Does anyone have tips / tutorials / heuristics for find bizarre numerical bugs like this?
I've never dealt with anything like this before but I'm hoping some of you have...
Thanks,
- Dan
[1] G. T. Wilson. "The Factorization of Matricial Spectral Densities". SIAM J. Appl. Math (Vol 23, No. 4, Dec. 1972)
I would recommend asking this question on the scipy-user mailing list, perhaps with an example of your code. Generally the people on the list seem to be highly experienced with numerical computation and are really helpful, just following the list is an education in itself.
Otherwise, I'm afraid I don't have any ideas... If you think it is a numerical precision/floating point rounding issue, the first thing you could try is bump all the dtypes up to float128 and see if makes any difference.
Interval arithmetic can help, but I'm not sure if performance will be sufficient to actually allow meaningful debugging at the matrix sizes of your interest (you have to figure on a couple orders of magnitude worth of slowdown, what between replacing highly-HW-helped "scalar" floating point operations with SW-heavy "interval" ones, and adding the checks about which intervals are growing too wide, when, where, and why).
I am trying to numerically integrate an arbitrary (known when I code) function in my program
using numerical integration methods. I am using Python 2.5.2 along with SciPy's numerical integration package. In order to get a feel for it, i decided to try integrating sin(x) and observed this behavior-
>>> from math import pi
>>> from scipy.integrate import quad
>>> from math import sin
>>> def integrand(x):
... return sin(x)
...
>>> quad(integrand, -pi, pi)
(0.0, 4.3998892617846002e-14)
>>> quad(integrand, 0, 2*pi)
(2.2579473462709165e-16, 4.3998892617846002e-14)
I find this behavior odd because -
1. In ordinary integration, integrating over the full cycle gives zero.
2. In numerical integration, this (1) isn't necessarily the case, because you may just be
approximating the total area under the curve.
In any case, either assuming 1 is True or assuming 2 is True, I find the behavior to be inconsistent. Either both integrations (-pi to pi and 0 to 2*pi) should return 0.0 (first value in the tuple is the result and the second is the error) or return 2.257...
Can someone please explain why this is happening? Is this really an inconsistency? Can someone also tell me if I am missing something really basic about numerical methods?
In any case, in my final application, I plan to use the above method to find the arc length of a function. If someone has experience in this area, please advise me on the best policy for doing this in Python.
Edit
Note
I already have the first differential values at all points in the range stored in an array.
Current error is tolerable.
End note
I have read Wikipaedia on this. As Dimitry has pointed out, I will be integrating sqrt(1+diff(f(x), x)^2) to get the Arc Length. What I wanted to ask was - is there a better approximation/ Best practice(?) / faster way to do this. If more context is needed, I'll post it separately/ post context here, as you wish.
The quad function is a function from an old Fortran library. It works by judging by the flatness and slope of the function it is integrating how to treat the step size it uses for numerical integration in order to maximize efficiency. What this means is that you may get slightly different answers from one region to the next even if they're analytically the same.
Without a doubt both integrations should return zero. Returning something that is 1/(10 trillion) is pretty close to zero! The slight differences are due to the way quad is rolling over sin and changing its step sizes. For your planned task, quad will be all you need.
EDIT:
For what you're doing I think quad is fine. It is fast and pretty accurate. My final statement is use it with confidence unless you find something that really has gone quite awry. If it doesn't return a nonsensical answer then it is probably working just fine. No worries.
I think it is probably machine precision since both answers are effectively zero.
If you want an answer from the horse's mouth I would post this question on the scipy discussion board
I would say that a number O(10^-14) is effectively zero. What's your tolerance?
It might be that the algorithm underlying quad isn't the best. You might try another method for integration and see if that improves things. A 5th order Runge-Kutta can be a very nice general purpose technique.
It could be just the nature of floating point numbers: "What Every Computer Scientist Should Know About Floating Point Arithmetic".
This output seems correct to me since you have absolute error estimate here. The integral value of sin(x) is indeed should have value of zero for full period (any interval of 2*pi length) in both ordinary and numeric integration and your results is close to that value.
To evaluate arc length you should calculate integral for sqrt(1+diff(f(x), x)^2) function, where diff(f(x), x) is derivative of f(x). See also Arc length
0.0 == 2.3e-16 (absolute error tolerance 4.4e-14)
Both answers are the same and correct i.e., zero within the given tolerance.
The difference comes from the fact that sin(x)=-sin(-x) exactly even in finite precision. Whereas finite precision only gives sin(x)~sin(x+2*pi) approximately. Sure it would be nice if quad were smart enough to figure this out, but it really has no way of knowing apriori that the integral over the two intervals you give are equivalent or that the the first is a better result.