Floating Point Programming, Today?

An anonymous reader asks: "I'm rather new with programming and stumbled across these twe articles: The Perils of Floating Point from 1996 and What Every Computer Scientist Should Know About Floating-Point Arithmetic from 1991. I tried some of the examples in these articles with Intel's Fortran Compiler and g77 and noted that some of those issue reported no longer seem valid whereas quite a few still very much are around. Could someone, please, give me a pointer to some newer thoughts and/or new facts surrounding floating point programming. What has been improved since those articles were written? What is still the same? How is the future, especially with the new platforms IA64 and AMD64? I am most interested in the x86 and x86-64 architectures. Thank you for your kind help."

Comments

The articles are quite up-to-date

[Mik!tAAt]

Both articles are still valid today, mostly because current processors use the same IEEE floating point format than the ones available in 96 (or 91).

Unsolvable problem

[Anonymous Coward]

Floating point stuff hasn't really changed much since then. Basic rule of thumb, if you want it to be accurate don't use floating point.

Much the same problem as you have with decimals. Many fractions cannot be evaluated evenly in certain bases. It will always cause you headaches if you don't realize this.

Try writing a bunch of numbers in hex but then do all of your calculations in decimal. you'll have the same problem.

Re:Unsolvable problem

[john_many_jars]

The use of floating point numbers isn't all bad. Those of use who use them are often solving problems with condition numbers that render the answer we get less accurate than the number of digits of accuracy provided.

Think about tan(89.99) versus tan(89.991) (which is very ill-conditioned around 90). Both numbers are not terribly truncated by floating point, but the results are different by about 1,000. Try it and you'll see floating point error isn't as dangerous as things like cancellation, ill-conditioning and the like.

Re:Unsolvable problem

[Phronesis]

Think about tan(89.99) versus tan(89.991) (which is very ill-conditioned around 90). Both numbers are not terribly truncated by floating point, but the results are different by about 1,000. Try it and you'll see floating point error isn't as dangerous as things like cancellation, ill-conditioning and the like.

tan(89.990) = -2.0460
tan(89.991) = -2.0408

perhaps you're thinking of

tan(1.571) = -4909.8
and
tan(1.578) = -138.8

Re:Unsolvable problem

[Admiral Burrito]

Try writing a bunch of numbers in hex but then do all of your calculations in decimal. you'll have the same problem.

Actually, you won't. You would the other way though.

The problem occurs when you try to represent a (properly reduced) fraction whos denominator has one or more prime factors not in common with your number base.

You can represent one tenth in base 10 because all the prime factors in the denominator (10: 5,2) are found within the factorization of the base (also 10: 5,2). You can not represent one sixth in base 10 because one of the factors of 6 is not found in the factorization of 10 (3). Likewise, you cannot represent one tenth in base 2, because the denominator (10) is a multiple of 5, which is a prime not found in the factorization of the base (2).

Because the factorization of 16 contains only primes that are in the factorization of 10 (2) all fractions that can be represented in hexadecimal can be represented in decimal. The reverse is not true, because 10 is the product of a prime (5) that is not found in the factorization of 16. So there is no way to get the "fifths" aspect of a decimal number into a hexadecimal number.

Platform and all

[Stary]

It all depends on what platform you program on and so on. Newer x86 processors do their floating point in an 80-bit format and only truncate when copying back to your original 32 or 64 bit floats. That saves you some precision but not that much. As others have said, there are probably situations where almost all of the material in those articles is valid.

Common mistake

[PD]

Don't count money as floating point. You'll just have rounding errors. Using long doubles instead of floats won't help you at all.

The solution is to count pennies instead, or if you need values bigger than 22 million dollars, use a BCD library. BCD is Binary Coded Decimal.

Re:Common mistake

[PD]

That's not the error I was addressing. Here's some definitions of a subtotal:

float subtotal; // wrong way to represent money
long subtotal_pennies; // right way to represent money

And, if you're at a gas station, you need to represent money like this:

long subtotal_mils; // gas per gallon has a 9/10 of a cent on the end - $1.34 9/10

The calculations that you perform on the money are a completely different story. There's no point in worrying about 4 decimal places of percentages if you don't start from the right place.

Here's an important one.

[Apuleius]

Using accurate arithmetics to improve numerical reproducibility and stability in parallel applications, also available here if you're one of the unwashed masses. It has algorithms to see if your app is facing floating point trouble.

If you need more precision...

[cfallin]

Hardware floating point is only so accurate - if you need more floating point (or integer) precision, use GNU MP - a library for C with bindings for many other languages too. It came in quite handy when I wrote some cryptography code with very large numbers.

Python-specific, but contains useful info for all.

[tdelaney]

Floating Point Arithmetic: Issues and Limitations.

Intervall Analysis

[mvw]

Ok, known issues with floating point routines that can be fixed (unintentional pun :-) should be fixed.

On the other hand it is clear that a finite representation of real numbers has tradeoffs. But only few seem to care about the cumulated errors.

My experience in engineering (simulation of casted turbine blades) was that people know that bad things can occur during complex floating point calculations but the matter was too complicated to be investigated.

Example: if during finite element simulation a timestep did not end up with a valid solution (the iterative/approximative solver of the large linear systems did not converge or even crash) just some control parameters were varied (time step, perhaps material curves) until the calculation seemed to produce some valid looking result. Needless to say, that that only obvious errors can be spotted that way.

The strange thing about all that is, that in the last years the mathematical discipline of interval analyis has been developed. Here every number is represented with its interval of known error bounds. These error intervall are kept and updated during calculations. Thus at the end of a large complex calculation, you know the error. That is a very valuable property.

More, in fact what one does so in many cases is not only a standard calculation but rather machine proof of error bounds.

This offers some unique properties, e.g. for rigorous global searches.

So we have far better technology available. Why is this stuff not used more widely?

As far as I know, only SUN puts interval analysis enabled data types in its FORTRAN and C/C++ compilers. But I have not seen that stuff in gcc, which would have a big impact.

Very strange.

To whom is interested, here is a homepage of the intervals community.

Regards,
Marc