Floating Point Programming, Today?

An anonymous reader asks: "I'm rather new with programming and stumbled across these twe articles: The Perils of Floating Point from 1996 and What Every Computer Scientist Should Know About Floating-Point Arithmetic from 1991. I tried some of the examples in these articles with Intel's Fortran Compiler and g77 and noted that some of those issue reported no longer seem valid whereas quite a few still very much are around. Could someone, please, give me a pointer to some newer thoughts and/or new facts surrounding floating point programming. What has been improved since those articles were written? What is still the same? How is the future, especially with the new platforms IA64 and AMD64? I am most interested in the x86 and x86-64 architectures. Thank you for your kind help."

Comments

The articles are quite up-to-date

[Mik!tAAt]

Both articles are still valid today, mostly because current processors use the same IEEE floating point format than the ones available in 96 (or 91).

Don't worry

[Hard_Code]

...those articles are only 99.99999891 percent true

Unsolvable problem

[Anonymous Coward]

Floating point stuff hasn't really changed much since then. Basic rule of thumb, if you want it to be accurate don't use floating point.

Much the same problem as you have with decimals. Many fractions cannot be evaluated evenly in certain bases. It will always cause you headaches if you don't realize this.

Try writing a bunch of numbers in hex but then do all of your calculations in decimal. you'll have the same problem.

Re:Unsolvable problem

[john_many_jars]

The use of floating point numbers isn't all bad. Those of use who use them are often solving problems with condition numbers that render the answer we get less accurate than the number of digits of accuracy provided.

Think about tan(89.99) versus tan(89.991) (which is very ill-conditioned around 90). Both numbers are not terribly truncated by floating point, but the results are different by about 1,000. Try it and you'll see floating point error isn't as dangerous as things like cancellation, ill-conditioning and the like.

Re:Unsolvable problem

[Daleks]

Basic rule of thumb, if you want it to be accurate don't use floating point.

Basic rule of thumb, determine what accuracy you need, then pick your number representation.

Platform and all

[Stary]

It all depends on what platform you program on and so on. Newer x86 processors do their floating point in an 80-bit format and only truncate when copying back to your original 32 or 64 bit floats. That saves you some precision but not that much. As others have said, there are probably situations where almost all of the material in those articles is valid.

Re:Platform and all

[norwoodites]

It only truncates when saving to memory, that is why you can get different results when optimizing than not optimizing with gcc (you can force gcc to truncate all the time by using -ffloat-store).
With gcc you can force the floating point calculations in the sse registers by -mfpmath=sse.

Common mistake

[PD]

Don't count money as floating point. You'll just have rounding errors. Using long doubles instead of floats won't help you at all.

The solution is to count pennies instead, or if you need values bigger than 22 million dollars, use a BCD library. BCD is Binary Coded Decimal.

Re:Common mistake

[AvantLegion]

Don't count money as floating point. You'll just have rounding errors.

But that's the point! And you transfer those fractions of cents (that just get rounded off anyway) into an account you control!

"back up in your ass with the resurrection...."

Here's an important one.

[Apuleius]

Using accurate arithmetics to improve numerical reproducibility and stability in parallel applications, also available here if you're one of the unwashed masses. It has algorithms to see if your app is facing floating point trouble.

If you need more precision...

[cfallin]

Hardware floating point is only so accurate - if you need more floating point (or integer) precision, use GNU MP - a library for C with bindings for many other languages too. It came in quite handy when I wrote some cryptography code with very large numbers.

Floating point operations are not that bad.

[stj]

Well, I have a lot of experience with that since I've been doing numerical computations for last 7 years. First of all, it's not all that bad. With 64 bit 'double' in C, you get around 15 decimal digits of accuracy (theoretically 18, but in practice don't count on the end). You have to understand that numbers are stored in logarithmic format: mantissa and a factor to multiply it (in computers exponent of 2). If there is no overlap between two numbers in addition (that is for example one number is 1.234*2^64, and the other is 1.234*2^-15), the smaller one is always lost. The are two ways to get around it:

extend mantissa so there is enough overlap - usually involves some kind of multiple precision libraries like mentioned in other post GNU MP and many others. I've implemented one for my own use, too. Generally means lots of overhead since there will be less than 5% of operations actually benefitting from greater precision.

postpone such operations until there is overlap - store such numbers together and do operations on them together, too. Sometimes additions in loops will add up small parts so actually there will be overlap with big part and additions can be done with enough precision.
On a side, interesting thing is that in computers multiplications and divisions are better (that is more accurate) than additions and subtractions because of logarithmic format.
I know that Sun was working on a variable precision floating-point CPU. I'm not sure how that project is going and what the end effect is, but I remember it being an interesting idea.

Multiple precision libraries are usually decent with only one problem, they are always slower by a couple orders of magnitude than regular CPU operations, so using them is just such a pain.

Huh?

[joto]

I tried some of the examples in these articles with Intel's Fortran Compiler and g77 and noted that some of those issue reported no longer seem valid whereas quite a few still very much are around.

Would you mind tell us what those "issues" where. Because the articles hardly deal with "issues" at all. What they deal with is the theoretic limitations that must exist in floating point, due to the fact that we have finite hardware, while real analysis assumes infinite precision. This should not have changed between 1991 and now (especially, since we have all standardized on IEEE floating point formats, but even if the article was from 1960, you should easily be able to "translate" it to your favourite floating point format (which is probably IEEE)).

Could someone, please, give me a pointer to some newer thoughts and/or new facts surrounding floating point programming.

There are very few new thoughts with regards to floating point programming, just as there are very few new thoughts on the use of "if-then-else"-branches or "while"-loops. Floating point programming is basically a solved problem. The only problem with it is that it sometimes flies in the face of intuition, and most programmers are ignorant about it. This has not changed since 1991 either.

The articles you mentioned are very good articles for understanding issues surrounding floating point. Just make sure you read them with your brain, instead of just feeding your favourite compiler with any examples you see.

What has been improved since those articles were written?

Speed. Computers have become faster. (It's possible that there also have been some minor software improvements such as an ISO C addendum clarifying tricky areas with rounding modes, or something like that.)

What is still the same?

Essentially, nothing have changed.

How is the future, especially with the new platforms IA64 and AMD64?

Very predictable. Nothing will change there either. Non-IEEE floating point vector instructions, or "multimedia" instruction sets will probably continue to be unstandardized and platform-dependent.

I am most interested in the x86 and x86-64 architectures

There is nothing special about those architectures with respect to floating point (well, the x86 reuses its floating point registers for MMX instructions, but you shouldn't need to know that unless you use assembler).

Real world and catastrophic failures

[Anonymous Brave Guy]

I would really like to know, if there are real world engineering examples, where simulations produced dangerous products, because the simulation was inadequate because of numerical errors. Perhaps in aerodynamics, who knows how they perform their flight simulations.

I've worked on a couple of projects where this is very important. One was writing control software for metrology equipment, industrial strength QA kit that measured manufactured parts down to fractions of a micron or even nanometres to make sure they were in spec. Another was a geometric modelling tool used in CAD applications and the like.

In neither case am I aware of any physical real world failure caused by a problem with the floating point calculations. You do have to be really careful with manipulating the numbers, though.

For example, the loss of significance when you subtract can be horrible if you've got two position vectors close together, and you're trying to calculate a translation vector from one to the other. The error in that translation vector can be enormous if the points you started with were very close: you might get only one or two significant figures, when the rest of your values have 15 or more. If you're interested in the direction of the vector, that can give you errors of +/- several degrees!

Inevitably, there are always going to be bugs in complex mathematical software, and I've seen plenty of wrong answers from programs like the above. Fortunately, it's normally possible to have checks and balances that at least identify and highlight inconsistencies so, in the worst case, at least nobody relies on them. You can also use ruthless automated testing procedures, which run zillions of calculations every night and flag the smallest changes in the results, so no-one accidentally breaks a verified algorithm with a change later. The combination makes it reasonably unlikely that any algorithm would fail catastrophically with the sort of consequences you're talking about.

The possibility is always there, of course, because programming is subject to human error. However, FWIW, I've worked on software that's used to design cars, and software that controls the QA machinery to make sure they're put together right, and I still drive one. :-)

News Flash: Go To considered harmful

[bellings]

I'm assuming that sometime in the next week one of the slashdot editors will be trolled with an article like:

I'm rather new with programming and stumbled across the article Go To Statement Considered Harmful from 1968. I tried some of the examples in this article, and noted that some of those issue reported no longer seem valid whereas quite a few still very much are around. What has been improved since the article was written? Will the new 64-bit architectures finally fix all the problems with Go To Statements, or is this something that the hardware designers still need to work on?