Slashdot Mirror
What Every Programmer Should Know About Floating-Point Arithmetic
-brazil- writes "Every programmer forum gets a steady stream of novice questions about numbers not 'adding up.' Apart from repetitive explanations, SOP is to link to a paper by David Goldberg which, while very thorough, is not very accessible for novices. To alleviate this, I wrote The Floating-Point Guide, as a floating-point equivalent to Joel Spolsky's excellent introduction to Unicode. In doing so, I learned quite a few things about the intricacies of the IEEE 754 standard, and just how difficult it is to compare floating-point numbers using an epsilon. If you find any errors or omissions, you can suggest corrections."
Excellent. The article may not be *that* specific or lengthy, but it helps out most people that have a simple problem. People with more than simple problems should know the standard word-per-word already ;)
Have you heard about SoylentNews?
I seems to me that this problem would pop up any time you worked with an irrational number.
Back in the early days the analog computer was used for things like ballistic calculations. I would think that they would be less prone to this type of problem.
Linearity may still be an issue (analog systems have their own set of problems).
Tisha Hayes
Floating point math should be properly verified using interval arithmetic: http://en.wikipedia.org/wiki/Interval_arithmetic
Damn...Missed it! lol
You really need to talk about associativity (and the lack of it). ie a+b+c != c+b+a, and the problems this can cause when vectorizing or otherwise parallelizing code with fp.
And any talk about fp is incomplete without touching on catastrophic cancellation.
Ian Ameline
This article should mention strictfp in the section on Java.
use BCD math. With h/w support it's fast enough...
Why don't any languages except COBOL and PL/I use it?
"I don't know, therefore Aliens" Wafflebox1
Somewhat a dup from seven years ago ...
Posted by Cliff on Thursday June 26 2003, @04:42PM
http://developers.slashdot.org/story/03/06/26/0044223/Floating-Point-Programming-Today ...ah well
"The floating-point types are float and double, which are conceptually associated with the 32-bit single-precision and 64-bit double-precision format IEEE 754 values and operations as specified in IEEE Standard for Binary Floating-Point Arithmetic , ANSI/IEEE Std. 754-1985 (IEEE, New York)."
http://java.sun.com/docs/books/jvms/second_edition/html/Overview.doc.html
Maybe in your list of solutions you could mention interval arithmetic, it's not very much used, but it gives "exact" solution.
Computers are about cheating.
Do it right and you won't let anyone input enough to get caught out by the bug in your codes hidden methods.
If you think that every language except Java implements IEEE-754 to the letter, you are sadly mistakenly. That fact is Java can be used just fine for floating point work in most applications.
Actually, the linked article says exactly the opposite, and up above I posted a link to the JVM definition that verifies it. So you are 100% incorrect.
What kind of terrible CS program doesn't teach floating-point arithmetic as part of a required course?
to floating point, please, everyone should've read Everything you ever wanted to know about C types and part 2 (which explains fp too).
this will save a lot of time & questions to most beginning (and maybe mediocre) programmers.
Given the great complexity of dealing with floating point numbers properly, my first instinct, and my advice to anybody not already an expert on the subject, is to avoid them at all cost. Many algorithms can be redone in integers, similarly to Bresenham, and work without rounding errors at all. It's true that with SSE, floating point can sometimes be faster, but anyone who doesn't know what he's doing is vastly better off without it. At the very least, find a more experienced coworker and have him explain it to you before you shoot your foot off.
float are over, we should go back to fixed point. With 64bits interger we already have more precision then 32bit float. Let use that 64bit adventage to drop the float madness and leave the 64bit float for a few niche applications.
I'm completely baffled by this (in Python):
>>> print (1/3)*3
0
I expected 1, and my FPU is from AMD, not Intel ;)
Check Goldberg's article, 'What Every Computer Scientist Should Know About Floating Point' for a more in depth discussion.
Here's a one-page intuitive description of floating-point, to give a feel for why multiplication and division are fairly benign, but addition and subtraction aren't. None of the other descriptions have made it as clear as this.
The article gives the impression that base 10 arithmetic is somehow "more accurate". It's not. You still get errors for, say, 1/3 + 1/3 + 1/3. It's just that the errors are different.
Rational arithmetic, where you carry along a numerator and denominator, is accurate for addition, subtraction, multiplication, and division. But the numerator and denominator tend to get very large, even if you use GCD to remove common factors from both.
It's worth noting that, while IEEE floating point has an 80-bit format, PowerPCs, IBM mainframes, Cell processors, and VAXen do not. All machines compliant with the IEEE floating point standard should get the same answers. The others won't. This is a big enough issue that, when the Macintosh went from Motorola 68xxx CPUs to PowerPC CPUs, most of the engineering applications were not converted. Getting a different answer from the old version was unacceptable.
Look, times are tough for programmers already. Knowing how to do things correctly - like proper floating point math - is one of the ways to separate the true CS professional from the wannabe new graduates. Articles like this just make everyone smarter, and make finding a job that much harder.
They would never have gotten to Slashdot front page if they did not include that xkcd reference.
PLEASE stop this thread - interval arithmetic and all that has been around as a research topic for decades now.
If you did not know: there are search engines providing lots of links...
I'm not sure whether that is factually true, but IEEE-754 isn't exactly perfect or without alternatives so I wouldn't base my language choice on it..
That'd be like not using Java because it doesn't represent ints using ones complement; if your code relies on the specific internal implementation of data primitives you're probably doing something wrong.
(Before I get replies: Of course sometimes these things really do matter, but not often enough to dismiss a multi-purpose langauge.)
// MD_Update(&m,buf,j);
Why don't you write it up yourself and give me a github pull request? :)
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
Java have a strictfp keyword for strict IEEE-754 arithmetic.
Another thing I think an article labelled "What every Programmer should know..." should mention is that, while "Nearly all hardware and programming languages use floating-point numbers in the same binary formats", implementations can actually be very inconsistent, in the results of computations.
For example, mac and windows programs will have different rounding behaviors, and come up with different computation results in floating point math (at least with the default compiler settings - does anyone know if it is possible to change this?). This is constantly causing us trouble, when trying to write Unit tests only to realize the expected output values are slightly different on different machines to run the tests on.
I also think that, as somebody posted above, the non-associativity of floating point operations is important and should be mentioned. But, otherwise, a well written and helpful article!
...should include a sample of some code that attempts to perform logical operations based on the outcome of floating point operations. Anyone who attempts to check the result of a floating point operation for equality without normalizing precision should be shown to the door and never called back.
This sounds a lot like the stuff i heard in CS/Mathematics classes more than 20 years ago (Hackbusch, Praktische Analysis, Summer term 1987/88). That course was mandatory then for any CS, Mathematics and Physics student. Has that changed yet? It's about the differences between a number at it's binary representation (and examples about consequences).
I completely agree, that every programmer should know about that. But this is nothing new, it was already important 40 years ago. I'm pretty sure some space probe/missile already got lost due to this problems.
CU, Martin
P.S. It's not only important for programmers (dealing with C, C++, Java, etc.). Several times i was called to debug some "Excel Error" when it turned out to be a floating point restriction.... It would be interesting if you can create a macro that checks for such conditions.
Don't you mean: 99.99999999904% incorrect?
I used to do algorithm ports for science grade code to operational grade for satellite data processing. The hardest part of the job was trying to convince the phds that considered themselves 'expert programmers' that we hadn't screwed something up in the port when the answers didn't exactly match. I'm not sure if its because they didn't think we had anything of value to contribute to the process, or if they were freaked out by the fact that all of the accuracy they had been touting was only perceived instead of actual. To their credit it is pretty difficult to convince someone that everything is working just fine when the algorithm takes a single sensor value representing an 800 sq m swath of land and spits out 'desert' on their system and 'tundra' on ours.
That's what he said, it just got rounded.
Because I don't know how to use github and it looks to me as a really, really complicated way to make a wiki..
On your page about comparison, you give the following code:
function nearlyEqual(a,b)
{
return a==b || Math.abs((a-b)/b) 0.00001;
}
if(nearlyEqual(a,b))
I think it will fail with a division by zero if e.g. a==2.0 and b==0.0.
Nothing at all.
Someone else has already written a library for the hard part of what you're doing, and their code *works*. Use it.
Well, I'll do it when I get around to it. Doing it as a wiki would mean that I'd have to deal with vandalism and spam, and it's really intended more as small self-contained site than a community thing.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
I remember when I first read it: it was all wrong, utterly confusing several notions. I flamed that donkey to death and he updated/changed his Unicode explanation. It was really a "ah ah, I just understood why Unicode is good, let's blog about it"-type of article, full of errors.
That guy is not a good programmer at all. Somehow commented here on /. a while ago that Joel Spolsky was approximately saying something interesting as often as sunlight shines in one's arse. Honestly the only thing that Spolsky said that I found interesting was when he said that he created stackoverflow.com because he was tired of "looking for good speakers" and finding endless "meta" discussion about the speakers-forum ("why is my comment in bold?") instead of finding discussion about speakers itself.
stackoverflow.com kinda works (it's a hive full of retards but it kinda works once you ask "advanced", like help for some elisp piece of code) and for that kudos to M. Spolsky...
But his articles are typically "ah ah moments" full of errors. He's not a good programmer and should not be quoted nor used as a reference.
People interested into floating point math will find some very interesting materials and horror stories in the documents collected at the home page of professor William Kahan, the man behind IEEE754 standard.
According to my personal experience the paper by David Goldberg cited in the post isn't that difficult after all. Plenty of interesting materials can also be found in the Oppenheim & Shafer textbook about digital signal processing.
Over at Evans Hall at UC/Berkeley, stroll down the 8th floor hallway. On the wall, you'll find an envelope filled with flyers titled, "Why is Floating-Point Computation so Hard to Debug whe it Goes Wrong?"
It's Prof. Kahan's challenge to the passerby - figure out what's wrong with a trivial program. His program is just 8 lines long, has no adds, subtracts, or divisions. There's no cancellation or giant intermediate results.
But Kahan's malignant code computes the absolute value of a number incorrectly on almost every computer with less than 39 significant digits.
Between seminars, I picked up a copy, and had a fascinating time working through his example. (Hint: Watch for radioactive roundoff errors near singularities!)
Moral: When things go wrong with floating point computation, it's surprisingly difficult to figure out what happened. And assigning error-bars and roundoff estimates is really challenging!
Try it yourself at:
http://www.cs.berkeley.edu/~wkahan/WrongR.pdf
Nice attitude. "Hey, I think this would be a good idea". "Shut up and do it yourself instead". Typical response from an open source developer; aggressive, smug and defensive.
Most of you are too young to have dealt with architectures like VAX and Gould PowerNode with their awful floating point implementations.
IEEE 754 has problems but it's a big improvement over what came before.
It's missing the irritating cutesy "humor".
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
I'm very sorry for whatever horrible things happened to you that makes you read that into my words. They were meant as an acceptance of the suggestion that a section about interval arithmetic would be a good addition to the site and an invitation to contribute.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
Python has a human-centric method of working with floating-point numbers: the Decimal class.
... to use proper test cases:
http://www.xkcd.com/217/
The real question is, if your coding safety critical applications why aren't you using the appropriate language e.g. Ada?
Java is a good programming language if you know how to use it and you can write some very efficient and small code. I'm tired of people attacking it for being slow when the people who use don't have a clue how to use it properly and just iterate their way through array lists (probably the most common mistake I see).
You should always use the right language for the right job.
I was brought in a bit after the start of a state project to write a system to track about a half billion dollars in money for elderly and disabled indigent care. I was horrified to find that all the money variables were float. After raising the issue and explaining the technical details, I proposed a Money class and if they didn't want that gave them the option of a simple fix: just change all the floats to long and keep the amounts in pennies, inserting the decimal point only when displaying numbers. The tech lead (nice guy, not the sharpest crayon in the box) and the DB architect (who was the sort for which Codd was God and for which the DB2 NUMERIC datatype was all we needed) determined that they could find no discrepancies, so the problems I cited must not apply to their code.
Fast forward several months later, when the numbers started to not add up, just as I explained. Killed a significant amount of time retrofitting the code to use BigDecimal. Even then, there were problems. Not the only reason the project was shut down before tackling the remaining planned features, but certainly a factor in it.
If anyone with ACM digital libray access wants the DOI link to the original article, rather than the edited version Sun/Oracle's site it's http://doi.acm.org/10.1145/103162.103163.
It is an old article though, so it's a 44 page scanned PDF.
Yeah, I had a sig once; I got bored of it.
True, but in many cases (those that don't require too much performance) it is probably better to use BigDecimal anyway. I've never used that keyword, IMHO it's for inner libs only. It's certainly not something I would learn starting programmers (just as you should not learn new programmers the use of wait() or sleep(), rather CountDownLatch and the different Queue's).
I think the original poster was referring to this piece by the father of floating point, William Kahan, and Joe Darcy
"How Java's Floating-Point Hurts Everyone Everywhere"
http://www.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
What is it that lets Java have such a bad name on SlashDot? Even articles that are typical flamebait like the one above get modded up. It's open source, it's relatively secure, it's got loads of libraries and IDE's and developer support. It's even pretty fast compared to most competitors. It's definitely not a closed eco system either. Of course it's got its drawbacks, but come on...
sound interesting - are there any formal automated means of testing / checking tho? (like a valgrind for numbers?).
From the guide: "If you really need your results to add up exactly, especially when you work with money: use a special decimal datatype."
No!
Use a decimal representation when you need representable numbers to match with the operator's/programmer's idea of what should be representable exactly (e.g. financial data).
Use arbitrary precision (which can be implemented with any base, but 2^32 is popular) when you need results to add up exactly.
Some Mathematica code:
The result: http://www.untruth.org/~josh/real-rounding.png
It's probably not perfect, but IMO the reason for sticking with 754 unless there's a real need for something else is:
Repeatability. If your code and language are standard-compliant, then you'll get the same floating-point math results as someone using another compliant language on any other platform. Not crucial for some tasks, but it certainly is for others, such as scientific work.
Hail Eris, full of mischief...
E pluribus sanguinem
Which, to most people, looks like the best way to convert a number from single to double precision in Java.
For this reason, you may also want to reference the forgotten class sun.misc.FloatingDecimal. And I still wonder why assigning floats to doubles in Java leads to a cast with no warning (which is just wrong), while they have the necessary conversion facilities available.
Example:
double myDoubleFromFloat = new FloatingDecimal(myFloat).doubleValue();
If you check the conversion code, the 3000+ lines in that class are totally not trivial.
Well, so much for my smug mathy reply! Amusingly, the reason it worked so well is that my "Evaluate" statement asked Mathematica to symbolically evaluate the function prior to graphing. So, the graph looked nice because Mathematica was just graphing the "Abs[x]" function!
Without the symbolic evaluation or requesting a particular precision level, the graph you actually get is this:
http://www.untruth.org/~josh/real-rounding-oops1.png
You can get a more reasonable looking answer by messing with the calculation precision and accuracy...
And I've seen the addition of two money columns defined in Access get magical values. I'm sure somebody here can explain the situation here better than I can, but I've seen $1.00+.50 become $1.49. But in MS Access's defense, a float is a poor way to define money especially in MS Access. I was just hired to bandaid the broken solution.
``What is it that lets Java have such a bad name on SlashDot?''
I can only speak for myself, but the beef I have with Java is that it was promoted as if it was a great step ahead in programming languages and really the right way to do things. A lot of people fell for the propaganda (including myself) and an enormous amount of time has been spent on Java since - writing software with it, improving the platform, re-implementing things from other platforms for Java, basing research on Java, etc.
In reality, Java wasn't actually all that great. None of the features that were touted with much fanfare at the time were actually new. The language had a pretty lame type system and a lame implementation. If, at the time, I had known the languages I know now, I never would have thought Java was a great language. But I didn't know many languages back then, so I swallowed the propaganda and was really enthusiastic about Java. I can only assume the same has happened to many other people. At any rate, Java took off and went on to become a dominant language (not to say _the_ dominant language) in the software business and many universities.
To be sure, Java has changed a lot since its inception, and there definitely has been improvement. The type system is much better now than it used to be, and there are now several implementations for different niches, some of which are very impressive indeed. Still, after 15 years of enormous exposure, success, and development, I still find the language and platform lacking. Many features or missing features in the language make it difficult to express certain programs in their most natural form, often requiring much more code to write in Java than in another language. The standard library, while indeed extensive, is full of rough edges, bad APIs, and long-unfixed bugs. Many features of the host platform are simply not available in Java. Very few things in Java or the popular frameworks strike me as done right.
Now, what bothers me is not that there are features of Java I don't like. Even if they could be shown to objectively make Java less good than some other platform or programming language, I wouldn't care very much. My problem is that Java has these shortcomings, has long had such shortcomings, and is _still_ being touted by many as the greatest platform out there, still receives enormous amounts of research and development effort, and is still extremely widely used _and_ taught to students. To me, that's a lot of energy, time, and money that we, as humanity, could have more effectively spent on other things. The way I see it, Java has been a colossal waste of time, and I blame the vocal advocacy for that.
Now, what any of this has to do with IEEE 754, I don't know. As far as I am concerned, fully IEEE-754-compliant is better than just IEEE-754-like, but only marginally so - in either case, your results will be inexact and there will be many gotchas. Java does have BigInteger and BigDecimal types, which I usually find better choices than floats.
Please correct me if I got my facts wrong.
I would REALLY like to see an "FPU" that keeps everything as fractions until explicitly forced to reduce to a floating point (if ever). Maintain a numerator and denominator register. Only reduce when the other choice is an overflow or where a common denominator operation is forced (addition, subtraction, or an explicit instruction) at least as an option!
It's truly stunning how often very detailed and compute intensive modeling where human lives depend on the correct answers will go astray because of brittle floating point. The very best case for that is when the errors compound quickly and cause obviously wrong answers, but I wonder how often the not so obviously wrong answer is accepted as correct. It's common enough in modeling to fiddle slightly with the start conditions to get the model over a rough patch and produce an answer. Perhaps go with 1.00001 second intervals rather than 1 second exactly. Perhaps bump something a micron to the left and so avoid a divide by zero on the 20,000th iteration. Or worst, compile with -O5 rather than -O6. Most of the time it works out OK, but it would be a lot more comforting if all that fiddling wasn't needed. You can only flip off the daemons of chaos so many times before you get squashed.
No adds or subtracts, but 128 invocations of a mysterious square root function that doesn't involve adds, subtracts, or divisions.
Therefore the square root function is the problem, and the reason is that it is implemented as a ghastly huge lookup table. This program never gets loaded into memory, let alone has a chance to actually run.
The "singularity theory" is just a red herring.
Can you be Even More Awesome?!
Java has forced objectification whilst at the same time lacking proper introversion. Since we most people here prefer lisping, especially the Pailinites that's just not acceptable.
=~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
Internal arithmetic always includes the exact solution, but only the rarest circumstances does it actually give the exact solution. For example, an acceptable interval answer for 1/3 would be [0.33,0.34]. That interval includes the exact answer, but does not express it.
FTA:
"Only 257 algebraic operations, so no hordes of rounding errors."
Say what?
We know where leadership by an anti-intellectual "strongman" who scapegoats minorities and likes boisterous rallies goes
Note that the cited paper location is docs.sun.com; this version of the article has corrections and improvements from the original ACM paper. Sun has provided this to interested parties for 20odd years (I have no idea what they paid ACM for rights to distribute).
http://www.netlib.org/fdlibm/ is the Sun provided freely distributable libm that follows (in a roundabout way) from the paper.
I don't recall if K.C. Ng's terrific "infinite pi" code is included (it was in Sun's libm) which takes care of intel hw by doing the range reduction with enough bits for the particular argument to be nearly equivalent to infinite arithmetic.
Sun's floating point group did much to advance the state of the art in deployed and deployable computer arithmetic.
Kudos to the group (one hopes that Oracle will treat them with the respect they deserve)
Can someone use a car analogy or even a Soviet Russia one?
Interval Arithmetic does *not* produce exact solutions, it provides upper and lower error bounds on the calculation. Most IA implementations introduce a bias on every bounds calculation to insure that the "exact" result is proven to be bounded by the interval. I think some SPARC processors had IA support.
Nice info, but as I have understood the sun.* package hierarchy is not part of standard Java and thus the FloatingDecimal class might not necessarily exist on a non-Sun JVM.
1. Some of it is just inevitable backlash. Many moons ago Java was hyped as the greatest thing ever that would magically solve all our problems. Of course it didn't, and anytime something gets overhyped, there is pushback.
2. Bad client experience. Java on the server is one thing, but client apps? Ugh. I've got four, and each one is picky about wanting its own version of the JRE. They're slow. They tend to look like Java apps rather then normal OS apps, so they don't really fit well and often don't follow conventions.
Some of that is just programmers, but I've never used a client side Java app that wasn't slow. These people who keep saying "Java isn't slow" need to point me to their magical fast JRE. Or maybe they mean it's not slow, except when doing UI stuff, because I've NEVER seen a UI that wasn't sluggish compared to a normal app.
3. It's not the shiny new thing anymore, and it's not buzzword compliant.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
Repeatability. If your code and language are standard-compliant, then you'll get the same floating-point math results as someone using another compliant language on any other platform. Not crucial for some tasks, but it certainly is for others, such as scientific work.
Wouldn't it be great if you could change a switch in your computer to change all double precision fp from 53 bit mantissa to 52 bit, and if your results are suddenly radically different then you know your first set of results couldn't be trusted?
Repeatability is highly overrated. It's no good if you get the wrong results, and a different computer system gets you identical wrong results.
Because I remember reading some programmer complain about this on bug reporting web site like 10 years ago.(I think it was bugwatch.com) The guy said he had 20 years of experience and ran into it and thought it was a bug in Java. Anyway to be serious this didn't really catch me because I've taken chemistry 101. Oddly enough part of that courses was significant figures. Once you're used to sig figs you kind of get why it's not a big deal that the values seem a bit "off".
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
You need a cock in your ass today.
I respectfully suggest that that would only worsen his mood...
i'd hit it so hard, if you pulled me out you'd be the king of britain [bash.org]
All be it predates IEEE 754 there is Semi Numerical Algorithms by Knuth. It is in the nature of everything you always wanted to know about floating point but were afraid to ask.The volume is very mathematical.
Actually the fundamental problem is very simple, there are an infinite number of numbers on the real number line (between any two numbers on the real number line there is another number...) However computers are
machines with a finite word length so that the basic theory of real numbers is not valid on the computer, because it is not necessarily the case that between any two floating point numbers on the computer
there is another number. Thus the computer can not represent all numbers which is the source of the problem. The approximation required each time a floating point number is manipulated case this "error' to propagate.
Of course this problem existed even when doing calculations with paper and pencil, since one does not do an infinite length calculation, but when working by hand the number of calculations is so much less that
the problem does not become evident.
Typical Spolsky: speaking with authority and getting everything wrong.
For the latest version of CityDesk, the web site management software published by my company, we decided to do everything internally in UCS-2 (two byte) Unicode, ... In C++ code we just declare strings as wchar_t ("wide char") instead of char and use the wcs functions instead of the str functions (for example wcscat and wcslen instead of strcat and strlen). To create a literal UCS-2 string in C code you just put an L before it as so: L"Hello".
UCS-2 is an obsolete character set; Windows actually uses UTF-16. And wchar_t is not a Unicode character.
Modders, this might look like a troll in shape but it is very insightful.
Even now, that article isn't very good at all. It isn't even passable. Go somewhere else for a good introduction to Unicode and why it is important.
This isn't an article but has far better information if somehow hurdled by the author's need to give us his opinions on the matter.
As a programmer, the one most important thing you need to know about floating point is never test for equality (even with zero), almost define how near the answer need be, and test against that. double a,b; double epsilon = 1.0d-10;
if (a==b){ // Bad practice
if ( abs(a-b) .lt. epsilon){ // Better practice
--- Mathematical Programming Feed @ Feed Distiller
how about the FACT .. that no matter how many have been taught to use it .. or how hard you want to believe in it .. decimal math is a LIE .. it is one of if not the main reason why .. there are so many MIS-CON-ceptions in our current UNDERSTANDING about life and the universe .. and the reason for the required need of approximation when using it ..
what we call life and the universe are fractional .. fractal and absolute .. and it does not matter to how may digits you can calculate pi .. it is an exact absolute fractional relationship .. there is NO decimal equivalent .. and can be nothing but an approximation ..
as for the concept of rational and irrational numbers .. again a MIS-CON-ception .. it is not that you can not divide by zero .. it is just you do not understand the the meaning and significance of the answer .. and so most persist in calling them irrational .. interesting but expect-able that these were the terms used to describe these kinds of numbers .. rational and irrational ..but of course we were naive enough to to BASE our computing system on binary computation(great for drawing straight line but nothing else) .. so i guess it was to be expected .. until we look at both 10 base and 12 base number systems and their interrelationships .. we will persist in our MISS-understandings about life in the BIG picture ..
and just to add insult to the injury .. particle physics is also a LIE .. there is NO part of MATTER smaller that the hydrogen ATOM .. looking for them is just a colossal waste of time money and human effort .. as a consequence of our MIS-CON-ception and MISS-understanding ..
best stop there ..
as we would not want those who have partaken in the knowledge of GOOD and EVIL gaining access to everything .. but then again they might figure out why it was such a bad idea in the first place ..
hint ..
it has a EVERYTHING to do with strict binary computation .. can draw straight lines in space .. but USELESS for plainer reference or relativity ..
The standard is essential as it allows compatibility across CPUs.
For example, we write Windows (hence Intel/AMD) applications that communicate to Analog Devices DSPs.
What are
* Catastrophic cancellation in complex arithmetic?
and
* Range reduction in trig functions?
Anyone that wants to know about the real dangers of computer precision just needs to watch Office Space!
I have left slashdot and am now on Soylent News. FUCK YOU DICE.
Floating point meth addicts need to stop asserting their point is not floating when it clearly is. Why can't I set a float to 1 and expect 1 == 1.00?. Your answer is an EXCUSE.
I'm sick of modular arithmitic, I'm sick of crappy floating point logic. I've used enough cell libraries to know why its done and why it works the way it does. It still doesn't explain why our favorite languages STILL force us to think in these crappy terms without having to resort to quirky external meth libraries to get any real work done.
I may be biased, but I prefer this article by numerical analyst and mathematician Cleve Moler.
``What is it that lets Java have such a bad name on SlashDot?''
I can only speak for myself, but the beef I have with Java is that it was promoted as if it was a great step ahead in programming languages and really the right way to do things. A lot of people fell for the propaganda (including myself) and an enormous amount of time has been spent on Java since - writing software with it, improving the platform, re-implementing things from other platforms for Java, basing research on Java, etc.
I've been a follower of Java since version 1, and I don't see any difference with other languages. If anything, Java has always been a C/C++ kinda language but much simplified and, indeed, objectified. There are a lot of things that have been done with this re-implementation, and in many cases, rightfully so.
In reality, Java wasn't actually all that great. None of the features that were touted with much fanfare at the time were actually new. The language had a pretty lame type system and a lame implementation. If, at the time, I had known the languages I know now, I never would have thought Java was a great language. But I didn't know many languages back then, so I swallowed the propaganda and was really enthusiastic about Java. I can only assume the same has happened to many other people. At any rate, Java took off and went on to become a dominant language (not to say _the_ dominant language) in the software business and many universities.
Actually, it was and is pretty great. Of course, looking in retrospect, it did some things wrong. Many of those things like the type system were actually thought of as "inefficient" at that time. Now most language purist would like Java to have each and every variable to be an object instance. If they had taken that route back then, they would have been scorned by most of the community.
To be sure, Java has changed a lot since its inception, and there definitely has been improvement. The type system is much better now than it used to be, and there are now several implementations for different niches, some of which are very impressive indeed. Still, after 15 years of enormous exposure, success, and development, I still find the language and platform lacking. Many features or missing features in the language make it difficult to express certain programs in their most natural form, often requiring much more code to write in Java than in another language. The standard library, while indeed extensive, is full of rough edges, bad APIs, and long-unfixed bugs. Many features of the host platform are simply not available in Java. Very few things in Java or the popular frameworks strike me as done right.
Most of the basic objects are now immutable. There are good ways of handling threads with a brilliant java.util.concurrent lib and pretty well worked out collection framework (which is a lot easier and safer than the boost libs, imho). The date and time API still is a mess, but that is being fixed. Java has a good regexp lib, good well defined string handling and many more well defined libs. Of course, new API's (of new languages) will build on this and hopefully be better than Java. But take a look and see how many fail; well designing core classes is no easy task. It's just to big to give all to James and Joshua.
Now, what bothers me is not that there are features of Java I don't like. Even if they could be shown to objectively make Java less good than some other platform or programming language, I wouldn't care very much. My problem is that Java has these shortcomings, has long had such shortcomings, and is _still_ being touted by many as the greatest platform out there, still receives enormous amounts of research and development effort, and is still extremely widely used _and_ taught to students. To me, that's a lot of energy, time, and money that we, as humanity, could have more effectively spent on other things. The way I see it, Java has been a coloss
Ah - a Microsoft Sharepoint programmer
"Cats like plain crisps"
Don't use float. Ever. Just use Decimal and Integer types. There's never a reason to use floating point math. They should have never invented it.
Catastrophic cancellation -- suppose you have C1 = a + bi, C2 = x + yi. Multiplying them yields (ax - by) + (ay + bx)i. Done "naively", those embedded subtractions/additions can lead to excess cancellation. The "easy" way to deal with this is to do the intermediate multiplications (ax, by, ay, bx) into higher (doubled) precision, do the add/sub in the higher precision, and then round down to get your answer. Works pretty well for single (because double is always available), not so well in double (because quad is sometimes architected, but not supported except in emulation). There are probably some tricks for doing this well with IBM Power's fused multiply-add.
Range reduction is substantially more subtle, and it requires that you understand a particular point of view. That POV is that each machine floating point number is exactly what it is -- the representation of pi (the 64-bit mantissa approximation that you get in 80-bit Intel FP) is expressly, NOT PI, it is a different number, very close to PI. So, consider sin(PI+x) for a moment, for very small x. That should be very nearly equal to -x. Now, suppose you take a 64-bit mantissa approximation of PI, call that "pie". I happen to know that it is slightly larger than PI, hence the "PI+x" formulation (bits 65-68 = hex C) (I may be making sign errors here). Suppose you want to computer sin(something), where something happens to be pie. The first thing you notice is that it is (a) larger than PI and (b) also larger than the 68-bit approximation of PI. So, either way, you need to do range reduction, subtracting either a sufficiently large approximation of PI, or subtracting a 68-bit approximation of PI. If you subtracted true PI, what you get is 64 (63?) zeroes in the mantissa, followed by 1 at the 64th position (because pie is rounded up) minus all the following digits of true PI. If, on the other hand, you subtract the 68-bit approximation of PI, you get 64/63 zeroes followed by binary 0100 (16-12), and all the rest is ZEROES. For the "pie is a real true number" POV, sin(pie) is not, not, not 4*2**-64 -- it's that number with all the (subtracted) digits of pi.
I thought I could get an example of this with comparing C and Java on an Intel box, but gcc on my Mac is too clever and keeps calling a subroutine instead (Apple was once very, very picky about floating point, and it would not surprise me if this is special to them).
Easy, java developers are apparently easy to troll. Every time there is a discussion where someone can take a shot at java, someone does in order to get a rise out of people.
I'm a good cook. I'm a fantastic eater. - Steven Brust
There are some decent points there, but a lot of them aren't really related to IEEE 754 compatibility. For example, bullet point #5 on their first-page list of five "gratuitous mistakes" is that Java doesn't support operator overloading. But by that standard, C sucks too, and yet is somehow used in lots of floating-point libraries.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Been dealing with some issues exactly like this myself. Is there ever a situation where you would work with floats rather than integers, related to what you were talking about? Only for presentation?
Selah.ca. Pause, and calmly think on that.
"Why is Floating-Point Computation so Hard to Debug whe it Goes Wrong?"
That's nthng. Strngs r hardr to dbug.
Generally speaking, it's never fine to use floats in financial calculations if you want exact results. For approximations it's ok, but why bother when you can get perfect precision using fixed point integer arithmetic. You might have concerns with bounds using traditional ints but that's why you use BigDecimal or similar.
I believe strictfp is the keyword you were looking for to get 100% IEEE compliant floating point in Java. Welcome to 12 years ago.
That is the biggest problem with Java.
1. To use Java well you need to be a good programmer.
2. A good programmer doesn't want to use Java.
The problems I've seen most programmers make is initializing and configuring objects (like full blown XML parsers, which takes over a ms to initialize and only a few us to use and reset) in the performance critical part of the program. This can be easily solved by creating object on demand, but putting them on a shared stack when you are done with it for reuse by an other thread.
This brings back nightmares involving macheps and cancellation for me...
It's weird because I actually have a mathematics degree, plus a lot of CS coursework, I took the numerical analysis class that covered most of this, and yet I work in a factory where I only ever use basic geometry and the ability to sort fractions very, very quickly. (Quick, is 5/32 more or less than 1/4? Now repeat similar exercises ~1000 times/night...).
Anyone need a jack-of-all-trades computer guy for something other than measuring and putting sticky labels on thousands of pieces of cut and tempered glass?
My mom also sometimes complains when I am not home in time - because of my inaccurate watch. I heard the internet was full of investigations into and solutions to this problem but I prefer DADT (don't ask, don't tell) tactics...
Correction: COBOL, PL/I and Ada. Ada has both fixed and floating point BCD arithmetic. And yes I too wonder why it is not in wider use. Perhaps it has to do with the ill conceived notion of "light weight languages" - most of which are not light weight at all any more once they are on the market for decade or so.
Martin
I'm not comparing BCD to floating point
BCD and floating point do not exclude each other. There is BCD floating point as well. And this is what we are talking about. Integer and fix point are always precise.
Martin
The notion of "precise" only comes into play when the number is presented to humans. And humans expect 1/5 to be precisely 0.2. And especially when that 0.2 is to appear on your bank statement.
If you never plan to display your data to humans then it does not matter which floating point you use.
Martin
Actually it's pretty important for multiplayer games to get deterministic results, because this is how many RTS and sports games are networked: starting with the same initial state they lockstep the player inputs forward in time, implicitly synchronizing the game without sending the state. For example most RTS games use this technique (eg. “Starcraft”, “Age of Empires”) also some physics simulation games like “Little Big Planet” do too, plus most sports games you play online as well.
And yes in this case, there is really no right or wrong. All you care about is the same result. *Exactly* the same result. Down to the bits. Unfortunately IEEE-754 makes no guarantee of this, and you'll find that in practice you'll get slightly different results across different architectures, compilers, hell - even between release and debug build.
http://gafferongames.com/networking-for-game-programmers/floating-point-determinism/
http://www.yosefk.com/blog/consistency-how-to-defeat-the-purpose-of-ieee-floating-point.html
Thanks for your thoughtful response.
To answer a few of the points and questions you raise:
``I've been a follower of Java since version 1, and I don't see any difference with other languages.''
That is sort of the point I was trying to make. Like every other language/platform, Java has its strengths and weaknesses. And that is to be expected, really.
I wrote: "In reality, Java wasn't actually all that great."
And then, you wrote: "Actually, it was and is pretty great.''
Right. I think I may not have expressed myself as clearly as I should have over there. What I meant to say is that Java wasn't (at the time) as great as the evangelists would have you believe. Like you said, you used Java since version 1, and don't see much of a difference with other languages. Java has its strengths and weaknesses. But that wasn't the word that was going around at the time. What I heard and read basically amounted to Java being the Messiah among programming languages, that it would deliver us from all the faults found in other programming languages, that we would write once and run anywhere, and so on. That's what I was referring to when I wrote Java wasn't actually _that_ great.
``Waste of time? Good gods, what other language would YOU have chosen 10 years back to develop major systems with? Yes, in retrospect you would not use basic types that overflow, single return values, have checked exceptions above unchecked ones, an underdeveloped module system, null-able references, mutable arrays ... the list goes on.''
Truth be told, I can't give you a good answer to that question, other than "I don't know." When Java was first released, the only languages I knew how to program in were a few BASIC dialects and x86 assembly. You asked about 10 years ago, so that would have been 2000. I also knew C and PHP, then. Compared to those, Java would have been the better choice for most application development work (though I preferred PHP for web development at the time). However, I note that, by 2000, such languages as Ruby and Objective Caml were available (which I prefer over Java now), and even in 1995, Ada, Common Lisp, Delphi, Dylan, Eiffel, Erlang, Haskell, Modula-3, Obective C, Python, Smalltalk, and Standard ML were already around. Many of those contain good ideas that a lot of people think were introduced by Java, and many of them include good ideas that I feel Java lacks.
The point I am trying to make here is that the folks behind Java certainly did a good job of getting their platform and programming language widely adopted, but in terms of making it the best platform and programming language out there, they could have been a lot better. And the thing that annoys me is not so much that Java didn't adopt all the great ideas out there (I know building a platform and programming language is hard and a lot of work), but that the world seems to have pretty much adopted Java to the exclusion of everything else, so that the great ideas that Java doesn't incorporate are still not mainstream 25 years later.
``I do try many "generic" languages out there (ECMA Script, C#, LUA, Go hell even Erlang, PHP and Perl and C++ to name a few).''
That is great, and I hope you keep doing it. And if you keep coming to the conclusion that Java is best for your needs, more power to you. You should definitely go and use it.
Please correct me if I got my facts wrong.
This is one of the reasons that Cobol refuses to die. It is all done in integer maths, ie no round-off unless you specifically ask for it.
You missed it too.
No... the #1 floating point rule is: do not compare for equality
"Video bona proboque; deteriora sequor." -- Ovid
If you want a good paper to reach as many people as possible, it's probably a better idea not to mention Joel Spolsky in the article at all. The camp is so split where he is concerned that you may lose half your readers at the very mention of his name.
BeauHD. Worst editor since kdawson.
Author: I think this article is simply very well written. I plan to send a link to all the CS professors I know.
If only the Python documentation were written half as well.
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
As I understand it there are two sides to the IEEE 754 issue.
The first is the data format. Most modern systems use the IEEE 754 formats so you can pass data arround pretty freely (there are some potential endian issues but those aren't too bad to deal with).
The second is rounding rules and the size of intermediate values etc. Afaict very few systems get these exactly as the standard specifies. For a well known example C compiled for the x87 will store values in floating point registers in much higher precision than the same value stored in memory in IEEE format.
Compilers will typically store variables both in memory (in the declared format) and in floating point registers (in the FPU internal format) depending on current register allocation pressure.
This is only a problem if you consider repeatability more important than accuracy.
Java is actually pretty unusual in giving you the option (though it doesn't do it by default) to force all intermediate values to be rounded to the variable's declared size.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Protip: Search for “strictfp” on http://floating-point-gui.de/languages/java/
Oh, and to the OP: EPIC FAIL!
Any sufficiently advanced intelligence is indistinguishable from stupidity.
You're such an idiot. Yeah, it "works" but the math is imprecise. When deterministic math is needed, fixed point is used. When dealing with money, a decimal fixed point, like BCD is used. Floating point doesn't just "work." You aren't a professional coder, are you?
Show me ONE language standard that requires 754 compliance? C doesn't. C++ doesn't. Hell, even Ada doesn't.