Slashdot Mirror
Ask Slashdot: How Reproducible Is Arithmetic In the Cloud?
goodminton writes "I'm research the long-term consistency and reproducibility of math results in the cloud and have questions about floating point calculations. For example, say I create a virtual OS instance on a cloud provider (doesn't matter which one) and install Mathematica to run a precise calculation. Mathematica generates the result based on the combination of software version, operating system, hypervisor, firmware and hardware that are running at that time. In the cloud, hardware, firmware and hypervisors are invisible to the users but could still impact the implementation/operation of floating point math. Say I archive the virutal instance and in 5 or 10 years I fire it up on another cloud provider and run the same calculation. What's the likelihood that the results would be the same? What can be done to adjust for this? Currently, I know people who 'archive' hardware just for the purpose of ensuring reproducibility and I'm wondering how this tranlates to the world of cloud and virtualization across multiple hardware types."
Use Fixed-point arithmetic.
In Mathematica make sure to specify your precision.
Look at 'Arbitrary-Precision Numbers' and 'Machine-Precision Numbers' for more information on how Mathematica does this.
The result is always the same, but the definition of reality is changing. The result of every single calculation is in fact 42 in some units. The hard part is figuring out the units.
Some drink at the fountain of knowledge. Others just gargle.
Floating point and integer operations are well defined.
Unless someone fucks up with implementing the floating point unit the result should be exactly the same.
Really, forget it. In 5 or 10 years cloud providers will not offer any compatibility to your current form of backup.
(And I really hope people miraculously realize that going to cloud does not solve any problems but creates more and more complex, and abandon the cloud in herds)
This problem is far broader than arithmetic. Any distributed system based on elements out of your control is bound to be somewhat unstable. For example, an app that uses google maps, or a utility to check your bank account. The tradeoff for having more capability than you could manage yourself, is that you don't get to manage it yourself.
First sentence seems stilted at best.
Some days it's just not worth
chewing through my restraints.
Just scroll down a couple of posts. "Quite soon the Wolfram Language is going to start showing up in lots of places, notably on the web and in the cloud."
Problem solved!
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
If the value your computing is so dependent of the details of float point implementation that you'er worried about it, you probably have an issue of numerical stability and the results you are computing are likely useless, so this is really a mute point.
What the title says - e.g. bignum for Python etc. It will be significantly slower, but the result is going to be stable at least for a given library version, and that is far easier to archive.
Mathematica in particular uses adaptive precision; if you ask it to compute some quantity to fifty decimal places, it will do so.
In general, if you want bit-for-bit reproducible calculations to arbitrary precision, the MPFR library may be right for you. It computes correctly-rounded special functions to arbitrary accuracy. If you write a program that calls MPFR routines, then even if your own approximations are not correctly-rounded, they will at least be reproducible.
If you want to do your calculations to machine precision, you can probably rely on C to behave reproducibly if you do two things: use a compiler flag like -mpc64 on GCC to force the elementary floating point operations (addition, subtraction, multiplication, division, and square root) to behave predictably, and use a correctly-rounded floating point library like crlibm (Sun also released a version of this at one point) to make the transcendental functions behave predictably.
Most of the time arbitrary precision is not necessary and it's easier (and faster) to just use a float. There are times when it matters, but for the most part people aren't doing things where it matters.
The submitter should know better about using integer operations for things that require precision though.
Different results on different hardware was a major problem up until CPU designers started to implement the IEEE754 standard for floating point arithmetic. IEEE754 conforming implementations should all return identical results for identical calculations
However, x86 systems have an 80-bit extended precision format and if the software uses 80-bit floats on x86 hardware and then you run the same code on an architecture that does not support the x86 80-bit format (say, ARM or Sparc or PowerPC) then you are likely to get different answers.
I think newer revisions of IEEE754 have support for extended precision formats up to 16-bytes, but you need to know your hardware (and how your software uses it) to make sure that you are doing equal work on systems with equal capabilities. You may have to sacrifice precision for portability.
When information is power, privacy is freedom.
another important question is what makes you think the numbers you are getting now
are "correct" or just what the computer is telling you they are...
if they are "mostly" correct who cares if the definition of mostly changes in the future.
If your calculations are processor-dependent, that's a bad sign for your code. If your results really depend on things that can be altered by the specific floating-point implementation, you need to write code that's robust to changes in the way floating-point arithmetic is done, generally by tracking the uncertainty associated with each number in your calculation. (Obviously you don't need real-time performance since you're using cloud computing in the first place.) I'm not an expert on Mathematica, but it probably has such things built in if you go through the documentation, since Mathematica notebooks are supposed to exhibit reproduceable behavior on different machines. (Which is not to say that no matter what you write it's automatically going to be reproduceable.
Archiving hardware to get consistent results is mainly used when there are legal issues and some lawyer can jump in and say, "A-ha! This bit here is different, and therefore there's some kind of fraud going on!"
I used to read Caltizzle. I was a lot cooler than you.
http://www.smbc-comics.com/?id=2999
It depends on what you mean by "cloud", which is sort of a catchall term. As you've pointed out, on SaaS clouds you're going to have no guarantee of consistency, even if no time passes -- you don't know that the cloud environment is homogeneous. For (P/I)aaS clouds, you can hopefully hold constant what software is running. For example, if you have your Ubuntu 12.04 VM that runs your software, when you fire up that VM five years from now, its software hasn't changed one bit. You of course have to worry about whether or not the form you have the VM in is even usable in five years. You would hope that, even with inevitable hardware changes, if none of the software stack changes, then you'll get the same results. I'd guess that if they're running all on hardware that really correctly implements IEEE floating-point numbers, than you will in fact get consistent results. But I wouldn't bet on it.
What you really need, unfortunately, is a library that abstracts away and quantifies the uncertainty induced by hardware limitations. There are a variety of options for these, since they're popular in scientific computing, but the overall point is that using such techniques, you can get consistent results within the stated accuracy of the library.
If you are depending on serious precision, floating point was not the way to go in the first place. Floating point implementations are not guaranteed to be exactly the same, nor exactly correct.
If you're worried about your program generating different results on different arch, you have some serious coding issues.
The math should be the same on all systems. If you're worried, try 2 different systems against a known or manually calculated result, that's how the Pentium-type bugs were discovered (if you remember).
Typically major issues in your processing units will be discovered quickly because of the ubiquity in the market. Unless you're using a custom built or compromised chip on eg primes, you shouldn't worry and even if it were compromised (the Chinese ARM chips or NSA-controlled crypto accelerators) you'll still get a valid result, just less secure.
Custom electronics and digital signage for your business: www.evcircuits.com
The problem of inconsistent floating point calculations between machines has been solved since 1985. I'm sure moving your app into the cloud doesn't suddenly undo 28 years of computing history.
The solution is to use a 3D printer to make your own cloud.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
So I'm supposed to do all my calculations without any Pi? How can you have any Pi if you don't eat your machine?
If you're not allowing for rounding errors, your result is invalid in the first place.
If you don't want rounding errors, use a packaged based on variable precision mathematics, like a BCD package.
I do not fail; I succeed at finding out what does not work.
Assuming there are no bugs like the Pentium fdiv bug, then there is only one way to simulate a floating point instruction correctly.
An x87 register has 80 bits and everyone simulating those with 64 bit doubles because he wants to use SSE instructions does it wrong.
I hope mathematica sets the rounding mode before performing calculations.
$ make
ERROR on line 7: Unable to reduce M_PI to a rational number; ran out of hard disk space trying to save the result.
I'm sorta kidding, but I'm also pointing out a serious flaw in this proposal.
If you really want it done right, use interval arithmetic and iterate each calculation until the error is within acceptable tolerance. This can also require insane amounts of storage space, but at least it will allow you to stop after a finite number of digits of e or pi based on what your computation requires in order to give an accurate answer.
Seeing as I get floating point math artifacts for simple arithmetic operations (e.g., balancing a household budget) in Google Doc spreadsheets...
This has pretty much been the bible for many, many, many years now: http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
If you haven't read it, you should - no matter if you're a scientific developer, a games developer, or someone who writes javascript for web apps - it's just something everyone should have read.
Forgive the Oracle-ness (it was originally written by Sun). Covers what you need to know about IEEE754 from a mathematical and programming perspective.
Long story short, determinism across multiple (strictly IEE754 compliant) architectures while possible is hard - and likely not worth it. But if you're doing scientific computing, perhaps it may be worth it to you. (Just be prepared for a messy ride of maintaining LSB error accumulators, and giving up typically 1-3 more LSB of precision for the determinism - and not only having to worry about the math of your algorithms, but the math of tracking IEEE754 floating point error for every calculation you do).
What you can do, easily, however is understand the amount of error in your calculations and state the calculation error with your findings.
floats are soft option, only gets us all in trouble.
remember
we are pentium of borg, division is futile
Responsible programmers store each value in the manner most suitable for that value. The reality is that very few applications actually care about the exact to-the-bit result of floating point ops, and floating point arithmetic should always be regarded as inexact.
Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
My first thought on seeing "tranlate" and "I'm research" was that it's only language, but then I read invalid and incorrect statements about how precision is defined in Mathematica. So now I'm not quite sure it's just language.
Archiving a whole virtual machine as opposed to the code being compiled and run is baffling to me.
Now if you are trying to archive the machine to run your old version of Mathematica and see if you get the same result, you may want to check your license agreement with Wolfram first. Second, you should be able to export the code and run the same code on new versions.
I'm really really confused on why you would want this to begin with though. Precision has increased quite a bit with the advent of 64bit hardware. I'd be more interested in taking some theoretical code and changing "double" to "uberlong" and see if I get the same results than what I solved today on today's hardware.
Unless this is some type of Government work which requires you to maintain the whole system, I simply fail to see any benefit.
Having "Cloud" does not change how precision works in Math languages.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
You ask:
Say I archive the virutal instance and in 5 or 10 years I fire it up on another cloud provider and run the same calculation. What's the likelihood that the results would be the same?
If the calculation is 2 + 2 I'd says the odds are pretty good you're going to get 4. I assume you're actually doing some difficult calculations that may push some of the edge cases in the floating point system. What I would do is make some test routines that stress the areas that you're interested in and run and check the results of those before doing any serious calculations. For the most part, you're going to have to assume that the basic functions work and there aren't simply specific combos like 17454423.2 + 99921234.1 that always gives the wrong answer since you can't check for those really but the usual concern is around the edge case handling and you should be able to define what you think is normal and make sure that your environment conforms to your definition of normal.
I currently have a Matlab script that produces slightly different FIR filter design coefficients each time I run it - when run on the same version of Matlab on the same machine. And this is with Matlab, whose primary selling point is its industrial-strength mathematical "correctness".
Also, I once used a C compiler that wouldn't produce consistent builds, and not just by a timestamp. The compiler vendor said that a random factor was used to decide between optimization choices that scored equally. We finally had to ask the vendor to remove that "feature" so we could reproduce a build, which was required as a condition for software release.
So, good luck reproducing math results in the cloud, and over many years.
Write a test suite that verifies all the behaviour you expect a system to provide to your code. Then, 10 years from now on whatever system you have to use, run those tests and make sure they pass.
Can't Mathematica be told to stick to an 80-bit precision output? If you can specify that in software, it shouldn't matter what code the underlying platform runs on.
Table-ized A.I.
Floating-point arithmetic will produce rounded results. The rounded result of a single operation will depend on the exact hardware, compiler etc. that is used. x86 compilers many years ago sometimes used extended precision instead of double precision, giving slightly different results (usually more precises). PowerPC processors and nowadays Haswell processors have fused multiply-add, which can give slightly different results (usually more precise). So the same code with the same inputs could give slightly different results.
The IEEE floating-point standard requires double precision with a 53 bit mantissa. They might have required a 54 or 52 bit mantissa, which would give slightly different rounding errors.
Now my point: If your code performing all these operations produces almost the same results on different implementations, then it is quite likely that your code is right. If you get vastly different results, then your code is likely wrong or the problem is very hard.
Some developers think that getting identical answers means that the answers are good. That's not true at all. If you have small differences due to slightly different rounding then there is a good chance that your results are good. Identical results guarentee nothing.
If you haven't already you may want to have a look at Interval arithmetic since it addresses some associated issues. It is supported in various development environments and libraries.
much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
I still have trouble with 1+1=10
If the math has been calculated with IEEE 754-2008, it is IEEE 754-2008 (aka ISO/IEC/IEEE 60559:2011). Should not matter what you are running it on...
To obtain consistency of results, you'll have to not rely on the hardware numerics at all. Use a software implementation of floating point (MPFR is a good one but there are others) that is included in your application. It is unlikely that the basic integer operations will change underneath you (but it is still possible). If you can't accept the hit for a full software implementation, rely on IEEE 754 arithmetic and code your algorithms very very carefully. Numerical analysis is difficult and full of traps for the naïve or unwary. Again, use a good library where possible.
Well written numeric code will be tolerant of slight deviations in results. e.g. the x86 uses 80 bit intermediaries while most other platforms use 64 and this causes subtle differences and a well designed algorithm will tolerate either, a poorly designed one will fail on one or both.
It is not like 1/3.
You need to go back to math class.
I could see one thing happening over time. Right now a lot of software does calculations involving decimal fractions in floating point. The problem with this is that in general you cannot precisely represent a decimal fraction using a binary floating point number. This is why you often see results like a-b = 0.19999999999999.
Well I think it is possible that we could see development of hardware arithmetic units that would internally use arbitrary precision fixed point calculations to do these sorts of calculations to eliminate these sorts of errors. So when you run your current programs on these processors the improved representation of decimal fractions would lead to slightly different results.
Actual mileage varies greatly. On x86 hardware C long double is 12 bytes, but on other hardware it's 16. If you're running an iterative process you are almost certainly not going to converge to the same result. Arbitrary precision arithmetic will mitigate the problem, but there are lots of opportunities to get in trouble. This is why regression tests are so important for mathematical codes.
You really need an analytic case you can check against. That can be really hard to come up with in many circumstances. It is hard to write code that will produce the same results with different revisions of a compiler on a single processor. It gets a lot harder when the processor changes. Numerical analysts worry a lot about such things. There have been papers which showed wildly divergent results for well written code on different processors. Compiler optimization can do things that have very unintuitive results.
You would do well to remember a quotation attributed to Richard W. Hamming: "The purpose of computing is insight, not numbers."
Qu'on me donne six lignes écrites de la main du plus honnête homme, j'y trouverai de quoi le faire pendre.
This issue was described far better than I can in William Kahan's essay, How Java's floating point hurts everyone everywhere
I remember a quote, attributed (likely incorrectly) to Seymour Cray: "Do you want it fast, or do you want it accurate?"
If you want absolutely exact arithmetic, code it entirely with arbitrary precision exact integer arithmetic. All rational real numbers can be expressed in terms of integers, and you can directly control the precision of approximation for irrational real numbers. Indeed, if your rational numbers get unwieldy, you can even control how they are approximated. And complex numbers, of course, are just pairs of real numbers in practice. (Especially if you stick to rectangular representations.) If you stick to exact, arbitrary precision integer arithmetic and representations derived from that arithmetic that you control, then you can build a bit-exact, reproducible mathematics environment. This is because integer arithmetic is exact, and you have full control of the representation built on top of that. Such an environment is very expensive, and not necessarily helpful. You can even relax the order of operations, if you can defer losses of precision. (For example, you can add a series of values in any order in integer arithmetic as long as you defer any truncation of the representation until after the summation.)
If you venture into floating point, IEEE-754 gives you a lot of guarantees. But, you need to specify the precision of each operation, the exact order of operations, and the rounding modes applied to each operation. And you need to check the compliance of the implementation, such as whether subnormals flush to zero (a subtle and easy to overlook non-conformance). Floating point arithmetic rounds at every step, due to its exponent + mantissa representation. So, order of operations matters. Vectorization and algebraic simplification both change the results of floating point computations. (Vectorization is less likely to if you can prove that all the computations are independent. Algebraic simplification, however, can really change the results of a series of adds and subtracts. It's less likely to largely affect a series of multiplies, although it can affect that too.)
And behind curtain number three is interval arithmetic. That one is especially interesting, because it keeps track at every step what the range of outcomes might be, based on the intervals associated with the inputs. For most calculations, this will just result in relatively accurate error bars. For calculations with sensitive dependence on initial conditions (ie. so-called "chaotic" computations), you stand a chance of discovering fairly early in the computation that the results are unstable.
Program Intellivision!
This assumption by the OP:
Mathematica generates the result based on the combination of software version, operating system, hypervisor, firmware and hardware that are running at that time.
... is entirely wrong. One of the defining features of Mathematica is symbolic expression rewriting and arbitrary-precision computation to avoid all of those specific issues. For example, the expression:
N[Sin[1], 50]
Will always evaluate to exactly:
0.84147098480789650665250232163029899962256306079837
And, as expected, evaluating to 51 digits yields:
0.841470984807896506652502321630298999622563060798371
Notice how the last digit in the first case remains unchanged, as expected.
This is explained at length in the documentation, and also in numerous Wolfram blog articles that go on about the details of the algorithms used to achieve this on a range of processors and operating systems. The (rare) exceptions are marked as such in the help and usually have (slower) arbitrary-precision or symbolic variants. For research purposes, Mathematica comes with an entire bag of tools that can be used to implement numerical algorithms to any precision reliably.
Conclusion: The author of the post didn't even bother to flip through the manual, despite having strict requirements spanning decades. He does however have the spare time to post on Slashdot and waste everybody else's time.
In x86 based processors we've had BCD (binary coded decimal) instructions for ages. I use those in my assembly project, or emulate unlimited bit length floating points with integer math in my big-num libs. However, modern languages do not rely on the hardware features like BCD.
In Matlab you should used fixedpoint math. That's pretty dumb, but it garauntees the precision will be the same on whatever platform.
Lacking a bignum lib with garaunteed behaviors, one could just use Java. Java emulates floating point values. That's why I don't use it: I NEED FPU speed. Java makes garauntees about its floating point operation behaviors -- which can varry by processor. The processor may have a 64 bit float type, but use 80 (or more) bits of internal representation, and only clip it to 64 bits on mem-write. You should treat hardware FPU calculations as imprecice -- This is why my physics engine has an epsilon (error bar) for equalities and such -- Without an error tolerance desynch on multiple clients is prevalent and minor rounding errors can lead to physics explosions when small values are divided beyond the precision of the machine. However, with Java your floating point behaviors are garaunteed. If you can't use fixed point or your application doesn't have support for binary coded decimal or equivalent bigint facilites with garauntees about precision, then USE JAVA DAMNIT. It's (mostly) cross platform -- That's its selling point: Write once, debug everywhere, but at least your slow as death floats will produce the same values.
> Floating point and integer operations are well defined. Unless someone fucks up
> with implementing the floating point unit the result should be exactly the same.
Not true in the real world. See http://slashdot.org/story/13/07/28/137209/same-programs--different-computers--different-weather-forecasts There was a scientic paper about the same weather model producing different forecast outputs on different machines.
I'm not repeating myself
I'm an X window user; I'm an ex-Windows user
Kahan, of course, is the authority on this.
Handling of floating point overflow is a big problem. Under Windows on x86, you can get exact (as in at the right instruction location) floating point exceptions, and I've used that to catch overflow in a physics engine. But on some CPUs, there's a speed penalty for enabling exact FPU exceptions. Java and Go don't support floating point exceptions; they return NaN or +INF or -INF or 0 (for underflow). One problem with IEEE floating point is that you don't have trichotomy. When you compare with a NAN, the result is always supposed to be false. So a != b and !(a == b) are not equivalent.
Doing a numerical compare against a NaN should raise an exception. That way, you can crunch your matrices at full speed, any operation with a NaN as an input has a NaN as an output, and if there's a NaN in the final results, code that uses it without checking for it faults out. But when IEEE floating point was designed, FPUs were separate chips. (In some cases, separate boards.) So the floating point design group didn't have the mandate to affect what the branching part of the CPU did.
As a result, you can generate a NaN, miscompare against it (all comparisons return false) and take the wrong branch in the code without recognizing the problem. Not many people care about this stuff, but where it matters, it's usually about something important.
The short answer is no. The long answer is no ... and a very long list of reasons why.
Start with reading Goldbergs classic paper "What Every Computer Scientist Should Know About Computer Arithmetic" Sun's floating point group made some improvements to the paper and paid for rights to redistribute. Oracle continues to do so. http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
If that isn't depressing enough, and you use trig functions, read http://www.scribd.com/doc/64949170/Ng-Argument-Reduction-for-Huge-Arguments-Good-to-the-Last-Bit you can get the source from netlib for "fdlibm" which is under a BSD flavor license.
If the purely software issues haven't made you realize that you haven't got much of a prayer, please note that different revs of the same intel chips sometimes provide slightly different results (sometimes intentionally, sometimes as a result of tweaking the order of execution in the out of order execution engine). Older x87 arithmetic was 80-bit, newer x64 arithmetic is pure 64-bit, providing no end of fun. Using the SSE instructions provides more variation.
If the pretty much (in principle) "simple" and potentially deterministic software issues aren't enough consider the reality of hw. Chessin has a very good, yet amusing, explanation of the key problems http://queue.acm.org/detail.cfm?id=1839574
Lest you think they only apply to a particular generation of boutique processor, most HPC ensembles are now built out of standard server motherboards and chips.
http://www.csm.ornl.gov/srt/conferences/ResilienceSummit/2010/pdf/michalak.pdf The issue of undetected soft errors is big and growing, as can be seen from the activity in the literature. SC13 "ACR: Automatic Checkpoint/Restart for Soft and Hard Error Protection" (which has lots of good citations of earlier work, including field data such as 27 soft errors per week leading to fatal node failures (that is, wrong enough results that while the hw didn't detect any problem, the issue caused the node to crash) on just one ensemble (ASC Q). its going mainstream in that HPCwire caught wind and in 31 Oct 2013 had a nice tabloidesqe writeup entitled "Addressing the Threat of Silent Data Corruption"
Neutron's don't only disrupt memory elements, but can hit logic as well. See the upcoming issue (already available via IEEE xplorer for member/subscribers) JOURNAL OF SOLID-STATE CIRCUITS, VOL. 49, NO. 1, JANUARY 2014 The 10th Generation 16-Core SPARC64 Processor for Mission Critical UNIX Server" which details the lengths some (but not many) go to ensure that there are no undetected errors (wide range of techniques, ranging from where wires are placed on the chip, ECC, parity, residue arithmetic, automatic retry, etc.). No doubt there are some good (similar) papers in the IBM Technical Journal.
No doubt a good literature search would turn up dozens of other papers, and circuit design textbooks cover some of the territory.
In principle, interval arithmetic could provide a solution (you might not get the same interval, but if the intervals nest, you have consistent results and if they are disjoint you have a bug ... and if they nest, the narrower one is "sharper" which is better). In practice, most algorithms haven't been reworked for good interval implementation, languages don't provide very good support, nor does most hardware. All fixable in principle, but unlikely to be the solution you seek for todays off the shelf virtual systems available cheaply.
I have seen some of the answers given by other people, and many seem to miss the point of floating point calculations. Floating point is by its very nature imprecise, and when you choose to use it, you have to keep that in mind - the task you want to perform must be one where a certain degree of imprecision does not matter. What you are after is not exact reproducibility, but simply that your results stay within accepted error margins, and depending on the nature of your calculations, these may be very wide - I believe you can still find astronimical measurements where ther error margin is something like +/- 200%.
However, it is a misconception to equate "maths" with "doing numbers", as only a fairly minor part of mathematics have to with numbers; and there are, in fact, computer tools out there for non-numerical calculations, like GAP (http://www.gap-system.org/). And although I haven't seen Mathematica for many years, I believe one of its main features is the ability to solve equations symbolically - ie without numerical caulculations - the result of which is going to be either correct and therefore precise, or incorrect.
The strictfp keyword in Java guarantees identical floating point calculations. Conveniently, many or most cloud services support Java. There may be equivalent ways in other languages.
One, hopefully helpful comment to the original question:
IEEE-754 is well-defined and reproducable, assuming its' user understands what he's doing, and understands to enforce IEEE-754 compliance on compilation.
There's another problem, though: IEEE-754 defines essentially only addition, subtraction, multiplication, division and square root as operations with precisely defined rounding. If your routine uses any other routine (say, from math.h), all bets are off. Multiple implementations of math libraries exist, and the results may even change when system libraries are updated. This causes functions as simple as sin() to possibly produce considerably different results on the least significant bits of a result, and of course these differences can amplify over the course of computation.
In short: know what you're doing, and don't make assumptions. Maybe only reasonable assumption to make is that a same process produces same result for same computation with same inputs... unless you're using specific versions of Java runtime, where JIT used different code for some transcendental functions inside JITted code and outside it. That was real pain in the ass.
But, in the end, "the cloud" is likely to be the smallest contributor to the problems you might encounter. Everything above virtualized hardware is really to blame, and should actually be responsible to know what's being done. You as user, are on top of that stack.
1. You can tweak your algorithms so that they minimize the error instead of accumulating it -- which you should be doing regardless of your need for reproducibility --, or
2. You can use alternative methods like software implementations of floating point, "decimal" (look at the System.Decimal type in .NET for an example), or even arbitrary-precision libraries.
How Reproducible Is Arithmetic In the Cloud?
As reproducible as you configure it to be. Fundamentally no different from running Mathematica (or a similar package) on a Beowulf cluster or in any assortment of machinery.
"I'm research the long-term consistency and reproducibility of math results in the cloud and have questions about floating point calculations. For example, say I create a virtual OS instance on a cloud provider (doesn't matter which one) and install Mathematica to run a precise calculation. Mathematica generates the result based on the combination of software version, operating system, hypervisor, firmware and hardware that are running at that time
And configuration, and choice of numeric data types, and choices of operators (.ie. division vs multiplication).
In the cloud, hardware, firmware and hypervisors are invisible to the users but could still impact the implementation/operation of floating point math. Say I archive the virutal instance and in 5 or 10 years I fire it up on another cloud provider and run the same calculation. What's the likelihood that the results would be the same? What can be done to adjust for this? Currently, I know people who 'archive' hardware just for the purpose of ensuring reproducibility and I'm wondering how this tranlates to the world of cloud and virtualization across multiple hardware types.
I doubt anyone is making such type of research. And the only way to ensure replicability of results is by strictly using fixed-precision numeric data types (instead of relying on floating point types.)
"Large" is a relative term. Original estimate for healthcare.gov was 5 billion. They went with the cheapest bidder for 1 billion.
Some of the Mersenne Prime hunters (who, incidentally, abuse the FPU to get arbitrary precision fixed point calculations on huge numbers, like millions of digits) once posted a reward, which has gone unclaimed for many years, for anyone who could show a practical consequence of the 80387 math bug which involved approximately the fifth significant digit.
The point is that nothing is built that precisely, and only a few standards measurements even care about that degree of precision....for example, the Ohm is only known to a few parts per million. This sensitive dependence on initial conditions is why weather forecasts, such as hurrican path forecasts, are based on ensemble averages.
OP, go, RTFM...and if you require bit-for-bit reproduction, then you need a bit-accurate system to run on and a way to double check if more than about 10^14 operations are required. There will be a cost to this reproducability, compared to running on the native floating-point units on the different machines used....whether you simulate the original machine instructions or use a language like Mathematica (or bignum, or, or, or...) that exactly specifies the arithmetic in arbitrary precision.
Otherwise, some statement needs to be made about the sensitivity of the calculation to its inputs and numerical errors, and those consequences followed through on. If the claim is that results are close if inputs are close, (that is, it converges in spite of small errors), then these small errors aren't important. Chaos (Lorenz style, and weather style), however, states that the small errors grow until eventually the bounds fill the entire space of possible outputs, and you will need to take the ensemble approach to get a distribuition.
If your results depend on hardware, software and so on, what you are doing is sampling from the solution space. You can then model that distribution and perform significance testing vs that distribution. What is the probability of your result being correct? your result belonging to the true distribution?
Statistics over mathematical proofs. That's what you want to do.