Slashdot Mirror
Why Improbable Things Really Aren't
First time accepted submitter sixoh1 writes "Scientific American has an excellent summary of a new book 'The Improbabilty Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day' by David J. Hand. The summary offers a quick way to relate statistical math (something that's really hard to intuit) to our daily experiences with unlikely events. The simple equations here make it easier to understand that improbable things really are not so improbable, which Hand call the 'Improbability Principle:' 'How can a huge number of opportunities occur without people realizing they are there? The law of combinations, a related strand of the Improbability Principle, points the way. It says: the number of combinations of interacting elements increases exponentially with the number of elements. The 'birthday problem' is a well-known example. Now if only we could harness this to make an infinite improbability drive!"
Mr. Hand
How improbable is the Heart of Gold?
And Zaphod stealing it...
The day before Fukashima happened I was writing a paper for an Industrial Safety class on the subject of Nuclear safety. My conclusion essentially made the argument that "Although individual improbable events are unlikely, the shear number of opportunities to experience an improbable event on a day to day basis are staggering." Any specific improbable event is highly unlikely to occur, but the occurrence of improbable events in general is a practical certainty.
The next day I saw on the news that mother nature had done her best to prove my point. The timing worked out to be an incredibly unlikely coincidence, but on a daily basis I rarely notice when unlikely coincidences fail to occur. :)
My theory of the question for life, the universe, and everything.
The books rely heavily on probability (even as far as powering the faster than light engine as alluded in the summary).
A pair of dice is one of, of not the most common symbol for probability, chance, and luck (at least in Anglo-American culture). And how many pips are on a pair of dice?
I'm out of my mind right now, but feel free to leave a message.....
I found it highly improbable that an article on that topic could be boring. It explained to me in laborious detail why I was wrong.
Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
Did someone else notice that if the chances for something to happen are exactly a million to one, there is a 1 to ten chance that it actually happens?
bickerdyke
Intuit has been in use as a verb for more than two hundred years. Your personal shortcomings do not and should not dictate what is acceptable in a Slashdot summary unless you happen to be the one writing said summary.
Why? Because there are 7 billion people on Earth.
There are well defined techniques for measuring the probability of events happening in industrial safety. Safety Integrity Levels or SIL are used to categorize the possibility of a life threatening event occurring.
The problem is how low a risk do you need and how much will it cost you to get there. Fukashima would probably not have happened if the sea wall had been higher, but the designers had to make the judgement that it was not worth the millions of cost required to build a bigger wall compared to risk of it being breached. Unfortunately decisions like that in hindsight always look flawed.,
Choose your allies carefully, it is highly unlikely you will be held accountable for the actions of your enemies
At any given time there are floods, fires, murders and any possible crime happening somewhere in the world. People, however are designed to react to what's happening in their community -- in their immediate environment.
Having every horror happening, nationwide, shoved down your throat 24X7 is equivalent to poisoning yourself.
Very interesting article on it http://www.lotterypost.com/new... been a long time since I've read it (bookmark), but this guy can tell which scratch tickets will pay off by by reading their serial numbers, winning wasn't as improbable as one is led to believe - and yes, of course he's a statistician.
I don't play the lottery, maybe a ticket twice a year, but my son likes the scratch tickets, I told him that they were predictable, he refused to listen; he wouldn't even pick up the link I printed out. He refused to imagine that it wasn't anything but random. It was just an odd reaction, I can't begin to explain the reasoning behind it.
The link is old so I imagine the serial number gig has been fixed (yet I have no clue one way or the other), but supports the improbability disclaimer.
Nothing is improbable until WE say it's improbable!! Was it improbable when the Germans bombed Pearl Harbor?
Wars are nature's way of teaching people geography. Until a bunch of states in North America federated.
Roll a dice. Each of the outcomes only has a probability of 16.p6 % (assuming a fair d6), which is fairly unlikely. Yet, there's a 100 % probability that you will obtain one of these unlikely results.
Claus
They didn't need higher sea wall. A wall can permanently hold water away and that is overdoing. What they needed to do was to make a watertight, anchored to the ground building for auxiliary generators and connect them to main building by undersea power cables. Basically, build a submarine on land, complete with snorkel. When there is a water surge, it holds generators safe and dry so that they function after the water recedes.
http://xkcd.com/1331/
Law of truly large numbers is the applicable law here, but the mistake is understandable.
one thing to remember about the birthday problem is that in a given classroom or other populated gathering, it's very likely that two people will have the same birthday... BUT... it says nothing about the possibility of any two people having any PARTICULAR birthday. so as long as you don't care what the date is, yes, two people will more than likely have one in common. but the odds that anyone will have a particular one or one that is the same as yours - those are still pretty big odds against.
>>...but what would I know, I'm just a lunatic conspiracy theorist.
Yes, indeed you are.
Law of truly large numbers is the applicable law here, but the mistake is understandable.
In fact fairly probable
Neoteric verbiage doth incrassate.
Just a bit of a nitpick. Mother nature did not "prove your point". Statistics infer data for a population from a sample. A single event from that sample does not prove or disprove anything about the population, nor the sample. Had there not been an event at Fukushima that day, your statement would not have been any more true or false, or any less proven. Your point is proven with statistical significance tests on the sample, not by taking one event and saying "here's proof". That's the opposite of statistics.
I understand what you're saying but I think much more care and precision is needed when articulating issues of probability. The bar in most discussions is set so hopelessly low that the general population - the people who least understand statistics and are most in need of some help - end up with insane theories as to how and why things occur.
It makes any rational discussion about risk impossible. I'm sure we've all heard some anecdote along the lines of "They say smoking causes cancer, but I've got an Uncle who smoked his whole life and lived to 102! Those stupid scientists don't know anything!"
People who are in a position to help with understanding these concepts do not clearly articulate the correct ideas, whether unintentionally (as in this post) or maliciously (politicians). We as a society need to become better at this.
Often when the probability of an event gets close to 1-in-100 people just say "impossible", i.e. they round down to zero.
They also forget that one can increase the chances of the event happening by repeating the trial. E.g. funding a 1-in-100 chances of blow-out-success company sounds like a risky bet, but if you fund 100 such companies, it is a rather safe bet. Hence VCs.
This is a counter-intuitive situation in which increasing the occurrences of the risky behaviour makes the whole situation safer. (Contrast this with Russian roulette in which increased trials is definitely a bad thing).
generate a small amounts of finite improbability .... to break the ice at parties by making all the molecules in the hostess's undergarments leap simultaneously one foot to the left, in accordance with the Theory of Indeterminacy.
Firstly, Pratchett's comment has nothing to do with a paradox something of the sort. It's a simple claim that scientists are bad at estimating very small probabilities, and typically get them wrong by a factor of hundreds of thousands. This is actually true and rather insightful in a the-emperor-has-no-clothes kind of way, and also not very deep at all.
The concept of the long tail is somewhat more interesting, but not that deep either. It's merely about realizing that many processes aren't Gaussian, unlike what students are lead to believe in highschool and various introductory courses which are not primarily about statistics.
However, your distinction between likely and unlikely events is confused. If you are going to label two events as likely and unlikely, then you are asserting that the likely event is to be observed with higher probability than the unlikely event. This is always true by definition.
What you are trying to say is that, if you restrict yourself to a particular family of events and you compare the probability of occurrence of an unspecified member of the family with the probability of occurrence of a single specified member, then the former can be larger.
As an example, consider the family of events {the hour of your death is N}. It is fairly unlikely that I can predict the hour of your death (not being a serial killer myself), so if I specify the event {the hour of your death is 12am} then the probability of occurrence is small. But if I do not specify the event, by saying {the hour of your death is N, where N is some hour in the day}, then that event is certain. Of course I haven't said anything interesting *with certainty*, whereas in the case of 12am I have said something interesting *with low probability*.
The tragedy of statistics is that the great majority of things we know with high probability aren't interesting, and the majority of things that are interesting have low probabilities or cannot be estimated accurately.
Everybody knows that that vital million to one chance happens nine times out of ten.
"Feynman discussed this ages ago. And I'm sure he did it better."
That's highly improbable.
Another example is in the curious case of Professor Meadows - a great paediatrician but a shite mathematician.
He endorsed the dictum that “one sudden infant death is a tragedy, two is suspicious and three is murder, until proved otherwise“. The trouble is, given enough numbers, multiple cot deaths are an inevitability.
Unfortunately, his expert testimony convicted an innocent woman. Fortunately, she was released on appeal when the math was reviewed.
--- "We've always been at war with Eastasia."
We all know that fusion happens in the core of the Sun because it is so hot and has such high pressure. Actually it's not hot enough, and the pressure is not high enough to initiate fusion. We get fusion anyway due to quantum tunneling. That is, particles can escape a potential well if there is a finite distance to another place of low potential. Imagine a marble rolling around in a bowl, but not energetically enough to pop over the rim. Quantum tunnelling provides that from time to time, the marble will spontaneously appear outside the bowl. It is for that reason that I never go near not a gun but a bullet. Quantum tunnelling: Fulminate of Mercury can spontaneously detonate.
Please mail me URLs of software employers.
BUT, was his research for an IMPROBABILITY DRIVE?
Apparently the only real danger is from falling whales and flower pots.
*Repent!Quit Your Job!Slack Off!The World Ends Tomorrow and You May Die!
Funny how people think that because we have sorted out a few complex puzzles of our reality, this should give us license to the rest. We are surrounded by things we won't ever be able to explain and the funniest part is that we can't even see most of this going on right in front of us. - The worst aspect of intelligence is being trapped by it
applied to debunking so-called "Intelligent Design". There are a few high profile proponents who claim that the probability of an organism as complex as humans evolving from single celled ancestors is so small as to be impossible, therefore we must have been "designed" by "someone" (a variation on the God of the gaps principle used by others for the same purpose). They like to point out eyes as organs that are so complex they could not have evolved, even though we have numerous living organisms that have organisms with photosensitive sensitive organs that aren't quite eyes, perhaps on their way to becoming eyes, many generations/mutations down the road.
In a single field of view under a microscope I can see tens of thousands of bacteria swimming around in a drop of water. Multiple that by all the drops of water in the world and you quickly realize that the number of living organisms is a HUGE number. With all that genetic replication (with errors that sometimes result) and gene swapping going on, and all the DNA floating around freely in the waters of the world, it seems inevitable that there will be enough mutations taking place to produce the variety of life we see on earth.
Fukashima was a multifactorial accident waiting to happen. Never improbable. e.g. Low seawall height, aux power location, tie in location, dense packed stations.
I still don't know what the relationship is between Quicken (the accounting software) and Quicken Loans
Pratchett is merely illustrating a narrative trick. If the storyteller really needs them to, all million to one chances will come in, because its a story. One of his characters goes on to say:
"Its a million to one chance - but it might just work"
Summary of article: Consider improbable event X. Repeat event X numerous times (10s, hundreds, thousands etc). Suddenly, event X is quite probable.
This is a point I bring up occasionally in regards to the so called "war on terror". The thing is these highly rare events, on average, don't happen. Your chances of ever encountering an attack is nearly nil. However, given long enough time spans, and large enough areas, they do happen with occasional frequency.
That is the thing, you can expect anything that could happen is going to happen occasionally given a large enough population that it could happen in and a long enough time for it to happen.
So if you set goals like preventing attacks where every single one that happens is a failure, if you are resigned that the next time one happens you will support this or that....then you have already resolved to support it, because it will happen, regardless of what you do.
Every liberty you are willing to curtail in the name of stopping the unstoppable is one you already lost.
"I opened my eyes, and everything went dark again"
Any type of working aux generator would have protected it. Although I think a better design would be to have one a few miles inland. That would provide better protection from other natural disasters, accidents, or intentional attacks.
The fact that people win big lotteries twice in a lifetime (sans any fraud) still blows my mind. If that can happen, just about anything can.
And yet it happened!
Honestly, I don't know, but this certainly isn't a new idea. I actually had arguments about this idea back in 2004, though I don't know how to look that far back in my post history. The reason I know it was in 2004, though, is because I have a couple of blog posts about it that are still live. It wasn't a new idea back then, either.
"Whoosh!"
Yep in the parallel universe where the Germans bombed Pearl Harbour, they used jets...
"The simple equations here make it easier to understand that improbable things really are not so improbable," [emphasis added]
Almost everyone who had birthday parties in school growing up knows SOME pair of kids with the same birthday. Anyone in America knows that "big lotteries" usually have at least a few winners a year. Helping people understand that such events happen isn't a big issue.
Helping them understand why they are expected to happen on the other hand....
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Summary of article: Consider improbable event X. Repeat event X numerous times (10s, hundreds, thousands etc). Suddenly, event X is quite probable.
You mean like a million monkeys typing for a million years will produce Shakespeare?
I just made that example up.
Simple: the former will tell you how much money in interest you've given to the latter. It'll even put it into a nice pretty graph for you.
Oh, a business relationship? Yeah, I'm not seeing that either.
You're special forces then? That's great! I just love your olympics!
One of the issues is the conflation of time with probability. A coin flip is half odds whether you flip it every second or every ten thousand years.
However, when you flip a coin every second for ten thousand years,you get different results. The million monkeys typing a million years to produce Shakespeare is a perfect example of how multiplying probability by time is like dividing integers by zero. Things get funky.
If the Fukushima risk analysis looks at one event per day versus one event per hour or one event per millisecond, you get different results for the 'same' amount of risk.
imo, it's why the Drake equations calculating the probability of life in the universe to be almost certain are almost certainly wrong. They calculated star formation PER YEAR for the entire age of the universe. That gives you a lot of events to sift through. A more accurate approach would be to avoid introducing time into the calculations and instead count the number of stars in the universe (10 to the 22nd power) and then realize that translates into life being unique in the entire universe if there are 22 events with a 1 out of ten chance of occurring.
I think one of the issues we are missing when teaching stats is that the bell curve (Gaussian distbtn) and the long tail (1/t curve) are both measures of populations.
;-)
Consider gun ownership in the US
There is approximately one gun per person in the nation; however _most_ people, more than half, do not own a gun. This situation is modeled like most any other unequal one (but it also matches the atomic configuration of atoms during a phase change). Most folks don't own a gun. The next largest group is those who have one gun. Fewer folks own 2, then 3, then 4. And there are a few folks who own tens or hundreds of guns (and we don't count 101, 102, 103 when performing the measurements).
The bell curve and the long tail curve are just different sides of the same coin (I hope that analogy doesn't confuse anyone
Why not? It depends on the probability that one monkey typing for one year can produce Shakespeare.
When the core is "shut-down" to prevent accidental thermal runaway (aka meltdown, or "china-syndrome") the system still contains a rather significant amount of heat for quite a while due to the secondary radioactive products, but this heat is not nearly enough to drive the normal steam turbine dynamos which generate the utility load - it takes a rather large amount of torque to generate megawatts of electric current. Until the heat is removed and the reactor core, fuel rods, and associated secondary decay radio-nucleotides reach a lower level, something needs to provide the power for the cooling pumps, and to ensure that the trapped hydrogen gas (byproduct of fission) is recycled and contained. There are various schemes to create "fail-proof" nuclear reactors, one of which happened to be the Chernobyl design (and we all know how well that one worked). It was supposedly "impossible!" for Cherynobyl to melt down because of the built-in systems, and the smart, but not smart-enough, engineers wanted to test those "fail-proof" systems...
When I first got into safety engineering I always imagined myself explaining the concept of a SIL level or the ALARP principle to a grieving widow. On an academic level I know exactly why it makes sense and that it's the best thing overall but it took a while to get rid of that feeling in my gut that it wasn't right.
Is 1563649 a prime number?
Or like 70,000K users playing the same game of pokemon and somehow making it past 2 gyms.
It's called 'probability'. Yes people don't understand it well, but inventing new terminology isn't the answer.
No whining yet on the misspelling of "Improbabilty", which should be Improbability?
We're talking about improbable things. Not inevitable things. Remember, this IS Slashdot.
Faster! Faster! Faster would be better!
What would really be funny is if two days later you got an F on the paper because the teacher didn't feel enough evidence existed to prove your point.
Sig. Sig. Sputnik
If they had put the generators behind (uphill) the main building AND put the generators in a water RESISTANT building, all would have been fine. If they had installed the hydrogen traps most of the problems (the earth shattering kaboom) would have been avoided. If they had followed their engineers advice and dumped sea water on the core, most of the bad problems would have been avoided.
All of those improbable problems would probably have been mitigated if TEPCO had competent upper management. Now, how likely is that?
Faster! Faster! Faster would be better!
But everyone knows that million-to-one shots occur nine times out of ten.
Proverbs 21:19
Yet this is the wrong calculation to consider because that probability—the probability that someone has the same birthday as you—is not what the question asked. It asked about the probability that any two people in the same room have the same birthday as each other. This includes the probability that one of the others has the same birthday as you, which is what I calculated above, but it also includes the probability that two or more of the other people share the same birthday, different from yours.
is very similar to the mental error(s) discussed by The Last Psychiatrist in his post "The Nanny State Didn't Show Up, You Hired It", and it's not a lack of mathematical skill or analysis:
It is this kind of example that trips up the "public" when judging things like Buckyballs because we don't think in large numbers and apply to one (statistics), we think in terms of ourselves and multiply by 6 billion (narcissism).
BUT, was his research for an IMPROBABILITY DRIVE? Apparently the only real danger is from falling whales and flower pots.
I find the idea of an infinite improbability drive to be very unlikely...
You have the right to remain sentient. If you give up the right to remain sentient, you will be elected to public office
I find the idea of an infinite improbability drive to be very unlikely...
But it's finitely unlikely. Perhaps there's a solution in there somewhere. Somebody get me a hot cup of tea.
I expect you'll find that Pascal discussed this ages ago, too.
Dr. Manhattan told the world in 1987, "Only what can happen does happen".
In short, you either need an infinite amount of monkeys, or an infinite amount of time to produce Hamlet.
"Not quite true."
Neither of those is quite true.
The test involved intentionally shutting down some of the safety systems. But when another power plant shut down, calling for more electricity from Chernobyl, the planned shutdown was postponed.
When the test was postponed, the Emergency Core Cooling System remained turned off (though that did not turn out to be a major factor in the eventual accident).
The whole thing is a long story, but in brief, it was a long chain of accumulated human error that caused the accident. Automatic systems were shut down. Manual corrections to the output were incorrect. Etc.
It doesn't matter how "safe" your automatic systems are, if they're turned off.
Correction: parent was basically true. GP was a bit off.
Really, Chernobyl all boiled down to human error. Not just one or two errors, but a whole string of errors, while simultaneously some of the automatic safety systems were turned off.
While I'm here, though, I want to mention that OP got it very wrong. The book isn't about why improbable things aren't really improbable. It's about why they ARE improbable, but happen anyway. Not the same thing.
Now if only we could harness this to make an infinite improbability drive!
From HHGTTG, quoting from here: Infinite Improbability Drive:
The principle of generating small amounts of finite improbability by simply hooking the logic circuits of a Bambleweeny 57 Sub-Meson Brain to an atomic vector plotter suspended in a strong Brownian Motion producer (say a nice hot cup of tea) were of course well understood
If ... such a [infinite improbability] machine is a virtual impossibility, it must have finite improbability. So all [one has] to do in order to make one is to work out how exactly improbable it is, feed that figure into the finite improbability generator, give it a fresh cup of really hot tea... and turn it on!
It must have been something you assimilated. . . .
Are there any well defined techniques for determining how may possibly fatal low probability events are extant? I doubt it. You may be able to say "Of the things we've considered, there are these things which each have this probability", but you can't calculate the things you haven't considered...which is most of the universe. Granted the liklihood of being stomped on by Godzilla is too low to consider, there are lots of things "metaphorically similar to Godzilla" that are more probable.
(I'm saying Godzilla rather than Zombie attack partially because this is WRT Japan, and partially because it's WRT nuclear power, and I've always considered Godzilla to be a metaphor for atomic power...though there are lots of other candidates, including Tsunami, that also fit "big, powerful, destructive...", and are thus metaphorically similar.)
You CAN'T calculate unconsidered risks. People normally dismiss them, because they aren't likely to happen within the lifespan of and within the sensory range of a homo habilis, and that's where we spent most of of recent evolutionary past. So we discount both future rewards and future costs.
I think we've pushed this "anyone can grow up to be president" thing too far.
Oh, well if you read about it then no one should never write about it again because clearly everyone know it.
The Kruger Dunning explains most post on
" magicians have calculated that million-to-one chances crop up nine times out of ten."
Terry Pratchett
The Kruger Dunning explains most post on
My point had nothing to do with the chemical formula of the primer. Lead styphnate is admittedly more stable than fulminate of mercury, but it is bound within a potential well within which it does not detonate. However every particle's wavefunction extends throughout the Universe, other than in regions whose potential is higher than the potential energy of the particle itself. While the amplitude of that wavefunction is very low in the portions of space that would lead to spontaneous detonation, it is still non-zero.
Please mail me URLs of software employers.
From human perception, there is no difference between these statements, and that's the problem addressed. The fact that something is statistically likely to "someone" (i.e.: not you) does not make something "probable" for you, which is included in the SA summary of the book.
"From human perception, there is no difference between these statements, and that's the problem addressed."
From a mathematics perspective, there is a world of difference. One is correct; the other is not.
I find the idea of an infinite improbability drive to be very unlikely...
But it's finitely unlikely. Perhaps there's a solution in there somewhere. Somebody get me a hot cup of tea.
IIRC, there was some cake in there too. Don't forget that.
McFly777
- - -
"What do people mean when they say the computer went down on them?" -Marilyn Pittman
This was probably discussed in Feynman's undergraduate textbook in "Statistics for dummies" from about 1935.
Intuitive gamblers have known that most people don't have a good understanding of the statistics of improbable events since Egyptian pyramid chisellers played dice over their lunch time beer and bread. They might not have been able to express it in mathematically rigorous form, but they understood it. And profited from it.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
... My conclusion essentially made the argument that "Although individual improbable events are unlikely, the shear number of opportunities to experience an improbable event on a day to day basis are staggering." Any specific improbable event is highly unlikely to occur, but the occurrence of improbable events in general is a practical certainty. ...
That is basically the description of "Murphy's Law". Which, contrary to some opinions, is -not- a joke.