Open Source Pioneer Michael Tiemann On the Myth of the Average

StewBeans writes: In a recent article, Michael Tiemann, one of the world's first open source entrepreneurs and VP of Open Source Affairs at Red Hat, highlights an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day." He uses this example to argue that IT leaders who think that playing it safe means being as average as possible in order to avoid risks (i.e. "Buy what others are buying. Deploy what others are deploying. Manage what others are managing.") may be making IT procurement and strategy decisions based on flawed data. Instead, Tiemann says that IT leaders should understand elements of differentiation that are most valuable, and then adopt the standards that exploit them. "Don't aim for average: it may not exist. Aim for optimal, and use the power of open source to achieve what uniquely benefits your organization."

Comments

Same thing happens in dating

[gurps_npc]

People do things like say "I only want someone with > average height, > average salary,

They get 3 people - one of which is married, one gay, and the other refuses to date someone as tall/fat/stupid/poor as them.

People just don't understand the selectivity of multiple and requirements.

Please Explain

[BradleyUffner]

" an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day"

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

Re:Please Explain

[Anonymous Coward]

FTA: "Using the size data he had gathered from 4,063 pilots, Daniels [Lt. Gilbert S. Daniels, who majored in physical anthropology at Harvard before joining the Air Force] calculated the average of the 10 physical dimensions believed to be most relevant for [optimal cockpit] design, including height, chest circumference and sleeve length. These formed the dimensions of the “average pilot,” which Daniels generously defined as someone whose measurements were within the middle 30 per cent of the range of values for each dimension. So, for example, even though the precise average height from the data was five foot nine, he defined the height of the “average pilot” as ranging from five-seven to five-11. Next, Daniels compared each individual pilot, one by one, to the average pilot.

Before he crunched his numbers, the consensus among his fellow air force researchers was that the vast majority of pilots would be within the average range on most dimensions. After all, these pilots had already been pre-selected because they appeared to be average sized. (If you were, say, six foot seven, you would never have been recruited in the first place.) The scientists also expected that a sizable number of pilots would be within the average range on all 10 dimensions. But even Daniels was stunned when he tabulated the actual number.

Zero."

Re:Please Explain

[Immerman]

One more reason at least basic probability should be taught as a high-priority subject, starting in high school at the latest. Probability is after all something that we all deal with almost constantly in informing our decisions, and by nature we're actually pretty lousy at assessing it.

If there's a 30% chance that someone will fall within the designated "average" range on any given dimension, and assuming the dimensions are roughly independent, there's only a roughly 0.30^10 = 0.0006% chance that someone will fall into that range for all dimensions. That's only one individual in 169,350. Nobody with the most basic grasp of probability would expect many people to qualify.

And these are scientists that expected otherwise? They should be ashamed of themselves.

Re:Please Explain

[KiloByte]

And here you made a basic mistake: the assumption of dimensions being independent is obviously false. Someone tall usually has long hands, and so on. Most dimensions have a rather high correlation.

Re:Please Explain

[OzPeter]

But even Daniels was stunned when he tabulated the actual number.

Zero."

Which has actually jack squat explanation as to why the pilots died.

For all the details we are given the dead pilots could have been flying bespoke aircraft with hand crafted cockpits that matched there personal ergonomics, but were flying against enemies with aircraft that were 10 times superior in flight ability, but were mass produced.

Without any further details, the idea that ergonomics killed those pilots is pure assumption.

Re:Please Explain

[Anonymous Coward]

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

... But even Daniels was stunned when he tabulated the actual number.

Zero."

"... at which point he flew into a homicidal rage, killing seventeen of the pilots before he could be subdued and sent to a mental institution."

Re:Please Explain

[Immerman]

Very true, thinking on my post I was actually hoping someone would bring that up. As you say it's obviously false that there's no correlation; however, unless you've actually collected the data to know what the actual degree of correlation is, assuming independence gives you a useful lower bound on probability to sanity-check your assumptions. You could also run the numbers for some "reasonable" guesses at correlation:

For example - lets say you only select individuals who have the first dimension within "acceptable" range - 100% of them, so we can ignore it. Furthermore we'll assume that all other dimensions have a freakishly high 80% correlation rate with the first of falling within their respective "average" range as well. You still end up with only 0.8^9 = 13% of candidates having all 10 dimensions within the average range.

So even with freakishly optimistic assumptions about correlation, you *still* end up with fewer than 1 in 8 candidates having all ten dimensions within your desired range.

Re:Please Explain

[PPH]

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

Software debugging methodology. It killed one pilot and management said to run it again and see if it does the same thing.

Re:Please Explain

[Immerman]

>So, all of common sense goes against probability.

To a very large extent, yes. Which is exactly why understanding the implications of basic probability is so important to behaving in a rational manner.

This isn't even anything complicated, venture into the wild and woolly world of false positives and the math only gets slightly more complicated, but the implications become radically less intuitive. For example testing positive for a rare disease on a test with only a 1% chance of a false positive, still means you almost certainly don't actually have the disease. (if 1 in 1000 people have the disease, then out of an average 1000 people, 10 will test positive, but only 1 will actually have it. A 90% chance that you're clean.)

Re:Please Explain

[spinozaq]

The quoted and linked article with the original article explains it a bit.... Here is a summary... When we started making jet fighters we had lots and lots of crashes, but they weren't from mechanical issues, so that seemed like pilot error, but the pilots didn't really think they were doing anything wrong. The cockpit design they were using was from 1926 and based on un-adjustable controls with positioning calculated from the "average pilot". While the upper management at the military was arguing about the costs of redesigning the planes engineerings invented adjustable seats and controls and pilots stopped crashing so much.

Re:Please Explain

[clovis]

" an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day"

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

The book, The End of Average by Todd Rose was misquoted.
First of all the exact quote from the first paragraph of the book was this:

At its worst point, seventeen pilots crashed in a single day

There is a huge difference between crashing and dying.

Anyway, he (Teimann) got the sequence of events wrong, but the general gist of what he said follows the intent of the book.
The crashing planes in the study were the in the 1940's. We're talking about planes like the P-80 and possibly the F-86. That was the first generation of jets and they had many many problems in design.
Here's where the average pilot comes in. Those planes had been designed for the average pilot's size as measured in 1926. The cockpit was non-adjustable, so The Army/Air Force sought pilots whose size fit the planes, but only that person who matched the average 1926 pilot would fit properly. In the highly demanding jets of the late 1940's, a pilot that didn't fit could have problems when split second control reactions were needed, and those planes needed it.

The study conducted by Lieutenant Gilbert Daniels in 1950 which examined modern average pilot sizes, was completed in 1952. The upshot of that study was that the Air Force immediately decided to take the study's recommendation: Everyone is different, and to get the maximum performance from people you adjust the environment to the soldier, not the soldier. The Air Force immediately mandated that the manufacturers make many elements of the cockpit be adjustable for the range of sizes from 5% to 95% of men from the seats, to pedal positions, to belts, and helmet straps, and so on. The result was that pilot performance soared and the US Air Force became the most dominant air force on the planet.

The book gives other example studies and goes on to say

Any system designed around the average person is doomed to fail

This is the gist of the book and what Michael Tiemann was getting at.

Anyway, the summary implied that the generation of planes designed in the 1950's were a generation of pilot killers.
This is wrong, the book said the opposite. The 1950's planes had the cockpit fit problems solved.

In other words, quit buying Red Hat Linux

[xxxJonBoyxxx]

>> Red Hat VP: IT leaders who think that playing it safe means being as average as possible in order to avoid risks (i.e. "Buy what others are buying. Deploy what others are deploying.")

Why isn't this article entitled "Red Hat Linux executive tells the sheeple to quit buying Red Hat Linux - there are plenty of identical and cheaper alternatives available?"

80/20

[Bengie]

Many look at a technology and say "it's good enough, it does 80% of what I need". Then they cobble together 9 other technologies the same way and you're left with 0.8^10 "enough", leaving you fighting fires from the lack of custom configuration that you need. Technical debt is multiplicative with other technical debt.

For every 1 person that reinvents the wheel, 9 others use an existing wheel for the wrong job or misconfigure the wheel because they don't understand their problem well enough. If you truly understand your current issue, you're smart enough to create a solution. Every time someone treats a tool like a black box of magic, looking at you programmers blindly using libraries without understanding how they work, it's because they don't understand the problem they're trying to solve.

P.S. Understanding what something is doing does not mean you know the exact details of the implementation.

Averages are misleading

[mark-t]

For example, the average person has approximately 1 testicle.

Re: Averages do exist

[iluvcapra]

When managers deploy "average" security solutions, they're not trying to protect against threats, they're trying to avoid getting fired.

If they deploy something unusual and it doesn't work, they'll be fired, regardless of how it failed or the merits. If they deploy something everyone else has deployed and it doesn't work, they will be commended for following "industry best practices."

Not all organizations work this way, but many do. When something breaks, there's a big temptation to avoid an investigation into exactly what happened- who knows what that could turn up! Much easier just to fire middle managers for prima facie reasons.

One KEEPS working and solving new problems. Diced

[raymorris]

> open source over proprietary
> ...
> Choose software because it works and solves a problem for you, not just because it is open source.

Proprietary software can get Diced, and there's a good chance that'll eventually happen to the one you choose.

At my last job, the first major project I was assigned to was replacing some proprietary software with open source for one of our critical systems. The proprietary system was originally chosen because it looked liked it worked (in vendor demos) and it seemed like it would solve most of the problems it needed to solve. Once it was actually used in production, they found that it mostly worked, and kinda solved a lot of problems, but created new problems. The vendor wasn't too enthusiastic about fixing things and certainly wouldn't add needed features because they were (once again) moving on to their "next generation platform". After a few years, our needs changed a bit, new requirements came up, and the old proprietary system really wasn't working well at all.

We downloaded the most recent version of an open source solution, called Moodle, and took a few minutes to set it up. Rather than watching the vendor's sales people demo it, we could actually try it out, and try loading our actual data into it. It actually worked with our real data, and could be integrated into our other systems. An idiosyncrasy or two was handled by adjusting the appropriate line of code and submitting the fix/improvement back to the FOSS project. After it went live in production, departments asked "can it do this? It would be great if it could do that.". For each feature, it it didn't already have the feature, I took a few hours to write a little plugin implementing the requested feature. A few years later, I'm even more confident FOSS was the right choice - it's basically impossible to have a major roadblock with the software because in the worst case we could just add a little plugin to have it do whatever we want.

In the very worst case, the project -could- completely change direction, and the organization using it could just keep using the version that already works well for them, applying any commits they want from the new version.

The best that can be said about any proprietary software you're thinking about adopting is that it looks like it will handle your needs, for the moment. Open source will continue to handle whatever needs come up, given that you form a relationship with either the sponsors of the project or any programmer of your choosing to handle the plugins and patches that you want to have as the need arises.

For that reason, open source is objectively better and more reliable on the "solves problems" and "it works" metrics, because it'll continue to work next year - it won't be dropped when the vendor is sold to Dice.

Of course, as you said, there is such a thing as bad proprietary software and bad open source software, and you want to avoid choosing bad software. Given to pieces of software that appear to be roughly equally good _at_the_moment_, the open source is better because the risk of insurmountable problems down the road and vendor lock in is essentially removed.

Skip this, read the article it references. Really

[Anonymous Coward]

The article misquotes an excerpt from a book here:
http://www.thestar.com/news/insight/2016/01/16/when-us-air-force-discovered-the-flaw-of-averages.html

This explains more of the story: the measurements were originally taken in 1926, but it wasn't until the 1950's that increased speeds for fighters made the design flaws apparent. The 17 deaths is an agile enterprise adaptation of 17 non-fatal crashes. Anyhow, it seems intuitive that body measurements would be correlated, so I'd say the big error was not checking that assumption. Kind of amazing bad science lasted that long.

Why pick popular

[XXongo]

From bitter experience, I'll put in a word for the value of picking software that multiple other people use rather than picking what optimally fits your needs.

Software that is popular with the most users is also the software that is least likely to be orphaned, leaving you to either keep obsolete machines running or else having to migrate some obscure data format into some different form.

Also, the most popular software is more likely to have the most annoying features "corrected" because so many users complain. (not to mention it has the most people posting work-arounds on the web for the things that don't work.)

Book misquoted; pilot crashes were in 1940's

[clovis]

The book "The End of Averages" by Todd Rose was misquoted
First of all the exact quote from the first paragraph of the book was this:

At its worst point, seventeen pilots crashed in a single day

There is a huge difference between crashing and dying.

Anyway, he (Teimann) got the sequence of events wrong, but the general gist of what he said follows the intent of the book.
The crashing planes in the study were the in the 1940's. We're talking about planes like the P-80 and possibly the F-86.
That was the first generation of jets and they had many many problems in design.

Here's where the average pilot comes in. Those planes (the 1940's) had been designed for the average pilot's size as measured in 1926. The cockpit was non-adjustable, so The Army/Air Force sought pilots whose size fit the planes, but only that person who matched the average 1926 pilot would fit properly. In the highly demanding jets of the late 1940's, a pilot that didn't fit could have problems when split second control reactions were needed, and those planes needed it.

The study conducted by Lieutenant Gilbert Daniels in 1950 which examined modern average pilot sizes, was completed in 1952.
The upshot of that study was that the Air Force immediately decided to take the study's recommendation:
Everyone is different, and to get the maximum performance from people you adjust the environment to the soldier, not the soldier.

The Air Force immediately mandated that the manufacturers make many elements of the cockpit be adjustable for the range of sizes from 5% to 95% of men from the seats, to pedal positions, to belts, and helmet straps, and so on. The result was that pilot performance soared and the US Air Force became the most dominant air force on the planet.

The book gives other example studies and goes on to say

Any system designed around the average person is doomed to fail

This is the gist of the book and what Michael Tiemann was getting at.

Anyway, the summary implied that the generation of planes designed in the 1950's were a generation of pilot killers.
The 1950's planes had the cockpit fit problems solved.
The crashing planes were in the late 1940's. The study was begun in 1950. Obviously, those crashes were not combat-related. Those planes were demanding and possibly evil, and a bad-fitting cockpit made it worse.