Open Source Pioneer Michael Tiemann On the Myth of the Average

StewBeans writes: In a recent article, Michael Tiemann, one of the world's first open source entrepreneurs and VP of Open Source Affairs at Red Hat, highlights an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day." He uses this example to argue that IT leaders who think that playing it safe means being as average as possible in order to avoid risks (i.e. "Buy what others are buying. Deploy what others are deploying. Manage what others are managing.") may be making IT procurement and strategy decisions based on flawed data. Instead, Tiemann says that IT leaders should understand elements of differentiation that are most valuable, and then adopt the standards that exploit them. "Don't aim for average: it may not exist. Aim for optimal, and use the power of open source to achieve what uniquely benefits your organization."

Comments

Averages do exist

[invictusvoyd]

they are called managers

Re: Averages do exist

[iluvcapra]

When managers deploy "average" security solutions, they're not trying to protect against threats, they're trying to avoid getting fired.

If they deploy something unusual and it doesn't work, they'll be fired, regardless of how it failed or the merits. If they deploy something everyone else has deployed and it doesn't work, they will be commended for following "industry best practices."

Not all organizations work this way, but many do. When something breaks, there's a big temptation to avoid an investigation into exactly what happened- who knows what that could turn up! Much easier just to fire middle managers for prima facie reasons.

Re:Averages do exist

[KGIII]

In the past, I've mentioned how we didn't really have much in the way of formal job titles at the company I sold. We didn't have a lead programmer, we didn't have a specific QA team, and people have often asked me how that worked or what that looked like. I suspect they ask out of morbid curiosity but... Well, no... That's really what I expect.

Anyhow, what it looked like is that people found and filled their own roles. I'm speaking pretty much of the developer side of things. We didn't have a whole lot of "users" in the traditional sense. There was a lot of cross over, including what you guys are calling "dev-ops." But, QA was done before changes were rolled out to production. Documentation was written - and done well. Research was done. A versioning system decided on (without my needing to intervene or even make suggestions). Code was written. I usually only approached a few people at once but they'd go back, relay the message, and organize and things got done.

There was no manager. In fact, the person who was doing much of the managing was one of the newest hires when I sold. Yet they didn't make more than anyone. They didn't do employee reviews nor could they have fired anyone. We didn't have HR but we had someone who did that sort of stuff - as well as liaison with the bookkeeping and payroll. They were more a secretary than HR - they couldn't hire or fire anyone, they probably didn't want to. Hell, they couldn't even write, issue, or get signed a disciplinary note for a person's files. Err... In all fairness, we never had such a thing.

People sometimes ask if I used scrum, agile, or witchcraft. No. I don't actually think the devs had a name for it. We called it, get this, "programming." They did it because I couldn't. I don't tell the mechanic how to fix my car. I don't tell my programmers how to do things that I don't know how to do or am not an expert at. If I'd been able to do it, I'd have kept doing it. I can't code for shit. Oh, I can... It's just... Well... It works. By some definition, we could say it worked well. And, well, it worked well enough for me to be able to hire professionals.

What system did they use? Buggered if I know. I told them what I needed, they gave me a time and date, and we revisited at that date and time. They then gave me an ETA. Sometimes, one of 'em might ask me to be a bit more clear. If they needed something, they let me know. They're programmers - trust me, they let me know. If it was going to be late, they let me know. Someone did a framework, someone maintained, someone did QA, someone did documentation, someone interacted with me, someone interacted with ops, someone kept it organized, and someone kept it disorganized. There might be multiple people wearing those hats, some wearing no hats at all, and some that probably didn't even want to wear any hat but the one that they liked best.

I'm not so sure that "trust and get out of the way so the experts you hired can actually do what you paid them to do without you slowing them down or adding needless complexity and largely letting them organize themselves and find their own role" has a three letter acronym so I never really know how to answer that question. "Buggered if I know" is not a very good system they tell me. Oddly enough, I'm still (fairly frequently) told that I did it all wrong. Oddly enough, I'm told that'd never work - to which I usually ask, "Well, have you tried it?"

It's amazing how well people do when left to their own devices with nothing but a goal and the tools they want to work with to meet the stated goals. I didn't even have to give them labels or separate pay grades. I don't know what happened between now and then but I suspect it would have been an abject failure if I'd tried to corral them. I didn't even have to tell them when to take breaks or go on lunch! Yet, they always met my goals - even when I pushed for the damned near impossible. Without them, I'd still have to work every day.

Re:Averages do exist

[KGIII]

Oh, ha! I forgot to add... That's the power of the "average" person - I think. They can do pretty well for themselves if you get out of the way and let them do it. They do even better if you ensure that they've got the tools they want. (Lesson learned: That's NOT always the tools suggested by a vendor. When they say they want a certain compiler - get it for them. Err... We had proprietary compilers back in those days.)

At any rate, the "average" seems to be able to do okay if given the ability to do so. The average can even have some impressive results in the right environment. Ego says that I should claim we were above average. I don't think that's true. While we were "industry leading" that's largely due to a combination of timing and having the willingness to work together and get shit done. No, we had some great people but I'd like to think we were mostly just average people who were willing to do good work.

Meh, I learned a lot from 'em. Mostly, I learned to get out of the way and take a lesson. They found what they were good at, on average, and did that job, on average. The guy writing the documentation today may not be (but probably is - unless there's a crunch of some type) someone who actually was hired to do documentation. Just like I'd been known to take out the trash or even clean up after a mess. I can assure you, I was not "hired" to clean up. But, it needed doing and I was there. I'd like to think we were largely average people. Otherwise, I fall into the trap of thinking I'm special and am where I am because of it and I know that's intellectually dishonest even if it is an appeal to the ego.

Same thing happens in dating

[gurps_npc]

People do things like say "I only want someone with > average height, > average salary,

They get 3 people - one of which is married, one gay, and the other refuses to date someone as tall/fat/stupid/poor as them.

People just don't understand the selectivity of multiple and requirements.

Re:Same thing happens in dating

[Archangel Michael]

People do things like say "I only want someone with > average height, > average salary,

Clueless people often say things like that. What they mean by that is "I have no idea what I want", which is exactly what they will get.

Re:Same thing happens in dating

[gurps_npc]

While I agree with your statement I have to respond with this:

If you are looking for someone with no 'undealt' personal issues, may I suggest you check Narnia, Atlantis, and Krypton - because you aren't living in the real world.

Please Explain

[BradleyUffner]

" an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day"

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

Re:Please Explain

[Anonymous Coward]

FTA: "Using the size data he had gathered from 4,063 pilots, Daniels [Lt. Gilbert S. Daniels, who majored in physical anthropology at Harvard before joining the Air Force] calculated the average of the 10 physical dimensions believed to be most relevant for [optimal cockpit] design, including height, chest circumference and sleeve length. These formed the dimensions of the “average pilot,” which Daniels generously defined as someone whose measurements were within the middle 30 per cent of the range of values for each dimension. So, for example, even though the precise average height from the data was five foot nine, he defined the height of the “average pilot” as ranging from five-seven to five-11. Next, Daniels compared each individual pilot, one by one, to the average pilot.

Before he crunched his numbers, the consensus among his fellow air force researchers was that the vast majority of pilots would be within the average range on most dimensions. After all, these pilots had already been pre-selected because they appeared to be average sized. (If you were, say, six foot seven, you would never have been recruited in the first place.) The scientists also expected that a sizable number of pilots would be within the average range on all 10 dimensions. But even Daniels was stunned when he tabulated the actual number.

Zero."

Re:Please Explain

[Immerman]

One more reason at least basic probability should be taught as a high-priority subject, starting in high school at the latest. Probability is after all something that we all deal with almost constantly in informing our decisions, and by nature we're actually pretty lousy at assessing it.

If there's a 30% chance that someone will fall within the designated "average" range on any given dimension, and assuming the dimensions are roughly independent, there's only a roughly 0.30^10 = 0.0006% chance that someone will fall into that range for all dimensions. That's only one individual in 169,350. Nobody with the most basic grasp of probability would expect many people to qualify.

And these are scientists that expected otherwise? They should be ashamed of themselves.

Re:Please Explain

[KiloByte]

And here you made a basic mistake: the assumption of dimensions being independent is obviously false. Someone tall usually has long hands, and so on. Most dimensions have a rather high correlation.

Re:Please Explain

[OzPeter]

But even Daniels was stunned when he tabulated the actual number.

Zero."

Which has actually jack squat explanation as to why the pilots died.

For all the details we are given the dead pilots could have been flying bespoke aircraft with hand crafted cockpits that matched there personal ergonomics, but were flying against enemies with aircraft that were 10 times superior in flight ability, but were mass produced.

Without any further details, the idea that ergonomics killed those pilots is pure assumption.

Re:Please Explain

[Daetrin]

That explains neither how the design killed so many people nor what the better alternative would be.

From the description of the problem the only issue i can imagine is them is making the design so exact, and not building in any adjustability at all, that pilots with shorter than average arms would have trouble reaching the controls and pilots with shorter than average legs would have trouble reaching the pedals and etc. However i find it hard to believe that anyone, even the military, even when aggressively aiming for an "optimum average", would be so dumb as to build in absolutely no methods for adjusting the seat/controls/etc. And if they were actually that dumb it ought to be stated explicitly, because it's not a conclusion that a lot of people would jump straight to.

Re:Please Explain

[Anonymous Coward]

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

... But even Daniels was stunned when he tabulated the actual number.

Zero."

"... at which point he flew into a homicidal rage, killing seventeen of the pilots before he could be subdued and sent to a mental institution."

Re:Please Explain

[Immerman]

Very true, thinking on my post I was actually hoping someone would bring that up. As you say it's obviously false that there's no correlation; however, unless you've actually collected the data to know what the actual degree of correlation is, assuming independence gives you a useful lower bound on probability to sanity-check your assumptions. You could also run the numbers for some "reasonable" guesses at correlation:

For example - lets say you only select individuals who have the first dimension within "acceptable" range - 100% of them, so we can ignore it. Furthermore we'll assume that all other dimensions have a freakishly high 80% correlation rate with the first of falling within their respective "average" range as well. You still end up with only 0.8^9 = 13% of candidates having all 10 dimensions within the average range.

So even with freakishly optimistic assumptions about correlation, you *still* end up with fewer than 1 in 8 candidates having all ten dimensions within your desired range.

Re:Please Explain

[PPH]

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

Software debugging methodology. It killed one pilot and management said to run it again and see if it does the same thing.

Re:Please Explain

[Immerman]

>So, all of common sense goes against probability.

To a very large extent, yes. Which is exactly why understanding the implications of basic probability is so important to behaving in a rational manner.

This isn't even anything complicated, venture into the wild and woolly world of false positives and the math only gets slightly more complicated, but the implications become radically less intuitive. For example testing positive for a rare disease on a test with only a 1% chance of a false positive, still means you almost certainly don't actually have the disease. (if 1 in 1000 people have the disease, then out of an average 1000 people, 10 will test positive, but only 1 will actually have it. A 90% chance that you're clean.)

Re:Please Explain

[Jawnn]

" an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day"

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

Yeah, some more detail there would have been nice, but as a pilot, I can easily see how this was certainly related to physical dimensions. Consider the non-average pilot with shorter than average arms. If he lost his grip on the stick during certain maneuvers, it's entirely possible that it could have moved to a position out of his reach. Same goes for rudder pedals for the pilot whose knees won't fit under the panel when he's thrown forward in the straps, or that won't allow them to bend enough to allow full travel of that control surface.

Re:Please Explain

[Eclectic Engineer]

No one seems to have pointed out this portion of TFA:

"During a time of great technical innovation in aviation — when fighters planes became fighter jets, and later super-sonic fighter jets — the design of the cockpit became more and more decisive in determining whether a pilot could stay ahead of the plane and successfully execute the mission, or whether the plane would get ahead of the pilot and fly out of control."

The nature and reliability of how this was determined is left unexplained, but it does at least clearly state the basis for the argument -- that cockpit dimensions played a large role in a pilot's ability simply to fly the plane.

Re:Please Explain

[spinozaq]

The quoted and linked article with the original article explains it a bit.... Here is a summary... When we started making jet fighters we had lots and lots of crashes, but they weren't from mechanical issues, so that seemed like pilot error, but the pilots didn't really think they were doing anything wrong. The cockpit design they were using was from 1926 and based on un-adjustable controls with positioning calculated from the "average pilot". While the upper management at the military was arguing about the costs of redesigning the planes engineerings invented adjustable seats and controls and pilots stopped crashing so much.

Re:Please Explain

[Coren22]

My grandfather flew B-52s in Korea. When he was up for the SR-71, it was deemed that the length of his legs was too great; he couldn't use the rudders or fit in the cockpit.

My grandfather is also quite tall, not sure exactly how tall, but his height wasn't even a concern, just the length of his legs was an issue.

It happens that different jets have different constraints on the dimensions of the pilots.

Re:Please Explain

[clovis]

" an example from the 1950s US Air Force where the "myth of the average resulted in a generation of planes that almost no pilots could reliably fly, and which killed as many as 17 pilots in a single day"

Did I miss the part of the story that explains HOW it managed to kill 17 pilots in one day?

The book, The End of Average by Todd Rose was misquoted.
First of all the exact quote from the first paragraph of the book was this:

At its worst point, seventeen pilots crashed in a single day

There is a huge difference between crashing and dying.

Anyway, he (Teimann) got the sequence of events wrong, but the general gist of what he said follows the intent of the book.
The crashing planes in the study were the in the 1940's. We're talking about planes like the P-80 and possibly the F-86. That was the first generation of jets and they had many many problems in design.
Here's where the average pilot comes in. Those planes had been designed for the average pilot's size as measured in 1926. The cockpit was non-adjustable, so The Army/Air Force sought pilots whose size fit the planes, but only that person who matched the average 1926 pilot would fit properly. In the highly demanding jets of the late 1940's, a pilot that didn't fit could have problems when split second control reactions were needed, and those planes needed it.

The study conducted by Lieutenant Gilbert Daniels in 1950 which examined modern average pilot sizes, was completed in 1952. The upshot of that study was that the Air Force immediately decided to take the study's recommendation: Everyone is different, and to get the maximum performance from people you adjust the environment to the soldier, not the soldier. The Air Force immediately mandated that the manufacturers make many elements of the cockpit be adjustable for the range of sizes from 5% to 95% of men from the seats, to pedal positions, to belts, and helmet straps, and so on. The result was that pilot performance soared and the US Air Force became the most dominant air force on the planet.

The book gives other example studies and goes on to say

Any system designed around the average person is doomed to fail

This is the gist of the book and what Michael Tiemann was getting at.

Anyway, the summary implied that the generation of planes designed in the 1950's were a generation of pilot killers.
This is wrong, the book said the opposite. The 1950's planes had the cockpit fit problems solved.

In other words, quit buying Red Hat Linux

[xxxJonBoyxxx]

>> Red Hat VP: IT leaders who think that playing it safe means being as average as possible in order to avoid risks (i.e. "Buy what others are buying. Deploy what others are deploying.")

Why isn't this article entitled "Red Hat Linux executive tells the sheeple to quit buying Red Hat Linux - there are plenty of identical and cheaper alternatives available?"

Re:In other words, quit buying Red Hat Linux

[RR]

Why isn't this article entitled "Red Hat Linux executive tells the sheeple to quit buying Red Hat Linux - there are plenty of identical and cheaper alternatives available?"

No, it’s more like: Buy Red Hat. They employ the largest number of influential Linux hackers (Poettering, Sievers, Molnar, etc.), so you can be assured that open source stuff works best in a Red Hat distribution.

80/20

[Bengie]

Many look at a technology and say "it's good enough, it does 80% of what I need". Then they cobble together 9 other technologies the same way and you're left with 0.8^10 "enough", leaving you fighting fires from the lack of custom configuration that you need. Technical debt is multiplicative with other technical debt.

For every 1 person that reinvents the wheel, 9 others use an existing wheel for the wrong job or misconfigure the wheel because they don't understand their problem well enough. If you truly understand your current issue, you're smart enough to create a solution. Every time someone treats a tool like a black box of magic, looking at you programmers blindly using libraries without understanding how they work, it's because they don't understand the problem they're trying to solve.

P.S. Understanding what something is doing does not mean you know the exact details of the implementation.

Re:80/20

[gstoddart]

Many look at a technology and say "it's good enough, it does 80% of what I need"

Having recently been a consultant at a company doing this ... that was their entire new strategy.

They were doing more with less, it was the new way forward, it was going to change the world, 80/20 was the new standard, no more of the white glove service, we'd focus on core functionality only and stop catering to the outliers.

Everyone not in management understood that to mean we're cutting budgets, and we're going to claim service levels won't suffer, while making policy which mandate service levels suffer because both of these can't be true.

What management understood that to mean wasn't clear -- either they believed they could do more with less, or it was doublespeak and they knew they were full of shit.

When average and mediocrity is the policy, WTF do you expect to get?

The people defining IT purchasing and driving strategy? They don't want anything BUT the safe middle of the road which won't get you in trouble ... and they want it to be managed in a foreign country as cheaply as possible, no matter how bad the service.

Averages are misleading

[mark-t]

For example, the average person has approximately 1 testicle.

Re:Averages are misleading

[onkelonkel]

And one ovary!

Re:Averages are misleading

[smellsofbikes]

For example, the average person has approximately 1 testicle.

Whenever people trot out the tired statistic that 70% (or whatever) of drivers believe they're better than the average driver, to mock the idea of how well people rate their own abilities, I trot out the statistic that 99.9% of humans have more than the average number of eyes. We assume gassian distributions with no idea if that's actually representative of the results.

With that said, at least the air force example is a case of misunderstanding of how averages concatenate, which is a slightly more complicated poor understanding of statistics. (It doesn't take long to calculate 0.3^10 and realize that it's an extremely small number, but that's not intuitive if you haven't taken statistics classes, or, in my case, lots of chemistry classes: when you have a complex synthesis that has seven steps, each with 80% yield, you realize you're going to need kilograms of starting material to even measure your final product.)

Re:Averages are misleading

[tommeke100]

That's because averages are prone to outliers. A better measure is the median ( or 50th percentile). The median for the US shows most people there have no balls. Same for Europe and Russia. People in China have 2 testicles though.

One KEEPS working and solving new problems. Diced

[raymorris]

> open source over proprietary
> ...
> Choose software because it works and solves a problem for you, not just because it is open source.

Proprietary software can get Diced, and there's a good chance that'll eventually happen to the one you choose.

At my last job, the first major project I was assigned to was replacing some proprietary software with open source for one of our critical systems. The proprietary system was originally chosen because it looked liked it worked (in vendor demos) and it seemed like it would solve most of the problems it needed to solve. Once it was actually used in production, they found that it mostly worked, and kinda solved a lot of problems, but created new problems. The vendor wasn't too enthusiastic about fixing things and certainly wouldn't add needed features because they were (once again) moving on to their "next generation platform". After a few years, our needs changed a bit, new requirements came up, and the old proprietary system really wasn't working well at all.

We downloaded the most recent version of an open source solution, called Moodle, and took a few minutes to set it up. Rather than watching the vendor's sales people demo it, we could actually try it out, and try loading our actual data into it. It actually worked with our real data, and could be integrated into our other systems. An idiosyncrasy or two was handled by adjusting the appropriate line of code and submitting the fix/improvement back to the FOSS project. After it went live in production, departments asked "can it do this? It would be great if it could do that.". For each feature, it it didn't already have the feature, I took a few hours to write a little plugin implementing the requested feature. A few years later, I'm even more confident FOSS was the right choice - it's basically impossible to have a major roadblock with the software because in the worst case we could just add a little plugin to have it do whatever we want.

In the very worst case, the project -could- completely change direction, and the organization using it could just keep using the version that already works well for them, applying any commits they want from the new version.

The best that can be said about any proprietary software you're thinking about adopting is that it looks like it will handle your needs, for the moment. Open source will continue to handle whatever needs come up, given that you form a relationship with either the sponsors of the project or any programmer of your choosing to handle the plugins and patches that you want to have as the need arises.

For that reason, open source is objectively better and more reliable on the "solves problems" and "it works" metrics, because it'll continue to work next year - it won't be dropped when the vendor is sold to Dice.

Of course, as you said, there is such a thing as bad proprietary software and bad open source software, and you want to avoid choosing bad software. Given to pieces of software that appear to be roughly equally good _at_the_moment_, the open source is better because the risk of insurmountable problems down the road and vendor lock in is essentially removed.

Re:One KEEPS working and solving new problems. Dic

[PPH]

Proprietary software can get Diced, and there's a good chance that'll eventually happen to the one you choose.

That isn't so! We just ported all our stuff to Windows 8 and re-trained all of our employees. It's a good system and will be around for the foreseeable future .... [Just a sec. Gotta deal with all these 'Upgrade Now!' popups].

Main problem...

[Chris Mattern]

Determining what's optimal is hard. Determining what everybody else is using is easy.

Skip this, read the article it references. Really

[Anonymous Coward]

The article misquotes an excerpt from a book here:
http://www.thestar.com/news/insight/2016/01/16/when-us-air-force-discovered-the-flaw-of-averages.html

This explains more of the story: the measurements were originally taken in 1926, but it wasn't until the 1950's that increased speeds for fighters made the design flaws apparent. The 17 deaths is an agile enterprise adaptation of 17 non-fatal crashes. Anyhow, it seems intuitive that body measurements would be correlated, so I'd say the big error was not checking that assumption. Kind of amazing bad science lasted that long.

Oh, he's missing the point

[hey!]

possibly intentionally.

The reason people are so conventional is what economists call "agency costs". They aren't minimizing risk to their employers, they're minimizing risk to themselves.

Height constraints

[XXongo]

My uncle is 6'2 and flew reconnaissance F4 Phantoms in Viet Nam. Just what is the height constraint?

http://work.chron.com/air-forc... states:
Pilots have to meet the Air Force’s height, weight and physical conditioning requirements. They must be 64 to 77 inches tall when standing, and 34 to 40 inches tall when sitting...

77 inches is 6 foot 5 inches, so looks like your uncle made it with three inches to spare.

Why pick popular

[XXongo]

From bitter experience, I'll put in a word for the value of picking software that multiple other people use rather than picking what optimally fits your needs.

Software that is popular with the most users is also the software that is least likely to be orphaned, leaving you to either keep obsolete machines running or else having to migrate some obscure data format into some different form.

Also, the most popular software is more likely to have the most annoying features "corrected" because so many users complain. (not to mention it has the most people posting work-arounds on the web for the things that don't work.)

Book misquoted; pilot crashes were in 1940's

[clovis]

The book "The End of Averages" by Todd Rose was misquoted
First of all the exact quote from the first paragraph of the book was this:

At its worst point, seventeen pilots crashed in a single day

There is a huge difference between crashing and dying.

Anyway, he (Teimann) got the sequence of events wrong, but the general gist of what he said follows the intent of the book.
The crashing planes in the study were the in the 1940's. We're talking about planes like the P-80 and possibly the F-86.
That was the first generation of jets and they had many many problems in design.

Here's where the average pilot comes in. Those planes (the 1940's) had been designed for the average pilot's size as measured in 1926. The cockpit was non-adjustable, so The Army/Air Force sought pilots whose size fit the planes, but only that person who matched the average 1926 pilot would fit properly. In the highly demanding jets of the late 1940's, a pilot that didn't fit could have problems when split second control reactions were needed, and those planes needed it.

The study conducted by Lieutenant Gilbert Daniels in 1950 which examined modern average pilot sizes, was completed in 1952.
The upshot of that study was that the Air Force immediately decided to take the study's recommendation:
Everyone is different, and to get the maximum performance from people you adjust the environment to the soldier, not the soldier.

The Air Force immediately mandated that the manufacturers make many elements of the cockpit be adjustable for the range of sizes from 5% to 95% of men from the seats, to pedal positions, to belts, and helmet straps, and so on. The result was that pilot performance soared and the US Air Force became the most dominant air force on the planet.

The book gives other example studies and goes on to say

Any system designed around the average person is doomed to fail

This is the gist of the book and what Michael Tiemann was getting at.

Anyway, the summary implied that the generation of planes designed in the 1950's were a generation of pilot killers.
The 1950's planes had the cockpit fit problems solved.
The crashing planes were in the late 1940's. The study was begun in 1950. Obviously, those crashes were not combat-related. Those planes were demanding and possibly evil, and a bad-fitting cockpit made it worse.