Slashdot Mirror
Flawed Analysis, Failed Oversight: How Boeing, FAA Certified the Suspect 737 MAX Flight Control System (seattletimes.com)
In one of the most detailed descriptions yet of the relationship between Boeing and the Federal Aviation Administration during the 737 Max's certification process, the Seattle Times reports that the U.S. regulator delegated much of the safety assessment to Boeing and that the analysis the planemaker in turn delivered to the authorities had crucial flaws. 0x2A shares the report: Both Boeing and the FAA were informed of the specifics of this story and were asked for responses 11 days ago, before the second crash of a 737 MAX. [...] Several technical experts inside the FAA said October's Lion Air crash, where the MCAS (Maneuvering Characteristics Augmentation System) has been clearly implicated by investigators in Indonesia, is only the latest indicator that the agency's delegation of airplane certification has gone too far, and that it's inappropriate for Boeing employees to have so much authority over safety analyses of Boeing jets. "We need to make sure the FAA is much more engaged in failure assessments and the assumptions that go into them," said one FAA safety engineer. Going against a long Boeing tradition of giving the pilot complete control of the aircraft, the MAX's new MCAS automatic flight control system was designed to act in the background, without pilot input. It was needed because the MAX's much larger engines had to be placed farther forward on the wing, changing the airframe's aerodynamic lift. Designed to activate automatically only in the extreme flight situation of a high-speed stall, this extra kick downward of the nose would make the plane feel the same to a pilot as the older-model 737s.
Boeing engineers authorized to work on behalf of the FAA developed the System Safety Analysis for MCAS, a document which in turn was shared with foreign air-safety regulators in Europe, Canada and elsewhere in the world. The document, "developed to ensure the safe operation of the 737 MAX," concluded that the system complied with all applicable FAA regulations. Yet black box data retrieved after the Lion Air crash indicates that a single faulty sensor -- a vane on the outside of the fuselage that measures the plane's "angle of attack," the angle between the airflow and the wing -- triggered MCAS multiple times during the deadly flight, initiating a tug of war as the system repeatedly pushed the nose of the plane down and the pilots wrestled with the controls to pull it back up, before the final crash.
[...] On the Lion Air flight, when the MCAS pushed the jet's nose down, the captain pulled it back up, using thumb switches on the control column. Still operating under the false angle-of-attack reading, MCAS kicked in each time to swivel the horizontal tail and push the nose down again. The black box data released in the preliminary investigation report shows that after this cycle repeated 21 times, the plane's captain ceded control to the first officer. As MCAS pushed the nose down two or three times more, the first officer responded with only two short flicks of the thumb switches. At a limit of 2.5 degrees, two cycles of MCAS without correction would have been enough to reach the maximum nose-down effect. In the final seconds, the black box data shows the captain resumed control and pulled back up with high force. But it was too late. The plane dived into the sea at more than 500 miles per hour. [...] The former Boeing flight controls engineer who worked on the MAX's certification on behalf of the FAA said that whether a system on a jet can rely on one sensor input, or must have two, is driven by the failure classification in the system safety analysis. He said virtually all equipment on any commercial airplane, including the various sensors, is reliable enough to meet the "major failure" requirement, which is that the probability of a failure must be less than one in 100,000. Such systems are therefore typically allowed to rely on a single input sensor.
Boeing engineers authorized to work on behalf of the FAA developed the System Safety Analysis for MCAS, a document which in turn was shared with foreign air-safety regulators in Europe, Canada and elsewhere in the world. The document, "developed to ensure the safe operation of the 737 MAX," concluded that the system complied with all applicable FAA regulations. Yet black box data retrieved after the Lion Air crash indicates that a single faulty sensor -- a vane on the outside of the fuselage that measures the plane's "angle of attack," the angle between the airflow and the wing -- triggered MCAS multiple times during the deadly flight, initiating a tug of war as the system repeatedly pushed the nose of the plane down and the pilots wrestled with the controls to pull it back up, before the final crash.
[...] On the Lion Air flight, when the MCAS pushed the jet's nose down, the captain pulled it back up, using thumb switches on the control column. Still operating under the false angle-of-attack reading, MCAS kicked in each time to swivel the horizontal tail and push the nose down again. The black box data released in the preliminary investigation report shows that after this cycle repeated 21 times, the plane's captain ceded control to the first officer. As MCAS pushed the nose down two or three times more, the first officer responded with only two short flicks of the thumb switches. At a limit of 2.5 degrees, two cycles of MCAS without correction would have been enough to reach the maximum nose-down effect. In the final seconds, the black box data shows the captain resumed control and pulled back up with high force. But it was too late. The plane dived into the sea at more than 500 miles per hour. [...] The former Boeing flight controls engineer who worked on the MAX's certification on behalf of the FAA said that whether a system on a jet can rely on one sensor input, or must have two, is driven by the failure classification in the system safety analysis. He said virtually all equipment on any commercial airplane, including the various sensors, is reliable enough to meet the "major failure" requirement, which is that the probability of a failure must be less than one in 100,000. Such systems are therefore typically allowed to rely on a single input sensor.
This judgement is going to run into 10 digits.
Yeah, but that costs extra, and making it an option allows Boeing to nickel-and-dime the airlines that want to look more professional.
And we can't have these costly things being mandatory in a free market neo-liberal economy!
This wasn't the 'stop trying to kill us button' this is like you are going up a hill and cruise control decides to slow down 1 mph per degree incline of the hill. so you keep pressing ACCL(+) until you are back to 65mph. except you suddenly come to a crest in the hill and you realize it too late and try to pull up on your ebrake but your ebrake doesn't disengage cruise control so you end up going down the hill at 100mph and careen off the side because you had no control
In reality there are a few faults here. A system designed to overcome aerodynamic flaws of larger engines is not a major failure scenario? But then again, Boeing offered options to provide detailed insight to these sensors that the customer opt'd out of. Who's at fault here?
Secondly, at what point does trim input negating MCAS (a) illicit a change in computing behavior and more importantly (b) illicit a change in human behavior? I read that and immediately what the fucked out loud because what pilot would trim up 21 times before disconnecting auto pilot and flying by hand while figuring out what is going on. This is showing the pilot is way to reliant on computers rather than hand flying the plane (something all American regulations enforce). Who's at fault here, Boeing or the country with lax pilot regulation?
-dk
it'll be safer to buy Delta tickets than find that other airlines are again allowed to put these Max planes back in the air
You said "Safer" and "Delta" in the same sentence, hmm...
This issue seems like something the pilots can work around if they know what is going on, which the U.S. pilots seem to.
I got an email from Southwest Cargo related to the Max, they stated:
While we remain confident in the MAX 8 after completing more than 88,000 flight hours accrued over 41,000 flights, we support the actions of the FAA and other regulatory agencies and governments across the globe that have asked for further review of the data
That's a lot of flights they have done with the plane, so it's not like the plane is inherently unsafe - there is a flaw in this system, which will get resolved one way or another. They'll be back in the air and as safe as any other place flying.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
In general if you have 2 sensors that disagree significantly, you disable all functions that rely on those sensors and issue an alarm.
You might be able to decide which sensor is correct from data from other systems, but that is another story
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
> "Going against a long Boeing tradition of giving the pilot complete control of the aircraft, the MAX's new MCAS automatic flight control system was designed to act in the background, without pilot input"
Or notify them either, it seems. Or be disabled when it erroneously kicks in over 20 times causing unexpected dives. Fuck everything about this system. Even if they fix it I'm not flying on any aircraft that has this.
> "this extra kick downward of the nose would make the plane feel the same to a pilot as the older-model 737s"
And that's also ridiculous. Because of the change in the engine configuration it is an aircraft that handles differently. "Compensating" so the pilot doesn't know the difference causes confusion, something you don't need when in charge of a passenger jet. Do they make 747s feel like you're flying a TriStar? Of course not.
"Wait. Something's happening. It's opening up! My God, it's full of apricots!"
Right, automation is good but when lives are on the line....one needs to take every precaution and think about failure cases. I saw a video elsewhere that said that there was an easy way to disable the sensor, but when the pilot only has a few seconds to respond and he is busy trying to keep the plane in the air... in either case, even if we agreed that 1 sensor is enough, 1 in 100K chance doesn't sound reliable enough to me.....I'd rather see 1 in a million minimum, 1 in a billion ideally.. You might need to 5 sensors where at least 3 of them must trigger fault to get super reliability. I'm not sure how expensive or tricky placing several of these sensors is.... In any case, non of us are pilots so its all speculation here.
Politics and economics wise, the US Air Force was reported to have recently chastened Boeing for QA issues. China and Europe, which want to dominate high tech airplanes have a vested interest in taking down Boeing. But, it sounds like Boeing did this all to themselves....perhaps cutting corners to increase time to market and production speed.
As for the FAA, I never have high expectations of any government agency to look out for public safety over vested national and economics interests. Letting companies get sued into bankruptcy with the CEO's unemployable when they massively screw up is a much more compelling and reliable way to ensure corners aren't cut.
Well, you may be right that this smells... And you may be right in your assumption that Boeing rushed through the certification process and the FAA failed in its oversight capacity and Boeing will be left liable for a pile of money... However, the implication that there was some kind of behind the scenes collusion deal between the FAA and Boeing though is a pretty heavy lift as you have crossed over from civil liability into criminal activity where the burden of proof moves from preponderance of evidence to beyond a shadow of a doubt.
But, the Civil liability problem here will be borne by Boeing's insurance companies and punitive damages will rack up some pretty big numbers for the victims as a result which will come out of Boeing's profits after being tied up in court for about a decade on appeal.
The end result will be that the aircraft will be rendered fit for service pretty quick and sales of the 737 MAX will resume unabated perhaps with a new name, with some PR efforts by Boeing and the airlines that fly these aircraft for a reason (they are cheaper to operate). There is nothing systemically wrong with the aircraft mechanically or aerodynamically and this flight control issue will be resolved, albeit by adding multiple sensors, cross checking of existing and redundant sensor data along with some software fixes and pilot training.
I'm no Boeing fan boy, but let's be reasonable here. Yes, this will hurt Boeing in the short term and the awards will initially be sizeable, with the punitive part getting appealed and appealed for at least a decade before they get paid. This will largely be paid by their insurance carrier and their premiums will be assured to rise. However, these awards pale in comparison to the cost of an aircraft development program and Boeing won't struggle to pay them when they come due. The aircraft system will be reevaluated and redesigned as necessary to account for lessons learned. Any folks who should have known better in the decision tree for fielding and certifying the 737 MAX will be rooted out, processes to make sure this kind of thing doesn't slip by again will be introduced and we will return to normal.
Where this mistake is bad, let's put it in prospective for the nations air safety. We've come a LONG way from the 60's when the accident rates where huge compared to now or even the 90's on air safety, when DC-10's where crashing right and left from Cargo doors blowing open and uncontained turbine failures. It's been a LONG time since the last major management mistake in air safety. A very long time. Humans make mistakes and flying is a risky business that quickly turns mistakes into tragedy, we won't avoid human error in the future, all we can do is try and catch it before it kills anybody.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Attitude is only one element of the aircraft's operation -- what about airspeed?
Surely if there was a large disparity between the aircraft's airspeed and its attitude (ie: it is accelerating beyond 500mph while the attitude sensor says it's in a steep climb) then the safety system ought to have recognized that there was a fault condition and triggered an alarm which would allow pilots to disable it with the simple flick of a switch.
Sadly, it seems that this system was never designed to be disabled -- because it was part of the FBW system used to modify the apparent flight characteristics of the new Max8 model so that it would fly like an earlier 737. This was done (so I understand) solely to make the plane more attractive to airlines that didn't want the extra expense of having to get their pilots "rated" for a new aircraft type.
When it comes to the mighty dollar versus safety -- you *know* which one wins :-(
Meanwhile, some people are still saying "it's only a matter of time before a drone brings down an airliner". I wish they'd shut up and focus on the *real* risks that are *actually* claiming hundreds of lives in the aviation industry.
What we see here is reflected somewhat in most major incident investigations through industry involving instrumented systems, the reliability of the equipment is not in question. Throughout the process industry some 80% of safety system failures were systematic. Poor design, poor maintenance, poor interaction, incorrect operation, etc. One in 100000 units failing is not what ultimately caused these planes to crash, it was a bunch of engineers who didn't think about how the system works in operation.
You could blame it on the Obama administration or the Trump administration, but it goes back a long time.
The Federal Aviation Act of 1958 was the original statute allowing FAA to delegate activities, as the agency thinks necessary, to approved private people employed by aircraft manufacturers. Although paid by the manufacturers, these designees act as surrogates for FAA in examining aircraft designs, production quality, and airworthiness. The FAA is responsible for overseeing the designees' work and determining whether the designs meet FAA requirements for safety.
Often old and simpler is far better....
Right until you look at outcomes. You're speaking emotionally from a recent tragic incident. You're not speaking based on data. The airline (along with others such as the process and automotive) industries have had a long downward trend of safety incidents. One of the primary drivers of that has been taking control away from people. As a Boeing noses down to prevent a stall, a car somewhere in the world saves a drive thanks to forward crash avoidance. An operator who mistakenly lowers the level from a high pressure separator is greeted by flashing alarms on his screen and a valve slamming shut in the field to prevent an explosion.
Humans make mistakes, giving them full control is not the answer. It's always worth remembering why this system was built, and how in the past pilots have through their own failure demolished plenty of planes due to putting the aircraft into a stall.
Sidenote: The thing that is really missing here which goes against industry trends is a lack of inherently safer design. A more stable plane is preferable to a plane that is only stable when a certain control system is active.
<quote><p>A system designed to overcome aerodynamic flaws of larger engines is not a major failure scenario?</p></quote>
<p>Of course it is, but what is the safe action? </quote>
The safe action is the one that nobody is talking about. The previous version of the 737 had engines so big that they had to flatten the intake on the bottom so it would fit under the wing. That should have been a clue that the existing 737 design was already at its limit. By putting even larger engines on it, they had to mess up the aerodynamic stability of the aircraft such that they had to implement this software fix just to get through the approvals. It's pretty obvious that someone should have said: "look, the 737 is great, but it's at end of life. We need to make a new aircraft design now."
Imagine if we were still flying the DC-3 with every new technological advance since it was designed kludged on to it even though it was never designed for them? At a certain point you need to realize that your design is at the end of its life and move on.
But that costs money, and apparently hundreds of lives.
In some sense nobody really made the decision to use this design without redundant sensors. According to the article, the system was approved with a relatively small amount of authority - it could only move the tail by 0.6 degrees. That wasn't a bad enough issue to warrant redundancy. The problem is that the authority was then increased to 2.5 degrees, more than 4 times larger, and the safety impact was simply never re-evaluated due to the rush to get it on the market. Even documents given to other country's air safety bodies still listed the 0.6 degrees. The explosive thing about this, which is why the article predates the second crash, is that this puts the whole process in doubt. How many other numbers in the documents are just fiction? How many other safety evaluation chains have not been updated due to the rush to market? Does this amount to fraudulent behavior on the part of Boeing? My expectation is that the engineer who upped the authority from 0.6 to 2.5 did so with the intent, possibly even documented, that the safety would be re-evaluated before the jet went to market.
It's also unclear why the authority was listed as 0.6 degrees when the system could repeatedly reset itself and do it again, effectively giving it infinite authority. That is more along the lines of your question, but I think it actually wasn't clear why the ability to reset was not included in the safety analysis. This really looks a lot like an updated safety analysis was planned, postponed, and then just never done until after the Lion Air crash.
The pilots thought it was relevant, they thought that without auto-pilot on there were no automatic systems overriding their controls.
Should they have treated it like any other trim failure, sure. Does the system betraying expectations increase the chance of cognitive dissonance and them failing to do so, of course.
You are. This crash merely shows yet again that a badly trained pilot - and many of them are - will crash the aircraft as soon as something unexpected happens. The cycle repeated for 21 bloody times yet the pilot kept fighting the aircraft instead of executing the correct procedure for a runaway stabiliser (essentily flicking two switches and manually cranking the stabiliser in the correct position).
Bad pilots are a fact of life, hence the only way to protect passengers from pilots is more automation, not less.
"It's such a fine line between stupid and clever" -- David St. Hubbins, Spinal Tap
There's a good chance the aftermath of this is going to bankrupt Boeing.
The evidence for gross engineering negligence is piling up, and they are not going to live through the results.
"One current FAA safety engineer said that every time the pilots on the Lion Air flight reset the (trim) switches on their control columns to pull the nose back up, MCAS would (reset its 0 degree reference and) have kicked in again and “allowed new increments of 2.5 degrees.”
“So once they pushed a couple of times, they were at full stop,” meaning at the full extent of the tail swivel, he said.
So in summary a system FAA-certified on the basis of being able to adjust nose-down trim by 0.6 degrees could actually, (after a few cycles of the pilot correcting it a little bit with trim up), command full nose-down trim, about 5 or 6 degrees tailplane tilt.
All of this relying on input from a single angle-of-attack sensor. Get this, the plane has two such sensors, one on each side, but the MCAS only uses input from one of them!!! ! !! ! ! ! ! What the hell? If you use two of them, then your software can check if they diverge, and disable systems relying on the input, and warn the pilots. That is some criminally bad development cost saving judgement there.
Where are we going and why are we in a handbasket?
Sandy Hook taught me that the non-regulated industries have lobbyists that prevent regulation.
It is ok for babies to be slaughtered in USA.
The couple of adult americans killed on an airplane is a sad event, but the reality is babies were slaughtered on USA soil and usa government did nothing.
I fully expect in light of this, that the entrenched Boeing politically connected will also walk free, maybe a small slap on wrist and a stern talking to.
In the end, they only have their reputation tarnished.
Thats it.
I'm not a big fan of MBAs, but this was a pretty long and complicated chain of errors. From what I gather: Boeing wanted to keep the 737's low ground clearance but needed to put bigger engines on to match the efficiency of the new A320s, which meant changing the aerodynamics. Boeing also wanted pilots to be able to do a simple difference training course, rather than have to recertify on a new aircraft, so they invented MCAS. The engineers must have figured that it was a supplemental system, and easy to turn off if it malfunctioned, so they chose to make it kick in aggressively rather than conservatively (either sensor says go, rather than both sensors say go). They also made it harder to turn off than the old system, probably by accident. Then Boeing decided not to mention the new system to pilots in that difference course, to avoid confusing them.
Lots of errors to go around. Some are definitely cost saving, but some are probably a result of not enough whole-system oversight. The decision to go based on one sensor is a bit mystifying. There are already two AoA sensors on the aircraft, and lots of other ways of cross checking them. In fact, Boeing is releasing a software update to add all that cross checking in, so it's not even a hardware limitation.
The 737 MAX isn't actually aerodynamically unstable in normal flight. Any airliner, including all the 737s, with the standard under-the-wing engines will have off-axis thrust that will add a bit of pitch up. The aircraft is designed to compensate for that in normal flight, but in a stall if the pilot gooses the engine it can make it impossible to recover. 737 pilots (including the older model) are trained NOT to increase throttle in a stall because of it. The MAX handles differently in that situation, so they added MCAS so the pilots wouldn't have to be trained in a new stall recovery procedure.