Boeing To Make Key Change in 737 MAX Cockpit Software (wsj.com)

Boeing is making an extensive change to the flight-control system in the 737 MAX aircraft implicated in October's Lion Air crash in Indonesia, going beyond what many industry officials familiar with the discussions had anticipated. From a report: The change was in the works before a second plane of the same make crashed in Africa last weekend -- and comes as world-wide unease about the 737 MAX's safety grows. The change would mark a major shift from how Boeing originally designed a stall-prevention feature in the aircraft, which were first delivered to airlines in 2017. U.S. aviation regulators are expected to mandate the change by the end of April.

Boeing publicly released details about the planned 737 MAX software update late Monday [Editor's note: the link may be paywalled; alternative source]. A company spokesman confirmed the update would use multiple sensors, or data feeds, in MAX's stall-prevention system -- instead of the current reliance on a single sensor. The change was prompted by preliminary results from the Indonesian crash investigation indicating that erroneous data from a single sensor, which measures the angle of the plane's nose, caused the stall-prevention system to misfire. Then, a series of events put the aircraft into a dangerous dive.

Comments

I guess the incredibly obvious question is...

[ZorinLynx]

Why the hell wasn't this the case before?

Aren't flight control systems supposed to be triple-redundant anyway? Everything I've read about them says they are; three systems and if there is incorrect data it uses the two that agree.

Re:I guess the incredibly obvious question is...

[lgw]

I assume they're talking about the sensor behind the pitot hole here. Making that the only sensor, and non-redundant, is particularly questionable. It's well known that pitot holes are very easily thrown off: an insect building a nest inside it (or ice forming, or etc) will throw off the sensor enough to crash a plane, if it's all you rely on.

Re:I guess the incredibly obvious question is...

[geekmux]

I assume they're talking about the sensor behind the pitot hole here. Making that the only sensor, and non-redundant, is particularly questionable. It's well known that pitot holes are very easily thrown off: an insect building a nest inside it (or ice forming, or etc) will throw off the sensor enough to crash a plane, if it's all you rely on.

I would assume you're correct here, but it still begs the question as to why this sensor was non-redundant, and how that SPOF design ultimately got approved.

Re:I guess the incredibly obvious question is...

[drinkypoo]

Yes, this is absolutely bananas. Even the accelerator pedal position sensor on cars with throttle-by-wire is a pair of pots, not just one. If one sweeps smoothly and the other doesn't, the PCM throws a code and only listens to the smooth input.

Re:I guess the incredibly obvious question is...

[bobbied]

Why the hell wasn't this the case before?

Aren't flight control systems supposed to be triple-redundant anyway? Everything I've read about them says they are; three systems and if there is incorrect data it uses the two that agree.

Well.. I believe the way the system works allows the control inputs of the pilots are able to overcome anything the system does. It's basically like an autopilot, where the pilot can override the system by applying pressure to the controls. This system is designed to apply backpressure as the aircraft approaches a stall, making it harder for the pilot to continue to increase the angle of attack and hopefully avoiding the stall. So you can still stall the aircraft, just pull harder and keep increasing the AOA...

The problem though, is that pilots are conditioned to change the trim to deal with unusual pressures for the desired pitch angle. So if the system believes the sensor and it's saying "STALL" but you are actually not, the system applies pressure to lower the nose, which the pilots will be conditioned to trim out. IF the stall doesn't go away, the system keeps the pressure there and unless the pilots realize what's going on they will keep trimming nose up. Eventually, the process ends up with an aircraft that's severely out of pitch trim which will be very confusing to the pilots, with really high control pressures required to do anything to the pitch. Thus "control problems" seems to describe exactly what I imagine was going on. It was a vicious cycle that makes the aircraft really hard to control.

So, I understand the engineering and using one AOA sensor. Kind of makes sense... Hey, the pilots can just override this anyway, we are stopping them from actually stalling the aircraft, just making it harder to do. We've don't this before in fighter aircraft and other fly by wire systems w/o any problems. But I think there wasn't enough thought given to what happens when that sensor fails and if they can implement some cross checks between airspeed, rate of climb, rate of turn, they might be able to more gracefully fail the system and disable it, or at least not get into the vicious cycle that leads to a pitch trim issue.

Re:I guess the incredibly obvious question is...

[Solandri]

Usually there are 3+ pitot tubes. Looks like the 737 has 5, with 3 of them dedicated to measuring airspeed. It's incredibly rare that a single fault causes a crash. Reporters just like to write up their stories that way to give their stories more impact, even if it twists the truth.

This isn't the first time faulty airspeed readings led to a flight computer has led to a crash. It isn't even the second time. In all previous cases, the plane was flyable. It was the confusion as the pilots tried to diagnose the problem based on the bizarre behavior of the plane and the flight control software and alarms which doomed the flights. It requires a deep and thorough understanding of when different flight protection modes in the software are triggered and kick in, to work backwards from the behavior you're seeing, to what problem(s) could be triggering those modes. If you've debugged software, you've encountered this. Unlike natural laws like physics, software can be designed arbitrarily. So your intuitive feel for how things should work becomes useless for tracking down the problem. You're totally dependent on how thoroughly you understand the software's arbitrary design.

Bear in mind that the stall warning is pretty much a "you're gonna die if you ignore me" warning. So it takes quite a bit of convincing before pilots will decide it's the warning that's faulty, not something else that they're doing wrong. That may be the cause of the reluctance of pilots to simply shut it off and fly the plane "by the seat of their pants" based on how the throttle settings, altitude, and attitude. So while theoretically the stall warning triggering incorrectly is a recoverable problem, it may take pilots a long time to diagnose and clear up the problem. Long enough for the plane to crash.

Re:I guess the incredibly obvious question is...

[Shotgun]

The question here is why is the computer listening to a sensor instead of the pilot. A plane can be flown just fine without any instrumentation other than the front window. Why does that sensor get to override the pilot?

Re: I guess the incredibly obvious question is...

[Bryansix]

There is a way to disable the systems but the procedure to do so is incredibly complicated. It's also hard to do anything when an emergency happens seconds after takeoff and the plane isn't even far off the ground yet.

Re:I guess the incredibly obvious question is...

[uncqual]

My lay person's understanding...

In order to increase fuel efficiency on the 737 MAX, the engine fan diameter was increased. These "underwing" engines would have been too close to the ground if mounted as on other 737 models. Thus, the engineers moved the engines forward and upward to achieve necessary ground clearance. This, along with some other changes, moved the force of thrust forward which made the plane more prone to lift its nose too high and stall. To guard against this, Boeing introduced the Maneuvering Characteristics Augmentation System (MCAS) which activates automatically when the autopilot is off in some conditions which include when the angle of attack (AOA) is too high. The MCAS system, when needed, attempts to prevent a stall by adjusting the horizontal stabilizer trim upward and will do this over, I believe, about 10 seconds or until the pilot overrides it or the angle of attack is within limits. If the pilot activates the trim control switch on the yoke, MCAS will be disabled -- but, five seconds after the switch is released, MCAS will reengage if the conditions call for it (esp. AOA). When MCAS is altering the trim, the manual trim controls on each side of the center "console" will be spinning away and, if a pilot looks down, they will see that motion as there is a white stripe extending outward from the center in order to make the movement obvious.

The best speculation I've heard about the Lion Air crash was that there was a problem with one of the AOA sensors. There are two such sensors - one on both side of the 737 Max.

As in most crashes, due to the redundancy of systems and procedures, it's rarely one thing that causes a crash but rather a cascade of events.

There had been problems with at least one of the AOAs on previous flights but maintenance attempts appear not to have solved the problem. So, first there was a failure of maintenance, but of course AOA sensors will fail from time to time, so one can't blame the crash on that failure.

I've not heard how MCAS handled conflicting AOA sensor readings but I suspect this is one of the big areas of change that they will push in the April "patch". But, it's likely that the failing AOA caused the MCAS to activate when it shouldn't have and push the nose down by adjusting the trim - but this actually pushed the plane's nose down too far. When the pilots tried to correct, they ended up disabling MCAS (although perhaps not explicitly aware that they were doing so) only to have it start undoing what they had accomplished five seconds after they released the trim control on the yolk - and this was a vicious loop.

Had the pilot recognized what was happening, they simply would have ran the "runaway trim" procedure (which would have disabled MCAS and some other automatic trim controls completely via a switch on the center "console") and flown the plane manually with no problems. Unfortunately, the pilots likely didn't figure out what was causing the problem and failed to execute the necessary procedure. So, that was a pilot error (and, that's probably what will be determined to be the main problem here, with contributing factors).

There is much debate on why the Lion Air pilots may have failed to recognize what was going on. Many pilots and their union claim that they were not told about the existence of MCAS. Boeing hasn't been talking a lot, but they seem to assert that there was no need to train the pilots on MCAS beyond what the manuals/training did as it was a classic "runaway trim" scenario and the training was sufficient to cause the pilots to detect that case and initiate the proper procedure. Boeing did, however, issue documentation updates to operators worldwide soon after the Lion Air crash.

After Boeing issued the documentation updates, every 737 MAX pilot should have been fully aware of MCAS and what to do if was doing the wrong thing. This, coupled with the witness reports that the Ethiopian Airline 737 MAX that crashed was spewing smoke and fire from the back of the plane a

Re:I guess the incredibly obvious question is...

[caseih]

The MCAS spins the same trim knobs that the pilot spins. So the pilot can trim the nose back and after MCAS spins it down. They might fight each other, but ultimately they are both adjusting (and potentially undoing) the same thing. I'm sure it's initially confusing to pilots for sure, especially because older planes would cancel the automatic trims when the stick was pulled on, but apparently this is not the case with MCAS. If it turns out the MCAS contributed to the Indonesian crash, then it was a matter of training. But Boeing screwed up the design.

Obvious

[Anonymous Coward]

if ( goingToCrash ) {
        dontCrash();
}

Next one will crash due to stall

[jfdavis668]

That would have been prevented by the current system.

Re:The Tesla People

[Geoffrey.landis]

It's funny how they point to aviation as nearly infallible when they talk about self-driving cars.

Well, it's a little less than one failure in four million flight hours, that's a pretty amazing safety record. If Tesla self-driving was one failure in four million driving hours, I'd call that very near infallible, compared to human drivers, anyway.

But when they do fail, it's spectacular, and makes news.

Source: http://planecrashinfo.com/caus...

Re:The Tesla People

[PvtVoid]

It's funny how they point to aviation as nearly infallible when they talk about self-driving cars.

Self-driving cars don't have to be infallible. They just have to be safer than the average person, which is a really low bar.

Re:Redundant Systems?

[bobbied]

million dollar aircraft brought down by a cheap sensor failure

Well that's better than the aircraft accident I helped to investigate... The pilot died because of a power switch position he specifically set in order to turn off the system that prevented his aircraft from departing it's "flying" envelope by applying back pressure to his control inputs. When he went to "break" during some ACM training looking over his shoulder at his opponent, he applied too much rudder input, the aircraft snap rolled as it stopped flying and started to tumble, his head was caught between the ejection seat and the canopy and he died of a broken neck before his aircraft hit the water.

That guy died because he wanted the competitive edge and specifically tried to cheat by putting the aircraft in a forbidden configuration....

I'd rather die from a sensor failure than by some stupid mistake I made to get an unfair advantge because I want to win some competition..

Additional sources

[Anubis IV]

Since the alternative source link in the summary appears to link to an article about stock prices, here's some alternative alternative links that actually contain more relevant information:
- Boeing press release
- Gizmodo
- Washington Post

Re:seems like the logic here is flawed.

[BostonPilot]

No, you're trying to grossly oversimplify the problem, and it's causing you to say things that are silly.

Having worked as a vendor to the avionics group at Boeing, and having had a student who wrote test code for the 777, I can tell you that the testing / verification process for their software is mind boggling. They've had decades to fine tune their processes for creating reliable computer software. Believe me, you sound idiotic second guessing them, and it doesn't sound like you're a pilot either...

The one thing I will agree with you about is that the system should trust the crew. However, I must say that some of my airline captain buddies would strongly disagree with that. Just look at Air France Flight 447 as a perfect example of why trusting the crew can go wrong. However, I still lean towards this... if you don't trust the crew then it's like the old joke about the perfect crew:

The ideal flight crew is a pilot and a dog.

The pilot is there to feed the dog, and the dog is there to bite the pilot if he touches anything.

Seriously, if the automation is so complicated and opaque that the crew can't tell what it's doing and why... that's a problem. The move towards more automation seems to be to make up for an inexperienced crew... I think more training / sim time is the right solution, not more automation. Still, both Airbus and Boeing seem to think more automation is the right way to go.

I'll be interested to hear what they learn from the FDR...

Re:The problem is normal and alternate control law

[uncqual]

A description of alternate law as it applies to aviation can be found here although this focuses on Airbus.

Re:seems like the logic here is flawed.

[K. S. Kyosuke]

I can tell you that the testing / verification process for their software is mind boggling. They've had decades to fine tune their processes for creating reliable computer software.

Haven't we had ample evidence by now that it's all too easy to make computer software that very reliably and very accurately does exactly the wrong thing?

Re: I guess the incredibly obvious question is..

[Hognoxious]

There is an option to disable the system.

It's a button in the aft toilet under a locked flap with "beware of the leopard" written on it.