Slashdot Mirror


Software Bug in F-35 Radar Causes Mid-Flight System Reboot

Reader Lisandro writes: The F-35 Fighter jet can't seem to catch a break. An advanced AN/APG-81 AESA F35 radar system has been found riddled with a software bug that causes it to degrade and stop working. The solution? Rebooting the system while in the air.

Major General Jeffrey Harrigian, director of the Air Force's F-35 integration office at the Pentagon, was quoted as saying "radar stability - the radar's ability to stay up and running. [...] What would happen is they'd get a signal that says either a radar degrade or a radar fail - "something that would force us to restart the radar." The issue was spotted in late 2015, and thankfully, it was caught during the testing period. The software version "3i" is affected. An update aimed to resolve the bug is expected to be delivered to the US Air Force by the end of March.

25 of 153 comments (clear)

  1. Nothing to see here by Freshly+Exhumed · · Score: 5, Funny

    Oh come on, who here hasn't had to reboot during air to air combat?

    --
    I deny that I have not avoided attaining the opposite of that which I do not want.
    1. Re:Nothing to see here by Feral+Nerd · · Score: 5, Funny

      Oh come on, who here hasn't had to reboot during air to air combat?

      ... a problem that is aggravated by system's insisting on the installation of innumerable update packages on every reboot.

      MISSILE LAUNCH DETECTED!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE IMPACT IMMINENT, EJECT! EJECT!
      Installing radar software update 4 of 68....
      For this update you need Microsoft Silverlight, install Silverlight [Y/N]:

    2. Re:Nothing to see here by sumdumass · · Score: 5, Funny

      You forgot we recommend you upgrade to Windows 10.

    3. Re:Nothing to see here by Coisiche · · Score: 3, Funny

      My first thought was alternate movie dialog.

      Maverick: Talk to me Goose... Where's the bogey?

      Goose: Uh... hang on a moment Maverick, we're just going through a reboot... any... minute... now...

    4. Re:Nothing to see here by Aqualung812 · · Score: 3, Insightful

      It looks like the F35 is the perfect military industrial complex aircraft, forever requiring upgrades and bug fixes and not just from the US government from every government required to fork over 2% of GDP tribute payment to the US military industrial complex.

      I'm not a fan of the F35 program, but this is still done in meatspace with aircraft as old as the B52.

      The aircraft manufacturers are constantly pushing new hardware updates, new specs on how each bolt should be tightened, etc. Those don't come for free.

      It is even possible that ongoing costs will be lower with something software-based rather than hardware. It turns out that way often.

      I don't expect it to get cheaper, but if it doesn't it is in spite of software updates, not because of them.

      --
      Grammer Nazis - I mod you "troll" unless you actually add something on-topic. Yes, I know I have mispellings in my sig.
    5. Re:Nothing to see here by dywolf · · Score: 4, Informative

      actually "rebooting", ie, flipping the power switch or circuit breaker, isn't at all uncommon on avionics equipment on military aircraft.
      we aren't talking about typical computers that go through a boot process anyway. this is ruggedized equipment that largely lacks any thing resembling an operating system or RAM or much else a typical Slashdot reader would be familiar with.

      anyone who's spent any time working on military aircraft as a maintainer, particularly the avionics systems, knows that inflight glitches are not at all infrequent. and when they pop up on the Master Caution* or elsewhere, often the first corrective action the pilot takes is to power cycle the specific piece of equipment. most every system is on its own breaker, and pilots are trained in what can and what cannot be power cycled in flight. the majority of the time, that's enough to fix the glitch.

      and typically the first thing that happens when the pilot returns is a rep from each of the main work shops (avionics, flightline, airframes, ordinance, life support) meets him as he is exiting his aircraft, in order to ask if any gripes came up during the flight. this way they can get a jump on it before the pilot even gets back to the maintenance control to write the maintenance order describing the glitch.

      there a thousands of wires, with hundreds of connectors, each connector a cannon plug consisting of several dozen pins, any one of which could have gotten slightly bent (or even broken) upon reconnection, making an imperfect electrical connection or faulty data bus signal (depending on system). Or a wire may fall out of the backend of the pin from a faulty installation of the retainer of the cannon plug. or the plug itself may be not quite fully seated; you'd think it would be easy, but there's a reason we have cannon plug pliers (aka "bi*ch grips"). There's also millions of solder joints and splices that can fatigue from vibration. sometimes a contact simply gets dirty cause oil or grease (we wipe everything constantly, but still happens).

      (*speaking of PITA to maintain: due its nature, being tied into EVERYTHING (hundreds, sometimes thousands, of feet of wiring, depending on aircraft type), the Master Caution Panel (MCP) itself is often the actual point of failure, throwing false indications. one of the first things we frequently did in tracing a gripe was to first eliminate the MCP itself)

      --
      The guy who said the election was rigged won the presidency with the second-most votes.
    6. Re:Nothing to see here by Noble713 · · Score: 3, Informative
      I know you are being somewhat facetious, but I suspect these positions aren't too far from your actual beliefs, therefore....

      Yep, Russia meddled and took part of Georgia because the evil U.S. was there. And they decided to steal part of the Ukraine for the same reason.

      Because the US State Department spending $5 billion to "influence" the political situation in a country directly on Russia's border couldn't *possibly* provoke a response, right? Crimea is one of Russia's few warm-water ports and an essential link to the Mediterranean. How do you think the US would respond to political instability in Panama, especially if it was caused by another major world power? Oh wait, we already know: https://en.wikipedia.org/wiki/...

      And those islands in the S. China Sea, why Vietnam and the other nations to which they are closer are only doing the bidding of the U.S. hence the need for China to militarize them.

      Something like 80% of the sea traffic going through the SCS is either to or from China. China, which is ~20% of humanity, compared to ~2% combined for Vietnam and the Philippines. Do the Needs of the Many and the Greater Good not apply? And US think tanks have written extensively about strangling raw material imports to China in the event of a conflict. Yeah, no way the Chinese might have a rational self-interest in securing the lifeline to their economy in their own backyard.

      . And Assad of Syria, we just know he was playing secret footsie with Americans before he decided to slaughter his people and chase a few million out of the country.

      Don't you think Assad would rather have a few million additional taxpayers contributing to his economy, even if significant portions of them are the unhappy Sunni majority? He certainly seemed to be getting along fine in 2010. Funny how the provision of funding, foreign fighters, and weapons from Saudi Arabia, Qatar, and Turkey (all Sunni states) roughly coincides with Assad's refusal to allow the planned Qatari/Saudi pipelines across his country. Pipelines to Europe that would undercut his patron Russia's economic interests.

      On a related note, do you express as much disgust at the suppression of popular dissent in Bahrain, or is that ok because King Hamad of Bahrain "is our man"?

    7. Re:Nothing to see here by Noble713 · · Score: 4, Insightful

      Because Russia invading a sovereign country is perfectly acceptable because they weren't doing what Russia wanted them to do. Of course, pointing to another country doing something kind of the same excuses what Russia is doing, it is all perfectly acceptable to annex the territory of another country.

      Great Powers do it all the time. I don't expect it to change. But I do aim to point out the hypocrisy and/or naivete of anyone who thinks the US's foreign policy has altruistic motives or that the "Other Guys" are inherently evil.

      Yeah, screw those other countries, China has a huge population, so they should just be able to steal territory that they have no valid claim to.

      Might makes right. Just ask any sovereign nation that's been subject to a US invasion. As an aside, note that no nuclear-armed state has suffered a regime change at US hands. And yet Americans are surprised when antagonists pursue nuclear arms? As for "stealing territory they have no valid claim to"....the validity of their claim stems from their ability to enforce their will. Hence the fortification of their man-made islands. Also note that the US has progressed to a uniquely insidious alternative to directly "stealing" territory: the Petrodollar system. But it requires constant enforcement by the US, and controlling/manipulating central banks, financial institutions, and the exchange of oil are all aspects of this enforcement.

      2000: Saddam was planning to switch sales of Iraqi oil from dollars to Euros. Within 3 years he was deposed.
      2009: Gaddafi was doing his best to reconcile with the West. Unfortunately for him, he also planned a gold-and-oil-backed Libyan currency. He was dead within 3 years of shaking Obama's hand. And the "rebels" sure were quick to set up a Central Bank (less than 2 months into the civil war).
      2012: Iran was planning to sell oil in exchange for gold. Despite having them bracketed with bases in both Afghanistan and Iraq, the US military was in no position to invade. So Iran instead found themselves promptly disconnected from global financial institutions: http://www.reuters.com/article...
      2014: Ukraine has a revolution....and suddenly all the gold is missing from their central bank. Now they are stuck with fiat currency and IMF obligations. http://www.zerohedge.com/news/... Meanwhile, Russia and China are buying up gold like crazy ( http://www.mining.com/china-ru... ), and started their own alternative-IMF (the Asian Infrastructure Investment Bank). Both have stated intentions to end the US's hegemonic influence. These are two nuclear-armed Great Powers that are closing the conventional military gap, and despite shaky economies, having been consistently moving to eliminate US dollar influence across their entire sphere of influence. Which, IMO, will eventually be a good thing for everyone, including the average (productive) American citizen.

      Oh, let us ignore all the people Assad was murdering, and that a large percentage of the population wants him out of office.

      If you have a problem with murderous heads of state that are unpopular, perhaps you should look a little closer to home before trying to solve other people's problems? https://theintercept.com/drone...

      Lets just prop up that dictator because he is our friend and is nice to us.

      Yes, the Russian relationship with Assad closely parallels the relationship the United States has with Saudi Arabia and the Gulf States. Ya know, the guys who are busy bombing the shit out of Yemen? These are also the same people who are VERY close ideologically to ISIS and al Qaeda....who we've spent the past 15 yea

  2. Re:Classic memory leak. by DamonHD · · Score: 4, Interesting

    Have you actually every tried writing a formal proof of correctness for any algorithm at all, let alone a non-trial one dependent on external subsystems and with huge amounts of state?

    Yes, I have tried, and raised funding, and managed in fact to run one layer of our formal modelling language in real time (slowly). But we decided that the proof languages (Z and ML, with a sprinkling of CCS) weren't up to the task, and nor were we.

    Rgds

    Damon

    --
    http://m.earth.org.uk/
  3. Re:Classic memory leak. by SeaFox · · Score: 3, Insightful

    It should have been caught in testing, but of course someone wanted to save money and then it's testing that gets shaved first.

    You mean like it says in TFS?

    The issue was spotted in late 2015, and thankfully, it was caught during the testing period.

  4. Only the radar system needs rebooting by wonkey_monkey · · Score: 4, Informative

    Software Bug in F-35 Radar Causes Mid-Flight System Reboot

    Alarmist headline.

    First of all, the bug doesn't cause a reboot. It requires a reboot to put the radar back into a useable state.

    Secondly, it is only the radar system that needs rebooting.

    --
    systemd is Roko's Basilisk.
    1. Re:Only the radar system needs rebooting by rikkards · · Score: 5, Insightful

      Third it's already been resolved. My dad worked on the Canadian Maritime Helicopter Project for years. During it the papers had an article talking about some snag that was hit and how horrible it was. I sent my dad the link and his response:
      "Yeah that was my thing and 8 months ago, it was pointed out and in less than a day resolved."

      Media exaggerating? Never...

  5. Re:Classic memory leak. by bloodhawk · · Score: 4, Insightful

    a formal proof for such a complex system is hideously, insanely ridiculously expensive. proper testing is sufficient. Yes a company could lose billions if their systems went offline as opposed to definitely spending billions to write verifiable software that will be out of date by the time they release it.

  6. Riddled with a single bug? by DrXym · · Score: 4, Insightful

    I'm sure it's a very serious bug but does it mean that the software is "riddled" with bugs? For all anyone knows it was an isolated issue that occurred in an atypical circumstance and was subsequently rectified. And it occurred during testing which is the reason that testing even exists as a thing - to find problems.

  7. "Hi, tech support?" by Snufu · · Score: 5, Funny

    "Yaeh, my jet is plummeting to earth at mach 3. Any suggestions?"

    "Have you tried turning off and on again?"

    1. Re:"Hi, tech support?" by DigiShaman · · Score: 3, Interesting

      777 or 747, can't remember which. Many years ago I took a flight from Chicago to Shanghai. Once we've boarded the plane, we sat there for a longer than usual amount of time. Eventually the pilot announced a delay due to "technical difficulties" and a technician was dispatched soon thereafter. After another 15 minutes or so, the pilot announced they had to reboot the system and we would be well on our way. Sure enough, all power was cut - the lights, seat displays, etc. I can't remember if they had the engines idling or not; either they were still running unaffected or they shut them down prior to the reboot. But the real fear I had was what happens if another reboot is needed 30k, feet in the air. I just hope the technician needed a hardware swap and not clearing an unknown unreproducible software glitch.

      --
      Life is not for the lazy.
  8. Reboot isn't a solution, that is a work around. by dsmatthews9379 · · Score: 4, Insightful

    A solution would be new code. It sounds like the test pilots are doing a great job of you know, testing.

  9. completely normal by at3matthewanderson · · Score: 3, Interesting

    This happens literally all the time with software updates on jets, anyone who's worked on any other generation fighter/attack aircraft in the "digital age" knows this. The interesting part, is that someone is publicly complaining about it, and making a software version with a bug, known to the public. Every radar system we've produced for 30 years has issues, again, this is completely normal. This article is about sounding some sort of political alarm, it shows that there's some dissent among the ranks, and I can assure you, that any experienced test pilot wouldn't even be remotely surprised to see this type of behavior, but the usual course of action is to document it and train the pilots in the short term, while releasing a new version of software in the long term. Pilots fly with "radar degrade" every single day. This is making a standard issue between contractors and military flight crews, into a public pentagon issue, to either ask congress for more money, rather than holding said contractors accountable for their failure to meet some sort of design goal, or to try to join the "anti F-35" team to advance his own career in some way. Our jets are flying way beyond their designed limitations right now, and the longer it takes to replace them, the worse off all of our military personnel are going to be.

  10. Version what? by ma++i+ude · · Score: 5, Funny

    The software version "3i" is affected.

    As a general rule, when your version numbering system needs to use complex numbers, something's going wrong with your project.

    --
    You can't shut us down! The Internet is about the free exchange and sale of other people's ideas!
  11. Re:Classic memory leak. by gtall · · Score: 4, Informative

    As others below have mentioned, it is very difficult to formally verify large complex systems. However, it is made even more complex in that there aren't enough research results to cover such a system in its complexity. Also, computer scientists tend to think the world revolves around their code, so if they get that correct, then the system will run correctly. The real world isn't like that, and it is not all captured in software, much of the system is hardware. Trying to capture the correct interaction between hardware and software is very, very hard...and it isn't clear that even if you could that you could verify the result before the universe dies.

  12. Re:F-35 is a "Little Turd" by PPGMD · · Score: 3, Informative

    I don't feel like watching the video, because based on the Defense Response it is the normal F-35 hachet job by journalists. I've heard the same arguments again and again.

    First off the F-35 was never meant to be a primary air supremacy fighter, it was meant to be a multi-role stealth strike thus the deficiencies in ACM are to be expected. It is sacrificing wing area to get an internal weapons bay large enough to fit a 2,000lb JDAM. Something that the F-22 can't do. It also has a much longer range than other fighter aircraft, literally 50% more on internal fuel than the F-16 with drop tanks, and the F-22 on internal fuel.

    Second though the specs don't seem impressive compared to the hot rod that is are the F-16, and F-15. But neither of those aircraft are going to reach their max speeds and altitudes with a war load. Those specs were tested with no armaments on the hardpoints and no drop tanks (with exception of the F-16 where wingtip missiles reduce wing flutter). While the F-35 going to be capable of nearly reaching its specs with an actual warload in the aircraft, and that includes super cruise. Granted the specs might be down rated somewhat from the initial contract specs, but that is to be expected as they often don't know the exact weights of all the third party systems to be installed on the aircraft (some were yet to be developed when the contract was written).

    Third the expected price of the aircraft is inline with the F-35 competitors, who are all non-stealth aircraft. The F-15SE is brought up as a replacement for the F-35 by some critics, saying it would be cheaper than the F-35. Well the F-15K which has a similar electronics suite as the F-35, cost the ROK $100M each. Compared to the current LRIP production cost of the F-35 at $90-100M each, with the full rate production cost to be in $90M. The Eurofighter cost just short $90M each. So the price for a stealth strike aircraft is actually inline with competitors.

    Fourth Canada made another stupid decision with canceling the F-35. They did so without selecting a replacement. The CF-18 nearly at the end of their service lives if they haven't reached it already. And likely will become another Sea King with the replacement used as political football between the ruling parties until if has killed enough aircrews that both parties agree "Perhaps we should actually replace these eh?"

  13. Re:Classic memory leak. by stevew · · Score: 3, Interesting

    The other detail missing here is that 3i isn't fielded yet. That is something like Block 2F which is only installed in the Marine Corp unit right now. Oh by the way - There is only one unit of Marine F-35s that are "on-duty" right now - the entire rest of the fleet is under test/development.

    --
    Have you compiled your kernel today??
  14. Re:Classic memory leak. by Quince+alPillan · · Score: 3, Interesting

    Extremely common, actually. It's one of the major pitfalls and difficulties of doing multi-threaded programming and one of the hardest things for programmers new to multi-threaded design to learn how to solve. It can also be extremely difficult to debug, even for experienced programmers.

    Improper garbage collection is another extremely common bug that becomes harder to find and debug with multi-threaded programming, and that can also lead to memory leaks.

    There are time tested techniques to mitigate these issues and strategies to find and squash the bugs, but as you said, they can be extremely hard to reproduce while testing.

  15. Re:Classic memory leak. by Waffle+Iron · · Score: 3, Interesting

    It's possible to write complex systems using modular design: do one thing and do it well.

    And you'll soon discover your simple modules start interacting in ways that you did not anticipate or understand. There are also unlikely to be any tools available to analyze how your set of modules work as a whole.

    Every nontrivial system has emergent behavior. You can't eliminate complexity with hand waving.

  16. Re:F-35 is a "Little Turd" by Lisias · · Score: 3, Interesting

    The F-104 was a fantastic jet. But a terrible military jet.

    That thing was made to be a fast, last minute, bomber interceptor. It was built to get there at Mach 2+, fill the bomber's ass with lead and get home. Originally, it hadn't provision even for missiles!

    But them, Pentagon changed the rules demanding a multi hole aircraft, and Lockheed started to hack the airframe. As a technical achievement, it was a formidable one. But again, as a military weapon, a questionable one.

    The best "worst" hack was the F-104G, made for Germany. They almost doubled the combat radius - but made the thing yet more harsh to handle. A lot of German women were made widows by this plane.

    Curiously, Italy was also an operator for this aircraft, but without a single recorded casualty (perhaps nobody managed to take it off! =P ).

    The bottom line I had read is: the F-104 is a formidable plane in the hands of formidable pilots. And a catastrophe waiting to happen in everybody else's.

    --
    Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org