Slashdot Mirror
In-Flight Reboot?
steelem writes "The Washington Post is running a story about how the F-22 Raptor's software requires in-flight reboots. Apparently the 2 million line software project is 93% done. Knowing most projects I've been on, it'll stay that way for another few years."
Welcome to Microsoft Airlines, your Stewardess today will be Steve Ballmer.
This is an ideal application for LinuxBIOS. The article says an average of 14 minutes per flight were spent rebooting computers. Even 36 seconds per reboot is too much, and would be totally unacceptable if it were say, a navigation computer on a 737 with a hundred civilians on-board.
Nasa has an interesting project called FlightLinux specifically geared for this sort of application. Unfortunately, they have yet to release code (export restrictions), but they supposedly use LinuxBIOS for their system.
Of course, having software that never crashes (no pun intended) would be best, but it never hurts to have a system that can boot up in just a couple seconds anyway.
when the contracting agency can't acocunt for $1 trillion? That's more than the rest of the world spent on their military last year. With that kind of accountability, I'm amazed any project gets over 80% done.
I'm much funnier now that I'm a subscriber.
damn, my job is so boring. I wish I was on the 'let's go kill people' software dev team.
The first hit on Google was this interesting take on the story.
Sheesh, evil *and* a jerk. -- Jade
Jeez, one would think that there would be built in redundancy so that if one system went down, it could be rebooted while the other system automatically takes over. Perhaps this is the way things are working, but the thought of rebooting during ACM makes me really nervous.
Visit Jonesblog and say hello.
Hi there soldier! You seem to have lost power to both engines secondary to a software malfunction, over hostile territory. Would you like me to help you reboot Windows?
Would it be too trollish to say this brings a whole new meaning to "The Blue Screen of Death"? Yeah, I thought so too.
Software like this should be able to reboot midflight without a hitch.
Flight control software has been rebootable on the fly since the earliest days of the space program.
If you're the test pilot you really got to hope they finished the code on the ejection seat at least, at 1,200 mph even a few seconds of reboot time is enough to turn you into part of the scenery at the test range.
"Now, admittedly, it's critical software. This is the 'let's go kill people' software."
Man, I need to get a new job.
this is a sig.
I've said it a hundred times and I will say it again. Software is getting way to complex for human management in developing bug-free code.
Life is not for the lazy.
Control: Destroy that incoming cruise missile. ETA 35 seconds.
Pilot: Got Radar Lock
Pilot: Hang on - just got to reboot. Will be ready in 36 seconds...
Please consider having Slashdot do a quick search, esp in the last 2-3 weeks. Even if this is done at the submittor level, then they could avoid this. I have no doubt that most submittors would prefer to avoid this. /., but more indicative of the problem that stories keep getting retold on the same news. Sad really.
Likewise, when viewing for submission, check the same search, so that you can see what the use saw
BTW, this is not really a problem with just
I prefer the "u" in honour as it seems to be missing these days.
What's funny is I always thought the guys writing this sort of software were uber-coders, and never had this sort of problem. Throw those few extra hundred million dollars at the coding effort, and I just thought this sort of problem went away. It's worrying though - isn't code which ever needed to be rebooted fundamentally flawed? Can you ever really fix that sort of code, or are we just waiting for the day whenever another edge test case comes along mid-flight, and an F-22 falls out of the sky? Even one of this sort of error seems like impending doom to me.
The software required to run the Raptor is insanely complicated. The plane itself was ambitious, but the contorl systems are the real innovation. Give these guys a break. The fact that the thing flies at all is amazing. The fact that it does everything it was designed to do is unbelievable. So there are a few bugs to work out. That's how it goes. We're not talking about "normal" programming problems here- this is Real Life stuff.
Good enough isn't. Stable code can be written. It merely takes talented engineers, design time to conceptualize and architech the product up front before coding it and giving QA what they need to test and committment to FIXING the issues that QA identifies.
I'm curious -- do you do development? Have you ever worked on a 2 million line program? No offense, but anyone who uses the word "merely" in a paragraph like that strikes me as someone with a tenuous grip on reality.
I am a senior engineer at a very big company. Applications I have written are in use by literally millions of people. And I'm scared stiff by the idea of writing the kind of software that powers the F-22. Software of this scale is the single most complicated project humanity has ever undertaken, and to belittle the efforts of the engineers involved by suggesting that they don't know what they're doing or aren't following responsible development guidelines shows a serious lack of understanding. I promise you, the software on the F-22 has been subjected to more rigorous QA than anything you or I have ever touched, but that still doesn't make it easy.
Humans aren't perfect, and as long as that continues to be the case, writing a multi-million line chunk of software will always be a ridiculously expensive and difficult proposition with no guarantee of success.
ZFS: because love is never having to say fsck
Second, I have seen this coming for about 10 years now. In the 70s and 80s I worked with digital control systems. Not avionics, but similar. In those days the systems were expected to work right, every time, for years at a time. 2 years between system restarts was considered "acceptable". If a system did fail, the manufacturer was expected to get its collective butt out to the site, figure out why, and issue a (solid!) fix pronto.
In the last 5 years, I have repeatedly been on brand-new airplanes at the gate when the pilot comes on and says "we are having a little problem with the system - don't be alarmed if the lights go off" followed by what is clearly a "reboot" of the airplane! When the fsk did it become acceptable to fix problems in avionics by rebooting the airplane?
And if the system designers really think the Microsoft Rebooting Disease is an acceptable way to handle system faults, how long before one of those faults occurs in the air?
I guess I am just old and crusty, expecting life-critical systems to work to spec 100.0% of the time.
sPh
Haven't read the article (typically of slashdot), but I do remember that the Apollo 11 computer nearly caused the first lunar landing to fail because it kept rebooting in-flight. Due to a configuration error that occurred shortly before flight, the computer repeatedly ran out of memory, but the software was designed so that the computer could reboot without catastrophe.
You can read more here.
Toronto-area transit rider? Rate your ride.
During WWI, pilots would signal the enemy if their machine guns jammed. Then it was considered the gentlemanly thing to do for the opponent to wait until the pilot had cleared the jam before resuming the dogfight.
I wonder if modern day pilots are going to need a way to signal their opponent that their computers are rebooting?
I am NOT a man!
I am a free number!
How about giving a whole new meaning to the term "three finger salute"?
Please help metamoderate.
I'm curious -- do you do development? Have you ever worked on a 2 million line program? No offense, but anyone who uses the word "merely" in a paragraph like that strikes me as someone with a tenuous grip on reality.
I think where people get thrown is that they see houses and cars and bridges and think, "If we can build those, why can't we build software? Programmers must be lazy"
Well, is every 2x4 in a house the exact same length? Are all the boards perfectly flush? A crooked door in a house will usually cause no problems, but the equivalent in a piece of software can cause a crash. Even computer hardware is never perfect. Does every 2.0 GHz processor run at EXACTLY 2.0 GHz? Not even close, but they are good enough. The problem with software is that it needs to be perfect to be perfect, and people aren't perfect.
The beauty of the F-22 system is that the developers realize this, and they designed the system knowing there would be flaws and that the software would crash. When some of the software crashes, the jet keeps right on going, which is the sign of ultimate stability.
Told to me by a pilot, I can't verify via a quick google.
this sig deleted by another sig
[_] Take off
[*] Land
[ok](cancel)
You must reboot your computer for the new settings to take effect...
I can already imagine the cockpit layout of a Raptor... Altimeter, speedometer, non-functional IFF indicator, roll indicator, yaw indicator, pitch indicator, three displays for tactical data, fuel indicator, HUD, control, alt, delete...
At least Windows would be fitting on an aircraft... It's easier to move a mouse cursor around with a joystick then to type "shutdown -r now" with it!
Hate me!
I've just re-re-read the article, and I can't find any mention that the software on board was Windows based.
Yes, you're all very droll, but the Microsoft bashing seems a little knee-jerk. It's insanely complicated to write software like this (as a few other posters have said, and I'm posting only because I have no mod points for them).
I doubt these errors are OS-based at all. Real-time systems like this are built on top of extremely well-tested embedded OSes. They reboot because they're writing pretty close to the bare metal, and mistakes are punished hard. Best practices are applied (interminable code reviews, fascist levels of regression testing, ungodly coding style standards), but not always followed, and even best practices don't always work.
I'd like to see a gradual shift to languages which enforce best practices (i.e. not C and assembly). Meantime, these pilots are pretty damn brave. But it's probably not Microsoft's fault, this time.
Go build me a pyramid. Without any modern machines. In the middle of the desert.
With ten thousand workers to help, a government that doesn't give a crap about death tolls or reasonable working conditions, and enough funding to bankrupt an empire, I'm sure I could manage.
The pyramids were gigantic, backbreaking undertakings, but I maintain my stance that software is the most complicated endeavor undertaken by mankind.
ZFS: because love is never having to say fsck
The F/A-22 does not need IFF with datalink and NCTR. Some USAF aircraft are not currently even equipped with IFF (the F-16 for example) and they have done quite well.
The APG-77 has a terrain following mode. And the widely spread weak emissions from it are much harder to detect than those from a conventional radar.
The Martin-Baker ACES II ejection seat can save a pilot's life from zero feet of altitude (that's why it's called a "zero-zero" ejection seat- effective down to zero altitude and zero speed)
Welcome to F22 Raptor version 3.1 (C)1990-2003 Microsoft Corp. Start Microsoft MiddleEast Explorer...Please Wait Target: Hussein, Saddam Located Would you like to: Copy/Delete/Return? Delete? Yes/Cancel Before you delete Hussein, Saddam, would you like to sign up for Microsoft .NET?
Trying to get a girlfriend to read /. is the most complicated endeavour undertaken by mankind.
Mode (3) smart-aleck mode. Press * to return to main menu.
Pilot: (Dialing microsoft support services while cruising at mach 50,000) Come on, pick up, pick up.
Pre-recorder message: We're sorry, all circuitys are busy now. Your call is very important to us, please stay on the line until an operator is availible.
Pilot: (Over enemy territory and ready to drop payload, toggling switches like a madman) Damnit, pick up.
Tech Support Person: Hi, This is Candice, how are you today. Pilot: (Engine failure light flashing) Can you can the chatter, I'm cruising over Eastern Kreblenkistan about to die at Mach 40,000.
Candice: There's no need to be rude sir. First I'll need to confirm that you're not using a pirated copy of our software, so will you please refer to the key sticker located on your computer. Pilot: (Frustrated, going down) I can't do that, I'm sort of in a plane right now, can you just tell me how to reboot the thing.
Candice: I'm sorry sir, but we can't be responsible for the failures of pirated software... (transmission ends, big fiery explosion)
I'm an advocate for a strong defense, and always have been. And advanced weapons programs always have major bugs. I'm a veteran, and I follow defense issure pretty closely. With that said, now I say kill the F-22 program.
Why? It's a problem program. It's been plagued with an abundance of serious unforseen engineering problems from the very beginning. This is just the latest one made public. Past problems have included repeated instances of various parts of the fuesalage (especially some wing and tailparts) cracking. Cost overruns have become endemic. When the ATF program (Advanced Tactical Fighter) was first launched in the mid-80's to find a successor to the legendary F-15 Eagle, the Air Force set a goal of a flyaway cost of no more than 35 million per copy. The cost is now up 200 million a copy, and before it goes into production, the F-22 might cost a quarter of a billion dollars FOR A SINGLE FIGHTER. No matter how rich a nation is, no Air Force in the world can afford to buy such fighters in effective quantities. Not even other Stealth projects have spiraled this far out of control. The F-117 NightHawk stealth fighter (really more of a small bomber), with a small inefficient production run of 64 aircraft, topped out at 61 million per copy.
Granted, not all of the cost overrun problems are the fault of the Air Force or of Lockheed Martin. Congress keeps screwing around with the production schedule, and reducing the total buy, which drives up the cost per aircraft. But Congress has done so in large part for three main reasons:
1- They ask "Do we really need this, or can upgraded F-15's do the job?" This is a valid question as no other nation, friend or foe, has an aircraft that equals the Eagle, save for Russia's SU-27 series of fighters. These have been produced in such small quantities that Congress still debates the need for an Eagle replacement.
2- The number and seriousness of technical problems has made Congress reluctant to commit to the project fully. This crosses party lines, as in the past few years, several powerful Republicans have tried to kill the program on the grounds that the Raptor is a lemon. Democrats seeking money for non-defense programs have joined them.
3- There are serious doubts emerging that the Raptor's massive complexity can ever truly be managed in an efficient manner. There are concerns that, even if the aircraft becomes operational and initial bugs are worked out, the aircraft will be unreliable, becoming what the Air Force calls a "Hangar Queen"; it looks pretty on the floor, but if it can't go up in the air regularly, how good is it? The Air Force has had aircraft before that they REALLY wanted, but turned out to be so expensive and maintenance intensive that they had to be retired early. And excellent example is the B-58 Hustler supersonic bomber, which had impressive performance...when it wasn't broken down. It was retired after only 10 years of frontline service.
Life is hard, and the world is cruel
Java F22: Pilot: Firing on target... Computer: "Starting Garbage Collector. Please Wait." Gentoo F22: Pilot: Firing on target... Computer: "Compiling Sidewinder Missile..." FreeBSD F22: Pilot: Firing on target... Computer: "Sidewinder Missile is dying..."
The vast majority of downed pilots, 80+% ?, never saw the attack coming. They were taken by surprise. The most successful aces avoided dogfights, they would try to surprise someone, if not they would disengage and look for someone else. Your account sounds like some romanticised story or an aberration that occurred in the earliest days of the war. WW1 pilots looked at battle the same way pilots do today. Give the other guy a chance and you may die, your wife a widow, your children fatherless.
By the time this thing ever gets into the air the only probable foes that it will ever face will be either SU-27 derivates or Mig-29 derivates, both of which cost far less than the F-22.
In pure features the Su-27 is an amazing plane. Anyone who has ever seen the Su-27 do the cobra manouver or the thrust vectored Su-30MKI or Su-35 do the 360 degree Kulbit manouver can attest to what these planes can do in close air combat. These are extreme manouvers that western planes cannot do for the simple reason that the engines in western planes receive no air at such high angles of attack and therefore often flame-out or stall. Not only this but the newer radars on the Su-30s and missiles are longer ranging than just about anything the west has with the exception of the F-14's AIM-54 Phoenix. As for stealth, newer Su-30's are coated with radar absorbant paint which reduce the advantages that a dedicated stealth fighter such as the F-22 would have in BVR combat.
In the hands of a good pilot I very much doubt that the Su-30 would automatically lose in combat. That however is the crux of the matter: Pilot training.
This has always been something that has been much better in the west with advanced simulators, top gun style combat training and long hours of aircraft experience. It is and has been a fallacy to believe that more modern high tech will always win the battle. It is almost always the quality of the pilots that decided the battle.
There is a good example of an air combat situation atht happened in the first gulf war. The only western plane to be shot down in air combat was an F-18 on an attack mission that was intercepted by an obviously experienced Iraqi Mig-25 pilot. The Mig-25 was already obsolete then in terms of technology but the sheer speed of the plane (Mach 2.8+) is unmatched by any other fighter. The Mig-25 went on after shooting down the F-18 to buzz an EF-111 raven that was providing ECM for the mission causing the raven to have to manouver to avoid the incoming missiles and drop back from the attack mission which was then unprotected by ECM and subsequently another F-18 was shot down by a SAM. No less than two F-15's and two F-16's all attempted to intercept the Mig-25, two of them firing missiles, but the Mig-25 used it's tremendous speed advantage to easily avoid the interceptors and reach its base.
This shows what a good plane , not necesserally the utterly most modern, can do in the hands of a good pilot. IMO the F-22 is an overexpensive white elephant.
Rather than the monolithic system which we all secretly love (which allegedly produces Blue Screens of Death when things go squiffy, although my own XP Home system has been thundering on with nary a problem for quite a while now), you build systems which can tolerate components restarting themselves. I don't care if you're RMS writing the purest code with GNU/Ada for the EFF Air Force, you're not going to write something that will never fail. Better to design and build an overall system which can tolerate minor interruptions, especially if you are going to be flying into a war zone.
In any case (I worked on some of the stuff on the fringes of the F22 program a long long time ago), there are a bunch of computers in the air vehicle; it's an airborne network. Saying "oh my god, I can't believe the plane is rebooting" is dissingenuous.(aside from the many Windows jokes). It's akin to "I had to power-cycle the printer twice today -- I can't believe the network stayed up for the 35 seconds it took the Lexmark to come back to life!".
Rebooting a subsystem computer works quite well in robotics too, which further leads into the concept of many small robots rather than one large beast screaming "Danger Will Robinson".
Cthulhu Barata Nikto
This was 1980.
It got fixed.
-dB
"It if was easy to do, we'd find someone cheaper than you to do it."
I used to work on avionics software and one of the biggest beefs of our main liason to the regulatory agencies was that there is currently no approved standard for generating system requirements. As a result there is no agreed-upon method for dealing with this single point of failure. In contrast, there is a well-defined and approved standard for software development: DO-178B.
This individual claimed that most of the mishaps she was aware of that were attributed to software were in fact due to faulty system requirements, and I have no reason to doubt her. Unfortunately I don't remember any specific cases that she cited.