Dublin Air Traffic Control Brought Down By Faulty NIC

Not so very long ago after passengers were left hanging by a similar glitch at LAX, Gilby4mPuck writes with another story of NIC failure leading to a disruption of air traffic, this time in Ireland, excerpting: "Data showing the location, height and speed of approaching planes disappeared from screens for 10 minutes each time. ... Thales ATM stated that in 10 similar air traffic control Centres worldwide with over 500,000 flight hours (50 years), this is the first time an incident of this type has been reported. ... '[They] confirmed the root cause of the hardware system malfunction as an intermittent malfunctioning network card which consequently overcame the built-in system redundancy,' said an IAA spokeswoman."

Comments

testing and QA

[hostyle]

Whatever happened to testing of installed hardware? You'd think they might csider that sort of thing important when it involves the lives of thousands of people. Then again, maybe they were drunk at the time.

Re:testing and QA

[Thanshin]

Testing doesn't confer prescience.

Re:testing and QA

[zach_d]

I think the issue is one of maintenance. things need to be replaced after their life-cycle is over, even if they seem to be functioning at the time.

Re:testing and QA

[MortenLJ]

The possiblity of failure can be reduced, but never completely removed. It's a simple matter of probabilities. E.g. a certain component fails on any day with probability p, if we add n redunndant fail-overs, the total system will fail with probability 1-p^n, an equation which will never be one, but it can get close.

Re:testing and QA

[tinkertim]

Whatever happened to testing of installed hardware? You'd think they might csider that sort of thing important when it involves the lives of thousands of people. Then again, maybe they were drunk at the time.

Well, when we set up some cheap NAS boxes with redundant nics .. some load balancers and other goodies .. we tested it by yanking cables on the bonded nics and making sure everything still worked.

This was for an e-commerce site.. I would agree in hoping more testing with real failures would be done on systems that monitor air traffic.

Also, we were very drunk when yanking cables during our test .. so I don't think intoxication is really a factor. In fact, turning a drunken monkey loose in a data center with a clearance to pull cables is _very_ good fail over testing :)

Re:testing and QA

[mcrbids]

The very best planned of redundant systems can be brought to its knees by hardware that "mostly works".

It's not hard to have system B check that system A is on/off line, and step in if the latter is the case. But what happens when A is *mostly* or *sorta* online? Does system B check that ALL functionality done by A is being done appropriately? Almost never.

And that's why, even in the best, most carefully designed, fully redundant high-availability systems, you never, ever see 100% uptime. It's just not possible to anticipate everything that can go wrong.

So design a system that fails gracefully! That's what nature did.

Take a look at your own body. It's a gorgeous example of a high-availability, high-redundancy system. There are literally BILLIONS of cells in your body, each operating as a semi-independent unit, such that any of them can fail without bringing down the whole, or even affecting it noticeably. Your body is an excellent example of a cheap, redundant, high-availability system.

Yet catastrophic failures still occur. Whether by cancer, diabetes, or heart disease, even a well-designed, tested-for-millions-of-years high-redundancy system with billions of individual, replaceable parts fails catastrophically from time to time.

It's the nature of the beast.

Mother nature has compensated by making not only the system redundant, but the need for the system also redundant. Rapid reproduction is nature's friend! Not just redundancy, but redundant redundancy.

High availability - it's much, much, MUCH harder than you thought.

Re:testing and QA

[boaworm]

Quite likely it did work at the time of FAT, SAT, Shadow operation and when going into live operation.

If it breaks down later on is another issue, that's not possible to test for beforehand. Isn't that pretty obvious? It is like testing a car to see if it will ever be in an accident. You sir, are the drunk one :)

Re:testing and QA

[Valehru]

I had an engineer stuck in Germany for three days due to this stupidity. He got his fill of beer, good hotel rooms and sightseeing done, so in his mind it was a decent holiday. The insane thing was that this issue happened before a few weeks earlier, there was an investigation however it did not discover then faulty NIC then either.

Re:testing and QA

[HungryHobo]

I'm inclined to trust the card which has been working fine for 5 years over a card which was put in yesterday.

Re:testing and QA

[kitgerrits]

The problem is not that redundancy wasn't implemented.
The problem is that redundancy doesn't handle 'flapping' hardware very well.
The NIC intermittently failed, causing the redundancy to switch cards several times.
This can play havoc on systems that work on a LAN and assume the MAC address to stay the same.
Also, a NIC that does not report an error, doesn't fail completely and simply swaps a few bits around can be nigh-on impossible to diagnose.

This could have been caught with real-time hardware and log-monitoring, but I have to confess even I only check the logs daily, not real-time. While some monitoring systems can mail the admin in the event of failure, not all systems are usually configured that way ('workstations' being a prime candidate).

There is a line you draw between monitoring and cost-effectiveness. Every company takes a claculated risk in this and they got bitten.

Re:testing and QA

[zach_d]

in a high noise/vibration/dust environment?

Re:testing and QA

[leuk_he]

If it works after 5 years. sure.

If not there always is a backup. isn't there? Well, in that case there is a backup of the backup.

Re:testing and QA

[Hal_Porter]

Only The Spice confers prescience.

Re:testing and QA

[DaedalusHKX]

The article says "it overcame the built in system redundancy"... how the hell does ONE failing card in a redundant setup "overcome" the redundant backup parts/systems ??

I call "CYA kissass excuse maker" to the stand!

Someone screwed up big, and they're Covering Their Asses now.

Re:testing and QA

[Hal_Porter]

This is very true. I worked on system where we had lots of redundancy in critical places. But given enough tests sometimes the bugs would get through, usually in ways that you hadn't thought of.

Re:testing and QA

[seifried]

Yes but if _one_ NIC can bring the entire system down what other single failures in a component could bring the entire system down? Obviously the system with the malfunctioning NIC can do any number of things that may result in a similar failure mode. Or what happens if the network switch it is attached to fails (I assume they use multiple paths... but if one nic can nuke it all, imagine if a switch went bonkers).

Re:testing and QA

[Anonymous Coward]

-- "The very best planned of redundant systems can be brought to its knees by hardware that "mostly works"."--

--NO, you are wrong there. What this indicates is that someone skimped. Techniques for processing and getting reliable signals through systems that only mostly work are very well known and used routinely. What this event means is that someone, either explicitly or implicitly assumed that NICs are binary - they either work or they don't, and designed accordingly.

What should have been used is multiple (more than two) parallel simultaneous communication paths with comparison voting at the far end to determine if the information received can be regarded as valid or not. Assuming that a NIC will fail gracefully is so boneheaded that in a safety critical application like this someone could likely be prosecuted for negligence.

Unfortunately, using the right techniques is expensive. Given that systems like this are provided by tender processes that favour low-bidders, it is not surprising that problems appear.

Of course, somebody may have done a cost-benefit analysis and decided the risk of one (or several) aircraft accidents didn't merit the extra expenditure. That's unlikely to be publicised 'though - although it would be a correct calculation to run.

As for testing - well running around pulling cables out at random doesn't really do it. Unplugging and plugging cables at various frequencies/intervals, swapping cables, plugging them into incorrect sockets, injecting noise, dropping the voltage on the power supply, overvoltage/spikes are all things that could and should be done - and in some cases mathematical formal proof that the system will work as required. All of this (and more) is done for safety critical applications.

Re:testing and QA

[tinkertim]

The problem is not that redundancy wasn't implemented. The problem is that redundancy doesn't handle 'flapping' hardware very well.
The NIC intermittently failed, causing the redundancy to switch cards several times.
This can play havoc on systems that work on a LAN and assume the MAC address to stay the same.

That's what got me curious, it looked like they were using takeover instead of bonding devices.

The most well engineered system in the world can not hope to escape a ~9 minute ARP cache upstream, which makes me wonder why it was designed the way that it was.

I'm not thinking in an antagonistic sense, I'm more wondering what changed in the network _after_ the system was deployed.

Re:testing and QA

[Hognoxious]

if we add n redunndant fail-overs, the total system will fail with probability 1-p^n

Any number raised to the power 0 is 1. So if you don't install anything, hence n is 0, it will always work since the probability of failure is 1-1 = 0.

Re:testing and QA

[TheThiefMaster]

If one fails with probability p, and you have n of them, a total system failure is probability p^n, not 1-p^n. Well technically it's Mult(p,1->n) where p1 is the probability of the first failing, p2 the probability of the second, etc, multiplying them all together to get the chance of a total system failure.
The probability of any one device in a redundant system failing is (1-((1-p)^n)). This equation rapidly approaches 1, so in larger setups failures will be a common occurrence, but they'll largely be harmless due to redundancy.

Of course this all assumes the failure mode of the device is "off" or "non-functioning". If it fails in a way which routes 15A of mains power into a network cable, redundancy might not help a whole lot.

Obviously that's not what happened, but it's not outside possibility for one device to take down an entire redundant system.

Re:testing and QA

[diskis]

Air traffice towers generally are not noisy or dusty. And in any case, disregarding the ports, the NIC card itself is practically eternal. Compared to the rest of the system, and the lifetime of the system that is.

Two lessons learned from years of technical support. The NIC isn't broken, unless the computer has been dragged from the network cable. And that the CPU is not broken as long as the system has not been overclocked, and the heatsink is still in place.

Re:testing and QA

[Phroggy]

The problem is, NICs can fail in all kinds of ways that yanking cables won't simulate. In this case it sounds like if they had yanked the cable, the backup system would have come online exactly like it was supposed to, but because the faulty NIC was kinda-sorta-almost-but-not-really working, it didn't. That's a difficult thing to test in the lab.

Re:testing and QA

[diskis]

> But what happens when A is *mostly* or *sorta* online?

You have dual NICs on system A, with a etherkiller connected to the second card.

When B takes over, it then can make sure that A stays down :)

Re:testing and QA

[mcrbids]

And this is where the "cheap" part of my comment "cheap, redundant, high-availability system" comes into play.

See, the likelihood of failure in a redundant system goes *up* as the number of units increases. But as the number of units in a redundant system increases, the likelihood of a *complete* failure drops to a number never equalling zero. In other words, no matter how much redundancy you build in, you'll never achieve zero downtime over the long haul.

The human body achieves zero downtime over a few decades in many cases, and close to 5 nines (%99.999) over 6-7 decades in most cases. This is very, very good uptime and is very noteworthy, but requires BILLIONS of redundant units and expensive external intervention (AKA the "hospital") to achieve.

You'll never get 100%. So get off it, already. Instead, prepare for the 1% to 0.1% of downtime and call it a day!

Re:testing and QA

[clickety6]

why stop with the drunken monkey...

http://www.the5thwave.com/gallery/comp_misc/677.html

Re:testing and QA

[Anonymous Coward]

Yes, because as soon as we get up in the morning before we even have breakfast we have a pint of Guinness and by the time we get to work we can barely even stand, never mind operate a radar system. Then again it might have been because we left the job up to the leprechauns who were to foucsed on protecting their crock of gold.

Re:testing and QA

[Thanshin]

Only The Spice confers prescience.

Actually, it confers "the ability to fold space. That is, travel to any part of universe without moving."

Well, at least they got the "not moving" part right.

Re:testing and QA

[Anonymous Coward]

if we add n redunndant fail-overs,

if we have n redundant fail-overs

the total system will fail with probability 1-p^n

the probability for a fail with n redundant fail-overs is

p^n

no?

Re:testing and QA

[xiox]

I always thought that CPUs could never be broken. We had an Athlon 64 processor 4600+, it was never overclocked and always used with a standard fan/heatsink, in a well ventilated case. After a year of work, it then started randomly crashing every few weeks. Replacing all the components except the CPU didn't fix the problem (different motherboard, memory, etc). Replacing the CPU did fix the problem. They can die randomly but it is very rare.

Re:testing and QA

[HJED]

Whatever happened to testing of installed hardware? You'd think they might csider that sort of thing important when it involves the lives of thousands of people. Then again, maybe they were drunk at the time.

Well, when we set up some cheap NAS boxes with redundant nics .. some load balancers and other goodies .. we tested it by yanking cables on the bonded nics and making sure everything still worked.

This was for an e-commerce site.. I would agree in hoping more testing with real failures would be done on systems that monitor air traffic.

Also, we were very drunk when yanking cables during our test .. so I don't think intoxication is really a factor. In fact, turning a drunken monkey loose in a data center with a clearance to pull cables is _very_ good fail over testing :)

it is point less to destroy a system testing it unless you have a big buget and can rebuild the system exactly the same
intoxcated monkey destroying data center != testing
intoxcated monkey destroying data center == (waste_of_money && destroying_data_center)
destroying_data_center == waste_of(time && money)
destorying_data_center != testing

Re:testing and QA

[mowall]

The article says "it overcame the built in system redundancy"... how the hell does ONE failing card in a redundant setup "overcome" the redundant backup parts/systems ??

I suspect it's because, as mentioned in the summary, it was "an intermittent malfunctioning network card". i.e. the failover system must have thought the card was functioning.

Re:testing and QA

[putaro]

That was in the movie. Read the book, it's much better.

Re:testing and QA

[jimicus]

Yes but if _one_ NIC can bring the entire system down what other single failures in a component could bring the entire system down? Obviously the system with the malfunctioning NIC can do any number of things that may result in a similar failure mode. Or what happens if the network switch it is attached to fails (I assume they use multiple paths... but if one nic can nuke it all, imagine if a switch went bonkers).

You don't need to bring the entire system down to cause havoc. What if there's a hitherto unknown bug in one of the CPUs which under some very specific set of circumstances causes aircraft altitude to be misreported on the operator's screen? As the GP said, most redundant systems only ensure that the components appear to be broadly working. They seldom check that all the components are doing something sensible.

Re:testing and QA

[david.given]

Actually, it confers "the ability to fold space. That is, travel to any part of universe without moving."

Actually actually, the space folding is done using the Holtzman drive, which is a perfectly ordinary machine. The Navigator merely navigates, plotting a safe path through the non-space/time foldspace. The spice grants the Navigator the limited prescience required to do this.

Eventually the Navigators become obsolete, replaced by Ixian semisentient machines known as Compilers that perform the same task without needing melange. A good thing too, because by that point Arrakis is rubble and sandworms are pretty much extinct.

Details courtesy of Wikipedia (and my lack of a social life).

Re:testing and QA

[TheSunborn]

Not according to the book. According to the book the spice is needed to predict what will happen when you arrive, that is: To ensure that you don't arrive inside a planet, or other dangerous place. The point is that when you do something that amount to traveling faster then light, the only way to know anything about where you arrive, is to predict the future.

This is also a big reason, that the guild newer took over Dune. They were so conditioned to always seek the safe path(Because that was what their ships needed) that they could not imagine starting an operation that was not know to be 100% safe for the guild.

(Only slightly offtopic)

Re:testing and QA

[kitgerrits]

Indeed.

I've seen Network Bonding in RedHat Enterprise Linux with HP hardware use a 'fake' MAC address that is bound to several interfaces to avoid just this problem.
Unfortunately, it may confuse the switch it is connected to, because of said ARP cache (CAM table, ours was 16 hours).

Really-HA systems require genuine engineers with tons of real-life experience, just to know what bits work and what bits you want to avoid.
I hope to become one, one day ;-)

Re:testing and QA

[somersault]

If you didn't apply the heatsink yourself, you don't know if it's been done correctly. On my first PC that I bought with my own money (well, half bought and my dad paid the rest), it kept locking up randomly, and after lots of IRQ and driver troubleshooting my dad removed the heatsink only to find that they hadn't applied it correctly. One reapplication of thermal paste and proper connection to the CPU later, and everything was fine (until that system got messed up in a lightning storm a few years later, but I still use the case when building my own machines)

Re:testing and QA

[somersault]

Sometimes, pure intuition can be more handy than maths.

Re:testing and QA

[Anonymous Coward]

My method is foolproof: I do processing with millionfold redundantly. I partition the nodes into segmented networks and when the results within a segment disagree (which is the usual case) I go with the plurality for that segment. The individual segments then get weighted scores proportionate to the number of units in that segment and finally the computation is performed by averaging the result of each segment weighted by the percentage of all nodes that are in that segment.

This computation is then compared (for historical reasons) with a separate result computed by nine "supreme" nodes, and finally I return the latter value, ignoring the former.

Foolproof, I tell you!

Re:testing and QA

[LiquidCoooled]

CPUs with correctly seated heatsinks which stay within their prime operating temperature usually have no problems.
However its rather easy to get the wrong amount of goop or something else wrong with the airflow, or just a marginal chip, etc

I had an AMD t'bird 1.4ghz which would NOT run happily at 1.4ghz no matter how much I tried.
In the end I gave up and ran it happily at 1.33 for years.

Re:testing and QA

[ebolaZaireRules]

I'm sure that its not merely the destination, but also the journey that needs to be known beforehand... 1 in 10 'disappeared', not ended up 1/2 parked inside of an asteroid.

Re:testing and QA

[xiox]

We did - we applied the heatsink several times, when we moved the CPU between different motherboards. Proper thermal transfer compound was used. The temperature of the CPU was fine.

Re:testing and QA

[methamorph]

The article says "it overcame the built in system redundancy"... how the hell does ONE failing card in a redundant setup "overcome" the redundant backup parts/systems ??

If the card had failed completely the redundant one would probably have kicked in. What I think happened is the card malfunctioned in a way causing the system to still think that the card is fine and there is no need for the redundant one to kick in.

Re:testing and QA

[witherstaff]

I recall some DEC NICS that when they started to fail, all got the same MAC. Talk about a fun thing to troubleshoot on a network! If it was a plain old switch using MAC switching, you can cause havoc pretty easily.

Re:testing and QA

[Anonymous Cowpat]

if you did apply it yourself, you can't be sure that it's been done correctly! On the first system I built; I put the heat sink on pi radians rotated from where it was supposed to be, and without thermal paste at all. (How was I supposed to know?)

Talk about expensive mistake.

Re:testing and QA

[dotancohen]

High availability - it's much, much, MUCH harder than you thought.

I would like nothing more that to be able to breed my way out of hardware failures.

"What? Another NIC failed? Honey, spread 'em!"

Re:testing and QA

[Z00L00K]

The big problem here was the intermittent function and that can happen anywhere in the lifecycle. Just replacing things won't help much and can cause a bigger problem.

What has to be improved is the redundancy system solution that has to be able to detect intermittent function and therefore do a complete failover.

And this isn't the first time this kind of problem have happened, and it's probably not the last time either. But in this case it was on a mission critical system.

Re:testing and QA

[colfer]

Lightning damage can come in through the NIC. On the network I saw, a Linksys router hooked to a satellite modem started sending damaging voltage down the ethernet after a storm. One computer lost its NIC, two others lost NIC + motherboard. After the storm was over, the Linksys box was still deadly to a laptop.

Re:testing and QA

[Hal_Porter]

One of the odd and very likable things about Dune is that there are occasionally implications that the society we read about is not the most advanced. Maybe their taboos are limiting them. Essentially the world we read about is actually in its own version of the Dark Ages where progress has all but stopped and feudalism is the only system. The Tleiaxu and the Ixians aren't in a Dark Age though. But we don't here too much about them because they are outside the known world because they violate the taboos that govern the know world.

Essentially it's a bit like reading history Taliban controlled Afghanistan, or unfortunately anywhere with an Islamic government. And I'm sure it's deliberate - Frank Herbert apparently was inspired by the Islamic uprisings against the British.

Or if you look at another way he wanted to write a hallucinogenic, retro sci fi epic, and he came up with a bunch of explanations - the Butlerian Jihad, the necessary for spice based prescience for interstallar travel, and the incompatibily between directed energy weapons and shields to explain why his universe was that way and not like conventional sci fi with ray guns, robots and open societies in the Popper sense.

Re:testing and QA

[vmcto]

Probably a cosmic ray.... http://www.scitech.ac.uk/PMC/PRel/STFC/Cosmic.aspx

Re:testing and QA

[Anonymous Coward]

Don't be a dick. You mentioned cheap specifically in the context of the human body, not in the overall context. An of course the likelihood of a component failure in a redundant system goes up as the component count increases: but we are talking about the system availability. Don't confuse the two.

You are right, however, in saying guaranteed 100% uptime is not achievable. Planning what to do when (not if) the system fails is always well worth doing. With luck, you'll never need the plans.

Re:testing and QA

[kaiidth]

Flew into Belfast instead last week because of this... when you're inconvenienced by technology it is very calming to know that what caused it was a real honest to goodness fuck-up, rather than a much less interesting case of human error :-)

Re:testing and QA

[Big Hairy Ian]

Hardware can still go down regardless of how often its tested. The issue here is it took 10 minutes to rectify the fault.

Re:testing and QA

[knightri]

If the NIC fails, the secondary NIC card will take over permanently. I see no reason why control would be given back to a card that has failed but now reads OK. At least thats how we do things in the power industry.

Re:testing and QA

[xalorous]

Yes, the books are better, and the spice does confer prescience in a small number of instances.

Re:testing and QA

[hostyle]

Data showing the location, height and speed of approaching planes disappeared from screens for 10 minutes each time.

The airport has been in chaos for the week or so since it started to happen. It took them that long to get back to full working capacity.

Re:testing and QA

[xalorous]

NIC's go bad. Fact of life. Equipment fails. This particular piece of equipment failed in a dramatic fashion that caused errors resulting in a denial of service to other network devices. The fix is monitoring at the switch level for these types of failures and disabling the port before the errors can cause cascading failures.

Do you run a full function check on every piece of hardware on your computer every time you turn it on? Or do you run it til it breaks then fix it? Also, consider how hard it is to trace this type of error. It is a partial failure, intermittent, and the failure brings down other machines. Any one of the machines that go down could be the culprit.

Kudos to the troubleshooters who found this beast!

Re:testing and QA

[Anonymous Coward]

faulty psu can usually cause cpu death.

Re:testing and QA

[Anonymous Coward]

This comment makes me wish multiple moderations were available. -1 Offtopic, +1 Funny

Re:testing and QA

[morgan_greywolf]

echo -e 'global _start \n _start: \n mov eax, 2 \n int 80h \n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a; a

Hmm....I wouldn't do that if you have 'ulimit' set to 'unlimited' number of processes allowed per user.

Re:testing and QA

[Dirtside]

So if you don't install anything, hence n is 0, it will always work since the probability of failure is 1-1 = 0.

Actually, it would be more accurate to say that it would never fail. ;)

Re:testing and QA

[ColdWetDog]

.echo -e 'global _start \n _start: \n mov eax, 2 \n int 80h \n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a; a

Hmm....I wouldn't do that if you have 'ulimit' set to 'unlimited' number of processes allowed per user.

You need to go OUTSIDE today, my man.

Re:testing and QA

[Anonymous Coward]

Indeed, to create a logic system B that can confirm _exactly_ whether or not a logic system A is functioning, one must encode completely the logic of system A into system B, thus creating a sort of recursive "who watches the watchers" type situation...

Re:testing and QA

[kilodelta]

Network Interface Cards and chips are probably the most failure prone components in a computer.

If you've been around networks long enough you've found jabbering nics, dead nics etc.

Re:testing and QA

[Ihmhi]

So putting in a faulty NIC card and seeing what happened wouldn't have done anything at all, huh?

Part of testing systems is trying to emulate what happens when a portion goes down.

Re:testing and QA

[ISoldat53]

Read Brain Herbert's books. They are prequels to the original Dune series and cover the history of most of the Dune races.

Re:testing and QA

[morgan_greywolf]

Yeah. Fortunately I just got back from going outside. OTOH, it was just raining, and I saw all the millions and millions of tiny water drops falling from the sky. Which made me think of Interrupt 80 and all those forked off-processes it would spawn with that code...

Re:testing and QA

[geminidomino]

Just have a solid supply of your own hallucinogen of choice if you plan to make it through them. Pee-ew

Re:testing and QA

[geminidomino]

On the first system I built; I put the heat sink on pi radians rotated from where it was supposed to be, and without thermal paste at all.

The word is "backwards," mate.

Re:testing and QA

[geminidomino]

intoxcated monkey destroying data center != testing
intoxcated monkey destroying data center == (waste_of_money && destroying_data_center)

destroying_data_center == waste_of(time && money)

destorying_data_center != testing

You forgot

intoxicated_monkey_destroying_data_center == really_fucking_funny

Re:testing and QA

[geminidomino]

Oh gods no... My bosses are highly fond of linksys for some reason.

My dick would fall off inside of 6 months. :'( I'm not a machine, dammit!

Re:testing and QA

[mr_3ntropy]

The problem is not that redundancy wasn't implemented.
[...]
This can play havoc on systems that work on a LAN and assume the MAC address to stay the same.

In other words, redundancy wasn't implemented.

Re:testing and QA

[skarphace]

So putting in a faulty NIC card and seeing what happened wouldn't have done anything at all, huh?

You keep a bunch of 'faulty' NICs around?

Re:testing and QA

[mpeg4codec]

Guess it really depends on where you are (or probably more poignantly, where you think you are) on the bathtub curve.

Re:testing and QA

[A440Hz]

Are you suggesting that NICs should reproduce?

Re:testing and QA

[bangwhistle]

If "redundancy" is clumsily implemented, then having multiple widgets doesn't help. Your monitor may think widget 1 is up when it really isn't. I've seen that problem in a common load balancing appliance.

Re:testing and QA

[seifried]

(Relatively) simple then, you have multiple systems, they vote on an answer, if someone is out they get voted off the island, you have another system with a different implementation also check to make sure they answer is sensible. Granted this is hideously expensive and probably only suitable for really expensive things like the space shuttle it is possible.

Re:testing and QA

[AnotherDaveB]

Frank Herbert apparently was inspired by the Islamic uprisings against the British.

Where did you hear that?

Re:testing and QA

[lgw]

Perhaps the fact that the leader of the Islamic updrisers was named "Maudi", or the fact that that uprising defeated a world-spanning empire with "desert power"? Or many other similar events?

Re:testing and QA

[lgw]

You can test kinda-sorta-almost-but-not-really working NIC by putting a jammer in-line with the test card. Systematically mess with bits important to every protocol layer intermittantly and look for potential issues, then focus there and do more tests.

It also helps to keep any bad NICs you fin around forever for this kind of testing, since they're far cheaper than jammers.

Re:testing and QA

[DaedalusHKX]

You would think on a NIC or some form of transmission interface, the status could be checked by simply pinging a known good host/point on the route the service normally takes... yep, and as soon as interface A doesn't provide adequate bandwidth, it is rerouted to B, or a better approach is to evenly share the load and to reroute failed/unconfirmed transmissions via a different interface than the last one used to try. After a certain amount of FAILED transmissions, a simple algorithm to compare how many fail on each NIC could simply update a VISIBLE reliability counter ina control panel available to any operator of the machine who could then say... HMMMM... packet A was resent 200 times through interface A and failed 70 times. Packet A was sent successfully each time through interface B. Packet B failed once or twice on interface B, failed over to interface A, failed once or twice, then went back to interface C and successfully sent through and confirmed.

There's a million samples of verbal flowcharting/pseudocode type stuff we did when I was in comp sci 101 and 102 back in college. Hell we sat around brainstorming server queue ideas for miniature servers to perform stupid little operations with no purpose. Call it "practice" since it was nothing but.

Anyways... it isn't that hard to handle in software as a means to combat subtle hardware failure, especially in a system that SHOULD be redundant and unbreakable. After all, they spent how many trillions on "anti terrorism" ?? And we had ONE plausible airborne terrorist attack in how many decades? Meanwhile we have HOW many fatal airliner wrecks in same decades? Perhaps the money is ill spent by self serving bureaucrats, but I could, of course, be mistaken. After all, we know self serving bureaucrats only have our best interests in mind. 6 figure salaries never figure into it... I'm sure.

Re:testing and QA

[kitgerrits]

It may have been implemented, even implemented correctly.
It may have signaled the problem and worked around it, the way it should.
The bug may also have show up because the software assumes there will never be a hardware failure, no matter how short.

Cisco switches implement redundancy by default with STP.
What they don't tell you that it may take over a minute before the network has fully recovered.
(that's why woy want to set up rapid STP for backbone switches and maybe tune the setting for userland).

Re:testing and QA

[kbahey]

I don't know about uprising against the British, but for sure, Herbert used many Arabic and Islamic themes in Dune. Some of the stuff is obscure historical terms, so he digged deeper than just current colloquial terms in use in the Middle East at the time.

Re:testing and QA

[Hognoxious]

It's like the old story that the more engines a plane has, the more likely it is to have an engine failure, so single engined planes are safest. It's not how many fail, what really matters is how many you have left working.

Re:testing and QA

[Ihmhi]

Give me two minutes and a soldering gun and I can make one.

The way the summary reads

[Centurix]

Makes it sound like the NIC was fighting against The Man! Go NIC!

There's only one way to solve this

[anomnomnomymous]

Put all those NIC's on the terror watchlist!

Re:There's only one way to solve this

[eclectro]

Put all those NIC's on the terror watchlist!

Why would anyone listen to you? Somebody who was just put on the terror watchlist by a bad NIC.

Re:There's only one way to solve this

[Anonymous Coward]

You joke, but we really should put the manufacturer on our personal list. It's a NIC! We should be able to rely on it. This is like paper turning black if you leave it in sunlight. Fuck them. Anyone know who the vender is?

Re:There's only one way to solve this

[Anonymous Coward]

*MICKs

redundant?

[Anonymous Coward]

if they were really smart they would have two separate machines dedicated to this information for more redundancy.

More scary stories.

[rixster_uk]

People - I am trying to collect airport related scary stories. I haven't got many yet but if you have some then please let me know - you can email me at admin@scareports.com or just visit the site (blatant pimping) here .

Re:More scary stories.

[DriedClexler]

Well, gee, I tried to email you the videofeed of every airport security line in the US, but the server rejected it for being too big.

Intermittent problems are the worst

[niks42]

I'd have to have some sympathy that it was an intermittent problem. They can really cause confusion to automated systems that are designed to cope with hard failures. I've had many occasions in my latter career in Service Delivery and support where it's taken human conviction to sort out issues caused by the cluster software trying to cope with intermittent connections

Re:Intermittent problems are the worst

[mikael]

In the early of office LAN's when there was just one big Ethernet cable with repeaters for a whole building, there were basically three ways a network card could fail:

o It just stopped working - no problem, just replace the card.

o It just kept jabbering ie. kept sending out random data or the same packet over and over again. This would jam the entire network segment. The only way to fix this was to detach each segment onto an alternative backbone until the rest of the office could get back to work.

o It just sent out the occasional runt - a small packet consisting of less than a header's worth of data packet. Most network cards would just ignore them. Others would keel over faster than a drunk parrot.

o Two cards had the same MAC address. I'm not sure how this happened, but one card just seemed to lose one bit of MAC address. Nobody really complained.

Re:Intermittent problems are the worst

[sjames]

That's why, in spite of all the marketing bullet points, often the best "failover" is manual. In case of failure, unplug this machine and plug that one in.

NICtzche

[cornjchob]

if this piece of hardware was capable of "overc[oming] the built-in system redundancy", perhaps its ilk ought to be patrolling the transistorized wunderplatz of interconnected morsels governing our most hubris means of transportation? I, for one, would certainly feel safer.

Well its a step above the old AppleTalk

[LM741N]

When I was administering a small network in Marin, every time we had a small earthquake, all of the AppleTalk connectors would come loose. Took hours to find the faults and push them together. I guess we should have used duct tape.

I suppose at an airport as each jet came in creating vibrations, those same connectors would have dislodged.

ten minutes

[Iamthecheese]

Ten minutes at a time? That doesn't sound like a "mostly broken" problem to me, that sounds like a 10 minute fail-over time. Shit happens, but if it takes you 10 minutes for your stuff to automatically start working again you're doing it wrong, especially since its all int one data center. And whatever hapened to redundant off-site systems? New law: As a conversation progresses, the chance of someone saying "terrorist" approaches 100%

Re:ten minutes

[wintermute000]

there are plenty of examples of 10 minute failover

Older cisco ATAs take 10 minutes to swing onto SRST if keepalives are lost to the callmanager cluster.

a complex routing protocol refresh (big BGP networks) can take many minutes

a faulty NIC can easily bring down a LAN segment, with or without redundant switching paths - and it makes it look like a router failure as the router overloads trying to deal with the broadcast storm

Re:ten minutes

[Anonymous Coward]

this doesn't make any sense.
protections at the switch level have been available for years.

a NIC can fail, leading it to spew thousands of garbage frames a second onto the network (broadcast, even).

or a NIC can fail by repeatedly flapping up/down; tying up switch resources.

any mid-to enterprise level switch most likely supports admin_disable of the port/interface based on configurable thresholds (e.g., 10 link flaps/minute), or CP/Rate-limits to auto-disable the interface if #broadcast packets/second threshold is exceeded.

how these protections were not put in place is astounding.

Re:ten minutes

[wintermute000]

Yeah but it might overload the local router before it exceeds the switch threshold.

Though most of the time I have seen this is with old/crappier models (26xx, 17xx), I've never seen this with 28xx or 18xx series before the switch err-disables the port.

Then again like 99% of workplaces, its probably 10 year old gear that just worked - until it borked. I'm on the network team of a fortune 500 and I swear there are sites larger than 10 people still using 10Meg HUBS that they are too cheap to replace. Heck our internet facing checkpoint Fws are 5 year old Nokia IP440s. No budget.

Re:Last time

[HungryHobo]

we don't have a prime minister and I'm fairly sure customs don't wear green.

Re:Last time

[JohnHegarty]

Any I am not entirely sure what a Lucky Charm is ...

It's a success story.

[Farmer Tim]

"...an intermittent malfunctioning network card which consequently overcame the built-in system redundancy"

But it's one of the lucky ones.

Every year, thousands of NICs fall victim to built-in system redundancy; if you know a card whose activity indicators are darkened and lifeless, it may have a redundancy problem. With your support and donations, we at Ethernetics Anonymous can help more network cards beat the scourge of built-in system redundancy, and make them feel like a useful part of society again.

Re:It's a success story.

[karnal]

Should have added:

"With your donation of only two bits per day"

Re:It's a success story.

[Farmer Tim]

I don't want to sound greedy, but we're trying to make packets.

My idea of fault tolerance

[Bromskloss]

in this case would be the ability to run air traffic control without all those fancy computrons, should the need arise.

Re:My idea of fault tolerance

[a_real_bast...]

Unfortunately, this NIC's fault showed up as the radar not working. What were they supposed to fail-over to? Binoculars?

Re:My idea of fault tolerance

[Bromskloss]

Unfortunately, this NIC's fault showed up as the radar not working. What were they supposed to fail-over to? Binoculars?

I suppose so, if it's possible to do it that way. Also, have the planes do the old-fashioned "circle the airport and keep an eye out for other traffic" if that works with big, heavy planes. It sure gives you (the pilot) a nice sense of being a free and sovereign person anyway, like on small airfields. :-)

Re:My idea of fault tolerance

[clickety6]

[quote]What were they supposed to fail-over to? Binoculars?[/quote]

And a giant relief model of the airport with young ladies pushing around little model aircraft with billiard cues. And a big glass panel with people marking up aircraft positions with wax crayons.

Re:My idea of fault tolerance

[a_real_bast...]

And gives the Comptroller a fit about the extra fuel expenditure? (",)

Re:My idea of fault tolerance

[zmollusc]

That is the stupidest plan ever, it is snooker cue _rests_ with which the ladies push the little model aircraft around.

Re:My idea of fault tolerance

[NoPantsJim]

(This is coming from a future air traffic controller)

You're forgetting a few things.

1. Not all air traffic control is done from airport towers. There are also TRACONs and ARTCCs, which is the type of facility you see in the movie Pushing Tin. Basically big dark buildings filled with radar screens and strung out people completely messed up on caffeine.

2. "circle the airport and watch for traffic" doesn't work for airplanes at FL350 doing 500+ knots. Usually that's IFR traffic, so the planes would have no chance of seeing each other anyway. Also, research has shown that at this speed, even on a cloudless day, two planes heading for each other would be unable to react and prevent a collision, no matter what.

3. If voice communication is going to be digitized as they plan to do with NextGen, a faulty NIC would silence all radios. The pilots would never even realize there had been a problem.

I wrote a research paper on NextGen and its faults this past Spring. The whole concept needs a serious overhaul if it's ever going to be safe.

Re:My idea of fault tolerance

[NoPantsJim]

Quick correction on an error. I said that anything over FL350 is 'usually' IFR traffic. What I meant to say was IFR conditions. Everything over 18,000ft is always IFR traffic.

Re:My idea of fault tolerance

[ddrichardson]

The default to scenario, where ever possible, would be to divert to other airports. This is perfectly viable in the UK given the relatively small distances between airports.

Re:My idea of fault tolerance

[Anonymous Coward]

I'm not familiar with this particular Air Traffic Control system vendor, but historically there have been various options used for fail-over when your radar goes wonky.

if its not an airport control tower ATC:
1) "shrimp boats" - commonly used before the introduction of radar-based (aka "positive control") ATC, the older green-screen ATC consoles seen in older movies were specially designed to be able to swing down from a vertical orientation to a horizontal one so that the controllers could revert to using "shrimp boats" when radar wasn't working (http://jurassicbark.blogspot.com/2008/05/fix-on-fail.html/).
2) redundant ATC systems complete with redundant feeds from the radar sites - the "DARC" with "RDP" referenced in the blog post at that URL is one such redundant system

for airport control tower ATCs:
1) you said it as a joke, but binoculars are (or at least used to be) an officially-planned fail-over method for use when the radars went out. Airport tower ATC is mostly visual-based control anyway (that's why they have/need all those panoramic windows) so its not as silly as you thought.

It really comes down to how many "nines" of availability Dublin ATC imposed as a requirement upon their ATC vendor. If their ATC operations were as disrupted by a single NIC failure as the article implies, they can't have asked for a highly available system but that doesn't mean they couldn't have had a fail-over ATC method available if they had wanted one.

In the queue

[davew]

I was due to fly the evening it all went wrong. Here's a lesson: if you're standing in a three-hour queue for the Ryanair desk, and they tell people to rebook on the web, and you take out a laptop and 3G modem, be prepared for a stampede.

Re:In the queue

[a_real_bast...]

You made two fatal mistakes:

1) You didn't do it where no-one could see you.
2) You flew Ryanair.

Re:In the queue

[Anonymous Coward]

3. you didn't display your premium usage rates

Re:In the queue

[bernywork]

On top of that..

You are flying ryan air, everything is an extra, I am suprised they don't charge to use the bathroom onboard (Having said that, they probably will now).

5 a person to rebook on your laptop, would have paid for a new laptop!

Re:In the queue

[Anonymous Coward]

Serves you right for flying Ryanair ( or Not So Easyjet for that matter )
Ryanair bumped me off a flight earlier this year. No warning except a refusal to let me board.
Ok they offered me a later flight but my almost 5 month pregnant partner was already on the flight and we missed our connecting flight.
They then called security when I kicked up a stink.
They Suck royally

Re:In the queue

[caluml]

You could make a pretty penny with that. £5 a shot, or whatever. Plus your keystroke logger would have tonnes of valid credit card information. :)

Re:In the queue

[heathen_01]

Did your laptop do you any good?

The Ryanair website was unusable (for me) during that time.

Re:In the queue

[davew]

1) I wasn't leaving that queue. :-)

2) The thing is - and believe me, I do shop around - none of the other options are very much better. :( The national flag carrier has remodelled itself to a low-cost airline and now matches ryanair feature for misfeature. BMI (Baby) are quite good where they fly, but to many other locations options seem to be very expensive, even accounting for ultimate cost including disasters like the above, or nonexistent.

Re:In the queue

[davew]

Unfortunately not. The flight I was on appeared not to be formally cancelled on the system, so I wasn't allowed to rebook at the time - and of course their system was jammers anyway. I tried to help out the people either side of me but was pretty unsuccessful.

Re:In the queue

[davew]

Actually, that's not true - it did do me good. I booked an Easyjet flight from Belfast for the next day. :-)

Re:In the queue

[a_real_bast...]

Aer Lingus has turned shit, yes - but you get marginally more legroom, still, and they aren't out to gouge you quite as hard. It's the difference between 5 turns of the thumbscrew and 4, I know, but it's still just that tiny bit nicer... plus it's not giving money to that tosser O'Leary, which is a bonus.

First time?

[Anonymous Coward]

âoeThales ATM stated that in 10 similar air traffic control Centres worldwide with over 500,000 flight hours (50 years), this is the first time an incident of this type has been reported.â

Is the LAX incident not of the same type, then?

Re:First time?

[Anonymous Coward]

Thales don't have any comparable ATC systems in the US.

One card "overcame the redundancy"???

[gweihir]

If they have good redundancy, they have two separate networks and two independent, preferrably different network cards, in all systems. Then they would do fail-over. Seems to me that if one card can bring this down, then the people that designed the redundancy screwed up badly.

Re:One card "overcame the redundancy"???

[jacquesm]

second that... sorry, I missed your post before I wrote mine. Whoever built the system goofed, and to screw up with flight control systems at this level should be grounds for termination and never ever to get work in mission critical systems again. There really is no room for error in systems like this.

I've worked a bit in the aerospace industry, specifically on software that would estimate the amount of fuel required for a flight taking into account alternative landing areas, winds and so on.

The amount of checking I did on that code bordered on the paranoid but I really could not live with some plane going down somewhere because of a stupid error in design.

Come to think of it, mission critical software should probably be open source, *always* so you can see what you're entrusting your life to and so that the 'many eyes' out there can point out the flaws. (assuming they're not eyes that will use that knowledge to bring down your system...).

Re:One card "overcame the redundancy"???

[gweihir]

Indeed. The open souce angle is also critical for fast and conclusive accident investigation.

Come to think of it, I have never worked on a really critical system, but I am in IT security, which shares the thinking about ways to break a system. One difference is that our "malfunctions" are intelligent and malicious. On the other hand, they typically cannot kill large numbers of people. I think I prefer that. Having software out there that can kill, would probably give me bad dreams....

Re:One card "overcame the redundancy"???

[Anonymous Coward]

no, ideally they have 4 cards, two bonded pairs on different networks.

Re:One card "overcame the redundancy"???

[Anonymous Coward]

Come come my good Negro, doest thou not know of "network teaming"?

Not to be corn-fused with "network teeming" of course.

Such a simplistic solution might well appeal to the simple-minded PHBs sure to be spearheading systems services in this sector.

Re:One card "overcame the redundancy"???

[Phroggy]

If they have good redundancy, they have two separate networks and two independent, preferrably different network cards, in all systems. Then they would do fail-over. Seems to me that if one card can bring this down, then the people that designed the redundancy screwed up badly.

It sounds to me like they DID have two separate networks. A faulty NIC was able to overcome that setup, by tricking the fail-over system into thinking that it didn't need to switch to the backup network.

Re:One card "overcame the redundancy"???

[Anonymous Coward]

It's all well and good to have redundancy. However many vendors will sell a redundant system but after the sale no monitoring is done to ensure that when one of the components goes off-line, it's immediately replaced.

Without monitoring & maintenance a system is only redundant until the first component fails.

I see this time & again with customers using RAID systems - they purchase off-the-shelf systems, don't do any training & possibly just don't have the tools for managing the device. The first they know there's a failure with any of this disks is when the 2nd or 3rd drive fails and takes the whole set out.

Was it running windows?

[iwein]

http://www.networkworld.com/community/node/29644?page=2&ts=

Re:Last time

[Anonymous Coward]

we don't have a prime minister and I'm fairly sure customs don't wear green.

Lordy, someone's lost the craic! Quick get this man a pint of the black stuff and a plate of annoying stereotype! /me dances a jig to up-beat folk music.

More seriously: had I said Taoiseach the joke would have flown over everyones heads. Secondly taoiseach is just Irish for prime minister, so get off your high horse. Thirdly, I can exploit Irish stereotypes for a cheap joke, because I am Irish. Personally my favourite examples of Irish stereotypes are the Monkey Dust sketches: Diary of Anne Frank and The Crusades, always worth a plug.

Why!?

[damburger]

I am flying to Florida tomorrow, it will only be my fifth plane flight in total and my first transatlantic flight. Despite being a rational scientist, who knows how safe it is statistically, I am having trouble suppressing my anxiety.

And at this point, fate sees fit to bombard me with horror stories about flying. This news about air traffic control comes on the heels of a headline I just saw on the front page of the Independent about pilots not reporting faults on aircraft and thus unsafe ones still flying about. I can't remember the exact wording because my brain parsed it as "TOMORROW YOU WILL DIE IN FLAMES"

Re:Why!?

[FrostedWheat]

A long time ago I went on a school trip to London, and it was the first time I had ever been on a plane so I was a bit nervous. In the airport shop there was a magazine (can't remember which now) with a plane in flames on the front cover, with the large headline "Why Planes Crash". Whoever put them out must have had an evil streak too, they had spread them out to fill the entire top shelf.

Re:Why!?

[damburger]

Damn thats cold

Your signature, however, gives me something else to focus on. Fucking software patents! Idiotic corporate pandering EU! Grrrrrr! I'm not afraid of flying, I'm angry about IP abuse!

Re:Why!?

[Anonymous Coward]

When I took a trip over to the Emerald Isle last, the night before we were due to fly over to Blighty I made everyone in the party watch a Discovery Channel programme on Helios Flight 522.

Needless to say, anxiety has never been an issue for me.

I got away with it by pointing out that there was no conceivable way in which watching a TV show could alter the probabilities of the plane dropping out of the sky the next day.

Re:Why!?

[damburger]

Depends, was the pilot at your house?

Re:Why!?

[Anonymous Coward]

Then I guess this won't help very much :)

See, this is why people are stupid, because all you see is this. Also, I've noticed that page contains too many "terrorists" words, really. Yes, there are bad things happening but those are the only ones that hit the news, the good stories where you were asked to go through the X-ray machine and you were just scared of X-rays so they let you pass don't make the news or they make the news as "guy walks through airport without being scanned! terrorists could do that."

Just ignore this /. story and move along because good things have always happened and will happen to you in the future and feeling good is all you should expect from flying. While you're up there, in stead of thinking of what could happen (and probably never will), think of what is really happening in that moment: you're flying!

Thanks for flying with us, have a nice trip ;)

Re:Why!?

[Anonymous Coward]

That's one small step for lawyers, one giant leap for software patents.

Re:Why!?

[dotancohen]

I can't remember the exact wording because my brain parsed it as "TOMORROW YOU WILL DIE IN FLAMES"

Just to reassure you, you are far more likely to die from the impact, or failing that smoke inhalation, or failing that drowning. The passenger compartment is rather well insulated from flame.

Re:Why!?

[adamofgreyskull]

--EXT: PLANE FLYING OVERHEAD
--INT: PLANE COCKPIT
PILOT #1: Oh wow, I really hope we don't have a crash.
PILOT #2: Me too.
PILOT #1: But they say it's safer than crossing the road!
PILOT #2: Yes, but we have to do that too.
PILOT #1: Best not to think about it.

Re:Why!?

[TheGratefulNet]

"TOMORROW YOU WILL DIE IN FLAMES"

its not the flames that kill you, its the long long LONG fall.

but don't think of it as an end; think of it as a really effective way to cut down on your living expenses.

Re:Why!?

[ddrichardson]

they had spread them out to fill the entire top shelf.

Top shelves aren't what they used to be.

Re:Why!?

[gad_zuki!]

>I am having trouble suppressing my anxiety.

What I do is I look at the flight crew and pilot and think "These people have been on hundreds if not thousands of flights each and they are still alive and uninjured. I'm just a beginner compared to them." That kinda kills the drama right there.

Re:Why!?

[Anonymous Coward]

There's one rather crucial point you're missing out on here... Air Traffic Control was "down" for 10 minute increments... guess what did NOT happen? Nobody was injured, no planes crashed.

What I take away from this is that even small system glitches in Air Traffic Control can be smoothly overcome.

As a rational scientist, shouldn't this make you even more comfortable?

Re:Why!?

[damburger]

Such things make me, the rational scientist, much more comfortable. However, they have little effected on the parts of my brain I inherited from flighty little rodents.

Re:Why!?

[LWATCDR]

Don't worry. I have flown a lot and nothing terrible has ever happened.
I don't know where you are flying into but let's say it is Miami. If the ATC for Miami airport goes down they can have you divert to Fort Lauderdale West Palm, Tampa or Orlando. If the ACT center goes down Patrick AFB, McDill AFB, Cape Canaveral AFS, or Key West NAS could take over using military systems.
If the weather is clear which it often is in Florida then it really isn't a big issue at all. If the weather is terrible there are alternate airports and Military bases you could divert to.
Rotten time to be coming to Florida. It is hot and humid here this time of year. Winter is when we have the nice weather.

Re:Why!?

[LWATCDR]

Funny but I have no fear of flying at all. I wonder if that has to do with the fact that the first flight I remember was with my dad in Piper PA-128-180. A little four seat puddle jumper.
I hope you have a nice stay here. It is hot and humid. Kennedy Space Center is a must see.

Re:Why!?

[gacl]

http://www.newsday.com/services/newspaper/printedition/friday/news/ny-bzair185767879jul18,0,1384201.story

Hope you feel better!

Re:Why!?

[damburger]

Seeing KSC first day after we land :)

I'm hoping to catch a glimpse of the giant finger they use to check the space shuttles centre of mass

Re:Why!?

[Anonymous Coward]

On eof my last flights, which went well, came with an in-flight video. The flight was about seven hours and the video looped two or three times during the trip. The whole thing was the physics and analysis of a recent plane crash. Not exactly a good thing to have playing while you're trying to nap.

Re:Why!?

[LWATCDR]

Well if you have your wife with you and your yungish I would suggest a trip to Ron Jons surf shop. It is the worlds largest surf shop and is right on Cocoa Beach, my wife loves it.
Also they have nice beaches around there as well.
I grew up south of there in Vero Beach FL. My family used to run up to the Merit Island mall twice a year. Back to school shopping and Christmas shopping. It also had the first Arcade I ever saw. I played Lunar Lander there:) Yes I am old.
My guess you are going to land in Orlando. I just had some people from the UK here for training. They where shocked by the distances here. Orlando is about a two hour drive from KSC depending where you are in Orlando. Try to get to KSC early since we often have afternoon thunder showers this time of year. In fact is pooring right now.

Re:Why!?

[damburger]

Sadly we aren't likely to be able to get to the beaches (although apparently my fiancees parents have arranged some form of dolphin-related activity for us).

Re:Why!?

[LWATCDR]

My sister took her little girl to that if it is the one in Orlando.
They liked it.
Too bad really Orlando is fun but it really doesn't give you much of the feeling of what the US or Florida is really like. A lot of it is just a giant tourist trap.
But just like Vegas if you have money you will never be board.
Oh when you got to KSC make sure you go to the Saturn V exhibit.

Re:Why!?

[Anonymous Coward]

so, did you die?

Re:Last time

[Anonymous Coward]

Customs do wear green, but it is hard to prove since the camera does not manage to capture them.

Beside that, it is not a area where photograping is allowed. If you do so anyway the X-ray machine will wipe your camera (It is not supposed to do that, but it will do anyway!)

that redundancy

[Anonymous Coward]

was not 'built in' properly, the system should have been isolated after the first fault and not put back into service until the fault was diagnosed and fixed.

pretty sloppy if you ask me...

Truth?

[Fri13]

"[They] confirmed the root cause of the hardware system malfunction as an intermittent malfunctioning network card which consequently overcame the built-in system redundancy,' said an IAA spokeswoman."

And when we edit littlebit, can we have the truth?:

They confirmed the root caused the hardware system malfunction using an intermittent malfunctioning network card wich consequently overcame the build-in system redundancy.

Re:Truth?

[ddrichardson]

The "overcoming" redundancy thing is unusual and I wonder if its linked to the specified time between failures. If the self test in the system operated at a periodicity as large as ten minutes it could miss such intermittent faults. This is why bus systems with self testing bus controllers are normally used in these circumstances. Either way, it shouldn't happen.

Confusing terminology

[ddrichardson]

I work in aviation and wonder if the terminology being used by the newspaper articles is correct.

It appears to be talking about mode S IFF (Interrogation Friend or Foe) or SIFF radar systems which identify aircraft and appends height data. The speed is the only thing that needs calculating, as it isn't encoded in the pulse train.

Why this is weird is because much older bus technologies are normally used to handle this data being transferred than current network technology, such as MIL-STD-1553.

This makes me wonder if it was one of two things - a system inputing to an ethernet PC system that calculates and displays the information or more likely they are talking about a DLTU type stub connector (or remote terminal) used in such typical buses. This is unlikely because the bus systems they are employed on, the bus controller would have picked up on the failure during continuous built in test and pulled in an alternative.

If its the former then someone needs shooting. ATC is a realtime application and the overhead involved here would be unacceptable. I'm not even sure of the benefit of a network, multiple self contained indiviual terminals would be safer.

Re:Confusing terminology

[ledow]

A quick google turns up:

http://en.wikipedia.org/wiki/Avionics_Full-Duplex_Switched_Ethernet

Which suggests that Ethernet-derived products are, indeed, used in critical systems (although this seems to be on-aircraft rather than in ATC). It (apparently) has seen wide deployment on common "famous" aircraft.

And the UK has been "upgrading" its air traffic control for years and years - so much so that they now appear to be nothing more than an office with some multi-head display if the footage shown on news-reports of a year or so ago are to be believed. It's concievable that this is truer than you would think.

However, I bow down to your knowledge as I know nothing about aviation at all.

Re:Confusing terminology

[ddrichardson]

While you're right, the key phrase from the article you give is:

ARINC 664 Specification which defines how Commercial Off-the-Shelf networking components will be used for future generation Aircraft Data Networks (ADN).

Specifically, this standard is aimed at use on aircraft not in ATC, in fact because of the weight reduction it offers.

Also not to split hairs but Dublin is not in the UK, this seems trite but is valid as there are different agencies involved. More over, the appropriation of new technologies is obsessive in the UK at present and has been for some time (except in the financial sector). There is a perception that newer is better and that answers to questions nobody asked are best solved by combining off the shelf components in a similar topology to older generation systems.

There is an argument to upgrade ATC due to higher volumes of aircraft but I can't help wonder if there is a bigger drive towards efficiency rather than safety.

Re:Confusing terminology

[Anonymous Coward]

I work in aviation and wonder if the terminology being used by the newspaper articles is correct.

If the news media are talking about technology, you can pretty much guarantee that there's a mistake in the details, if not the "big picture".

 

/former broadcast engineer, who regularly shakes his head at what goes out over the air.

annoying..

[Anonymous Coward]

This is the first time in 50 years that this has happenend. And the first time they had accurate information on screens was 10-15 years ago..

Re:Last time

[Anonymous Coward]

Outside ireland, lucky charms is an american-style (sickly sweet and artificial, featuring sort of freeze-dried marshmallow lumps) breakfast cereal, with a hollywood-irish-accented mascot "lucky the leprechaun" (akin to "tony the tiger" from "frosties"), with a surrounding advertising campaign that could be considered vaguely offensive (on grounds of nauseating cutesyness if nothing else), at least if irish people were excessively thin-skinned (fortunately, they're generally not, and since they're also white-skinned people in america probably wouldn't care if they were upset anyway). It's not hugely offensive, it's not "Bloody Sunday Breakfast Snacks" or something, but all the same, it's not sold in the Republic of Ireland, and was withdrawn from the UK market fairly rapidly upon introduction.

As part of that campaign, the leprechaun obsessively worries about people trying to take his lucky charms, or at least used to, now he just seems to be resigned to thieving kids running off with it.

Irish Examiner, ha!

[PinkyDead]

Everyone in Ireland knows that the Irish Examiner used to be the Cork examiner - and they never miss an opportunity to point out how Dublin is doing a bad job.

This is because Cork thinks that it's the centre of the friggin' universe. The 'Real Capital', my arse! Just a bunch of thunderin' ejits, living in their little Blarney fantasy land. Sure they can't even talk right. What the hell is a 'langer', anyway. They wouldn't even know how to spell NIC.

The fact that they are right is quite beside the point.

(For a North American cultural equivalent, please see http://en.wikipedia.org/wiki/South_Park:_Bigger%2C_Longer_%26_Uncut)

Anyone who mods me down is from Cork - believe it!

Re:Irish Examiner, ha!

[Anonymous Cowpat]

but Cork is the only bit of Ireland that will still float if the country falls into the sea!

Re:Irish Examiner, ha!

[cwgatling]

Indeed. A well-balanced Cork man has a chip on both shoulders.

No system, no failure

[duyn]

G-GP:

if we add n redunndant[sic] fail-overs, the total system will fail with probability 1-p^n

GP:

Any number raised to the power 0 is 1. So if you don't install anything, hence n is 0, it will always work since the probability of failure is 1-1 = 0.

P:

Sometimes, pure intuition can be more handy than maths.

Only if you're not good at the math.

The way the G-GP described the system, the number of redundant fail-overs includes the primary system. With n=0, you have no system in place. No system, no possibility of system failure.

Re:No system, no failure

[somersault]

No system, no possibility of system failure.

That was my point.

Re:No system, no failure

[Hognoxious]

And mine!

Zing Zang Zoom

[worldcitizen]

It's a cover-up for Zing Zang Zoom rolling out a rootkit protection

Speaking as a sysadmin of just such a network...

[Interfacer]

...it is not so black and white.

I administer a network like that. Pharmaceutical plant to be precise.
All machines on the production network have 2 independent PCI nics, connecting to 2 identical but separate networks, using separate routers and switches. The critical servers are stratus high availability servers which have dual redundant everything, driving all components in lock-step and correcting errors on the fly.

If something happens to cause a network switch over, there is a bulk of network traffic to deal with it, because sockets have to be opened and closed, state has to be transferred, system control message flow has to be restarted so that all controllers go back to the normal state, ... And at application level, everything is RPC and DCOM based, so this will cause a significant disruption for the running services, since COM objects and RPC marshalling have to be destroyed and recreated, reinitialized, ...

The whole thing is very complex from a systemwide point of view, and causes a significant disruption.

Now, switchovers can be triggered by different things, like a maintenance techician replacing a controller and causing a timeout halfway through a message flow, network cable that has to be unplugged for some reason, ...
When that happens, performance of the system sucks for the next couple of minutes.
Critical signals will work within defined tolerances, but anything else will simply stop responding during the switchover.

I can perfectly imagine that IF you have a nic which is failing in just such a way that makes the network switch back and forth, it'll dirupt and eventually kill the entire network.
Unless of course the switches are smart enough to detect this and disable the physical port. But even then, we are not talking about DDoSing, but just minor errors at exactly the right time to trigger network failover.

Network redundancy of complex systems (and air traffic is much more complex than a pharma plant) can protect very well against single or predictable failures.
But there will always be very specific failures, depending on dozens of variables, which have the power to bring down the system. Since is the first time such an event occurred with that air traffic control system, it is likely to be one of those corner cases.

It's the same with other proecedures with aviation. everything is supposed to be double and triple redundant, but still National Geographic has enough material to create the series 'Seconds from disaster' where one unlikely error amplifies another, and in the end, the plane hits the building / swamp / ground.

Re:Last time

[Bloke down the pub]

More seriously: had I said Taoiseach the joke would have flown over everyones heads.

I'll have you know, my good man, that not all of slashdot's readers are American.

But what is a "contol"?

[rbanffy]

What is a "contol" and why is this so important?

Re:Speaking as a sysadmin of just such a network..

[bepe86]

Ever heard of a nice thing called Spanning Tree Protocol and the Campus Model for network design? With proper network design, this would not be an issue.

Blame it on the sales guy

[Midnight Thunder]

I think this is a case of Sales guy vs web dude.

Bloody Sunday Breakfast Snacks

[Anonymous Coward]

I prefer Bobby Sands Diet Bars myself.

Success vs failure

[sjbe]

No system, no possibility of system failure.

No possibility of success either. Brilliant! Let's all just go back to bed and forget about trying anything that might possibly fail...

Re:Speaking as a sysadmin of just such a network..

[gweihir]

If something happens to cause a network switch over, there is a bulk of network traffic to deal with it, because sockets have to be opened and closed, state has to be transferred, system control message flow has to be restarted so that all controllers go back to the normal state, ... And at application level, everything is RPC and DCOM based, so this will cause a significant disruption for the running services, since COM objects and RPC marshalling have to be destroyed and recreated, reinitialized, ...

That strikes me as the wrong approach. Use the two physical NICs as interfaces of a router and route everything from/to a logical NIC. Then there is no failover disruption. I think that today there is no need to do this so that the applications actually notice or (worse) need to be able to deal this type of failure.

I can perfectly imagine that IF you have a nic which is failing in just such a way that makes the network switch back and forth, it'll dirupt and eventually kill the entire network.

The well established way to deal with that is lockout periods, where only a failure of the second network and a re-availability of first will cause a switch-back before a certain time has passed. Note that this is needed regardless of switchover delay, since the system can ''oscillate'' faster for faster switchovers.

I see three possible problems: Either this system was not state-of-the art and has serious issues simply because of its old and unflexible deaign, or the designers screwed up, or it was indeed some very complex failure that had several unexpected sideeffects. Avionic people are vey good in learning from past desasters. Their history of prevention and prediction is not nearly as good.

Oops!

[cashman73]

Sounds like they spilled Guinness on the servers again! Either that, or it's those damned pesky Leprechauns! :-)

Re:testing and QA: and dynamic behavior

[pruneau]

It's hard enough designing redundant system, but when you have an intermittent failure, the complexity steps up a bit, because redundant system design generally involves people thinking along the lines: ok, it this fails, how best can the system react ? But when people think "If this fails", they practically always assume "if this fails, _and_stay_that_way_".

That's why intermittent failures are so bad, because they introduce a dynamic element into an already very complicated equation, and usual testing strategies don't go as far as this.

For example, just try to see how any routing algorithm reacts to a router that's up and down, up and down, up and down...

My first flight and a book from the wife

[Dareth]

I was on my first flight on an airplane and was excited and a bit nervous. I decide to read the book my wife gave me at the last minute for some reading material. The first line was, "I was going down fast..." regarding a crashing plane. My wife claims she had no idea of course. After the fact, I can appreciate the irony at least.

How we do it

[Anonymous Coward]

I work in a company that makes stuff for ATC. Our systems have 2 networks (2 NICs in each box, 2 sets of cables, switches etc)

Every packet is sent on both networks and alarms are set off if a packet turns up somewhere without being received on the other network.

Iffy connectors etc. are discovered pretty quickly, but it is still be possible for the system to fail if, for example, a switch on the 'red' network failed and a box had a dodgy NIC on the 'green' network -- this kind of thing happens when somebody chooses to ignore an alarm because "it's working fine, we'll get round to replacing that card next week"

Re:Last time

[geminidomino]

Lucky Charms never pissed me off so much as Trix did. I remember one commercial where the rabbit actually *bought his own cereal* and the kids took it because "trix are for kids". I wanted to see him mow the little fuckers down for home invasion or something...

Silly rabbit, supersonic lead is for thieving, speciesist little pricks.

Network Access Control to the rescue?

[TheSync]

I swear I've heard about something like this happening before.

Can't a Network Access Control (NAC) enabled switch with some kind of "reasonable NIC operation metrics" shut down the port the bad NIC card is on? (While notifying the network admin, of course)

Looks like they caught the error...

[Anonymous Coward]

...just in the NIC of time.

you don't understand what "open source" means

[Anonymous Coward]

"Open source" is not equivalent to "source code is available to the end-user for inspection" even though that is one of the consequences of making software open source.

"Open source" means that the author places little-to-no restrictions on what the end-user may do with the software.

There are many, MANY other ways to supply a software's source code to its end-users including under contracts which explicitly places heavy restrictions on what an end-user may do with the supplied source code. I've received (from Sun) the source code to Solaris 2.6, but that was certainly never open source!

If you'd developed actual flight control systems (or even air traffic control systems such as the one which failed here) you'd know that such systems are always subjected to review by 'many eyes' -- they're just drawn from a restricted set. Think about all the havoc experienced when some hacker is the first to find a 0-day exploit in IIS or Apache and then imagine how much worse it would be if hacker-Ivan was the first to find an exploitable bug in a nuclear reactor control system or air traffic radar processing.

No. Odds are exceptionally high that whomever built this system built it to exactly the level of system availability the Irish ATC asked them to provide and the folks at-fault are still Ireland's ATC for failing to specify a more reliable system.

Whose NIC?

[herbierobinson]

I know of one NIC (that was never recalled) that mysteriously stops taking interrupts every so often. I am prohibited by NDA to mentioning that manufacturer's name. Anybody want to guess who it is?