Slashdot Mirror
Dublin Air Traffic Control Brought Down By Faulty NIC
Not so very long ago after passengers were left hanging by a similar glitch at LAX, Gilby4mPuck writes with another story of NIC failure leading to a disruption of air traffic, this time in Ireland, excerpting: "Data showing the location, height and speed of approaching planes disappeared from screens for 10 minutes each time. ...
Thales ATM stated that in 10 similar air traffic control Centres worldwide with over 500,000 flight hours (50 years), this is the first time an incident of this type has been reported. ...
'[They] confirmed the root cause of the hardware system malfunction as an intermittent malfunctioning network card which consequently overcame the built-in system redundancy,' said an IAA spokeswoman."
Testing doesn't confer prescience.
I think the issue is one of maintenance. things need to be replaced after their life-cycle is over, even if they seem to be functioning at the time.
Put all those NIC's on the terror watchlist!
When you shoot a mime, do you use a silencer?
The possiblity of failure can be reduced, but never completely removed. It's a simple matter of probabilities. E.g. a certain component fails on any day with probability p, if we add n redunndant fail-overs, the total system will fail with probability 1-p^n, an equation which will never be one, but it can get close.
Whatever happened to testing of installed hardware? You'd think they might csider that sort of thing important when it involves the lives of thousands of people. Then again, maybe they were drunk at the time.
Well, when we set up some cheap NAS boxes with redundant nics .. some load balancers and other goodies .. we tested it by yanking cables on the bonded nics and making sure everything still worked.
This was for an e-commerce site.. I would agree in hoping more testing with real failures would be done on systems that monitor air traffic.
Also, we were very drunk when yanking cables during our test .. so I don't think intoxication is really a factor. In fact, turning a drunken monkey loose in a data center with a clearance to pull cables is _very_ good fail over testing :)
The very best planned of redundant systems can be brought to its knees by hardware that "mostly works".
It's not hard to have system B check that system A is on/off line, and step in if the latter is the case. But what happens when A is *mostly* or *sorta* online? Does system B check that ALL functionality done by A is being done appropriately? Almost never.
And that's why, even in the best, most carefully designed, fully redundant high-availability systems, you never, ever see 100% uptime. It's just not possible to anticipate everything that can go wrong.
So design a system that fails gracefully! That's what nature did.
Take a look at your own body. It's a gorgeous example of a high-availability, high-redundancy system. There are literally BILLIONS of cells in your body, each operating as a semi-independent unit, such that any of them can fail without bringing down the whole, or even affecting it noticeably. Your body is an excellent example of a cheap, redundant, high-availability system.
Yet catastrophic failures still occur. Whether by cancer, diabetes, or heart disease, even a well-designed, tested-for-millions-of-years high-redundancy system with billions of individual, replaceable parts fails catastrophically from time to time.
It's the nature of the beast.
Mother nature has compensated by making not only the system redundant, but the need for the system also redundant. Rapid reproduction is nature's friend! Not just redundancy, but redundant redundancy.
High availability - it's much, much, MUCH harder than you thought.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
People - I am trying to collect airport related scary stories. I haven't got many yet but if you have some then please let me know - you can email me at admin@scareports.com or just visit the site (blatant pimping) here .
Quite likely it did work at the time of FAT, SAT, Shadow operation and when going into live operation.
If it breaks down later on is another issue, that's not possible to test for beforehand. Isn't that pretty obvious? It is like testing a car to see if it will ever be in an accident. You sir, are the drunk one :)
Probable impossibilities are to be preferred to improbable possibilities.
Aristotele
I'd have to have some sympathy that it was an intermittent problem. They can really cause confusion to automated systems that are designed to cope with hard failures. I've had many occasions in my latter career in Service Delivery and support where it's taken human conviction to sort out issues caused by the cluster software trying to cope with intermittent connections
if this piece of hardware was capable of "overc[oming] the built-in system redundancy", perhaps its ilk ought to be patrolling the transistorized wunderplatz of interconnected morsels governing our most hubris means of transportation? I, for one, would certainly feel safer.
We now have confirmed reports from an informed Orange County minister that Ethel is still an active communist.
When I was administering a small network in Marin, every time we had a small earthquake, all of the AppleTalk connectors would come loose. Took hours to find the faults and push them together. I guess we should have used duct tape.
I suppose at an airport as each jet came in creating vibrations, those same connectors would have dislodged.
Ten minutes at a time? That doesn't sound like a "mostly broken" problem to me, that sounds like a 10 minute fail-over time. Shit happens, but if it takes you 10 minutes for your stuff to automatically start working again you're doing it wrong, especially since its all int one data center. And whatever hapened to redundant off-site systems? New law: As a conversation progresses, the chance of someone saying "terrorist" approaches 100%
If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
I had an engineer stuck in Germany for three days due to this stupidity. He got his fill of beer, good hotel rooms and sightseeing done, so in his mind it was a decent holiday. The insane thing was that this issue happened before a few weeks earlier, there was an investigation however it did not discover then faulty NIC then either.
I'm inclined to trust the card which has been working fine for 5 years over a card which was put in yesterday.
The problem is not that redundancy wasn't implemented.
The problem is that redundancy doesn't handle 'flapping' hardware very well.
The NIC intermittently failed, causing the redundancy to switch cards several times.
This can play havoc on systems that work on a LAN and assume the MAC address to stay the same.
Also, a NIC that does not report an error, doesn't fail completely and simply swaps a few bits around can be nigh-on impossible to diagnose.
This could have been caught with real-time hardware and log-monitoring, but I have to confess even I only check the logs daily, not real-time. While some monitoring systems can mail the admin in the event of failure, not all systems are usually configured that way ('workstations' being a prime candidate).
There is a line you draw between monitoring and cost-effectiveness. Every company takes a claculated risk in this and they got bitten.
"I was in love with a beautiful blonde once, dear. She drove me to drink. It's the one thing I am indebted to her for."
we don't have a prime minister and I'm fairly sure customs don't wear green.
in a high noise/vibration/dust environment?
If it works after 5 years. sure.
If not there always is a backup. isn't there? Well, in that case there is a backup of the backup.
Only The Spice confers prescience.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
"...an intermittent malfunctioning network card which consequently overcame the built-in system redundancy"
But it's one of the lucky ones.
Every year, thousands of NICs fall victim to built-in system redundancy; if you know a card whose activity indicators are darkened and lifeless, it may have a redundancy problem. With your support and donations, we at Ethernetics Anonymous can help more network cards beat the scourge of built-in system redundancy, and make them feel like a useful part of society again.
Blank until
in this case would be the ability to run air traffic control without all those fancy computrons, should the need arise.
Swedish plasma phys. PhD student; MSc EE; knows maths, programming, electronics; finance interest; seeks opportunities
The article says "it overcame the built in system redundancy"... how the hell does ONE failing card in a redundant setup "overcome" the redundant backup parts/systems ??
I call "CYA kissass excuse maker" to the stand!
Someone screwed up big, and they're Covering Their Asses now.
" What luck for rulers that men do not think" - Adolf Hitler
This is very true. I worked on system where we had lots of redundancy in critical places. But given enough tests sometimes the bugs would get through, usually in ways that you hadn't thought of.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
I was due to fly the evening it all went wrong. Here's a lesson: if you're standing in a three-hour queue for the Ryanair desk, and they tell people to rebook on the web, and you take out a laptop and 3G modem, be prepared for a stampede.
Yes but if _one_ NIC can bring the entire system down what other single failures in a component could bring the entire system down? Obviously the system with the malfunctioning NIC can do any number of things that may result in a similar failure mode. Or what happens if the network switch it is attached to fails (I assume they use multiple paths... but if one nic can nuke it all, imagine if a switch went bonkers).
If they have good redundancy, they have two separate networks and two independent, preferrably different network cards, in all systems. Then they would do fail-over. Seems to me that if one card can bring this down, then the people that designed the redundancy screwed up badly.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
http://www.networkworld.com/community/node/29644?page=2&ts=
Show a man some news, distract him for an hour. Show a man some mod points, distract him for the rest of his life.
-- "The very best planned of redundant systems can be brought to its knees by hardware that "mostly works"."--
--NO, you are wrong there. What this indicates is that someone skimped. Techniques for processing and getting reliable signals through systems that only mostly work are very well known and used routinely. What this event means is that someone, either explicitly or implicitly assumed that NICs are binary - they either work or they don't, and designed accordingly.
What should have been used is multiple (more than two) parallel simultaneous communication paths with comparison voting at the far end to determine if the information received can be regarded as valid or not. Assuming that a NIC will fail gracefully is so boneheaded that in a safety critical application like this someone could likely be prosecuted for negligence.
Unfortunately, using the right techniques is expensive. Given that systems like this are provided by tender processes that favour low-bidders, it is not surprising that problems appear.
Of course, somebody may have done a cost-benefit analysis and decided the risk of one (or several) aircraft accidents didn't merit the extra expenditure. That's unlikely to be publicised 'though - although it would be a correct calculation to run.
As for testing - well running around pulling cables out at random doesn't really do it. Unplugging and plugging cables at various frequencies/intervals, swapping cables, plugging them into incorrect sockets, injecting noise, dropping the voltage on the power supply, overvoltage/spikes are all things that could and should be done - and in some cases mathematical formal proof that the system will work as required. All of this (and more) is done for safety critical applications.
The problem is not that redundancy wasn't implemented. The problem is that redundancy doesn't handle 'flapping' hardware very well.
The NIC intermittently failed, causing the redundancy to switch cards several times.
This can play havoc on systems that work on a LAN and assume the MAC address to stay the same.
That's what got me curious, it looked like they were using takeover instead of bonding devices.
The most well engineered system in the world can not hope to escape a ~9 minute ARP cache upstream, which makes me wonder why it was designed the way that it was.
I'm not thinking in an antagonistic sense, I'm more wondering what changed in the network _after_ the system was deployed.
Any number raised to the power 0 is 1. So if you don't install anything, hence n is 0, it will always work since the probability of failure is 1-1 = 0.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
If one fails with probability p, and you have n of them, a total system failure is probability p^n, not 1-p^n. Well technically it's Mult(p,1->n) where p1 is the probability of the first failing, p2 the probability of the second, etc, multiplying them all together to get the chance of a total system failure.
The probability of any one device in a redundant system failing is (1-((1-p)^n)). This equation rapidly approaches 1, so in larger setups failures will be a common occurrence, but they'll largely be harmless due to redundancy.
Of course this all assumes the failure mode of the device is "off" or "non-functioning". If it fails in a way which routes 15A of mains power into a network cable, redundancy might not help a whole lot.
Obviously that's not what happened, but it's not outside possibility for one device to take down an entire redundant system.
Air traffice towers generally are not noisy or dusty. And in any case, disregarding the ports, the NIC card itself is practically eternal. Compared to the rest of the system, and the lifetime of the system that is.
Two lessons learned from years of technical support. The NIC isn't broken, unless the computer has been dragged from the network cable. And that the CPU is not broken as long as the system has not been overclocked, and the heatsink is still in place.
The problem is, NICs can fail in all kinds of ways that yanking cables won't simulate. In this case it sounds like if they had yanked the cable, the backup system would have come online exactly like it was supposed to, but because the faulty NIC was kinda-sorta-almost-but-not-really working, it didn't. That's a difficult thing to test in the lab.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
I am flying to Florida tomorrow, it will only be my fifth plane flight in total and my first transatlantic flight. Despite being a rational scientist, who knows how safe it is statistically, I am having trouble suppressing my anxiety.
And at this point, fate sees fit to bombard me with horror stories about flying. This news about air traffic control comes on the heels of a headline I just saw on the front page of the Independent about pilots not reporting faults on aircraft and thus unsafe ones still flying about. I can't remember the exact wording because my brain parsed it as "TOMORROW YOU WILL DIE IN FLAMES"
If we can put a man on the moon, why can't we shoot people for Apollo-related non-sequiturs?
> But what happens when A is *mostly* or *sorta* online?
:)
You have dual NICs on system A, with a etherkiller connected to the second card.
When B takes over, it then can make sure that A stays down
And this is where the "cheap" part of my comment "cheap, redundant, high-availability system" comes into play.
See, the likelihood of failure in a redundant system goes *up* as the number of units increases. But as the number of units in a redundant system increases, the likelihood of a *complete* failure drops to a number never equalling zero. In other words, no matter how much redundancy you build in, you'll never achieve zero downtime over the long haul.
The human body achieves zero downtime over a few decades in many cases, and close to 5 nines (%99.999) over 6-7 decades in most cases. This is very, very good uptime and is very noteworthy, but requires BILLIONS of redundant units and expensive external intervention (AKA the "hospital") to achieve.
You'll never get 100%. So get off it, already. Instead, prepare for the 1% to 0.1% of downtime and call it a day!
I have no problem with your religion until you decide it's reason to deprive others of the truth.
why stop with the drunken monkey...
http://www.the5thwave.com/gallery/comp_misc/677.html
----------------------------------- My Other Sig Is Hilarious -----------------------------------
Only The Spice confers prescience.
Actually, it confers "the ability to fold space. That is, travel to any part of universe without moving."
Well, at least they got the "not moving" part right.
"[They] confirmed the root cause of the hardware system malfunction as an intermittent malfunctioning network card which consequently overcame the built-in system redundancy,' said an IAA spokeswoman."
And when we edit littlebit, can we have the truth?:
They confirmed the root caused the hardware system malfunction using an intermittent malfunctioning network card wich consequently overcame the build-in system redundancy.
I work in aviation and wonder if the terminology being used by the newspaper articles is correct.
It appears to be talking about mode S IFF (Interrogation Friend or Foe) or SIFF radar systems which identify aircraft and appends height data. The speed is the only thing that needs calculating, as it isn't encoded in the pulse train.
Why this is weird is because much older bus technologies are normally used to handle this data being transferred than current network technology, such as MIL-STD-1553.
This makes me wonder if it was one of two things - a system inputing to an ethernet PC system that calculates and displays the information or more likely they are talking about a DLTU type stub connector (or remote terminal) used in such typical buses. This is unlikely because the bus systems they are employed on, the bus controller would have picked up on the failure during continuous built in test and pulled in an alternative.
If its the former then someone needs shooting. ATC is a realtime application and the overhead involved here would be unacceptable. I'm not even sure of the benefit of a network, multiple self contained indiviual terminals would be safer.
A thistle is a fat salad for an ass's mouth...
I always thought that CPUs could never be broken. We had an Athlon 64 processor 4600+, it was never overclocked and always used with a standard fan/heatsink, in a well ventilated case. After a year of work, it then started randomly crashing every few weeks. Replacing all the components except the CPU didn't fix the problem (different motherboard, memory, etc). Replacing the CPU did fix the problem. They can die randomly but it is very rare.
Whatever happened to testing of installed hardware? You'd think they might csider that sort of thing important when it involves the lives of thousands of people. Then again, maybe they were drunk at the time.
Well, when we set up some cheap NAS boxes with redundant nics .. some load balancers and other goodies .. we tested it by yanking cables on the bonded nics and making sure everything still worked.
This was for an e-commerce site.. I would agree in hoping more testing with real failures would be done on systems that monitor air traffic.
Also, we were very drunk when yanking cables during our test .. so I don't think intoxication is really a factor. In fact, turning a drunken monkey loose in a data center with a clearance to pull cables is _very_ good fail over testing :)
it is point less to destroy a system testing it unless you have a big buget and can rebuild the system exactly the same
intoxcated monkey destroying data center != testing
intoxcated monkey destroying data center == (waste_of_money && destroying_data_center)
destroying_data_center == waste_of(time && money)
destorying_data_center != testing
null
The article says "it overcame the built in system redundancy"... how the hell does ONE failing card in a redundant setup "overcome" the redundant backup parts/systems ??
I suspect it's because, as mentioned in the summary, it was "an intermittent malfunctioning network card". i.e. the failover system must have thought the card was functioning.
That was in the movie. Read the book, it's much better.
Yes but if _one_ NIC can bring the entire system down what other single failures in a component could bring the entire system down? Obviously the system with the malfunctioning NIC can do any number of things that may result in a similar failure mode. Or what happens if the network switch it is attached to fails (I assume they use multiple paths... but if one nic can nuke it all, imagine if a switch went bonkers).
You don't need to bring the entire system down to cause havoc. What if there's a hitherto unknown bug in one of the CPUs which under some very specific set of circumstances causes aircraft altitude to be misreported on the operator's screen? As the GP said, most redundant systems only ensure that the components appear to be broadly working. They seldom check that all the components are doing something sensible.
Actually, it confers "the ability to fold space. That is, travel to any part of universe without moving."
Actually actually, the space folding is done using the Holtzman drive, which is a perfectly ordinary machine. The Navigator merely navigates, plotting a safe path through the non-space/time foldspace. The spice grants the Navigator the limited prescience required to do this.
Eventually the Navigators become obsolete, replaced by Ixian semisentient machines known as Compilers that perform the same task without needing melange. A good thing too, because by that point Arrakis is rubble and sandworms are pretty much extinct.
Details courtesy of Wikipedia (and my lack of a social life).
Everyone in Ireland knows that the Irish Examiner used to be the Cork examiner - and they never miss an opportunity to point out how Dublin is doing a bad job.
This is because Cork thinks that it's the centre of the friggin' universe. The 'Real Capital', my arse! Just a bunch of thunderin' ejits, living in their little Blarney fantasy land. Sure they can't even talk right. What the hell is a 'langer', anyway. They wouldn't even know how to spell NIC.
The fact that they are right is quite beside the point.
(For a North American cultural equivalent, please see http://en.wikipedia.org/wiki/South_Park:_Bigger%2C_Longer_%26_Uncut)
Anyone who mods me down is from Cork - believe it!
Genesis 1:32 And God typed
Not according to the book. According to the book the spice is needed to predict what will happen when you arrive, that is: To ensure that you don't arrive inside a planet, or other dangerous place. The point is that when you do something that amount to traveling faster then light, the only way to know anything about where you arrive, is to predict the future.
This is also a big reason, that the guild newer took over Dune. They were so conditioned to always seek the safe path(Because that was what their ships needed) that they could not imagine starting an operation that was not know to be 100% safe for the guild.
(Only slightly offtopic)
Indeed.
I've seen Network Bonding in RedHat Enterprise Linux with HP hardware use a 'fake' MAC address that is bound to several interfaces to avoid just this problem.
Unfortunately, it may confuse the switch it is connected to, because of said ARP cache (CAM table, ours was 16 hours).
Really-HA systems require genuine engineers with tons of real-life experience, just to know what bits work and what bits you want to avoid. ;-)
I hope to become one, one day
"I was in love with a beautiful blonde once, dear. She drove me to drink. It's the one thing I am indebted to her for."
If you didn't apply the heatsink yourself, you don't know if it's been done correctly. On my first PC that I bought with my own money (well, half bought and my dad paid the rest), it kept locking up randomly, and after lots of IRQ and driver troubleshooting my dad removed the heatsink only to find that they hadn't applied it correctly. One reapplication of thermal paste and proper connection to the CPU later, and everything was fine (until that system got messed up in a lightning storm a few years later, but I still use the case when building my own machines)
which is totally what she said
Sometimes, pure intuition can be more handy than maths.
which is totally what she said
CPUs with correctly seated heatsinks which stay within their prime operating temperature usually have no problems.
However its rather easy to get the wrong amount of goop or something else wrong with the airflow, or just a marginal chip, etc
I had an AMD t'bird 1.4ghz which would NOT run happily at 1.4ghz no matter how much I tried.
In the end I gave up and ran it happily at 1.33 for years.
liqbase
I'm sure that its not merely the destination, but also the journey that needs to be known beforehand... 1 in 10 'disappeared', not ended up 1/2 parked inside of an asteroid.
The Bible: Historically verifiable fact from an observers point of view
We did - we applied the heatsink several times, when we moved the CPU between different motherboards. Proper thermal transfer compound was used. The temperature of the CPU was fine.
G-GP:
if we add n redunndant[sic] fail-overs, the total system will fail with probability 1-p^n
GP:
Any number raised to the power 0 is 1. So if you don't install anything, hence n is 0, it will always work since the probability of failure is 1-1 = 0.
P:
Sometimes, pure intuition can be more handy than maths.
Only if you're not good at the math.
The way the G-GP described the system, the number of redundant fail-overs includes the primary system. With n=0, you have no system in place. No system, no possibility of system failure.
The article says "it overcame the built in system redundancy"... how the hell does ONE failing card in a redundant setup "overcome" the redundant backup parts/systems ??
If the card had failed completely the redundant one would probably have kicked in. What I think happened is the card malfunctioned in a way causing the system to still think that the card is fine and there is no need for the redundant one to kick in.
I recall some DEC NICS that when they started to fail, all got the same MAC. Talk about a fun thing to troubleshoot on a network! If it was a plain old switch using MAC switching, you can cause havoc pretty easily.
It's a cover-up for Zing Zang Zoom rolling out a rootkit protection
if you did apply it yourself, you can't be sure that it's been done correctly! On the first system I built; I put the heat sink on pi radians rotated from where it was supposed to be, and without thermal paste at all. (How was I supposed to know?)
Talk about expensive mistake.
FGD 135
...it is not so black and white.
I administer a network like that. Pharmaceutical plant to be precise.
All machines on the production network have 2 independent PCI nics, connecting to 2 identical but separate networks, using separate routers and switches. The critical servers are stratus high availability servers which have dual redundant everything, driving all components in lock-step and correcting errors on the fly.
If something happens to cause a network switch over, there is a bulk of network traffic to deal with it, because sockets have to be opened and closed, state has to be transferred, system control message flow has to be restarted so that all controllers go back to the normal state, ... And at application level, everything is RPC and DCOM based, so this will cause a significant disruption for the running services, since COM objects and RPC marshalling have to be destroyed and recreated, reinitialized, ...
The whole thing is very complex from a systemwide point of view, and causes a significant disruption.
Now, switchovers can be triggered by different things, like a maintenance techician replacing a controller and causing a timeout halfway through a message flow, network cable that has to be unplugged for some reason, ...
When that happens, performance of the system sucks for the next couple of minutes.
Critical signals will work within defined tolerances, but anything else will simply stop responding during the switchover.
I can perfectly imagine that IF you have a nic which is failing in just such a way that makes the network switch back and forth, it'll dirupt and eventually kill the entire network.
Unless of course the switches are smart enough to detect this and disable the physical port. But even then, we are not talking about DDoSing, but just minor errors at exactly the right time to trigger network failover.
Network redundancy of complex systems (and air traffic is much more complex than a pharma plant) can protect very well against single or predictable failures.
But there will always be very specific failures, depending on dozens of variables, which have the power to bring down the system. Since is the first time such an event occurred with that air traffic control system, it is likely to be one of those corner cases.
It's the same with other proecedures with aviation. everything is supposed to be double and triple redundant, but still National Geographic has enough material to create the series 'Seconds from disaster' where one unlikely error amplifies another, and in the end, the plane hits the building / swamp / ground.
High availability - it's much, much, MUCH harder than you thought.
I would like nothing more that to be able to breed my way out of hardware failures.
"What? Another NIC failed? Honey, spread 'em!"
It is dangerous to be right when the government is wrong.
What has to be improved is the redundancy system solution that has to be able to detect intermittent function and therefore do a complete failover.
And this isn't the first time this kind of problem have happened, and it's probably not the last time either. But in this case it was on a mission critical system.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Lightning damage can come in through the NIC. On the network I saw, a Linksys router hooked to a satellite modem started sending damaging voltage down the ethernet after a storm. One computer lost its NIC, two others lost NIC + motherboard. After the storm was over, the Linksys box was still deadly to a laptop.
One of the odd and very likable things about Dune is that there are occasionally implications that the society we read about is not the most advanced. Maybe their taboos are limiting them. Essentially the world we read about is actually in its own version of the Dark Ages where progress has all but stopped and feudalism is the only system. The Tleiaxu and the Ixians aren't in a Dark Age though. But we don't here too much about them because they are outside the known world because they violate the taboos that govern the know world.
Essentially it's a bit like reading history Taliban controlled Afghanistan, or unfortunately anywhere with an Islamic government. And I'm sure it's deliberate - Frank Herbert apparently was inspired by the Islamic uprisings against the British.
Or if you look at another way he wanted to write a hallucinogenic, retro sci fi epic, and he came up with a bunch of explanations - the Butlerian Jihad, the necessary for spice based prescience for interstallar travel, and the incompatibily between directed energy weapons and shields to explain why his universe was that way and not like conventional sci fi with ray guns, robots and open societies in the Popper sense.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
I'll have you know, my good man, that not all of slashdot's readers are American.
It's true I tell you, feller at work's next door neighbour read it in the paper.
What is a "contol" and why is this so important?
http://www.dieblinkenlights.com
Ever heard of a nice thing called Spanning Tree Protocol and the Campus Model for network design? With proper network design, this would not be an issue.
Probably a cosmic ray.... http://www.scitech.ac.uk/PMC/PRel/STFC/Cosmic.aspx
I think this is a case of Sales guy vs web dude.
Jumpstart the tartan drive.
Flew into Belfast instead last week because of this... when you're inconvenienced by technology it is very calming to know that what caused it was a real honest to goodness fuck-up, rather than a much less interesting case of human error :-)
Hardware can still go down regardless of how often its tested. The issue here is it took 10 minutes to rectify the fault.
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
No system, no possibility of system failure.
No possibility of success either. Brilliant! Let's all just go back to bed and forget about trying anything that might possibly fail...
If something happens to cause a network switch over, there is a bulk of network traffic to deal with it, because sockets have to be opened and closed, state has to be transferred, system control message flow has to be restarted so that all controllers go back to the normal state, ... And at application level, everything is RPC and DCOM based, so this will cause a significant disruption for the running services, since COM objects and RPC marshalling have to be destroyed and recreated, reinitialized, ...
That strikes me as the wrong approach. Use the two physical NICs as interfaces of a router and route everything from/to a logical NIC. Then there is no failover disruption. I think that today there is no need to do this so that the applications actually notice or (worse) need to be able to deal this type of failure.
I can perfectly imagine that IF you have a nic which is failing in just such a way that makes the network switch back and forth, it'll dirupt and eventually kill the entire network.
The well established way to deal with that is lockout periods, where only a failure of the second network and a re-availability of first will cause a switch-back before a certain time has passed. Note that this is needed regardless of switchover delay, since the system can ''oscillate'' faster for faster switchovers.
I see three possible problems: Either this system was not state-of-the art and has serious issues simply because of its old and unflexible deaign, or the designers screwed up, or it was indeed some very complex failure that had several unexpected sideeffects. Avionic people are vey good in learning from past desasters. Their history of prevention and prediction is not nearly as good.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Sounds like they spilled Guinness on the servers again! Either that, or it's those damned pesky Leprechauns! :-)
Yes, the books are better, and the spice does confer prescience in a small number of instances.
TANSTAAFL GIGO Acronyms to live by!
Data showing the location, height and speed of approaching planes disappeared from screens for 10 minutes each time.
The airport has been in chaos for the week or so since it started to happen. It took them that long to get back to full working capacity.
Caesar si viveret, ad remum dareris.
NIC's go bad. Fact of life. Equipment fails. This particular piece of equipment failed in a dramatic fashion that caused errors resulting in a denial of service to other network devices. The fix is monitoring at the switch level for these types of failures and disabling the port before the errors can cause cascading failures.
Do you run a full function check on every piece of hardware on your computer every time you turn it on? Or do you run it til it breaks then fix it? Also, consider how hard it is to trace this type of error. It is a partial failure, intermittent, and the failure brings down other machines. Any one of the machines that go down could be the culprit.
Kudos to the troubleshooters who found this beast!
TANSTAAFL GIGO Acronyms to live by!
That's why intermittent failures are so bad, because they introduce a dynamic element into an already very complicated equation, and usual testing strategies don't go as far as this.
For example, just try to see how any routing algorithm reacts to a router that's up and down, up and down, up and down...
[Pruneau
I was on my first flight on an airplane and was excited and a bit nervous. I decide to read the book my wife gave me at the last minute for some reading material. The first line was, "I was going down fast..." regarding a crashing plane. My wife claims she had no idea of course. After the fact, I can appreciate the irony at least.
I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling
I work in a company that makes stuff for ATC. Our systems have 2 networks (2 NICs in each box, 2 sets of cables, switches etc)
Every packet is sent on both networks and alarms are set off if a packet turns up somewhere without being received on the other network.
Iffy connectors etc. are discovered pretty quickly, but it is still be possible for the system to fail if, for example, a switch on the 'red' network failed and a box had a dodgy NIC on the 'green' network -- this kind of thing happens when somebody chooses to ignore an alarm because "it's working fine, we'll get round to replacing that card next week"
echo -e 'global _start \n _start: \n mov eax, 2 \n int 80h \n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a; a
Hmm....I wouldn't do that if you have 'ulimit' set to 'unlimited' number of processes allowed per user.
My blog
So if you don't install anything, hence n is 0, it will always work since the probability of failure is 1-1 = 0.
Actually, it would be more accurate to say that it would never fail. ;)
"Destroy science and religion. Science would re-emerge exactly the same; but not religion." - Penn Jillette, paraphrased
You need to go OUTSIDE today, my man.
Faster! Faster! Faster would be better!
Network Interface Cards and chips are probably the most failure prone components in a computer.
If you've been around networks long enough you've found jabbering nics, dead nics etc.
So putting in a faulty NIC card and seeing what happened wouldn't have done anything at all, huh?
Part of testing systems is trying to emulate what happens when a portion goes down.
Random Thoughts From A Diseased Mind (Not For Dummies)
Read Brain Herbert's books. They are prequels to the original Dune series and cover the history of most of the Dune races.
Yeah. Fortunately I just got back from going outside. OTOH, it was just raining, and I saw all the millions and millions of tiny water drops falling from the sky. Which made me think of Interrupt 80 and all those forked off-processes it would spawn with that code...
My blog
Just have a solid supply of your own hallucinogen of choice if you plan to make it through them. Pee-ew
On the first system I built; I put the heat sink on pi radians rotated from where it was supposed to be, and without thermal paste at all.
The word is "backwards," mate.
intoxcated monkey destroying data center != testing
intoxcated monkey destroying data center == (waste_of_money && destroying_data_center)
destroying_data_center == waste_of(time && money)
destorying_data_center != testing
You forgot
intoxicated_monkey_destroying_data_center == really_fucking_funny
Oh gods no... My bosses are highly fond of linksys for some reason.
My dick would fall off inside of 6 months. :'( I'm not a machine, dammit!
The problem is not that redundancy wasn't implemented.
[...]
This can play havoc on systems that work on a LAN and assume the MAC address to stay the same.
In other words, redundancy wasn't implemented.
Lucky Charms never pissed me off so much as Trix did. I remember one commercial where the rabbit actually *bought his own cereal* and the kids took it because "trix are for kids". I wanted to see him mow the little fuckers down for home invasion or something...
Silly rabbit, supersonic lead is for thieving, speciesist little pricks.
So putting in a faulty NIC card and seeing what happened wouldn't have done anything at all, huh?
You keep a bunch of 'faulty' NICs around?
Bullish Machine Tzar
Guess it really depends on where you are (or probably more poignantly, where you think you are) on the bathtub curve.
Are you suggesting that NICs should reproduce?
I swear I've heard about something like this happening before.
Can't a Network Access Control (NAC) enabled switch with some kind of "reasonable NIC operation metrics" shut down the port the bad NIC card is on? (While notifying the network admin, of course)
If "redundancy" is clumsily implemented, then having multiple widgets doesn't help. Your monitor may think widget 1 is up when it really isn't. I've seen that problem in a common load balancing appliance.
(Relatively) simple then, you have multiple systems, they vote on an answer, if someone is out they get voted off the island, you have another system with a different implementation also check to make sure they answer is sensible. Granted this is hideously expensive and probably only suitable for really expensive things like the space shuttle it is possible.
Where did you hear that?
Perhaps the fact that the leader of the Islamic updrisers was named "Maudi", or the fact that that uprising defeated a world-spanning empire with "desert power"? Or many other similar events?
Socialism: a lie told by totalitarians and believed by fools.
You can test kinda-sorta-almost-but-not-really working NIC by putting a jammer in-line with the test card. Systematically mess with bits important to every protocol layer intermittantly and look for potential issues, then focus there and do more tests.
It also helps to keep any bad NICs you fin around forever for this kind of testing, since they're far cheaper than jammers.
Socialism: a lie told by totalitarians and believed by fools.
You would think on a NIC or some form of transmission interface, the status could be checked by simply pinging a known good host/point on the route the service normally takes... yep, and as soon as interface A doesn't provide adequate bandwidth, it is rerouted to B, or a better approach is to evenly share the load and to reroute failed/unconfirmed transmissions via a different interface than the last one used to try. After a certain amount of FAILED transmissions, a simple algorithm to compare how many fail on each NIC could simply update a VISIBLE reliability counter ina control panel available to any operator of the machine who could then say... HMMMM... packet A was resent 200 times through interface A and failed 70 times. Packet A was sent successfully each time through interface B. Packet B failed once or twice on interface B, failed over to interface A, failed once or twice, then went back to interface C and successfully sent through and confirmed.
There's a million samples of verbal flowcharting/pseudocode type stuff we did when I was in comp sci 101 and 102 back in college. Hell we sat around brainstorming server queue ideas for miniature servers to perform stupid little operations with no purpose. Call it "practice" since it was nothing but.
Anyways... it isn't that hard to handle in software as a means to combat subtle hardware failure, especially in a system that SHOULD be redundant and unbreakable. After all, they spent how many trillions on "anti terrorism" ?? And we had ONE plausible airborne terrorist attack in how many decades? Meanwhile we have HOW many fatal airliner wrecks in same decades? Perhaps the money is ill spent by self serving bureaucrats, but I could, of course, be mistaken. After all, we know self serving bureaucrats only have our best interests in mind. 6 figure salaries never figure into it... I'm sure.
" What luck for rulers that men do not think" - Adolf Hitler
It may have been implemented, even implemented correctly.
It may have signaled the problem and worked around it, the way it should.
The bug may also have show up because the software assumes there will never be a hardware failure, no matter how short.
Cisco switches implement redundancy by default with STP.
What they don't tell you that it may take over a minute before the network has fully recovered.
(that's why woy want to set up rapid STP for backbone switches and maybe tune the setting for userland).
"I was in love with a beautiful blonde once, dear. She drove me to drink. It's the one thing I am indebted to her for."
I don't know about uprising against the British, but for sure, Herbert used many Arabic and Islamic themes in Dune. Some of the stuff is obscure historical terms, so he digged deeper than just current colloquial terms in use in the Middle East at the time.
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
It's like the old story that the more engines a plane has, the more likely it is to have an engine failure, so single engined planes are safest. It's not how many fail, what really matters is how many you have left working.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I know of one NIC (that was never recalled) that mysteriously stops taking interrupts every so often. I am prohibited by NDA to mentioning that manufacturer's name. Anybody want to guess who it is?
An engineer who ran for Congress. http://herbrobinson.us
Give me two minutes and a soldering gun and I can make one.
Random Thoughts From A Diseased Mind (Not For Dummies)