Message Storm Knocks NYSE Offline
ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."
I as well as many others in my office got royally screwed here, getting stuck with quite sizeable unhedged positions overnight. It's bad enough that order routing went down, but they failed to open up for a final print (as originally proposed) later in the afternoon. Very bad.
It sounds like a distributed systems failure, alright.
Here is something about the system that might have broken. I'm wondering if the thing that failed really is the thing mentioned here -- the stuff the stuff Birman did. His new book on distributed systems is out, by the way.
Somone will get flying ninja-kicked in the nuts for this, you can be sure.
http://www.thebricktestament.com/the_law/when_to_
"the main system and its backup were swamped with error messages, Reuters reports"
Which is kinda funny, since it was *probably* a reuters feed that was spewing the errors in the first place....
Somewhere, in a secret underground lair wallpapered with 100 dollar bills, Dick Grasso is laughing maniacally.
No, linux had nothing to do with it. I work for SIAC and I was there when the crash happened. Most of our operations systems are HP|UX and Linux with a Windows box here and there. I can't say what exactly happened (being fired for a /. posting isn't really what I want) but I can say that it had to do with a bridge type connection.....