Message Storm Knocks NYSE Offline

ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."

Comments

SCO, of course

[Anonymous Coward]

Immediately claimed the message storm to be the work of linux hackers

Big deal...

[Anonymous Coward]

It's called "leaving early from work".....everyone does it.

Nice copyright violation

[Anonymous Coward]

...of a wikipedia text. (You didn't follow the terms of the GNU Free Documentation License.)

Re:Nice copyright violation

[mcguyver]

I never thought the day would come when someone posts a joke and the respone, on /. of all places, references copyright restrictions. How ironic, if not a sad sign of how times have changed.

Re:Nice copyright violation

[jesterzog]

I never thought the day would come when someone posts a joke and the respone, on /. of all places, references copyright restrictions. How ironic, if not a sad sign of how times have changed.

I don't see why it's ironic. As uninformed as some slashdot posts are, there are also a lot of users who recognise that copyright makes a lot of sense, and is actually useful. It's the enforcement of copyright that allows the GPL and the GFDL to work. What many people here do complain about is the never-ending extentions of copyright, arguably against the general public interest, and allegedly because corporations have bought off politicians.

This may be a joke, but it was copied verbatim without providing the copyright notice, which is required by the GNU Free Documentation Licence. It's a copyright violation, and to ignore it as irrelevant would be hypocritical and ironic in itself. (Not to mention illegal.)

Re:Details?

[afidel]

A "message storm" is a storm of data that overwhelms a system, kind of like a DDOS, but legitimate traffic. In this case it sounds like a large number of error messages overwhelmed the message queueing system (probably MQ from IBM), which likely set off an even larger storm of error messages when backed up messages started to expire.

Transcript of the message storm

[Chairboy]

JoeTrader: dood, chk out MSFT, 12m volume
XyxyZ: wtf i sold on margin
-- NASDUCK has entered the channel
JoeTrader: rofl!
NASDUCK: whatsup?
JoeTrader: sam sold msft on margin before the spike
NASDUCK: HAHAHA!
JoeTrader: werd
XyxyZ: screw you guys
JoeTrader: OMG roflrofldolololo!!!!!
NASDUCK: you are such a tool, sam
JoeTrader: brb, gotta tell the office
-SYSTEM- JoeTrader has left the channel (sam in a tool)
-SYSTEM-:NASDUCK has changed the subject to "XyxyZ sold MSFT before the spike today!!!:D:D:D"
XyxyZ: fu duck. i hope my boss isn't online
XyxyZ: ops
XyxyZ: +ops
-SYSTEM- Hot2Trade has joined the channel
NASDUCK: nice try, only way to erase that is to crash the server
Hot2Trade: Sam, I heard that you got the horns of the bull shoved up where the bear don't shine
XyxyZ: dude this sux hard
-SYSTEM- JOHN@MLYNCH has joined the channel
NASDUCK: nice one Hot2Trade. asl?
Hot2Trade: fu hippy, this is Jerry in at prudential
NASDUCK: fuc sorry, didn't recognize you :O
XyxyZ: So if I can down the server, I can erase the subject?
Hot2Trade: no worries I just changed my nic
NASDUCK: XyxyZ, you got pwned by the bull
JOHN@MLYNCH: SAM! HAHAHA I TOLDYOU NOT TO SELL!
JOHN@MLYNCH: YOU AER
JOHN@MLYNCH: SUCH A SP
XyxyZ: i got s cript
JOHN@MLYNCH: AZZZ!!!!!!!!!!!!!!!!!!!
XyxyZ: take this bitches
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
- SYSTEM - NASDUCK (quit(connection reset by peer))
- SYSTEM - JOHN@MLYNCH (quit(connection reset by peer))
- SYSTEM - Hot2Trade (quite(connection reset by peer))
- SYSTEM - error(91) - rebooting

Re:Great!

[Atzanteol]

I never saw the slashdot editors claim they were 'late breaking' or 'first on the scene'. Where in hell did you people ever get that illusion?

pain in the ass

[rcamera]

as a trading engine developer/support guy for a financial firm in ny, i can't stress enough what a pain in the ass this was. the day after the nyse crash, it took hours upon hours of verifying (by hand) trades that the nyse says we were filled on that we never say (because all nyse trading lines were down).

this type of 'message flood' occurs from time to time, but not on the nyse in a while. it's generally the ecms trading otc stocks that have rouge programs blast orders in an infinite loop. when this happens to an ecm, they slow down but generally don't lose the ability to trade. the nyse, who toutes the importance of their rapists^H^H^H^H^H^H^Hspecialists because they add 'stability' to the system, was dead in the water. this crash goes to show how useless the specialists really are - without the technology working, they can do nothing. if this is the case, why not just replace them altogether with electronic trade matching?

interestingly enough, the nyse announced mere months ago that they are 'merging' with archipelago - a large ecm. perhaps this merger will be the beginning of the end of the specialists.

Re:no final print

[gbasin]

Frankly, you have no clue what you are talking about nor to whom. I don't "invest", I'm a professional trader - mostly US treasury future triangular cash/future arbitrage but also quite a bit in stocks (mostly listed) doing M&A arbitrage and occasionally make markets. Technical difficulties are expected, however, if you knew what you were talking about you would understand that many on the street were expecting a final print (including specialists) and they failed at providing it. Your understanding of trading is that of an amateur, there is another side to this business buddy. When the final print didn't come, a large number of professional traders who get out of all or most of their positions by the end of the day got stuck with tied up capital, interest costs, and overnight risk that needed to be hedged as a result of no fault of their own.

Re:no final print

[gbasin]

no problem, I write C/C++ as a hobby, love these boards :) yes, interest rate. you have to remember that I am a proprietary trader, I rarely have much capital, if any, at my firm (usually just what I have made so far in the current month): I'm buying on credit and am stuck paying a little over prime if I hold overnight. You may think this is not a lot, but consider that I may have $20,000,000 in positions (most hedged against general market movement by long or short SPY or futures). Participating in the open is similar to what traders similar to myself do on closing imbalances. When NYSE approaches opening, the specialists collect orders into their book and as the time approaches begin giving indications as to where they think they will open the stock (and match all current OPG or market on open orders). My goal is to grab the other side of an imbalance, usually taking the form of an indication a large distance from the previous day's closing price, the same side that the specialist will be taking. I do this by putting orders for blocks of stock every couple pennies in the direction I want to take - I would do this on dozens of stocks as the open approaches, as much as I can handle. Even though there is practically no chance of even a fraction of my outstanding orders (sometimes approaching or exceeding a million shares), my buying power is still reduced. By entering the day with less buying power, I can't put in as many orders and therefore can not expose my self to as much opportunity.