Slashdot Mirror
Message Storm Knocks NYSE Offline
ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."
They will begin beating the squirrels at precisely 3:55 EST from now on.
"Rocky Rococo, at your cervix!"
Immediately claimed the message storm to be the work of linux hackers
Why not just say, "Unsinkable" instead?
"...Well, there's egg and bacon; egg sausage and bacon; egg and spam; egg bacon and spam; egg bacon sausage and spam..."
1. If anything can go wrong, it will. (see Murphy's law)
2. Systems in general work poorly or not at all.
3. Complicated systems seldom exceed five percent efficiency.
4. In complex systems, malfunction and even total non-function may not be detectable for long periods (if ever).
5. A system can fail in an infinite number of ways.
6. Systems tend to grow, and as they grow, they encroach.
7. As systems grow in complexity, they tend to oppose their stated function.
8. As systems grow in size, they tend to lose basic functions.
9. The larger the system, the less the variety in the product.
10. The larger the system, the narrower and more specialized the interfaces between individual elements.
11. Control of a system is exercised by the element with the greatest variety of behavioral responses.
12. Loose systems last longer and work better.
13. Complex systems exhibit complex and unexpected behaviors.
14. Colossal systems foster colossal errors.
-KISS
It's called "leaving early from work".....everyone does it.
I haven't got the time or I would look myself - does anyone have any more informative sources on the specific information about the cause of the problem? And WTF is a "Message Storm"? God - another catchphrase - great!
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d Capitalization really works: i helped my uncle jack off a horse
And if Linux was involved, everything would be ok? Oh, that's right. I don't know what I was thinking.
It sounds like a distributed systems failure, alright.
Here is something about the system that might have broken. I'm wondering if the thing that failed really is the thing mentioned here -- the stuff the stuff Birman did. His new book on distributed systems is out, by the way.
Somone will get flying ninja-kicked in the nuts for this, you can be sure.
http://www.thebricktestament.com/the_law/when_to_
...of a wikipedia text. (You didn't follow the terms of the GNU Free Documentation License.)
JoeTrader: dood, chk out MSFT, 12m volume :O
XyxyZ: wtf i sold on margin
-- NASDUCK has entered the channel
JoeTrader: rofl!
NASDUCK: whatsup?
JoeTrader: sam sold msft on margin before the spike
NASDUCK: HAHAHA!
JoeTrader: werd
XyxyZ: screw you guys
JoeTrader: OMG roflrofldolololo!!!!!
NASDUCK: you are such a tool, sam
JoeTrader: brb, gotta tell the office
-SYSTEM- JoeTrader has left the channel (sam in a tool)
-SYSTEM-:NASDUCK has changed the subject to "XyxyZ sold MSFT before the spike today!!!:D:D:D"
XyxyZ: fu duck. i hope my boss isn't online
XyxyZ: ops
XyxyZ: +ops
-SYSTEM- Hot2Trade has joined the channel
NASDUCK: nice try, only way to erase that is to crash the server
Hot2Trade: Sam, I heard that you got the horns of the bull shoved up where the bear don't shine
XyxyZ: dude this sux hard
-SYSTEM- JOHN@MLYNCH has joined the channel
NASDUCK: nice one Hot2Trade. asl?
Hot2Trade: fu hippy, this is Jerry in at prudential
NASDUCK: fuc sorry, didn't recognize you
XyxyZ: So if I can down the server, I can erase the subject?
Hot2Trade: no worries I just changed my nic
NASDUCK: XyxyZ, you got pwned by the bull
JOHN@MLYNCH: SAM! HAHAHA I TOLDYOU NOT TO SELL!
JOHN@MLYNCH: YOU AER
JOHN@MLYNCH: SUCH A SP
XyxyZ: i got s cript
JOHN@MLYNCH: AZZZ!!!!!!!!!!!!!!!!!!!
XyxyZ: take this bitches
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
- SYSTEM - NASDUCK (quit(connection reset by peer))
- SYSTEM - JOHN@MLYNCH (quit(connection reset by peer))
- SYSTEM - Hot2Trade (quite(connection reset by peer))
- SYSTEM - error(91) - rebooting
I never saw the slashdot editors claim they were 'late breaking' or 'first on the scene'. Where in hell did you people ever get that illusion?
"Ignorance more frequently begets confidence than does knowledge"
- Charles Darwin
as a trading engine developer/support guy for a financial firm in ny, i can't stress enough what a pain in the ass this was. the day after the nyse crash, it took hours upon hours of verifying (by hand) trades that the nyse says we were filled on that we never say (because all nyse trading lines were down).
this type of 'message flood' occurs from time to time, but not on the nyse in a while. it's generally the ecms trading otc stocks that have rouge programs blast orders in an infinite loop. when this happens to an ecm, they slow down but generally don't lose the ability to trade. the nyse, who toutes the importance of their rapists^H^H^H^H^H^H^Hspecialists because they add 'stability' to the system, was dead in the water. this crash goes to show how useless the specialists really are - without the technology working, they can do nothing. if this is the case, why not just replace them altogether with electronic trade matching?
interestingly enough, the nyse announced mere months ago that they are 'merging' with archipelago - a large ecm. perhaps this merger will be the beginning of the end of the specialists.
Wave upon wave of demented avengers March cheerfully out of obscurity into the dream
Frankly, you have no clue what you are talking about nor to whom. I don't "invest", I'm a professional trader - mostly US treasury future triangular cash/future arbitrage but also quite a bit in stocks (mostly listed) doing M&A arbitrage and occasionally make markets. Technical difficulties are expected, however, if you knew what you were talking about you would understand that many on the street were expecting a final print (including specialists) and they failed at providing it. Your understanding of trading is that of an amateur, there is another side to this business buddy. When the final print didn't come, a large number of professional traders who get out of all or most of their positions by the end of the day got stuck with tied up capital, interest costs, and overnight risk that needed to be hedged as a result of no fault of their own.
Was your problem a "fill on close" issue? I.e. did you have trades that were supposed to go through at the closing price but didn't because there was no close that day? If so I can see how this would screw a lot of traders.
I'm not a day trader, but some (most?) day traders follow the rule that you close out all positions at the end of the day and don't leave yourself vunerable overnight.
By your logic it would be best to only open the market once a month. But that would make the market much less liquid and that does materially affect the fundamentals of the stock market system. Even if you are a day trader and accept a much higher risk, the risk of the market not working at all is still outside the norm.
I'm finding it tough to find anything to agree with in your post - my gut tells me even the statement of "large fund managers-- don't speculate on minute-to-minute fluctations" is wrong. People with huge risk try to minimize that risk by hedging short term (even minute-minute) flucations in stocks, options, and currency.
Somewhere, in a secret underground lair wallpapered with 100 dollar bills, Dick Grasso is laughing maniacally.
I work in Technology for a Wall Street firm (you've heard of them). Stuff like this happens all the time -- systems go down and are usually back up pretty quickly, some route to some exchange will bounce for a few min. This time it was worse in that it affected NYSE and not one of the smaller exchanges at the end of the trading day. If you look at any graph showing trading volumes, the last few minutes of trading are always the heaviest.
99.9% of the time, things bounce back very quickly and with the exception of a few internal emails, nobody cares, things go on.
Yes, on the average day I completely hedge myself using MOC (market on-close) orders, as well as open up some new positions against market imbalances on close to be closed off on the ECNs a few minutes after the close.
T-Shirt:
"I built NYSE IT backbone and all I got was this lousy t-shirt"
CV (Some poor teenager applying for underpaid software QA position):
Work Experience:
- Leading developer in development of mission critical heavily distributed and absolutely fault tolerant system that can handle 20m transactions per second (NYSE).
The final closing price on the NYSE for a stock. Anyone who wants to trade on the final print, can, using market on close orders (to buy or sell). Primarily, it allows one to get out of any positions and get flat for the night at whatever the final closing price is (decided upon by the specialist). Secondly, it allows traders like me to "help out the specialist" meet the demands of a buy or sell imbalance on close, getting paid to take on the same risks the specialists do. Finally, professional traders have their daily profit and loss "marked to market", where even if they have positions open, receive credit/debit to their account overnight based on where their positions closed (the final print) - this is also used to calculate overnight interest in positions as well as margin needed to maintain the position. The fact that they simply stopped trading for the day forced me to keep several positions overnight, costing me some change (maybe $100) in interest but more importantly tieing up close to $5mil in buying power which I didn't get back until after the open the next day hurting my ability to participate in the opening.
no problem, I write C/C++ as a hobby, love these boards :)
yes, interest rate. you have to remember that I am a proprietary trader, I rarely have much capital, if any, at my firm (usually just what I have made so far in the current month): I'm buying on credit and am stuck paying a little over prime if I hold overnight. You may think this is not a lot, but consider that I may have $20,000,000 in positions (most hedged against general market movement by long or short SPY or futures).
Participating in the open is similar to what traders similar to myself do on closing imbalances. When NYSE approaches opening, the specialists collect orders into their book and as the time approaches begin giving indications as to where they think they will open the stock (and match all current OPG or market on open orders). My goal is to grab the other side of an imbalance, usually taking the form of an indication a large distance from the previous day's closing price, the same side that the specialist will be taking. I do this by putting orders for blocks of stock every couple pennies in the direction I want to take - I would do this on dozens of stocks as the open approaches, as much as I can handle. Even though there is practically no chance of even a fraction of my outstanding orders (sometimes approaching or exceeding a million shares), my buying power is still reduced. By entering the day with less buying power, I can't put in as many orders and therefore can not expose my self to as much opportunity.
No, linux had nothing to do with it. I work for SIAC and I was there when the crash happened. Most of our operations systems are HP|UX and Linux with a Windows box here and there. I can't say what exactly happened (being fired for a /. posting isn't really what I want) but I can say that it had to do with a bridge type connection.....
"It's black Tuesday all over again. Everybody sell, sell, sell! The market is about to collapse!"
It's possible that people made money overnight by accidentally holding positions that they intended to close. It's also possible that the error could compound the next day by creating problems at opening. It could conceivably cost a LOT of money to hold a position that you wanted out of when the market closed.
Don't gamble with borrowed money.
-fb Everything not expressly forbidden is now mandatory.