Slashdot Mirror
Message Storm Knocks NYSE Offline
ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."
1. If anything can go wrong, it will. (see Murphy's law)
2. Systems in general work poorly or not at all.
3. Complicated systems seldom exceed five percent efficiency.
4. In complex systems, malfunction and even total non-function may not be detectable for long periods (if ever).
5. A system can fail in an infinite number of ways.
6. Systems tend to grow, and as they grow, they encroach.
7. As systems grow in complexity, they tend to oppose their stated function.
8. As systems grow in size, they tend to lose basic functions.
9. The larger the system, the less the variety in the product.
10. The larger the system, the narrower and more specialized the interfaces between individual elements.
11. Control of a system is exercised by the element with the greatest variety of behavioral responses.
12. Loose systems last longer and work better.
13. Complex systems exhibit complex and unexpected behaviors.
14. Colossal systems foster colossal errors.
-KISS
...of a wikipedia text. (You didn't follow the terms of the GNU Free Documentation License.)
I bet you say that to all the transexual prostitutes.
A "message storm" is a storm of data that overwhelms a system, kind of like a DDOS, but legitimate traffic. In this case it sounds like a large number of error messages overwhelmed the message queueing system (probably MQ from IBM), which likely set off an even larger storm of error messages when backed up messages started to expire.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
as a trading engine developer/support guy for a financial firm in ny, i can't stress enough what a pain in the ass this was. the day after the nyse crash, it took hours upon hours of verifying (by hand) trades that the nyse says we were filled on that we never say (because all nyse trading lines were down).
this type of 'message flood' occurs from time to time, but not on the nyse in a while. it's generally the ecms trading otc stocks that have rouge programs blast orders in an infinite loop. when this happens to an ecm, they slow down but generally don't lose the ability to trade. the nyse, who toutes the importance of their rapists^H^H^H^H^H^H^Hspecialists because they add 'stability' to the system, was dead in the water. this crash goes to show how useless the specialists really are - without the technology working, they can do nothing. if this is the case, why not just replace them altogether with electronic trade matching?
interestingly enough, the nyse announced mere months ago that they are 'merging' with archipelago - a large ecm. perhaps this merger will be the beginning of the end of the specialists.
Wave upon wave of demented avengers March cheerfully out of obscurity into the dream
Frankly, you have no clue what you are talking about nor to whom. I don't "invest", I'm a professional trader - mostly US treasury future triangular cash/future arbitrage but also quite a bit in stocks (mostly listed) doing M&A arbitrage and occasionally make markets. Technical difficulties are expected, however, if you knew what you were talking about you would understand that many on the street were expecting a final print (including specialists) and they failed at providing it. Your understanding of trading is that of an amateur, there is another side to this business buddy. When the final print didn't come, a large number of professional traders who get out of all or most of their positions by the end of the day got stuck with tied up capital, interest costs, and overnight risk that needed to be hedged as a result of no fault of their own.
no problem, I write C/C++ as a hobby, love these boards :)
yes, interest rate. you have to remember that I am a proprietary trader, I rarely have much capital, if any, at my firm (usually just what I have made so far in the current month): I'm buying on credit and am stuck paying a little over prime if I hold overnight. You may think this is not a lot, but consider that I may have $20,000,000 in positions (most hedged against general market movement by long or short SPY or futures).
Participating in the open is similar to what traders similar to myself do on closing imbalances. When NYSE approaches opening, the specialists collect orders into their book and as the time approaches begin giving indications as to where they think they will open the stock (and match all current OPG or market on open orders). My goal is to grab the other side of an imbalance, usually taking the form of an indication a large distance from the previous day's closing price, the same side that the specialist will be taking. I do this by putting orders for blocks of stock every couple pennies in the direction I want to take - I would do this on dozens of stocks as the open approaches, as much as I can handle. Even though there is practically no chance of even a fraction of my outstanding orders (sometimes approaching or exceeding a million shares), my buying power is still reduced. By entering the day with less buying power, I can't put in as many orders and therefore can not expose my self to as much opportunity.