Message Storm Knocks NYSE Offline

ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."

Comments

The Swear They've Got It Fixed

[geomon]

They will begin beating the squirrels at precisely 3:55 EST from now on.

Re:The Swear They've Got It Fixed

[MyNymWasTaken]

The squirrels must first be covered in hot butter sauce.

Re:The Swear They've Got It Fixed

[Stanistani]

Here Try Again - Poor Squirrels.

Re:The Swear They've Got It Fixed

[geomon]

Yes.... I've heard the immigrants from Africa saying this before.... It went something like this:

"Ubuntu.... Ubuntu..... Ubuntu......"

Re:The Swear They've Got It Fixed

[nametaken]

Oops, someone forgot to shut off the Messenger Service.

Damnit! Who sent 40,000 print jobs to the laser printer in the pit?!

Re:The Swear They've Got It Fixed

[flithm]

What the hell were you thinking when you wrote that? Seriously, I'd like to know.

Re:The Swear They've Got It Fixed

[spun]

I'd say he was thinking of covering squirrels in hot butter sauce. Kinda like Natalie Portman and hot grits. Start a Slashdot meme by saying something dumb but sorta funny.

I've got noodles in my armpits! Yow!

Re:The Swear They've Got It Fixed

[MyNymWasTaken]

Start a Slashdot meme by saying something dumb but sorta funny.

Close, but no cigar. I was bringing an established local meme into slashdot.

SCO, of course

[Anonymous Coward]

Immediately claimed the message storm to be the work of linux hackers

Re:Any Linux role?

[MasterDirk]

Linux gives the sysadmins indemnity? That's nice.

Re:Any Linux role?

[superpulpsicle]

I knew someone who worked in the tech area of the NYSE floor. They used to run alot of M$ windows clusters. Not sure if it's still true.

Re:YACSF

[coolsva]

And, what is the point in making a very specialized system that they have into open source. Like, we would have a swarm of people trying to help NYSE. Please remember, OSS only applies when the software in question is needed by the many so that the some can derive value in enhancing it

I used to monitor the NYSE for IBM. (www.nyse.com)

[Thaidog]

It is "ultra reliable' but you've got to remember the amount of hits this site takes a day... it makes /. Trolling look like a fairy Godmother!

no final print

[gbasin]

I as well as many others in my office got royally screwed here, getting stuck with quite sizeable unhedged positions overnight. It's bad enough that order routing went down, but they failed to open up for a final print (as originally proposed) later in the afternoon. Very bad.

Re:no final print

[gbasin]

Frankly, you have no clue what you are talking about nor to whom. I don't "invest", I'm a professional trader - mostly US treasury future triangular cash/future arbitrage but also quite a bit in stocks (mostly listed) doing M&A arbitrage and occasionally make markets. Technical difficulties are expected, however, if you knew what you were talking about you would understand that many on the street were expecting a final print (including specialists) and they failed at providing it. Your understanding of trading is that of an amateur, there is another side to this business buddy. When the final print didn't come, a large number of professional traders who get out of all or most of their positions by the end of the day got stuck with tied up capital, interest costs, and overnight risk that needed to be hedged as a result of no fault of their own.

Re:no final print

[wayne606]

Was your problem a "fill on close" issue? I.e. did you have trades that were supposed to go through at the closing price but didn't because there was no close that day? If so I can see how this would screw a lot of traders.

Re:no final print

[krbvroc1]

While its true that minute to minute trading is speculation, there are many legitimate reasons that people dont want outstanding trades overnight. We are still talking trader/speculators, not investors. If something bad had happened overnight it could have been a disaster to the economy.

I'm not a day trader, but some (most?) day traders follow the rule that you close out all positions at the end of the day and don't leave yourself vunerable overnight.

By your logic it would be best to only open the market once a month. But that would make the market much less liquid and that does materially affect the fundamentals of the stock market system. Even if you are a day trader and accept a much higher risk, the risk of the market not working at all is still outside the norm.

I'm finding it tough to find anything to agree with in your post - my gut tells me even the statement of "large fund managers-- don't speculate on minute-to-minute fluctations" is wrong. People with huge risk try to minimize that risk by hedging short term (even minute-minute) flucations in stocks, options, and currency.

Re:no final print

[gbasin]

Yes, on the average day I completely hedge myself using MOC (market on-close) orders, as well as open up some new positions against market imbalances on close to be closed off on the ECNs a few minutes after the close.

Re:no final print

[bluGill]

No, it would be best to open the stock market once every 5 years[1]! These are companies you are buying, not paper. They have value for fundamental reasons, not because the stars are stacked a different way today.

I suppose I should say anything, traders like you are what give students of Benjamin Graham the extra boost to our portfolio in the long run.

[1]Warren Buffet, second richest man in the world (these positions vary from time to time of course) said this. Though I don't recall the extact timeframe he said. He made all his money as a trader. He was Benjamin Graham's most successfull student.

Re:no final print

[gbasin]

I assume this question pertains to the NYSE (all other markets are fully electronic). The answer is yes and no, a normal order enters the specialists book where his job is to match it, usually quite quickly. You can also use an NX feature for round lots to bypass specialist intervention - used when speed is more important than price.

Re:no final print

[gbasin]

I take very measured risks, I do not trade based on "technical analysis" and other crap. I do not work on wall street, I am private. I make a comfortable 7 figures, on track to do almost 8 this year - this is all from trading profits, not salary. People like you take for granted the value of real traders such as myself, who get paid by the market to make it more efficient (think arbitrage, market making). I love it how people shoot down others when they have no idea what they're talking about, how do you plan to buy your 1000 shares of GE once every 3 years when no one will sell it to you for a reasonable price? Oh yea, me. Idiots. http://themarkettruth.blogspot.com/

Re:no final print

[Duhavid]

What, if I may be so bold as to ask, if a "final print" and what does that do for you? Why was it important?

Re:no final print

[willis]

Hi.

You know those large fund managers - where do you think they can get their stock from? How do you think, say, Fidelity can sell 500,000 shares of IBM to Prudential? They can't just call up Pru and say "yo, I've got 500,000 IBM to go - what will you pay", because then Pru will think "Damn, he's got a lot to sell, I'll bid really low because he has to sell it to someone". Instead, Fidelity slices up the orders over minutes, hours, days, and over time, the shares change hands.

Because it's not just Fidelity, it's 50 other funds and pension plans and moms and dads moving in and out of positions over time, the net result is tons of small trades. Some banks will buy a block of stock from one fund, and hold onto it, parcelling it out to other people over time. Other people stand in the middle and "make markets", taking the absorbing the imbalences due to bad timing (say Pru sells and Fidelity hasn't started to buy yet).

It's really complicated, and can be fascinating.
This book : http://www.amazon.com/exec/obidos/tg/detail/-/0195 144708 explains things very well, if you're interested in learning and broadening your view of markets, give it a look.

Re:no final print

[gbasin]

The final closing price on the NYSE for a stock. Anyone who wants to trade on the final print, can, using market on close orders (to buy or sell). Primarily, it allows one to get out of any positions and get flat for the night at whatever the final closing price is (decided upon by the specialist). Secondly, it allows traders like me to "help out the specialist" meet the demands of a buy or sell imbalance on close, getting paid to take on the same risks the specialists do. Finally, professional traders have their daily profit and loss "marked to market", where even if they have positions open, receive credit/debit to their account overnight based on where their positions closed (the final print) - this is also used to calculate overnight interest in positions as well as margin needed to maintain the position. The fact that they simply stopped trading for the day forced me to keep several positions overnight, costing me some change (maybe $100) in interest but more importantly tieing up close to $5mil in buying power which I didn't get back until after the open the next day hurting my ability to participate in the opening.

Re:no final print

[Duhavid]

"overnight interest in positions"...

interest as in 'interest rate' or 'interesting'?

And how do you participate in the opening?

Thank you for your answers thus far.

Re:no final print

[gbasin]

no problem, I write C/C++ as a hobby, love these boards :) yes, interest rate. you have to remember that I am a proprietary trader, I rarely have much capital, if any, at my firm (usually just what I have made so far in the current month): I'm buying on credit and am stuck paying a little over prime if I hold overnight. You may think this is not a lot, but consider that I may have $20,000,000 in positions (most hedged against general market movement by long or short SPY or futures). Participating in the open is similar to what traders similar to myself do on closing imbalances. When NYSE approaches opening, the specialists collect orders into their book and as the time approaches begin giving indications as to where they think they will open the stock (and match all current OPG or market on open orders). My goal is to grab the other side of an imbalance, usually taking the form of an indication a large distance from the previous day's closing price, the same side that the specialist will be taking. I do this by putting orders for blocks of stock every couple pennies in the direction I want to take - I would do this on dozens of stocks as the open approaches, as much as I can handle. Even though there is practically no chance of even a fraction of my outstanding orders (sometimes approaching or exceeding a million shares), my buying power is still reduced. By entering the day with less buying power, I can't put in as many orders and therefore can not expose my self to as much opportunity.

Re:no final print

[Archangel_Azazel]

It's threads like this that remind me the stocks are a lot more in depth than what I played with in 7th grade...

Good luck and thanks for the info!

A.A

Re:no final print

[ras_b]

say, man- you do know that there are several other markets that trade pretty much 24/7 (or at least 24/5) and would allow you to hedge any stuck overnight positions. the CME index futures come to mind. i was trading those when the nyse went down that day. there was plenty of time before the close of regular trading to open a position there. after the futures close at 3:15 central time, they open back up at 3:30 and trade all night until 3:15pm the next day. a position could have been entered and held overnight- just a thought.

Re:no final print

[gbasin]

Not enough liquidity to square off the kind of positions I had without costing me too much. Different cost structure as well (although I get close to clearing rate at CME)

Re:no final print

[cellocgw]

Sorry, but I really don't give a rat's ass what happens to a bunch of trigger-happy day traders. And that's what you are if you plan your moneymaking schemes around closing or opening positions to a precision of minutes, let alone hours.
I just want to be able to make some investments, in long-term positions, and not worry about a bunch of insiders gaming the system to their personal benefit.

And spare me the "you don't understand how the market works" lecture. The whole point is that us outsiders shouldn't have to know, and shouldn't have to worry about volatility on a time scale that we simply don't have the resources to react to.

Re:no final print

[gbasin]

Honestly, are you retarded? "I just want to be able to make some investments, in long-term positions, and not worry about a bunch of insiders gaming the system to their personal benefit." How do you plan to make these investments? I don't "game the system", I make it more efficient, to the benefit of naive idiots like YOU. Again, when you go to buy stock, it's people like me that take the risk of undercutting the market makers and sell it to you.

Re:no final print

[pyite]

Maybe you don't understand what it is the OP does. Professional traders are not the same as day traders. Professional traders are one of the reasons the market actually functions. Then again, certain important people in the industry, such as David Shaw have said that "A lot of people in the business do things that should be done by computers." No doubt we keep getting closer to that point, but professional traders are and probably will be needed.

Re:no final print

[cellocgw]

I remain unconvinced that there's any reason to close deals within the next 5 minutes. If there really is money to be "made" there, there's money to be lost as well.
Then again, I am absolutely opposed to any sort of margin purchases too so call me a conservative if you want.

Re:no final print

[pthisis]

Frankly, you have no clue what you are talking about nor to whom. I don't "invest", I'm a professional trader - mostly US treasury future triangular cash/future arbitrage

Yes, like I said, you try to make money timing small market fluctuations.

When the final print didn't come, a large number of professional traders who get out of all or most of their positions by the end of the day got stuck with tied up capital, interest costs, and overnight risk that needed to be hedged as a result of no fault of their own.

It's not like they "got stuck" with them. They bought them, and hadn't yet sold them.

"Ultra Reliable".. heh..

[Chordonblue]

Why not just say, "Unsinkable" instead?

To NYSE

[mcguyver]

1. If anything can go wrong, it will. (see Murphy's law)
2. Systems in general work poorly or not at all.
3. Complicated systems seldom exceed five percent efficiency.
4. In complex systems, malfunction and even total non-function may not be detectable for long periods (if ever).
5. A system can fail in an infinite number of ways.
6. Systems tend to grow, and as they grow, they encroach.
7. As systems grow in complexity, they tend to oppose their stated function.
8. As systems grow in size, they tend to lose basic functions.
9. The larger the system, the less the variety in the product.
10. The larger the system, the narrower and more specialized the interfaces between individual elements.
11. Control of a system is exercised by the element with the greatest variety of behavioral responses.
12. Loose systems last longer and work better.
13. Complex systems exhibit complex and unexpected behaviors.
14. Colossal systems foster colossal errors.
-KISS

Re:To NYSE

[myov]

Designing a system to not fail means that it will fail in a way that you didn't plan for.

Twice in the last year, for example, both my primary and backup systems have failed within a week of each other.

Re:To NYSE

[matt2004]

15. Colossal lists make me lose interest in parent quickly, despite "Informative" mod.

Re:To NYSE

[LaCosaNostradamus]

In other words, "ultra-reliable" was just a marketing term to support whatever bonuses were paid out that year. Anything "ultra-reliable" is probably too simple to use in the case where people care about ultra-reliability. A spoon is ultra-reliable and I've never suffered a spoon failure ... but I can't fix a car transmission with a spoon. My spoons could become 100 times more unreliable, and I'd barely even notice. They're too simple to even care about.

Re:Anyone notice

[EnronHaliburton2004]

Looks like high unemployment to drive down the cost of labour is the new in thing.

It's not new unfortunately. I remember saying something very similar when GM laid off a ton of people in the early 90s and their stock went up. And this sort of behavior predates my statement by a very long time.

The irony here: At the same time that GM & Ford were shutting down their factories in my state, Toyota & Honda were opening factories. So their whole argument about an "unfriendly business environment" doesn't hold much water.

If I want to be a patriot and buy American, I'm going to buy from the company who actually employ Americans thank you Mr. Ford & Mr. GM.

Big deal...

[Anonymous Coward]

It's called "leaving early from work".....everyone does it.

Details?

[CrackHappy]

I haven't got the time or I would look myself - does anyone have any more informative sources on the specific information about the cause of the problem? And WTF is a "Message Storm"? God - another catchphrase - great!

Re:Details?

[afidel]

A "message storm" is a storm of data that overwhelms a system, kind of like a DDOS, but legitimate traffic. In this case it sounds like a large number of error messages overwhelmed the message queueing system (probably MQ from IBM), which likely set off an even larger storm of error messages when backed up messages started to expire.

Re:Details?

[MightyMartian]

Did you notice the ticker going "PR0N... V1AGRA... DIPLOMAS... PR0N... V1AGRA..."?

Re:Details?

[photon317]

I've never understood the corprorate programmers' obsession with purchasing message queue systems like MQSeries. You can design and code something that does the 90% that you actually need of what MQSeries does yourself in a week tops. MQSeries is so universal and overly complex for what most people want out of it, which is just reliable transactional networked message queueing with the options of in-order delivery, multiple queuers and dequeuers, and disk persistence.

Re:Details?

[conteXXt]

well of course ... when you put it that way.

shoot, I'll just go and whip one up before bedtime.

Re:Details?

[Bender0x7D1]

The term "Message Storm" has been around for a long time. I used to work on an event management system and we had to carefully manage our reporting rate to avoid this problem.

In a lot of cases a "catastrophic" failure will trigger a whole series of alarms. If a power supply goes, you can get a heat warning, then a voltage warning, then a backup activation warning, which shut down non-critical systems, which generate heartbeat failure messages, etc. So a single event generates multiple, (sometimes thousands), of individual alarms, informative and management messages.

You wouldn't believe the flood of errors that can happen when a TOXIC_LEAK_DETECTED alarm is generated. (This is a real standard alarm type. Check IEEE Standards.)

Re:Details?

[bigberk]

I figure we're not getting the entire story. Remember, the NASDAQ had a "mysterious glitch" within the past few weeks as well (quotes off by multiples). Two of the best run, most important stock exchanges in the world suffering unusual and silly sounding errors?

Re:Details?

[caluml]

OMG!! It's teh digital Perl Harbour!!! Lol!!11

Re:Details?

[ras_b]

something to keep in mind about systems like these is that they are basically handling intentional DDOS attacks all day. what i mean is, picture thousands of traders all clicking away at a computer screen at the exact same millisecond (kind of like a slashdot effect), frequently sending multiple orders each. this often happens because traders listen to the same news and trade off the same events. so the exchange's systems are designed to handle major DDOSing all day. and they are required to handle the requests in milliseconds or it is considered an order delay (don't know the exact but something like > 300 milliseconds is considerd slow). don't know if i'm doing a great job explaining it, but i'm actually kind of impressed by the systems. they get beat down all day and do a pretty damn good job of handling it. some technical difficulties are expected. if i as a trader don't prepare for system failures in advance with backups and other ways to get out of positions, i'm a foolio.

Re:Details?

[photon317]

Well we use HTTP because we want to interoperate with code written by other people. Most uses of MQSeries are within an architecture controlled by one company, as an internal message-queueing method between different peices of code (threads or processes of various kinds, possibly on different machines from each other, it varies a lot). Considering how large an effort it is to engineer the type of software that MQSeries gets used in, it would be pretty trivial to throw in your own custom queueing software that's better suited to your own needs.

Re:This is messed

[MightyMartian]

Actually, I'm changing tact. I've decided to blame Alfred Hitchcock. I think it's healthy to spread blame around. Next week, I've decided everything will be your fault.

Re:Any Linux role?

[taped2thedesk]

Was there any Linux role here? If not someone should be fired.

And if Linux was involved, everything would be ok? Oh, that's right. I don't know what I was thinking.

Let me guess

[tripslash]

They had to hire outside *nix coders when the in-house MS crew couldn't integrate the existing WinLAN into the (unsupported, shortsighted) Linux rollout last month.

What is the System?

[putko]

It sounds like a distributed systems failure, alright.

Here is something about the system that might have broken. I'm wondering if the thing that failed really is the thing mentioned here -- the stuff the stuff Birman did. His new book on distributed systems is out, by the way.

Somone will get flying ninja-kicked in the nuts for this, you can be sure.

Nice copyright violation

[Anonymous Coward]

...of a wikipedia text. (You didn't follow the terms of the GNU Free Documentation License.)

Re:Nice copyright violation

[mcguyver]

I never thought the day would come when someone posts a joke and the respone, on /. of all places, references copyright restrictions. How ironic, if not a sad sign of how times have changed.

Re:Nice copyright violation

[Detritus]

Whether or not someone's copyright has been infringed, it is plagiarism.

Re:Nice copyright violation

[Lshmael]

The GNU Free Documentation License is not about restrictions, but about giving the viewers of your post the same rights that Wikipedia gave you, thereby allowing anyone to continue to spread your "joke."

Re:Nice copyright violation

[jesterzog]

I never thought the day would come when someone posts a joke and the respone, on /. of all places, references copyright restrictions. How ironic, if not a sad sign of how times have changed.

I don't see why it's ironic. As uninformed as some slashdot posts are, there are also a lot of users who recognise that copyright makes a lot of sense, and is actually useful. It's the enforcement of copyright that allows the GPL and the GFDL to work. What many people here do complain about is the never-ending extentions of copyright, arguably against the general public interest, and allegedly because corporations have bought off politicians.

This may be a joke, but it was copied verbatim without providing the copyright notice, which is required by the GNU Free Documentation Licence. It's a copyright violation, and to ignore it as irrelevant would be hypocritical and ironic in itself. (Not to mention illegal.)

Re:Nice copyright violation

[Riddlefox]

Why is it 100% certain that the OP took the text from Wikipedia? If you google for the phrases, there are many places with that same list. It's unlikely that all of those places ripped off Wikipedia.

It's more likely that all of those sites ripped off John Gall's book, Systematics. However, not having read that book, I can only speculate.

One of those statements in the list (A system tends to oppose it's own purpose) seems to be a re-wording of Le Chatelier's Principle, who died in the 1930's. Who violated who's copyright?

I think I'd consider the original post a copying of a portion of text for conversational purposes. A citation would've been nice, but I think it's hardly necessary.

Re:Nice copyright violation

[JamesGecko]

However, the Wikipedia article copied those from John Gall's book, Systemantics. Does the GPL apply to copies of already copyrighted work?

Re:Nice copyright violation

[triskaidekaphile]

One of those statements in the list (A system tends to oppose it's own purpose) seems to be a re-wording of Le Chatelier's Principle, who died in the 1930's. Who violated who's copyright? Rewording does not violate a copyright.

Re:Nice copyright violation

[Maggott]

Ironic or not, it's still anal.

Re:Anyone notice

[Anonymous Coward]

Proof that democracy DOES exist on slashdot. Enough ppl complained and they removed it, hooray! I can tell you I have a good enough eye-sight (don't wear glasses, don't need 'em as of 3 months ago). Some images were tougher to read then others. At least 2 out of 6 times I just couldn't read the image

I bet you say that to all the transexual prostitutes.

I washed my shirt with Ultra Tide.

[ARRRLovin]

That doesn't mean it will never get dirty.

Transcript of the message storm

[Chairboy]

JoeTrader: dood, chk out MSFT, 12m volume
XyxyZ: wtf i sold on margin
-- NASDUCK has entered the channel
JoeTrader: rofl!
NASDUCK: whatsup?
JoeTrader: sam sold msft on margin before the spike
NASDUCK: HAHAHA!
JoeTrader: werd
XyxyZ: screw you guys
JoeTrader: OMG roflrofldolololo!!!!!
NASDUCK: you are such a tool, sam
JoeTrader: brb, gotta tell the office
-SYSTEM- JoeTrader has left the channel (sam in a tool)
-SYSTEM-:NASDUCK has changed the subject to "XyxyZ sold MSFT before the spike today!!!:D:D:D"
XyxyZ: fu duck. i hope my boss isn't online
XyxyZ: ops
XyxyZ: +ops
-SYSTEM- Hot2Trade has joined the channel
NASDUCK: nice try, only way to erase that is to crash the server
Hot2Trade: Sam, I heard that you got the horns of the bull shoved up where the bear don't shine
XyxyZ: dude this sux hard
-SYSTEM- JOHN@MLYNCH has joined the channel
NASDUCK: nice one Hot2Trade. asl?
Hot2Trade: fu hippy, this is Jerry in at prudential
NASDUCK: fuc sorry, didn't recognize you :O
XyxyZ: So if I can down the server, I can erase the subject?
Hot2Trade: no worries I just changed my nic
NASDUCK: XyxyZ, you got pwned by the bull
JOHN@MLYNCH: SAM! HAHAHA I TOLDYOU NOT TO SELL!
JOHN@MLYNCH: YOU AER
JOHN@MLYNCH: SUCH A SP
XyxyZ: i got s cript
JOHN@MLYNCH: AZZZ!!!!!!!!!!!!!!!!!!!
XyxyZ: take this bitches
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
- SYSTEM - NASDUCK (quit(connection reset by peer))
- SYSTEM - JOHN@MLYNCH (quit(connection reset by peer))
- SYSTEM - Hot2Trade (quite(connection reset by peer))
- SYSTEM - error(91) - rebooting

Re:Transcript of the message storm

[udderly]

Wish I had some mod points--that's too funny! Thanks for the giggle.

Re:Transcript of the message storm

[modecx]

Haha, that's too good. Thanks for the laugh.

NASDUCK... hehe.

Re:Transcript of the message storm

[celerityfm]

Being an ex-IRC junky I get most of this, but the day trader stuff I don't get. Can anyone provide a translation for maximum funnay?

Re:Great!

[Atzanteol]

I never saw the slashdot editors claim they were 'late breaking' or 'first on the scene'. Where in hell did you people ever get that illusion?

pain in the ass

[rcamera]

as a trading engine developer/support guy for a financial firm in ny, i can't stress enough what a pain in the ass this was. the day after the nyse crash, it took hours upon hours of verifying (by hand) trades that the nyse says we were filled on that we never say (because all nyse trading lines were down).

this type of 'message flood' occurs from time to time, but not on the nyse in a while. it's generally the ecms trading otc stocks that have rouge programs blast orders in an infinite loop. when this happens to an ecm, they slow down but generally don't lose the ability to trade. the nyse, who toutes the importance of their rapists^H^H^H^H^H^H^Hspecialists because they add 'stability' to the system, was dead in the water. this crash goes to show how useless the specialists really are - without the technology working, they can do nothing. if this is the case, why not just replace them altogether with electronic trade matching?

interestingly enough, the nyse announced mere months ago that they are 'merging' with archipelago - a large ecm. perhaps this merger will be the beginning of the end of the specialists.

Re:pain in the ass

[Saeger]

perhaps this merger will be the beginning of the end of the specialists.

Fewer unnecessary middlemen is always a good thing; now they can specialize in something that's more useful than being an overpaid floor trader. Mmm.. automation cuts from both sides.

Re:pain in the ass

[madaxe42]

What trading engine? Let me guess - you're with eSpeed?

Re:pain in the ass

[rcamera]

incorrect. i'm on a prop trading floor (but we hired a small sales team last year)

Reuters?

[ajs]

"the main system and its backup were swamped with error messages, Reuters reports"

Which is kinda funny, since it was *probably* a reuters feed that was spewing the errors in the first place....

Re:Reuters?

[Inigo Montoya]

Probably not. I used to work for TIBCO, which had at that time a market data distribution system (i.e. middleware) that redistributed Reuters data. The Reuters data came directly from the trading floors. The NYSE didn't use Reuters data, they sold it to Reuters, who then re-sold it to their customers (who often used TIBCO software to get it to their in-house trading floors).

Re:YACSF

[DogDude]

If you're gonna troll, you really should do it logged in, so that more people will see your post. Granted, it would've been modded down to -1 in about a nanosecond, but still... any yahoo can troll as an AC.

At least...

[niteskunk]

At least this didn't happen at the begining of the trading day, the very last thing we need is strain on the economy...

Re:At least...

[jgold03]

What are you talking about?? I was watching Fox news and they said the economy is doing great! Oh wait, that was in early November.

tibco?

[synx]

I wonder if NYSE uses tibco rendevouz for their message transport "bus". My work uses this software, and our usage of it has stressed it to extremes and you can end up with message storm issues.

FYI this system is a multicast-based publish-subscribe system. The multicast thing tends to be a wash IMHO, especially since many people use it for queues, rather than true 1 to many messaging.

Re:tibco?

[beeblebrox]

I worked at tibco, but not in the messaging side of engineering - it's only a smallish part of what they do. Here's what I remember, modulo time and lack of coffee:

- NASDAQ uses tibrv (formerly Rendezvous). This featured a lot in tibco's pitch about reliability/scalability etc. NYSE, I *think*, uses MQ Series.

- Rendezvous used to only run over IP broadcast, with software "router" daemons for crossing subnets. This has not been required for years now - remember, the history of Rendezvous stretches back to the late 80s.

- The/a key choking point for tibrv-over-multicast is (hardware) router multicast group table space.

Re:tibco?

[fupeg]

Tibco's implementation definitely leaves a lot to be desired. Their success has always came from their ties with Reuters (who used to own a huge stake in the company) and thus their use in high profile environments. Its never been because of their technology.

Re:tibco?

[synx]

Ah yes, we've seen the multicast group table problem. Basically you run into it and your system instantly crashes and suffers huge difficult to recover problems. I wasn't directly involved, but we have put in place migration paths away from tibco. Of course we were doing several things wrong:

- Dont have HUGE subject names - significantly reduces performance I have been told.
- dont use multicast for point to point RPC-like services.

Of course its intensely attractive since you can just run a program, not worry about sockets and stuff, and send and receive messages with no problems. But the very same attractiveness has allowed us to build a big pile of crappy middleware ontop of rv. arrrghh!

As for the software "router" daemons (rvrd), we use them to cross over WAN links. This in my experience has been the worst part, especially when you reach low-grade packet loss. What ends up happening is your entire message stream is sent on 1 TCP socket, and it ends up with a max thruput of about 20-30kB/sec so you end up with huge buffering which rvrd ends up unable to recover from. It has made me cry a few times.

Multicast storm

[thelordx]

This was actually most likely the result of a multicast or "slow consumer" storm. In a multicast network environment, often desktops are overloaded by all of the filtering they must perform (multicast sends nearly everything to everybody). Sometimes some desktops will miss a packet and ask for a retransmission. Often, this involves retransmitting in multicast-form - that is to all of the consumers. If this happens too many times, you get a storm. No matter what the NYSE does (unless they buy our technology - email for info if you'd like), they will not be able to avoid this happening in the future. It is a condition of middleware systems that were not meant to handle the volumes of modern stock markets, rather than a simple glitch in the network infrastructure.

Actually Linux is common at NYSE

[ad0gg]

IBM is also supplying Linux workstations optimized for high-resolution graphics, but consume less power. Traders will use the workstations to pipe information from the exchange floor to trading desks upstairs in real-time.

Source

NYSE is an IBM shop, using DB2, websphere. Its competitor, NASDAQ is using a Microsoft solution. Not a good week to be IBM.

Re:Great!

[Atzanteol]

If you *expect* things to be late, or at least not up-to-the-minute, then why in the hell are you complaining about it?

Frankly I'm rather sick of the "This was on foo.com last week - old news!" and "again slashdot is reporting old news!" posts.

Your post fit the "whining" portion of my sig.

Re:Anyone notice

[Monf]

i think the biggest problem was that they had images that contained uppercase and lowercase, but the answers had to be in lowercase...

TIBCO

[ManDude]

"The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems."

I think they are using TIBCO for their data messaging bus.

. . . the reliability of a network described as "ultra reliable" . . .

The use of the word network doesn't seem to fit. They won't be calling cisco.

The Dude

Re:TIBCO

[otis wildflower]

They also use MQ Series, at least for some messaging. I had to configure an iBGP link to SIAC to support MQ, for some process..

However, it's been over 6 months since I've had to deal with that shit thank Eris, so my memory on details is wonderfully blank.

Revenge of the Dick!

[Joe Jarvis]

Somewhere, in a secret underground lair wallpapered with 100 dollar bills, Dick Grasso is laughing maniacally.

This happens all the time

[Ececheira]

I work in Technology for a Wall Street firm (you've heard of them). Stuff like this happens all the time -- systems go down and are usually back up pretty quickly, some route to some exchange will bounce for a few min. This time it was worse in that it affected NYSE and not one of the smaller exchanges at the end of the trading day. If you look at any graph showing trading volumes, the last few minutes of trading are always the heaviest.

99.9% of the time, things bounce back very quickly and with the exception of a few internal emails, nobody cares, things go on.

Re:This happens all the time

[lpangelrob2]

This is true... I (now) work at a company located in the CBOT, and it seems there's more of a "errors are more okay" attitude. Which is interesting, when you consider that when an exchange connection (or even a connection for just one major contract) goes down, millions of dollars are at stake.

This contrasts with my previous experience at a struggling airline company, where such failures were tagged with a price tag and flagged as "never, ever do this again". :-) Even if they weren't struggling, though, the tolerance of failure seemed much less.

Re:This happens all the time

[bigberk]

Stuff like this happens all the time... 99.9% of the time, things bounce back very quickly

Boy are you guys going to have a fun time when there is a real market "event", not just a random glitch but a systemic problem. Those "discontinuities" look scary on graphs.

There is the other spin of the story

[KZigurs]

T-Shirt:
"I built NYSE IT backbone and all I got was this lousy t-shirt"

CV (Some poor teenager applying for underpaid software QA position):
Work Experience:
- Leading developer in development of mission critical heavily distributed and absolutely fault tolerant system that can handle 20m transactions per second (NYSE).

collisions, broadcast storms!

[ImaLamer]

I told them to throw away all those hubs and upgrade to a nice set of Layer-3 switches, no one ever listens to me though...

Natural Selection

[handy_vandal]

NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem.

No, the remedial work is designed to cull out less adaptive problems, thus preparing the digital ecosystem for the emergence of tougher problems.

-kgj

Re:Great!

[Doc Ruby]

You're really nuts. I posted only

" As usual, old news to geeks is new news to Nerds."

That's not a complaint. It's not even about Slashdot. It's a sarcastic comment about Nerds. What's your problem?

Re:I used to monitor the NYSE for IBM. (www.nyse.c

[BengalsUF]

Except, of course, that we're not talking about the public website here, but some of the trading systems themselves. A little bot of a difference there.

Re:I used to monitor the NYSE for IBM. (www.nyse.c

[Thaidog]

Yes those systems are not IBM's... they are SIAC.

Re:Any Linux role?

[Eastree]

> Using Linux means that you are ... as pure as the Virgin Mary ... Well, this *IS* Slashdot XD Of course, we're not pure *minded*, though.

NYSE/OS

[poor_boi]

Your NYSE/OS C: drive appears to be getting full. You have approximately 4% disk space free. NYSE/OS recommends having at least 15% free disk space to properly conduct Big Buck Stock Exchanges.

NYSE/OS has searched your drive for under-used and overly-large files and generated the following report.

Under-used
----------
c:\tickerlogs\SUNW.log 942 bytes

Over-sized
----------
c:\logs\NYSEerrors.log 543 Gb

Re:Anyone notice

[Sj0]

Buy a used car.

Re:True story

[TeraCo]

If only you'd been using Exchange..

What scares me the most...

[jd]

...is that I could actually see that being exactly what happened and even what was said...

Re:Any Linux role?

[Anonymous Coward]

No, linux had nothing to do with it. I work for SIAC and I was there when the crash happened. Most of our operations systems are HP|UX and Linux with a Windows box here and there. I can't say what exactly happened (being fired for a /. posting isn't really what I want) but I can say that it had to do with a bridge type connection.....

number of error messages a common fault?

[jago25_98]

I seem to remember the repetition of error messages as common in these high reliability systems.

I think the same sort if thing happened with the US power grid shutdown?

Time to put in something to slow down the maximum number of error messages.

Re:Oh no!

[fishbowl]

"It's black Tuesday all over again. Everybody sell, sell, sell! The market is about to collapse!"

It's possible that people made money overnight by accidentally holding positions that they intended to close. It's also possible that the error could compound the next day by creating problems at opening. It could conceivably cost a LOT of money to hold a position that you wanted out of when the market closed.

Don't gamble with borrowed money.

Re:Any Linux role?

[CmdrGravy]

I used to work for on a helpdesk for a company supporting a number of prominent investment banks and we would get a lot of questions about how to use VB in Excel, this probably explains why !

Unfortunately with one or two exceptions the bankers had the mental age of 8 year olds and would never work out to do what they wanted in Excel no matter how many times you explained it to them, this then became 'your fault' and enabled them to do what they were apparently best at which was throwing tantrums, stamping their feet and crying that you wouldn't do their work for them. In short 98% of investment bankers are in fact utter tools.

The messages:

[Lost Penguin]

SELL SCOXE

Re:Any Linux role?

[skarphace]

My guess would be an infinite loop problem, where packets keep getting passed between two bridges bridging the same 2 networks.

Re:Anyone notice

[carlos_benj]

What if you just want to be f***ing a**hole?

It appears you've already filled the position.....