Dow Jones Plunge Fueled by Overwhelmed Computers

cloudscout writes "The Dow Jones Industrial Average dropped over 400 points today. While there were various valid financial reasons for such a decline, some of the blame is being placed on computer systems that couldn't keep up with the abnormally high volume at the New York Stock Exchange and the resulting tremor as they switched over to a backup system."

Comments

Blaming?

[VincenzoRomano]

Computers never make errors.
Humans do, at least in designing, manufacturing and sizing computer systems.
This one seems to me like blaming at a knife once you cut your fingers.

Re:Blaming?

[pilgrim23]

In the 1929 crash the problem was partially blamed on the ticker tape running at times up to 1 hour late. Before computers there were people you could blame.

Re:Blaming?

[Nerdfest]

Perhaps a poor analogy ... this is more like cursing a knife when it won't cut through a tree fast enough.

Re:Blaming?

[tha_mink]

Computers never make errors. Humans do, at least in designing, manufacturing and sizing computer systems. This one seems to me like blaming at a knife once you cut your fingers. Nobody is blaming the computers, the blame is being plopped on the computer "system", which at last I checked, is the hardware and software.

Re:Blaming?

[stevesliva]

Computers never make errors.

Sure they do... just need to bombard them with alpha particles.

Re:Blaming?

[skoaldipper]

I found this bit interesting...

"I've never seen a collapse like that, and I've only been doing this for 47 years," said Alfred E. Goldman

Huh? I remember Black Monday in 87. That was a huge (freakin' huge I tell ya!) % drop. I thought they had fixed these systems to just temporarily shut down in such a scenario. I wonder if that backup switch was part of that system.

Re:Blaming?

[skoaldipper]

Oops. I should have continued reading the article...

There are also more draconian measures that weren't invoked Tuesday. Known as "circuit breakers," these safeguards force traders to take a time-out. The Big Board developed these measures following the October 1987 crash and a mini-crash in October 1989.

Seems they didn't trip the switch here and went to plan B instead.

Re:Blaming?

[LunaticTippy]

Bombarding a computer with alpha particles is a user error.

You'd have to have an independently-evolved AI robot bombard the computer with alpha particles.

Re:Blaming?

[Mr Z]

I think he was referring to the ginormous cliff that happened at almost precisely 3PM. Take a look.

Re:Blaming?

[PRC Banker]

Shutting down a system would be bad. It would cause suspicion and speculation (on the unquoted markets, which would impact the quoted ones). "Why is the system being shut down?" "Sell [on the unquoted markets], it's a crash of all comprehension!" And the markets crash because of suspicion. Self fulfilled.

Re:Blaming?

[Anonymous Coward]

In that case, the fault lies either with the person who built the computer that couldn't withstand alpha particles, or with the person who deployed a non-alpha-particle-resistant computer in a universe with alpha particles.

Re:Blaming?

[C0y0t3]

Computers don't kill people - network lags kill people.

Computer errors exist because people program computers and it is possible to tell a computer perform a function incorrectly/inadequately. The error is in the model of the problem or in the pipe - me, I usually blame the pipe or the user. The pipe doesn't get offended and fire you, though.

Re:Blaming?

[Agent Green]

Well ... the drop sounds big, but percentage wise was nothing. There wasn't enough to trip the breaker (in CDV):

Decline, Before 1:00 P.M., Before 2:00 P.M., 2:00 - 2:29 P.M., 2:30 p.m. or Later
1250-point Decline in the Dow, 1 Hour Halt, 1 Hour Halt, 1/2 Hour Halt, No Halt
2500-point Decline in the Dow, 2 Hour Halt, 1 Hour Halt, Close For Day, Close For Day
3700-point Decline in the Dow, Close For Day, Close For Day, Close For Day, Close For Day

Oh yeah ... trading collars too:

Trading-collars, which restrict index-arbitrage trading to stabilizing, will be triggered during First-quarter 2007 when the DJIA moves 180 points or more above or below its closing value on the previous trading day and removed when the DJIA is above or below the prior day's close by 90 points.

Re:Blaming?

[crabpeople]

"Computers never make errors."

Sure they do. Bad ram, overheated components. If computers didnt make errors, you wouldn't need CRC checking - to name but one common error identifier.

Lets try and think more, go for frosty pist less. Ok?

Re:Blaming?

[Anonymous Coward]

This is referring specifically to the 200 point drop which happened almost instantly. The overall drop was no big deal percentage wise. The 200 point near-instant drop was caused by a computer error. The main system was lagging so they switched to a backup system, this caused about a half hours worth of data to get processed all at once. At the time no one knew what was happening though and were appropriately freaked out.

Re:Blaming?

[timeOday]

Computers never make errors. Humans do, at least in designing, manufacturing and sizing computer systems.

OK, but everybody knows that. Do you think the people behind the NYSE computers are getting off the hook by saying "it's not our fault, blame our computers!" I've never seen it work that way. When a system fails, the blame always falls on people.

Re:Blaming?

[Blackknight]

Remember the Pentium divide by 0 bug?

Re:Blaming?

[Austerity Empowers]

Nonsense, that tape is clearly defective.

Re:Blaming?

[AaronW]

It wasn't divide by zero, but just plain floating-point division. After I learned of the bug I wrote a simple program to just brute force test division and it very quickly detected the problem. Certain values would cause errors. See http://en.wikipedia.org/wiki/Pentium_FDIV_bug for some more information. As far as I know, there never was a problem with the division by zero logic.

Re:Blaming?

[Anonymous Coward]

From Frederick Lewis Allen's Only Yesterday , a fascinating history of the 1920s:

Once more the ticker dropped ridiculously far behind, the lights in the brokers' offices and the banks burned till dawn, and the telegraph companies distributed thousands of margin calls and requests for more collateral to back up loans at the banks. Bankers, brokers, clerks, messengers were almost at end of their strength; for days and nights they had been driving themselves to keep pace with the most terrific volume of business that had ever descended upon them. It did not seem as if they could stand it much longer. But the worst was still ahead. It came the next day, Tuesday, October 29th.

The big gong had hardly sounded in the great hall of the Exchange at ten o'clock Tuesday morning before the storm broke in full force. Huge blocks of stock were thrown upon the market for what they would bring. Five thousand shares; ten thousand shares appeared at a time on the laboring ticker at fearful recessions in price. Not only were innumerable small traders being sold out, but big ones, too, protagonists of the new economic era who a few weeks before had counted them. selves millionaires. Again and again the specialist in a stock would find himself surrounded by brokers fighting to sell--and nobody at all even thinking of buying. To give one single example: during the bull market the common stock of the White Sewing Machine Company had gone as high as 48; on Monday, October 28th, it had closed at 11 1/8. On that black Tuesday, somebody--a clever messenger boy for the Exchange, it was rumored--had the bright idea of putting in an order to buy at 1--and in the temporarily complete absence of other bids he actually got his stock for a dollar a share! The scene on the floor was chaotic. Despite the jamming of the Communication system, orders to buy and sell-mostly to sell--came in faster than human beings could possibly handle them; it was on that day that an exhausted broker, at the close of the session, found a large waste-basket which he had stuffed with orders to be executed and had carefully set aside for safekeeping-and then had completely forgotten. Within half an hour of the Opening the volume of trading had passed three million shares, by twelve o'clock it had passed eight million, by half-past one it had passed twelve Million, and when the closing gong brought the day's madness to an end the gigantic record of 16,410,030 shares had been set. Toward the close there was a rally, but by that time the average prices of fifty leading stocks, as compiled by the New York Times, had fallen nearly forty points. Meanwhile there was a near-panic in-other markets--the foreign stock exchanges, the lesser American exchanges, the grain market.

Re:Blaming?

[3fiddy]

Computers don't kill people
oh yes they do...they eat old people's medicine for food.
better get your insurance policy up to date.

Re:Blaming?

[C0y0t3]

no, no, this is nothing to be concerned about. the market will sort it all out - do not panic.

Re:Blaming?

[Mr Z]

I realize that the cliff was due to delayed updates. Everybody does now. Still, to a trader who's spent 40+ years on the floor, seeing that 200 point drop in a matter of seconds without knowing what's going on would do far more than give one pause. It might even require a change of pants afterwards.

Re:Blaming?

[R.D.Olivaw]

Humans do, at least in designing, manufacturing and sizing computer systems.

Let's try and read a post fully before replying. Ok?

Re:Blaming?

[default luser]

You'd have to have an independently-evolved AI robot bombard the computer with alpha particles.

Or, you'd just have to be crazy enough to send your computer up into space. What, did you think they used mice to run those satellites and space probes?

Solar / cosmic radiation is a real problem for computer systems that can't take advantage of the earth's magnetic field.

NYSE: Microsoft (R) powered (R)

[hxnwix]

It looks like somebody forgot to change the gigantic SQL batteries embedded in the side of the NYSE building...

Re:NYSE: Microsoft (R) powered (R)

[Anonymous Coward]

as someone who has worked on the floor and systems at NYSE, I can assure you that none of the machines used for trading by specialists or brokers are windows machines, nor are the backends.

all of the machines you see in pictures are dumb terminals connected to a number of master unix systems. they are configured for that special keyboard set used for deals and also touchscreen inputs. however, some (not all) can be booted into windows for personal use or to use firm-specific software.

a side note -- the monitors used by NYSE (you can see them in the pictures on CNN) are ridiculous. huge size, almost perfect 180 degree visibility from side to side and top to bottom, touchscreen, and dust/fingerprint repellent. pretty nasty stuff.

as far as the backends go, the standard system and the backup system are kept running in two locations -- on site, and in a backup location outside of manhattan (not to hard to figure out where, if you try). one can function without the other, on both counts. The QA done to ensure validity is INSANE. Nothing can even be brought out on to the floor without making sure it won't corrupt any of the systems (including the bluetooth system that has become the lifeline of the trading floor -- largest (in size and traffic) private bluetooth network in the world, when I was working the floor)

Re:NYSE: Microsoft (R) powered (R)

[Yoooder]

I was just thinking how funny it is that Microsoft has helped "THE NYSE ACHIEVE RECORD RELIABILITY!!!" The slow switch-over is a feature though, it allows the emulated 1.5 bit vacuum tubes in the Windows Flux-capicator kernel module DLL INI to resync with electromagnetic partition in the data-time continuum

Re:NYSE: Microsoft (R) powered (R)

[Anonymous Coward]

Actually, SQL Server 2005 is used by NASDAQ, not the NYSE - they are two completely different trading systems. The Dow Jones is an index of select stock listed on the NYSE.
The machines that calculate the Dow Jones Industrial Average are IBM hardware running Unix.
SQL Server running NASDAQ: http://www.windowsfs.com/eNews/tabid/112/articleTy pe/ArticleView/articleId/933/Securities-NASDAQ-Mig rates-to-SQL-Server-2005.aspx
IBM Unix Machines running NYSE and calculating the Dow Jones, as pointed out by another poster: http://www-03.ibm.com/industries/financialservices /doc/content/news/pressrelease/1567015103.html

Re:NYSE: Microsoft (R) powered (R)

[Not The Real Me]

Fair enough, from your link "The System p5 570 servers running AIX feature IBM's POWER5 processors, which deliver 64-bit computing power...The system is designed to scale with both UNIX and Linux enterprise applications on demand. Dow Jones Electronic Publishing and an extensive IBM team worked closely to enhance Dow Jones application performance. "

Since IBM is one of the biggest boosters of Java and Java application servers, might we not infer that this is a hiccup in IBM's Java application servers (aka IBM WebSphere)?

I have to wonder if this is spam related?

[erroneus]

It's a very strange and vague situation. I'm not sure we'll ever hear what actually happened. "Computer related" does tend to make one a bit frightened as to what that means exactly. It would be over-estimating the impact of stock-spam quite a bit, I should think even to consider whether or not that had any relation to the problems of yesterday. But if it did, perhaps we can now see some real action against spam... we can only hope.

Re:I have to wonder if this is spam related?

[gunnk]

From what I read (NY Times, I believe) the system didn't really have a big an impact as some headlines would lead you to believe.

Apparently the system that computes and displays the current Dow couldn't keep up with the systems that process the transactions when the number of transactions became very large. The display system caught up a bit later making it *appear* that the market at suddenly dropped something like 250 points in a few seconds.

In reality, the decline was fast but steady. It was just the exchange's version of "Damn lag!"

Re:I have to wonder if this is spam related?

[LGV]

All of the stock spam I get is for companies that don't trade on the NYSE or NASDAQ, it's for much, much, much smaller companies that trade on other exchanges. The companies that make up the Dow Jones average (30 of them) are way too heavily traded to be swayed by pump and dump spam, so they don't bother.

Re:I have to wonder if this is spam related?

[DrWho520]

Change in Chineese trading market regulations were the cause of this drop. A massive sell off occurred with the beginning of a crackdown on questionable and illegal trading on China's stock market. This rippled to every other market in the world. Asian, North America, South American and European markets were all affected. Blaming the computer systems is damage control.

Re:I have to wonder if this is spam related?

[Jonny do good]

The drop was really fueled by a number of causes. China's 9% decline the night before was the primary trigger. Sub-prime loans have been leading to trouble for a number of firms lately with the housing decline fueling those problems. The market has been in bull form for quite some time with no corrections leading to a large number of stock prices not supported by their fundamentals. The durable goods sales reports are expected to show under 3% growth when it has been up in the 4% range and this always spooks investors. Any economic indicators showing any sign of change spark massive changes on the market.

The computer problems experienced were really just a lag between the DJIA being calculated and the massive volume of trades being made. Individual stock prices were being reported correctly but the index wasn't keeping up. When the computers caught up they did it over a single minute dropping about 300 points while in reality by the time the index caught up the market had started to rebound a bit. All of the value stock buyers saw the deals becoming available when the landslide hit and started buying a bit. Kind of like today, the market is rebounding because many are looking at stocks that were overpriced yesterday and thinking they are cheap. It's not really as big of deal as the press makes it out to seem. It's not like the '87 crash where 500 points was like 20% of the market. 500 points off th dow is under 4%.

Another trigger for the sudden decline could have been the headline on The Drudge Report (linking to the New York Times article by the same headline) stated that Greespan predicted an imminent recession when his words were as they always have been and that people should be carefull because the economy has been growing for longer than the average growth cycle by about 12 months. Greenspan didn't say anything about a recession being imminent.

Re:I have to wonder if this is spam related?

[Knara]

Can you post more info / links about this?

Re:I have to wonder if this is spam related?

[DerekLyons]

Change in Chineese trading market regulations were the cause of this drop. A massive sell off occurred with the beginning of a crackdown on questionable and illegal trading on China's stock market. This rippled to every other market in the world. Asian, North America, South American and European markets were all affected. Blaming the computer systems is damage control.

if you actually bother to read TFA - you'll find they don't blame the computers for the drop. They blame the computers for creating the perception of a cliff rather than a steep slope.
 
This is actually something fairly important to consider as more and more of our life interfaces with displays mediated by computer. (Even down to the mundane. I've discovered the digital controls in my oven seem to use a time weighted average - which works fine in convection mode, but it enlarges the deadband in normal mode.)

Re:I have to wonder if this is spam related?

[g-to-the-o-to-the-g]

Can you post more info / links about this? Here and here

Re:I have to wonder if this is spam related?

[DrWho520]

I prefer Bloomberg for my financial news. Straight down the middle.
http://www.bloomberg.com/apps/news?pid=20601103&si d=arfY.lBSosN4&refer=news

You mean "Yesterday"

[ReidMaynard]

Today it's up over 60 points so far...

Re:You mean "Yesterday"

[Short Circuit]

When he submitted the story, it was still "Today". I've seen it take as many as four days between when a story was submitted, and when it was posted.

Keep an eye on the Firehose, if you're a /. subscriber, and you'll see what I mean.

Re:You mean "Yesterday"

[ivan256]

You don't have to be a subscriber to see the Firehose.

I ran out of points ages ago, and I can still see it.

Re:You mean "Yesterday"

[Short Circuit]

Odd. It used to be a subscriber feature.

Re:You mean "Yesterday"

[scottv67]

Keep an eye on the Firehose, if you're a /. subscriber, and you'll see what I mean.

I've never been a subscriber yet I am invited to drink from the Firehouse on a weekly basis.

This is why...

[tomstdenis]

I spend a lot more than I save :-)

The Dow may go down, but a pizza is still as tasty hehehehe.

I wonder how much of this load is due to low volume day trader movement?

Tom

Re:This is why...

[Erwos]

You joke, but in high inflation economies, this is not uncommon.

I'd also point out that a one-day blip, even a big one, doesn't really mean very much. "Savers" win and lose in the stock market by long-term trends, not the short-term.

Re:This is why...

[pkulak]

Once again, the conservative, sandwich-heavy portfolio pays off for the hungry investor!

Re:This is why...

[bberens]

Maybe, but my portfolio just took a dump on the order of about 4% of total value yesterday. It will likely take several months to get back that single day's loss. If I were near retirement and had $1 million in there I'd be pretty upset at losing $40k in a day (almost a year's worth of retirement money). This of course ignores the fact that at or near retirement my money would all be in bonds for the most part and I'd be fairly impervious to these spikes...

See...

[ZonkerWilliam]

This is what happens when I sell one lousy share of google!

Chaos theory, anyone?

[pzs]

1. Computer switch-over is a bit slow

2. Market starts to waiver

3. Other parts of the market see this tremor so market waivers a lot

4. Panic ensues

5. Indices drop 10%

6. a pension company goes bust

7. my grandpa doesn't get to eat.

The last few steps are somewhat hypothetical, but still. The stock market must be one of the most immediately visible examples of chaos theory kicking humans in the nuts.

Peter

Re:Chaos theory, anyone?

[Nykon]

A butterfly flaps it's wings in Asia , and the dow drop 400 points in the US... brilliant.

Your Nobel Prize is in the mail. Don't call us. We'll cal you.

Re:Chaos theory, anyone?

[pzs]

> A butterfly flaps it's wings in Asia , and the dow drop 400 points in the US... brilliant.

Of course, the usual moron response to this is to say "why don't we just kill all the butterflies in Asia?"

Peter

Re:Chaos theory, anyone?

[Lord Ender]

If an index has been trading near a certain level for a while, then a "panic" event causes a huge drop without changing the fundamental underpinnings of the market, traders view this as a HUGE signal to BUY BUY BUY, on margin if possible. In a few weeks, the index is back where it started before the panic event.

At least, that has been my observation. I can't WAIT for leveraged index ETFs... come on, ProFunds!!

Oh, and your Grandpa's pension would not go bankrupt over a panic event. That's absurd.

Of course, anyone who relies on pension companies for retirement has bigger problems...

Re:Chaos theory, anyone?

[Phreakiture]

8. . . .

9. Profit!

Re:Chaos theory, anyone?

[stefanlasiewski]

0. Diversify your holdings, especially your retirement accounts, to protect against market fluctuations. The stock markets will always go up and down for a thousand different reasons-- computer glitch, bad news, hummingbirds, whatever.


1. Computer switch-over is a bit slow
2. Market starts to waiver
3. Other parts of the market see this tremor so market waivers a lot
4. Panic ensues
5. Indices drop 10%


5a. If your investments are diversified, you will survive when the Indexes drop 10%. This is especially true for long-term investments.
6. Buy low.
7. Wait for a while.
8. Sell high.
9. Profit!
10. Enjoy your retirement.

Re:Chaos theory, anyone?

[UbuntuDupe]

As opposed to investing in the stock market yourself?

Saving has become very, very simple recently. There are now "target-date" or "life cycle" funds that divide your investmetns into asset classes appropriate for your intended retirement date. Simply put your savings in the date closest to when you plan to retire, and it automatically diversifies for you and keeps your appropriate asset mix given how much time you have. They're available in most 401(k) plans (though some, *cough cough* assume an older workforce than they really have).

Re:Chaos theory, anyone?

[FirstOne]

"A butterfly flaps it's wings in Asia , and the dow drop 400 points in the US... brilliant."

More likely a disk drive started to fail, but was able to recover. They can do upward of several hundred retries over 5 to 10 seconds before they return a fatal error.

If the disk drive completes the operation after just a couple of dozen retries, they'll keep on going without reporting an error. If no error is reported back to the controller the raid functionality usually won't kick in (depends on the design). The disk drives data rate and performance during these soft failures drops drastically(100x) resulting in very slow processing times.

Re:Chaos theory, anyone?

[Anonymous Coward]

Bacon!

Re:Chaos theory, anyone?

[vertinox]

Oh, and your Grandpa's pension would not go bankrupt over a panic event. That's absurd.

I can't help but think the same thing was said during 1929 and then again in 1989.

Re:Chaos theory, anyone?

[narsiman]

You forgot the ZonkerWilliam selling one share of Google that was the root cause of this Chaos. Personally I dont like chaos. I would like to blame one person and let that one assign the blame on others!!

Re:Chaos theory, anyone?

[Lord Ender]

Banks don't invest 90% of their customers' savings in the stock market these days. Banks also carry insurance. So 1929 doesn't apply.

I wasn't trading in 1989, but I don't remember stepping over starving retirees on my way to school. I think kids these days know that they should keep most of it in index funds when they are young, then move it to government bonds when they are reaching retirement. To rely on a single company that relies on the stock market /while in retirement/ seems nuts to me.

Re:Chaos theory, anyone?

[science_gone_bad]

It's already been done!
Benoit Mandelbrot wrote it up in Scientific American a few years ago (Feb 1999...sorry I didn't link, the only one I could find was pay-per-view). He was even able to match the stock market trends from 1929 - today.

What I learned from that is the BEST way to play the stock market is completely random since ANY change to the starting conditions of a chaotic system changes the system overall. I'd always wondered why those News shows where they gave $5000 to a stock broker and picked $5000 of stock randomly were always won by the random method.

Not that I do the stock market outside of my 401k, etc. I just don't have the gamblers mindset, and don't want to randomly throw money around.

Re:Chaos theory, anyone?

[science_gone_bad]

Sorry forgot to get the name of the paper:

"A Multifractal Walk down Wall Street"

Re:Chaos theory, anyone?

[UbuntuDupe]

You're terminology's a bit imprecise. "Index funds" are not mutually exclusive with "government bonds". There are government-bond index funds. What I think you meant was that you should keep most of your money in stock index funds, and then gradually transition to high-grade bond index funds. Of course (and you didn't specifically contradict this) you should always have some in stocks for the capital growth so as to keep up with inflation.

To rely on a single company that relies on the stock market /while in retirement/ seems nuts to me.

Exactly. Unfortunately, that's not as obvious as you might think it is, for most people. Some were raised on a "every big corporation will always reap huge profits" mentality...

Re:Chaos theory, anyone?

[Lord Ender]

I said "ETFs." Not the same.

Re:Chaos theory, anyone?

[auspiv]

levereged index etfs-

Double weight SPY - SSO
Double weight QQQQ - QLD

The Short QQQQ - PSQ
and Short S&P - SH
and Short DJI - DOG

I'm actually suprised

[jhfry]

Considering the amount of, and importance of, data that flows through that system... I am surprised that it's not routinely well ahead of the needs at peak capacity.

I'd say that someone, likely the one in charge of the IT budget approval, keeps tight purse strings. Of course, he's not the one getting reamed, it's the CIO and his crew who are taking the blame even though they have repeatedly requested the funds to improve the system. Just speculation, but likely spot on.

Just another piece of ammo when I start a new job and demand a reasonable budget.

Re:I'm actually suprised

[Overzeetop]

Imagine how easily this could have been avoided had they funneled just half of the board's compensation into IT expenditures.

Re:I'm actually suprised

[gEvil (beta)]

Yeah, but where are they going to get money to put into their budget? I mean, it's not like they can just funnel money into whatever accounts they want... : p

Re:I'm actually suprised

[Darth_brooks]

Considering the amount of, and importance of, data that flows through that system... I am surprised that it's not routinely well ahead of the needs at peak capacity.

The NYSE has historically been behind the curve as far as capacity to trade is concerned. I don't have links to back it up, but i recall a TV program that detailed how, before electronic trading was introduced, the exchange had to be closed during the week to allow transaction processing to catch up with the previous week's trading.

Re:I'm actually suprised

[L0rdJedi]

You probably saw it on the History channel. That's where I saw it. And it wasn't that they didn't introduce the technology sooner, it was that the technology wasn't available sooner. It wasn't really until the advent of the personal computer that they were actually able to do it. Before then, yes, the NYSE was closed for a few days each week so the people that were processing the trades could catch up with that weeks trades. Yes, it was PEOPLE doing the processing, not COMPUTERS like they have today.

Re:I'm actually suprised

[chad.koehler]

Data rates have doubled at the exchanges in the recent past, and they are likely to double again the the near future. Keeping ahead of this curve is not as easy as one might think...

Just one stock for instance, APPL will have millions of transactions in a single days trading... Including just trades and quotes you can see close to 100Mb of activity for a single symbol in a day.

We are constantly trying to increase capacity, but we're near a point where the only meaningful upgrades for capacity planning are new hardware. To top this off,most of the downstream (client side) applications are limited not by the hardware but by the available bandwidth.

Re:I'm actually suprised

[DerekLyons]

Considering the amount of, and importance of, data that flows through that system... I am surprised that it's not routinely well ahead of the needs at peak capacity.

Generally it *is* ahead of things at peak capacity - but there was a massive spike yesterday that briefly exceeded peak capacity.
 
One of the problems Wall Street has been dealing with for decades is a tail chase with itself - every time they up their computer capacity, trading volume expands to consume the capacity. What was shortly before a comfortable capacity margin is now consumed in the regular course of business - which means the ability to deal with peaks over average goes down. They upgrade their computer sysems and.... lather, rinse, repeat.

Re:I'm actually suprised

[jhfry]

You speak as though you are somehow involved... if so, your admittance that data rates have doubled and will likely double again means that they are aware that something needed to be done and have not done it.

In their industry, the NYSE cannot afford to be complacent... there is little reason for this to have become an issue, I could have predicted it years ago.

I realize that upgrades for that type of system are not something taken lightly, however they should have had a plan in place, and implemented it before things reached the point where the hardware couldn't keep up with demand. I don't care if that meant a full rewrite of the software and complete overhaul of the hardware, something should have been implemented well before the problem presented itself.

As a network manager, it's my responsibility to assess our future needs and ensure that the system I have will meet those needs. If not, I immediately begin to plan an upgrade, perhaps even implementing it if budget allows. The systems I manage are not "as critical", but I still treat them as such.

With current technology, there is really nothing preventing them from developing a truly scalable system, that can easily grow as needed. Or they could at least deploy a system with 10x the current demand and start planning it's replacement immediately thereafter.

What they experienced should NEVER happen on a critical financial system... too much of our economy is dependent upon it. The resources to prevent it are there, I'm sure the Fed would even offer a loan if they had to. I am confident that, if this issue was indeed caused by an overloaded system, someone dropped the ball in a bad way.

Re:I'm actually suprised

[chad.koehler]

I am involved only tengentally, so I can't speak for exactly what NSX ran into. I'm only saying the issues that I am currently seeing with our systems. We are taking every precaution to make sure that the same thing DOESN'T happen to us. One thing to note, these systems are (in general) running on the most cutting edge systems that money can buy... Hardware upgrades are usually deployed as soon as they are certified, which may only be weeks after a release. When you are already paying 6 figures for a single machine, there isn't much vertical mobility in terms of hardware selection.

Re:I'm actually suprised

[jhfry]

I realize that when your dealing with big iron or even high end server solutions there is little more to do than wait for the next best thing... or innovate.

Put out an RFP for someone to design a cluster based system that will perform the same functions the system currently does, but using OTS hardware in a massive single or distributed cluster configuration. Perhaps they use virtualization and have one VM for every symbol... then purchase enough hardware that only the least active symbols share machines while those with the most activity are alone on a machine. They could even be dynamically moved about as loads fluctuate.

Sure it'd be expensive... but a massive cluster of low end machines, even at $2000 a pop can still bring a lot of power and stability to the mix... and best of all, adding more power is as simple as plugging a new one in.

Anyway, it's besides the point... I just believe that someone somewhere really screwed up to let things get so bad. Perhaps it's the PM who selected the software vendor, perhaps its the CIO who hasn't done what needed to be done, or perhaps it's those with the money who didn't listen to their IT people. I personally think it's the latter case, as I have seen that attitude at all levels of business... even a 1 man company can fail to allot the proper funds to IT and let essential services go ignored (like backups).

I have begged and pleaded with bosses that it's necessary to increase capacity prematurely... but often they counter with, come back when something is broken. Then when I do, it's "What do you mean it will take 3 months to implement a fix? Didn't you see this coming?" I still believe this is what happened. Sure someone saw the problem, and knew how to fix it.. but they couldn't convince the man with the cash to cough some up and get the ball rolling. I bet the vault is open now though!

Re:I'm actually suprised

[chad.koehler]

You're right, of course. Someone dropped the ball. However, almost every company in the industry is taking notice and sparing no expense to increase their performance. These things to take time though, and now its a race against the volume. Should be an exciting (from a development perspective at least), couple of years.

Re:I'm actually suprised

[caffeinex36]

What CIO? haha

The Highly Reliable Times

[maharg]

Dow Jones & Co., the media company that manages the well-known index of 30 blue chip stocks, said it discovered shortly before 2 p.m. that its computers weren't properly handling the day's huge volume in trades at the New York Stock Exchange.

It switched to a backup computer, and the result was a massive swoon in the index as the secondary system took over processing shortly before 3 p.m.

Highly Reliable Indeed !!!!

Re:The Highly Reliable Times

[Jello B.]

Say, why'd you get me a subscription to The Highly Reliable Times? That's not a reputable journal of opinion.

Just like the computers in 1929!

[Rob T Firefly]

Sure, blame the computers. After all, it's not like the market system has shown any penchant for random and pseudo-random ups and downs ranging from negligible to ultraviolence since long before before any figures were computed by machines rather than pencils.

wonder what the reactions would have been like if a "computer glitch" knocked the thing up 500 points instead of down.

Re:Just like the computers in 1929!

[UbuntuDupe]

wonder what the reactions would have been like if a "computer glitch" knocked the thing up 500 points instead of down.

Um ... *pulls aside*

You mean like what happened two weeks ago?

Re:Just like the computers in 1929!

[Rob T Firefly]

Neat! Ask and you shall receive! Thanks for the answer.

Re:Just like the computers in 1929!

[UbuntuDupe]

Glad to help :-) Sorry if I seemed rude, I was going for dramatic, but it doesn't always carry through in ascii.

Re:Just like the computers in 1929!

[PPH]

Sure, blame the computers.

Typical corrupt capitalist reaction. In Soviet Russia, computers blame you!

Fueled nothing

[AlphaNuRho]

I don't see how you could say that the computer problem fueled the plunge. My understanding of the events is that the only problem was with the system that calculated the Dow Jones Industrial Average Index (the number that is around 12,200). There wasn't a backup or delay in execution of trades or anything like that. The decline was real, but it was spread out over an hour instead of the 2 minutes reflected by the DJIA.

Traders still bought and sold stocks at their real value in real time. The calculation of the sum of their activity was the only thing delayed.

Re:Fueled nothing

[Maxo-Texas]

Actually, if you look at the chart, as soon as the computers caught up and showed how low the market had gone, folks started buying.

They were selling based on the false impression that the market was not down enough yet.

Re:Fueled nothing

[complexmath]

From the article:

"Some of the books froze up," he said, referring to the systems in which traders place their orders. "You couldn't really trade. You couldn't really make sales." He said orders appeared to become backed up. "Once they unfroze the Dow fell."

It sounds like at least some of the trade processing systems rely on the DJIA calculator in some way. Perhaps they query it to report the DJIA at the time of the transaction or some such. So trades were being processed but the feedback mechanism was frozen so people weren't sure if their trades were actually going through. It likely took them a while to realize this, so there was a backlog of trades in the system once people stopped to figure out what was going on. Timely feedback is obviously crucial to trading because decisions are made based on current market conditions. It's a good thing more automated "sell" triggers weren't fired when the DJIA caught up or this could have turned into quite a mess.

Re:Fueled nothing

[nelsonal]

Most of the folks with any real money (the ones who can move markets when they start buying) are calcing the S&P themselves, because it's faster than waiting for the index to update. My first internship was writing stuff that did that, it was cool to see the S&P a few 10ths faster than it was on the television. Almost no one in the investment world looks at the DJIA except for sentimental value. It's a terrible index but really easy to calculate (compared to other more complex and useful indicies). Besides, it was obvious there was a problem the Dow was only down 2% but almost all stocks were down 3% or more by 2 in the afternoon.

not quite

[flynt]

The computer systems weren't responsible for the overall drop, but rather the rate of the drop during the few minutes of switching over to the backup computers. This queued up trades, and at the current volume of the switchover, caused a large drop when they caught up. At least that's how I understand it.

Re:not quite

[micah_hainline]

The issue with the computer systems did not create the decline, and the drop it reported was accurate, but by delaying notification of the size of the downturn and then updating it all at once, it made it look like there was suddenly a dramatic increase in the momentum of the decline. As we know, the market is driven by confidence in the market. This sudden drop scared some into selling a few more shares. This was just a very small part of the overall decline however.

Real Cause?

[green453]

I'm all for looking at things from the tech/computer geek side of things as much as anybody on Slashdot, but isn't the summary taking things a bit far? It was mentioned that there may have been other causes that combined with computer glitches, but wouldn't the fact that markets in China dropped a whopping 9% yesterday seem to be the real cause? I'm sure swithcing computer systems may have scared a few people, but I doubt it was the primary cause of a 400pt drop. That said, it is interesting to think about the effect of computer systems on the financial markets. I've always maintained that it isn't the politicians or the business owners or the economists that run the world, it's the engineers. Think about what would happen if there was a complete shutdown of the systems that run the markets. See if all the Wall Street profiteers pay their geeks a pittance of their "annual bonus" then...

Re:Real Cause?

[zuiraM]

There may be some truth to this, but there is nothing near that level of coordination in the IT world at present. If there were, I'm pretty sure something would be done about it. Fast.

Either way, giving any indication that you are aware of and/or have considered this line of thought, is a pretty surefire way to make them want to tighten your leash, or fire you outright.

Re:Real Cause?

[TubeSteak]

My hypothesis is that the lag in the system took the perfect out of "perfect information".

It might seem somewhat simplistic, but since 'the market' didn't know it was in free fall, 'the market' couldn't correct.

The lack of information kept people from buying while stocks were steadily slumping, China or not.

Dropping prices versus dropping data

[G4from128k]

The problem was obvious to anyone watching the markets. A trace of the Dow versus the S&P showed that the Dow's drop was NOT keeping pace with the drop in the S&P (they are normally tightly correlated, especially when big moves occur). It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop. When that backup server cut in, the Dow data suddenly reflected the true state of affairs that was obvious from people watching the S&P and the broader market.

The Dow did NOT drop 200 points in minutes, the data simply caught up with the drop that had already occurred.

Re:Dropping prices versus dropping data

[richg74]

From what I've read, and learned from talking to a couple of former colleagues (I worked in IT on Wall Street for 20+ years), your note is almost but not quite right. You correctly point out that the Dow's decline was, for a time, seriously out of line with the decline in other market indices, such as the S&P 500. However, you went on to say:

It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop.

In fact, the problem appears to have been that the systems at Dow-Jones -- which owns the DJIA index and calculates it -- could not keep up with the volume. When the backup system came online, the reported index showed a significant drop in a very short time. In actuality, the decline was real, but it had already happened over a longer period of time. As it says in the original article:

"The market's extraordinary trading volume caused a delay in the Dow Jones data systems," said Dow Jones spokeswoman Sybille Reitz. "We decided to switch over to the backup system, and the result was a rapid catch-up in the published value of the Dow Jones industrial average."

There's no indication that there was a problem with trade reporting by the NYSE, which would be a much more serious problem. But the data feed from the exchange reports prices trade-by-trade as they occur. So, if an external system like Dow-Jones's gets behind, it usually has to plow through the update stream to get current, which can result in a sort of "compressed time" effect.

Re:Dropping prices versus dropping data

[rcamera]

all right except one point - the nyse didn't have a problem during that time. dow jones had the problem (they are a seperate company). if nyse had a problem reporting trades, then the s&p would have shown the same problem. i like to pick on nyse as much as the next guy (you should see some of our slippage numbers...), but this time it was not their fault. why is everyone under the impression that they are?

Re:Dropping prices versus dropping data

[C_Kode]

why is everyone under the impression that they are?

Because a lot of people don't know the difference.

Don't see how they could.

[Kadin2048]

I wonder how much of this load is due to low volume day trader movement?

I would guess, virtually none, since they're by definition low-volume?

This blaming it on computers seems mostly a red herring. The markets in Asia (particularly Shanghai) tanked, and as a result, the markets in the US tanked, because companies in the US are heavily invested in China.

I think the only lesson here, in case there was anyone left who didn't get it, is that we all float or sink together. For better or worse, the US has tied itself pretty tightly to the Asian markets, and if they collapse, we're going to be seriously hurting.

Re:Don't see how they could.

[tomstdenis]

low volume means how much stock you are moving.

Do you really think the traffic is higher if you move more units of stock? What I meant is there are probably millions upon millions of people moving $200 worth of stock here and there. Versus the tens of thousands moving significant quantities.

Tom

Re:Don't see how they could.

[Kadin2048]

Yes, but what I'm questioning is whether the Dow Jones' computers really had anything to do with this whole market movement at all. At most, all they did was slow down, so that the DJI lagged behind the real world for a while, and then suddenly caught up when their backup system went on-line and took over.

I think that it's more of a symptom and less of a cause. The cause of the market movement was in Asia; that made people sell, people selling caused the DJIs computers to suck. Now, perhaps the DJIs computer slowdown, and consequent large jump when they fixed the problem and got the backup running, caused more people to sell, but this seems specious. The slump was already in progress by the time that the computer slowdown occurred, because the slowdown was driven by high trade volume.

So my point is mostly that I don't see how it matters, really. People are looking towards computer glitches as the cause for the 3% drop in the market (or whatever it was), and that's just not true. The computer glitch might have made the drop look worse, or more precipitous, than it actually was at one point during the day, but it didn't cause or really drive it in any significant way. Even if the DJI folks' computers had worked perfectly, the market would still be sucking. In fact, computerization and the consequent flow of information is what links markets; it's only in the last few decades that the Asian and US markets have felt each other's pain so closely, so in a way, you can blame the computers for working too well in general, when you get these domino-effect deflations.

Re:Don't see how they could.

[illumin8]

Now, perhaps the DJIs computer slowdown, and consequent large jump when they fixed the problem and got the backup running, caused more people to sell, but this seems specious.

No, what really happens is that a lot of the big money has automated trading systems. These automated trading systems look for key criteria like quick movements in stock and buy or sell to make money (or avoid losses) on the trend. When a big movement happens in a short period of time, it creates a feedback loop where big movement causes automated trading systems to issue sell orders, and a glut of sell orders causes the downward movement to increase, thus triggering another round of automated sales orders. FYI, automated trading systems were blamed for the October 1987 crash as well.

The moral of the story is this: When automated trading systems take the market data as gospel truth (they have to, what else will they trust) and that data turns out to be inaccurate, bad things happen. Thus the need for more accurate and timely market data.

Re:Don't see how they could.

[Lawrence_Bird]

er.. not in this case. Their 'explanation' is that they had fallen behind, which in fact was the case as when DJIA was "down" 295, futures were already down 400 and sinking. In fact, I commented to another trader what a nice arb was out there if only one could short on the downtick in the cash market. Note also that there were already some limitations in effect on automated program trading (tick rules)

In the event of a 180-POINT DECLINE in the NYA, all index-arbitrage sell orders of the S&P 500 stocks must be stabilizing for the remainder of the day. Collar will be removed if the NYA moves back to within 90 points of the previous days close.

At one point the NYA was down 400 pts. This 'blame it on the computer' or 'it was china' are BS. Stocks were over extended. While any one thing might have some effect, the time was ripe and the buyers could no longer hold the bid in the face of china sell off, weaker econ data, Greenspan, Afghanistan, Irock, etc etc simultaneously. There is too much leverage in use, too much complacency and just too much liquity. When all rush for the door in the face of trouble this is what happens.

Re:Don't see how they could.

[illumin8]

In the event of a 180-POINT DECLINE in the NYA, all index-arbitrage sell orders of the S&P 500 stocks must be stabilizing for the remainder of the day. Collar will be removed if the NYA moves back to within 90 points of the previous days close.

Thanks for the explanation. I was mostly guessing of course based on past history. I studied this a bit although I'm not a trader by any stretch of the imagination. Looks like they've instituted safeguards (the collars) to prevent people from making too much money off of the fluctuations. Otherwise automated professionals would have cleaned up yesterday, just as is the case in most downturns.

"Load"? That's exactly what I was thinking

[Mateo_LeFou]

yesterday's news was annoying as hell. Everyone and their dog chimed in on what caused this horrible crash, what investors should do now, how bad it might get, etc. People: the market's been soaring for months. This is a perfect example of broadcasters' attempt to get you afraid and addicted to "news".

Re:"Load"? That's exactly what I was thinking

[tomstdenis]

What you say? They done lie to us to drum up bidnez?

You know this, I know this, some others may know this, but the problem is people let themselves get caught up in it. Very few people really give a damn about gay rights [for example, one way or another] but they'll sure as damned have an opinion about it [usually supplied by the media or some ass on the medium],

People know the news is fake, at some level, they just *need* to believe it. So long as the bad guy is someone I don't understand, and all those "bad things" are happening elsewhere, I feel safe, warm, and fuzzy by comparison.

Tom

Re:"Load"? That's exactly what I was thinking

[A_Non_Moose]

yesterday's news was annoying as hell. Everyone and their dog chimed in on what caused this horrible crash

WOOF! NYSE crash after failed Vista Migration BARK!

Details soon. Don't touch that dial GRRRRR! Bad viewer, bad!!

Hey, you're right, that's what my dog told me!

Cause Was China

[Bellum Aeternus]

There may have been a computer error that made things look interesting, the real culprit was China's bad day at the stock market. It created a World stock slump which drove down stock markets all over the globe. So for those of you who don't RTFA, there wasn't anything nefarious going on.

Re:Cause Was China

[internetizen]

is some FUD by the pundits as i see it. It was a *rumoured* change in regulations, the fact that they might start instituting a capital gains tax, which was quickly dismissed in the morning. The best commentary so far is from Tom Easton from The Economist, on NPR's Morning Edition. The Chinese gov't is the biggest shareholder, and to suggest that happened overnight was ludicrous. Easton mentioned that foreign ownership of the chinese market is about 10 billion dollars, a drop in the bucket (160 billion was lost in the chinese markets yesterday) Fundamentals didn't change overnight, and the Chinese economy didn't suddenly stop growing, it was more the fact that the image of China as a mystical market is now flawed. More importantly, investors - retail and institutional - need to understand how disconnected the stock market is from the economy in China. It is a speculative market, and for the developed markets to blame and react to China shows clearly how ignorant or jittery the western markets really are.

Re:Cause Was China

[Valdrax]

Just and aside, but I'm frequently bemused by/frustrated with people who seem to search for the single point of failure in any problem.

Re:Ok...

[BanjoBob]

I thought they were still on Windows Server 2003 running MS SQL Server.

Now, had they been running Linux and MySQL on an IBM monster box things would have probably been different

Editors? We don't need no stinking editors.

[dougman]

The DOW is up today. Can't you at least get the most basic facts right? The drop was yesterday.

Twilight Zone: Tom Clancy

[maroberts]

As well as crashing planes into buildings, it seems "Debt of Honor" is getting good at being an oracle of modern times.

Re:Twilight Zone: Tom Clancy

[Da Fokka]

I'd hope not - one of the terrorist acts described in the book is a major outbreak of Ebola in the United States.

Re:I smell conspiracy

[radish]

Errmmm....if trades were halted, how exactly would said investors pull out?

IBM backend w/ Linux Workstations used here?

[xxxJonBoyxxx]

If I'm reading this right, an IBM back-end system (mainframe) with lots of IBM-delivered Linux workstations were in the mix here. Anyone know for sure (i.e. work there)?

http://www.internetnews.com/bus-news/article.php/3 447741

Sooner than think.

[Anonymous Coward]

Several economists have said that America is bankrupt. In addition, they have pointed out that countries are quietly dropping dollars. Once it is realized by the general public, many will want to shed theirs, probably moving to euros joining Bill Gates, warren buffet, and other billionaires. At that point, Berneke will have no choice but to slowly bump prime up until it is 11-20% to get investors to stay. At that point, our economy will truly slow. Hopefully, at that time, GWB will be forced to get a real backbone and deal with China and get them to untie their money from ours.

Re:Sooner than think.

[Red Flayer]

At that point, our economy will truly slow. Hopefully, at that time, GWB will be forced to get a real backbone and deal with China and get them to untie their money from ours.

1. China has already unpegged from the dollar, they now peg to an index of currencies (I know, with the dollar still the main component).

2. What makes you think that's going to happen during the current administration? We both know that any negative actions will be held off on until the current "what-me-worry?" administration is gone. They'll let someone else take the heat.

3. And raising prime that far will have the beneficial effect of causing inflation, which will devalue our debt. The downside will when ere high prime will cause inflation, but the returns on business investment lag behind government securities causing a shrinking economy (stagflation). We've been through it before, it's not pretty -- and it's what gave us Reagan in the White House, for better or worse.

Where by various financial reasons...

[Etherwalk]

The Chinese market took a bit of a nosedive (around 9%) after (1) profit-taking from a record-high the day before when their market reached a psychologically nice place, and (2) the rumor that they were going to start charging capital-gains tax, which we go here. China's place in the world market is important enough that that triggered a worldwide economic hiccup that wiped out most of this year's economic gains worldwide.

Other contributors were low durable goods orders and Greenspan's warning on monday of a recession as early as the end of the year.

Tell that ...

[vlad_petric]

To super-duper, high energy alpha particle and gamma rays. Why do you think that ECC is so widespread? Not only for RAM - most processors these days have their caches ECC-ed as well.

Not just Dow Jones

[mi]

Trying to access my account at Fidelity is also slow and with intermittent failures today and yesterday.

The "snowballing" is not in the prices moving too much up or down, but simply in the increased activity as a lot of people are trying to check their accounts and trade in and out of positions...

It's voodoo

[PIPBoy3000]

Yep - that sounds just about right.

Imagine a series of database transactions, with each step getting queued up and waiting for the system to finish processing it. The actual DOW number reflects fully completed transactions, but not pending transactions that might impact the outcome. This is probably a good thing, as a transaction might end up being rejected, so you only want to show the outcome of completed transactions. Once the backup system came online, the transactions quickly finished being completed, resulting in the dramatic drop.

The amazing thing to me is that the system is robust enough that transactions can survive the loss of their main computer system and bringing up a secondary one. That's database, networking, and coding voodoo, all wrapped into something pretty awe-inspiring.

Re:It's voodoo

[baffled]

If the system is a black box, and trade activity is the input, then build two independent black boxes and connect them in parallel with the input. If one system goes kooky, try the other one! I think the space shuttle systems were built the same way.

Re:It's voodoo

[Aceticon]

The amazing thing to me is that the system is robust enough that transactions can survive the loss of their main computer system and bringing up a secondary one. That's database, networking, and coding voodoo, all wrapped into something pretty awe-inspiring.

Pretty much the number one consideration when designing a technical architecture for a mission critical is to guarantee that no transaction is lost.

Such an architecture is not too hard to design.

The really hard part is to make one that can process millions of transactions per minute.

Re:It's voodoo

[WeblionX]

Nah, the space shuttle has five computers. It's an entirely different kind of computing; all together!

Re:It's voodoo

[D-Fens]

It's an entirely different kind of computing.

Re:It's voodoo

[C_Kode]

Pretty much the number one consideration when designing a technical architecture for a mission critical is to guarantee that no transaction is lost.


With todays databases like Oracle losing a translation just doesn't happen. (for the most part anyhow) What is cool is to have a fail-over that keeps in sync with the production DB, will fail-over, run the logs, and come online in mere minutes with a system this size and with this load. No matter what, it's impressive even if it's effect was visible to the users. Of course, they always could do better. :)

Re:It's voodoo

[Ctrl-Z]

It's an entirely different kind of computing.

Re:It's voodoo

[yoha]

wrong - transactions were not delayed. Dow Jones is a media company - they publish the results of transactions. They're reporting was delayed.

What system?

[UnknowingFool]

It was never mentioned which system crashed. I guess the NYSE doesn't want to embarass the company. I wonder if it was their Windows 2003 with SQL Server system.

Re:What system?

[SuseLover]

Didn't the NYSE use Solaris running Oracle or other heavyweight db in the past. I heard they were switching to WinTel systems in recent years. If true, that shows how robust Solaris/Sun/Oracle is compared to the MS solution.

Re:What system?

[rcamera]

Didn't the NYSE use Solaris running Oracle or other heavyweight db in the past. I heard they were switching to WinTel systems in recent years. If true, that shows how robust Solaris/Sun/Oracle is compared to the MS solution

how does this say anything about the systems at nyse? nyse was flooding dow jones with too many messages. nyse didn't have a problem - dow jones had the problem. it seems like a lot of folks posting to this story don't know the difference (no - i'm not new here).

i also don't see why this is such a big deal. anyone who actually trades based on index values calculates the values themselves instead of waiting 15 seconds for an index quote to be broadcast. as evidence of this, note that index futures and spdrs (dia in this case - i think this should work. if not, compare it to the ^DJI) were in-line with the actual dji price instead of the broadcast price for the entire time.

Hal? HAL!!!

[Malakusen]

Let me put it this way, Mr Amer. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error.

Re:Ok...

[Phillup]

Now, had they been running Linux and MySQL on an IBM monster box things would have probably been different What do you think the "backup system" was running... and how they knew the primary system wasn't working properly?

;-)

Seriously tho, I seem to remember reading at one point in time about an exchange that moved their *nix based system over to an MS based system... and kept the old system as a backup.

Don't know if this is similar.

But... we do know that they had another system running, and the two systems didn't jive... and the primary system could not keep up.

Switching over to the backup system cleared the log jam almost immediately.

Makes me question which one should be the primary...

Re:I smell conspiracy

[rcamera]

not to mention the fact that none of the names in the dow were halted yesterday.

Glitch didn't fuel anything

[bgutierrez]

I can't see any way this glitch fueled any part of the market "collapse." We should remember that the Dow Jones is only a composite of actual stocks. It's an indicator. Would we say that it got cold outside because a thermometer got stuck and then suddenly plunged? The Chinese market dropped 9%, everyone got scared, and the DJI was a little slow in reporting the real values. It was corrected, and showed the market regain a little of what it lost. Up until 3pm, I was a little surprised at how well the Dow was doing in comparison to other indicators. If anything, this should have given investors a little confidence instead of making them sell so quickly.

Re:Switch to backup system... because of load!?

[funwithBSD]

Real companies with real computers have duplicate systems as backups.

Capacity on mission critical systems backups where I work is identical to the primary system.

Re:Switch to backup system... because of load!?

[Denis The SQL Menace]

Windows?? Are you sure? http://www-03.ibm.com/press/us/en/pressrelease/194 05.wss

Re:Switch to backup system... because of load!?

[misleb]

Real companies with real computers have duplicate systems as backups.

That isn't the WTF here...

Capacity on mission critical systems backups where I work is identical to the primary system.

This is the WTF. What is the point of switching to a backup, which has identical capacity, when the main is overloaded? You just end up with an overloaded backup. :-P

-matthew

Re:Ok...

[Denis The SQL Menace]

Oops did you say IBM? http://www-03.ibm.com/press/us/en/pressrelease/194 05.wss

Re:Switch to backup system... because of load!?

[funwithBSD]

One presumes that the overload is because of a fault, otherwise you are right, you would gain nothing.

Many of our systems will survive the failure of a single part and fence it off, like a cpu, SAN controller cards, or memory.

However, it will degrade performance.

Sorry I was not clear on how real computers work.

how often are WAG really spot-on?

[fool36]

Just speculation, but likely spot on.

Thanks for your in-depth analysis.

I second that...
I'd be interested to see just how likely it is that wild ass guesses turn out to be exactly correct.

Re:Awe-inspiring? Nah.

[flynt]

How is this in any way awe-inspiring?

It was designed to do this. It was implemented. It worked.


You answered your own question!

Damn it!

[Rohan427]

Darn WGA and Automatic Updates bit us again! ;)

PGA

The real problem

[Nirvelli]

The computers freezing didn't cause any drop in the stocks.
The real problem was that everybody was selling, so prices were dropping, but the computers didn't show the drop. Once the backup system kicked in, the displays all instantly caught up to what had been going on, and what everyone saw was a big drop.
Graph

Oblig Futurama quote

[Cervantes]

Dr. Zoidberg: Once again, the conservative, sandwich-heavy portfolio pays off for the hungry investor!

The Dow's technical glitch, dissected

[rrohbeck]

http://money.cnn.com/2007/02/28/markets/tech_glitc h/index.htm

The publisher of the Dow industrials said that a system problem starting at 1:50 p.m. ET on Tuesday, amid unusually heavy trading volume, caused a 70-minute lag during which the value of the market measure lagged the declines in the underlying stocks.

The subsequent downward spike in the Dow occurred when the problem was corrected as the company switched to its backup system at around 3 p.m.

Just before the switch, the Dow was showing about a 160-point drop. But then the blue-chip barometer appeared to tumble some 200 points in the blink of an eye as the newly available data was correctly reflected in the average.

Re:The Dow's technical glitch, dissected

[randomplanck]

When they say they coped with volume by switching to a backup system don't they probably mean a fall-back system? In other words, there was something wrong with their on-line production system? Either that or they have backup hardware with more horsepower than their day-to-day hardware which I think is unlikely. Or, by backup system they mean they added some scalable nodes. They probably had deployed something on the wrong day. - anyone agree?

In other news:

[Lost Penguin]

It seems the stock market completed the switch over to Vista...... /We don't need to wait on no sticking service pack 1

Which computer and OS are at fault?

[Anonymous Coward]

What is the brand of the computer that malfunctioned in calculating the Dow Jones index? Also, what operating system was the offending computer running?

I hope that the answer is not "Linux on top of HP hardware."

Re:Which computer and OS are at fault?

[MrAnnoyanceToYou]

It appears that the NYSE is running a Service Oriented Architecture. That leads to the second question as to which service failed and how to remedy that in the future.... It looks like they are using Java, so I would expect a Linux based architecture, although my searches mentioned IBM's Rational platform. Honestly, the IBM software I'm currently working with (ClearQuest) shows pretty extensive performance issues as well, so I'm not all that surprised.

Re:Which computer and OS are at fault?

[jgiltner]

Based on some reading the NYSE was running on IBM mainframes running z/OS as the back end system, the clearing system and billing system.. But early last year they announce they were replacing all with pSeries server running AIX. This was to save money. I am not sure if they have migrated, but one article stated they had 1600 zSeries MIPS, which although is quite a bit, is no where near the capacity of the top end z9.

To err is human ...

[140Mandak262Jamuna]

... to blame it on the computers is even more human.

Even the linked article says the excuse is bull.

[Erris]

From the fine article:

To the chagrin of many investors, the drop was indeed real.

I read the NYT article too. One of their sources denied that there was a computer problem. The truth may come out in a day or so, just as soon as all the market watch people quit selling everything they own .... AHHH!

I don't get it

[Anivair]

There were several valid financial reasons that the market took a dive. But chief among them was that it was doing too well for the computers to handle? Don't they realize how contrary that sounds?

Doesn't Matter

[bill_mcgonigle]

It was never mentioned which system crashed. I guess the NYSE doesn't want to embarass the company. I wonder if it was their Windows 2003 with SQL Server system.

I don't think they care about embarrassing the company really. To say, "Oh, a Windows system crashed and caused it," while might be true, casts doubt on the ability of the NYSE to do their job properly (that is specify and operate robust financial systems). Which appears would be justified. They'd be doing the wrong thing by pointing fingers - this was an NYSE system crash.

If you see a bunch of vagrants on Wall Street next week with MCSE pins, you'll know why. If instead it's a bunch of guys muttering about garbage collection spikes in Java 1.5, you'll know why too. Either way, the opposing camp gets to put their system and reputations on the line for the next go around.

Yen carry

[Anonymous Coward]

London closed down sharply yesterday, over a few hundred points. The issue is Yen carry. Check the ForEx charts for USDJPY and GBPJPY - the carry boys are running for the exists. This happened in the late nineties.

There is abnormally high volume due to hedge funds unloading highly leveraged positions secured on Yen loans.

Disclaimer: I'm carrying a large GBP/JPY short on the March future. (short at at 236.01)

Disclaimer: This post represents an opinion and should not be considered as financial advice. Please consult a financial adviser or a person authorized and regulated to give financial / investment advice in your local jurisdiction.

Java???

[gcav]

The more they start using Java for trading systems the worst it will get...

date...

[Presidential]

umm...that happened yesterday, folks.

today: DOW up 12268.63 +52.39 (0.43%)

Australia also in the firing line

[AcidAUS]

Similar issues happened in Australia - the financial services regulator is investigating whether the online share brokers are liable. http://www.smh.com.au/news/business/watchdog-probe s-share-trade-meltdown/2007/03/01/1172338759235.ht ml