Dow Jones Plunge Fueled by Overwhelmed Computers

cloudscout writes "The Dow Jones Industrial Average dropped over 400 points today. While there were various valid financial reasons for such a decline, some of the blame is being placed on computer systems that couldn't keep up with the abnormally high volume at the New York Stock Exchange and the resulting tremor as they switched over to a backup system."

Comments

Blaming?

[VincenzoRomano]

Computers never make errors.
Humans do, at least in designing, manufacturing and sizing computer systems.
This one seems to me like blaming at a knife once you cut your fingers.

Re:Blaming?

[pilgrim23]

In the 1929 crash the problem was partially blamed on the ticker tape running at times up to 1 hour late. Before computers there were people you could blame.

Re:Blaming?

[stevesliva]

Computers never make errors.

Sure they do... just need to bombard them with alpha particles.

Re:Blaming?

[Mr Z]

I think he was referring to the ginormous cliff that happened at almost precisely 3PM. Take a look.

Re:Blaming?

[crabpeople]

"Computers never make errors."

Sure they do. Bad ram, overheated components. If computers didnt make errors, you wouldn't need CRC checking - to name but one common error identifier.

Lets try and think more, go for frosty pist less. Ok?

Re:Blaming?

[Anonymous Coward]

This is referring specifically to the 200 point drop which happened almost instantly. The overall drop was no big deal percentage wise. The 200 point near-instant drop was caused by a computer error. The main system was lagging so they switched to a backup system, this caused about a half hours worth of data to get processed all at once. At the time no one knew what was happening though and were appropriately freaked out.

Re:Blaming?

[Anonymous Coward]

From Frederick Lewis Allen's Only Yesterday , a fascinating history of the 1920s:

Once more the ticker dropped ridiculously far behind, the lights in the brokers' offices and the banks burned till dawn, and the telegraph companies distributed thousands of margin calls and requests for more collateral to back up loans at the banks. Bankers, brokers, clerks, messengers were almost at end of their strength; for days and nights they had been driving themselves to keep pace with the most terrific volume of business that had ever descended upon them. It did not seem as if they could stand it much longer. But the worst was still ahead. It came the next day, Tuesday, October 29th.

The big gong had hardly sounded in the great hall of the Exchange at ten o'clock Tuesday morning before the storm broke in full force. Huge blocks of stock were thrown upon the market for what they would bring. Five thousand shares; ten thousand shares appeared at a time on the laboring ticker at fearful recessions in price. Not only were innumerable small traders being sold out, but big ones, too, protagonists of the new economic era who a few weeks before had counted them. selves millionaires. Again and again the specialist in a stock would find himself surrounded by brokers fighting to sell--and nobody at all even thinking of buying. To give one single example: during the bull market the common stock of the White Sewing Machine Company had gone as high as 48; on Monday, October 28th, it had closed at 11 1/8. On that black Tuesday, somebody--a clever messenger boy for the Exchange, it was rumored--had the bright idea of putting in an order to buy at 1--and in the temporarily complete absence of other bids he actually got his stock for a dollar a share! The scene on the floor was chaotic. Despite the jamming of the Communication system, orders to buy and sell-mostly to sell--came in faster than human beings could possibly handle them; it was on that day that an exhausted broker, at the close of the session, found a large waste-basket which he had stuffed with orders to be executed and had carefully set aside for safekeeping-and then had completely forgotten. Within half an hour of the Opening the volume of trading had passed three million shares, by twelve o'clock it had passed eight million, by half-past one it had passed twelve Million, and when the closing gong brought the day's madness to an end the gigantic record of 16,410,030 shares had been set. Toward the close there was a rally, but by that time the average prices of fifty leading stocks, as compiled by the New York Times, had fallen nearly forty points. Meanwhile there was a near-panic in-other markets--the foreign stock exchanges, the lesser American exchanges, the grain market.

NYSE: Microsoft (R) powered (R)

[hxnwix]

It looks like somebody forgot to change the gigantic SQL batteries embedded in the side of the NYSE building...

Re:NYSE: Microsoft (R) powered (R)

[Anonymous Coward]

as someone who has worked on the floor and systems at NYSE, I can assure you that none of the machines used for trading by specialists or brokers are windows machines, nor are the backends.

all of the machines you see in pictures are dumb terminals connected to a number of master unix systems. they are configured for that special keyboard set used for deals and also touchscreen inputs. however, some (not all) can be booted into windows for personal use or to use firm-specific software.

a side note -- the monitors used by NYSE (you can see them in the pictures on CNN) are ridiculous. huge size, almost perfect 180 degree visibility from side to side and top to bottom, touchscreen, and dust/fingerprint repellent. pretty nasty stuff.

as far as the backends go, the standard system and the backup system are kept running in two locations -- on site, and in a backup location outside of manhattan (not to hard to figure out where, if you try). one can function without the other, on both counts. The QA done to ensure validity is INSANE. Nothing can even be brought out on to the floor without making sure it won't corrupt any of the systems (including the bluetooth system that has become the lifeline of the trading floor -- largest (in size and traffic) private bluetooth network in the world, when I was working the floor)

Re:NYSE: Microsoft (R) powered (R)

[Anonymous Coward]

Actually, SQL Server 2005 is used by NASDAQ, not the NYSE - they are two completely different trading systems. The Dow Jones is an index of select stock listed on the NYSE.
The machines that calculate the Dow Jones Industrial Average are IBM hardware running Unix.
SQL Server running NASDAQ: http://www.windowsfs.com/eNews/tabid/112/articleTy pe/ArticleView/articleId/933/Securities-NASDAQ-Mig rates-to-SQL-Server-2005.aspx
IBM Unix Machines running NYSE and calculating the Dow Jones, as pointed out by another poster: http://www-03.ibm.com/industries/financialservices /doc/content/news/pressrelease/1567015103.html

I have to wonder if this is spam related?

[erroneus]

It's a very strange and vague situation. I'm not sure we'll ever hear what actually happened. "Computer related" does tend to make one a bit frightened as to what that means exactly. It would be over-estimating the impact of stock-spam quite a bit, I should think even to consider whether or not that had any relation to the problems of yesterday. But if it did, perhaps we can now see some real action against spam... we can only hope.

Re:I have to wonder if this is spam related?

[gunnk]

From what I read (NY Times, I believe) the system didn't really have a big an impact as some headlines would lead you to believe.

Apparently the system that computes and displays the current Dow couldn't keep up with the systems that process the transactions when the number of transactions became very large. The display system caught up a bit later making it *appear* that the market at suddenly dropped something like 250 points in a few seconds.

In reality, the decline was fast but steady. It was just the exchange's version of "Damn lag!"

Re:I have to wonder if this is spam related?

[LGV]

All of the stock spam I get is for companies that don't trade on the NYSE or NASDAQ, it's for much, much, much smaller companies that trade on other exchanges. The companies that make up the Dow Jones average (30 of them) are way too heavily traded to be swayed by pump and dump spam, so they don't bother.

Re:I have to wonder if this is spam related?

[DrWho520]

Change in Chineese trading market regulations were the cause of this drop. A massive sell off occurred with the beginning of a crackdown on questionable and illegal trading on China's stock market. This rippled to every other market in the world. Asian, North America, South American and European markets were all affected. Blaming the computer systems is damage control.

Re:I have to wonder if this is spam related?

[Jonny do good]

The drop was really fueled by a number of causes. China's 9% decline the night before was the primary trigger. Sub-prime loans have been leading to trouble for a number of firms lately with the housing decline fueling those problems. The market has been in bull form for quite some time with no corrections leading to a large number of stock prices not supported by their fundamentals. The durable goods sales reports are expected to show under 3% growth when it has been up in the 4% range and this always spooks investors. Any economic indicators showing any sign of change spark massive changes on the market.

The computer problems experienced were really just a lag between the DJIA being calculated and the massive volume of trades being made. Individual stock prices were being reported correctly but the index wasn't keeping up. When the computers caught up they did it over a single minute dropping about 300 points while in reality by the time the index caught up the market had started to rebound a bit. All of the value stock buyers saw the deals becoming available when the landslide hit and started buying a bit. Kind of like today, the market is rebounding because many are looking at stocks that were overpriced yesterday and thinking they are cheap. It's not really as big of deal as the press makes it out to seem. It's not like the '87 crash where 500 points was like 20% of the market. 500 points off th dow is under 4%.

Another trigger for the sudden decline could have been the headline on The Drudge Report (linking to the New York Times article by the same headline) stated that Greespan predicted an imminent recession when his words were as they always have been and that people should be carefull because the economy has been growing for longer than the average growth cycle by about 12 months. Greenspan didn't say anything about a recession being imminent.

Re:I have to wonder if this is spam related?

[DerekLyons]

Change in Chineese trading market regulations were the cause of this drop. A massive sell off occurred with the beginning of a crackdown on questionable and illegal trading on China's stock market. This rippled to every other market in the world. Asian, North America, South American and European markets were all affected. Blaming the computer systems is damage control.

if you actually bother to read TFA - you'll find they don't blame the computers for the drop. They blame the computers for creating the perception of a cliff rather than a steep slope.
 
This is actually something fairly important to consider as more and more of our life interfaces with displays mediated by computer. (Even down to the mundane. I've discovered the digital controls in my oven seem to use a time weighted average - which works fine in convection mode, but it enlarges the deadband in normal mode.)

This is why...

[tomstdenis]

I spend a lot more than I save :-)

The Dow may go down, but a pizza is still as tasty hehehehe.

I wonder how much of this load is due to low volume day trader movement?

Tom

See...

[ZonkerWilliam]

This is what happens when I sell one lousy share of google!

Chaos theory, anyone?

[pzs]

1. Computer switch-over is a bit slow

2. Market starts to waiver

3. Other parts of the market see this tremor so market waivers a lot

4. Panic ensues

5. Indices drop 10%

6. a pension company goes bust

7. my grandpa doesn't get to eat.

The last few steps are somewhat hypothetical, but still. The stock market must be one of the most immediately visible examples of chaos theory kicking humans in the nuts.

Peter

Re:Chaos theory, anyone?

[Nykon]

A butterfly flaps it's wings in Asia , and the dow drop 400 points in the US... brilliant.

Your Nobel Prize is in the mail. Don't call us. We'll cal you.

Re:Chaos theory, anyone?

[pzs]

> A butterfly flaps it's wings in Asia , and the dow drop 400 points in the US... brilliant.

Of course, the usual moron response to this is to say "why don't we just kill all the butterflies in Asia?"

Peter

Re:Chaos theory, anyone?

[Lord Ender]

If an index has been trading near a certain level for a while, then a "panic" event causes a huge drop without changing the fundamental underpinnings of the market, traders view this as a HUGE signal to BUY BUY BUY, on margin if possible. In a few weeks, the index is back where it started before the panic event.

At least, that has been my observation. I can't WAIT for leveraged index ETFs... come on, ProFunds!!

Oh, and your Grandpa's pension would not go bankrupt over a panic event. That's absurd.

Of course, anyone who relies on pension companies for retirement has bigger problems...

Re:Chaos theory, anyone?

[stefanlasiewski]

0. Diversify your holdings, especially your retirement accounts, to protect against market fluctuations. The stock markets will always go up and down for a thousand different reasons-- computer glitch, bad news, hummingbirds, whatever.


1. Computer switch-over is a bit slow
2. Market starts to waiver
3. Other parts of the market see this tremor so market waivers a lot
4. Panic ensues
5. Indices drop 10%


5a. If your investments are diversified, you will survive when the Indexes drop 10%. This is especially true for long-term investments.
6. Buy low.
7. Wait for a while.
8. Sell high.
9. Profit!
10. Enjoy your retirement.

I'm actually suprised

[jhfry]

Considering the amount of, and importance of, data that flows through that system... I am surprised that it's not routinely well ahead of the needs at peak capacity.

I'd say that someone, likely the one in charge of the IT budget approval, keeps tight purse strings. Of course, he's not the one getting reamed, it's the CIO and his crew who are taking the blame even though they have repeatedly requested the funds to improve the system. Just speculation, but likely spot on.

Just another piece of ammo when I start a new job and demand a reasonable budget.

Re:I'm actually suprised

[chad.koehler]

Data rates have doubled at the exchanges in the recent past, and they are likely to double again the the near future. Keeping ahead of this curve is not as easy as one might think...

Just one stock for instance, APPL will have millions of transactions in a single days trading... Including just trades and quotes you can see close to 100Mb of activity for a single symbol in a day.

We are constantly trying to increase capacity, but we're near a point where the only meaningful upgrades for capacity planning are new hardware. To top this off,most of the downstream (client side) applications are limited not by the hardware but by the available bandwidth.

Re:I'm actually suprised

[chad.koehler]

I am involved only tengentally, so I can't speak for exactly what NSX ran into. I'm only saying the issues that I am currently seeing with our systems. We are taking every precaution to make sure that the same thing DOESN'T happen to us. One thing to note, these systems are (in general) running on the most cutting edge systems that money can buy... Hardware upgrades are usually deployed as soon as they are certified, which may only be weeks after a release. When you are already paying 6 figures for a single machine, there isn't much vertical mobility in terms of hardware selection.

Re:I'm actually suprised

[jhfry]

I realize that when your dealing with big iron or even high end server solutions there is little more to do than wait for the next best thing... or innovate.

Put out an RFP for someone to design a cluster based system that will perform the same functions the system currently does, but using OTS hardware in a massive single or distributed cluster configuration. Perhaps they use virtualization and have one VM for every symbol... then purchase enough hardware that only the least active symbols share machines while those with the most activity are alone on a machine. They could even be dynamically moved about as loads fluctuate.

Sure it'd be expensive... but a massive cluster of low end machines, even at $2000 a pop can still bring a lot of power and stability to the mix... and best of all, adding more power is as simple as plugging a new one in.

Anyway, it's besides the point... I just believe that someone somewhere really screwed up to let things get so bad. Perhaps it's the PM who selected the software vendor, perhaps its the CIO who hasn't done what needed to be done, or perhaps it's those with the money who didn't listen to their IT people. I personally think it's the latter case, as I have seen that attitude at all levels of business... even a 1 man company can fail to allot the proper funds to IT and let essential services go ignored (like backups).

I have begged and pleaded with bosses that it's necessary to increase capacity prematurely... but often they counter with, come back when something is broken. Then when I do, it's "What do you mean it will take 3 months to implement a fix? Didn't you see this coming?" I still believe this is what happened. Sure someone saw the problem, and knew how to fix it.. but they couldn't convince the man with the cash to cough some up and get the ball rolling. I bet the vault is open now though!

Fueled nothing

[AlphaNuRho]

I don't see how you could say that the computer problem fueled the plunge. My understanding of the events is that the only problem was with the system that calculated the Dow Jones Industrial Average Index (the number that is around 12,200). There wasn't a backup or delay in execution of trades or anything like that. The decline was real, but it was spread out over an hour instead of the 2 minutes reflected by the DJIA.

Traders still bought and sold stocks at their real value in real time. The calculation of the sum of their activity was the only thing delayed.

not quite

[flynt]

The computer systems weren't responsible for the overall drop, but rather the rate of the drop during the few minutes of switching over to the backup computers. This queued up trades, and at the current volume of the switchover, caused a large drop when they caught up. At least that's how I understand it.

Real Cause?

[green453]

I'm all for looking at things from the tech/computer geek side of things as much as anybody on Slashdot, but isn't the summary taking things a bit far? It was mentioned that there may have been other causes that combined with computer glitches, but wouldn't the fact that markets in China dropped a whopping 9% yesterday seem to be the real cause? I'm sure swithcing computer systems may have scared a few people, but I doubt it was the primary cause of a 400pt drop. That said, it is interesting to think about the effect of computer systems on the financial markets. I've always maintained that it isn't the politicians or the business owners or the economists that run the world, it's the engineers. Think about what would happen if there was a complete shutdown of the systems that run the markets. See if all the Wall Street profiteers pay their geeks a pittance of their "annual bonus" then...

Dropping prices versus dropping data

[G4from128k]

The problem was obvious to anyone watching the markets. A trace of the Dow versus the S&P showed that the Dow's drop was NOT keeping pace with the drop in the S&P (they are normally tightly correlated, especially when big moves occur). It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop. When that backup server cut in, the Dow data suddenly reflected the true state of affairs that was obvious from people watching the S&P and the broader market.

The Dow did NOT drop 200 points in minutes, the data simply caught up with the drop that had already occurred.

Re:Dropping prices versus dropping data

[richg74]

From what I've read, and learned from talking to a couple of former colleagues (I worked in IT on Wall Street for 20+ years), your note is almost but not quite right. You correctly point out that the Dow's decline was, for a time, seriously out of line with the decline in other market indices, such as the S&P 500. However, you went on to say:

It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop.

In fact, the problem appears to have been that the systems at Dow-Jones -- which owns the DJIA index and calculates it -- could not keep up with the volume. When the backup system came online, the reported index showed a significant drop in a very short time. In actuality, the decline was real, but it had already happened over a longer period of time. As it says in the original article:

"The market's extraordinary trading volume caused a delay in the Dow Jones data systems," said Dow Jones spokeswoman Sybille Reitz. "We decided to switch over to the backup system, and the result was a rapid catch-up in the published value of the Dow Jones industrial average."

There's no indication that there was a problem with trade reporting by the NYSE, which would be a much more serious problem. But the data feed from the exchange reports prices trade-by-trade as they occur. So, if an external system like Dow-Jones's gets behind, it usually has to plow through the update stream to get current, which can result in a sort of "compressed time" effect.

Don't see how they could.

[Kadin2048]

I wonder how much of this load is due to low volume day trader movement?

I would guess, virtually none, since they're by definition low-volume?

This blaming it on computers seems mostly a red herring. The markets in Asia (particularly Shanghai) tanked, and as a result, the markets in the US tanked, because companies in the US are heavily invested in China.

I think the only lesson here, in case there was anyone left who didn't get it, is that we all float or sink together. For better or worse, the US has tied itself pretty tightly to the Asian markets, and if they collapse, we're going to be seriously hurting.

Re:Don't see how they could.

[Kadin2048]

Yes, but what I'm questioning is whether the Dow Jones' computers really had anything to do with this whole market movement at all. At most, all they did was slow down, so that the DJI lagged behind the real world for a while, and then suddenly caught up when their backup system went on-line and took over.

I think that it's more of a symptom and less of a cause. The cause of the market movement was in Asia; that made people sell, people selling caused the DJIs computers to suck. Now, perhaps the DJIs computer slowdown, and consequent large jump when they fixed the problem and got the backup running, caused more people to sell, but this seems specious. The slump was already in progress by the time that the computer slowdown occurred, because the slowdown was driven by high trade volume.

So my point is mostly that I don't see how it matters, really. People are looking towards computer glitches as the cause for the 3% drop in the market (or whatever it was), and that's just not true. The computer glitch might have made the drop look worse, or more precipitous, than it actually was at one point during the day, but it didn't cause or really drive it in any significant way. Even if the DJI folks' computers had worked perfectly, the market would still be sucking. In fact, computerization and the consequent flow of information is what links markets; it's only in the last few decades that the Asian and US markets have felt each other's pain so closely, so in a way, you can blame the computers for working too well in general, when you get these domino-effect deflations.

Re:Don't see how they could.

[Lawrence_Bird]

er.. not in this case. Their 'explanation' is that they had fallen behind, which in fact was the case as when DJIA was "down" 295, futures were already down 400 and sinking. In fact, I commented to another trader what a nice arb was out there if only one could short on the downtick in the cash market. Note also that there were already some limitations in effect on automated program trading (tick rules)

In the event of a 180-POINT DECLINE in the NYA, all index-arbitrage sell orders of the S&P 500 stocks must be stabilizing for the remainder of the day. Collar will be removed if the NYA moves back to within 90 points of the previous days close.

At one point the NYA was down 400 pts. This 'blame it on the computer' or 'it was china' are BS. Stocks were over extended. While any one thing might have some effect, the time was ripe and the buyers could no longer hold the bid in the face of china sell off, weaker econ data, Greenspan, Afghanistan, Irock, etc etc simultaneously. There is too much leverage in use, too much complacency and just too much liquity. When all rush for the door in the face of trouble this is what happens.

"Load"? That's exactly what I was thinking

[Mateo_LeFou]

yesterday's news was annoying as hell. Everyone and their dog chimed in on what caused this horrible crash, what investors should do now, how bad it might get, etc. People: the market's been soaring for months. This is a perfect example of broadcasters' attempt to get you afraid and addicted to "news".

Re:Ok...

[BanjoBob]

I thought they were still on Windows Server 2003 running MS SQL Server.

Now, had they been running Linux and MySQL on an IBM monster box things would have probably been different

Re:Just like the computers in 1929!

[UbuntuDupe]

wonder what the reactions would have been like if a "computer glitch" knocked the thing up 500 points instead of down.

Um ... *pulls aside*

You mean like what happened two weeks ago?

Editors? We don't need no stinking editors.

[dougman]

The DOW is up today. Can't you at least get the most basic facts right? The drop was yesterday.

Twilight Zone: Tom Clancy

[maroberts]

As well as crashing planes into buildings, it seems "Debt of Honor" is getting good at being an oracle of modern times.

IBM backend w/ Linux Workstations used here?

[xxxJonBoyxxx]

If I'm reading this right, an IBM back-end system (mainframe) with lots of IBM-delivered Linux workstations were in the mix here. Anyone know for sure (i.e. work there)?

http://www.internetnews.com/bus-news/article.php/3 447741

It's voodoo

[PIPBoy3000]

Yep - that sounds just about right.

Imagine a series of database transactions, with each step getting queued up and waiting for the system to finish processing it. The actual DOW number reflects fully completed transactions, but not pending transactions that might impact the outcome. This is probably a good thing, as a transaction might end up being rejected, so you only want to show the outcome of completed transactions. Once the backup system came online, the transactions quickly finished being completed, resulting in the dramatic drop.

The amazing thing to me is that the system is robust enough that transactions can survive the loss of their main computer system and bringing up a secondary one. That's database, networking, and coding voodoo, all wrapped into something pretty awe-inspiring.

Re:It's voodoo

[D-Fens]

It's an entirely different kind of computing.

Re:It's voodoo

[Ctrl-Z]

It's an entirely different kind of computing.

Hal? HAL!!!

[Malakusen]

Let me put it this way, Mr Amer. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error.

Oblig Futurama quote

[Cervantes]

Dr. Zoidberg: Once again, the conservative, sandwich-heavy portfolio pays off for the hungry investor!

The Dow's technical glitch, dissected

[rrohbeck]

http://money.cnn.com/2007/02/28/markets/tech_glitc h/index.htm

The publisher of the Dow industrials said that a system problem starting at 1:50 p.m. ET on Tuesday, amid unusually heavy trading volume, caused a 70-minute lag during which the value of the market measure lagged the declines in the underlying stocks.

The subsequent downward spike in the Dow occurred when the problem was corrected as the company switched to its backup system at around 3 p.m.

Just before the switch, the Dow was showing about a 160-point drop. But then the blue-chip barometer appeared to tumble some 200 points in the blink of an eye as the newly available data was correctly reflected in the average.

Re:Sooner than think.

[Red Flayer]

At that point, our economy will truly slow. Hopefully, at that time, GWB will be forced to get a real backbone and deal with China and get them to untie their money from ours.

1. China has already unpegged from the dollar, they now peg to an index of currencies (I know, with the dollar still the main component).

2. What makes you think that's going to happen during the current administration? We both know that any negative actions will be held off on until the current "what-me-worry?" administration is gone. They'll let someone else take the heat.

3. And raising prime that far will have the beneficial effect of causing inflation, which will devalue our debt. The downside will when ere high prime will cause inflation, but the returns on business investment lag behind government securities causing a shrinking economy (stagflation). We've been through it before, it's not pretty -- and it's what gave us Reagan in the White House, for better or worse.

Which computer and OS are at fault?

[Anonymous Coward]

What is the brand of the computer that malfunctioned in calculating the Dow Jones index? Also, what operating system was the offending computer running?

I hope that the answer is not "Linux on top of HP hardware."

Australia also in the firing line

[AcidAUS]

Similar issues happened in Australia - the financial services regulator is investigating whether the online share brokers are liable. http://www.smh.com.au/news/business/watchdog-probe s-share-trade-meltdown/2007/03/01/1172338759235.ht ml