Slashdot Mirror


Dow Jones Plunge Fueled by Overwhelmed Computers

cloudscout writes "The Dow Jones Industrial Average dropped over 400 points today. While there were various valid financial reasons for such a decline, some of the blame is being placed on computer systems that couldn't keep up with the abnormally high volume at the New York Stock Exchange and the resulting tremor as they switched over to a backup system."

18 of 215 comments (clear)

  1. NYSE: Microsoft (R) powered (R) by hxnwix · · Score: 5, Funny

    It looks like somebody forgot to change the gigantic SQL batteries embedded in the side of the NYSE building...

    1. Re:NYSE: Microsoft (R) powered (R) by Anonymous Coward · · Score: 5, Interesting

      as someone who has worked on the floor and systems at NYSE, I can assure you that none of the machines used for trading by specialists or brokers are windows machines, nor are the backends.

      all of the machines you see in pictures are dumb terminals connected to a number of master unix systems. they are configured for that special keyboard set used for deals and also touchscreen inputs. however, some (not all) can be booted into windows for personal use or to use firm-specific software.

      a side note -- the monitors used by NYSE (you can see them in the pictures on CNN) are ridiculous. huge size, almost perfect 180 degree visibility from side to side and top to bottom, touchscreen, and dust/fingerprint repellent. pretty nasty stuff.

      as far as the backends go, the standard system and the backup system are kept running in two locations -- on site, and in a backup location outside of manhattan (not to hard to figure out where, if you try). one can function without the other, on both counts. The QA done to ensure validity is INSANE. Nothing can even be brought out on to the floor without making sure it won't corrupt any of the systems (including the bluetooth system that has become the lifeline of the trading floor -- largest (in size and traffic) private bluetooth network in the world, when I was working the floor)

  2. Re:Blaming? by pilgrim23 · · Score: 4, Insightful

    In the 1929 crash the problem was partially blamed on the ticker tape running at times up to 1 hour late. Before computers there were people you could blame.

    --
    - Minutus cantorum, minutus balorum, minutus carborata descendum pantorum.
  3. See... by ZonkerWilliam · · Score: 5, Funny

    This is what happens when I sell one lousy share of google!

  4. Chaos theory, anyone? by pzs · · Score: 5, Insightful

    1. Computer switch-over is a bit slow

    2. Market starts to waiver

    3. Other parts of the market see this tremor so market waivers a lot

    4. Panic ensues

    5. Indices drop 10%

    6. a pension company goes bust

    7. my grandpa doesn't get to eat.

    The last few steps are somewhat hypothetical, but still. The stock market must be one of the most immediately visible examples of chaos theory kicking humans in the nuts.

    Peter

  5. I'm actually suprised by jhfry · · Score: 4, Insightful

    Considering the amount of, and importance of, data that flows through that system... I am surprised that it's not routinely well ahead of the needs at peak capacity.

    I'd say that someone, likely the one in charge of the IT budget approval, keeps tight purse strings. Of course, he's not the one getting reamed, it's the CIO and his crew who are taking the blame even though they have repeatedly requested the funds to improve the system. Just speculation, but likely spot on.

    Just another piece of ammo when I start a new job and demand a reasonable budget.

    --
    Sometimes the best solution is to stop wasting time looking for an easy solution.
  6. Fueled nothing by AlphaNuRho · · Score: 4, Informative

    I don't see how you could say that the computer problem fueled the plunge. My understanding of the events is that the only problem was with the system that calculated the Dow Jones Industrial Average Index (the number that is around 12,200). There wasn't a backup or delay in execution of trades or anything like that. The decline was real, but it was spread out over an hour instead of the 2 minutes reflected by the DJIA.

    Traders still bought and sold stocks at their real value in real time. The calculation of the sum of their activity was the only thing delayed.

  7. not quite by flynt · · Score: 5, Informative

    The computer systems weren't responsible for the overall drop, but rather the rate of the drop during the few minutes of switching over to the backup computers. This queued up trades, and at the current volume of the switchover, caused a large drop when they caught up. At least that's how I understand it.

  8. Dropping prices versus dropping data by G4from128k · · Score: 5, Insightful

    The problem was obvious to anyone watching the markets. A trace of the Dow versus the S&P showed that the Dow's drop was NOT keeping pace with the drop in the S&P (they are normally tightly correlated, especially when big moves occur). It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop. When that backup server cut in, the Dow data suddenly reflected the true state of affairs that was obvious from people watching the S&P and the broader market.

    The Dow did NOT drop 200 points in minutes, the data simply caught up with the drop that had already occurred.

    --
    Two wrongs don't make a right, but three lefts do.
    1. Re:Dropping prices versus dropping data by richg74 · · Score: 4, Interesting
      From what I've read, and learned from talking to a couple of former colleagues (I worked in IT on Wall Street for 20+ years), your note is almost but not quite right. You correctly point out that the Dow's decline was, for a time, seriously out of line with the decline in other market indices, such as the S&P 500. However, you went on to say:

      It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop.
      In fact, the problem appears to have been that the systems at Dow-Jones -- which owns the DJIA index and calculates it -- could not keep up with the volume. When the backup system came online, the reported index showed a significant drop in a very short time. In actuality, the decline was real, but it had already happened over a longer period of time. As it says in the original article:

      "The market's extraordinary trading volume caused a delay in the Dow Jones data systems," said Dow Jones spokeswoman Sybille Reitz. "We decided to switch over to the backup system, and the result was a rapid catch-up in the published value of the Dow Jones industrial average."
      There's no indication that there was a problem with trade reporting by the NYSE, which would be a much more serious problem. But the data feed from the exchange reports prices trade-by-trade as they occur. So, if an external system like Dow-Jones's gets behind, it usually has to plow through the update stream to get current, which can result in a sort of "compressed time" effect.
  9. Re:Just like the computers in 1929! by UbuntuDupe · · Score: 4, Informative

    wonder what the reactions would have been like if a "computer glitch" knocked the thing up 500 points instead of down.

    Um ... *pulls aside*

    You mean like what happened two weeks ago?

  10. Editors? We don't need no stinking editors. by dougman · · Score: 5, Informative

    The DOW is up today. Can't you at least get the most basic facts right? The drop was yesterday.

  11. Re:I have to wonder if this is spam related? by gunnk · · Score: 4, Informative

    From what I read (NY Times, I believe) the system didn't really have a big an impact as some headlines would lead you to believe.

    Apparently the system that computes and displays the current Dow couldn't keep up with the systems that process the transactions when the number of transactions became very large. The display system caught up a bit later making it *appear* that the market at suddenly dropped something like 250 points in a few seconds.

    In reality, the decline was fast but steady. It was just the exchange's version of "Damn lag!"

    --
    Life is short: void the warranty.
  12. Twilight Zone: Tom Clancy by maroberts · · Score: 5, Interesting

    As well as crashing planes into buildings, it seems "Debt of Honor" is getting good at being an oracle of modern times.

    --

    Donte Alistair Anderson Roberts - hi son!
    Karma: Chameleon

  13. Re:I have to wonder if this is spam related? by DrWho520 · · Score: 5, Interesting

    Change in Chineese trading market regulations were the cause of this drop. A massive sell off occurred with the beginning of a crackdown on questionable and illegal trading on China's stock market. This rippled to every other market in the world. Asian, North America, South American and European markets were all affected. Blaming the computer systems is damage control.

    --
    The cancel button is your friend. Do not hesitate to use it.
  14. It's voodoo by PIPBoy3000 · · Score: 4, Insightful

    Yep - that sounds just about right.

    Imagine a series of database transactions, with each step getting queued up and waiting for the system to finish processing it. The actual DOW number reflects fully completed transactions, but not pending transactions that might impact the outcome. This is probably a good thing, as a transaction might end up being rejected, so you only want to show the outcome of completed transactions. Once the backup system came online, the transactions quickly finished being completed, resulting in the dramatic drop.

    The amazing thing to me is that the system is robust enough that transactions can survive the loss of their main computer system and bringing up a secondary one. That's database, networking, and coding voodoo, all wrapped into something pretty awe-inspiring.

  15. Re:I have to wonder if this is spam related? by Jonny+do+good · · Score: 5, Informative

    The drop was really fueled by a number of causes. China's 9% decline the night before was the primary trigger. Sub-prime loans have been leading to trouble for a number of firms lately with the housing decline fueling those problems. The market has been in bull form for quite some time with no corrections leading to a large number of stock prices not supported by their fundamentals. The durable goods sales reports are expected to show under 3% growth when it has been up in the 4% range and this always spooks investors. Any economic indicators showing any sign of change spark massive changes on the market.

    The computer problems experienced were really just a lag between the DJIA being calculated and the massive volume of trades being made. Individual stock prices were being reported correctly but the index wasn't keeping up. When the computers caught up they did it over a single minute dropping about 300 points while in reality by the time the index caught up the market had started to rebound a bit. All of the value stock buyers saw the deals becoming available when the landslide hit and started buying a bit. Kind of like today, the market is rebounding because many are looking at stocks that were overpriced yesterday and thinking they are cheap. It's not really as big of deal as the press makes it out to seem. It's not like the '87 crash where 500 points was like 20% of the market. 500 points off th dow is under 4%.

    Another trigger for the sudden decline could have been the headline on The Drudge Report (linking to the New York Times article by the same headline) stated that Greespan predicted an imminent recession when his words were as they always have been and that people should be carefull because the economy has been growing for longer than the average growth cycle by about 12 months. Greenspan didn't say anything about a recession being imminent.

  16. Re:I have to wonder if this is spam related? by DerekLyons · · Score: 4, Interesting

    Change in Chineese trading market regulations were the cause of this drop. A massive sell off occurred with the beginning of a crackdown on questionable and illegal trading on China's stock market. This rippled to every other market in the world. Asian, North America, South American and European markets were all affected. Blaming the computer systems is damage control.

    if you actually bother to read TFA - you'll find they don't blame the computers for the drop. They blame the computers for creating the perception of a cliff rather than a steep slope.
     
    This is actually something fairly important to consider as more and more of our life interfaces with displays mediated by computer. (Even down to the mundane. I've discovered the digital controls in my oven seem to use a time weighted average - which works fine in convection mode, but it enlarges the deadband in normal mode.)