Slashdot Mirror

← Back to Stories (view on slashdot.org)

The London Stock Exchange Goes Down For Whole Day

Posted by ScuttleMonkey on from the at-least-no-one-died-...-yet dept.

Colin Smith writes "TradElect, the Microsoft .Net based trading platform for the London Stock Exchange, was offline for about seven hours, meaning that their 5-nines SLAs are shot for approximately the next 100 years. The TradElect system was launched back in June of 2007 and was designed for increased speed and system capacity."

20 of 792 comments (clear)

  1. 99.9967% Uptime if up the next 100 years by xmas2003 · · Score: 5, Informative

    Assuming 8.5 hour trading day (0700-1530) and 250 trading days/year. Maybe a squirrel caused the problem ... ;-)

    --
    Hulk SMASH Celiac Disease
  2. Ugly Day by pyite · · Score: 4, Informative

    It was an ugly day of finger-pointing and near-fixes, but in the end, it just left all the financial firms standing there staring at the Exchange. Definitely was a big deal--and it seemed like a lot of volume spilled over to US markets, creating volume related issues here.

    --

    "Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman

  3. single page by Anonymous Coward · · Score: 5, Informative

    I wish people would get into the habit of linking to the single page version of the FA.

  4. Misleading summary by denoir · · Score: 5, Informative

    The summary implies that TradElect was responsible for the shutdown, but according to the stock exchange itself, it wasn't the case. They say instead it was a network problem.

    1. Re:Misleading summary by Hyppy · · Score: 4, Informative

      if it was a network problem, then they're in more trouble than the summary implies. It's relatively simple to get 100% uptime (minus a dropped packet or two) in a network. The key here is redundancy. If you throw enough hardware at it, yes, it will not break.

      Internal? Dual(+) homed servers, redundant switches, redundant AC, redundant power.
      External? BGP on 2 or more transits on separate physical runs.

      What, you say that you need to account for natural disasters? Then get a second site, at least a few hundred miles away, and repeat.

      Virtual 100% uptime is a solved problem in the networking world.

    2. Re:Misleading summary by IgnoramusMaximus · · Score: 5, Informative

      Any company that includes Linux is RTP/STP should go out in the street with them. Though at least you got Solaris correct.

      You have no clue. When people mention Linux in these environments they mean Linux running on one of these, not a home-brew distro running on a $150 PC.

    3. Re:Misleading summary by Cillian · · Score: 4, Informative

      Everybody wasn't dropped. A few people had issues, and so they had to completely stop trading, else the people without issues had an unfair advantage.

      --
      -- All your booze are belong to us.
  5. 5-nines SLA by skeeto · · Score: 4, Informative

    "5-nines SLA"

    I had to look this up, so I imagine other people didn't know it either (I thought was was a stock exchange term). First Google search result reveals the answer,

    The Battle With "3 Nines" and The Goal of "5 Nines"

  6. Re:How many failures before.. by KernelMuncher · · Score: 5, Informative

    When I worked in academia I used to collaborate on a research project with a data architect from one of the major electronic exchanges. His whole shop is MS and .NET. I asked him why he didn't run Linux / Unix. He said that with competent guys the MS boxes had great uptime. Wall Street can afford to pay the top salaries so they attract guys who really know their stuff. Not just semi-competent people who managed to sit through an MSCE exam. [his words not mine]

    Also he said support was crucial for his company. If something went down, he wanted to be able to call someone immediately. He couldn't afford to just post a question on a message board and hope someone replies. He wanted contracts with 3rd party support that had experience with similar huge enterprise systems that he had.

    When I said there were companies who could provide excellent Linux support, he said his ass was on the line if something broke so he wanted to be able to justify his software choice to the the C-level guys. And those guys knew the name Microsoft. So he didn't see anything else as an option.

  7. Re:Oh, my. by im_thatoneguy · · Score: 5, Informative

    Which from the sounds of this article http://www.computerweekly.com/Articles/2008/06/12/231031/agile-trading-software-critical-to-london-stock-exchange.htm was the intent.

    One very interesting note is at the end of the article:

    Timeline for Tradelect upgrades

    18 June 2007: Tradelect launched, reducing the time taken to process trades from 140 milliseconds to 10 milliseconds. Capacity increased from 593 to 2,500 orders a second.

    November 2007: Version 2 upgrade. Trading time reduced from 10 milliseconds to about 6 milliseconds. Capacity increased by 70% from 2,500 to 4,200 orders a second. Introduced full suite of Mifid-compliant services.

    September 2008: Planned migration of Italian trades to Tradelect platform.

    September 2008: Tradelect Version 2 to launch. Plans to double trading capacity to 10,000 continuous messages per second. Aims to cut average time taken to complete a trade by half from 6 milliseconds to 3 milliseconds.

    Coincidence that this month was when they intended to release a new version?

  8. Re:100 years? by julesh · · Score: 5, Informative

    5 nines does not mean what you think it means.

    No, you're right. By my calculation, the actual figure is more like 360 years.

    (Remember, this is a system that only operates 7.5 hours per day, 250 days per year)

  9. Re:In other NEWS... by Cap'nPedro · · Score: 5, Informative

    No, he'd waggle his arse .
    A fanny would be a vagina in Britain.

    Come on +5 informative!

  10. Bad upgrade by JShadow21 · · Score: 5, Informative

    The article here blames it on some sort of botched upgrade.

  11. Re:Still don't know why... by Dan667 · · Score: 4, Informative

    Leaky abstractions (look it up, it is a good read). A lot of times for kitchen sink platforms like .Net and Java you get burned by the bugs buried in the underlying platform. If to many of these system are stacked it becomes really difficult to have any stability.

  12. To be fair by EmbeddedJanitor · · Score: 4, Informative
    Five 9s does not mean achieving five 9s at every installation. It means five 9s averaged across all installations. Having a 0.0001% chance of being hit by a bus is hardly consolation for the person that actually does get hit by the bus.

    Of course it is very unlikely that MS achieves five 9s on any installation, let alone as an average.

    --
    Engineering is the art of compromise.
  13. Link to incident status page by alexmin · · Score: 5, Informative

    Here: http://www.londonstockexchange.com/en-gb/products/membershiptrading/tradingservices/Incident/LIVE
    Notice that there were several unsuccessful attempts to bring it back up.
    What's really pitiful, LSE has just a fraction of data/trade volume of major US exchanges like Nasdaq or NYSE and still, their systems are regularly getting hosed, albeit not as much as today's meltdown.
    Hopefully in coming years LSE will lose market share to Nasdaq/Europe, BATS/Europe, Chi-X and other electronic markets - that should teach them well.

  14. Re:How many failures before.. by metamatic · · Score: 4, Informative

    Can you point to any smaller exchanges that use Linux?

    No, but I can point to the New York Stock Exchange, which uses AIX and Linux.

    --
    GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
  15. Re:Potentially misleading summary by Angostura · · Score: 5, Informative

    Well, the Reuters article does say that trading started normally, but some traders were unable to connect, so the whole exchange was bought down to avoid unfair advantage/disadvantage occurring, so actually both stories are consistent.

  16. Using Microsoft for a 5-nines SLA? Is that a joke? by MrJerryNormandinSir · · Score: 5, Informative

    That was the their first mistake. What were they thinking? You need a 3 highly available Unix clusters with three SANs. You need three to elect a quorum. If you don't know what a quorum is you shouldn't be attempting to design system that is supposed to deliver on a 5-nine SLA. Each geographic location should include 1 cluster and 1 SAN. All three locations networked with dark fiber. fiber routing should be set up so that a cluster can fail over to a SAN in another location. As far as Hardware is concerned, I would go with a cluster of IBM P6-570 and use an EMC Symmetrix DMX SAN at each site.
    Who the heck designed this? .Net trading platform.. I have to laugh! Microsoft .net = 5.none SLA! .Net is only good for people who would like to create a light duty website. Under a load it breaks. The London Stock Exchange proves my point.

  17. Re:Using Microsoft for a 5-nines SLA? Is that a jo by Aceticon · · Score: 4, Informative

    I work in London as a freelancer in IT in Investment Banking. My professional experience was mostly with IT Products/Services companies.

    Although I haven't worked in the LSE, from the places I've worked in around here I came out with the impression that most people in IT in this industry are amateurs (and that includes those in other geographical locations).

    Any kind of more advanced IT concepts such as technical analysis, software/hardware architecture, iterative software development processes are pretty much either not done or done by people you don't have clue about what they're doing.

    I'm hardly surprised with what happened in the LSE.