Slashdot Mirror
London Stock Exchange Price Errors 'Emerged At Linux Launch'
DMandPenfold writes "Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources. Thomson Reuters, Interactive Data and Netbuilder are among the largest data vendors, providing share prices to traders, that have been displaying pricing problems on some stocks throughout the week. Even the LSE's own data vendor, ProQuote, experienced problems. Concerns are being raised that there could be mistakenly setup connections or incorrect software interfaces at some of the large data vendors. Alternatively, there may be a data caching issue at the LSE that means data going out is not properly synchronised between different systems."
Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources.
Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.
Google, Amazon, IBM, Redhat, ... Take your pick :-)
Turn it off and on again.
Don't forget the US DoD, The US Navy Submarine Fleet, the FAA, and Cisco:
http://www.focus.com/fyi/information-technology/50-places-linux-running-you-might-not-expect/
I don't see IBM running Watson on Windows 2008 Server because, well, you know....they want it to work.
My heart goes out to the devs "working long hours and night shifts" to suss this out.
That being said, the line that catches my eye most is: "The fact the majority of smaller vendors were fine demonstrated that those having trouble had made mistakes." In my experience, that means one of two things:
1) The devs configuring the system didn't properly account for the sheer scale of the stress on their systems.
2) The smaller vendors took the change more seriously, and being smaller and more flexible, successfully updated their systems to interact properly with the new systems.
Or, of course, both.
The amusing part of this troll is that it would be modded up to 5, instantly, if it were attacking Windows. The day when I see a pro-linux comment modded "troll" is the day that I take anything posted on Slashdot seriously. Until then, it's just self-important, self-congratulatory, mindless groupthink.
I work on large scale air traffic control systems which run Linux and I don't envy the LSE in their task. Most of our interfaces are relatively simple and go out to organisations with a good history of validating interfaces. This trading system seems to have to interface to a lot of little offices around the place running various implementations. Its no surprise some of the interfaces weren't tested to the point where they are known to work 100%, though they may be 100% correct.
http://michaelsmith.id.au
Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.
Quote from the project manager:
"Dang. If only we had read Tubal-Cain's post before going live. Who would have thought to run test data through this darn thing?"
Article is a a bit misleading....
Reuters, Netbuilder et al are quote services that run externally and fetch quotes (price/volume) from the LSE and are *not* part of what would be considered the main trading platform. The problem isnt the actual OS (Linux) or tradiing platform, which has been heralded as a big step forward from the MS solution, but rather how the resulting trade data that is pushed from the exchange to the 3rd party sites and then ultimately to the end users.
using a solid os won't protect you from human stupidity. i thought that was a given. perhaps letting microsoft setup everything would be better for stupid people like lse. more expensive but atleast no stupid errors.
meanwhile, intelligent people continue using linux to their advantage.
Wealth is the gift that keeps on giving.
I thought I'd stuffed up big time when I hooked all of our users (around 1000 users at the time) up to test market prices rather than live data (Australian Stock Exchange) over market opening one morning (our product is aimed at day traders too). My stuff up seems minor in comparison to this.
As someone on the receiving end of this problem (as head of Development for one of the largest UK stockbrokers), it has been immensly frustrating trying to get an admission of blame from anyone involved in this issue. Our clients are complaining to use because elements of the data we're displaying on our site are just so out of whack they are laughable. In the end we've just had to remove those elements of data from our website, and they remain removed until the fix is confirmed working on Monday. This is after several attempts to fix the issues during the week.
It's also causing serious issues for things like our black box trading engine which is used to drive automatic execution of client orders when the price moves into range for execution. The data being bad means these systems just have to be turned off, and clients are being advised to simply delete their orders.
This is, frankly, a pretty appalling situation as the service to our clients is being impacted pretty severely.
For an exchange that is attempting to position itself as the true 'global' exchange, this is doing an immense amount of damage to it's reputation.
As with most major issues there's bound to be a big ol' postmortem on this. As head of Dev you've probably got a unique insight into this, I'm curious as to your perspective on this, what you think the cause of failure might be? More strategic or more technical? Poor interface specification? Inability to handle queries under full load? From TFA there was supposedly 15 months of testing. So I'm curious as to why it failed, whether the testing simply wasn't realistic and/or thorough enough, or it was something that just wouldn't come up except on the live system?
If it didn't crash and didn't drop its network connections, Linux was doing its job.
If the application software had bugs, then the application software developers are to blame.
The drop in stock price after the announcement of the Nokia Microsoft venture was sharp for a single day. It's also one of the highest volume days in the last 5 years.
However, the drop is insignificant compared to the drop from late 2007 to now. Market Cap at Nokia is 1/4 what it was. Stock prices have dropped from mid 40's / share to ~ 10/share.
Besides, the 8.5B drop in market cap isn't nearly as important to the bottom line as the billions in cash coming in from Microsoft as part of this deal.
An attempt to rubbish Linux.
After 20 seconds they should have realised that they should have tested more, and fired the programmers for allowing such mistakes. This is nothing to do with Linux.
You'd think the LSE would have learnt from their last computer system rollout which also had massive problems. No quality control = management problem.
Take Nobody's Word For It.
That quoted phrase 'emerged at linux launch' is NOWHERE in the article. The article says "London Stock Exchange price data failures ‘emerged immediately at Millennium launch’". This is obviously an application issue, why is the summary sort of blaming the operating system?
WTF am I doing replying to an AC at 5 A.M on a Friday night?
No surprise here for anyone working with them. Their marked data specs have hundreds of pages while Chi-X EU have about 20. Chi-X EU quoting and trading volume is far greater. Coincidence? I say no - people voting with their feet and going from LSE to Chi-X EU (biggest trading platform in EU now), BATS EU, and other MTFs. Maybe the only thing keeping LSE still afloat is a fact that FTSE 100 index is computed using LSE prices. Once that gone, potentially as soon as MIFID II comes out, LSE is going to circle in the crapper.
I would wager that the vast majority of problems in Windows are also due to user error, yet Linux fanatics instantly claim it's the OS.
At very least you'd expect a large scale dry-run with simulated data that should bring these errors out before becoming live.
Maybe. Such testing only tests the type of errors you have thought of and are anticipating. I was once blessed with a competent and skilled QA department and months of pre-release testing, simulation, fuzzing, beta testing, and yet when a product goes live to millions bugs become apparent. You just can't think of and test for everything in a sufficiently complex system. The real failure of this LSE rollout may have been not running the old and new systems in parallel while looking for discrepancies in the publicly reported data.