Slashdot Mirror
London Stock Exchange Price Errors 'Emerged At Linux Launch'
DMandPenfold writes "Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources. Thomson Reuters, Interactive Data and Netbuilder are among the largest data vendors, providing share prices to traders, that have been displaying pricing problems on some stocks throughout the week. Even the LSE's own data vendor, ProQuote, experienced problems. Concerns are being raised that there could be mistakenly setup connections or incorrect software interfaces at some of the large data vendors. Alternatively, there may be a data caching issue at the LSE that means data going out is not properly synchronised between different systems."
Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources.
Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.
Once again we witness how Linux destroys companies, One after another (Novell, SCO, Nokia, etc) each of them adopted Linux and then fell victim to the unreliability, unmarketability and anti-business attitude of the linux zealots.
Who's next?
Turn it off and on again.
My heart goes out to the devs "working long hours and night shifts" to suss this out.
That being said, the line that catches my eye most is: "The fact the majority of smaller vendors were fine demonstrated that those having trouble had made mistakes." In my experience, that means one of two things:
1) The devs configuring the system didn't properly account for the sheer scale of the stress on their systems.
2) The smaller vendors took the change more seriously, and being smaller and more flexible, successfully updated their systems to interact properly with the new systems.
Or, of course, both.
Did they reboot it 3 times? I've heard that works.
Oh wait this isn't windows 2008.
I expected to come in hear reading how this was proof Linux sucks and how it's not ready for mission critical apps. I was just _sure_ that would be the tenor of the comments, as that was the tenor previously when a different system was used and had issues. Odd, I guess I just guessed wrong.
Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.
Quote from the project manager:
"Dang. If only we had read Tubal-Cain's post before going live. Who would have thought to run test data through this darn thing?"
Hmmm.. So .Net was bad ? Bot a good start on Linux
You get what you pay for with Linux.
If they had gone with MS for the upgrade it would have been bulletproof, as they have a reputation to keep up.
Article is a a bit misleading....
Reuters, Netbuilder et al are quote services that run externally and fetch quotes (price/volume) from the LSE and are *not* part of what would be considered the main trading platform. The problem isnt the actual OS (Linux) or tradiing platform, which has been heralded as a big step forward from the MS solution, but rather how the resulting trade data that is pushed from the exchange to the 3rd party sites and then ultimately to the end users.
botNet wab bab? Bot a bood bart on Binux
I thought I'd stuffed up big time when I hooked all of our users (around 1000 users at the time) up to test market prices rather than live data (Australian Stock Exchange) over market opening one morning (our product is aimed at day traders too). My stuff up seems minor in comparison to this.
Data is always seems to be the least respected item in the IT stack (TBV). Yet its the lifeblood. Referential integrity, blah. At your peril.
The offending code was traced back to :
$stockprice = power( 2, rand()*10 + 0.0001 );
Data vendors quickly addressed the issue by cutting the LSE out entirely, and rolling their own rand() function.
Fucking stock markets... .what a joke!
-Billco, Fnarg.com
Go ahead, downvote, I know you will.
On the other hand, a moon mission interfaced a host of complex systems including the lives of the astronauts. Difference is the level of integration of QA into the project.
...We'd have heard a million nerds orgasming on Slashdot, reminding us how much Windows sucks.
As someone on the receiving end of this problem (as head of Development for one of the largest UK stockbrokers), it has been immensly frustrating trying to get an admission of blame from anyone involved in this issue. Our clients are complaining to use because elements of the data we're displaying on our site are just so out of whack they are laughable. In the end we've just had to remove those elements of data from our website, and they remain removed until the fix is confirmed working on Monday. This is after several attempts to fix the issues during the week.
It's also causing serious issues for things like our black box trading engine which is used to drive automatic execution of client orders when the price moves into range for execution. The data being bad means these systems just have to be turned off, and clients are being advised to simply delete their orders.
This is, frankly, a pretty appalling situation as the service to our clients is being impacted pretty severely.
For an exchange that is attempting to position itself as the true 'global' exchange, this is doing an immense amount of damage to it's reputation.
is the concept of "high speed trading",
This horsecrap has to stop, the stock market has to be dismantled
I am curious which part of the Linux (kernel) they'll be going to blame and how Slashdot will turn it into FUD with adverts around it... xD
It's only a stock exchange. Why the fuck should they bother testing it first, right? This just goes to underline that QA does not exist in the software world.
Seven puppies were harmed during the making of this post.
As with most major issues there's bound to be a big ol' postmortem on this. As head of Dev you've probably got a unique insight into this, I'm curious as to your perspective on this, what you think the cause of failure might be? More strategic or more technical? Poor interface specification? Inability to handle queries under full load? From TFA there was supposedly 15 months of testing. So I'm curious as to why it failed, whether the testing simply wasn't realistic and/or thorough enough, or it was something that just wouldn't come up except on the live system?
I of course don't make mistakes, but what do you think would happen if for example you wiped 8.5 billion dollars off of the value of your company?
http://finance.yahoo.com/q/bc?s=NOK+Basic+Chart&t=3m
Do we have a world record here Mr. Elop?
Deleted
If it didn't crash and didn't drop its network connections, Linux was doing its job.
If the application software had bugs, then the application software developers are to blame.
An attempt to rubbish Linux.
After 20 seconds they should have realised that they should have tested more, and fired the programmers for allowing such mistakes. This is nothing to do with Linux.
You'd think the LSE would have learnt from their last computer system rollout which also had massive problems. No quality control = management problem.
Take Nobody's Word For It.
You just can't find qualified programmers for Linux. Sure, you can find many out of work in russia but that's a worse plan. The question then is, how much will it cost to re-train real programmers to work on its "free" operating system?
I bet many of them are 100% windows shops with a significant investment in Microsoft.
Would it really be so far fetched that MS supporters would try to embarrass a massive publicized Linux rollout?
From TFA there was supposedly 15 months of testing. So I'm curious as to why it failed, whether the testing simply wasn't realistic and/or thorough enough, or it was something that just wouldn't come up except on the live system?
Well, he had 15 months to test it, but spent 14 of them on /. instead of actually testing the push feed.
http://www.linuxjournal.com/node/1000121
That quoted phrase 'emerged at linux launch' is NOWHERE in the article. The article says "London Stock Exchange price data failures ‘emerged immediately at Millennium launch’". This is obviously an application issue, why is the summary sort of blaming the operating system?
WTF am I doing replying to an AC at 5 A.M on a Friday night?
No surprise here for anyone working with them. Their marked data specs have hundreds of pages while Chi-X EU have about 20. Chi-X EU quoting and trading volume is far greater. Coincidence? I say no - people voting with their feet and going from LSE to Chi-X EU (biggest trading platform in EU now), BATS EU, and other MTFs. Maybe the only thing keeping LSE still afloat is a fact that FTSE 100 index is computed using LSE prices. Once that gone, potentially as soon as MIFID II comes out, LSE is going to circle in the crapper.
Seems to me like a few Slashdot users got into positions of authority at the London Stock Exchange. .NET & M$ Windoze has bad performance. We should rewrite with Linux and Opens Sores for Maximum 1337.
Since these were Open Sores blowhards instead of good developers they now have a stock exchange where you can't trust the numbers
They typically have decent to excellent central systems, and it feels like they're treating it seriously and spending lots of money to maintain and upgrade them.
Now at the periphery ... the interconnections, for electronic payment and so on, they're a disgusting mess. It doesn't help that some protocols are truly horrible and should have died 40 years ago, but even when they've moved into the IP era, they keep on using outdated shit and/or idiotic settings.
At very least you'd expect a large scale dry-run with simulated data that should bring these errors out before becoming live.
Maybe. Such testing only tests the type of errors you have thought of and are anticipating. I was once blessed with a competent and skilled QA department and months of pre-release testing, simulation, fuzzing, beta testing, and yet when a product goes live to millions bugs become apparent. You just can't think of and test for everything in a sufficiently complex system. The real failure of this LSE rollout may have been not running the old and new systems in parallel while looking for discrepancies in the publicly reported data.
Canadians have bigger issues with the LSE
As one of the AC posters above, who was working for one of the involved companies suggested, it's a bit rough to suggest that the programmers are the ones who should be sacked.
The companies involved have systems that are built through years of acquisitions and management decisions that favour short-term 'wins' over long-term stability.
The focus in such an organisation is often on shipping at all costs (there are, after-all, regulatory, compliance and business obligations that MUST be met), and obviously in those situations, given a relatively constant set of team members skilled enough to do the work, the quality will suffer. The more systems added in to the mix from various acquisitions, the harder it gets to guarantee a quality outcome. Adding more people doesn't necessarily help on a project with a fixed-timeline, as training the new team members distracts those who actually know what they're doing, as do the required communications/meetings to co-ordinate everybody.
A software developer working in an environment like that has far less control over the 'quality' variable than a complete outsider might expect, and I for one hope that the management wear their fair portion of blame when the time for reckoning comes.
Zero, since programmers of Linux don't get paid.
This is strictly an application server implementation issue, not a Linux one. They had more (though probably different) problems with their previous MS Server infrastructure, which is, as I understand it, the reason behind the migration to Linux. So, don't tie this fiasco to the use of Linux, but rather to inadequate testing of the new systems in a real-world load scenario. This is really difficult for large scale distributed systems such as this is. I spent a LOT of time and engineering effort to build a testing framework for a major manufacturing execution system. It took the efforts of a fair number of really talented software engineers to emulate and resolve many of the issues (race conditions, hardware/software failures, etc) before we had a dead-bang reliable system that can run a major semiconductor FAB on a 365x24 basis. You must start thinking about these things during the design phase - not after the system has been implemented.
Sometimes, real fast is almost as good as real-time.
Microsoft has has had a lot of bad press lately as linux and java seem to be eating its lunch in the new world (high-performance, web base computing, small form factor and mobile computing). I suspect these issues are true but exaggerated.
Remember that the LSE had serious problems using the MS-based system before. Remember the curious wording of the recent problems with the .NET-based system. Somehow, they tried to link this to the fact that the Linux-based system was being tested (of course it had nothing to with that).
Windows Phone 7 is seriously buggy (many basic functions still don't work). Android (java) and IPhone (freebsd) have a lock on the smartphone market and MS's feeble attempt is too little, too late. Selling low-quality crap only works if you have a monopoly and it must be hard for them to see that they can't compete in the new emerging markets.
Even Microsoft has to provide its updates via Akamai linux-based servers. It's hotmail servers couldn't handle the load when switched to Windows Server and had to be switched back.
All high-end Web-based services (facebook, twitter, google, yahoo, etc) are all run on linux or freebsd systems.
Does it surprise anyone that an issue with a high-profile linux system is going to be big news for the Windows world?