London Stock Exchange Price Errors 'Emerged At Linux Launch'

DMandPenfold writes "Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources. Thomson Reuters, Interactive Data and Netbuilder are among the largest data vendors, providing share prices to traders, that have been displaying pricing problems on some stocks throughout the week. Even the LSE's own data vendor, ProQuote, experienced problems. Concerns are being raised that there could be mistakenly setup connections or incorrect software interfaces at some of the large data vendors. Alternatively, there may be a data caching issue at the LSE that means data going out is not properly synchronised between different systems."

Comments

Re:Testing?

[PNutts]

Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.

From TFA: "Meanwhile, the private view at the exchange is that the 15 month testing window should have been plenty of time for the vendors to interface perfectly with it."

Re:Linux fails... AGAIN

[TooMuchToDo]

Don't forget the US DoD, The US Navy Submarine Fleet, the FAA, and Cisco:

http://www.focus.com/fyi/information-technology/50-places-linux-running-you-might-not-expect/

I don't see IBM running Watson on Windows 2008 Server because, well, you know....they want it to work.

either sympathy or accusation

[DreadPirateShawn]

My heart goes out to the devs "working long hours and night shifts" to suss this out.

That being said, the line that catches my eye most is: "The fact the majority of smaller vendors were fine demonstrated that those having trouble had made mistakes." In my experience, that means one of two things:

1) The devs configuring the system didn't properly account for the sheer scale of the stress on their systems.
2) The smaller vendors took the change more seriously, and being smaller and more flexible, successfully updated their systems to interact properly with the new systems.

Or, of course, both.

Re:either sympathy or accusation

[micheas]

Or the big vendors were doing excessive cost cutting to maintain profits during the recession.

As long as this is just a one weekend nightmare, the execs that ordered the QA not done are looking like great business monkeys. they took a risk that cut costs and managed to defect the blame to the exchange when it blew up, and it looks like nothing bad will happen to the big vendors, unless one of the small vendors gets a really unethical sales monkey that steals a large chunk of business from a couple of the big vendors.

Re:either sympathy or accusation

[Anonymous Coward]

As I work for one of the vendors cited in the article, I can assure you that the reason is freaking simple: the systems that consume the feeds to move them into the trader's desktop are absolute mess of epic proportions riddled with mind boggling wtfs. Missing data, blank values, wrong numbers are common on those systems (root cause are mainly race conditions everywhere, and assumptions about missing data), assembled from random left and right acquisitions. There is no way to really know how they will behave under stress, and corporate culture will not allow managers to actually fix the issues (extremely costly, and corporate bonuses are not aligned to quality).

In fact, it is much cheaper for everyone to just pretend this is a one time occurrence, to fire a couple of middle managers and continue frantically fixing bugs in the mess until it sorta works.

Re:either sympathy or accusation

[ArchieBunker]

We don't like High Frequency Trading because it fucks up the economy for everyone else.

Re:Linux fails... AGAIN

[MichaelSmith]

I work on large scale air traffic control systems which run Linux and I don't envy the LSE in their task. Most of our interfaces are relatively simple and go out to organisations with a good history of validating interfaces. This trading system seems to have to interface to a lot of little offices around the place running various implementations. Its no surprise some of the interfaces weren't tested to the point where they are known to work 100%, though they may be 100% correct.

If only they had thought of that

[raftpeople]

Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.

Quote from the project manager:
"Dang. If only we had read Tubal-Cain's post before going live. Who would have thought to run test data through this darn thing?"

Re:Testing?

[cronius]

From this article a couple of days earlier http://www.computerworlduk.com/news/it-business/3261625/frustrations-mount-over-london-stock-exchange-data-interface-problems/ :

Publicly, the LSE and the data vendors say they are working together as engineers scramble to fix the price data problems, even though some statements to clients suggest elements of blame.

But behind the scenes, sources close to the several of the parties, including the exchange, told Computerworld UK they were immensely frustrated at the reputational impact of the problems.

The LSE is taking the position that its data feeds are working correctly. Industry sources said the exchange had placed a great deal of emphasis on the launch, which was largely providing successful high-speed trading, and that it had allocated sufficient time - 15 months - for the vendors to be fully prepared for the new system.

The LSE statement appeared to place some blame with the vendors. "Unfortunately a couple of market data vendors have experienced some specific issues aligning to the new Millennium Exchange platform and we are actively working with them to help resolve their issues," it said.

"All other trading customers and vendors are successfully trading on the new platform, benefiting from Millennium Exchange's superior functionality and speed."

Emphasize mine. Looks to me like the exchange is getting the flack for a couple of amateurish (or saboteurish?) vendors.

Quote Services

[Anonymous Coward]

Article is a a bit misleading....
Reuters, Netbuilder et al are quote services that run externally and fetch quotes (price/volume) from the LSE and are *not* part of what would be considered the main trading platform. The problem isnt the actual OS (Linux) or tradiing platform, which has been heralded as a big step forward from the MS solution, but rather how the resulting trade data that is pushed from the exchange to the 3rd party sites and then ultimately to the end users.

Re:Quote Services

[Anonymous Coward]

Hi,

The NYSE Arca FIX docs are not quite relevant to this - as what we're talking about here is the market data API.

The FIX protocol (although it does support market data transactions), is typically not the protocol of choice for market data. Typically it'll be non standard provider specific protocols which are optimised for delivering partial updates of information in a very bandwidth efficient manner.

In the parent post someone mentions that Reuters, Netbuilder etc fetch quotes - which is also not entirely correct either. They accept push feeds from the exchange, which contain tick updates to the information about the instruments traded on that exchange. The generation of these ticks is triggered when a match occurs on the order book - which may (or may not) involve a change in price data. They then typically augment this information with other data - such as other identifiers, or derived information such as previous days close value / percentage etc.

It is with these derived fields that the ongoing issues seem to lie, although for a certain period of time, the exchange was actually pushing out zero values for the bid/offer fields. As you can imagine, these plays merry hell with systems that attempt to drive automatic execution based on current price.

I am informed by Netbuilder that they will have pushed a final fix to live within a subset of their servers - but it will probably be Tuesday before this is available across their whole system. This is over a week since the original identification of the issue, not good!

Effect on stockbrokers

[Anonymous Coward]

As someone on the receiving end of this problem (as head of Development for one of the largest UK stockbrokers), it has been immensly frustrating trying to get an admission of blame from anyone involved in this issue. Our clients are complaining to use because elements of the data we're displaying on our site are just so out of whack they are laughable. In the end we've just had to remove those elements of data from our website, and they remain removed until the fix is confirmed working on Monday. This is after several attempts to fix the issues during the week.

It's also causing serious issues for things like our black box trading engine which is used to drive automatic execution of client orders when the price moves into range for execution. The data being bad means these systems just have to be turned off, and clients are being advised to simply delete their orders.

This is, frankly, a pretty appalling situation as the service to our clients is being impacted pretty severely.

For an exchange that is attempting to position itself as the true 'global' exchange, this is doing an immense amount of damage to it's reputation.

Re:Testing?

[CaymanIslandCarpedie]

I think you can take off the tinfoil hat on the sabotage thoughts. For one LSE's own quote providing solution is experiencing the same issue. So unless they have some diabolical plan to sabotage themselves, I think that fact should rule that out. Also, this is effecting large players. These are billion dollar companies whose sole propose is to provide accurate data and provide it fast. Without that they simply don't exist. You'd have to live in a very conspiracy theory driven existence to think they'd all really all throw their own companies and livelihoods away in some dark plan to get the LSE or Linux.

Now would all these large players all have made the exact same errors in their interfaces so they are seeing these same issues while the smaller players seem to have got it right? Sure. But the simplest explanation, of course is they are all experiencing the same issue because of some upstream problem with scaling to their large volumes. Those actually involved seem to point to some caching issue. Is that true? No clue, but they are certainly in a better position to offer analysis of the issue than you or I. That doesn't mean Linux is bad for christ's sake (so you don't have to go on some crazy conspiracy hunt to explain it away). If it is true (and we certainly don't know that it is, though the information currently available may point that way), you know what? There is software which just happens to be running on Linux, which may have an issue. No big deal. Happens every day on every platform. Not some big black eye on Linux if some software that happens to run on Linux has issues and no need to start jumping to conspiracies to explain it away.