How To Lose $172,222 a Second For 45 Minutes

An anonymous reader writes "Investment firm Knight Capital made headlines in 2012 for losing over $400 million on the New York Stock Exchange because of problems with their algorithmic trading software. Now, the owner of a Python programming blog noticed the release of a detailed SEC report into exactly what went wrong (PDF). It shows how a botched update rollout combined with useless or nonexistent process guidelines cost the company over $172,000 a second for over 45 minutes. From the report: 'When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. ... During the deployment of the new code, however, one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.'"

Comments

There should be a mandatory one second delay.

[Anonymous Coward]

This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.

It can be stopped. And it should be stopped. And the only reason it is not being stopped is because too many rich and powerful people make too much money on it.

Re:There should be a mandatory one second delay.

[ShanghaiBill]

You are confused. "Algorithmic trading" and "High Frequency Trading" are two different things, used for two different purposes. This was caused by AT, and you are complaining about HFS. That makes no sense.

Failure of best practices

[onyxruby]

No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive.

If I'd done this kind of thing when I was working with the exchanges I would have been fired in a heartbeat. Whoever failed to utilize best practices, or whoever failed to allow the utilization of best practices had damn well better have been fired. This is incompetence of the highest level and a perfect example of why ITIL based best practices were born.

Re:Failure of best practices

[Billly Gates]

Dude, Wall Street traders demand software engineers get things done in minutes and hours!

There is no process as it all had to be done yesterday if you ask anyone who has worked in that environment.

They pay top dollars and change core algorithms on the fly as each millisecond costs money to a competitor who has a more efficient trading algorithm that steals from their own HTC network.

So when when skims and manipulates the prices in milliseconds they steal from those who have process engineered designs who are too slow to react.

If it messes up they get bailed out by the SEC anyway and they can just fire the programmer.

IT Debt

[Anonymous Coward]

They had some code that processed orders in a special way. There was a flag on the order they could set that would trigger that code. We will call this Power Peg. They later moved away from that functionality but it still existed in the system. It sat there for years untested and unused. 9 years later they added new functionality and decided to reuse that same flag. The new code also disabled Power Peg.

When they pushed the new code into production, they missed a server. That missed server still had Power Peg looking for that flag. Orders started setting that flag and it was processed correctly on all but one server. But that last server was placing orders incorrectly. The logic that Power Peg used was not valid anymore. In a panic they rolled back the code on the servers. Not knowing that Power Peg was the issue, they now had all the servers running Power Peg again.

Re:The efficiency of capitalism

[TsuruchiBrian]

The wealth that the money represented was not "lost", but rather redistributed. Efficient redistribution of wealth is a strength of the private sector not a weakness. The private sector is good at redistributing money where it needs to go for economic growth. This company was not exercising an appropriate level of caution and it's money was redistributed elsewhere.

This is what I like best about /.

[rsilvergun]

In the same thread where I can find 1000 people going on about how efficient capitalism is I can find another (sometimes the same) 1000 people complaining all the dumb things their companies do. Well, which one is it? It doesn't work both ways people. Could it be that people are people, no matter what banner they're organized under?

Re:This is what I like best about /.

[Anonymous Coward]

Capitalism produces large corporations that are very efficient at doing dumb things.

Re:This is what I like best about /.

[Anonymous Coward]

The money wasn't destroyed, it went to other traders. Maybe they will donate it to children with cancer?

Re:This is what I like best about /.

[Tablizer]

The public and private sector do different things better. While the private sector is more efficient in terms of service or widgets per hour, there's also heavy pressure to trick, mislead, and rip off consumers and customers.

Such "trick" pressure is noticeably lower in the public sector. There is relatively little pressure for a line worker and middle manager to rip off or trick customers because they don't have to compete with other cut-throat competition and are not sweating over narrow margins. They generally don't want the drama of customers sending or phoning complaints and so do an adequate job, even if a bit slow at it.

Certain kinds of services are better under one versus the other. If it's a service where it's easy to fool customers, especially over longer-term features, then it's probably best done by or heavily regulated by the gov't. If it's something that's relatively easy for a consumer to verify the quality of, then it's probably best done by the private sector.

Re:The efficiency of capitalism

[techno-vampire]

I think you're being a tad too literal here. As far as Knight was concerned the money was lost because they didn't have it any longer and they had nothing to show for it.

Re:Garden Variety Upgrade SNAFU

[Alan Shutko]

There were a number of errors made here.

1. They failed to deploy to one of eight servers
2. They failed to automate the deployment to the servers such that it would be impossible to deploy to all servers without knowing.
3. They didn't have a step between code deployment and production activation where they could validate all 8 servers. For instance, in our company, we deploy the prod code to the prod servers but leave them in a "stage" environment, where the production load balancer doesn't hit those instances. Once we've validated, we then switch the load balancers to point to the correct instance.
4. They failed to quickly back out a change when they realized it was having problems. In fact, they backed out the part on seven servers but not the flag that was being sent to the servers, which made things worse.
5. They failed to have a risk-mitigation backstop in place which would have prevented these orders from being submitted once they hit a certain amount, and which was required by SEC Rule 15c3-5(c)(1)(i).

There were a lot of places that you could put in a control to prevent or limit the effect of these kinds of errors, and that's the lesson people need to learn. Yes, mistakes happen! But try to make it hard to make a mistake, easy to recover from a mistake, and really easy to NOTICE a mistake.

Re:private dumb: $20K. Govt dumb: $400 billion

[rsilvergun]

Um, Microsoft used borderline slave labor and a complete disregard for the environment to build the Surface pro. And not the kind of disregard that we get to ignore. It's the kind that makes 'Cancer Villages' (google it)

And the reason your gov't is making an agency to do a contract to get something done isn't inefficiency, it's socialism. Seriously. It's make work projects to try and spread some money around because otherwise the natural tendency if for money to be hoarded at the top by a few wealthy oligarchs (we call 'em the 1% these days).

The US Postal Service has God like efficiency. Just ask Netflix. Or just mail a letter and watch it travel across the breath of the United States in less than a week for 33 1/3 cents. The Space programs were (and are) amazing, and did things private industry couldn't hope to accomplish and that we're still reaping the benefits from today. The Gov't can be plenty efficient when it wants to be. It can also be very inefficient when it wants to. There are times when that's a good thing.

Re:Combining nerdism with capitalism

[happyhamster]

>> Capitalism is very effective in what it does

Pumping the money from the poor to the rich.