Slashdot Mirror
How To Lose $172,222 a Second For 45 Minutes
An anonymous reader writes "Investment firm Knight Capital made headlines in 2012 for losing over $400 million on the New York Stock Exchange because of problems with their algorithmic trading software. Now, the owner of a Python programming blog noticed the release of a detailed SEC report into exactly what went wrong (PDF). It shows how a botched update rollout combined with useless or nonexistent process guidelines cost the company over $172,000 a second for over 45 minutes. From the report: 'When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. ... During the deployment of the new code, however, one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.'"
See, the private sector can blow money faster than the public sector (OmabaCare site).
Table-ized A.I.
I'm sure someone will chime in and claim to be the em-effing Change MASTER! but this seems like an ordinary error, the sort I've seen a hundred times before, where one server is a tiny bit wonky, and during the change, something doesn't happen as expected. Normally it's an "inexpensive" error, where some of your VPN users get randomly disconnected... ...and sometimes it's the sort of error where you lose half a billion dollars an hour by HFT trading... ...badly.
This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.
It can be stopped. And it should be stopped. And the only reason it is not being stopped is because too many rich and powerful people make too much money on it.
Correction: "Obama" not "Omaba". Sorry 'bout that. It was not intentional (although there's probably a mean joke in there somewhere.)
Table-ized A.I.
Damn the validation. Full speed to prod!
No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive.
If I'd done this kind of thing when I was working with the exchanges I would have been fired in a heartbeat. Whoever failed to utilize best practices, or whoever failed to allow the utilization of best practices had damn well better have been fired. This is incompetence of the highest level and a perfect example of why ITIL based best practices were born.
So...where did the money go?
They had some code that processed orders in a special way. There was a flag on the order they could set that would trigger that code. We will call this Power Peg. They later moved away from that functionality but it still existed in the system. It sat there for years untested and unused. 9 years later they added new functionality and decided to reuse that same flag. The new code also disabled Power Peg.
When they pushed the new code into production, they missed a server. That missed server still had Power Peg looking for that flag. Orders started setting that flag and it was processed correctly on all but one server. But that last server was placing orders incorrectly. The logic that Power Peg used was not valid anymore. In a panic they rolled back the code on the servers. Not knowing that Power Peg was the issue, they now had all the servers running Power Peg again.
Capitalism is very effective in what it does
Nerdism, on the other hand, is very detail in what it does
Combining both and you will get an invincible beast
Unfortunately (or fortunately, depending on how you look at it), the investment firm "Knight Capital" does not treasure the nerds enough to put them into position that can have effective oversee powers over technology deployment
I hope the 400 million loss will wake them up
No more we nerds should work under them capitalists --- they need us MUCH MORE than we need them
Muchas Gracias, Señor Edward Snowden !
Knight had no written procedures that required such a review. And as the deity of your choice only knows, you are not required to expect uncommon sense from your employees.
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
In the same thread where I can find 1000 people going on about how efficient capitalism is I can find another (sometimes the same) 1000 people complaining all the dumb things their companies do. Well, which one is it? It doesn't work both ways people. Could it be that people are people, no matter what banner they're organized under?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Didn't RTFA, but summary makes me go WTF in several places:
1. Python. I thought all the quants liked C, assembler, and even VHDL for their high frequency stuff. No matter
2. "2nd technician to review". If this were flight hardware or a bridge or skyscraper, there would be a second "technician" to review and at least one "engineer" to personally sign off that what was built/deployed is a) done right and b) is what you want
3. "no written procedures". There are a very small number of things in life about which it is absolutely imperative to keep a rod firmly up one's ass: a. moving machinery, b. formal mathematics, and c: hundreds of millions of dollars of your clients and shareholders' money.
Actually, they asked the SEC to bail them out and got the boot. They had to do a round of private investment and diluted shareholders' value quite a bit. It's generally cited as the "right way" to deal with companies that fuck up and lose billions. Shame we can't do the same with the banks.
If I have been able to see further than others, it is because I bought a pair of binoculars.
$400 billion per year in interest payments is "essentially zero"? That number is actually almost manageable, the BIG problem is that it's snowballing.
You'll recall Obama has been saying that if he wasn't allowed to borrow more, the government would default - wouldn't be able to pay the interest due. We're borrowing to the interest on our borrowing. That's when you know you're fucked, when you're maxing out one credit card to pay the minimum payment on another card. That's essentially what the US government is doing. We're in a trap the we're borrowing more and more in order to make the payments on existing debt, so the debt and the interest just keeps getting bigger and bigger until 100% of our tax money goes to pppay interest, leaving no money for essential government services.
Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates. Eight years ago, our debt to GDP ratio was about 35%. In 2014, it should hit 70%. That tells us we are about six to eight years from becoming Greece.
The difference between Greece and though, is that Greece is small enough to be bailed out. Nobody has $5 trillion to bail the US out.
And nothing of value was lost.
Alogrithmic trading and HFT are the same thing.
No they aren't.
That is the only reason to use a computer in the first place; otherwise you just need a handheld calculator
Algorithmic trading predates the invention of digital computers. People with (you guessed it) handheld calculators would crunch numbers according to prescribed rules, and then submit their orders to the trading pits.
Two things. First, the free market does things efficiently, including dumb things. Microsoft built the Surface efficiently - $390 per tablet. Government spent over $1,000 each buying them.
Secondly, there's dumb, and there's government dumb.
A dumb corporation requires three people to do a job that one person could do. A government gets the same job done by creating a new agency which contracts with three companies at $320 million each, requires that it be done in Pascal, and prohibits any testing until the project is "complete", at which time it's six years after tube event that the project was created to handle.
I just saw one of our most efficient government agencies, one that wins efficiency awards, spend over $19,000 on a project I could do in two days. Funny thing is, I WORK for that agency, so it would have cost them almost nothing to have me do it.
We don't need no steenking process!!!!
Cheers,
Dave
They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
Ben
No. You're basically wrong on every point you bring up.
"Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates."
No, there is no magic debt/GDP limit: http://krugman.blogs.nytimes.com/2013/05/26/reinhart-and-rogoff-are-not-happy/
"That tells us we are about six to eight years from becoming Greece."
No, Greece doesn't have its own currency, so we are in no way the same as Greece: http://krugman.blogs.nytimes.com/2013/10/18/the-china-debt-syndrome/?_r=0
The problem is that the federal reserve is specifically designed to siphon capital out of america, and also force the government to borrow to issue new money.
Banks have shares in the federal reserve, and those banks, even if they are foreign owned, are entitled to a statutory dividend.
I worked recently for a financial company doing algorithmic trading. Here's how it worked:
Every morning at 7 AM, Master.xlsx would run an "UpdateOrders" macro to check for the prior day's events, and possibly change some green circles to red. Whenever an account under that system needed to sell something, the red circles would go first. Once each year, new green circles were added and red ones were eliminated.
11.5740741 microhertz is indeed high frequency, compared to something like 413 nanohertz.
You do not have a moral or legal right to do absolutely anything you want.
the body of some programmer is at the bottom of a river.
Shoes for Industry. Shoes for the Dead.
To err is human. To screw up 100,000 things per second requires root.
Of all the things that probably shouldn't be tested in production...! I want to write my own trading bot, but I'm not going to give it control of my entire portfolio before I thoroughly test it.
I don't always test my code, but when I do, I do it in production!
>> 11.5740741 microhertz is indeed high frequency, compared to something like 413 nanohertz.
actually, no. it's very, very, VERY, low frequency.
11.5740741 microhertz is equivalent to a time period of 11574 seconds and 413 nanohertz to 413000000000 seconds.
that means one action at every X seconds.
science.
Suddenly 400 million dollars disappearing in nanoseconds
Oh, you don't say? It just disappeared, huh?
Here's a clue - it didn't disappear. Somebody made money off their mistake. So I fail to see what the issue is you are all whining about. A company messed up and paid a huge price for that error, and someone else profited from it.
Which all goes to prove that managing directors still do not understand computers and love to cut IT departments
It was not designed for gambling, which is what it has become.
Speculation and market making are natural parts of the market even when they aren't intended. And it's not gambling if there is as usually is the case a reasonable expectation for profit.
So in your book, seven orders of magnitude increase in frequency is not relatively high frequency.
Indeed. There should be a rule that says that any money you own now, you should spend in the coming ten years, or you lose it. Kind of like those cellphone contracts, where you buy a bundle of text-messages that is valid for only one month. We should give capitalism a taste of its own medicine.
If Pandora's box is destined to be opened, *I* want to be the one to open it.
At least in the context of the whole world. The fact that the process has broken down, especially in America, is as a result of previous exclusion of large parts of the world's population from economic growth. Now that the factors that were preventing their sharing in the benefits have been removed, we are seeing a RAPID reduction in rates of absolute poverty around the world. See http://www.brookings.edu/research/interactives/2013/ending-extreme-poverty for the data. Which is not to claim that capitalism is perfect; the tendency for monopolistic behaviour is a major danger, but the potential for effective relief of poverty, is, according to world experience in the past 60 years, clearly in favour of capitalism.
Both might work.
In Latin America at least the reason that the poor are less poor is the rise of socialist and semi-socialist governments that have put tight controls on corporations and the very wealthy. The GNP of the countries of South America has almost tripled in the decade since the rise of 'socialismo' across the continent while the percentage of that rise going to the poor and middle class has (IIRC) quadrupled. You can't have a healthy economy without it being driven by the economic patterns of the lower and middle classes, and unbridled capitalism concentrates wealth and power in the hands of the capital holders.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
There's a reference in K&R 2nd edition to Orwell's 1984. Turns out that in Winston's world the "C" language was the language of technocrats.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
Oh man my wife will be pissed then since we were saving that for retirement. New sports car and a really nice 40 in the north woods here I come.
Time to offend someone
> but there is no mechanism to ensure that it DOES do things efficiently.
Sure there is. Inefficient companies lose in the market to more efficient competitors.
Unless of course the more inefficient company is much better in some other respect.
> Except that it can be smart to borrow to pay off debt--if the rates on new new debt are lower than the rates of the old debt.
Yeah, "borrowing" to PAY OFF a higher interest debt isn't _really_ borrowing, it REDUCES your total liability.
That's not what the government is doing. The government is borrowing (increasing debt) to pay INTEREST on debt.
The more they borrow, the more interest there is. The more interest there is, the more they borrow to pay the interest.
That's a vicious cycle spiraling to collapse.
> Maybe this is why the TEA party people think it wouldn't be so bad to "default." Maybe they imagine it simply means not making interest payments
> to China, the Social Security administration, and JP Morgan, and forget that it also means not paying for new Humvee tires or civil service salaries.
I think it's the opposite. Debt service on existing debt is mandatory spending - you pay your debts. That's very much part of the conservative ethic.
On the other hand, there's a lot of spending that is not justified, in their eyes. Some are downright stupid - bridges to nowhere. Some are nothing
but giveaways to donors - in just the last few months, THREE times executives have walked away from "green" companies financed by the
taxpayer. The latest (ECOtality) took $99.8 million of our money and walked away. Others are well intentioned programs being run so poorly that they need a complete overhaul. We're spending twice as much on food stamps as eight years ago, and 20% - 40% of that is fraud. Some recipients of
government aid claim they have a new baby every 2-3 months. How does that work?
From 1900 to 1940, government spending ranged between 6% - 20% of GDP, Today it's 39%.
Does the government really need to be almost half of the country?
By reducing fraud and utter waste by just 50%, we could pay off the debt, which would free up $415 BILLION dollars,
14% of the federal budget, for tax relief and worthwhile projects while at the same time eliminating the massive weight
we're saddling our children with.
* (Considering only _federal_ government spending, 1900-1940 it ranged 2% - 10% of GDP. Today it's 23%.)
There's another important difference between Greece and the US - Greece doesn't have its own currency. The US can inflate itself out of debt. Greece was as trapped as an individual who maxed out their credit cards.
Oolite: Elite-like game. For Mac, Linux and Windows
> There is no point at which a "death spiral" occurs.
Let's do some simple grade school math. The yield on government bonds, the cost of borrowing, has ranged from 2% to 15%.
To make the math easy while being generous to your claim, let's predict 5% as a long term average. Let's say the debt is 20X GDP.
5% of that is 1 X GDP - everything the country produces goes to pay interest on debt.
When the entire GDP is spent on interest payments, nothing is left for the people to eat, get shelter, education, etc.
Everybody's entire paycheck is sent to China and other creditors. That sounds like collapse to me.
So we've established via grade school math that collapse occurs by the time you reach 20 X GDP. If you can pass
4th grade math, you can see there IS a point where it collapses. The only debatable question is exactly where
that point is.
Those, like you, who advocate for big government spending say that the government needs to spend at least 10% - 30% of GDP
on services and entitlements. That leaves 70% for interest payments, so debt can't be more than 14 X GDP.
Hmm, people need to eat, and pay utilities, etc., so they'll need to keep half of their pay check. To leave people will
half of their paycheck, that's a max of 7 X GDP for government debt. This is still basic arithmetic and we've seen that
anything above 7 X GDP collapses. We could get a more specific number by doing advanced things like decimal division,
but I think the point is clear - there absolutely is a point of collapse, and with the debt doubling every 8 - 10 years, it
won't be long before we hit it.
> Post WW2, Britain had a 220% debt to GDP ratio. Post WW2, the US had a 121% debt to GDP ratio. Both countries were fine.
It's arguable whether the US is "fine", Obama says it's "definitely NOT fine", but you're confusing a one time event with a habit of chronic over spending. WWW2 was a one time event, like buying a house. What we have today is chronic, permanent overspending every single year - like a person
who makes $5,000 / month and spends $7,000 / month every single month.
> The US can inflate itself out of debt.
Only if defaulting is okay. Paying a 10€ trillion debt with 1€ trillion worth of bills is defaulting.
You'd also need chronic long term inflation of at least 14% to get there. That would be a problem.
Yes, definitely when your beloved government mama is blowing money left and right and someone points that out and tries to do something about it, the smart thing is to fire them.
You are the problem.
Er we were not fine. I'm not sure who you know that went through the post WW2 years in Britain but according to my grandparents it was pretty austere. A large chunk of debt was to the USA / Canada, and we only finally paid your WW2 loan off in 2006. If you don't mind your future generations being under the yoke of debt until 2075 or so then go ahead...
Phillip.
Property for sale in Nice, France
*pfft*...amateurs...I would have only lost them 50k per second...
Why has nobody modded this post up? he is right, this is econ 101 and it is stupid not to borrow money at ultra low rates if you can create an asset with a return greater than the rate
Lol, who made you the Value Police? I see no value in about 75% of the things people do. What value is created by selling a baseball card for $2M?
Exactly. Someone valued those stocks and had the disposable income to purchase them.
So it's a not a glitch in the algorithm per se but a wrong integration by a technician which is basically lack of an automated procedure to upgrade and roll out important software. OK