Slashdot Mirror
How To Lose $172,222 a Second For 45 Minutes
An anonymous reader writes "Investment firm Knight Capital made headlines in 2012 for losing over $400 million on the New York Stock Exchange because of problems with their algorithmic trading software. Now, the owner of a Python programming blog noticed the release of a detailed SEC report into exactly what went wrong (PDF). It shows how a botched update rollout combined with useless or nonexistent process guidelines cost the company over $172,000 a second for over 45 minutes. From the report: 'When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. ... During the deployment of the new code, however, one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.'"
See, the private sector can blow money faster than the public sector (OmabaCare site).
Table-ized A.I.
I'm sure someone will chime in and claim to be the em-effing Change MASTER! but this seems like an ordinary error, the sort I've seen a hundred times before, where one server is a tiny bit wonky, and during the change, something doesn't happen as expected. Normally it's an "inexpensive" error, where some of your VPN users get randomly disconnected... ...and sometimes it's the sort of error where you lose half a billion dollars an hour by HFT trading... ...badly.
This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.
It can be stopped. And it should be stopped. And the only reason it is not being stopped is because too many rich and powerful people make too much money on it.
Correction: "Obama" not "Omaba". Sorry 'bout that. It was not intentional (although there's probably a mean joke in there somewhere.)
Table-ized A.I.
Damn the validation. Full speed to prod!
No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive.
If I'd done this kind of thing when I was working with the exchanges I would have been fired in a heartbeat. Whoever failed to utilize best practices, or whoever failed to allow the utilization of best practices had damn well better have been fired. This is incompetence of the highest level and a perfect example of why ITIL based best practices were born.
Computers that create no value for mankind now make more in a second than you to all year.
So...where did the money go?
They had some code that processed orders in a special way. There was a flag on the order they could set that would trigger that code. We will call this Power Peg. They later moved away from that functionality but it still existed in the system. It sat there for years untested and unused. 9 years later they added new functionality and decided to reuse that same flag. The new code also disabled Power Peg.
When they pushed the new code into production, they missed a server. That missed server still had Power Peg looking for that flag. Orders started setting that flag and it was processed correctly on all but one server. But that last server was placing orders incorrectly. The logic that Power Peg used was not valid anymore. In a panic they rolled back the code on the servers. Not knowing that Power Peg was the issue, they now had all the servers running Power Peg again.
Stubs that don't bring the system to a halt if called? This is why I assert, C is superior.
Capitalism is very effective in what it does
Nerdism, on the other hand, is very detail in what it does
Combining both and you will get an invincible beast
Unfortunately (or fortunately, depending on how you look at it), the investment firm "Knight Capital" does not treasure the nerds enough to put them into position that can have effective oversee powers over technology deployment
I hope the 400 million loss will wake them up
No more we nerds should work under them capitalists --- they need us MUCH MORE than we need them
Muchas Gracias, Señor Edward Snowden !
What is neglected in this article is the story how the FEC compensated them for their loss and reversed the transactions.
So nice they get the same protections that you (slashdot readers) and I get when we invest. I wonder if any if it is tax based. Better get to work to pay for someone elses mistakes then.
http://saveie6.com/
Knight had no written procedures that required such a review. And as the deity of your choice only knows, you are not required to expect uncommon sense from your employees.
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
In the same thread where I can find 1000 people going on about how efficient capitalism is I can find another (sometimes the same) 1000 people complaining all the dumb things their companies do. Well, which one is it? It doesn't work both ways people. Could it be that people are people, no matter what banner they're organized under?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
"Knight had no written procedures that required such a review."
They will now!
Alogrithmic trading and HFT are the same thing. HFT needs an (insert term here) in order to do its trades. That is the only reason to use a computer in the first place; otherwise you just need a handheld calculator or at most a Python sctipt for portfolio management.
All trades shall be entered ONLY by a human and signed for in blood!
If there are any illegal problems with your trade. We come get the rest of your blood.
Didn't RTFA, but summary makes me go WTF in several places:
1. Python. I thought all the quants liked C, assembler, and even VHDL for their high frequency stuff. No matter
2. "2nd technician to review". If this were flight hardware or a bridge or skyscraper, there would be a second "technician" to review and at least one "engineer" to personally sign off that what was built/deployed is a) done right and b) is what you want
3. "no written procedures". There are a very small number of things in life about which it is absolutely imperative to keep a rod firmly up one's ass: a. moving machinery, b. formal mathematics, and c: hundreds of millions of dollars of your clients and shareholders' money.
I think your calculations are off.
172,000*60(seconds in a minute)*60(minutes in an hour)*24(hours in a day)*365(days in a year)*20 = $108,483,840,000,000.
I may have not really used math all that much in the past 10-years but I think I still remember this much.
transaction....into an account that they created!
$400 billion per year in interest payments is "essentially zero"? That number is actually almost manageable, the BIG problem is that it's snowballing.
You'll recall Obama has been saying that if he wasn't allowed to borrow more, the government would default - wouldn't be able to pay the interest due. We're borrowing to the interest on our borrowing. That's when you know you're fucked, when you're maxing out one credit card to pay the minimum payment on another card. That's essentially what the US government is doing. We're in a trap the we're borrowing more and more in order to make the payments on existing debt, so the debt and the interest just keeps getting bigger and bigger until 100% of our tax money goes to pppay interest, leaving no money for essential government services.
Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates. Eight years ago, our debt to GDP ratio was about 35%. In 2014, it should hit 70%. That tells us we are about six to eight years from becoming Greece.
The difference between Greece and though, is that Greece is small enough to be bailed out. Nobody has $5 trillion to bail the US out.
And nothing of value was lost.
Two things. First, the free market does things efficiently, including dumb things. Microsoft built the Surface efficiently - $390 per tablet. Government spent over $1,000 each buying them.
Secondly, there's dumb, and there's government dumb.
A dumb corporation requires three people to do a job that one person could do. A government gets the same job done by creating a new agency which contracts with three companies at $320 million each, requires that it be done in Pascal, and prohibits any testing until the project is "complete", at which time it's six years after tube event that the project was created to handle.
I just saw one of our most efficient government agencies, one that wins efficiency awards, spend over $19,000 on a project I could do in two days. Funny thing is, I WORK for that agency, so it would have cost them almost nothing to have me do it.
We don't need no steenking process!!!!
Cheers,
Dave
They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
Ben
No. You're basically wrong on every point you bring up.
"Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates."
No, there is no magic debt/GDP limit: http://krugman.blogs.nytimes.com/2013/05/26/reinhart-and-rogoff-are-not-happy/
"That tells us we are about six to eight years from becoming Greece."
No, Greece doesn't have its own currency, so we are in no way the same as Greece: http://krugman.blogs.nytimes.com/2013/10/18/the-china-debt-syndrome/?_r=0
The problem is that the federal reserve is specifically designed to siphon capital out of america, and also force the government to borrow to issue new money.
Banks have shares in the federal reserve, and those banks, even if they are foreign owned, are entitled to a statutory dividend.
"2013:Q2: 100.46407 Percent of GDP" -- According to the St. Louis Fed. http://research.stlouisfed.org/fred2/series/GFDEGDQ188S
Of course, that doesn't include the unfunded liabilities (Special T-bills) in Social Security.
the body of some programmer is at the bottom of a river.
Shoes for Industry. Shoes for the Dead.
To err is human. To screw up 100,000 things per second requires root.
I hope they would of lost all their money.... Making money without making any REAL work, but only on the fact that they have money to begin with.
This is what is wrong with the world today.
Of all the things that probably shouldn't be tested in production...! I want to write my own trading bot, but I'm not going to give it control of my entire portfolio before I thoroughly test it.
lots of people posting stuff that most in their right mind would never say.
seems to be a lot of paid shills on here
I don't always test my code, but when I do, I do it in production!
Which all goes to prove that managing directors still do not understand computers and love to cut IT departments
It was not designed for gambling, which is what it has become.
Speculation and market making are natural parts of the market even when they aren't intended. And it's not gambling if there is as usually is the case a reasonable expectation for profit.
because then they can call replay and make out they never lost money:
http://www.bloomberg.com/news/2013-08-20/goldman-says-exchanges-working-to-resolve-options-order-mishap.html
At least in the context of the whole world. The fact that the process has broken down, especially in America, is as a result of previous exclusion of large parts of the world's population from economic growth. Now that the factors that were preventing their sharing in the benefits have been removed, we are seeing a RAPID reduction in rates of absolute poverty around the world. See http://www.brookings.edu/research/interactives/2013/ending-extreme-poverty for the data. Which is not to claim that capitalism is perfect; the tendency for monopolistic behaviour is a major danger, but the potential for effective relief of poverty, is, according to world experience in the past 60 years, clearly in favour of capitalism.
Both might work.
You'll recall Obama has been saying that if he wasn't allowed to borrow more, the government would default - wouldn't be able to pay the interest due. We're borrowing to the interest on our borrowing.
When ordinary humans talk about "defaulting," they're almost always talking about not paying their mortgage or other significant loan because those are really the only contractual payments most people have. The government has a lot more - interest payments, sure, but also social security, salaries, contracted purchases. When the feds talk about "default" they don't mean "stop making interest payments," they mean "stop making payments." Interest payments are $400b, but total payments exceed total income by about $900b. Even if the feds were able to prioritize interest payments and preferentially destroy the value of the dollar, without continued borrowing, there is not enough income to pay daily operating expenses.
Maybe this is why the TEA party people think it wouldn't be so bad to "default." Maybe they imagine it simply means not making interest payments to China, the Social Security administration, and JP Morgan, and forget that it also means not paying for new Humvee tires or civil service salaries. The right solution to an imbalance in income and spending is almost never to stop meeting your obligations. If that imbalance has built up over 30 years, you better accept that it's going to take 30 more to correct.
In Latin America at least the reason that the poor are less poor is the rise of socialist and semi-socialist governments that have put tight controls on corporations and the very wealthy. The GNP of the countries of South America has almost tripled in the decade since the rise of 'socialismo' across the continent while the percentage of that rise going to the poor and middle class has (IIRC) quadrupled. You can't have a healthy economy without it being driven by the economic patterns of the lower and middle classes, and unbridled capitalism concentrates wealth and power in the hands of the capital holders.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
I once talked with a military psychiatrist tasked with creating a project that profiled military personnel so that they could be matched to positions that were psychologically suited for them.
The projected efficiency was greater than the private sector so it was scrapped as a military more efficient than capitalism would dominate the society and we would become a hyper efficient militarized society.
Perhaps the future is just such a society?
Our loss of civilian freedoms to the military politically industrial google complex is showing us the way, perhaps the machines will even tell us what to think and when.
There's a reference in K&R 2nd edition to Orwell's 1984. Turns out that in Winston's world the "C" language was the language of technocrats.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
You'll recall Obama has been saying that if he wasn't allowed to borrow more, the government would default - wouldn't be able to pay the interest due. We're borrowing to the interest on our borrowing. That's when you know you're fucked, when you're maxing out one credit card to pay the minimum payment on another card.
Except that it can be smart to borrow to pay off debt--if the rates on new new debt are lower than the rates of the old debt. That's why a lot of people get consolidation loans: you get one big loan at (say) 2-3% instead of trying to pay off your various credit cards which charge over 15%.
Current T-bill rates are about 0.03 percent (0.0003) over three months, and even a two-year bond pays a third of a percent (0.003):
http://www.bloomberg.com/markets/rates-bonds/government-bonds/us/
In finance terms, that's basically "free" money. I know people who would love to get their mortgages converted to rates that the US government can get for debt. Ten-year bonds are currently at 2.49%; most ten-year mortgages (in Canada where I am) are at least 4.29% and go up to 6.75%. Thirty-year bonds are 3.69%. The Treasury department would be stupid not to borrow as much money as it can at these rates to pay off debt at presumably higher rates.
There was actually a short period of time (2008/2009?) where the rates were actually negative for T-bills: people were basically paying the US government to hold their money because they didn't think it was safe to keep it anywhere else since all these financial houses were threatening collapse.
Now the question is, if you can can borrow money at 0.03%, what do you do with it? If you were smart, you'd invest it in something with a return on investment that's larger than that. Perhaps on infrastructure to increase trade (and thus tax revenue). Or education so that people get higher paying jobs in the future (which increases income tax revenue).
Not all deficity spending is bad: it's simply a matter of doing spending that will help improve the economy.
just another example
> but there is no mechanism to ensure that it DOES do things efficiently.
Sure there is. Inefficient companies lose in the market to more efficient competitors.
Unless of course the more inefficient company is much better in some other respect.
Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates.
Post WW2, Britain had a 220% debt to GDP ratio. Post WW2, the US had a 121% debt to GDP ratio. Both countries were fine.
Japan currently has an over-200% debt to GDP ratio, and it's fine (and was actually suffering from deflation for many years).
There is no point at which a "death spiral" occurs. There has only ever been one (non-peer reviewed) paper that made such a claim ("Growth in a Time of Debt" by Reinhart and Rogoff, 2010) and it has been quite thoroughly debunked. I challenge you to give any economic paper or economic model that predicts this--something which I know cannot be done, because there are no such things.
> Except that it can be smart to borrow to pay off debt--if the rates on new new debt are lower than the rates of the old debt.
Yeah, "borrowing" to PAY OFF a higher interest debt isn't _really_ borrowing, it REDUCES your total liability.
That's not what the government is doing. The government is borrowing (increasing debt) to pay INTEREST on debt.
The more they borrow, the more interest there is. The more interest there is, the more they borrow to pay the interest.
That's a vicious cycle spiraling to collapse.
$400 billion per year in interest payments is "essentially zero"?
It is when the interest is in your own currency.
You'll recall Obama has been saying that if he wasn't allowed to borrow more, the government would default - wouldn't be able to pay the interest due.
You're using the conservative definition of "default" - which consists solely of paying interest and principal due on the federal debt, regardless of other bills due. The Treasury Department has stated that it has no legal authority (or processes) to prioritize debt payments over bills due - so a default would've occurred either by paying some other bills instead of debt payments, or in the broader "universal default" sense of not meeting your financial obligations and commitments.
We're borrowing to the interest on our borrowing.
Not particularly true. Though, in a philosophical sense, money is fungible - so *any* borrowing and subsequent paying of interest could be construed as such. Likewise, if you have a mortgage and use your credit card to buy a TV - it could be said that you're borrowing money from your credit card to pay your mortgage (though I don't think most people would come to that conclusion).
We're in a trap the we're borrowing more and more in order to make the payments on existing debt, so the debt and the interest just keeps getting bigger and bigger until 100% of our tax money goes to pay interest, leaving no money for essential government services.
The US government never has to worry about running out of US dollars (absent political games, of course). Since they are the unlimited and sole supplier of US dollars, this isn't even in the same ballpark as a household budget (even though the terms are the same, which seems a bit unfortunate since it's often hard to separate from our "intuitive" understanding of finance).
The "debt" is really just an accounting game betting on future increases in productivity and a way to match the number of dollars in circulation to the actual capacity of the economy without causing unwanted inflation or deflation. When the government "borrows", they're removing short term money (dollars) in exchange for long term (bonds) - deferring the economy's need to produce to a later date. When they "repay", they're trading long term money (bonds) for short term money (dollars) - causing the economy to produce now instead of later. When they "tax", they're destroying short term money (dollars) - removing the economy's need to produce at all. When they "spend", they're causing the economy to produce now. Balancing these actions to produce "full employment" (maximum productivity in the economy) without "high inflation" (too much short term money) is what the Fed does, and the debt is how they do it.
In this framework, your argument about the debt is really that, in the future:
(a) investors will demand a lot more long term money (bonds) for their short term money (dollars) - possibly higher than we're willing to pay (aka, high interest rates)
(b) this glut of short term money will outstrip the ability of the economy to produce (aka, high inflation and low unemployment)
(c) we will not destroy the short term money (aka, raise taxes)
(d) government will not "force" the economy to produce for its services instead of the short term money (aka higher inflation)
This is all possible - and in fact, goes a long way to explaining the stagflation of the 1970s (except substitute a lowered capacity causing the short term money glut) and even Weimar-like hyperinflation scenarios (except substitute reparations for the investors and throw in some exchange rates). However, note that there's ample other avenues out of that scenario and that all indications (low interest rates, low inflation, low taxes, high unemployment) are that we are currently as far from that scenario as we have ever been.
Or, as a slight alternative, you *could* be arguing tha
> Maybe this is why the TEA party people think it wouldn't be so bad to "default." Maybe they imagine it simply means not making interest payments
> to China, the Social Security administration, and JP Morgan, and forget that it also means not paying for new Humvee tires or civil service salaries.
I think it's the opposite. Debt service on existing debt is mandatory spending - you pay your debts. That's very much part of the conservative ethic.
On the other hand, there's a lot of spending that is not justified, in their eyes. Some are downright stupid - bridges to nowhere. Some are nothing
but giveaways to donors - in just the last few months, THREE times executives have walked away from "green" companies financed by the
taxpayer. The latest (ECOtality) took $99.8 million of our money and walked away. Others are well intentioned programs being run so poorly that they need a complete overhaul. We're spending twice as much on food stamps as eight years ago, and 20% - 40% of that is fraud. Some recipients of
government aid claim they have a new baby every 2-3 months. How does that work?
From 1900 to 1940, government spending ranged between 6% - 20% of GDP, Today it's 39%.
Does the government really need to be almost half of the country?
By reducing fraud and utter waste by just 50%, we could pay off the debt, which would free up $415 BILLION dollars,
14% of the federal budget, for tax relief and worthwhile projects while at the same time eliminating the massive weight
we're saddling our children with.
* (Considering only _federal_ government spending, 1900-1940 it ranged 2% - 10% of GDP. Today it's 23%.)
There's another important difference between Greece and the US - Greece doesn't have its own currency. The US can inflate itself out of debt. Greece was as trapped as an individual who maxed out their credit cards.
Oolite: Elite-like game. For Mac, Linux and Windows
> There is no point at which a "death spiral" occurs.
Let's do some simple grade school math. The yield on government bonds, the cost of borrowing, has ranged from 2% to 15%.
To make the math easy while being generous to your claim, let's predict 5% as a long term average. Let's say the debt is 20X GDP.
5% of that is 1 X GDP - everything the country produces goes to pay interest on debt.
When the entire GDP is spent on interest payments, nothing is left for the people to eat, get shelter, education, etc.
Everybody's entire paycheck is sent to China and other creditors. That sounds like collapse to me.
So we've established via grade school math that collapse occurs by the time you reach 20 X GDP. If you can pass
4th grade math, you can see there IS a point where it collapses. The only debatable question is exactly where
that point is.
Those, like you, who advocate for big government spending say that the government needs to spend at least 10% - 30% of GDP
on services and entitlements. That leaves 70% for interest payments, so debt can't be more than 14 X GDP.
Hmm, people need to eat, and pay utilities, etc., so they'll need to keep half of their pay check. To leave people will
half of their paycheck, that's a max of 7 X GDP for government debt. This is still basic arithmetic and we've seen that
anything above 7 X GDP collapses. We could get a more specific number by doing advanced things like decimal division,
but I think the point is clear - there absolutely is a point of collapse, and with the debt doubling every 8 - 10 years, it
won't be long before we hit it.
> Post WW2, Britain had a 220% debt to GDP ratio. Post WW2, the US had a 121% debt to GDP ratio. Both countries were fine.
It's arguable whether the US is "fine", Obama says it's "definitely NOT fine", but you're confusing a one time event with a habit of chronic over spending. WWW2 was a one time event, like buying a house. What we have today is chronic, permanent overspending every single year - like a person
who makes $5,000 / month and spends $7,000 / month every single month.
> The US can inflate itself out of debt.
Only if defaulting is okay. Paying a 10€ trillion debt with 1€ trillion worth of bills is defaulting.
You'd also need chronic long term inflation of at least 14% to get there. That would be a problem.
Yes, definitely when your beloved government mama is blowing money left and right and someone points that out and tries to do something about it, the smart thing is to fire them.
You are the problem.
Er we were not fine. I'm not sure who you know that went through the post WW2 years in Britain but according to my grandparents it was pretty austere. A large chunk of debt was to the USA / Canada, and we only finally paid your WW2 loan off in 2006. If you don't mind your future generations being under the yoke of debt until 2075 or so then go ahead...
Phillip.
Property for sale in Nice, France
*pfft*...amateurs...I would have only lost them 50k per second...
http://erikkrogstad.com/how-to-create-a-disaster-recovery-plan/
So it's a not a glitch in the algorithm per se but a wrong integration by a technician which is basically lack of an automated procedure to upgrade and roll out important software. OK