Slashdot Mirror
Knight Capital Fined $12M For a Software Bug That Cost $460M
Mark Gibbs writes "Knight Capital monumentally fouled up a software update. According to the SEC, 'Knight did not have supervisory procedures to guide its relevant personnel when significant issues developed.' In other words, not only was Knight's code management inadequate but their human management processes were just as bad. The fine for what could have been a biblical financial disaster? A measly $12 million."
The cost to them was $472 M. I *think* that will discourage them.
What they did was criminal negligence, plain and simple. And they did it out of greed. As long as mismanagement this severe has no personal consequences for the perpetrators, nothing will change.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
They were FINED 12M, and they LOST 460M discovering the bug. This cost them a total of 476M.
I am not understanding the outrage. Why should the SEC care if Knight Capital wanted to lose a big pile of money.
Can someone tell me why these financial institutions are never forced to compensate the *individuals* that suffer from these events?
For instance in the mortgage fraud scandal they were allowed to settle fraudulent foreclosures for pennies on the dollar. Why are these companies never required to make the people they hurt whole again? Individuals that paid thousands of dollars simply got a small payment while banks just had to deal with "the cost of doing business."
I think I know the answer (lobbying/congresscritters in their pockets) but I think it's one of the most scandalous aspects of the financial mess of 2008.
That $460 million came out of Knight Capital's pockets too...and is far more effective than any fine the SEC could levy. Why should the SEC pile on, aside from the populist outrage that goes along with people handling billions of dollars?
Disinfect the GNU General Public Virus!
This isn't a slap on the wrist. This is a pat on the back for inflicting harm with egregious negligence.
Therefore this was probably engineered as an assault.
Businessweek weighs in.
The dangers of knowledge trigger emotional distress in human beings.
Look at the bottom of the Wikipedia page.
Some as far back as 2002.
I think a bit more than a fine is due. Why isn't FINRA on these folks?!
As a proprietary trading firm, they were working entirely with their own money. They had no external investors or whatnot (like hedge funds do). So, they made a mistake and they paid for it dearly. It's not clear to me that they should have paid any fine.
The article's whole argument seems to be made by comparing the size of the trading loss to the size of the fine, but no logical reasoning is given for why the one should have any relation to the other.
TFA sucks.
I'm not joking when I say that procure number one when money is flying out of your servers is to Shut Them Down instantly. I would have pulled the cables out so fast the CPU might have been yanked out with the network cable. Or a good old shutdown -h now !!!!! (The exclamation marks speed up the shutdown)
And I wouldn't have done this one server at a time it would have been all the servers at the same time. I suspect they would lose money by not having the servers up but not at the firehose rate that they were losing money as they were.
The worst part is that the admins were probably following some procedure in their book and were refusing to just pull the plug in some vain attempt for 99.9 percent up time or other admin related metric instead of the clear "Don't Lose $48 Million a minute!!!!" metric. So probably another clear case of IT's priorities getting way out of sync with the company's actual priorities.
TFA states that the $460 million was lost by Knight Capital themselves. If they'd been fined $12M for stealing $460M, I'd be as outraged as the article author, but from where I'm standing it looks like the SEC turned a $460M loss into a $472M loss.
Sure, they're idiots, they've punished themselves amply!
foo mane padme hum
What does HFT have to do with this?
Ezekiel 23:20
This had absolutely jack to do with bad code, that wasn't the problem. The problem was a failure to adhere to best practices that would have prevented the bad code from ever seeing production to begin with. The lack of a process for the distribution of code to production made a failure for bad code inevitable.
This was sheer incompetence of the highest magnitude and should have been readily caught in the lab. This is what happens when cowboys run the show and ITIL is considered a four letter word. Take your younger staff, the wannabe cowboys and make them read this report. Let them learn at others incompetence. As for getting your management to read this, that's an entirely different story.
I am a bit numbed by the number of failures of software systems at big companies (& governments) who should know better.
If you are designing critical systems, there has to be an incredibly detailed master system describing fallbacks, trip points and fail safe conditions, let alone a gross shutdown (seen multiple times recently.) What do these failures in both checking and security and logic mean for trusting large institutions and government?
The question: What overview system of principles of software design are going to be needed to properly organize a major software program from day one to prevent, at least, the obvious failure modes? There is something inherently wrong by design when hundreds to thousands of security breaches occur in the US on public websites and databases each year.
Furthermore, why would millionaires trust their money to a company that is getting pilloried in the press for fundamental failures of management, not to mention development practices?
Cutting corners on developing the software that handles your money: penny wise and pound foolish.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
It's a bad thing...
Just make sure they suffer all the pain caused by the $450 Million loss
In other words: don't allow them to pass any of this loss on to their customers by drawing funds from their accounts.
Install Windows on it. Total annihilation. No survivors.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
Most all Wall St firm's systems are bloody awful. There are many reasons for this. First, the true business is sales/brokerage so the engineering side, though it is a strategic asset, is often neglected. This includes putting clueless business side people in charge of IT system. Second, the boom and bust cycles of tech investment are a bad way of building tech systems. It's like not watering your garden all summer except for one day when you use a high-pressure fire hose on it. Third, as part of the boom/bust cost cutting they have no employee longevity in tech so no one understands how the mind-bogglingly complex and obscure layers of technology work. Fourth, and more recently for cost cutting, they've dispersed their dev teams around the globe so communication and teamwork are seriously compromised. Fifth, when there is a boom they try to build their systems so quickly that they take all sorts of dangerous engineering short cuts. All this adds up to engineering disaster.
Given the cronyism masquerading as capitalism in USA, you should be glad this behavior is considered bad enough to be punished. Be glad they did not get the contract to "improve" healthcare.gov
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
It's what Knight Capital was in the business of doing when they gave away $460 million to other players.
Apart from the exact victims here, there is a huge chilling effect.
http://www.forbes.com/sites/christophersteiner/2012/08/02/knight-capitals-algorithmic-fiasco-wont-be-the-last-of-its-kind/
Knight Capital is a trading company so the money they used belonged to other people.
FTA: "Such an episode would take down not only the traders, but likely the brokerage house that gives them access to electronic markets and perhaps even other clients of that brokerage. It could completely subvert the little amount of trust the public still has in our stock markets."
The dangers of knowledge trigger emotional distress in human beings.
I'm going to quote a bumpersticker that I think is fitting here: "I'll beleive corporations are people when Texas executes one"
My thought exactly. What seems to be the problem? The money were not lost. Just redistributed.
That's like saying mugging is not a crime. Your money is just being redistributed to the needy.
Who cares? Nobody died. This isn't the Tacoma friggin' Narrows Bridge, or even Therac-25. Hell, the mistake was cheaper than the Mars Climate Impactor. Total impact in the real world outside of Wall Street was zero, unless you were directly related to Knight Capital in some way.
Doh! I should have stripped off the parameters:
http://www.nytimes.com/2011/03/26/business/26nocera.html [In Prison for Taking a 'Liar Loan' - Joe Nocera - NY Times; may require registration, or try reaching it through a search engine.]
and there is a follow-up:
http://www.nytimes.com/2012/06/02/opinion/nocera-the-mortgage-fraud-fraud.html
The National Council of Examiners for Engineering and Surveying sets the standards for licensing engineers.
Their certification tests include:
Architectural
Chemical
Civil: Structural
Civil: Transportation
Electrical and Computer: Computer Engineering
Electrical and Computer: Electrical and Electronics
Nuclear
Petroleum
SOFTWARE
Structural
Tell you what, if 12 Mil is measly to you, then I'm sure you wouldn't even notice if half a mil went missing. And I sure could use half a million dollars . . .
In a bit of shameless internet panhandling, I accept Litecoin Donations at Lbd2oH9QsthD1GfuUXPyka12YxvWJYnBVf
But this wasn't Knight mugging someone, this was Knight giving up the money by their own choice.
The people who were hurt were Knight's owners, which is an incentive for them to do a better job of oversight in the future.
I refer to the trading losses, not the fine.
It's not 100% certain what happened when but my understanding was that the original fire continued to burn and, eventually, melted those high pressure gas lines. Only at that point was total destruction of the rig inevitable.
Had the original fire burned itself out due to lack of fuel (and I was wrong, it was oil that continued to be pumped into the initial fire that kept it burning, not gas) then the high pressure gas lines would not have melted and there would not have been the second explosion that destroyed the rig.
God said, "div D = rho, div B = 0, curl E = -@B/@t, curl H = J + @D/@t," and there was light.
Seriously, there *was* no bad code. What happened was that one of their systems didn't get upgraded and they re-used a variable that was previously used to make systems to keep buying until they were told to stop by a master system. When the server that didn't get upgraded got that variable switched, it just started buying and nobody told it to stop. They knew something was wrong for 45 minutes and kept on letting it buy stuff, didn't just switch it off because there was nobody authoritative that could make that decision available. This was not caused by the code at all, purely procedure and bad organizational design.
I was promised a flying car. Where is my flying car?
The SEC (Slashdot Effeciency Committee) have released their findings and conclude that:
Slashdot pushed their new code to all but one app server. That one app server reposted the same Knight story as yesterday. Slashdot has been fined 12 karma.
Texas will not only terminate a corporation, they charge a $40 fee for the paperwork processing. Such inhumane treatment. Next they will start charging the inmate's estate for the injections.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
The Obamacare website fiasco should be featured on the next episode of Modern Marvel's "Engineering Disasters" on the History Channel.
These are all elementary software mistakes:
1) You never reuse a flag for a working code, because it makes it impossible to revert back to older deployment.
2) You always double check deployments to make sure it actually succeeded.
I don't get what's the purpose of these "remedial sanctions," especially coming from the SEC.
If the SEC is doing this to deter Knight's management from being un-diligent with Knight's owners' assets, then it ought be a fine or prison time for the people who were responsible, not the company (the owners, who were also the victims). That's like punishing someone for the crime of being raped, while talking about how irresponsible the rapist was.
If the SEC is doing this to deter Knight's owners (who worked through their agents, the management) from making poor decisions that will cost them money, then (like everyone else is posting) it seems like the loss itself, is punishment enough. That's like punishing someone for the crime of suicide.
"Believe me!" -- Donald Trump
After all, they needed more money to continue funding KITT and that damn trailer that drives it around.
Michael Knight doesn't come cheap, either.
Contrary to popular opinion, sanitation engineering is a real thing, and is not the same as throwing bags of trash into a garbage truck. Sanitation engineers are civil engineers who design landfills (and/or sewage treatment plants, but those are more often called "hydraulic engineers" these days). They are most likely required to be licensed.
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
In theory, any managers or executives who cost their company half a billion dollars would never be hired by anyone else, since they've clearly demonstrated that they aren't competent at their jobs.
In theory.
Cool! I have a bunch of throwing rocks. I see a big bonus from the Glass Doctor in my future.
That's like saying mugging is not a crime.
No it isn't. It is like saying that going up to people in a dark alley and stuffing money into their pockets is not a crime.
[HFT is] what Knight Capital was in the business of doing when they gave away $460 million to other players.
This had nothing to do with HFT. "Algorithmic Trading" and "High Frequency Trading" are two different things. Before anyone says "but they both use a computer": Algorithmic Trading predates the invention of the digital computer. In the early 1900's it was common to use human "computers" (they were actually called that) to perform calculations by hand according to specific rules, and then forward the buy/sell orders to traders in the pits.
One is a subset of the other, and Knight Capital does the subset that is high frequency - plus other things of course.
Hence why GETCO, another large HFT player, swallowed them up. And why they describe themselves with terms like:
"""Knight Capital Group, Inc. (NYSE Euronext: KCG) a leading liquidity provider in global markets specializing in high frequency trading""" - http://www.knight.com/careers/summerInternship.asp
The question: What overview system of principles of software design are going to be needed to properly organize a major software program from day one to prevent, at least, the obvious failure modes?
Going to be needed? Those principles have been needed for 50 years, and they've been known for nearly that long. The Mythical Man-Month by Frederick Brooks, Jr. discusses a great many of those principles, lessons learned from the design and implementation of OS/360 by IBM. It was first published in 1975. There are others.
The solutions cost money and take time. The losses for failing to spend money and time are suffered by other people, not the companies that own the failing systems. Therefore there is no incentive to solve the problem beyond the barest of band-aide patches. Nothing will be done as long as "government regulation" is a bad word, and it will remain a bad word for as long as lobbyists are paid to maintain it. And lobbyists will be paid to maintain it because there's plenty of money available to spend on lobbyists. Engineers? Nah. They're a cost center, and must be eliminated at every turn.
Sanitation engineers may also have exciting newer duties: designing recycling centers, and reversing effects of contamination and pollution from soil and water.