Slashdot Mirror
Knight Capital Fined $12M For a Software Bug That Cost $460M
Mark Gibbs writes "Knight Capital monumentally fouled up a software update. According to the SEC, 'Knight did not have supervisory procedures to guide its relevant personnel when significant issues developed.' In other words, not only was Knight's code management inadequate but their human management processes were just as bad. The fine for what could have been a biblical financial disaster? A measly $12 million."
The cost to them was $472 M. I *think* that will discourage them.
A fine?
If you make bad decisions (as a company or as a person), you kinda have to live with 'em. That should be as far as it goes.
The result of this should be 'you fucked up with all the money, now go away'.
It cost THEM $460M. That money was not lost, HFT is zero-sum. The market gained 460M, they should receive a bonus from their competitors not a fine.
What they did was criminal negligence, plain and simple. And they did it out of greed. As long as mismanagement this severe has no personal consequences for the perpetrators, nothing will change.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
They were FINED 12M, and they LOST 460M discovering the bug. This cost them a total of 476M.
I am not understanding the outrage. Why should the SEC care if Knight Capital wanted to lose a big pile of money.
Can someone tell me why these financial institutions are never forced to compensate the *individuals* that suffer from these events?
For instance in the mortgage fraud scandal they were allowed to settle fraudulent foreclosures for pennies on the dollar. Why are these companies never required to make the people they hurt whole again? Individuals that paid thousands of dollars simply got a small payment while banks just had to deal with "the cost of doing business."
I think I know the answer (lobbying/congresscritters in their pockets) but I think it's one of the most scandalous aspects of the financial mess of 2008.
That $460 million came out of Knight Capital's pockets too...and is far more effective than any fine the SEC could levy. Why should the SEC pile on, aside from the populist outrage that goes along with people handling billions of dollars?
Disinfect the GNU General Public Virus!
This isn't a slap on the wrist. This is a pat on the back for inflicting harm with egregious negligence.
Therefore this was probably engineered as an assault.
Businessweek weighs in.
The dangers of knowledge trigger emotional distress in human beings.
Look at the bottom of the Wikipedia page.
Some as far back as 2002.
I think a bit more than a fine is due. Why isn't FINRA on these folks?!
What they did was criminal negligence, plain and simple.
In what way were the interest of other people or society's at large harmed ? At most you could say they have an obligation to the KCG shareholders, but they also have the obligation to be greedy. They've put in the cheapest system they could build, and it failed losing allot of money. Sounds like regular management to me, in no way criminal. Nowadays Microsoft does it more often than not.
As a proprietary trading firm, they were working entirely with their own money. They had no external investors or whatnot (like hedge funds do). So, they made a mistake and they paid for it dearly. It's not clear to me that they should have paid any fine.
The article's whole argument seems to be made by comparing the size of the trading loss to the size of the fine, but no logical reasoning is given for why the one should have any relation to the other.
TFA sucks.
I'm not joking when I say that procure number one when money is flying out of your servers is to Shut Them Down instantly. I would have pulled the cables out so fast the CPU might have been yanked out with the network cable. Or a good old shutdown -h now !!!!! (The exclamation marks speed up the shutdown)
And I wouldn't have done this one server at a time it would have been all the servers at the same time. I suspect they would lose money by not having the servers up but not at the firehose rate that they were losing money as they were.
The worst part is that the admins were probably following some procedure in their book and were refusing to just pull the plug in some vain attempt for 99.9 percent up time or other admin related metric instead of the clear "Don't Lose $48 Million a minute!!!!" metric. So probably another clear case of IT's priorities getting way out of sync with the company's actual priorities.
TFA states that the $460 million was lost by Knight Capital themselves. If they'd been fined $12M for stealing $460M, I'd be as outraged as the article author, but from where I'm standing it looks like the SEC turned a $460M loss into a $472M loss.
Sure, they're idiots, they've punished themselves amply!
foo mane padme hum
The fine for what could have been a biblical financial disaster?
Perhaps it shouldn't be possible for single trader or trading company to create a biblical financial disaster. Perhaps this is indication of larger problems in the marketplace.
If you're interested in this sort of thing, there's a great book about the rise and fall of "Long Term Capital Management", a hedge fund that just about wiped out the world financial markets - for REAL (none of this biblical hyperbole!) http://en.wikipedia.org/wiki/Long-Term_Capital_Management.
This had absolutely jack to do with bad code, that wasn't the problem. The problem was a failure to adhere to best practices that would have prevented the bad code from ever seeing production to begin with. The lack of a process for the distribution of code to production made a failure for bad code inevitable.
This was sheer incompetence of the highest magnitude and should have been readily caught in the lab. This is what happens when cowboys run the show and ITIL is considered a four letter word. Take your younger staff, the wannabe cowboys and make them read this report. Let them learn at others incompetence. As for getting your management to read this, that's an entirely different story.
I am a bit numbed by the number of failures of software systems at big companies (& governments) who should know better.
If you are designing critical systems, there has to be an incredibly detailed master system describing fallbacks, trip points and fail safe conditions, let alone a gross shutdown (seen multiple times recently.) What do these failures in both checking and security and logic mean for trusting large institutions and government?
The question: What overview system of principles of software design are going to be needed to properly organize a major software program from day one to prevent, at least, the obvious failure modes? There is something inherently wrong by design when hundreds to thousands of security breaches occur in the US on public websites and databases each year.
It's the way the story was reported.
When a firm loses money, it's their customers' money. And quite a few of those are small investors ()212 of them.). Theoretically, this fine would come out of the pockets of the actual firm's equity - i.e. the owners' pockets.
But 12 million for people like this is a month's rent for their mistress' Park Avenue penthouse.
Furthermore, why would millionaires trust their money to a company that is getting pilloried in the press for fundamental failures of management, not to mention development practices?
Cutting corners on developing the software that handles your money: penny wise and pound foolish.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
It's a bad thing...
Just make sure they suffer all the pain caused by the $450 Million loss
In other words: don't allow them to pass any of this loss on to their customers by drawing funds from their accounts.
Install Windows on it. Total annihilation. No survivors.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
Most all Wall St firm's systems are bloody awful. There are many reasons for this. First, the true business is sales/brokerage so the engineering side, though it is a strategic asset, is often neglected. This includes putting clueless business side people in charge of IT system. Second, the boom and bust cycles of tech investment are a bad way of building tech systems. It's like not watering your garden all summer except for one day when you use a high-pressure fire hose on it. Third, as part of the boom/bust cost cutting they have no employee longevity in tech so no one understands how the mind-bogglingly complex and obscure layers of technology work. Fourth, and more recently for cost cutting, they've dispersed their dev teams around the globe so communication and teamwork are seriously compromised. Fifth, when there is a boom they try to build their systems so quickly that they take all sorts of dangerous engineering short cuts. All this adds up to engineering disaster.
Given the cronyism masquerading as capitalism in USA, you should be glad this behavior is considered bad enough to be punished. Be glad they did not get the contract to "improve" healthcare.gov
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
When will the other traders get their bad trades annulled and suffer only a minor slap? Are they too big to fail or something? What the hell is going on?
Why is it that when a company like Knight screws up and costs $460M they are penalized but when the government spends more on a useless site (heathcare.gov) nothing happens? Considering that the SEC fined them for not "having adequate safeguards in place" this should apply to other debacles. I guess we should have lower expectations for our elected officials and we can all be friends and sing kumbaya.
Apart from the exact victims here, there is a huge chilling effect.
http://www.forbes.com/sites/christophersteiner/2012/08/02/knight-capitals-algorithmic-fiasco-wont-be-the-last-of-its-kind/
Knight Capital is a trading company so the money they used belonged to other people.
FTA: "Such an episode would take down not only the traders, but likely the brokerage house that gives them access to electronic markets and perhaps even other clients of that brokerage. It could completely subvert the little amount of trust the public still has in our stock markets."
The dangers of knowledge trigger emotional distress in human beings.
I'm going to quote a bumpersticker that I think is fitting here: "I'll beleive corporations are people when Texas executes one"
No, you can't fine them. You have to reward them: their paychecks are a hundredfold that of hoi polloi because of the large responsibility they bear. And no responsibility is greater than that of fscking up cool half a billion dollars in less than an hour.
That kind of responsibility comes at a price the shareholders should be happy to pay. Would you want to be responsible for that kind of loss? Certainly not. But those guys are pissing responsibility through every hole in their Armani suits.
Who cares? Nobody died. This isn't the Tacoma friggin' Narrows Bridge, or even Therac-25. Hell, the mistake was cheaper than the Mars Climate Impactor. Total impact in the real world outside of Wall Street was zero, unless you were directly related to Knight Capital in some way.
Doh! I should have stripped off the parameters:
http://www.nytimes.com/2011/03/26/business/26nocera.html [In Prison for Taking a 'Liar Loan' - Joe Nocera - NY Times; may require registration, or try reaching it through a search engine.]
and there is a follow-up:
http://www.nytimes.com/2012/06/02/opinion/nocera-the-mortgage-fraud-fraud.html
millions of people are losing their insurance and being forced to buy from the exchange which is non-existent?
Are we going to see articles about the lack of oversight on Slashdot and populists with torches demanding the White House do something about it?
I think not.
"Criminal negligence"? Care to explain that or did you just rely on the fact that other angry people would mod you up for it?
"Out of greed"? Of course it was out of greed. Everything a for-profit business does is out of greed (self-interest). What exactly is the problem? Do you want to outlaw profit-making so that the only organization left doing it is government?
Moderators, please stop voting up these worthless "anger" posts. There is absolutely nothing of value in them, unless you see some kind of value in anger. Hell, this one doesn't even offer the slightest explanation or rationale for the anger. Nothing to learn, no knowledge to gain, no insight, nothing but fist-pumping and chest-beating. I guess that makes some people feel good, but for the grownups among us, it's useless noise.
The National Council of Examiners for Engineering and Surveying sets the standards for licensing engineers.
Their certification tests include:
Architectural
Chemical
Civil: Structural
Civil: Transportation
Electrical and Computer: Computer Engineering
Electrical and Computer: Electrical and Electronics
Nuclear
Petroleum
SOFTWARE
Structural
"cheaper than the Mars Climate" (I assume you meant "Orbiter")
Are you seriously comparing a website and a collection of databases to a spacecraft with rocket science? Even with hundreds of thousands of parts (on the ground and in space) and probably obscene amounts of code a single issue (spacecraft using metric, Ground controllers using English units) was all that went wrong. By all reports the Healthcare website is nothing but issues, bad code, poor implementation, incomplete databases, the list goes on.
As a proprietary trading firm, they were working entirely with their own money.
High frequency traders are not working with their own money. They are siphoning money off from the actions of actually serious buyers and sellers. Basically one expects them to act as medicinal leeches keeping the blood flowing. Not like sharks in a feeding frenzy. Even though most of the blood in the water afterwards will be from the sharks themselves, it tends to frighten the tourists.
Tell you what, if 12 Mil is measly to you, then I'm sure you wouldn't even notice if half a mil went missing. And I sure could use half a million dollars . . .
In a bit of shameless internet panhandling, I accept Litecoin Donations at Lbd2oH9QsthD1GfuUXPyka12YxvWJYnBVf
Piper Alpha was a cascade of multiple design and procedural failures. However, while it's true the gas feeds from other rigs were not shut off, the lines were so highly pressurized that it would have taken hours for them to bleed off anyway, and the entire disaster unfolded in less than half an hour. So that aspect had very little to do with the final death toll.
Seriously, there *was* no bad code. What happened was that one of their systems didn't get upgraded and they re-used a variable that was previously used to make systems to keep buying until they were told to stop by a master system. When the server that didn't get upgraded got that variable switched, it just started buying and nobody told it to stop. They knew something was wrong for 45 minutes and kept on letting it buy stuff, didn't just switch it off because there was nobody authoritative that could make that decision available. This was not caused by the code at all, purely procedure and bad organizational design.
I was promised a flying car. Where is my flying car?
The SEC (Slashdot Effeciency Committee) have released their findings and conclude that:
Slashdot pushed their new code to all but one app server. That one app server reposted the same Knight story as yesterday. Slashdot has been fined 12 karma.
Texas will not only terminate a corporation, they charge a $40 fee for the paperwork processing. Such inhumane treatment. Next they will start charging the inmate's estate for the injections.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
These are all elementary software mistakes:
1) You never reuse a flag for a working code, because it makes it impossible to revert back to older deployment.
2) You always double check deployments to make sure it actually succeeded.
I don't get what's the purpose of these "remedial sanctions," especially coming from the SEC.
If the SEC is doing this to deter Knight's management from being un-diligent with Knight's owners' assets, then it ought be a fine or prison time for the people who were responsible, not the company (the owners, who were also the victims). That's like punishing someone for the crime of being raped, while talking about how irresponsible the rapist was.
If the SEC is doing this to deter Knight's owners (who worked through their agents, the management) from making poor decisions that will cost them money, then (like everyone else is posting) it seems like the loss itself, is punishment enough. That's like punishing someone for the crime of suicide.
"Believe me!" -- Donald Trump
No, NASA meant "orbiter", and ended up with "impactor" when Conway's Law kicked in.
And who's talking about the health care website? You seem to have a one-track mind. We're talking about a relatively small error in Knight Capital's algo trading setup that caused them to buy high and sell low. Lots of moving parts in a chaotic system when you're dealing with the stock market, and a slipup can cause money to burn shockingly fast. But it's not like anyone got killed because of it; most people wouldn't even notice if it wasn't for the media highlighting a little schadenfreude for the rest of us.
Typical attitude of the software "engineer"...oops! we f*cked up, we can fix this in the next service pack. Send the widow a copy of the fixed version. We can name a variable after the folks savings we threw away.
Quote: Who cares? Nobody died. This isn't the Tacoma friggin' Narrows Bridge, or even Therac-25. Hell, the mistake was cheaper than the Mars Climate Impactor. Total impact in the real world outside of Wall Street was zero, unless you were directly related to Knight Capital in some way.
After all, they needed more money to continue funding KITT and that damn trailer that drives it around.
Michael Knight doesn't come cheap, either.
Contrary to popular opinion, sanitation engineering is a real thing, and is not the same as throwing bags of trash into a garbage truck. Sanitation engineers are civil engineers who design landfills (and/or sewage treatment plants, but those are more often called "hydraulic engineers" these days). They are most likely required to be licensed.
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
In theory, any managers or executives who cost their company half a billion dollars would never be hired by anyone else, since they've clearly demonstrated that they aren't competent at their jobs.
In theory.
because then they could have called a take-backsies:
http://www.bloomberg.com/news/2013-08-20/goldman-says-exchanges-working-to-resolve-options-order-mishap.html
hell, the SEC would probably have GIVEN them money as an apology for the inconvenience
The question: What overview system of principles of software design are going to be needed to properly organize a major software program from day one to prevent, at least, the obvious failure modes?
Going to be needed? Those principles have been needed for 50 years, and they've been known for nearly that long. The Mythical Man-Month by Frederick Brooks, Jr. discusses a great many of those principles, lessons learned from the design and implementation of OS/360 by IBM. It was first published in 1975. There are others.
The solutions cost money and take time. The losses for failing to spend money and time are suffered by other people, not the companies that own the failing systems. Therefore there is no incentive to solve the problem beyond the barest of band-aide patches. Nothing will be done as long as "government regulation" is a bad word, and it will remain a bad word for as long as lobbyists are paid to maintain it. And lobbyists will be paid to maintain it because there's plenty of money available to spend on lobbyists. Engineers? Nah. They're a cost center, and must be eliminated at every turn.
The incentives in terms of cost versus benefit do not always favour doing a thorough job when a half-ass job gets done quicker as well as cheaper, and this has tangible benefits when pursuing the all-important quarterly-earnings-targets. Futher, a lot of business opportunities are time-sensitive to the degree that trying too carefully and diligently will lose your window.
Of course there's also something to be said for the fact that rich organizations generally have premium access to legal services and politicians to cover their problems if they do fuck up badly....
Sanitation engineers may also have exciting newer duties: designing recycling centers, and reversing effects of contamination and pollution from soil and water.