Slashdot Mirror
Wall Street and the Mismanagement of Software
CowboyRobot writes "Last week, a bug in high-frequency trading software from Knight Capital Group resulted in erroneous trades costing almost a half-billion dollars. So, what went wrong and how can they, or any other software developer, prevent something similar from happening again? In hindsight, it's clear that the developers did not verify the code under enough conditions. But the real issue is how these high-frequency trades work in the first place. Robert Dewar at Dr. Dobb's suggests the financial industry needs to take a page from the avionics rulebook, which has very strict guidelines about what code can be implemented due to the high cost of failure in that field. 'High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week's incident. It doesn't come for free, but it certainly costs less than $440M.'"
First 100 trades in a day: free
Next 1000, taxed at 0.02%
Next 1000, taxed at 0.1%
And so on.
This would do wonders for the problem.
Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:
http://www.theregister.co.uk/2012/08/08/knight_capital_analysis/
http://www.nanex.net/aqck2/3525.html
http://www.zerohedge.com/news/what-happens-when-hft-algo-goes-totally-berserk-and-serves-knight-capital-bill
It wasn't HFT software, it was regular trading software. The developers created a build of the software that included a module that generated lots of silly trades as test data. The software was hooked up to the exchange as a live test to ensure it would talk to the exchange correctly. Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.
It wasn't actually a bug; everything worked perfectly. It was more of a configuration management problem.
The only relevance it has to HFT is that if the NYSE limited the rate of trades then a lot less money would have been lost.
Maybe it should serve as a warning to executives to not release buggy software. I know a lot of shops that push things out the door before they're fully baked.
In terms of the stock market, I don't see a problem. The long-term market wasn't affected, no value was created or destroyed, and those who played the game improperly lost out big time. Short term trades on the exchange are gambling. Anyone who tells you otherwise just wants your money. Don't forget, there's always a buyer *and* a seller. Just because Knight lost $450m doesn't mean other people didn't gain $450m.
Is it me or are we continuously using "profits" as a excuse for bad *anything* and pushing that idea to an extreme?
Blair K., Certified Master of the Scrum, responded: "Well, that doesn't sound like a very agile process to me! "Certified" and "compliant with a standard" sound pretty waterfallish. Why not just have a 15-minute standup and decide to launch the plane? At last the aerospace industry could deliver aircraft on time and under budget."
Customer wants their plane painted hot pink? We can totes do that, bros! Shouldn't take more than 24 hours to get to Home Depot and get a few cans of spraypaint. Delivered! And if bits of paint peel off at altitude and get sucked into the engine, gluing themselves to the turbine blades until catastrophic failure of an engine, well, we can just patch the paint recipe in the next sprint! Paint that's "hot pink" is part of this sprint. The user story about engines that don't fail is part of the next sprint.
The real problem with aircraft design is that all our little user stories are in a big clunky database. If we printed out the database's contents (by hand!) on little 3x5 index cards, then we'd be using the best practices of both Scrum and Kanban. Our planes would be so damn agile they'd have turning radii measured in inches.
When a senior engineer piped up that an aircraft with a turning radius measured in inches would kill everyone on board due to G-forces measured in the thousands of Gs, and would likely tear itself apart because the centripetal force far exceeds the tensile modulus of steel, titanium, carbon fiber, or anything else available, he was terminated because "switching from traditional tube-construction to blended-wing-body design made of unobtanium" was part of the next epic.
This isn't the equivalent of $500m of infrastructure burning down. The money was lost by some investors, but gained by other investors.
The really troubling thing (to my mind) is not so much the 'what does Wall St. produce?' question(which, as you note, is ostensibly 'capital allocation'; but the 'how efficiently do they actually produce it?' question.
In a non-pathological market situation, you would hope to see Wall St.'s share of the economy as a whole be static or declining(as newer technology makes allocating capital easier and less expensive) and the demand for 'capital allocation' exist only so far as other business sectors find that more efficient capital allocation makes them more efficient and productive(in the same way that you would want to see any other support function of a business kept in line with the business overall. You wouldn't want your IT group consuming a greater percentage of your total economic output every year). Trouble is, that isn't what the numbers reflect.
Instead of acting as other suppliers do, and having their health and size depend on the success of the customers, the financial services sector has managed to capture a steadily increasing share of the value relative to other sectors. Absolute growth would be one thing, if the economy as a whole is growing; but relative growth, in terms of percentage of total output captured, suggests a substantial increase in the market(and regulatory) power of financial services without necessarily any increase in the value of their product to their customers. That is bad.
The spectacular "losses" are the path by which the money leaves investors, and is syphoned off to pay the disproportionate brokerage fees.
As a point of reference see gold - the value of gold traded between speculators each day is 1,000 times the value actually sold into industry/jewelery/etc - so a 0.25% brokerage fee on both buyer an seller is worth 5* the value of the gold that enters or leaves the market each day. Obviously, the additional margin actually paid by real life users for the benefits of a liquid market is a tiny fraction of the value of the gold - sometimes there has to be a "software problem" or "rogue trader" or, to use the colloquial term "scapegoat", to provide the money distributed on brokerage fees.
I am not suggesting any one person is responsible for this - the banks have colluded to look the other way while this system has developed in an ad-hoc manner. In all probability, many of the so called "masters of the universe" are too stupid to understand what is going on, as it requires an understanding of the laws of maths, which they generally don't have. The few "whizz-kids" who could understand are "highly motivated" to look the other way.
Sent from my ASR33 using ASCII
America once had a great capitalism. Now we have the system where no matter what risk the rich insiders take, all the profits are theirs and all the losses are ours. A system where the ruling elites are protected from the consequences of their actions, where they can rig the game so that they win no matter what, is how societal collapse begins. Jared Diamond's book "Collapse" discusses specific case studies showing how it collapses. Greenland colonization from Iceland, Pueblo Indians, Easter Island were what he discusses in great detail.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
From the article I read recently, they had just put a new system into place, but they weren't even watching what it did, and had no "kill switch" in place to stop it immediately if it started acting up. Crazy. Maybe the programmers screwed up, but whoever was supervising the installation and running of the thing was a moron.
which is totally what she said
This increase in the relative growth of the financial sector was one of the predictions in Karl Marx's Das Kapital: He saw bond markets and stock markets as the natural and predictable outgrowth of the role of a capitalist, which in his view was somebody who made their living not by producing stuff but by buying the means of production and making other people produce stuff. Bonds in particular simply abstract the concept completely away from any actual work: The capitalist now doesn't even do the buying and managing himself, but buys bonds allowing somebody else to do that work and demands a portion of the proceeds of the firm in return. As the capitalist class gains more and more wealth, the trade in bonds and other financial instruments goes up as a percentage of the economy, while the industrial and agricultural production becomes less important.
(And no, I'm not arguing that workers of the world should unite and revolt, just that Marx was a serious economist who made some good points about how capitalism works.)
I am officially gone from
It's NOT the same as airplane full of people being destroyed or an ambulance failing to show up. In fact, all the money the summary suggests pouring into perfecting HFT software would be a waste and a loss to the economy overall. The real question is how can we fix the incentives to get those HFT developers back working on avionics or 9/11 call centers or something else with real value?
But the real problem issue here isn't even the buggy software IMO. It was the way the software was put into place, not monitored, and nobody was ready to just shut it off if it started going haywire. According to the article I read, Knight hadn't even noticed a problem themselves - it was pointed out to them by the stock exchange that they were doing a very high amount of trades compared to usual, and it still took them half an hour or more to shut everything down. There should have literally been a big red STOP button in place to shut things down if they went wrong.
which is totally what she said