Slashdot Mirror
Wall Street and the Mismanagement of Software
CowboyRobot writes "Last week, a bug in high-frequency trading software from Knight Capital Group resulted in erroneous trades costing almost a half-billion dollars. So, what went wrong and how can they, or any other software developer, prevent something similar from happening again? In hindsight, it's clear that the developers did not verify the code under enough conditions. But the real issue is how these high-frequency trades work in the first place. Robert Dewar at Dr. Dobb's suggests the financial industry needs to take a page from the avionics rulebook, which has very strict guidelines about what code can be implemented due to the high cost of failure in that field. 'High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week's incident. It doesn't come for free, but it certainly costs less than $440M.'"
Back in the late 90s when I was system admin for a trading company, they recruited me from a place that did 911 computer aided dispatch software. My shop, at least, recognized that some of the same reliability issues were at stake, so some people get it.
First 100 trades in a day: free
Next 1000, taxed at 0.02%
Next 1000, taxed at 0.1%
And so on.
This would do wonders for the problem.
Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:
http://www.theregister.co.uk/2012/08/08/knight_capital_analysis/
http://www.nanex.net/aqck2/3525.html
http://www.zerohedge.com/news/what-happens-when-hft-algo-goes-totally-berserk-and-serves-knight-capital-bill
So, sue the developer for the cost he caused.
That should teach him a lesson. ;)
It wasn't HFT software, it was regular trading software. The developers created a build of the software that included a module that generated lots of silly trades as test data. The software was hooked up to the exchange as a live test to ensure it would talk to the exchange correctly. Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.
It wasn't actually a bug; everything worked perfectly. It was more of a configuration management problem.
The only relevance it has to HFT is that if the NYSE limited the rate of trades then a lot less money would have been lost.
It's already such a waste that so much talent is getting thrown at problems that seek to make money while producing absolutely nothing. HFT is cleverly sanding in the middle of a river in an eddy and dipping your hand in to tap power without getting pushed downstream. What does Wall Street actually produce? What is their product? Why should we care that they periodically lose their minds and shirts? If anything HFT should be taxed into oblivion so that excellent minds aren't recruited to deliver nothing of social value.
"There are some people that if they don't know, you can't tell them." ~ Louis Armstrong
Want to stem the flood of HFT software into the Wall Street environment? Pass a law that requires any such software to be written in Ada. Think that one through...
All the world's an analog stage, and digital circuits play only bit parts.
Maybe it should serve as a warning to executives to not release buggy software. I know a lot of shops that push things out the door before they're fully baked.
In terms of the stock market, I don't see a problem. The long-term market wasn't affected, no value was created or destroyed, and those who played the game improperly lost out big time. Short term trades on the exchange are gambling. Anyone who tells you otherwise just wants your money. Don't forget, there's always a buyer *and* a seller. Just because Knight lost $450m doesn't mean other people didn't gain $450m.
Is it me or are we continuously using "profits" as a excuse for bad *anything* and pushing that idea to an extreme?
The SEC usually gives them a mulligan when a software boo-boo costs a big institution a lot of money. I never get a do-over when I make a bad trade though...
Blair K., Certified Master of the Scrum, responded: "Well, that doesn't sound like a very agile process to me! "Certified" and "compliant with a standard" sound pretty waterfallish. Why not just have a 15-minute standup and decide to launch the plane? At last the aerospace industry could deliver aircraft on time and under budget."
Customer wants their plane painted hot pink? We can totes do that, bros! Shouldn't take more than 24 hours to get to Home Depot and get a few cans of spraypaint. Delivered! And if bits of paint peel off at altitude and get sucked into the engine, gluing themselves to the turbine blades until catastrophic failure of an engine, well, we can just patch the paint recipe in the next sprint! Paint that's "hot pink" is part of this sprint. The user story about engines that don't fail is part of the next sprint.
The real problem with aircraft design is that all our little user stories are in a big clunky database. If we printed out the database's contents (by hand!) on little 3x5 index cards, then we'd be using the best practices of both Scrum and Kanban. Our planes would be so damn agile they'd have turning radii measured in inches.
When a senior engineer piped up that an aircraft with a turning radius measured in inches would kill everyone on board due to G-forces measured in the thousands of Gs, and would likely tear itself apart because the centripetal force far exceeds the tensile modulus of steel, titanium, carbon fiber, or anything else available, he was terminated because "switching from traditional tube-construction to blended-wing-body design made of unobtanium" was part of the next epic.
What they really mean is they were too lazy to write "catch code." They spend all that time making it hyperintelligent then lazy out and don't write any checks and stops and tests into it. Just a tiny bit of heuristics would have detected that as a trade that maaaaybe might need a human to review for a second first. Even every version of Halo is the same way. Yes, is "shouldn't" happen but put in a check in the movement engine to see if the player is currently moving faster than running or falling should allow. Tada, no "superbounce" glitch. For MW3, if someone gets a score of 15 kills and 0 deaths with 100% accuracy, MAYBE they might be cheating and should be booted from the game. A little AI goes a long way, people. But nope, programmers are just too lazy. Once the product is "done," they're out of there!
It should be called high frequency gambling and taxed as such. It has nothing to do with the (perceived) value but only with a gamble on what the sentiment and competing algorithms will produce as the next stock or derivative price.
Any derivative trade and any stock trade that is done within 28 days of purchase should be taxed as gambling. It's nothing more or less than that so it's fair if these big online casino's get their profits taxed so the rest of us can profit too.
I was promised a flying car. Where is my flying car?
This isn't the equivalent of $500m of infrastructure burning down. The money was lost by some investors, but gained by other investors.
"Anyway, no drug, not even alcohol, causes the fundamental ills of society. If we're looking for the source of our troubles, we shouldn't test people for drugs, we should test them for stupidity, ignorance, greed and love of power." -- P. J. O'Rourke
"Can there be a Klein bottle that is an efficient and effective beer pitcher?"
The really troubling thing (to my mind) is not so much the 'what does Wall St. produce?' question(which, as you note, is ostensibly 'capital allocation'; but the 'how efficiently do they actually produce it?' question.
In a non-pathological market situation, you would hope to see Wall St.'s share of the economy as a whole be static or declining(as newer technology makes allocating capital easier and less expensive) and the demand for 'capital allocation' exist only so far as other business sectors find that more efficient capital allocation makes them more efficient and productive(in the same way that you would want to see any other support function of a business kept in line with the business overall. You wouldn't want your IT group consuming a greater percentage of your total economic output every year). Trouble is, that isn't what the numbers reflect.
Instead of acting as other suppliers do, and having their health and size depend on the success of the customers, the financial services sector has managed to capture a steadily increasing share of the value relative to other sectors. Absolute growth would be one thing, if the economy as a whole is growing; but relative growth, in terms of percentage of total output captured, suggests a substantial increase in the market(and regulatory) power of financial services without necessarily any increase in the value of their product to their customers. That is bad.
Sorry but this whole idea doesn't 'fly'. Avionics are *NOT* the type of cutting edge technology used in stock market matching engines or HFT engines. Avionics are designed to be utter reliable to such a degree that they wind up using older tech. The deployment and approval cycle is also long enough that 'new' for a plane is probably years out of date.
HFTs on the other hand, are bleeding edge systems with essentially a cost-is-no-factor approach. You're talking about a world where microseconds are very litterally counted. 10G and higher network connections - not for data throughput but because it lowers latency but a small but appreciable amount. No, they obviously don't want the FUBAR situaion Knight had because of pushing tech but to assume the stock market is using tech with any resemblance to what's in the DreamLiner shows a lack of understanding of both worlds.
Let's use a car analogy! Sure you can make a race car utterly reliable and safe - it's called a Volvo.* It will undoubtely get you to the finish line for race after race after race with no maintenance while the cars meant for the race break down, crash, need maintenance and so on. Just like race car accidents, you don't usually hear about trading mistakes unless they're spectacular.
*substitute your own preferred car mfg
You can get rich if you own a politician, but you have to be rich to buy one in the first place.
Stop bailing these bastards out every time they screw up.
The purpose of avionics is to get a plane from one point to another without incident.
The purpose of automated stock trading software is to make as much money as possible while screwing the other guy if you get the opportunity.
You'll never make automated stock software 'safe'. Its purpose is inherently risky and combative. You're not up against the laws of physics and the occasional thunderstorm; you're up against other people who have similar software and an urge to hurt you. This is Wall Street PvP (that's Prick-versus-Prick). It's unsafe by its nature.
You cannot make competitions entirely 'safe'. What you can do is pen them in so that they do not hurt bystanders. Just like putting crash walls around a NASCAR track, we need to put up regulations around Wall Street so their blood combat does not spill out and harm the larger economy. Re-implementing Glass-Steagall is the least that we can do to keep Wall Street's fiery crashes from hurting the common people. There are probably more reforms we could make to wall them off properly.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
I'm not sure I'd agree with that part about taking money away from every stockholder, and all. Back in the old days before high frequency trading was in vogue, you would place a stock trade with your broker. Depending on the brokerage firm, it was amazing how many trades that were made claimed to be near the low of the day if you were selling or near the high of the day if you were buying. Of course proving anything was impossible, but the people actually doing the trading were still making a killing.
Now, when you place a stock order you will in most cases in normal markets get it filled at the price that was being quoted when you hit the sell or buy, even if the stock has moved several dollars per share over the course of the day. This is largely due to HFT. Yes, they are making fractions of a cent each share traded and are making lots of money. But they are also providing the opposite side of trades that you want to make at a price near what is being quoted, That lets you, as an individual investor, get the price you have considered to be fair for the security.
Yes, I know all about limit orders and other options to try to control the price you pay or receive and that is always an option, but HFT - when working correctly - has largely eliminated the need for them.
All software trading algorithms are dangerous and can easily cause massive problems when they play against each other or feed on each other. I don't necessarily like them and wouldn't mind them going away. But to say they provide nothing of value is not true -even for a common stock trader who only does a few trades every few months.
This increase in the relative growth of the financial sector was one of the predictions in Karl Marx's Das Kapital: He saw bond markets and stock markets as the natural and predictable outgrowth of the role of a capitalist, which in his view was somebody who made their living not by producing stuff but by buying the means of production and making other people produce stuff. Bonds in particular simply abstract the concept completely away from any actual work: The capitalist now doesn't even do the buying and managing himself, but buys bonds allowing somebody else to do that work and demands a portion of the proceeds of the firm in return. As the capitalist class gains more and more wealth, the trade in bonds and other financial instruments goes up as a percentage of the economy, while the industrial and agricultural production becomes less important.
(And no, I'm not arguing that workers of the world should unite and revolt, just that Marx was a serious economist who made some good points about how capitalism works.)
I am officially gone from
High-frequency automated trading are destroying the stockmarket. With transactions on a stock happening at thousands of times a second, when things go bad they go bad very quickly. When things go well, the normal trader gets reamed.
IMHO High-frequency automated trading should be outlawed, stocks should be bought and sold at a rate comparable to human interaction. Say 1 per second. Then if things go bad, it goes bad over the course of a day and people can react. Brokers normally buy and sell bulk amounts of stock, so this would be no different.
It would level the playing field, and put the normal investor at less of a disadvantage compared to the big companies. If a stock is particularly hot, then some trades won't get made by the end of the day. This is in other words reverting to the stock market of old, where the market could be looked at, and expected to stay the same over the course of a minute.