Slashdot Mirror
SEC Blames Computer Algorithm For 'Flash Crash'
Lucas123 writes "The US Securities and Exchange Commission and the Commodity Futures Trading Commission today issued an 87-page report (PDF) on the results of a months-long investigation into the May 6 'flash crash' that sent the Dow tumbling almost 1,000 points in a half hour. The Commissions are holding a single trading firm's automated trade execution platform responsible for the crash, saying it dumped 75,000 sell orders into the Chicago Mercantile Exchange over a period of minutes causing an already volatile market to come crashing down. The SEC has already enacted some quick rules to pause trading if a stock price should rise or fall by 10% in a five minute period, but the regulators said they expect the results of the investigation to prompt additional rules limiting the functions of automated computer trading systems."
Here's the way this went down. Because this malfunction dumped market (not limited) sell orders without matching buys, they quickly soaked up all of the buy bids that were on the market, leaving only outrageously low buy bids that usually don't see the light of day at the top of the pile. Those got filled too, and suddenly you've got everything trading at 90ish% off what it was a moment before. CNBC and other instant media realizes that something's amiss... Jim Cramer happened to be making his regular afternoon visit to the daytime programming and shouted out a pretend limit buy order for the stock he was scheduled to say was overvalued... he then "sold" that order a few moments later to show there was instant profits to be made by somebody. This selloff was nonsense, and the market quickly recovered to where it was before minus some losses for the fact that some of the investing public was losing faith in the system.
Now, since this was a malfunction, the people who lost 90% instantly and the people in the other side of those trades who made 80% did so by foul play. The flash crash trades were busted (market regulators ordered them undone) and the world went on like this never happened.
There used to be rules that if there was nonsense at the NYSE, the specialist on the floor would ask questions and stop processing trades. If there was no news to make a fundamental change in the stock and there were suddenly sellers but no legit-priced buyers... just shout out that this was going on and some buyers would be sure to show up.
But now, with many electronic places competing with the NYSE, an NYSE-only stop to computers damage that needs to be routed around, and the crash continued at these exchanges. So, the SEC at its level over all of these systems is establishing rules under which every exchange has to stop processing trades in the affected issues until there's enough time for the news of the event has spread and everybody's had a chance to react.
Market rules are based on trying to give everybody involved a fair chance to trade. Trading on information you have that isn't public yet is not allowed. Martha Stewart went to jail because she had money in IMClone, and was called before the news was out by somebody telling her an FDA trial had a failed result. She sold immediately, and then we she realized she had fouled, faked her phone records about the call. Gotta play fair... they are watching.
"Prevent" is such a strong word. They're good at keeping bad things from happening, just not perfect.
Remind me, why do we have such a fragile system at the very core of modern civilisation?
Hope the person(s) who wrote that algorithm aren't writing nuclear reactor code. I'll admit though that I'm a bad programmer too. Back when I did write code I used such gems as DIM TotalSales AS INTEGER. That didn't work so well.
"This food is problematic."
This was one of those things that happened outside of the rules.
RIP America
July 4, 1776 - September 11, 2001
It was a Solaris backend using a database on Linux that had a Java front end on a Windows PC. The trader monitoring the system was watching porn his Macbook Pro and didn't notice when things went kaflooey.
RIP America
July 4, 1776 - September 11, 2001
I agree that "prevent" could be considered wrong.
What I detect is that the smartest & most motivated people do NOT inhabit the regulatory agencies and indeed probably do not inhabit the exchanges themselves.
The various brokers hire people at much higher salaries &/or bonuses and pay them VERY well to find the tactics, some would say loopholes, to allow quick profits each day. That in itself is not what the original intent of share ownership markets were about.
I wonder when the word "Day Trader" was invented, but it certainly was quite awhile back but it didn't include the ability to do tens of thousands of trades out of one broker in a matter of seconds, and it certainly wasn't considered why we needed a share exchange in the first place. Exchanges were to allow companies to raise funds and to promote their value based on earnings and assets over time and allow a company to achieve an immortal status that an individual person could not achieve.
I think governments as the regulatory overseer are flawed, but then recognize the brokers are also very self-interested, so the whole mess needs more transparency.
That sort of transparency has been achieved with the likes of Linux.
I wonder if open sourcing the rules of the share markets could achieve the results where everyone knows the rules of the game & small individual investors have the same info that the large brokers do?
The worst thing in the world for a share market is to eliminate the small investor leaving only the whales to thrash about.
It is a big problem to solve and the self-interest of the big brokers cause all sorts of broken arms in WDC, if I guess right (meaning $s passed behind between arms).
Transparency is the only solution I see.
Seriously. I think most people will admit it isn't perfect, and it looks like they are trying to improve the system as a result of this. However fundamentally, it works. It helps money move around more, so that businesses can get financing, individuals can invest and so on.
The reason why you find that prosperous nations have things like a stock market and other capitalist features is because they work. Doesn't mean you ignore them or let them run totally wild, but fundamentally they get the job done, where as a command and control economy does not. While it may add instability it also adds flexibility and that is important.
This wasn't a BSOD or General Protection Fault kind of crash, or even a DIV/0. It was an order that didn't have any of the safety measures that should have been there, any one of which could have prevented this from making news. So, there's a new rule... if somebody yells "SELL!" and no buyers show up... they don't fill orders for a penny, they stop, report there's a crazy man in the room, and then there's an auction held to determine who the lucky bidders are.
So; it would have been fine had they used *BSD ;^)
Great minds think alike; fools seldom differ.
The worst thing in the world for a share market is to eliminate the small investor leaving only the whales to thrash about.
Frankly, 99.9% of all people who "invest" in the markets do not have sufficient training in the ways they can be screwed by people who know what they're doing, and are therefore not the sort of reasonable actors that would tend to create rational markets, but are instead cattle to be slaughtered by manipulation. The prices are bogus, nothing more than bait to lure them into the pen where their trading accounts are drained and the bolt stamps a hunk of their skull into their brain.
The best thing for the markets would be to require investors to be certified to put their money there.
But the people running the markets don't want the best thing for the markets, they want the best thing for themselves, and they can afford to buy enough votes in Congress to make sure it stays that way, at least until they make a mistake and show a little of what's behind the curtain, as they did here.
Remind me, why do we have such a fragile system at the very core of modern civilisation?
Define 'core of civilization.' I don't view stock markets as that kind of thing. Regardless, I believe the reasoning they allow it is that -- like everything else in that crazy place of Wall Street -- it can help you make or lose money. This wasn't the only investigation where an algorithm screwed up. I submitted a story that wasn't accepted about an algorithm that lost one company a million dollars in five seconds.
So, you know, before you sign up to let a high frequency trader manage your trades, take note of the risks you are accepting. In the story I reported, the company that lost the money just fired the guy who wrote the algorithm and keeps doing it.
If it's like margin trading where people were taking loans and lost it all and everything died because everyone was doing it, then it's bad. The question is whether or not these micro translations are going to suddenly force everyone all at once to realize their losses. I don't think that's the case but the 'flash crash' might be proof otherwise.
In defense of high frequency trading, I don't see it as anymore of a gamble than regular trading. You are shifting money around to make more money. So you shift tinier amounts faster and for shorter periods of time to get better returns. I'm not doing it so if it turns out to be bad for the people doing it then I'm going to benefit. If it turns out to be good for the people doing it then I bitch because I don't have that same benefit. If the HFTs are putting everyone at risk, I'd like to hear precisely how that logic follows because right now it's looking like it sporadically injects chance volatility that we've dealt with before.
My work here is dung.
High Frequency Trading algorithms are most likely written by a very small number of people, who probably even know each other. The approaches to creating these algorithms should be very similar.
So it should not come as a surprise that given the same set of market data (news/some stocks going up/some going down/interest rates velocity/housing data/confidence/M1/Mx/etc.) the algorithms used by different HFT houses would respond in a similar fashion.
Imagine HFT House 1, 2, 3.
Now if one of them (1) noticed the market data at the same time *(and they saw that Japan was doing something funky with currency at that time) it started calculating probability of stocks going up or down and decided to play it safe (which at that time meant moving out of equities and commodities into cash), so it started to sell.
Now the other one (2) noticed the same thing about the market and noticed that (1) is selling, so it (2) upped the probability that stocks will go down and also decided to 'play safe' and started moving into dollars.
Same with the last one (3).
Now everybody is trying to sell at increased velocities. First they do their normal 5000 transactions/second post and cancel routine, but eventually they would actually stop canceling, prompting the rest of the market to start selling, triggering the automatic retail stops etc.
The HFT algorithms are really synchronized when it comes to overall data and they magnify the resulting movement by each others' actions.
BTW., you'll soon notice that bad news will no longer cause stock markets to go down, but instead they'll go up and so will commodities, that's because it is now recognized that bad news = more quantitative easing = more inflation = weaker dollar. So who wants to buy dollar on bad news, if bad news really means that the Fed will print more dollars? Same with bonds, buying bonds is stupid, they'll eventually figure it out. Buying bonds is like buying dollars to be received a number of years into the future. BUT if you don't want dollars that are inflating NOW, why would you want the inflated dollars in the FUTURE? Makes no sense, so bonds will also go down upon bad news eventually.
You can see these mini flash events caused by HFT in different market segments through the day, if one bank goes down in a very short time frame, then you can be almost certain that most of them went down by the same amount, and then they'll all come back to almost the original levels minus the retail auto stops that'll be eaten. Don't be a sucker, move your money to commodities or foreign equities.
You can't handle the truth.
One of the thing that was made clear to me over the last few years was that the price of stock is whatever the last person bid for it.
The price of ANYTHING is the price of the last accepted bid. Always has been, always will be.
It isn't based on the book value of the company.
Not directly, no. Really stock prices are usually based on a collective opinion of the future profit making prospects of the company. Sometimes though they are based on things that have little or even nothing whatsoever to do with profits. (Exhibit A is the dot com bubble in the late 1990s) The stock market is really not much different than any other form of betting and it only secondarily has anything to do with the actual finances of the company.
Value is a subjective thing. I'm an accountant in my day job and I'll be the first to tell you that valuation is probably more of an art than a science. Opinion plays a huge role because the same thing can be worth very different amounts to different people.
Except that stock speculation has NOTHING to do with investment anymore. Wall street does NOT invest, it speculates. It is gambling on the minute by minute perceived loss and gains in the world with a hefty amount of making events happen.
Take the recent event of a speculator simply buying up chocolate to drive up the price. What has that got to do with investment or making money go around? Nothing at all.
You have the idea that the stock market is still the old idea of buying a share in a ships voyage when this was first made official in Holland centuries ago.
Yes, if you buy shares in a company hoping to get dividend from it in the future, then you are investing. When you are shorting stocks on the difference in value over a period of minutes, that is NOT investment.
Stop pretending that it is.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
According to the CME Group:
"The 75,000 contracts represented 1.3% of the total E-Mini volume of 5.7 million contracts on May 6 and less than 9% of the volume during the time period in which the orders were executed. The prevailing market sentiment was evident well before these orders were placed, and the orders, as well as the manner in which they were entered, were both legitimate and consistent with market practices. These hedging orders were entered in relatively small quantities and in a manner designed to dynamically adapt to market liquidity by participating in a target percentage of 9% of the volume executed in the market. As a result of the significant volumes traded in the market, the hedge was completed in approximately twenty minutes, with more than half of the participant's volume executed as the market rallied – not as the market declined. Additionally, the aggregate size of this participant's orders was not known to other market participants.
Additionally, the most precipitous period of market decline in the E-Mini S&P 500 futures on May 6 occurred during the 3½ minute period immediately preceding the market bottom that was established at 13:45:28. During that period, the participant hedging its portfolio represented less than 5% of the total volume of sales in the market."
http://cmegroup.mediaroom.com/index.php?s=43&item=3068&pagetemplate=article
The SEC/CFTC report is typical of something that we tend to see come out of government agencies (low quality analysis). Also, they didn't make any meaningful recommendations. It seems like that just tried to rush something out as quickly as possible to say, "Everyting is fine. Retail investors can hazard their capital again. We caught the evil, responsible financial firm and will sort them properly."
As a buy-and-hold investor, why do you care whether high-frequency trading exists at all? The flash crash was largely erased shortly thereafter, so it wasn't like it artificially destroyed your wealth. As a person who believes that a core value of our moral system should be those things that do not impinge on the rights of others should be allowed (with notable and obvious exception), I find banning high value trading simply because we are afraid the market will do strange things is silly.
When it comes down to it, the flash crash was a boon for the buy and hold investor, since you got an opportunity to buy things at great prices. And, when it comes times to sell, having a bazillion automated trades in the system ensures your trade will get lost in the liquidity, practically guaranteeing a fair price. Wipe out market liquidity and you are suddenly at the mercy of whoever happens to want to buy that day.
-Ryan
AUWYHSTOT (Acronyms are Useless When You Have to Spell Them Out Too)
What I detect is that the smartest & most motivated people do NOT inhabit the regulatory agencies
That is by design. The agency wasn't ever going to go away, but their efficacy sure did in the holy pursuit of unfettered Capitalism. What has that gotten the majority of Americans who believed in the wisdom and efficacy of deregulation?
-Banking system on national life support.
-Consumers with no confidence in many forms of economic activity.
-A series of economic bubbles
It never works out and yet voters are more than willing to get screwed again under the new mantra of "fiscal austerity." That's more pocket picking for the recovering Capitalists living in your parent's basement.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
Numbers below are courtesy Peter Schiff:
October 1 2010
Gold: new high
Silver: new 30 year high
Gold stocks hit 52 week high
Oil: strong day and strong week
Dollar: dropped 13 percent from peak 3 months ago
September is done, media says: this is best September in 71 years. Dow gained 7.7%, SMP gained 8.8%.
However this month of September.
CRB Index (commodities): gained 8.7% - beat DOW and just under SMP
Soy beans: up 9.5% - beat SMP
Copper: up 10% - beat SMP
Rice: up 10% - beat SMP
Oil: up 11% - beat SMP
Corn: up 12% - beat SMP
Silver: up 13% - beat SMP
Frozen concentrated orange juice: up 13% - beat SMP
Cotton: up 17.5% - beat SMP
Sugar: up 19.3% - beat SMP
Currencies:
Swiss Frank: up 4.6%
Euro: up 7%
Australian Dollar: up 9% - beat SMP
--
Realize that this so called 'best September' is no such thing, what you are observing is huge, very fast inflation.
Beware of USD and US bonds.
Fed says that this inflation is still too low, to slow, prices are not rising fast enough for the Fed. Fed wants your prices to go up up up up up up up.
Buy sugar and get out of the dollar.
You can't handle the truth.
I just finished reading through the whole report. It's fascinating, if you're into this.
First, none of this involved a "bug" . All systems involved functioned as designed.
What's going on here is a logical consequence of the way the markets are set up. The Chicago Mercantile Exchange ("CME", the futures market, which started by trading grain) has a tradeable commodity called the "E-mini", which is a derivative security based on the S&P 500 stocks. Anyone can buy or sell contracts in E-minis, and can also buy or sell the underlying stocks. This generates a frantic amount of short-term trading from market players trying to profit from the differences between the two, which keeps the price of the E-mini close to the prices of the S&P 500 stocks.
None of this is productive activity, of course.
There's a consolidated feed from all markets that everybody gets. It has a few seconds of lag. To obtain an advantage in fast trading, some of the players buy direct exchange feeds with an average of 8ms (yes, 8 milliseconds) of lag.
What started the crash was that a fundamentals trader (one who actually pays attention to the companies involved) was selling $4 billion in stocks. Ordinarily, this isn't a big deal. They had a program throttling their rate of sale to 9% of market volume in the last minute, to avoid depressing the market. That's normal. So far, so good.
However, in response to this sale, the "high-frequency traders" started frantically trading back and forth to balance their portfolios. Their net effect didn't move prices much, but it pushed volume up. So the big seller started selling faster.
This generated enough volatility that some market players started dropping out, decreasing liquidity. That generated market imbalances which other traders started to exploit. Then, because of all this frantic trading, the consolidated market feed and the millisecond feed differed enough that some trading firms had data quality alarms and dropped out of trading. Since traders who are "market makers" are required to maintain buy and sell bids in the market, they defaulted to their default bids - buy at $0.01, sell at $100,000. Some trades actually took place at those prices. 895 shares of Apple stock were sold at $100,000. The price of Accenture fell from $30 to $0.01 in seven seconds, then recovered within the next minute.
Then "At 2:45:48, trading on E-Mini was paused for five seconds when the CME Stop Logic Functionality was triggered in order to prevent a cascade of further price declines". Yes, a 5-second automatic trading halt. That was enough to start to stabilize the E-mini contract trading on the CME. But by then, the E-mini was enough out of sync with the underlying stocks (mostly on the NYSE) that trading on the NYSE started to move stocks there to resync with the E-mini.
The NYSE still has a trading floor, which slows it down. This didn't help. But that's another story.
Nothing failed. Nobody did anything wrong. The original seller's strategy for unloading $4 billion in stock was reasonable. This is all a consequence of normal market operation. The report concludes that speeding up the consolidated market feed to get the 5-second lag (which was more than fast enough before program trading) down should be done. That's it.
Whether or not society should support an "efficient market" system to this extent is an question one is not supposed to ask.
Here are few important facts:
1. Waddell & Reed is the company whose aggressive selling triggered drop in S&P 500 futures price. The company is not HFT shop but rather long-term investment hedge fund. More here: http://www.bloomberg.com/news/2010-09-30/waddell-reed-e-mini-trades-are-said-to-have-helped-trigger-may-6-crash.html
2. According to SEC report, HFT traders played their intended role: smooth out short-term price volatility. However, due to enormous size of Waddell & Reed selloff (about $4 billion dollars in 75000 futures contracts during 20min.) they can do only that much. W&R just cut right through the order book on CME.
3. Slowing down the trading on NYSE did not help but rather hurt by locking up liquidity. Shitty NYSE Arca systems that handle ETFs overloaded and further exacerbated the problems.
4. At the end of day market returned to pre-crash levels. Long term investors were not hurt, W&R payed between 100 and 200 millions for their mistake.
5. Overall, market worked as expected.
The problem was there's similar rules for crashes of individual stocks, but those rules were only at the NYSE and not everywhere. Now they're everywhere. Problem solved.
The algorithm didn't fail, Wall Street as an institution has failed. The simplistic view of why capitalism works is that individuals and institutions making informed decisions results in good allocation of resources. The profitable thrive and the unprofitable die, and on the whole society benefits.
None of the preconditions for capitalism exist in the current setup. The big entrenched special interests change the nature of the system so that they take profit and are shielded from risk. The technical term for this is "moral hazard". The TARP bailout is the perfect example of this. All the big Wall St. firms made huge amounts of money by playing insider games with mortgage back securities (MBS) and collateralized debt obligations (CDO), and then when their gambling resulted in failure, the were bailed out to the tune of ONE HALF TRILLION DOLLARS, and the government is left with the bad assets. And the people who caused the mess are still in charge and got to keep all the money they stole during the bubble, as well as the money they got from the Treasury. Does the phrase "moral hazard" seem sufficient to describe this behavior, or would "rape, pillage and burn" seem more appropriate?
Programmed high frequency trading (along with hedge funds) are another mechanism for taking wealth from the system that breaks the capitalistic model. The claim is the it "increases fluidity" and therefore make the market "more efficient". The plain English translation of "more efficient" is theft, and "increases fluidity" is like saying "magic pixie dust".
The real world value of a company cannot change at millisecond resolution. The only things of economic value that change that fast are electronic abstractions of money. Therefore, high frequency trading is completely disconnected from real world value, so no capitalism is involved. The system is built so that insiders can become personally wealthy because they are the insiders, not because they do an efficient (good) job of allocating resources and benefiting society.
This is identical to the MBS/CDO monstrosity, in that there is no clear real world description of how value is created. For MBS/CDOs there was a lot of math that no one making decisions really understood, but somehow mortgages from buyers who were previously unqualified could become AAA securities. For flash trading there is "fluidity" and algorithms that traders don't understand. It is the same kind of scam.
As long as the stock market allows high frequency trading it will be intrinsically unstable, because this kind of trading is about manipulating the abstract system, not about real world issues. No set of rules will change things, because computationally based trading is about taking advantage of rules to get advantage via manipulation.
The only kind of rule changes that will help are things like increasing the cost of individual trades or keeping electronic traders from placing trades that they cannot or do not intend to make. (Trading algorithm determine price points by placing lots of orders and seeing which ones get responses.) Or electronic traders must be forced to honor trades or hold assets that they are trading, so they are exposed to the market risks of the underlying securities. Right now there is no cost for these traders for any manipulative practices, which effectively decouples risk from reward. All these kinds proposals move this kind of trading back towards actual capitalism.
It will be very hard to get meaningful changes to high frequency trading because the powerful and personally corrupt Wall St. insiders don't want a capitalistic system, they want their guaranteed profits. It is much closer to a Mafia style protection racket then a system that enables real business activity.
Why is Snark Required?
As I said, the market is the big guys.
There isn't enough money in the little guys to even make a market, nor so much that if the little guys were removed from direct contact from it they'd make a difference.
Seriously. The little guys don't even know that they're buying nothing when they enter the market. They get a meaningless portion of the votes of a corporation. They might get a meager dividend, but that's one of the carrots used to lure them in; it's certainly no significant piece of the profits. They get no access to the company. They won't be let into the building. They can't see any proprietary information. They don't get a discount. They don't even get the CEO's phone number. Some "owner" that makes them.
The stock market isn't investing. It's gambling. It's an ornate casino run by people who know full well that there are a million suckers born every day, and each one of them has a lifetime of earning power to be squeezed into the funnel.
Introduce a damping factor. Tax the earnnings by a small amount (say between 1 and 2 percent).
Every physicist, every engineer knows that dynamic systems without a damping factor tend to instability
Use this tax revenue to get the poorest economies out of their misery.
Sometimees those economist remind me of early proto-engineers (say Western Europe, Middle Ages). Quite capable, but blind-sided by ideology.
Why is this moderated Funny?
Wouldn't Interesting be a better fit?