Slashdot Mirror
Mandelbrot Suggests A Hunt For Financial Patterns
Phoe6 writes "Wired is carrying an Open Letter of Benoît Mandelbrot, the father of the fractal, to the wizards of Wall Street, calling on them to recognize a pattern in the finantial and economic trends in the world. Mandelbrot says, If we can map the human genome, why can't we map how a man loses his livelihood? If millions can contribute a few cycles of their PCs to the search for a signal from outer space, why can't they join a coordinated search for patterns in financial markets?" I'd like to see a debate between Mandelbrot and Friedrich Hayek.
Considering that Hayek has been dead for over 10 years, I think that debate would definitely be worth seeing.
Mandelbrot watches Pi, has idea...
Won't there be problems with predicting what will happen, then acting on the predictions? Almost to the point of being self-fulling prophecy?
Also, I remember very vagually that there are laws about getting a computer buying and selling automatically, to try to curb this?
What would be the outcome? How can markets continue to exist, if their highs and lows can be predicted? I think that the very act of prediction will change the outcome... basically making this impossible to practically achieve.
Visit the Game Programming Wiki!
Technical analysis of markets is a waste of time. When a pattern is found, it is exploited by many, which changes whatever "meaning" the pattern had before.
Send lawyers, guns, and money. Dad, get me out of this.
using either linear algorithmic models or a parrallel neural network approach.
However while most people agree that past performance is indicative of the future nothing can predict what is going to happen. Things such as politics and current events have a huge impact but are not easily factored in to a computer program.
There are many sophisticated solutions to recognising and predicting complex patterns but with the stock market there is an element of trying to predict the lottery.
If the lottery is run properly then every draw should be completely random, any pattern detected in past draws should be about a useful as picking your numbers out of a hat.
Are you familiar with Elliot cycles? Probably not. He came up with an idea like this around a decade ago. (Reading back issues of NewScientist can do wonders for you knowledge like this.) These ideas keep getting thrown around but never come to fruition because at their core they are inaccurate. As simple as that. Whether it's based on power laws, or assumptions about the nature of price spikes (up-up-down-up-up-down), trying to reduce markets to mathematical patterns invariably fails.
8:14 read slashdot /. is the language of nerds.
8:15 restate my assumptions:
1.
2. Everything around us can be represented and understood through discussion threads and trolls.
3. If you graph these numbers, karma emerges.
Therefore: There are karma whores everywhere in nature.
8:17 Press Submit
Better then to focus on economies, and what fundamentals control them. Many of those fundamentals seem to be known, i e you know what things are "good" (low taxes on work, flexible labor market, well educated work force, good infrastructure, good governance and legislation wrt to right of ownership, free trade). Problem is quantifying them and make them interact with all the other less known factors...
This concept was explored in an Aussie movie of a couple of years back, called The Bank. A person previously wronged by a bank was employed to investigate stock market trends, using 'chaos theory' and 'fractal geometry'. Quite an interesting movie to watch.
I lose nothing by running Seti@HOME, and I have nothing (or at least little) to gain. Let's say that my computer is the one that finds the "alien signal" paving way to a real sustainable contact, visits, technology exchange and what have you with an alien civilization. I'd be lucky to end up in a history book.
Similarly, the research groups working on the signal processing, detection, filtering and what have you, will freely share information - again because they have nothing to gain by refraining therefrom.
But financial markets? If my computer can detect that in a few weeks General Electric's shares will plummet - why would I want to give that information away? Would I get a reward from the research group (at a financial institution somewhere most likely) that could (and of course would) benefit from this information?
Would the algorithms even be developed? Why would one group (at Citibank for example) share their information with another group (at GE Capital or whatever)?
There would not be sharing of knowledge. There would not be sharing of results. Simply because the potential gain you have by keeping the information confined is too great.
If you could forecast financial markets reliably on a large scale, imagine how powerful you could become. You could buy the planet.
And this, ladies and gentlemen, is why shit like this won't happen. Not as long as financial markets deal in things that have material value.
If we can map the human genome, why can't we map a man's livelihood? If millions can contribute a few cycles of their PCs to the search for a signal from outer space, why can't they join in a coordinated simulation of Friedrich Hayek?
...finding aliens will have great importance for society. Solving AIDS or cancer or other great medical vices of our time will have great importance for society. But financial markets?
Who'd profit from that? Remember that most of the money made in the stock market is made off the losses of other stock holders - one person's loss is another's gain. If you alone can find the pattern, you profit. If everyone finds the pattern, it has very little value.
Something that made everybody run twice as fast, wouldn't in any real way change sports. The fastest would still be the fastest. Have you truly achieved anything then? I don't see this as a useful cause to dedicate my clock cycles to, do you?
Kjella
Live today, because you never know what tomorrow brings
I don't think he understands that financial and economic markets are linked to world events. Therefore, in order to accurately predict the movement of financial markets, you will need to be able to predict the future! Do you think that a computer could have predicted 9/11 or the Iraq war or elections of world leaders or the Microsoft settlement or a myriad of other news events which effect the direction of the markets? If anyone actually believes that we will be able to design a computer to do this, feel free to reply to me because I have a bridge to sell you!
(Before all you chart-heads jump on me, I do not think technical analysis or charting has any validity, so please do not waste any time trying to convince me otherwise, because it won't work!)
------
www.moneybythenumbers.com
The very nature of stock trading is that no money is created, it's only moved.
Not quite. Transaction fees are the friction in this system. Buying and selling stocks is not a zero sum game. The brokers and exchanges always come out ahead.
You can't just map all the data in the stock market and look for a pattern. The amount of socio-financial data to pattern would be... incomprehensible. This is truly a "butterfly effect" situation. Millions of rumors swirl throughout the trading floor - some make it to us as news - and mapping all that data... it's not that it is impossible, but conceiving of that today is unlikely.
You'd almost need some kind of impartial Bayseian analysis to traffic through millions of petabytes of data. There are patterns surely, just like the weather. Someday, they will be just as apparent.
We can't just "find the pattern" in the stock market, since its created by people in the first place. To attempt to find said pattern it to say that human beings act in a particular way by nature, not volition, and that even in the presence of external force, people will act the same way. The problem becomes harder the more people you have working on it. One person may make a model which is accurate in the broad sense, precisely because he is unaware of the other stock market players, and vice versa. If everyone in the market got together to try and figure out the pattern, then the pattern would be whatever everyone wanted.
I want to delete my account but Slashdot doesn't allow it.
As soon as a structure usable to make money has been found, so many people exploit it that it instantaneously disappear. This is especially true when such structures are explained to the public and not kept ultra-secret in some bank basement.
The problem with the market is that the knowledge humans have about it modifies it.
IMHO, it is not possible to predict the actions of millions of users (not to speak the 1000s of program trading systems) over a long period of time. However, given a sufficiently small window, it may be possible to predict the motion of a security with a better than random probability; and if you have a direct link to trading systems (i.e. low fees), you might be able eke out a meaningful return on investment.
As with most other things, you'd need a hefty investment to pull this off.
Ever heard of Long Term Capital Management? They tried some sophisticated modeling in the late 90's, and at first it went great, but the Asian crisis wiped them out when their models fell apart. All modeling based on historical trends falls apart eventually. I tend to fall in to the camp that believes the mere act of predicting markets makes them less predictable.
When "chaos theory" (better to call it "field", or "approach to complex systems", maybe?) broke upon the world in 1970s, it's not like finance people yawned and ignored it! Au contraire, there was tremendous interest for the subsequent decade, as everyone searched for power laws, fractal dimensions and attractors. But now 30 years have passed, and the conclusion that was reached after research, and not in ignorance, as Mandelbrot suggests, is that financial markets are not predominantly chaotic. In other words, the "randomness" that we see in financial series is unlikely to be generated by underlying repeated actions of some simple nonlinear system; instead, it's really stochastic, really comes from un-anticipatable non-deterministic shocks. And if you really think about how the world works, with companies cheating and countries defaulting on debt, you'll find that intuitive, as well.
Here's one overview of the current state of the matter: Barnett and Serletis, 1998 (disclaimer: found after a 5 minute search of RePEC, there are likely even better papers).
"I am just a customs officer; but I, too, wish to understand what is going on" -- Bertold Brecht
Let me recap Mandelbrot's point here.
In the April 2003 settlement of postbubble fraud charges, the biggest Wall Street firms agreed to cough up $432.5 million to fund "independent" research. Mandelbrot then makes the distinction between two kinds of research. One is the kind of research where analysts study a publicly traded company, and then give recommendations to buy, sell or hold. The other kind is fundamental economic research.
Mandelbrot then suggests that at least five percent of the settlement money be directed toward fundamental research. He does not say that we should look for a way to predict the markets with absolute certainty - that would be impossible, as many here have redundantly pointed out. (He would probably be insulted to know that so many here think that's what he advocated. He's not stupid, you know.)
He's talking about giving a boost to the kind of fundamental economic research that's already taking place. Stuff like risk management, for example. If you read the article, maybe you noticed that in the beginning he clearly gave examples of what's wrong with our present models in risk management.
Bernstein probably read Mandelbrots essays about exactly the same topic and draw its conclusions about that. It was Mandelbrot, who in 1963 suggested that all statistical analysis done so far yielded just one result: Markets are unpredictable by looking at historical data. He coined the term 'scale invariant' to describe the fact, that without a numbered scale you can't tell what period a random example of stock market data describes.
;)
(Mandelbrot (1963): The variation of certain speculative prices. Mandelbrot (1963): New methods in statistical economics.)
So I guess, Mandelbrot knew already 40 years ago what Bernstein wrote
That financial markets are stochastic is an assumption - with extreme levels of 'deterministic noise' it can appear stochastic - a small difference when specifying an equation to predict a future level can make a massive difference: this is chaotic. Stochastic modelling works OK because of the large levels of noise, most modern finance is built on the assumption errors are Gaussian, a framework formalised in the 60s but assumed even earlier. Sadly today anyone can trot out a Black Scholes thesis, support the group think in academia, and get a PhD from a decent institution (a disease in many disciplines other than economics/finance).
Think of financial markets like the weather - tomorrow is more likely to be the same as today, some minor variance, but saying it is going to be largely similar will mean you're right most of the time: this doesn't help pedict a storm. Likewise tomorrow the stockmarkets will likely be the same as today, but this doesn't help predict a crash. As weather the stock market model can be refined, but in the end it is chaotic and hugely deterministic (many agents looking at each other and others actions).
Many agents looking at each other's actions is important - the market does not exist without agents (buyers/sellers) - it is an endogenous process. A co-operative solution will not work - someone always has to do worse for anyone to do better than the market - the market and the economy is just the sum of the actions of participants, participants cannot move against the market as they, in sum, are the market. Calling on the market to recognise a pattern is folly - a small participant can take advantage of any pattern if discovered, but the market as a whole cannot because if they stop their present action to follow/takeadvantage of/neutralise the pattern they have stopped taking action that creates the pattern.
Patterns in financial markets have long been looked into, a good starting point on current thought, if interested, is technical analysis and elliott-wave theory
Can we, I rather doubt it... You only can predict some already occurred situations and patterns.
But that does not mean we already have encountered every possible (well probably there are infinite) situations to a crash.
The problem is, that many of these patterns only show one aspect of such a problem, the numbers, but forget about the root. Sociological factors. An overheated stock market does not necessarily need to lead to a total 1929 like crash, as does high stocks have to cause a crash (in most cases they do but not in all)
Sorry this is the search for the holy grail all over again. What will come out of it is simulation patterns for certain kind of situations which can give you a prewarning system. (But probably will be exploitet by a few to become richer and thus the patterns will be rendered useless again)
All I can see here is that in the end you will get as a result a probability based system again which even is questionable.
Of course a mass investigation into this implies a mass awareness of the result. That awareness though would immediately change the pattern as they try to act on it. Better try to figure it out on your lonesome then aye?
...of the stock market are fear and greed. Once Mandelbrot can find a pattern in these two uniquely human traits, his problem will have solved itself.
Mandelbrot wasn't the first to model financial markets using fractals. Elliott was.
(Note: I have degrees in mathematics and finance.) Mandelbrot is not talking about making price predictions in the stock market, e.g. "if the price goes up on Monday, it will go up on Tuesday". As many ./ers have already noticed, any such scheme would be self-defeating: if people began to anticipate a Tuesday price rise, they would buy on Monday, driving the price up a day early -- and erasing the pattern.
For this reason, most financial models assume that stocks follow a kind of stochastic (i.e. random) process called a "martingale", meaning that returns are uncorrelated over time, so you can never beat the market with a strategy like the one above. However, this leaves open the question of which probability distribution the returns follow.
The earliest models, such as the Black-Scholes model for option pricing, assume that stocks follow geometric Brownian motion -- this means that returns follow a normal probability distribution, i.e. the usual bell curve. However, real world markets do not follow a normal distribution: the tails of the distribution are much "fatter", meaning that the Black-Scholes model underestimates the risk of extreme market moves. Therefore, this is a bad risk model, and a company full of Nobel prize winning PhDs, called Long Term Capital Management, followed it off a cliff in the late 90's, nearly bringing down the US financial markets. (For a captivating account of LTCM's rise and fall, check out the book "When Genius Failed".)
This is what Mandelbrot means when he "encourage[s] the study and adoption of more-realistic risk models". We need a better model for the statistical dynamics of markets in order to properly understand (and price) risk -- i.e. to be able to compute accurately the probability that such-and-such price move will happen -- not to make simple-minded stock predictions.
If you're interested in Mandelbrot's own mathematical work on the subject, I'd recommend his book "Fractals and Scaling in Finance". For a great read about the inherent unpredictability of the markets, try the books "A Random Walk Down Wall Street" or "Fooled By Randomness".
Cheers,
IT
Power corrupts. PowerPoint corrupts absolutely.
I gathered data on daily stock market returns on 5000 companies listed in Standard and Poor's listing of U.S. publicly traded companies.
I normalized then crunched the data through a fractal analysis tool that quantified the level of chaos (randomness) in the changes of each company's stock market value from one day to the next.
I understand the professors studied the data to determine any correlation between each company's chaos metric and the company's eventual bankruptcy.
Now, IANAM* and I have never read any of the resulting research papers, so I cannot tell you many details of the professors' findings.
However, I understand that the hypothesis was that changes in market value characterized by a high amount of chaos (randomness) would correlate to a robust, or healthy business model, just as the life sciences have found that a high degree of chaos correlates with a healthy system. As business managers' actions become constrained by the costs of bankruptcy, so the theory goes, the daily variation in stock market returns become less and less random.
I do recall that they did find a correlation between eventual bankruptcy and suppressed chaos. However, I seem to recall they also found a correlation between a successful turnaround and suppressed chaos.
I guess (IIRC) you could say that suppressed levels of chaos could be a predictor of future business distress, but not necessarily of future business failure.
A list of the research supported by the fractal study follows:
Lindsay, D.H. and A. Campbell. A Fractal Approach to Bankruptcy
Prediction. Business Research Yearbook: Global Business
Perspectives, (2), 1995, 13-17.
Lindsay, D.H. and A. Campbell. The Effect of Deregulation Upon
the Chaotic Properties of Stock Market Time Series Returns.
Business Research Yearbook: Global Business Perspectives, (3),
1996, 13-17.
Lindsay, D.H. and A. Campbell. A Chaos Approach to Bankruptcy
Prediction. Journal of Applied Business Research, (12)4, Fall,
1996, 1-9.
Lindsay, D.H. and Campbell, A. The Effect of Changes in
Proportional Institutional Ownership upon the Chaotic Properties
of Stock Market Time Series Returns. Business Research Yearbook:
Global Business Perspectives, (4), 1997, 267-271.
Lindsay, D.H. and Campbell, A. Beta and the Chaotic Properties of Time Series Returns. Business Research Yearbook: Global Business Perspective, (5), 1998, 7-11.
Lindsay, D.H. and Campbell, A. Public Pension Funds: The Effect of Negative Public Announcements on Chaotic Properties of Returns Business Research Yearbook: Global Business Perspective, (6), 1999,
Lindsay, D.H. and Campbell, A. Risk and Financial Distress: A New Approach, Business Research Yearbook: Global Business Perspective, (7), 2000,
*IANAM - I am not a mathematician. (I am an accountant. Accountants don't need math. We have tables.)
I'm time traveling, right now
Really? In all the near catastophes cited by Madelbrot (http://www.wired.com/wired/archive/12.08/view.htm l?pg=2?tw=wn_tophead_7), a common theme resonates: irresposible and/or corrupt government regulation of banking systems. But...we've known that for years haven't we - that irresponsible government banking regulation precedes financial catstrophe?
Here, let me make a prediction - AND YOU REMEMBER IT: Argentina will recover from it's current financial crises only to again borrow massive sums of money and yet again crash it's economy. But...along the way, Argentina's economy will be hailed as a "Tiger". Then, after the crash, the Argentines will blame thier irresponsibility on an "International Jewish Banking Conspiracy". It happens EVERY TIME.
See...corrupt government regulation at work. I don't need a model to predict THAT future.
I remember when I was in grad school for fluid dynamics at the Von Karaman Institute there was a big fad on modelling turbulence using fractals. While it is true that turbulence is more accuratley described with fractal as opposed to Euclidian geometry, this doesn't necessarily mean that useful predictive model can be produced. The many attempts at modelling turbulence with fractals didn't really produce models more effective then the usual stochastic models that were used. There may have been more progress made in the past few years, but I'm not aware of any major breakthroughs. Granted turbulence is a different problem than the financial markets, but the deterministic chaotic behavior is the simular. Also if you could model the dynamics of the stock market, it is unlikely that I'd be willing to donate CPU cycles just so a few day traders can get rich. Mark
After all, if you're going to replace war with flows of money, what makes you think you are going to have honest scientific discourse in the field of economics?
Its eat or be eaten.
The real solution to war isn't to replace it with economics but rather to direct it against acquisitors who steal from creators. If you do that, then there is a positive sum environment and war becomes far less necessary.
Seastead this.
"Technical analysis of markets is a waste of time."
When you made this comment, were you aware that people apply technical analysis to problems other than forecasting market direction?
What Mandelbrot is suggesting is not the development of a predictive model for entry and exit of positions. Rather, he's suggesting a better model for evaluating the risk in a portfolio once the positions have already been established by whatever means the investor is using. Since risk in this context refers to the risk of unfavorable trade outcomes and trade outcome most definitely *is* a function of price, ((selling price - buying price) * shares transacted), technical methods are applicable.
"When a pattern is found, it is exploited by many, which changes whatever "meaning" the pattern had before."
Your second point is considerably closer to the mark. Any sort of pattern like "if the market closes on a high and it's Thursday..." is unlikely to be of any practical use. However, this doesn't mean all technical analysis is useless. Every trader has to decide whether to be long, short or out, when to get in, when to close the trade and how many shares to transact. Totally disregarding prices while making these decisions is reckless, at best. The alternative is to take price information into account when making these decisions which is, by definition, technical analysis.
As far as "exploited by many" is concerned, it can depend a lot on the payoff structure for the pattern. A pattern that wins every time will quickly be assimilated into common practice but a pattern that is psychologically difficult to trade will remain profitable for much longer. For example all of these patterns have the common trait of having been profitable for decades despite being public knowledge for most of that time. The reason why is that they result in winning trades only about a third of the time, making them emotionally stressful to use, keeping people away. Nevertheless, they make money because the average win is far larger than the average loss.
Besides all that, even if there were a holy grail of technical analysis, most people wouldn't use it. They'd insist that it was just an illusion, so the irony is that skepticism toward technical methods would prevent that holy grail from being crowded out, causing it to last longer than it would in a truly efficient market.
The state is the great fiction by which everyone tries to live at the expense of everybody else. ~F. Bastiat
Some forms of basic research should be publicly funded because they have no inherent reward. For example, Research on Black Holes.
Research on stock markets, is a whole different kettle of fish. He who achieves a superior understanding of the operation of markets may choose the nature and amount of his reward. This type of research will be amply funded by the private sector, and it is. Every major bank employs a large staff of PhD's in Finance, Economics, Physics, and Math, to research these issues. They are handsomly paid and very well supported. (one of these banks is the biggest APL shop around).
Dr. Mandelbrodt's request that the SEC should use public money to fund research on markets shows that he does not understand the distinction between these types of research. The SEC should not use money as he proposed. The SEC should use money to help it discharge its fundamental duty, which is the protection of investors.
A couple of prominent recent examples are high pressure sales of investments to soldiers and selling non-tradeable real estate trust shares to retirees [the Wall Street journal Story was much better, but is on their subscription only site that I cannot hack]. When the SEC figures out how to spot these types of scams before the newspapers, which are reactive organisms too, then they can start worrying about esoterica like the underlying mathmatcal basis of markets. Of course, by then chickens will have lips.
In the land of the blind, the one-eyed man is king.
someone always has to do worse for anyone to do better than the market - the market and the economy is just the sum of the actions of participants, participants cannot move against the market as they, in sum, are the market.
Here you have quite adequately restated the problem; However, at the rsik of attempting to contribute to a topic growth weary of overcontemplation,
Proposed:
That the solution sought is not to permit the whole to move against the market - but to increase the effeciency of the market.
The Effeciency is the amount of value created by the act of corporation relative to the costs in overhead.
In theory, if we all buy into a well, we can dig deeper and get water which as individuals in out of reach.
The cost, is that our individual contributions may be wiped out by unforseen market forces.
For example, We invest in a mutual fund - the mutual fund invests in several drilling companies, and they and 12 other companies all reach water on the same day - we get our water, but not at the theoretical savings possible under an ideal transaction.
The difference in effeciency between an ideal market, and a real market is available as a net gain to all participants if it can be better attained.
I would suggest as a solution, a blend of Short term patents, a market for command economy directives, and a method of evaluating the relative worth of proprietary ideas.
Briefly as applied:
In 1980, about a hundred companies were working on hard disk technology. Some had good ideas, which they kept to themselves. they were all working on parallel projects, which as a result, muted the effeciency of the investment generally, and actually harmed the entire sector, because the result was less than optimal return on investment.
If companies with good ideas, were able to register them for short-term secret protection, and have them evaluated by trusted confidants elected by investors to look at all the competing ideas, and render an early "Command Market Directive" in other words - decide which company has the right to continue research.
In return, the companies with inferior technology can get early warning that they should spend their investors money on research in which they have a core competancy and protect the investors from blind parallel risk.
Companies would not be "forced" to heed the ruling of the common investors group, but investors would be warned, and this would have much the same effect.
Note however, that this solution has little to do with guessing the next days numbers, but rather by opening up the secret doors of publicly traded companies, and warning investors about high rates of overlapping aims.
We laid down too much dark fiber - for much the same reason, and those companies, and the wealth they represent were lost - not because of bad decisions, but because of current spending for access to an oversaturated future market.
The point again - is that the IS room to improve the market and it has little to do with understanding chaos, and more to do with understanding why large companies continue large investments in losing propositions.
AIK
Wallstreet is already doing this, and throwing some pretty powerful brain power and computer power at the problem. Just don't expect the players to share their information with us.
I have a friend who got his PhD in mathematics a few years back. His career options were: 1) Teach (there are far fewer teaching positions than there are candidates), 2) Work for the NSA, 3) or go work for a Wallstreet firm.
He lucked out and got a teaching position at a local city college.
It's not offtopic, dumbass. It's orthogonal.
Sorry, Benoit, but this is just crap. The whole point of the markets is that those who have knowledge end up with the money of those who are ignorant. No matter how good your study is, the markets will change and adapt in response to its conclusions. You won't stop any people losing their livelihoods with such a study unless you give the results to them and nobody else (and then, others will lose their livelihoods instead).
Economics is not hard mathematics. It is not hard science. There are absolutely no axiomatic truths to be had; you cannot even build a wooden shithouse of logic on the foundations that are available, because the foundations shift with every new insight -- and the advantage goes to those who can predict how long that insight will be valuable for; when the uninformed will still believe in it but the cognoscenti will have abandoned it.
You can't map the market, because the market depends on people and world events. You would have to predict or compensate for natural disasters, russia defaulting on loans, rogue traders, etc.
The analogy of the market to the human genome isn't a good one, IMHO, simply because the human genome is essentially in a vacuum, and the market isn't. The market today is very different than it was 20 years ago, but the human genome is still the same as it was 200 years ago.