Slashdot Mirror
Stock-Picking Computers
eldavojohn writes "A while ago, Slashdot ran an article on Algorithms used to augment or replace analysts. Today, the NY Times is running an article on stock-picking computers with quotes from the lovable Ray Kurzweil." From the article: "'Investment firms fall over themselves advertising their latest, most esoteric systems,' said Mr. Lo of M.I.T., who was asked by a $20 billion pension fund to design a neural network. He declined after discovering the investors had no real idea how such networks work. 'There are some pretty substantial misconceptions about what these things can and cannot do,' he said. 'As with any black box, if you don't know why it works, you won't realize when it's stopped working. Even a broken watch is right twice a day.'"
Why spend all that money when a monkey and a dartboard can outperform most analysts already? Surely you can get a monkey and a good set of darts for less than what they were going to spend?
Learning HOW to think is more important than learning WHAT to think.
While the idea of stock picking algorithms is neat; market history suggest it won't work a a way to predict performance. What would be interesting is to better search for arbitrage opportunities to exploit faster than others. Of course, eventually others do the same and it becomes an arms race.
I'm a consultant - I convert gibberish into cash-flow.
to see stockbrokers being made redundant by machinery.
init 11 - for when you need that edge.
Discssing this on /. is futile...
Anyone who really knows anything about this subject wil not post. Too much going on in trading land...
Hence AC post...
If you have a giant set of data, and you set a computer on it for long enough, it should be able to come up with some rather solid paterns/corelations/etc. You come up with these things, and if you don't have to wory about paying the stock broker what ever cut for each transaction (because you are the broker), then constantly trading on these thigns seems like agreat idea.
However, it still can't predict things that a human can (yet). I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments. That was the one thing that the article didn't really talk about.
So I doubt we will see these "black boxes" replacing brokers, simply suplimenting them.
Do Or Do Not, There Is No Spoon, There Is Only Zuul. Everything in the above post is probably opinion.
Most companies provide free trading stations but the tools/software to predict what might happen in the market cost thousands of dollars, yet these tools are never 100% accurate. So, does anyone know where one can grab Open Source versions of these? Thanx.
Forget quotes about neural networks, program trading WORKS. Firms like Goldman Sachs have pulled in hundreds of millions in profits with program trading and will likely continue to do so. Its about replicating human trader behaviours, except with lightning-sharp reflexes.
There are so many theoretical problems with automated stock picking systems that I could spend all day working on this post. Instead, I'll ignore all but two.
First, no matter how well you can predict based on patterns, when you are picking individual stocks, there is such a huge influence from the chaos of human nature that, from day to day, no matter how appropriate your predictions are (based on history), they may have nothing at all to do with reality.
Additionally, if you get enough of these stock picking systems in operation, they can actually change the dynamics of the market, keeping them from being accurate for years as they all try to account for the activities associated with each others' predictions.
The problem with stocks is that in order to know how they are going to perform, you have to know not only what the company is going to do and how their customers are going to respond, but also how the investing public is going to take that news. It's an odd mix of fundementals and faith, in my experience.
The CB App. What's your 20?
While there are probably measurable patterns. Many aspects of a market are choice based. For example, today central bankers are in a real bind. If they raise rates, it could crash housing and the whole economy with it. If they lower rates, there could be a panic out of the dollar to currencies that offer a higher return on interest rates. Once that move starts, than even investments that don't do anything at all (like gold) will rocket, making the panic even worse. In the end a human is making those choices at the helm, and a only a human can have the intuitive understanding that you better buy gold (but not on margin), even if it looks like a crappy investment. Another thing, if someone knows that the computer reads pattern X Y Z as a sell, then they might try to force the stock price to make that pattern against the market to force a computer sell and get in on it at a good price.
There's moneybee: http://uk.moneybee.net/ or maybe wealthlab http://www.wealth-lab.com/
Or you could try genetic algorithms, download the info on the whole market plus historical info, give the algorithms access to the lot, plus downloaded financial news, classify the financial news as good or bad for a stock using a bayesian classifier, add that to the pot and then use evolution to see which algorithms survive best in the market to date.
You may need to build a supercomputer to run enough algorithms to perform the search. Look at it this way though... NOW you have an excuse.
Deleted
[ Yes, I am joking. I'm quite sure Mr. Lo is brilliant -- just maybe a touch too honest. :-) ]
John
'I'll build a stock market neural network for a couple thousand (you'll just need five thousand years of market data to train it before it actually works).'
Where did this quote come from? I can't find it in the article.
Anybody remember Long-Term Capital Management? They were a hedge fund, heavy with experts, even had a Nobel Economics Laureate on their team. They lost 4.6 billion dollars in four months time, having to be bailed out by the big banks to prevent them from crashing the whole market. PBS (I think Frontline) did a documentary on the whole thing. Fascinating stuff - I wish I could find a link for it.
No folly is more costly than the folly of intolerant idealism. - Winston Churchill
Let's see... we want to design a neural net that predicts the summed trading behavior of, oh, several tens of millions of human brains? Beyond the obvious hardware scaling problems, we must think of the software used by these brains, each of which has different biases for selecting stocks. Some choose based on omens they call 'fundamentals'. Some choose based on some form of scrying on the shapes of squiggles of price history charts. Some trade decisions are made on how the trader's 'gut' happens to feel about it. But, in fact, everything stored in these neural nets has some contribution to the outcome of an investing choice. In a close decision, it may be that the memory of having been turned down on an offer to go to the prom might provide the unconscious final impetus to invest or not.
Kinda reminds me of a fanciful and ambitious kid who declares he's going to build an interstellar spaceship out of stuff laying around in the garage.
and other such real world variables into how it makes judgments.
It also can't take into account stock-pumping spam, which has graced so many of our inboxes lately.
The last work of Mandelbort (the 'fractals' father) 'The (Mis)behaviour of markets' http://www.amazon.com/Misbehavior-Markets-Benoit-M andelbrot/dp/0465043550/ is quite interesting.
Sigh, markets are chaotic, much more chaotic than current market analisis states.
What's in a sig?
...we need "buy and hold" time limits on stock transfers, a year or two. Right now it isn't investing in the truly classic sense, it's a combo of manipulation and being the tail wagging the dog with the big houses and this computerised trading swindle. The skimmers and middlemen (the ultimate non producers) actually *ruin* the market for 99.999% of the people out there who might want to actually "invest" in company x,y,z and it forces those Cxx bozos in company x,y or z to do ridiculous "this quarter" stunts. And fucking derivatives? Second biggest con and dangerous economic practice out there, right after the fiat credit based central bank money system.
Wouldn't bother me a bit to see the economy temporarily collapse from that crap and those gents who push that and promote it and the pirate globalism manipulations undergo a little "harsh people's social readjustment" from the people who are going to ultimately suffer from it. And the collapse is going to happen, too, just watch. We are in larger peril than before 1929 right now.
However, it still can't predict things that a human can (yet).People are really bad at predictions. Managed funds rarely even keep up with simple index linked funds. I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments. No this really should be pretty simple. We already have an algorithm that can classify a message as spam or not spam with 99.5% + accuracy, basically good or bad. The same algorithm could be used to classify any news article as good or bad for a particular stock by looking at the direction the stock takes after the news appears. You could even train the classifier automatically by looking a couple of days in the past. Simply classify every single financial announcement against every single stock, eventually it'll get to the point that the classifier will be able to say that an announcement is 80% probably good news for this stock, 92% bad news for that stock.
Deleted
Computer simulated electronic trading systems exist. I know because I use one to manage a hedge fund. I developed this system with my father over the last two years with only 20 years of back-data. We used a certain company for thier data, another company for thier market simulation machine, and another company to provide us with a market timing indicator. All my father and I did was come up with a mathematical algorithim that produces a real world result of 130% return (on average).
Last year we ran our simulation right in-line with our real world account and came out with 152% and this year we are at around 91%. Currently we are fully invested and with full margin in the stock market. Currently the equity in the account is around $6 million. Add full margin, take into consideration the performance, and you can see how much money can be made.
And, to top it all off, this system only makes me have to work about 30 minutes a day and do only about 185 trades a year on average.
Trader 1: It's hit rock bottom. Come on, let's buy.
Trader 1 (on the phone): Buy May belly contracts at...
STOCKBOT:- That's a big mistake, Sir.
Trader 2: Why shouldn't we buy now, STOCKBOT?
STOCKBOT:- The price is going to keep going down.
Trader 1: Randolph, this isn't Monopoly money we're playing with.
Trader 2 (on the phone): This is Randolph. Hold that belly order a moment.
Trader 2: Tell me why you think the price of pork bellies is going down.
STOCKBOT:- It's Christmas time. Everybody's uptight.
Trader 1: Could we please buy now?
STOCKBOT:- If you want to lose money go ahead.
Trader 2: What are you trying to say? Please Explain.
STOCKBOT:- Pork belly prices have been dropping all morning. So everybody's waiting for them to hit rock bottom so they can buy cheap. The people with pork belly contracts are thinking, "Hey, we're losing all our money and Christmas is coming. I won't be able to buy my son the GI Joe with the Kung Fu grip, and my wife won't make love to me 'cause I ain't got no money." They're panicking, screaming, "Sell, sell." They don't want to lose all their money. They are panicking right now. I can feel it. Look at this graph.
Trader 2: He's right, Mortimer, my God, look at it.
STOCKBOT:- I'd wait till you get to 0.35 then buy. You'll have cleared out all the suckers by then.
Trader 2: Do you realise how much money he just saved us?
Trader 1: Money isn't everything, Randolph.
When our name is on the back of your car, we're behind you all the way!
But if someone else uses the same algorithm and beats you to market on your trades by even a fraction of a second your information is outdated.
If it was really possible to predict price behaviour on trading patterns these opportunities would be fully exploited until such a prediction would not work.
Why not simply buy a good profitable company?
Or a company that will be good and profitable?
Really, if you don't know what you're doing then you are the prey. I don't, and I recognise I'd get eaten by the big guys. Better idea to take up a known to work investment strategy and invest for the longer term. You can still beat the market average and the bank savings rates but you won't be making 100% per day.
There are several very simple to manage portfolio strategies which will give you very good returns whether the markets are going up or going down and with little risk. It's not magic, it's common sense, long term investing and frankly a bit dull.
Deleted
This was accomplished in 1998 by Maximillian Cohen, however, shortly thereafter, he drilled a hole in his head and couldn't remember how he accomplished it.
There is quite good theoretical work done on this subject and there are plenty of known (not) working algorithms out there. ;-)
In a nutshell: The problem is that every tranaction changes the market itself. If you just train a neural network, then the neural network only trains for a market without it. Once more and more people start using neural networks, their predictions will become really bad.
Markowitz was way ahead of that in 1956. He asked himself "if there is an algorithm to find a perfect portfolio and everyone in the market is using it, what would that algorithm be like?". And he came up with an impressive solution, read more on wikipedia .
With his algorithm you can calculate the maximum yield portfolio for a given risk if you provide the volatility, estimated yield and corelations of the stocks. These are of course statistical values with a great error, not to mention the crap quality of existing stock market data providers.
But in the end, stock markets aren't rational. They heavily depend on psychological reactions on news. My idea would be to train the neural network with stock data AND text from several big news-pages. Maybe there is a significant connection between keywords and stock movements that can be exploited by a software.
Anyone willing to donate a Teraflop computer? (I would bring beer and pizza
Why anyone on this earth would believe anything a full service broker or fund manager says now a days - a believer would have to have a few screws loose.
According to Prof. Moshe Milevsky (in his book Money Logic) - a stock picker or fund manager or broker would have to be right 9 times out of 10, year after year, in order to be classified as someone who has skill. Anyone - even a five year old - will be right 50% of the time (you know flipping coins - heads versus tails). Only one or two fund managers can beat their competitive index 7 out of 10 times - let alone 9 out of 10 times
So it is laughable when the main stream media (NY Times et al) or the investment charlatans themselves start shouting about a 20% gain in one year. And I'll bet 5 cents that not one so-called computer generated investment factoid caller - is right 90% of the time - all the time. I'll bet that throwing darts will be just as successful
It's odd how online poker is illegal in the United States, but the stock market (which is largely just gambling) is not.
Oolite: Elite-like game. For Mac, Linux and Windows
Now you tell me that someone thinks they can write software that predecits the things that other sofware is trying to conceal. I submit that the market may not be random, but it is also not predictible. Ask yourself this: If you are buying a stock because you think it is going to rise, what is the guy selling it thinking? Perhaps he just wants the money to buy a house, but just as likely he thinks the stock is going to fall. Why are you smarter than him?
-cliff
This is the last thing that's holding me back from adopting Linux. If it came with stock pick software, I would use it.
There are different levels of analysis that can be performed on market data - not all of them as sophisticated as 'stock picking'
For example, you can analyze the price motion of a particular fixed income instrument against some other fixed income instrument (eg some Treasuries.) You may learn that certain bevavior of one is a reliable predictor of the same behavior in the other. Then you could potentially predict a slight movement in the price before it happens, and buy into that security.
You may only win a few pennies per bond here, but if you do this enough and with a large enough pool of money, this can be an earner.
As a matter of fact, this is being done today.
Mock Tech Interviews & Free Resume Review
not using linux because it wont run there favorite game.
The Kruger Dunning explains most post on
Does this bother anyone else? Sometimes it seem to me the stock market isn't about investing anymore, but trading-for-profit, get rich quick style. You know, large scale gambling and playing the odds.
The worst part is when people pour their retirement money into the stock market. (401K, etc.) Then the players and day-traders come along and extract from that pool, skimming off the crests and troughs.
I don't know if there is a way to fix it though, of if it needs to be fixed. Maybe, different currencies: play-money for the traders, and work-credits for everybody else. Or maybe there should be a new market, one just for real investing (in companies) without all the playing around. Or maybe the most practical solution is to not put your retirement in the stock market, unless you have some good players managing it...
Surely you can get a monkey and a good set of darts for less than what they were going to spend?
Getting the required import permits, the actual price of purchase and the care and feeding of a Monkey probably exceeds the price of an analyst's consulatation services. After all, a monkey is far more valuable and useful than, say, that crazy Cramer dude on CNBC (seems like Ballmer's long lost brother doesn't he?).
It doesn't take a very sophisticated algorithm to outperform the average analyst. I suggest a far cheaper solution: My parents still have a lot of stuff from my childhood and I know that stowed somewhere in my basement is that Coleco ADAM. It has more than enough computing power to perform the task. What once came in handy for homework (it's BASIC was applesoft compatible yet the ADAM was much cheaper than an Apple) could be put to use as my investment advisor! I even have 5.25" floppies with SmartCALC spreadsheets of my Dad's portfolio as it stood in late 1986 I could use as a starting point!
In theory, making money in stock market should be easy. Buy low, sell high is a guaranteed successful investment strategy and yet few people follow it. Why is that? By definition, an undervalued stock is an unpopular stock. Doing the unpopular thing is very difficult. Quite simply, human beings are a social animals and going against the crowd is often not in our best interest. Investing in unpopular stocks almost always feels wrong.
However, if you take a rational look at popular and unpopular stocks, you will see that the unpopular stocks are the ones likely to be the real money makers. Consider a mythical corporation that we'll call XYZ Corp. Let's assume that there is a lot of popular interest in the stock. The analysts all believe that great things are in this company's future. Should you invest in it? No, because the stock's price already reflects belief in the company's rosy future. The price of the stock always reflects everything that is known about the company. This concept is important to grasp.
Now, let's consider the future. The future is, of course, impossible to predict, but two outcomes are guaranteed. Either the company will live up to the investors' expecations or it will not. If it does live up to the company's expecation, the price of the stock may stay flat or rise only moderately since the stock price already reflects the high expectations of the investors. If the company fails to meet investor expectations, the price of the stock will drop in response to lowered expectation. Popular stocks don't have very much upside potential in the long run.
Now, let's assume the converse. XYZ Corporation has been mismanaged and had some bad luck. Investors and analysts now consider this stock a "dog." What are the possible outcomes for this stock? Currently the stock price is depressed because it reflects all the pessimism folks have about the company. In the future, either XYZ Corp will continue to run the company poorly or it will change its business strategies and turn itself around. The price of stock will stay flat in the first case and rise in the second case. Unpopular stocks have a great deal of upside potential and little downside since the price already reflects low expectations.
Level 1 or 2 has got nothing to do with the delay of the market data (realtime or delayed).
With Level 1 access you can only see the best bid and the best ask in the market. Level 2 market data includes market depth. It gives you the 10 (depends from exchange to exchange) best bids and asks.
This way you can see, whether a good bid is just a single trader trying desperately to buy a few shares or if it's just the tip of an iceberg.
Delayed level 2 market data is quite common. It just doesn't make too much sense.
bye, Paul.
OMG FIRST POST!!!!!
I am not aware of any real long term benefit in Elliot waves or so called Fibonacci trading. However as with any theory where there are a number of adherants, you can simply follow the flow. This will generally work until there is a major market adjustment.
See my journal, I write things there
I'd pitch a data mining bid based on business domain expert's experience as a system that will produce better results, and is agile and tunable by the business itself, as neural networks are detached from the "real world".
A HUGE amount of the market is in retirement funds or investment funds designed to be liquidated through retirement. My generation will be retiring soon, and withdrawing their funds from the market. If the USA keeps producing, it may be a good place for foreign investment that will buy the stocks being sold. However, there's about 10 Trillion dollars of off-balance-sheet liabilities in Social Security, Medicaid and Medicare out there. (If the USA was Enron or Worldcomm, the last 8 Presidents and all the legislators would be in prison today.) I suspect that the government will fund these liabilities by inflating the currency and raising taxes, making the USA attractive investment opportunity for only China. The gambling schemes, including the derivatives you talk about, will fold. The world will get hungry...invest in food.
I support the investor's right to be as stupid as they wish. A buy and hold regulation won't do as much good as something like the Fair Tax Plan, increased personal debt reduction, a balanced budget, and reduced spending. Presumably, investments are property that an investor should be able to divest anytime. Leave the government out of it.
"The mind works quicker than you think!"
First of all, the article summary seems a bit misleading because people might think we're talking about "stock picking" as in analyst opinions/monkeys picking stocks. That's not so, this is about software driven trading. Often this is of a very short term nature, so in a way the growing use of program trading is the "new day trading" fad.
In the industry it's called "program trading" and refers to automated, algorithmic trading of instruments such as stocks, futures, forex. This is regularly done by many banks and large funds, and also small investors. In fact there is a discount brokerage which I'll just call IB here, that has an API which lets anyone program their own computerized trading. It's a bit "too easy" to do.
That doesn't mean it's always profitable in the long term, but without a doubt people are profiting at least in the short term. The software has multiple strategies, well documented approaches and algorithms. Generally the trading robot is trying to ride trends.
As someone who follows these things, here are a few criticisms I'm aware of:
1. These short term trading activities require high leverage, because trades have to be for large amounts of money to make them worthwhile. You need large amounts of money to make this work, because things like trading costs eat into profits tremendously. Again, like day trading.
2. High leverage is risky because one big mistake or unpleasant event could wipe out tons of past small gains. Risk management becomes a key issue. Some would argue that perceived risk in markets these days is unreasonably low. Does this unbalance the risk/reward equation?
3. Market-wide, we know program trading has increased dramatically on US exchanges. Add to this the undocumented program trading (smaller traders who don't have to report it to anyone) and basically there are a ton of computer algorithms out there today trading stocks. Everybody can't make money at the same time, so to profit the participants have to use even greater leverage = more risk.
4. Programming flaws, bugs, or improper risk management could have tremendous market-wide implications. Take for example the huge market moves in 1987; the drop was a "20-sigma" event and not anywhere within the realm of possibility back then. Obviously the models failed to handle it. Similarly, the next time we have a "big event" in markets, today's algorithms might fail. If a large number of computers choke while trading, could bad things happen?
5. So under unstable market conditions, the program trading could lead to increased volatility (like daytrading caused volatile markets during the crash). But under stable market conditions, like we have today, program trading seems to smooth out daily movements. Notice that the US markets hardly move as much as 1% in a day; trends are smooth and volatility is extremely low. The VIX, a volatility measure, has hit historic lows.
"Even a broken watch is right twice a day."
Except those much-less-accurate 24 hour watches...
If you think about the number of symbols being tracked from the key exchanges, we have over 250,000. A publish an subscribe system usually bottoms out at 20,000 topics. (In the old days, SmartSockets had trouble with 2,000.)
The number of events/updates comming through, at least 7 years ago, was about 800 every second. That's a lot of information.
So, what can we do with that much information that is updated so frequently?
Well, implementing time and sales for just transactions takes lots of disk space. Gig for a day. Now, try and keep yearly data. Now you have huge, absolutely freaking huge, data storage issues. And search... wow. Neural doesn't seem appropriate.
So where do we actually implement automated trading?
Look for a few commodity stocks, high volume, no dividend. These will follow certain daily trends. Apply some fuzzy logic - threshold and objective functions. In the morning, we are high. We drop until mid day, and then we go up a bit. Drop again in the after noon, and we're usually high or low at end of day more along our daily trend.
And then there is NASDAQ-OTC, or any pink sheeted company. Here we take advantage of stupid people and penny stocks. If the stock is worthless, there is a large float, and there is news, and suddenly there are a bunch of transactions, we can start our automated shorting. News flows a lot less frequently. And you know a dump is comming after a reverse split. This is highly automatable.
Any fund managers want to give me money to write it for them? It has been done before...
/\/\icro/\/\uncher
... was never right.
So now we have an idea on how to make the money:
Step 1: Find source of stock data.
Step 2: Pipe the data into charts.
Step 3: ??? - Make BS comments on the data. (It's currently working for other investors).
Step 4: Profit.
Crap.... someone help me with part 1.
You didn't think the 370 Trillion in derivatives was traded by humans did you?
Of course, the entire planet's GDP is only 60 trillion, so even a little mistake means complete global meltdown.
.
- Adam L. Beberg - The Cosm Project - http://www.mithral.com/
This sort of creeps me out. Soon we'll have some utilitarian computer deciding upon the most desireable allocation of resources, which essentially won't be much different from some totalitarian planned economy in nature -- just that this one will be "perfect". You will not be able to fight its choices in any way, because if you did, you'd find yourself out of "investment" pretty quickly. There will be no room for any kind of altruistic humanity either, as this will be, well, inefficient...
I can already hear the Libertarians wetting their pants...
I want to play Free Market with a drowning Libertarian.
Having the cash to manipulate the market and/or having insider information is the best way to make money. Take this a step further. If you can gain access to information about the trading programs that some of these big funds are using and have a reasonable idea of how they will react to certain movements, you could manipulate the market intentionally to induce a response that you can profit from. And of course, if you don't have the cash to influence the market and don't have any way to access inside information, you can simply run a pump and dump spam scheme to achieve very quick results.
At the end of the day, spending millions of dollars and lots of effort developing complex systems to predict market movements isn't the way to go about making the real cash. The real cash is in making the actual market and/or knowing the material information that is going to affect the market before it becomes public knowledge.
I've always wondered something. When you get a return on investment greater than the rate of inflation, is the money you're receiving entirely money lost by other investors elsewhere? In other words, is investment a zero-sum game? ("Zero" being relative to inflation.)
Melissa
"Screw Sun, cross-platform will never work. Let's move on and steal the Java language." - Visual J++ Product Manager
No, the economy is not a zero-sum game. Real-world development (building more factories, getting better tech., etc.) translates into money in the overall pool. Even just buying a diverse, standardized index (like the S&P500) and sticking with it will do significantly better than inflation.