Internet Data Mining for Investment Analysis

CaroKann writes "Reuters is reporting on a Wall Street investment research company, Majestic Research, that is using web crawling techniques to track business performance. Instead of attempting to estimate business conditions by talking to company management, or pounding the pavement visiting stores, this company uses data mining systems to collect real-time sales data and other information on companies that have a web presence. Using this data, Majestic attempts to estimate company earnings more accurately than traditional research outfits."

Comments

Traditional Wall Street Research?

[eldavojohn]

But the New York-based research firm is winning converts among hedge funds who say its brand of Web-based quantitative analysis can be more accurate than traditional Wall Street research forecasts.

Possibly because "traditional Wall Street research" involves reading tea leaves and throwing down chicken bones while watching Alan Greenspan do a rain dance to the gods in hopes that our economy will pick up.

Economics and future fiscal predictions are completely theoretical. There are just too many variables involved, folks.

Re:Traditional Wall Street Research?

[nelsonal]

The only factor Greenspan had in most models was "the Greenspan put" which impacted growth only indirectly (because it freed an awful lot of risk capital). Macro forecasts are not worthe the paper they are printed on, my models were designed to interrpet when others were missing things like market share shifts and competitive advantages that were forming or decaying. The economy played a relativly small role in how those dynamics shifted (ie Dell was a better competitor than HP in good times 1998-2000, and bad 2001-2003) but to what extent did investors give Dell too much credit and HP too little credit for their successes and failures. Spotting that early is what most analysts are paid to do.

Re:Traditional Wall Street Research?

[Uber Banker]

Macro forecasts are not worthe the paper they are printed on, my models were designed to interrpet when others were missing things like market share shifts and competitive advantages that...

Good for you. And when you get your fixed interest rate mortgage, when a company takes out a loan for investment, when you have a great investment idea but need to buy a cascade of 10 year futures (invest in a gas pipeline but want to invest in the quality of the management rather than take risks on anything and everything affecting the gas price and the second order effects on your investment) to mitigate your risk and make the bucks you planned, you are relying on macro forecasts.

They are broad-brush efforts, very rarely would a 'macro person' deny that. But they are essential to the modern running of our global economy. The point of 'analysts' [stock market analysts, rather than economic analysts] is to have ideas about individual companies, perhaps sectors vs. sector or region vs. region if pushed to it. Consensus Forecasts (which accumulate individual stock market analysts across companies, markets and countries) are laughable when looked at on a macro level, even more laughable than economist estimates. There are some excellent analysts, but they are usually the ones that accept estimation is, to varying degrees, educated guesswork, rarely will there be one sure choice; and if there is, that's because time, dedication, luck, restraint and insight have been combined.

If any readers fancy a quick run-down of macro economic policy, I recommend today's publication of the Bank of England Inflation Report, or more specifically, the live press briefing where topics are discussed in a straight-forward way. Hopefully Bernake will implement something similar at the Federal Reserve to improve on Greenspan's smoke and daggers approach to public statements.

Now that the companies know that...

[drgonzo59]

They can create bogus pages to feed to the Majestic bot like in the BMW vs. Google case.

Cue the web spam...

[Rob T Firefly]

We can expect yet another huge rise in fake blogs, fake product reviews on Amazon and such, and paid shills in chats and message boards. Swell.

My data mining results

[CaptainFork]

based on manually mining (eg reading) Slashdot I determine a spike in Majestic's share price about now...

I call Bull

[spectrokid]

TFA mentions data about drug prescriptions by hundreds of physicians. Is that lying around unorganised on the net? Tell me which algorithm you are going to use to predict how many XBOX365 are going to get sold next month by webcrawling??? You think supermarkets post their sales-figures to public webpages? Wallmart is said to have more data off-line than is available on the entire public section of the net. Now give me access to that.. But on the other hand; if you work for the sales-tax administration (in Europe) and all the big companies file their invoices weekly, that is also a good starting point...

mining online news stories for word connotations

[Cryofan]

I wrote a project in perl some years ago that would download online financial news stories and count the critical words and weigh their connotational weight, and compare that to the direction of the stock market. For example, if the words "stocks" and "down" started showing up a lot in sentences in online news stories, you might expect a downward trend.

I posted the preliminary code online in the perl newsgroup.

google "data mining" "news" "perl" etc

Realtime News Analysis

[Anonymous Coward]

A friend of mine has developed software that goes even further. It parses streaming news stories for good/bad news and executes orders before humans even finish reading. That advantage is enough to make this company a mint.

This is great

[RoboSpork]

Computers should be able to give a much more unbiased assessment of the economy than any person ever could. People are essentially incapable of interpreting economic data in a straightforward way, political agendas always seem to work their way into economists opinions about the economy. By using algorithms to do the analysis (and allowing market forces to refine those algorithms), we should be able to get a much better understanding of the REAL economy.

This is a good thing for mankind.

the rise of the machines...

[DeveloperAdvantage]

This is interesting stuff. I would like to learn more about the algorithms they use to analyze their data - the article has very few details. It is neat how systems like this are becoming favored over traditional human analysts (or at least reducing the need for people).

I remember back in grad school in the late 90s I worked on a major project to design an intelligent agent based system including the same functionality, but, in addition to pulling information off the internet, it could also take into account whatever other information could be gathered and interfaced into it (for example, there is also a lot of content on TV which could be fed into a system, in addition to the online data). It was a design project though and not implemented, perhaps I will need to resurrect it!

I do think the whole area of quantitative or at least semi-quantitative analysis of information, both textual and numerical, is going to explode over the next few years, driven by vast amounts of incredibly cheap computing power and bandwidth. Computer applications do amazing stuff right now, but five years from now truly "intelligent" applications will exist. The term "artificial intelligence" has fallen out of fashion, perhaps a sign of how common place these systems have now become.

As an example, our local phone company has a voice recognition system which actually works reasonably well, much, much better than anything 5-10 years ago. We are certainly making progress.

Wait a sec

[DrSbaitso]

For example, if the words "stocks" and "down" started showing up a lot in sentences in online news stories, you might expect a downward trend.

So you wrote a program that would read some stories that said the stock market was going down, and it told you the market was down? Did your program also see if weather news reports contained words like "rain" and "downpour" and hence "predict" rain?

Ties to Majestic 12?

[Anonymous Coward]

Does anyone know whether Majestic Reasearch has any connections to Majestic 12 (http://www.majestic12.co.uk/)? For those who don't know, Majestic 12 is a distributed search engine. The distributed part is in that they have a bunch of people donate CPU cycles and bandwidth to run a web crawler in a SETI at home fashion. Now i thought this was a good thing to join, because we kind of need some independent alternatives to google. But if it turns out i'm sponsoring some marketing firm, well... i'd feel pretty stupid.

Stock Memes - I posted this one year ago

[broward]

BEHOLD! The Power of the Meme!

http://www.realmeme.com/Main/miner/stock.jsp?start up=http://www.realmeme.com:80/Main/miner/investmen t/AMZNDejanews.png

Will it work?

[Arwing]

Nope, it won't, because even if it does, everyone will start using it and render it useless. There is only one trend in stock market that is backed up by statistics over long run and that is the stock market drifts upward overtime. My professor did a exmeripment using computer modeling, basically using a random number generator to decide if the stock market goes up or down, adding the 'upward drift' factor using historical data and comparing it to the actual data over last 75 yrs, and two data looks almost identical. I know it doesn't "prove" my point, but it does show that playing stock market short term is basically a flip of coin.

Several groups do this

[JimDog]

I once interviewed with a group in San Francisco that did stuff like this. They weren't clear about who they were working for, but I do remember some of the techniques they mentioned during the interview. Some of these were actually implemented, others were just ideas:

- An eBay crawler that could estimate the number of auctions and average selling price to predict whether eBay would make their earnings target or not. eBay quickly blacklisted their IP space, so they started using a bunch of open proxies they found.

- By analyzing client/server communication for the Sims Online, they discovered that each connection was assigned a sequentially incrementing connection ID number. By looking at the rate at which the connection ID numbers were increasing each time they logged in, they determined that the Sims Online wasn't going to be nearly as popular as Electronic Arts was forecasting.

- They talked about placing a camera somewhere in Union Square (in SF) to monitor the entrace to Tiffany's during the holiday shopping season, and doing image analysis to determine what percentage of shoppers left the store with a Tiffany's bag in hand.

- Monitoring wireless carriers' spectrum to determine what percentage of GSM/CDMA channels were in use for data vs. voice. The communication itself is encrypted of course, but you can still tell whether a channel is carrying voice or data. They wanted to determine if wireless carriers forecasts about revenue from data services were accurate.