Internet Data Mining for Investment Analysis

CaroKann writes "Reuters is reporting on a Wall Street investment research company, Majestic Research, that is using web crawling techniques to track business performance. Instead of attempting to estimate business conditions by talking to company management, or pounding the pavement visiting stores, this company uses data mining systems to collect real-time sales data and other information on companies that have a web presence. Using this data, Majestic attempts to estimate company earnings more accurately than traditional research outfits."

Comments

Traditional Wall Street Research?

[eldavojohn]

But the New York-based research firm is winning converts among hedge funds who say its brand of Web-based quantitative analysis can be more accurate than traditional Wall Street research forecasts.

Possibly because "traditional Wall Street research" involves reading tea leaves and throwing down chicken bones while watching Alan Greenspan do a rain dance to the gods in hopes that our economy will pick up.

Economics and future fiscal predictions are completely theoretical. There are just too many variables involved, folks.

Cue the web spam...

[Rob T Firefly]

We can expect yet another huge rise in fake blogs, fake product reviews on Amazon and such, and paid shills in chats and message boards. Swell.

My data mining results

[CaptainFork]

based on manually mining (eg reading) Slashdot I determine a spike in Majestic's share price about now...

I call Bull

[spectrokid]

TFA mentions data about drug prescriptions by hundreds of physicians. Is that lying around unorganised on the net? Tell me which algorithm you are going to use to predict how many XBOX365 are going to get sold next month by webcrawling??? You think supermarkets post their sales-figures to public webpages? Wallmart is said to have more data off-line than is available on the entire public section of the net. Now give me access to that.. But on the other hand; if you work for the sales-tax administration (in Europe) and all the big companies file their invoices weekly, that is also a good starting point...

mining online news stories for word connotations

[Cryofan]

I wrote a project in perl some years ago that would download online financial news stories and count the critical words and weigh their connotational weight, and compare that to the direction of the stock market. For example, if the words "stocks" and "down" started showing up a lot in sentences in online news stories, you might expect a downward trend.

I posted the preliminary code online in the perl newsgroup.

google "data mining" "news" "perl" etc