Slashdot Mirror

Interview with Rayid Ghani, Chief Scientist Obama 2012 Campaign: Q. How did you use facebook and other social networks as part of modeling? A. We used facebook for a few different purposes: We used facebook to reach young voters who were hard to reach using traditional channels such as phone, direct mail, and door-to-door canvassing. We built models using data from users who authorized our facebook app that allowed us to ask our supporters to contact their friends for specific reasons (voter registration, volunteering, going to vote, etc.). Our hypothesis was that getting their friends to ask them was more effective than us asking them directly by broadcasting on our facebook page. We also used facebook to determine people's interest and send them messages that were relevant to them and hence increase their likelihood of taking action.

While still not on par with Tufte's graphic quality, there are quite a few packages that attempt to produce new displays with high information density for a variety of contexts. See for instance Discovery and some other tools of the same kind: KDNuggets

Excel is very widely used data mining tool (see: http://www.kdnuggets.com/polls/2006/data_mining_an alytic_tools.htm).
It is used both to analyze smaller sets (less than 100 million rows :-) and preliminary reports from larger sets. In practice, the data is seldomly in the DB when it is being mined because that is just so darn slow (unless you have bought a multi million dollar systems - in which case you absolutely have to use is even though it sucks as otherwise somebody has made a stupid desision to buy it...). How long does it take to open small (60 000 rows - to be able to do it today) data set in Excel and then for example do a pivot to it to see all the combinations of the values? Not time at all. Now, import that to your relational database and do the same report from there.

The 65k rows limit was very limiting. The first time I tried OpenOffice I immediately checked the limit there... my surprise was huge when the limit was 32 000 rows (not even 2^15 or 2^16 as in Excel). And as Excel 2000 otherwise beats OO in usability, I didn't look at OO any more.

So, get rid of all the limits! Why should it not be more than about million rows? I can see no reason. (Yeah, if the computer is too small then it doesn't fit into the memory, but that's a different thing.)

"FOSDEM: The Mozilla Foundation's partnership with Google has kept it afloat for the past few months, and is now allowing it to hire more staff"

Seems to suggest that the google deal came through roughly at the same time. however that headline was misleading to suggest google was keeping Mozilla foundation afloat. see

As long as google sticks to gathering information from me only when i use google I am happy enough, it's when you get into alexa type activitys i am not.

Although Alexa does go hand in hand with the internet archive. (damn conflicts with something I do like)

If your interested in Datamining in general http://www.kdnuggets.com/dmcourse/other_lectures/i ntro-to-data-mining-notes.html or "knowledge discovery" then that link looks interesting

I like google but they are slipping wtf are all the landing sites doing high in the rankings. you know if google could derank hits based on how quickly someone went back to google after following a duff link it should progressively improve

Ok.. maybe I'm missing something here.. Are these things sold with a mandatory music-download-service subscription or something like that, in order to subsidize the price of the hardware? Or what?

A quick google search reveals that a professional camera shop/mail order company sells a 4 GByte microdrive for $370-$500, while the MP3 player is expected to retail for $250. The difference is due to market pricing, as professional photographers are used to paying thousands for a professional camera, while the average consumer is used to paying hundreds for a portable digital product. The casing of the product hides the fact that both products used the same core component. Eventually the market will realize this and take action- perhaps choosing MP3 players with removable microdrives

This isn't any different from Amazon's price discrimination for books.

There are "clustering" techniques which attempt to identify similar bunches of data, without respect to any pre-determined bins, but the are not as useful for programmatically dealing with information. This is simply because you don't know what the clusters will contain, so you cannot make assumptions about what you will want to do with each cluster.

Classification systems are used when you WANT to fit things into one of a number of bins that you already have decided what to do with (e.g. SPAM - delete, From Mistress - show now, From Boss - file for later, From Debt collector - return "Deceased", etc.) Bayesian filtering is simply one form of classification.

For more information and ideas, check out KD Nuggets

Nice work on the newsbot, BTW.

The larger pattern this fits into is one that's seen lots of research recently: we don't like other people getting what we perceive to be "better" deals, even if their deals come at no marginal cost to us. The social disapproval at these "cheaters" who don't "pay their share" is pretty strong.

So consumers get pissed when Amazon tries differential pricing, and people will moan about how they should have bought a Powerbook this month, and not last month. Combine this with the "all MP3s and software should be free" crowd, and that will generate a lot of /. posts!

There is a field dealing with this. It's called Knowledge Discovery in Databases (KDD). It's been around for a few years now. Go here for a slightly more technical overview. The posted article is aimed more toward the business people rather than the technical people.