Slashdot Mirror

← Back to Stories (view on slashdot.org)

Why Is Data Mining Still A Frontier?

Posted by ryuzaki0 on from the let's-get-together dept.

bbsguru writes "How much do we know that we still don't know? A story in The Register points out that little has changed since Francis Bacon proposed combining knowledge to learn new things 400 years ago, despite all the computer power we now have. Scientific (and other) data is still housed in unrelated collections, waiting for some enterprising Relational Database Programmer to unlock the keys to understanding. Is RDBMS still a Brave New Frontier, or will Google make the art obsolete once they finish indexing everything?"

4 of 223 comments (clear)

  1. Shot in the dark: by Spazntwich · · Score: 5, Insightful

    Either
    a) There's not enough money in it to make it worthwhile

    or

    b) It doesn't work.

    1. Re:Shot in the dark: by Disavian · · Score: 5, Insightful

      How about

      c) our ability to produce data far outstrips our ability and/or willingness to analzye it

    2. Re:Shot in the dark: by Coryoth · · Score: 5, Informative

      a lot of entities public and private are throwing a lot of money at data mining research, reasonably expecting a big payoff, and sometimes it gets very good results indeed. The basic problem is that, as with any worthwhile CS question, doing it well is hard. It is very easy to come up with false connections between data. Sorting the wheat from the chaff in any kind of automated or even semi-automated fashion, OTOH, is an enormous challenge.

      I would suggest that, in practice, the real difficulty is that the problems that need to really be solved for data mining to be as effective as some people seem to wish it was are, when you actually get down to it, issues of pure mathematics. Research in pure mathematics (and pure CS which is awfully similar really) is just hard. Pretending that this is a new and growing field is actually somewhat of a lie. It's avery very old field which people have been working on for a very long time, to the point where the problems that remain to be solved are incredibly difficult. What is new is someone other than pure mathematicians taking much interest in these problems. Do a search for "non linear manifold learning" on Google and you'll see what I mean.

      Jedidiah.

      --
      Craft Beer Programming T-shirts
  2. I tell you why (from a bioinformatics viewpoint) by Neil+Blender · · Score: 5, Insightful

    Programmers have no idea of context. Biologists have no idea about programming. It is very hard to mix the two. You can be the shit-hottest dba in the world but if you have no relevant (deep) biology background you are guaranteed to produce crap. Almost every piece of biological software is a POS because of this.