Slashdot Mirror

← Back to Stories (view on slashdot.org)

Extreme Complexity of Scientific Data Driving New Math Techniques

Posted by Soulskill on from the how-do-you-process-twelve-billion-data-points dept.

An anonymous reader writes "According to Wired, 'Today's big data is noisy, unstructured, and dynamic rather than static. It may also be corrupted or incomplete. ... researchers need new mathematical tools in order to glean useful information from the data sets. "Either you need a more sophisticated way to translate it into vectors, or you need to come up with a more generalized way of analyzing it," [Mathematician Jesse Johnson] said. One such new math tool is described later: "... a mathematician at Stanford University, and his then-postdoc ... were fiddling with a badly mangled image on his computer ... They were trying to find a method for improving fuzzy images, such as the ones generated by MRIs when there is insufficient time to complete a scan. On a hunch, Candes applied an algorithm designed to clean up fuzzy images, expecting to see a slight improvement. What appeared on his computer screen instead was a perfectly rendered image. Candes compares the unlikeliness of the result to being given just the first three digits of a 10-digit bank account number, and correctly guessing the remaining seven digits. But it wasn't a fluke. The same thing happened when he applied the same technique to other incomplete images. The key to the technique's success is a concept known as sparsity, which usually denotes an image's complexity, or lack thereof. It's a mathematical version of Occam's razor: While there may be millions of possible reconstructions for a fuzzy, ill-defined image, the simplest (sparsest) version is probably the best fit. Out of this serendipitous discovery, compressed sensing was born.'"

6 of 107 comments (clear)

  1. Amazing intuition by ZeroPly · · Score: 5, Funny

    "They were trying to find a method for improving fuzzy images, such as the ones generated by MRIs when there is insufficient time to complete a scan. On a hunch, Candes applied an algorithm designed to clean up fuzzy images,[...]"

    Wow! That would be the last thing I thought of in that situation...

    --
    Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
    1. Re:Amazing intuition by Anonymous Coward · · Score: 2, Funny

      They were trying to reach him to talk to him. Oh a hunch, the Nobel committee applied a phone designed to reach people.

    2. Re:Amazing intuition by Anonymous Coward · · Score: 4, Funny

      But it's even more amazing than that.

      The Nobel committee only had the first three digits of his phone (the area code), so they applied the same algorithm, and bam! Turns out it works just as well for phone numbers.

      They got him on the first ring too. But that part is just coincidence.

  2. Re:I dunno about you... by almitydave · · Score: 4, Funny

    Yeah, my doctor couldn't see enough detail in my head x-ray, so he used Photoshop's "content-aware fill" to fix it, and now apparently I need surgery to remove the 3rd half of my brain. I get to keep the 2 extra eyeballs, though.

    (actually, I really really want to see that applied to medical x-rays)

    --
    my, your, his/her/its, our, your, their
    I'm, you're, he's/she's/it's, we're, you're, they're
  3. Re:I dunno about you... by lgw · · Score: 4, Funny

    OF course it works. "Zoom! Enhance!" If TV hasn't taught me that "enhance" works reliably, then TV has taught me nothing.

    --
    Socialism: a lie told by totalitarians and believed by fools.
  4. Old news on old news by key45 · · Score: 3, Funny

    4 years ago, Slashdot ran this exact same story http://science.slashdot.org/story/10/03/02/0242224/recovering-data-from-noise about Wired running this exact same story: http://www.wired.com/magazine/2010/02/ff_algorithm/all/1