Slashdot Mirror
Finding a Needle in a Haystack of Data
Roland Piquepaille writes "Finding useful information in oceans of data is an increasingly complex problem in many scientific areas. This is why researchers from Case Western Reserve University (CWRU) have created new statistical techniques to isolate useful signals buried in large datasets coming from particle physics experiments, such as the ones run in a particle collider. But their method could also be applied to a broad range of applications, like discovering a new galaxy, monitoring transactions for fraud or identifying the carrier of a virulent disease among millions of people." Case Western has also provided a link to the original paper. [PDF Warning]
82.67% of all statistics are made up anyway...
I can't even find my keys some days.
A good strategy for finding useful information in oceans of data is to reduce the data set by sloughing off large chunks of irrelevant data. For example, if you want to find useful info in /., you would want to start by excluding all stories submitted by Roland Piquepaille.
"But their method could also be applied to a broad range of applications, like discovering a new galaxy, monitoring transactions for fraud or identifying the carrier of a virulent disease among millions of people."
When asked about more advanced applications for the technology, researchers replied it will probably be "quite a while" before the technology could be used for extremely high noise environments. Said one, "I mean, it's going to be a long time before we're up to finding finding useful comments on Slashdot or something."
Sounds like they've been watching Numb3rs ;-)
The Case team discovered a technique that is built on the principle of comparing a set of summary characteristics for any sub region of the observations with the background variation. From these characteristics, attempts are made to find small regions that appear significantly different from the background--a difference that cannot simply be attributed to random chance
So, basically its the one search engine that can only find the words "horny teen nekkid" if it is NOT on a pr0n-page. I can see uses for that. Not for me, but I'm sure SOMEONE is interested in finding other kinds of pages once in a while.
Someone asked me to give ten different ways to find a needle in a haystack, these are my thoughts:
1) INDUSTRIAL MAGNENT
2) BLIND LUCK
3) BURN THE HAY, PICK UP THE NEEDLE
4) STATISTICAL ANALYSIS (SINCE NEEDLES IN HAYSTACKS ARE NOT PLACED AT RANDOM, THEY ARE SUBJECT TO REGRESSION ANALYSIS)
5) OFFSHORE TO CHINA WHERE LABOR IS CHEAPER, SEARCH THE HAY WITH 10000 OF WORKERS.
6) WAIT YEARS UNTIL THE HAY DECAYS, PICK UP THE NEEDLE
7) SPREADOUT THE HAY, HIRE BAREFOOT HAY WALKERS
8) TAKE ALL THE HAY, PUT IN A POOL OF WATER - HAY WILL FLOAT, AND NEEDLE WILL SINK
9) LET COWS EAT THE HAY, X-RAY ALL THE COWS!
10) TRIAL AND ERROR - ONE PERSON
"This isn't a study in computer science, its a study in human behavior"
Does Google have the technology to do this kind of scientific searches yet?
;-)
It's only in Beta thus it's not useful
Actually, its not two paper clips together. It's a fat man holding a surf board. Look for yourself
"It just refused to load for me."
Maybe your interest in the story was deemed statistically insignificant.
Beauty is in the eye of the beerholder.
But can it find potential girlfriends for Slashdotters?
Wow. There really are't any out there. Check it out on google yourselves.
The same results come back in images, groups, news, etc. Man. What a sad bunch.
Where have you been?
Don't you know the editors are in cahoots with the the Beatles Beatles guy now?
Please, try to keep up with the conspiracy theories, mkay? Jeez!
1) INDUSTRIAL MAGNET
DBAs everwhere are cringing and covering their data.
"What does God use to tag a galaxy with though?"
Are you telling us there's such a thing as Intelligent Tagging?
An article posted by Roland Piquepaille with no links back to his site???
WTF? Roland? You feeling OK?
No folly is more costly than the folly of intolerant idealism. - Winston Churchill
Here's a couple TB of data. Find me all the top quark candidates by tomorrow.
Bah! Not even in Froogle... ;)
Did you see the pool? They flipped the bitch!