Slashdot Mirror
Extreme Complexity of Scientific Data Driving New Math Techniques
An anonymous reader writes "According to Wired, 'Today's big data is noisy, unstructured, and dynamic rather than static. It may also be corrupted or incomplete. ... researchers need new mathematical tools in order to glean useful information from the data sets. "Either you need a more sophisticated way to translate it into vectors, or you need to come up with a more generalized way of analyzing it," [Mathematician Jesse Johnson] said. One such new math tool is described later: "... a mathematician at Stanford University, and his then-postdoc ... were fiddling with a badly mangled image on his computer ... They were trying to find a method for improving fuzzy images, such as the ones generated by MRIs when there is insufficient time to complete a scan. On a hunch, Candes applied an algorithm designed to clean up fuzzy images, expecting to see a slight improvement. What appeared on his computer screen instead was a perfectly rendered image. Candes compares the unlikeliness of the result to being given just the first three digits of a 10-digit bank account number, and correctly guessing the remaining seven digits. But it wasn't a fluke. The same thing happened when he applied the same technique to other incomplete images. The key to the technique's success is a concept known as sparsity, which usually denotes an image's complexity, or lack thereof. It's a mathematical version of Occam's razor: While there may be millions of possible reconstructions for a fuzzy, ill-defined image, the simplest (sparsest) version is probably the best fit. Out of this serendipitous discovery, compressed sensing was born.'"
You win the internets
...but I don't think I'd want my doctor working from a "fuzzy logic" MRI if I had (God forbid) a BRAIN TUMOR or something...
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
Ya, know, filter out all the noise and controversy, and get an output which is purely liberal computer geek rants all in unified, biased agreement. That would be amazing!
For fuck's sake.
These techniques of dealing with incomplete and unstructured data have existed for decades.
AI researches hyping absolutely everything about their field to get some funding is starting to get on my nerves.
Make assumptions
"They were trying to find a method for improving fuzzy images, such as the ones generated by MRIs when there is insufficient time to complete a scan. On a hunch, Candes applied an algorithm designed to clean up fuzzy images,[...]"
Wow! That would be the last thing I thought of in that situation...
Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
They were trying to find a method for improving fuzzy images, such as the ones generated by MRIs when there is insufficient time to complete a scan. On a hunch, Candes applied an algorithm designed to clean up fuzzy images, expecting to see a slight improvement.
I think I'm having difficulty comprehending these two sentences. This person was trying to accomplish a task, and 'on a hunch' used a tool specifically designed to accomplish that task. How is that a hunch? Isn't that basic reasoning? If I needed to drive a nail into a wall, I would pick up a claw hammer, because it's a tool designed specifically for that task. It wouldn't be a 'hunch,' and I certainly wouldn't be surprised to find that the nail went into the wall.
Fir__ P___
While there may be millions of possible reconstructions for a fuzzy, ill-defined image, the simplest (sparsest) version is probably the best fit."
Of the millions of possibilities, the sparsest is MOST likely. Perhaps it's twice as likely as any other possibility. That still means it's 99.999% likely to be wrong.
As for the MRI, that fuzzy part is probably noise that can be deleted, except when it's a tumor.
"
Not that that is new !! Math is hard !! And not just for girls !! No one can understand GPL and you expect them to understand MATH ?? No Way Hozay !!
Interpolate and extrapolate
Laughter is the Spackle of the Soul.
R...
B......
T..
L....
I like some of the more subtle details in the title and summary: new math "techniques", "researchers need new mathematical tools", etc.
I find it hard to believe that our sciences are driving the math fields, as mature and well-developed as the math community is. But it is true that existing knowledge and tools from mathematics drive huge advances in the sciences when they are brought to bear. The sad truth is that scientists just don't play terribly well with others (maybe no one does): interdisciplinary work is rare and difficult, and so we end up re-inventing the wheel over and over again. The reality is that the "wheel" being created by the biologist in order to interpret their data is a poor copy of the one already understood by the physicist across campus.
What can we do about this? I'm not sure, but I think it's safe to say that our greatest scientific advances in the next few decades will be the result of novel collaborations, and not novel math or (strictly speaking) novel science.
Does Slashdot ONLY promote junk science and maths. Frauds and cons? Yes, we all know about the concept of fractals, and how fractals can be used to create FAKE data that seems semantically pleasing to the Human eye. However, there is a world of difference between producing FAKE data that matches the pattern expectations of our brains, and thus looks correct, and REAL data that science (including medicine) requires.
I am reminded of a true story. In the early days of 'image enhancement', one city in the UK was regularly getting convictions for robbery from surveillance footage that was VERY unclear. They used a so-called specialist image processing company who took the footage, 'enhanced' it (just like you see in CSI) and proved in court the boys-in-blue had caught the guilty party. One problem. It later came to light that the police were giving the enhancement company a photograph of the suspect, and the enhancement company were merging a scan of this photograph with the so-called enhanced video frame. They CLAIMED that this was a legitimate mathematical technique.
Needless to say, with yet another proven case of forensic fraud in the UK 'justice' system, all previous convictions based on such 'evidence' were overturned. The UK police and prosecution bodies have a LONG history of faking forensic evidence- and using prosecution 'experts' spouting absolute nonsense fill with sciencey sounding terms to bamboozle a naive jury (or judge).
Filling in gaps in data DOES have a legitimate place in 'art', like when an old photograph or frame of a movie has a region (damaged or empty down to something like post-conversion 3D) that needs to be filled with a 'texture' that will look to the viewer like it belongs in shot. It has NO, repeat ***NO*** place in science.
Sadly 99.9% of scientists are VERY poor scientists with an extremely limited grasp of fundamental concepts of logic and mathematics. They can quote, repeat, and remember, but they cannot understand. Being the vast majority of ALL scientists, casual sheeple hear more NOISE (fake facts) from the crappy scientists than ever they hear true facts from the good ones. The pseudo science of EUGENICS, for instance, was proven to be total garbage by first class scientists in the UK and Europe at the beginning of the 20th Century, for instance, but was universally adopted by American scientists (who then directly influenced Hitler's scientists), and accepted as proven 'fact' by the population of the USA up through into the 1960s. Even today people like Bill Gates fund the same US Eugenic organisations that operated in the early 20th century, partnered with the Nazis, and pushed for forced sterilisation and medical experimentation on 'inferior' Humans.
Bill Gates continues the work of Eugenics via far more modern methods of societal manipulation of those Gates refers to as 'inferior'. Gates is behind the 'Common Core' curriculum dumbing down the US schools where most ordinary people send their kids, and Gates partnered with RUPERT MURDOCH of Fox News to create the inBloom full surveillance database of every child in the USA
Bill Gates created MSNBC, and you sheeple are constantly told that MSNBC is the very opposite to Fox News. Yet when Bill Gates partners with Murdoch to record intimate details about your child (including sexual development) and your family, and provides that information to any pervert who wants to target children with minimum risk of detection or punishment, you sheeple are supposed to NOT notice this unexplainable arrangement?
Gates even named the company overseeing the database inBloom- which is a pedophile codeword describing a child considered ripe for abuse.
Loading a MRI in to Photoshop and using a sharpening tool- novel concept!
love is just extroverted narcissism
The harder you have to dig.
who are these moderators who object to america first?!
These liberals never want to see anyone excel. The insist on equal outcomes not equal opportunity. Everyone has the chance to be first, several times a day here on Slashdot yet the conspiritors will not be happy until they crush your dreams under a hailstorm of moderation that would make lenin blush.
What is Big Data? They say it is when your problem grows faster than your resources.
Yet, since the 70's we have the concept of NP-Hard: again your problem grows faster than your resources. We have always had "Big Data."
.: Semper Absurda
Anyone who manage to invent a new methods for analyzing such a big data will get job...you guess where....
This! is the kind of article I joined slashdot to find out about.
I wish there was a way to mod actual articles +1 or -1 instead of just modding comments; or to at least toss the submitter a karma point or something.
Could someone point at some pictures he tried to clean up with these algorithms, and how they got cleaned up, and what existing picture manipulation tools use the same algorithms? And saying what the algorithms were would be good too.
4 years ago, Slashdot ran this exact same story http://science.slashdot.org/story/10/03/02/0242224/recovering-data-from-noise about Wired running this exact same story: http://www.wired.com/magazine/2010/02/ff_algorithm/all/1
Given the overall percentage of libertarians (1%?) and the overall percentage of liberals (48%?), clearly it isn't anywhere near "all libertarians". This proves that:
The liberals are completely wrong.
That's the only conclusion that can be drawn by anyone who can follow simple logic. People who can follow simple logic knew that already, though.
I'm KIDDING you hyper-sensisitive liberal weenie who is furiously clicking the "reply" button. Sometimes liberals are right, even Obama. Obama was right when he said the lack of a federal budget was a sign of no leadership from the president. Obama was right when he said if the economy isn't back on track in early 2012 he shouldn't be re-elected. Obama was right when he said it would be irresponsible of him to run for president because a presidential candidate should "know what you're doing". Liberals are very often right.
The whole article is just a sales job:
The first place to look when people make such claims is at their publications, neither Gunnar Carlsson nor Simon DeDeo have significant publications that show that their approach works on real data or standard test sets. The statements in the article that these kinds of approaches are new are also bogus (I don't know whether they are deceptive or ignorant).
Lastly, from a Stanford math professor, I would expect better citation statistics overall; I don't know what's going on there.
http://scholar.google.de/citations?user=nCGwiu0AAAAJ&hl=en
http://scholar.google.de/scholar?as_ylo=2009&q=author:%22gunnar+carlsson%22&hl=en&as_sdt=0,5
Have gnu, will travel.
Just plain lame. Nice marketing graphics though. What the hell is this crap doing on /.?
snakeoil salesmen
I applaud the work, seriously. But in some departments of mathematics, statisticians are referred to in the same breath as politicians and liars. I'm not calling the OP a liar, but generalization can lead to incorrect conclusions. Unless it's lupus of course.
Why?
what is Blade Runner
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
At the risk of repeating Lord Kelvin's folly: science is almost over, that's the root of the "data" problem. Data is so complex, because we exhausted simple systems, and we are trying to tackle irreducible systems.
It's a fallacy.
I do not believe in karma. "Funny"=-6. Do good and forbid evil. Yours, Oft-Offtopic Flamebaiting Troll.
If you love trusting these kinds of compression algorithms, I have a Xerox machine I'd like to introduce you to...
Cmoon If you say smething like: "On a hunch, Candes applied an algorithm designed to clean up fuzzy images, expecting to see a slight improvement. .. What appeared on his computer screen instead was a perfectly rendered image...", your readers (at least me) expect to see this image and to hear about hunch-algorithm.
Anyway, here (many clicks later) is original article - https://www.simonsfoundation.org/quanta/20131004-the-mathematical-shape-of-things-to-come/ (but still no pics)
turns out on the professor's web page Emmanuel Candes, there is a link to Some old talks that shows an example of the kind of transforms / cleanup they're talking about (they're lengthy PDFs, but worth skimming if you're curious about the kidns of images). Nothing like real world pictures; synthetic examples with some shapes (almost like something you could mock up with MS Paint), but the premise is rather interesting.
And I just saw this like on the Candes web page above: this does have some interesting more real-world pictures. Fill in the Blanks: Using Math to Turn Lo-Res Datasets Into Hi-Res Samples (wired, 2010)