Slashdot Mirror
GZipping Life Forms: Deflate Reveals Bare-Bones
An anonymous reader writes "To distinguish images derived from living vs. non-living sources, USC and NASA JPL researchers report today using the standard gzip compression utility. As a measure of overall pattern complexity, they find that the inherent pixel content of biologically generated fossils produces higher image compression ratios [more data redundancy], compared to their non-biological counterparts. The more the file shrinks, the more likely it is that a living process was involved. A test is live online here. This extends the simple, but powerful, uses of gzip to biogenic fossil detectors, in addition to spam cop filters, DNA sequence comparisons, digital camera image crunchers, etc. In nine months, the two Mars rovers will send back the first microscopic-scale images of Mars rocks, which should be amenable to some of these same techniques: thus gzipping is apparently pretty zippy."
Lifeforms seem to be built on patterns afterall. Patterns are easily compressible.
... therefore I am.
:-)
I'm not sure I should be flattered that the best way to tell a picture of me from a picture of a rock is that I have more redundant image data.
So when we compress the ultimate, super-duper intelligent life form we get a two byte file containing "42"
Trolling is a art,
that this has something to due with patterns and image continuity. If so (enlighten me!), then it would be a decent filtering tool, but reliability would be a major problem. Geological (or whatever) patterns could fool the algorithm. Finally, the most compressible image consists of monochrome - is it alive?
(Mods: the last line was a joke, intended to point out a particularly simple example of a problem - not a troll)
No more sniffing when i'm checking items in the refrigerator - is it 'alive' ? gzip is the answer!
Doesn't gzip only look for patterns in one dimension? Assuming they are using these for pictures, they are missing the boat on at least one more area of complexity!
then we will find out if he truly is the borg!
this sig steers like a cow. and i can prove it
OK, so if I have this right: Life is less random, and more predictible (more compressable)than non-life.
So that tells me that life contains less data then non-life.
Perhaps sophisticated life (human life?) contains even less data than non-sophisticated life. So the smarter we get, the more predictable we get, and the less data we contain.
Perhaps we will someday get smart enough to be totally compressed to one bit. In the time I thought about this concept, I think my gzip file got even more compressed. Hmm....
The Magic School Bus is true!
From excellent karma to terible karma with a single +5 funny post...
... if it could find life forms in my doom wad's?
Has anyone checked if bzip2 is better or worse in detecting biological products?
/usr/src/linux/kernel/sys.c, 24957 bytes uncompressed).
After all, they have quite different compression characteristics (on one hand, compression of a megabyte of zeroes is much better in bzip2, OTOH adding the same file on top of itself and then compressing gives much less additional compressed size with gzip than with bzip2 - tested with
The Tao of math: The numbers you can count are not the real numbers.
Although I'm certainly no compression expert, I think this makes sense. Many (most?) natural systems have fractal structures on some level so it only makes sense for them to compress better (ie: have more self-similar features) than systems which don't have this feature.
...
Then again, what do I know? Maybe something more immersed in this field can tell us whether there's a seed of truth to my ramblings
Greetings
--> R
Every one of us is incredibly redundant, and I don't just mean in our posts on slashdot!
Simply consider that you can have a reasonably good duplicate of yourself, with only the DNA contained in a single cell!
You may need most of your parts to be functional but, information-wise, it all comes down to 1 germ cell (say, a spermatozoid) and the aparatus needed to move it into proximity of another compatible germ cell ;)
This post can't be compressed.
at the comparison page attached to the article that lets you run the same test on images that the researchers tried. In a startling discovery that is sure to earn me a Nobel Prize for Physics, Chemistry, Biology and Marital Relations, I was told the following:
"Answer: Image 1 [the Mars image](1.43702451394759 % compression) has a higher complexity measure than image 2[the image of my wife] (0.773501341151519 % compression), and thus image 1 is more probably biogenic."
Not only does this prove that there was once life on Mars, but it also proves that my wife is some sort of robot. Further research will be undertaken pending receipt of my prize money.
Carousel is a lie!
.. thought of being gzipped is quite disturbing.
Mad Scientist: "Fire up the GZip Continueum Transfunctioner!"
Operator: "Okay, Boss"
*Bizzzttt*
"Never let the truth get in the way of a good story..."
One of the posters brings up an interesting point. Although meaningful data has more information than pure noise, it also has less than a blank signal. When you download pictures, regardless of the "meaning" they have to you, their compression can vary a considerable amount. And you've probably heard the statistic that the english language is 50 percent redundant. That figure may vary a bit too, but the point is that english's meaning to us is independent of its information content. And the probability that an image of a life form with more information will also have more "meaning" is probably just as uncertain.
Roughly, Kolmogorov Complexity is a measure of randomness - the measure is how long a computer program needs to be to reproduce data (pardon an oversimplification).
-Mark
The problem here is that your wife is wearing clothes. Clothes are man made.
If you send me a picture of your unclothed wife, I'll be happy to, uhm, test this theory.
It was pretty simple... Images over a certain size contained lightning, the others were mostly black, therefore smaller. Once I filtered it that way, manually filtering out the better images was easy.
BTW, if you want to be file name independent, you can useThis way, gzip doesn't see the file name, and therefore doesn't include it into the
The Tao of math: The numbers you can count are not the real numbers.
Read about it in _the_ book (http://www.cwi.nl/~paulv/kolmogorov.html) or check out the web site here (http://www.hutter1.de/kolmo.htm). For a more succint idea of the approach, these articles by one of the gurus on the topic (http://www.cs.ucsb.edu/~mli/focs.ps and http://www.cwi.nl/~paulv/papers/ecml97.ps).
"Consensus" in science is _always_ a political construct.
ahh, but the picture of your wife contains a lot of inanimate objects. I'm sure if you cropped the picture down to just her (or reasonably close) she would fare better in this comparison.
I compress to binary 0, therefore I am not.. :(
zip is a fine thing, but it's not a pattern-recognition program!
This is the loopiest thing I've heard of since Rosenblatt reported that his Perceptrons could distinguish between music composed by Bach and music composed in imitation of Bach.
Good heavens, any picture that's slightly out of focus will now be declared to be evidence of "biological processes."
I'm guessing that the researchers are not as nutty as they sound and that they've done more than is being reported, but still...
Reminds me of the researchers in the sixties who were publishing analyses of data that supposedly showed "biological clocks." It turned out that they were using smoothing algorithms that, basically, were filters that had a 24-hour peak in the frequency domain--so their analysis was creating the patterns they claimed to be detecting. A debunking article was published in Science in which another research used data from a random number table (the "unicorn" data) and showed that the same analysis techniques showed that the unicorn had a biological clock.
"How to Do Nothing," kids activities, back in print!
I myself have successfully used gzip for factoring large prime numbers, sorting the men from the boys, unblocking the kitchen sink and cracking safes. I'm currently trying to locate Osama Bin Laden by compressing Al Jazeera footage, but all I come up with are reports of Elvis sightings.
If I seem short sighted, it is because I stand on the shoulders of midgets
This whole thing is slightly dodgy, and I begin to wonder whether it was released a day early by mistake.
The big problem is the use of JPEG source images. Unless you've stuck it up to the maximum size on quality, then the jpeg artifacting (which is in effect repeating blocks of image data after transitions) will probably mask any hidden level of complexity in the images - the human brain is a much better tool at pattern recognition than most computer algorithms (especially those algorithms not designed for the task!).
Throw high-resolution bitmap files at it, and I'd be more persuaded that there is a genuine effect. Until then, I suspect it's more of a happy coincidence that the files they've thrown at it give results they are excited about.
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
doesnt bzip2 outperforms gzip ?
gzip might be preferable because it works more locally. It only keeps track of the last n bytes of data and does substitutions based on patterns seen in those n bytes.
bzip2 uses a markov predictor and the chain length is typically much longer than gzip uses, so the compression is less local. That's great if you're going for compression but for this work, it might be misleading.
That said, gzip doesn't know about image formats, so I wonder if these guys are getting some false positives on scanline wraps and other non-image data.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Wait wait wait you have a wife? Dude, this is Slashdot; are you sure you're not a misdirected user???
Karma whorin' since 1999
42 is one byte.
This is not surprising at all really. Gzip and other compression utilities can be used to get upper bound for real/nonredundant information content.
<p>I'm not sure if above is public knowledge, but I have used it as a one additional feature for certain pattern recognition tasks for a while.</p>
It was an interesting coffee break discussion with one of my professors, we were arguing if there is neat way to estimate the semantic content of a neural network after training it, I recall suggesting to compress the value of the weights of all layers and the less compressible the more this neural network is trained.
Could this be what you're after?
I envision a whole array of compression algorithms.
Each algorithm could be fine tuned for a paticular type of pattern.
Is that an elephant or a giraffe?
Does it compress better with the elephant algorithm or the giraffe algorithm?
It is a well-known fact that any compression algorithm will cause some files to increase in size when 'packed'. If this were not the case, then '42' would be the compressed version of some other file, say 'Wdugiu*6x9', which in turn would be the compressed version of DNA's DNA, which in turn might be the compressed version of the answer to life, the universe, and everything. Furthermore, everybodies DNA would compress down to the same file '42' (since we all contain the answer within ourselves, presumable mice would compress down to something else), which would mean we were all clones, which means that I am the Pope and you are CowboyNeal (and vice versa). QED.
I doubt this is very accurate for marking photos as hits or misses directly. This kind of thing may be useful more for detecting the lack of life rather than the presence of it. If compression rates are low, maybe you don't have to look at this photo so much. If they're high, maybe you want to examine it more closely. If you're dealing with truck loads of data and you're looking for a needle in a haystack, a mechanism for ruling out uninteresting data is invaluable.
That having been said, it sounds good in theory that 'organisms are highly patterned and therefore compress better', but then why would you use gzip? Why not take that theory and build something a little more adept at locating particular types of patterns you're interested in, or ruling out the ones you know are going to create false positives?
So, THAT having been said, I'm forced to wonder if somebody forgot that March has 31 days. Lord knows I can never keep track.
Interesting. For genome analysis Hidden Markov Models have been used in a lot of software.
Maybe if you could have an image recognition system do the Hard Machine Vision probelm of generating a schematic of the picture, and then fed the "leg bone is connected to the hip bone" kinda data into a HMM you could work out which fossils are ancient Cambrian crustations and which ones are Trogdor the Burninator.
I wonder if viruses (sorry - didn't RTFA) would compress like living life forms or if they would be more similar to nonliving.
Just a thought.
Mathematician, n.:
Someone who believes imaginary things appear right before your i's.
There are other techniques for measuring the level of chaos in a set of data, and they'd probably yield more consistent results than running the data through an algorithm meant for an entirely different purpose.
What about compress? Or even good old "compact". Ah I remember the days when we had 20% compression
and were glad of it and some of the old timers could have been confused with non living processes
even without the help of gzip anyway!
feed a business technology proposal through gzip
A similiar technique has been used by italian mathematicians to differentiate pages from various authors by using zip. A nature article can be found here. After a request from a dutch newspaper they were able to identify one author (Marek van der Jagt, which made his first debut) to be the same as an already well-known author (Arnon Grunberg).
Using CGI as the user hit the web page it took pictures at different shutter speeds. Working up from the slowest shutter speed the first JPG over 20K bytes was the right exposure and was shown on the page.
"This extends the simple, but powerful, uses of gzip to biogenic fossil detectors..."
The problem with gzip is that doesn't preserve data very well. Now tar, it preserves fossil data quite well.
Did a test run with some default images in windows xp. Windows XP's "Purple Flower.jpg" is apparently more "alive" than Windows Xp's "Tulips.jpg" but "Windows XP.jpg" is more alive than both of them!
--- If we knew half the things we shouldn't we'd stop wishing we knew it all