Slashdot Mirror
Digital Big Bang — 161 Exabytes In 2006
An anonymous reader tips us to an AP story on a recent study of how much data we are producing. IDC estimates that in 2006 we created, captured, and replicated 161 exabytes of digital information. The last time anyone tried to estimate global information volume, in 2003, researchers at UC Berkeley came up with 5 exabytes. (The current study tries to account for duplicating data — on the same assumptions as the 2003 study it would have come out at 40 exabytes.) By 2010, according to IDC, we will be producing far more data than we will have room to store, closing in on a zettabyte.
And half of that is porn...
"Banking establishments are more dangerous than standing armies." -Thomas Jefferson
Without Slashdot dupes.
I am a believer of momentum and curves.
The furry porn gets deleted first.
... times does the Library of Congress fit in that? Exabytes simply don't speak to me.
Alternatively, you can also answer in anime episodes, or mp3 files.
I left cat /dev/urandom
running
"You mortals are so obtuse." -Q
We won't be running out of space just like we didn't run out of food. New technology will allow us to store ever more data.
I imagine that a lot of this is web traffic logs. What if the US government really does force ISP's to keep records detailing the sites visited by their customers? Will my ISP rates increase to pay for all of that disk space?
So the sum total of data has increased by a factor of more than 30 since 2003? I knew Brent Spiner was putting on weight, but damn.
Web server log files with the history of people clicking around. My address stored by everybody I ever bought anything on line from. It's more an information land-fill than an information warehouse.
Leave the gun, take the cannoli -- Clemenza, The Godfather
What's really striking is how little data was available in machine-readable form well into the computer era. In the 1970s, the Stanford AI lab got a feed from the Associated Press wire, simply to get a source of machine-readable text for test purposes. There wasn't much out there.
In 1971, I visited Western Union's installation in Mawah, NH, which was mostly UNIVAC gear. (I worked at a UNIVAC site a few miles away, so I was over there to see how they did some things.) I was shown the primary Western Union international gateway, driven by a pair of real-time UNIVAC 494 computers. All Western Union message traffic between the US and Europe went through there. And the traffic volume was so small that the logging tape was just writing a block every few seconds. Of course, each message cost a few dollars to send; these were "international telegrams".
Sitting at a CRT terminal was a woman whose job it was to deal with mail bounces. About once a minute, a message would appear on her screen, and she'd correct the address if possible, using some directories she had handy, or return the message to the sender. Think about it. One person was manually handling all the e-mail bounces for all commercial US-Europe traffic. One person.
Is that the size of the next MS OS?
I'm sorry, how stupid is this?
"producing far more data than we will have room to store"
That's like saying, for the last 2 months, my profit has increased by 10%. If my profit keeps increasing at 10% per month, then pretty soon I'll own all the money in the world, and then I'll own more money than exists! Damn I must stop making money now before I destroy the world economy!!!
Who are these people who draw straight lines on growth curves? Why do people print the garbage they write and why weren't they the first against the wall after the dot com bust?
The only things that seem certain are death, taxes, entropy and stupid people...
"The weirdest thing about a mind, is that every answer that you find, is the basis of a brand new cliche" -
Data that cannot be stored will not be produced because all data that is produced must be stored. Data that is not stored (for however short a time) is not really produced.
Then again the past no longer exists anyway, the future doesn't exist yet and the present has no duration- so maybe the data never existed anyway. Maybe you don't exist?!?! Awe man maybe I *~/ disappears in a puff of logic*
----
Kudos to Augustine and Adams
Ok, so we generate some staggering amount of computerized data every year. This is one of those stories where I can't remember hearing about it before, but it really doesn't feel like "news".
My question is how much of this data is actually being used? I'm horrible for constantly downloading e-books, movies, software, OSes, and other stuff that I'm *intending* to do something with, but often don't get around to. I end up with gigabytes of "stuff" just sucking up disc space or wasting CDs. I burned a DivX copy of Matt Stone and Trey Parker's popular pre-South Park indie film "Orgazmo" in about 2001. I've since seen the film 2 or 3 times on TV. I STILL haven't watched the DivX version I have, and now I can't find the CD I put it on. I know I'm not the only one who does this either, as many of my friends are using up loads of storage space on files they've just been too busy to have a look at.
Right now I'm on a project digitizing patient files for a neurologist. We're going up to 10 years deep with files for over 18,000 patients. Most of this is *just* for legal purposes and nobody is EVER going to open and read the majority of these files. The doctor does electronic clinics where he consults the patient and adds new pages to their file, which will probably sit there undisturbed until the Ethernet Disk fails someday.
I think a more interesting story (although probably MUCH more difficult to research) would be "How much computerized data is never used beyond it's original creation on a given storage medium?"
Comment removed based on user account deletion
As interesting as the sheer volume is, most of it is garbage. I'd rather have 50 terabytes of organized and accurate information than 500 exabytes of data that isn't organized, and even if it were, it's accuracy is questionable at best. In essence, even if you manage to find what you want, the correctness of that information is likely to be very low.
I've long said we are not in the information age, we are in the data age. The information age will be when we've successfully organized all this crap we're storing/transmitting.
It's well and fine to have a statistic like 161 exabytes
...was that water just running down the pipe into the sewar or
of data, but what's the point. Is that data any more useful
to people than the selective data that was used to run the world
50, 60 or 100 years ago?
We as individuals are only capable of assimilating a limited amount
of information so most of those exabytes are just rolling around
like so many gears in an old machine. If they are minimally used or
never used they simply become a storage liability.
As an example, the internet has not made *better* doctors.
Even with all the latest information at thier finger tips
professionals are still only the sum of what they can
mentally absorb. Too much data, or wrong data (ie: wikipedia)
can lead to the same levels of inefficiency seen prior to
the 'information age'. What would a single doctor do with
160 exabytes of reading material, schedule it into the work day?
Also, if the amount of information is rated purely on bytes
but not in *useful content* the stats get skewed. Things like
movies and music should be ranked by the length of script
and/or notation. That would make the numbers much less than
160 exabytes.
Saying that the whole world produced 160 exabytes of information
is like saying the whole world used 50 billion tonnes of water.
did somebody actually drink it to sustain life?
Mechanistic stats are stupid.
So DR Evil, after emerging from his suspended animation, would demand a computer big enough to store 100 Megabytes of evil data.
Again, RTFS: "IDC estimates that in 2006 we created, captured, and replicated 161 exabytes of digital information."
If a NMEA lat-lon string gets spit out of the serial port of a GPS and there's nothing there to capture it, it is not part of their count. They're not counting bitrate on data generators and multiplying times bandwidth. They're counting discrete blocks of saved data. You cannot arrive at the latter from the former, just like you can't tell how much water is behind Hoover Dam on average during the year by measuring the average daily flow rate of the Colorado river and multiplying by 365.
If a job's not worth doing, it's not worth doing right.
50 Exabytes = (50)1024 petabytes = (50)1048576 terabytes:
RAID6 (24 Drives -2{Parity} -1{Hot Spare} = 21) 750GB, 13.48TB ZFS/Solaris:
93,345,048 750GB Hard Drives: $17,735,559,120
3,889,377 Areca ARC-1280ML: $4,317,208,470
1,944,689 Motherboards/Mem/CPU: $766,207,466
1,944,689 5U Rackmount Chassis's: $4,546,682,882
194,469 4 Post 50U Racks: $45,700,215
3,684 528-port 1Gbps Switches: $374,294,400
40 96-port 10Gbps Switches: $11,424,000
1,948,935 Network Cables: $2,020,812
? Assembly Robots/Misc. $111,000,000
Sub Total: $27,910,097,365
Tax/Shipping: $2,645,915,779
Grand Total: $30,556,013,144
$470 billion cheaper then the IRAQ war.