Slashdot Mirror
"Digital Universe" Enters the Zettabyte Era
miller60 writes "In 2010 the volume of digital information created and duplicated in a year will reach 1.2 zettabytes, according to new data from IDC and EMC. The annual Digital Universe report is an effort to visualize the enormous amount of data being generated by our increasingly digital lives. The report's big numbers — a zettabyte is roughly a million petabytes — pose interesting questions about how the IT community will store and manage this firehose of data. Perhaps the biggest challenge isn't how much data we're creating — it's all the copies of it. Seventy-five percent of all the data in the Digital Universe is a copy, according to IDC. See additional analysis from TG Daily, The Guardian, and Search Storage."
A zettabyte is more data than you generate during your whole lifetime. It's pointless to have so much space.
Hmm - thinking that I'd like to pop over to cnet or tigerdirect or fry's and pick up a zettabyte drive. I'm sure that's "more than enough storage" for all my digital files...
Are they on sale for $149.00 yet?
The Kai's Semi-Updated Website Thingy
"In 2010 the volume of digital information created and duplicated in a year will reach 1.2 zettabytes, according to new data from IDC and EMC. The annual Digital Universe report is an effort to visualize the enormous amount of data being generated by our increasingly digital lives. The report's big numbers -- a zettabyte is roughly a million petabytes -- pose interesting questions about how the IT community will store and manage this firehose of data. Perhaps the biggest challenge isn't how much data we're creating -- it's all the copies of it. Seventy-five percent of all the data in the Digital Universe is a copy, according to IDC."
With Z already in place and with [not so] recent inline deduplication feature, I think ZFS should do it.
"You talk to a kid these days and they have no idea what a kilobyte is. The speed things progress, we are going to need many words beyond zettabyte."
Sure they do. When their download speed drops under 100 kilobytes, they start sending "seed pls" comments to the Pirate Bay.
i know some people here frown down on those
Since this is EMC, let me tell you...
EMC loves to tell you to use RAID1. - 2 copies of your data
If it's important, you should use timefinder (snapshots), 1 more copy of the data.
If you want DR, then you should implement SRDF, 1 more copy of the data (this one is remote)
If you want to do data warehousing on what you just replicated, you run timefinder on the remote copy, 1 more copy.
So that makes it 5 copies of my data on disk.
Oh, and to protect myself from data corruption (or a deleted file) being replicated to all these copies, it's still recommended that I backup to tape/VTL/MAID.
Total of 6 copies of data. That is if I'm using dedup on my VTL or TSM (which stores versions of a given file). If i'm using a traditional (daily incrementals plus weekly fulls) I could have lots of duplications within my tape infrastructure.
Ever wonder why EMC stands for Endless Mirroring Company.
Only 75%? Considering that all DVD's are copies, all local caches are copies, I wouldn't be surprised if that number was much larger.
Also, cutting out all the copies would only reduce the problem to .3 zettabytes. For day-to-day IT purposes, that's about the same number.
The ______ Agenda
That we have all become good citizens, backing up all our data. I presume the data recovery firms are all panicking now that all their potebtial customers have backups of everything, and thus no longer need their services.
Not bad to have a global backup ratio of >1:1
Personally I use RAIM (Redundant Array of Instant Messages) to back up all my important notes and communications. It only works as long as all my friends log everything too, of course.
Love over Gold.
I was told about 10 years ago that "70% of the world's digital data is stored under MVS" which surprised me a bit, even then.
After some thought when you consider that almost all commercial transactions (banks, telcos etc) whould have been running MVS then it may have been true.
SETI and CERN and other large scientific endeavours are small fry in comparison.
" Perhaps the biggest challenge isn't how much data we're creating — it's all the copies of it. "
Why is that a challenge? Digital media is somewhat unique in that you can carefully craft media or information (reports, programs, videos much in the same way you'd carve a chair) but risk instantly and nearly irrecoverably lose it (much unlike a chair).
Copies of data are a safeguard by redundancy. A website gets taken offline, well good thing there is a mirror. My camera breaks or my hard drive disk fails, well good thing I have an external backup or copies on my DVDs.
while(1) attack(People.Sandy);
In the world of home storage, 75% is definitely way too low. The average personal desktop probably has 20 to 40 gigabytes of used storage, with far less than 1 gigabyte being original content. If they also back up this data, the fraction grows even lower.
Everything on their DVR is also not original.
Now, in the business world things are a bit different. Here you can expect the same 20 to 40 gigabytes of used storage on the median machine, but backed by a massive networked database of original uptime-critical content with at least a couple mirrors.
It is this second category that is clearly driving their estimate.
"His name was James Damore."
640K ought to be enough for anybody
Reply to That ||
Information storage was expensive.
At some point we started word processing on the desktop.
Information storage was still expensive.
Files were still small and the majority of the bytes in each file was information.
As time progressed and Microsoft Office has permeated the work area, the information content of each file hasn't changed much.
Each release seemed to take more space to store the same information.
Today, the portion of the file consumed in making it pretty though has gone through the roof.
We could always go back to just plain text files that were easy to search and cheap to store. Project Gutenberg has taken that approach for saving books. Keep it simple. Of course if we did that the productivity might go through the roof and layoffs might be high.
Guess pretty isn't so bad, even if it is part of the zettabyte problem.
If every piece of digital data doesn't have a copy made of it, it is one hardware failure away from non-existence. Most of the storage space used in businesses that I administrate is not for the original data, but for multiple backup copies. Copies are not a bad thing, in the business we call them redundancy.
# Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)
* Torvalds, Linus (1996-07-20).
How many libraries of congress is that?
A typical individual wouldn't have a whole lot of unique information to store in the first place.... Basically, a collection of photos and some video from a few vacation trips or holidays, and some handwritten notes .... Maybe some artistic works (a few original songs or paintings, or ?) if he/she was interested in such endeavors. Oh, and your tax records and resume. But let's face it. Most of us are FAR more of content consumers than creators. Content creation usually results in mass re-distribution of the original work, as others want to enjoy a copy of it.
I don't see any harm with this either, since duplication is the best way to protect against data loss. (When my parents were trying to trace their family history, they reached a dead-end because a library had burnt up in a fire that contained the only known records of some of the people they needed to research. With so much data going digital, on media that's practically EXPECTED to fail after less than 10 years of regular use? You better believe we need lots of duplicates out there!)
I have often spoken to a many engineers from gmail and hotmail....pertaining to the data they store and how they could improve their
systems by having pointers to emails instead of actual copies per storage account. if someone sends a joke email from one gmail account to all his friends which have 80% gmail accounts (so let's say, 25 in 30) you would only still have one copy of that joke email sitting on their server accessible by all who have that pointer reference, but in fact looks like they all have their own copy, unless of course someone modifies that email, which then becomes its own version , so splitting into many copies....
If you were to add that to many more things in life like cloud computing for storage purposes, across the board, you would have a lot less bandwidth being used, imagine being able to reference first by using lookup pointer tables, and instead of downloading, you already know someone on your own network has that same service pack for xxx application, so the download supposedly goes quicker because you are not streaming the packets from outside the network to get the info. etc...etc...
Anyways, I can't wait for our first zetta drive being sold at a futureshop near you, i will be the first in line to buy one!
In a few years i think.....lol
"a zettabyte is roughly a million petabytes" ...?
/me ducks
Surely that should read *exactly*
1.21 zettabytes? Great Scott!
This beautifully illustrates how idiotic the concept of "copy right" and IP in general is in the digital universe. When 75% of 1.2 zettabytes is mostly untracked copies of other information, just storing the licenses alone would be an impossible task.
How do you maintain a business model built on the exclusive right to copy information in world where everything is a infinitely copied and copyable? It's like trying to legislate and sell access to saltwater while floating on a raft in the middle of the pacific.
HD home movies and photographs are far more than 1gb
my karma will be here long after I'm gone
The comment about all the duplication of storage makes me think of the current pop culture obsession with hoarding.
I'd guess that all slashdotters have known someone who obsessively downloads music - to the point that they've got more music stored than they could possibly listen to.
kinda makes me wonder what it would take to store the non-digital universe.
It hurts my head.
1.1ZB of that is porn.
I can't believe none of you caught this... mod funny please, not interesting. Too bad there's no +1 Redundant.
- 1 zettabyte / 1.44MB floppy disk = approx 694,444,444,444,444 floppy disks.
- 694,444,444,444,444 * 3.5 inches per disk = 2,430,555,555,555,550 inches if you laid the floppies end to end.
- 2,430,555,555,555,550 inches / 63360 inches per mile = 38,361,040,965 miles
- 38,361,040,965 miles / 2.7 billion miles to pluto = approx 7 round trips to Pluto via floppy disk.
In conclusion: Don't kill NASA yet, President Obama. We've found a way to get to Pluto!.
No problem...
zfs set dedup=on tank
there... that should do the trick.
75% of everything I have on disk is a copy of something else, but unfortunately I usually have lost the copy somewhere in the process of moving, moving from one machine to the next, or trying to clear up disk space so I can download more stuff to leave on my disk.
One day I feel I'm ahead of the wheel / the next it's rolling over me / I can get back on / I can get back on
By definition. And since EMC is a storage company, they're almost certainly using the SI prefixes properly.
The author of the summary is, I think, confusing zettabytes and petabytes with the base-2 units, zebibytes and pebibytes. For all of the binary prefix haters, when you get up into these sizes the difference between base 2 and base 10 units is more than big enough to justify the effort to use the correct terms. The difference between one zebibyte and one zettabyte is over 180 exabytes.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Everything should be backed up ...... Right?!?!
Then I would expect 100% duplication, or higher should be the norm.
75% of 1.2ZB = 1E14 megabytes = 150 GigaCDs. At arround 10 tracks per CD costing each 22000USD, that makes it 34 petadollars in lost sales for the music industry !
The average person doesnt have the ability to take HD home movies because they dont even own the equipment necessary.
I've seen you project your geek lifestyle onto the world before.
"His name was James Damore."
"75% percent of all the data in the Digital Universe is a copy"
Slashdot Editors unavailable for comment.
The average personal desktop probably has 20 to 40 gigabytes of used storage
This is slashdot, i doubt many here qualify as average.
I myself fill my drives with 680GB of stuff, with 40-50GB easily being original.
O.o
Identical meaning everything down to the create date and last modified date.
a zettabyte isn't ROUGHLY a million petabytes - it is EXACTLY a million petabytes, that being its definition, and all.
Actually, information is useful stuff. The internet, and the world is saturated with useless stuff (data, noise). Also, the world's stuff is considerably smaller when de-duplicated. And then if you remove redundancies, and different ways of saying the same thing...
I am pretty sure that all the world's Information can be contained on a single petabyte. That would include all the world's literature, and all the newspapers, magazines, etc. If you include pictures, maybe significantly more.
Part of the Data problem is tons of data which has not been analyzed. Of the analysis could keep up with the data collection rates, then no problem. And no zetabytes needed, either.
wake up and hold your nose
"the big numbers pose interesting questions about how the IT community will store and manage this firehose of data" - just like the construction community will house and manage the firehose of over 6 billion people, so to speak.
Everybody takes care of their own bit(s) & backups; there is no single entity dealing with managing 1.2ZB.
Questions not so interesting. Move on.
Yesterday was the time to do it right. Are we having a REVOLUTION yet?
a zettabyte isn't ROUGHLY a million petabytes - it is EXACTLY a million petabytes, that being its definition, and all.
Depends on whether you're using SI & IEC units (Z/zeta- and Zi/zebi- for 1000^7 / 1024^7) or extrapolating JEDEC (which would come up with Z/zeta- for 1024^7).
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Are you kidding? I bought, at retail, a Sony cam, with 60GB internal drive and HD resolution 2 years ago for $900 from Costco. Most people buying new camcorders have ready access to HD quality cameras. Given the purchasing behavior of my cousin and brother-in-law; I think far more people cycle their personal tech more frequently than you suspect.
Is what percentage of it ISN'T backed up AND should be (which will be something less than 25% but much greater than 0%).
Seriously, what is up with the mods on this one?!?
"In 2010 the volume of digital information created and duplicated in a year will reach 1.2 zettabytes, according to new data from IDC and EMC. The annual Digital Universe report is an effort to visualize the enormous amount of data being generated by our increasingly digital lives. The report's big numbers -- a zettabyte is roughly a million petabytes -- pose interesting questions about how the IT community will store and manage this firehose of data. Perhaps the biggest challenge isn't how much data we're creating -- it's all the copies of it. Seventy-five percent of all the data in the Digital Universe is a copy, according to IDC."
HD home movies and photographs are copies, even if only one digital copy exists.
Morpheus, God of Dreams.
"In 2010 the volume of digital information created and duplicated in a year will reach 1.2 zettabytes, according to new data from IDC and EMC. The annual Digital Universe report is an effort to visualize the enormous amount of data being generated by our increasingly digital lives. The report's big numbers -- a zettabyte is roughly a million petabytes -- pose interesting questions about how the IT community will store and manage this firehose of data. Perhaps the biggest challenge isn't how much data we're creating -- it's all the copies of it. Seventy-five percent of all the data in the Digital Universe is a copy, according to IDC. "
So when this is duped in a few hours will that be irony or just funny?
Paying taxes to buy civilization is like paying a hooker to buy love.
http://www.enterprisestorageforum.com/ipstorage/news/article.php/3879726
a zettabyte is roughly a million petabytes
No, a zettabyte is exactly a million petabytes.
A zebibyte is roughly a million pebibytes.
Sheesh.
I have just a new terabytes sitting around the house plus back up on various computers, USB sticks here and there. I have also tried to find a nice unified storage device, say RAID 5, that can collect it all and remove all duplications (but keep a kinda of symbolic link to the one true copy and only fork it if it is changed). So if I have two files A and B in two different directories but they where the exact same file then they could be just stored a one file on the disk. If I decide to change A then I should be ask if I should update it or fork a copy of it, leaving B alone.
My dads probably got more music (excluding the stuff he downloads) than he will ever get round to listening to. He is almost 50 and I seriously doubt he will double that and his entire basement manages to fit 2x chair + HI-FI + 90% record collection + 75% CD collection. If I go stay the rest of the records and some CDs make it impossible to get into my bed without knocking a stack over.
This is all after getting rid of tapes/MD and quite a lot of the vinyl/CD sometime I think he hold up the entire UK music scene on his own.
My point is you don't need to download to have to much music just spend all your time in record shops.
Most Damage is done by people who are AWAKE
The number is meaningless, because "duplication" is arbitrary. Where do you draw the line? If duplication means "copying data from one place to another" then data is duplicated every time function parameters are pushed on the stack, every time memcpy() is called, every time something is loaded from disk into RAM. I could write a simple loop that copies a 32-bit quantity from EAX to EBX three billion times per second. If you include all that shit going on, I bet their number would be higher by a factor of a thousand or more. What the fuck is the use of this metric? How do you justify the arbitrary stopping point?
Where have you been, stuck in 2000? I said HD home movies and photographs. First, average joe can't hardly find a camera that's not digital since walmart only sells 2 cameras that still use film.
Second, you can't buy a camcorder that's not flash or hard disk. Yep, you heard me: Walmart only sells 2 camcorders that record directly to DVD, the other 150+ are all flash and hard drive. The camcorder offering the smallest hard drive capacity is still 80gb for a paltry sum of $350 and HD camcorders start at only $89.
So it is not I that is projecting my geek lifestyle on the world, it is you who is out of touch with modern consumer electronics.
my karma will be here long after I'm gone
Someone please Mod parent +Funny - Did no one get the irony?
Authority questions you. Return the favor.
In the world of home storage, 75% is definitely way too low. The average personal desktop probably has 20 to 40 gigabytes of used storage, with far less than 1 gigabyte being original content. If they also back up this data, the fraction grows even lower.
Also, it implies that there a lot of un-backed up data out there.
Are you kidding? I bought, at retail, a Sony cam, with 60GB internal drive and HD resolution 2 years ago for $900 from Costco.
Yeah.. thats means everyone has a Sony cam with a 60GB hard drive.. oh wait.. NO IT DOESNT
It means that you are a geek. Most people do not own *ANY* camcorder.
You heard me. Most people do not own a camcorder.
Let me repeat that one more time. Most people do not own any camcorder.
"His name was James Damore."
Average Joe does not even LOOK for a camera, let alone buy one.
You are projecting your own lifestyle onto others. The average person does not own a digital camera, and most of the ones that do are sporting one integrated into their pay-as-you-go $30 cell phone.
The average person does not have a smart phone. The average person does not have a camcorder. The average person does not have a digital camera. The average person doesnt even have a game console. They have a laptop which they send email with. Thats it. A laptop for email.
"His name was James Damore."
You know people who don't own a digital camera? Really? I don't know anyone who _doesn't_!
Anybody have a link to the torrent?
Jesus was all right but his disciples were thick and ordinary. -John Lennon
LOL he's joking, obviously. Average Joe does not have a digital camera, smartphone, camcorder, game console... all they own is a laptop for email
my karma will be here long after I'm gone
ha yeah it was pretty dead-pan humour (not that that's a bad thing...)
gmhowell, do you even have a CSC or CIS degree? That's a better question. Answer = No. What exactly then qualifies you as somekind of expert on the subject material of computing on this forums then?? Your talking??? LMAO, not.
gmhowell, do you even have a CSC or CIS degree? No. What exactly then qualifies you as somekind of expert on the subject of computers around here then?? Your talking??? LMAO, not. You're just another dime-a-dozen slashdot ne'er do well.
1.2 ZBy is about 1/8th of a gram mol of bits. With molecular memory, it all ought to fit on a few grammes of carbon.
No it means he bought a camcorder sometime in the last 5 years.
In plainer language it is one with 21 zeros. I personally didn't know petabyte off the top of my head.
Democracy Now! - your daily, uncensored, corporate-free
See subject and of course not. gmhowell = yet another "slashdot wannabe expert", albeit with no degrees in computer science or computer information systems.
I dunno, my anecdote agrees with the other guy. My dad had a digital camcorder and was looking into upgrading. This is the man who screw up a mostly locked down laptop he only used for web browsing and email (o.k. mostly instead of entirely locked down was my mistake.)
I have a 20 year old sub $10/hour employee with a mid-range blackberry, and the rest of them probably average $200 phones (a couple of geezers like me and two single moms bring that average way down).
My thinking is look at the prices and availability of them, that suggests a fairly high number of people have them, I'd guess (just that a guess) well over 20%, but probably not as high as 60%, say 1/4 to 1/3.
Mycroft
https://signup.leagueoflegends.com/?ref=4c3ed6600b6ea
Do you have a degree in CSC or CIS squiggleslash? No?? We thought not. You're by no means an expert to comment on anything in the art and sciences of computing then. The same things you note have happened to others that write good softwares, such as Dr. Mark Russinovich of Microsoft even and also Nir Sofer of Nirsoft, to name only a couple. Your lack of expertise is showing itself squiggleslash. Go get a degree and then get back to us, because until then? Your credibility as an expert is non-existent.
Do you have a degree in CSC or CIS gmhowell? No?? We thought not. You're by no means an expert to comment on anything in the art and sciences of computing then. The same things you note have happened to others that write good softwares, such as Dr. Mark Russinovich of Microsoft even and also Nir Sofer of Nirsoft, to name only a couple. Your lack of expertise is showing itself gmhowell. Go get a degree and then get back to us, because until then? Your credibility as an expert is non-existent!
This has been debunked multiple times already. Alex Peter Kowalski's code wasn't fleetingly described as malware, it is currently classified this way by multiple anti-malware organizations, who have placed it under that category for several years.
You want to talk about libel? What about pretending that Russinovich's software is comparable to that written by an author of software widely considered malware who promotes snake-oil anti-virus solutions.
And continue pretending that nobody's qualified to slap you down. Until you can show that any of us also writes tools considered malware, and "anti-virus guides" that, by your own admission, do not work (USE IT AND YOU STILL GET TWO VIRUSES A MONTH?!), you don't have a leg to stand on.