Slashdot Mirror
"Digital Universe" Enters the Zettabyte Era
miller60 writes "In 2010 the volume of digital information created and duplicated in a year will reach 1.2 zettabytes, according to new data from IDC and EMC. The annual Digital Universe report is an effort to visualize the enormous amount of data being generated by our increasingly digital lives. The report's big numbers — a zettabyte is roughly a million petabytes — pose interesting questions about how the IT community will store and manage this firehose of data. Perhaps the biggest challenge isn't how much data we're creating — it's all the copies of it. Seventy-five percent of all the data in the Digital Universe is a copy, according to IDC. See additional analysis from TG Daily, The Guardian, and Search Storage."
Hmm - thinking that I'd like to pop over to cnet or tigerdirect or fry's and pick up a zettabyte drive. I'm sure that's "more than enough storage" for all my digital files...
Are they on sale for $149.00 yet?
The Kai's Semi-Updated Website Thingy
Maybe you won't but then you are not CERN or the Hadron Collider
America, Home of the Brave.
"In 2010 the volume of digital information created and duplicated in a year will reach 1.2 zettabytes, according to new data from IDC and EMC. The annual Digital Universe report is an effort to visualize the enormous amount of data being generated by our increasingly digital lives. The report's big numbers -- a zettabyte is roughly a million petabytes -- pose interesting questions about how the IT community will store and manage this firehose of data. Perhaps the biggest challenge isn't how much data we're creating -- it's all the copies of it. Seventy-five percent of all the data in the Digital Universe is a copy, according to IDC."
A zettabyte is more data than you generate during your whole lifetime. It's pointless to have so much space.
For those wondering: 1,000,000,000,000,000,000,000-ish, or some 10^21
I think therefore I am... a Linux geek.
A zettabyte is more data than you generate during your whole lifetime. It's pointless to have so much space.
Speak for yourself.
The Future of Human Evolution: Autonomy
With Z already in place and with [not so] recent inline deduplication feature, I think ZFS should do it.
Since this is EMC, let me tell you...
EMC loves to tell you to use RAID1. - 2 copies of your data
If it's important, you should use timefinder (snapshots), 1 more copy of the data.
If you want DR, then you should implement SRDF, 1 more copy of the data (this one is remote)
If you want to do data warehousing on what you just replicated, you run timefinder on the remote copy, 1 more copy.
So that makes it 5 copies of my data on disk.
Oh, and to protect myself from data corruption (or a deleted file) being replicated to all these copies, it's still recommended that I backup to tape/VTL/MAID.
Total of 6 copies of data. That is if I'm using dedup on my VTL or TSM (which stores versions of a given file). If i'm using a traditional (daily incrementals plus weekly fulls) I could have lots of duplications within my tape infrastructure.
Ever wonder why EMC stands for Endless Mirroring Company.
Only 75%? Considering that all DVD's are copies, all local caches are copies, I wouldn't be surprised if that number was much larger.
Also, cutting out all the copies would only reduce the problem to .3 zettabytes. For day-to-day IT purposes, that's about the same number.
The ______ Agenda
That we have all become good citizens, backing up all our data. I presume the data recovery firms are all panicking now that all their potebtial customers have backups of everything, and thus no longer need their services.
Not bad to have a global backup ratio of >1:1
Personally I use RAIM (Redundant Array of Instant Messages) to back up all my important notes and communications. It only works as long as all my friends log everything too, of course.
Love over Gold.
I was told about 10 years ago that "70% of the world's digital data is stored under MVS" which surprised me a bit, even then.
After some thought when you consider that almost all commercial transactions (banks, telcos etc) whould have been running MVS then it may have been true.
SETI and CERN and other large scientific endeavours are small fry in comparison.
" Perhaps the biggest challenge isn't how much data we're creating — it's all the copies of it. "
Why is that a challenge? Digital media is somewhat unique in that you can carefully craft media or information (reports, programs, videos much in the same way you'd carve a chair) but risk instantly and nearly irrecoverably lose it (much unlike a chair).
Copies of data are a safeguard by redundancy. A website gets taken offline, well good thing there is a mirror. My camera breaks or my hard drive disk fails, well good thing I have an external backup or copies on my DVDs.
while(1) attack(People.Sandy);
In the world of home storage, 75% is definitely way too low. The average personal desktop probably has 20 to 40 gigabytes of used storage, with far less than 1 gigabyte being original content. If they also back up this data, the fraction grows even lower.
Everything on their DVR is also not original.
Now, in the business world things are a bit different. Here you can expect the same 20 to 40 gigabytes of used storage on the median machine, but backed by a massive networked database of original uptime-critical content with at least a couple mirrors.
It is this second category that is clearly driving their estimate.
"His name was James Damore."
Yes, 640 petabytes should to be enough for anybody.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
640K ought to be enough for anybody
Reply to That ||
Didnt it say that they are talking about created data (and assorted copies of it)... not how much data we can store
Information storage was expensive.
At some point we started word processing on the desktop.
Information storage was still expensive.
Files were still small and the majority of the bytes in each file was information.
As time progressed and Microsoft Office has permeated the work area, the information content of each file hasn't changed much.
Each release seemed to take more space to store the same information.
Today, the portion of the file consumed in making it pretty though has gone through the roof.
We could always go back to just plain text files that were easy to search and cheap to store. Project Gutenberg has taken that approach for saving books. Keep it simple. Of course if we did that the productivity might go through the roof and layoffs might be high.
Guess pretty isn't so bad, even if it is part of the zettabyte problem.
If every piece of digital data doesn't have a copy made of it, it is one hardware failure away from non-existence. Most of the storage space used in businesses that I administrate is not for the original data, but for multiple backup copies. Copies are not a bad thing, in the business we call them redundancy.
# Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)
* Torvalds, Linus (1996-07-20).
A zettabyte is more data than you generate during your whole lifetime. It's pointless to have so much space.
Ooooh, it sounds like somebody's pr0n collection seems a bit inadequate today.
John
How many libraries of congress is that?
I have 10^21 + 1 desktop shortcuts
www.RacquetUp.org - Helping Detroit Youth
My Hot Fresh Brewed Coffee shot out my nose. It still burns a little, but that joke was totally worth it.
Glad I could assist in the steam cleaning of your nasal cavities.
Just another service we offer in addition to sarcasm.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
That's everyone, not per person.
That's ~1/7th TB per person
Man am I over quota.
Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).
A typical individual wouldn't have a whole lot of unique information to store in the first place.... Basically, a collection of photos and some video from a few vacation trips or holidays, and some handwritten notes .... Maybe some artistic works (a few original songs or paintings, or ?) if he/she was interested in such endeavors. Oh, and your tax records and resume. But let's face it. Most of us are FAR more of content consumers than creators. Content creation usually results in mass re-distribution of the original work, as others want to enjoy a copy of it.
I don't see any harm with this either, since duplication is the best way to protect against data loss. (When my parents were trying to trace their family history, they reached a dead-end because a library had burnt up in a fire that contained the only known records of some of the people they needed to research. With so much data going digital, on media that's practically EXPECTED to fail after less than 10 years of regular use? You better believe we need lots of duplicates out there!)
I have often spoken to a many engineers from gmail and hotmail....pertaining to the data they store and how they could improve their
systems by having pointers to emails instead of actual copies per storage account. if someone sends a joke email from one gmail account to all his friends which have 80% gmail accounts (so let's say, 25 in 30) you would only still have one copy of that joke email sitting on their server accessible by all who have that pointer reference, but in fact looks like they all have their own copy, unless of course someone modifies that email, which then becomes its own version , so splitting into many copies....
If you were to add that to many more things in life like cloud computing for storage purposes, across the board, you would have a lot less bandwidth being used, imagine being able to reference first by using lookup pointer tables, and instead of downloading, you already know someone on your own network has that same service pack for xxx application, so the download supposedly goes quicker because you are not streaming the packets from outside the network to get the info. etc...etc...
Anyways, I can't wait for our first zetta drive being sold at a futureshop near you, i will be the first in line to buy one!
In a few years i think.....lol
Yes, or roughly a million petabyte, where a million is roughly 10^6 and peta is roughly 10^15.
I'll make roughly one post about this matter.
Don't you mean
Cwm, fjord-bank glyphs vext quiz
1.21 zettabytes? Great Scott!
OK, I laughed when I read that, and then I wondered: How much of that would be porn? NOBODY needs a porn collection that big. And I considered making a Library of Congress type comparison, but saying the LoC would be a single hair...... it got creepy. Can we just, as a collective, hit the delete key a couple dozen more times a day? And keep the porn down to under a terabyte?
This beautifully illustrates how idiotic the concept of "copy right" and IP in general is in the digital universe. When 75% of 1.2 zettabytes is mostly untracked copies of other information, just storing the licenses alone would be an impossible task.
How do you maintain a business model built on the exclusive right to copy information in world where everything is a infinitely copied and copyable? It's like trying to legislate and sell access to saltwater while floating on a raft in the middle of the pacific.
HD home movies and photographs are far more than 1gb
my karma will be here long after I'm gone
The comment about all the duplication of storage makes me think of the current pop culture obsession with hoarding.
I'd guess that all slashdotters have known someone who obsessively downloads music - to the point that they've got more music stored than they could possibly listen to.
- 1 zettabyte / 1.44MB floppy disk = approx 694,444,444,444,444 floppy disks.
- 694,444,444,444,444 * 3.5 inches per disk = 2,430,555,555,555,550 inches if you laid the floppies end to end.
- 2,430,555,555,555,550 inches / 63360 inches per mile = 38,361,040,965 miles
- 38,361,040,965 miles / 2.7 billion miles to pluto = approx 7 round trips to Pluto via floppy disk.
In conclusion: Don't kill NASA yet, President Obama. We've found a way to get to Pluto!.
No problem...
zfs set dedup=on tank
there... that should do the trick.
75% of everything I have on disk is a copy of something else, but unfortunately I usually have lost the copy somewhere in the process of moving, moving from one machine to the next, or trying to clear up disk space so I can download more stuff to leave on my disk.
One day I feel I'm ahead of the wheel / the next it's rolling over me / I can get back on / I can get back on
... and lolcats
75% of 1.2ZB = 1E14 megabytes = 150 GigaCDs. At arround 10 tracks per CD costing each 22000USD, that makes it 34 petadollars in lost sales for the music industry !
The average person doesnt have the ability to take HD home movies because they dont even own the equipment necessary.
I've seen you project your geek lifestyle onto the world before.
"His name was James Damore."
Identical meaning everything down to the create date and last modified date.
Actually, information is useful stuff. The internet, and the world is saturated with useless stuff (data, noise). Also, the world's stuff is considerably smaller when de-duplicated. And then if you remove redundancies, and different ways of saying the same thing...
I am pretty sure that all the world's Information can be contained on a single petabyte. That would include all the world's literature, and all the newspapers, magazines, etc. If you include pictures, maybe significantly more.
Part of the Data problem is tons of data which has not been analyzed. Of the analysis could keep up with the data collection rates, then no problem. And no zetabytes needed, either.
wake up and hold your nose
No, what I meant specifically was "(Score:5, Funny)".
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
"the big numbers pose interesting questions about how the IT community will store and manage this firehose of data" - just like the construction community will house and manage the firehose of over 6 billion people, so to speak.
Everybody takes care of their own bit(s) & backups; there is no single entity dealing with managing 1.2ZB.
Questions not so interesting. Move on.
Yesterday was the time to do it right. Are we having a REVOLUTION yet?
a zettabyte isn't ROUGHLY a million petabytes - it is EXACTLY a million petabytes, that being its definition, and all.
Depends on whether you're using SI & IEC units (Z/zeta- and Zi/zebi- for 1000^7 / 1024^7) or extrapolating JEDEC (which would come up with Z/zeta- for 1024^7).
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Is what percentage of it ISN'T backed up AND should be (which will be something less than 25% but much greater than 0%).
Ah, so you've finally taken my advice and started cleaning those up. Thanks.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
HD home movies and photographs are copies, even if only one digital copy exists.
Morpheus, God of Dreams.
NOBODY needs a porn collection that big.
You're forgetting backups.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
So when this is duped in a few hours will that be irony or just funny?
Paying taxes to buy civilization is like paying a hooker to buy love.
http://www.enterprisestorageforum.com/ipstorage/news/article.php/3879726
lol - now that's funny
If you can read this... 01110101 01110010 00100000 01100001 00100000 01100111 01100101 01100101 01101011
My dads probably got more music (excluding the stuff he downloads) than he will ever get round to listening to. He is almost 50 and I seriously doubt he will double that and his entire basement manages to fit 2x chair + HI-FI + 90% record collection + 75% CD collection. If I go stay the rest of the records and some CDs make it impossible to get into my bed without knocking a stack over.
This is all after getting rid of tapes/MD and quite a lot of the vinyl/CD sometime I think he hold up the entire UK music scene on his own.
My point is you don't need to download to have to much music just spend all your time in record shops.
Most Damage is done by people who are AWAKE
The number is meaningless, because "duplication" is arbitrary. Where do you draw the line? If duplication means "copying data from one place to another" then data is duplicated every time function parameters are pushed on the stack, every time memcpy() is called, every time something is loaded from disk into RAM. I could write a simple loop that copies a 32-bit quantity from EAX to EBX three billion times per second. If you include all that shit going on, I bet their number would be higher by a factor of a thousand or more. What the fuck is the use of this metric? How do you justify the arbitrary stopping point?
Where have you been, stuck in 2000? I said HD home movies and photographs. First, average joe can't hardly find a camera that's not digital since walmart only sells 2 cameras that still use film.
Second, you can't buy a camcorder that's not flash or hard disk. Yep, you heard me: Walmart only sells 2 camcorders that record directly to DVD, the other 150+ are all flash and hard drive. The camcorder offering the smallest hard drive capacity is still 80gb for a paltry sum of $350 and HD camcorders start at only $89.
So it is not I that is projecting my geek lifestyle on the world, it is you who is out of touch with modern consumer electronics.
my karma will be here long after I'm gone
Someone please Mod parent +Funny - Did no one get the irony?
Authority questions you. Return the favor.
Are you kidding? I bought, at retail, a Sony cam, with 60GB internal drive and HD resolution 2 years ago for $900 from Costco.
Yeah.. thats means everyone has a Sony cam with a 60GB hard drive.. oh wait.. NO IT DOESNT
It means that you are a geek. Most people do not own *ANY* camcorder.
You heard me. Most people do not own a camcorder.
Let me repeat that one more time. Most people do not own any camcorder.
"His name was James Damore."
Average Joe does not even LOOK for a camera, let alone buy one.
You are projecting your own lifestyle onto others. The average person does not own a digital camera, and most of the ones that do are sporting one integrated into their pay-as-you-go $30 cell phone.
The average person does not have a smart phone. The average person does not have a camcorder. The average person does not have a digital camera. The average person doesnt even have a game console. They have a laptop which they send email with. Thats it. A laptop for email.
"His name was James Damore."
Funny, me too, I'm at 7/1 TB rather than 1/7
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
It's not that much really. Only about a millionth of a hellabyte.
You know people who don't own a digital camera? Really? I don't know anyone who _doesn't_!
Zettabyte, pettabyte...pish pah, When they come up with the Teraquad then I will be impressed.
Most amusing to me is that 1,048,576 (if that's what he meant) is fewer characters than "roughly a million"
And 2^20 is even fewer!
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Anybody have a link to the torrent?
Jesus was all right but his disciples were thick and ordinary. -John Lennon
Do you have Asperger's?
No. Do you have Alzheimer's?
I'm assuming that's how we play this game. I haven't seen it before, though, so I'm just guessing.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
LOL he's joking, obviously. Average Joe does not have a digital camera, smartphone, camcorder, game console... all they own is a laptop for email
my karma will be here long after I'm gone
ha yeah it was pretty dead-pan humour (not that that's a bad thing...)
NOBODY needs a porn collection that big.
You're forgetting backups.
Just another kind of porn, I guess.
Hell, 640 petabytes isn't enough for me. I started burning the porn on DVDs!
1.2 ZBy is about 1/8th of a gram mol of bits. With molecular memory, it all ought to fit on a few grammes of carbon.
Umm, the article is pretty much about the fact that we did generate it in our lifetime
No it means he bought a camcorder sometime in the last 5 years.
In plainer language it is one with 21 zeros. I personally didn't know petabyte off the top of my head.
Democracy Now! - your daily, uncensored, corporate-free
I dunno, my anecdote agrees with the other guy. My dad had a digital camcorder and was looking into upgrading. This is the man who screw up a mostly locked down laptop he only used for web browsing and email (o.k. mostly instead of entirely locked down was my mistake.)
I have a 20 year old sub $10/hour employee with a mid-range blackberry, and the rest of them probably average $200 phones (a couple of geezers like me and two single moms bring that average way down).
My thinking is look at the prices and availability of them, that suggests a fairly high number of people have them, I'd guess (just that a guess) well over 20%, but probably not as high as 60%, say 1/4 to 1/3.
Mycroft
https://signup.leagueoflegends.com/?ref=4c3ed6600b6ea
For those new to this, the troll is someone claiming to be a defender of a certain Alexander Peter Kowalski, the author of a tool, apkapp2backgrounddaemonprocessengine, generally considered malware by a large number of anti-malware companies and organizations.
CA
PestPatrol
SpywareDB ("Dangerous!")
Freedom Anti-Spyware
Spycheck (Spanish-language) - "Recomendacion: DESACTIVAR Y ELIMINAR"
Spyware No-More("Threat risk: High risk", "Advice: Remove This is a very high risk threat and should be removed immediately as to prevent harm to your computer and / or to protect your privacy")
Mr Kowalski, or his admirer, got upset because someone had the audacity to link to a threat describing Kowalski's attempts to remove some embarrassing comments posted under his name. Rather than deal with it maturely, this person has been attempting to stalk said poster and those who pointed out Kowalski wasn't doing himself any favors.
So if you see these comments posted as replies to clone, squiggleslash, Tom Hudson, or GM Howell, now you know why they're appearing. And if you feel like joining in, well, come on in, the water's lovely!
You are not alone. This is not normal. None of this is normal.