Slashdot Mirror
The Past and Future of the Hard Drive
Snags writes "Brian Hayes of American Scientist has written a nice little historical review of hard drive technology, from the first hard drive (nice pic) made by IBM in 1956 to what may be available in 10-15 years. He muses on how to fill up a 120 TB hard drive with text, photos, audio, and video (60,000 hours of DVD's). Kind of ironic that this came in my mailbox today considering IBM's announcement."
I remember going to the Trenton Computer Fair 8 years ago and finding a great bargain! a 5.25 full height drive that could hold 1gb!!! our bbs was gonna rock(loved them 0-day warez d00d).. we paid 20 dollars for it..which at the time was like a dream.. so it was probably hot... but it worked :)
I don't care how much space you give me.. its never enough. I'm running around 300 gigs right now and it mostly full. Why? because I do video work and motion graphics design, and I'm not very organized. I've got tons of source files from archive.org and I really don't want to go through the trouble of burning CD's or backing up to tape - I love to have it all accesible quickly via harddrive.
So of course I'm not going to fill all that space with typed notes, but even if I dont do as the article suggest and "document every moment of our lives and create a second-by-second digital diary". I still want that space for massive amounts of easily accesible data.. There's no reason I should ever have to delete anything ever again..
Uh. Except that I cant find anything I'm looking for anymore.. Can't this search function go faster?
air and light and time and space
120TB seems like an enormous capacity, but multi-terabyte storage mediums will be a neccessary in the future with REAL streaming media. What I'm worried about is the network connection - that has been my current bottleneck (even at 1.5Mbps). When we eventually combine HDTV PVR's, MP3 players, DVD archives, and pictures into a giant media database, the numbers don't look as staggering. But transferring that much data from one machine to another may prove to be the hardest part of all. 120Tbps network connection -now THAT'S impressive!
It was 5 MB
I remember trying to cram as much as possible on my little 20mb hard drive -- I had DoubleSpace and then Stacker running to compress that thing to the utter maximum -- and I backed it up on floppies, too!
;-) collections, but for those of us who actually store mostly normal files, a lot of source code, etc, it'd be great!
What my real question is, with as much abstraction going on between the ide controller and the drive platters as there is now, why haven't we seen more in the way of harddrive firmware except for better bad-sector remapping and the like? What about hardware compression? A little flash memory or a dedicated on-disk area and a compress/expand chip and we could probably fit quite a bit more on existing physical head/platter technology with not much speed loss -- In fact, we might see some speed GAIN if we only have to pull 100 bytes off of the disk to return 1000 bytes of data, etc.. Of course, it wouldn't give you any more space for your DivX
I guess, unfortunately, it requires a bit (ok a lot) of work getting an OS to play nice with such a gimmik, but would installing a "driver" for your hard drive compression electronics really be all that much different than for your video card or drive controller itself? -- then of course there is the question of a filesystem that can handle the indeterminate capacity of this kind of system. IE you couldn't necessarily delete a 100M file and fit another 100M file in the same physical storage space...
ideas. something to muse over
He muses on how to fill up a 120 TB hard drive
Let's see - if the history of the internet serves as an apt model - 120TB drives probably won't meet consumer demands for long.
First harddrives will start to fill up with fully-imersive holo-pr0n, followed quickly, due to adaptive marketing trends by fully-imersive unsolicitted holo-spam.
There... that solves that ol' capacity problem quite nicely then.
:)
Sure it can holds 60,000 hours of DVD. That's why when the time comes we'll come up will a more precise format. I don't believe we will had enoguht space until it can store a lifetime's worth of information in a format indistinguistable from reality to human senses. Until that point we will always be able to make it a little more lifelike, a little longer, there will always be somrething else to eat up the space, clockcycles and bandwidth. My 1 GB filled up just as fast as my 256 MB and I'm sure that my 60 GB will fill up as well. We still have a long way to go before we have "enough" space.
I stole this Sig
My dad would come home from work and tell me that the computer (probably something 8 bit) at work would fail several times a day. Mainly the hard drive (which was as huge as an ATX full tower case and only stored 10 mb) would stop accessing so they would have to reboot the computer. How did they do that? They gave a swift kick to the hard drive and the machine started right back up.
A well-known corollary of Parkinson's Law says that data, like everything else, always expands to fill the volume allotted to it.
I don't think this extends to distributed computing; I hardly think the collective drivespace of the WWW has been filled to the brim. Even a few percent free space per drive per server equates to huge amounts of unfilled sectors.
There are already a number of Terra satellites downlinking data at about 4GB/hr, circling from pole to pole in orbits lasting under 2hrs.
.
There are multitudes of airborne surveys churning out digital snapshots at 400MB a frame.
Mosaiced together at 1m resolution with R,G,B and mean height above sea level, how much storage will a single global snapshot of the earth take ?
Then consider for historical and environmental reasons, most urban/semi rural areas deserve a mosaiced snap at least once a year.
120 TB is just the start . .
std::disclaimer<std::legalese> sig=new std::disclaimer; sig->dump(); delete sig;
15 years ago, I was speaking with a freind of my fathers about these new exotic 'hard drives' you could get. He was a big time computer guru, and I was a little kid.
I said to him that I thought a 10 megabyte hdd would be perfect, and that someday we would be able to buy one for less then $1,000. He scoffed.
I still remember this very clearly. "Why in heavens name would you ever need that much space in a hard drive"
I worked it out, and every concievable program I'd use, including saved files, all told would only need 2 megs. It was just impossible to think up enough applications at the time that a home PC really needed.
I'd love to run into him again and ask him if he remembers that conversation. In 10 years, 100 Terrabyte drives will seem 'quaint'.
The Internet is generally stupid
I do question some of his statements, however, particularly about human creativity:
Now it seems we face a curious Malthusian catastrophe of the information economy: The products of human creativity grow only arithmetically, whereas the capacity to store and distribute them increases geometrically. The human imagination can't keep up.
Or maybe it's only my imagination that can't keep up.
I'd say that the bolded part above is very likely. He states that he individually can't think of what to do with 120 TB, but collectively I'm confident we'll find a use for it. I've read through a dozen posts already where we've come up with some suggestions for use. Not to be critical or anything, but the surface has barely been scratched. It's not going to be all about data warehouses, streamable content, and how many dvd movies we can rip. Tell the whole world that 120 TB is available for storage, and a variety of uses will come up.
I'm pretty convinced that the actual consumer use of 120 TB will be for something that, if suggested now, we'll all laugh at and ask why the hell we'd want to pursue such an insane idea. For instance, the article mentioned mounting tiny cameras on eyeglasses to document one's whole life. The article also mentions home digital media hubs. Both are probably uses, but I actually think they are rather conservative ideas in the grand scheme of things. A successful idea now to think of an interesting and probably use of that much space in the future is more likely to come out of the mouth of some random guy while intermittently taking puffs out of a giant bong than any mature, prominent engineer.
Then again, some founders of successful companies were allegedly (I don't have any factual evidence) pretty fond of the herb.
"Thus the 120-terabyte disk will hold some 60,000 hours worth of movies; if you want to watch them all day and all night without a break for popcorn, they will last somewhat less than seven years." Most people can't even last seven minutes with high-res pr0n playing, much less seven YEARS.
Job? I don't have time to get a job! Who will sit around and bitch about being broke and unemployed then?
Did you even read the article?
When conventional disk technology finally tops out, several more-exotic alternatives await. A perennial candidate is called perpendicular recording. All present disks are written longitudinally, with bit cells lying in the plane of the disk; the hope is that bit cells perpendicular to the disk surface could be packed tighter. Another possibility is patterned media, where the bit cells are predefined as isolated magnetic domains in a nonmagnetic matrix. Other schemes propose thermally or optically assisted magnetic recording, or adapt the atomic-force microscope to store information at the scale of individual atoms.
-Kevin
I used to think that my files would expand to fill all available space, but not anymore. Different tasks, different tools, and different personalities mean different thresholds, but I think everyone has a threshold above which they won't keep their disks full. For me, my disks stopped being full around 10 gigs. My wife's antique PC (running msdos 2.0 and used strictly as a text editor) had a 15 meg drive and never went above 10% full. Obviously your threshold is higher, but I'm sure it exists.
Even after they hit their thresholds, most people's use will grow over time, but slowly. We'll also start writing things in ways that don't try to conserve disk space. Compression will be used almost exclusively for data transmission. Future filesystems will probably keep every version of every file ever saved. (Hopefully with an option to delete the occasional residue of an indiscretion and accidental copy of /dev/hda). But even these things won't increase our use by us more than a factor of 10 or so. If we really do get 120TB drives, we won't talk about buying new ones very often.
A cynic's retort might be that installing the 2012 edition of Microsoft Windows will take care of the rest, but I don't believe it's true. "Software bloat" has reached impressive proportions, but it still lags far behind the recent growth rate in disk capacity.
I think they would have no problem occupying 20% of a 120 TB HDD, MS products have done that to my drives for years...
A friend burns CDs like there's some sort of deadline. I've only just got myself a CD burner. Maybe if I downloaded MP3s through work I might run out of space a bit faster, but the 20gig I mentioned at the top of this comment is an IDE drive in a removable bay with a USB connection. If I need more storage I just buy another cheap drive and another tray.
At work, and I am fairly new here so I might get this wrong, we currently have about 6Gig of network storage for students which is more or less full, but our solution is simply to have Zip drives in every PC. Students have a quota and really their data is their own problem. Staff have 30Gig to play with and 5 is still free. And 9 of that is a backup of the Ghost images of the student labs that I made when I arrived.
Speaking of backups, it's been my experience that hard drive space is useless without the same amount of backup space available. DDS3 tapes only go up to something like 12/24 Gig if they haven't changed in a year or so, meaning that cheap and easy backup really ends at 20Gig. Personally, my 20Gig hard drive is more or less the backup, with data burnt to CD the same time it's moved from my portable's 4Gig to the 20Gig drive. But burning CDs isn't the best solution for stuff that changes frequently. (Although it seems to be the best option for Mac-oriented graphics designers who have to live in an otherwise PC-only corporate enviroment.)
My interest in retro gaming also probably helps keep my storage requirements low. Recently I burnt an 8Mbyte Dreamcast image that had an Atari 2600 emulator and over 100 (Public Domain) ROMs.
And I mean, who needs 120TB of random access storage? Seriously. I mean, sure it's nice to be able to skip instantly to a particular chapter of a movie, but how often do you do it, really? And the "Random" button on any given MP3 player is fun, but if you had to listen to the tracks in a particular order it wouldn't be the end of the world (imagine a DDS3 MP3 player - that's 12 days of music, solid, and it would probably be able to be smaller than the original Sony Walkman).
Wow, that was a long post.
They are the key players. In America.
Mmmm.... future pr0n.
sic transit gloria mundi
It'd be nice to have an optical disk capacity comparable to hard drives again so that it is practical to do backups.
Please consider making an automatic monthly recurring donation to the EFF
If we really do get 120TB drives, we won't talk about buying new ones very often
I don't know, how much space does full motion holography take up? Seriously, we'll fill the space with something. Not everyone will fill a 120TB dive of course but there are those of us who will change our habits subconsciously to use more disk space. We just need a "killer" app.
Think about it, you'd upload a file to Freenet and it would never disappear, every Freenet node that would have ever received the file could keep it cached for a long time.
With such capacity Freenet or distributed file systems would become the ultime backup tool, you'd never have to loose data again. All movies, music and books could be stored online and would be readily available from a nearby Freenet node.
But bigger HDDs will be needed so that even if most of the world is destroyed, the most important data online is preserved on single nodes.
A more optimist use for having all of the world's data on a single computer, is for sending the data along on space ships to far away galaxy's. Perhaps for humons on the ship to enjoy themselves during a trip that would take years (or generations), or for exchanging our culture with alian civilisations in outer galaxies.
Whenever I see an announcement of the newest harddisc with record breaking capacity I think "How do you backup that beast?".
Whenever harddisc manufactureres manage to double the number of bits that can be written on an inch of a track they get an four fold increase of capacity. But unless you increase the rotational speed of the plattern the time to read the whole content of the harddisc will double as well since the recording/writing speed is proportional to the linear density.
And the rotational speed only increases very slowly - we recently saw the small jump from 5400 to 7200 RPM for the "standard" consumer (IDE) harddisc, the first for several years (I personally stick with 5400 for the cheapo IDE drives for the next few years. Reliability, you know --- see IBM)).
Given that the lower limit for the time to make a full backup of harddiscs will increase roughly with the square root of the growths of their sizes over the time.
The other problem is that backup devices and media affordable for the home user can't keep pace with the harrdiscs, so in my eyes the traditional full backup get's more and more inpractical.
One of the most cost effective backup devices for a harddisc today is another harddisc, but it still needs hours to mirror the content of one disc to another. RAID or something that keeps two discs in sync automagically in the background is no solution - it saves you from data loss by harddisc failures (good if you use IBM GXPs and the like) but it won't help you if you or your software have/has destroyed some important files you have created over the past few weeks/months.
Well, I don't know what other people do, but I stopped doing full backups of the whole disc: Thank God a large amount of files on my harddiscs is not backup worthy since in case of loss I won't miss it (think swap space or contents of a web cache), can easily recreate it (like mp3s made from my own CDs or object files - if you track netbsd-current you keep them around to save some time on future inkremental builds and don't delete them after "make install") or get it back from another of my machines, CDROM or the net.
So I only have to backup a fraction of my discs and the good news is that in absolute numbers the amount of data to backup doesn't grow nearly as fast as harddisc capacities. In my case with compression it easily fits on an older harddisc for complete backups and for weekly incremental backups I can still use and old 1GB DAT (DDS) tape I've got for free. It's not the best solution since recovering from desaster needs some time and a lot of manual work but I can sleep better than those who don't do backups at all...
Later, in 1986, I bought an external 20 meg (HDSC20) for my Mac Plus for $1,200 and couldn't believe how cheap they had gotten. That same year I spent $700 for two one meg SIMMs for that computer.
I also remember around 1984 seeing my first b/w 2-bit porn pic on my Mac and being amazed at the quality!
The old days sucked, but it was also kinda nice to be around during that time. I can really appreciate how good I got it now. I just bought a 160 gig external firewire drive for $400 for example. Sweet...
No more backup worries either, I just buy firewire disks and tack on as needed, and rsync nightly...
I can just imagine the Canadian high school trying to flip enough burgers for that shiny new 5 petabyte mp3 player in 2010.
$100 down, $200,000 to go... just another 200 years and I'll have it!
1920 x 1080 pixels
16 bits/pixel x RGB = 6 bytes
60 frames/second
Yields 746,496,000 bytes/second. (Or about 8 parallel gigabit ethernet cards)
Do this at full bore, and you get 160,751 seconds of video, less than 2 days worth!
Sure, I know you could compress the video, but I've seen 1080p up close and personal, I noticed the artifacts in the video on the monitor of the broadcast quality HDTV demo, and the sales guy finally confessed that they just had to compress it to make it feasable to record it on tape.
If I noticed it right off the bat, someone will pay to have this quality level.
So... when do we get the petabyte storage?
--Mike--
I some letter to the editor of some computer magazine.
:-) and one CD-RW drive on my TiBook which I would use for backup except that the drivers from Retrospect don't seem to quite be working.
Now I own a 20MB & some 4.5, 6, 10, 15s, 19 & 30 GB drives, and you know what? I still can't backup my stuff worth crap.
I have one CR-RW drive for my Linux box (which IS backing up my domain,
Its never been IF the drives fail (I've hung lots of opened up drives on my cubicle wall,) but WHEN. And nobody has EVER addressed that issue properly.
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
I remember 17 years ago when a Seagate 20 MB hard drive went for US$500. Those were also the days when Maxtor introduced their 650 MB 5.25" full-height SCSI hard drive, which cost US$6,000 back then. (eek!)
Nowadays, that same US$500 buys you 320 gigabytes of storage on two 160 GB ATA-100 3.5" 1/3 height hard drives. (thud)
I'm surprised no one else here has ripped a DVD to their HD before. 2 gig per hour seems like a minimum to me.
And even if it isn't, by the time the 120 TB disk come out, you think we're still going to be using the worst DVD format? 720p is supposed to be coming out in a few years. I can't find the page with the details now, but 720p will require a new type of DVD disk, one that can store up to 24 GB.
That's right, your DVD collection is going to become outdated and worthless as companies republish into the new "Hi-def" DVD format.
Let us say the average DVD uses 70% of the space, 16.8 GB. That's about 7,100 hours. Compression might be less effective on these disks, since the entire point of having Hi-Definition DVDs is the extra detail.
And I've come up with another use for a 120 TB drive. The biggest, most kick-ass TIVO in the world. Imagine having any TV show that was shown in your lifetime available to view.
Democrats or Republicans. They are both taking us to the same place and they are not afraid of us anymore.
First off, The future 120TB drives will be cool. Second, does anybody read? IBM is NOT getting out of the HD business. They are moving most of their HD business into a joint venture with Hitachi. This new company will be 70% owned by Hitachi and 30% owned by IBM. (IBM is NOT selling 70% of its HD business). IBM is going to supply most of the technology (and employees. According to a CBS Marketwatch article) and Hitachi will manage most of the business. IBM will still be a leader in HD technology, they just don't have to take on as much burdon in the poor HD market right now.
Yes, I _WAS_ refering to Betacam and DV, and not even thinking about the HDTV versions. Heck, Umatic is probably as good as DVD (Just a different "Kind" of error)
As I said in another thread, I'm a geek for a network - who else has 4 million hours of video sitting on the shelves? (One of my projects tracks all those tapes)
And your right, film blows them ALL away
-- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
If the average file size increases to something more like a half gig instead of a couple hundred k won't disk io be prohibitively slow? I mean copy an entire 80 GB hard drive right now takes forever....even if it is at 15k rpm. I wish that disk io could at least attempt to keep up with processor speeds and storage capacity
this sig is deprecated
People wil probably want *personal* storage of a thousnd hours of media. Beyond that you start forgeting it or never re-using it. We already have this capicty for books and music. A few terabytes gives a thousand hours of conventional TV. Some day they'll be forms of 3D TV, and that will really consume storage.
t_t_b
I'm on PJ's "enemies" list! Are you?
I can't see spinning disk drives reachin the 120 tb capacity talked about in the article. Although I think the capacity will surely exist in 10 years, I'd place my money on either:
- some form of persistent RAM
- holographic storage media
Both have come a long way. The holographic option had a usable storage time of close to a year the last time I checked; not good enough yet for sale, but given that it started on the order of *hours* this is a damned impressive advance.
Plus, the holographic model being talked about now is a cube which you could *actually pop out of the machine*, put into your pocket, and take with you. Imagine going to a friend's house and being able to snap in your entire hard drive as easy as you do a floppy....
Max
My god carries a hammer. Your god died nailed to a tree. Any questions?
Sure, a terabyte on a single disk would be great, especially for large supercomputers like ASCII White or the new Linux supercomputer. But wouldn't this result in lowered overall memory throughput?
Let me explain. If I were to build a supercomputer with a 1TB storage array, I would probaby use 100 or so 10GB drives rather than ten 100 GB drives. Creating a RAID 0 array with 100 drives would probably be much faster than with 10 drives, even though the 100GB drives transfer data internally faster than thier 10GB counterparts (assuming the same RPM and number of platters). I realize the cost of supporting 100 disks as opposed to 10 is much greater, but you must make a tradeoff. Likewise, a single 1TB harddrive would not be as fast as ten 100GB drives.
Of course, if you are rich like IBM and most major universities (all they have to do is bump up tuition... again), you can just buy a bunch of the biggest drives available and make a super-fast, super-big array.
--- At my sig, unleash hell.
perhaps i'm just low on creativity, but I can't imagine any way of capturing vide from real life with a system that isn't functionally equivalent to some finite number of video cameras.
A hologram isn't made by taking lots of photos from different angles. I don't know why you'd suggest it for motion holography. You need to look up how holograms are made.
You honestly can't see how a holodeck-like virtual recreation of a medieval battle (or whatever) would take up a petabyte? How about a hundred recreations?