Slashdot Mirror
Mass Storage Leaves Microchips in the Dust
Roland Piquepaille writes "This article from Wired Magazine looks at storage with a new angle. 'Right now I am sitting in front of a whirring 60-gigabyte hard disk that cost less than $100. Do the math: If back then 10 megabytes cost $1,000, then 60 gigabytes would have cost x, where x = $6,000,000 and "back then" = 18 years ago. I'm sitting in front of $6,000,000 worth of mass storage, measured at mid-1980s prices. We have Moore's law for microprocessors. But who's coined a law for hard disks? In mass storage we have seen a 60,000-fold fall in price -- more than a dozen times the force of Moore's law.' DeLong also looks at a non-distant future when a $100 mass storage device will hold a full terabyte. He also thinks that with disk space becoming cheaper and cheaper, we'll be tempted to archive everything about ourselves, including pictures and videos. This is in fact the goal of the Gordon's Bell project, MyLifeBits. You can learn more about the MyLifeBits project by reading this NewsFactor Network article. Check this column for more details."
No only the price, the size of the drives. 18 years ago a 40 Mb HD has the size of a toaster...
The package said "Windows XP or better. Pentium Class Processor or better"... So I got a Mac with OS X
" 'Right now I am sitting in front of a whirring 60-gigabyte hard disk that cost less than $100. Do the math: If back then 10 megabytes cost $1,000, then 60 gigabytes would have cost x, where x = $6,000,000 and "back then" = 18 years ago. I'm sitting in front of $6,000,000 worth of mass storage, measured at mid-1980s prices. We have Moore's law for microprocessors. But who's coined a law for hard disks? In mass storage we have seen a 60,000-fold fall in price --"
You mean that all this time we could have had much faster computers just by using magnetic media?
"Derp de derp."
we have jon's law.
as in the toilet.
note to harddrive manufacturers: i'm not impressed. i'm still waiting on my data to "move around".
This is just common knoledge. And if he paid 100$ for a 60gig drive, he got screwed! Thats why there cheap, cause dumbasses pay too much for drive, and the manufacs pass the savings on to ME.
In general the problem is that while capacities have lept up, the rate at which we can read/write to those drives has not kept pace. It's not so bad for the iPod in particular, but at some point it's going to be a real problem for desktops and laptops, assuming our appetite for capacity grows as the capacity does.
Can't you see that everyone is buying station wagons?
I don't want to see Gordon Bell's "lifebits"
--"The perfect example of the man of action is the suicide." - William Carlos Williams
Moore's law says nothing about price though. If you are going to compare hard disks to processors in the same general terms using Moore's law, shouldn't you compare increase in storage size to increase in processing power?
"I turn away with fright and horror from the lamentable evil of functions which do not have derivatives."
Bloat will kill the increase in storage available - one way or the other. It'll be a 3gig version of word, or windows movie maker that will only save in raw, non-compressed video. Anything to drive the market. We've seen it with processor speeds, if HD prices keep dropping I'm sure well see it with storage as well.
Come on, is XP is SO far ahead of NT 4 that it requires 4x the ram? Of course not. But what MS reccomends, PC manufacturers will have to yield to.
This really helps to put into perspective the ass-whipping I got when I installed Wing Commander 2 on my Dad's new hard drive.
"THAT 800 MB HARD DRIVE COST ME 500 DOLLARS, AND THAT GAME TAKES 72 MB?!!!"
"But dad, in 15 years that will only be 25 cents of space!"
Why should I argue rationally with someone being irrational? I'll just mock them instead.
A lot of the developments that have made disks so high capacity came from spintronics research. Here is a link to an article on Scientific American about how it works: http://www.sciam.com/article.cfm?articleID=0007A73 5-759A-1CDD-B4A8809EC588EEDF
He's predicting only a tearbyte for a $100 in 2012. Right now desktop drives are about a dollar a GB. So, he's predicting about a 10 fold increase in the next 9 YEARS!!! What have we seen in the last 9 years, about 100 fold increase?
but wait until the RIAA starts charging in advance for piracy. They can do $15 for an album, or charge $15000 per song ammounting to $1,485,000 for a single recordable disc (99 possible tracks).
You can't judge a book by the way it wears its hair.
If that rate continues, some day hard drives will become so large that processesors will not have the power to process it all....
I will know that day has arrived when the length of my winamp playlist rolls over into negative integers. :)
Muerte
I have to admit that the notion that it is now techincally possible to mpeg-1 every moment of one's existence is a staggering one.
If you accept that Blogs satisfy some previously underestimated human desire for self-expression, think of what might happen if one could clip a web cam to one's collar, wear a storage device on one's waist and synch that with an online VidLog every night like a Palm Pilot?
I am going out back to sit among the dandelions.
The best way to do is to be.
An interesting feature of OpenNap is that it tells you exactly how many MB of files are out there for your downloading pleasure. I used to be blasted away at the large number. Sometimes I could get it up to 1 or 2 pedabytes.
A terabyte is 1000 gigs. You can get a terabyte of storage today for $1000 dollars. One dollar per gig. It's insane. Soon it will be a dollar a terabyte. We wont need things like divx anymore. We'll be looking for ways to increase the quality of our recording devices so that the video, image and audio files will take up more space. Nothing else really requires a large amount of storage.
The one limited is network speed. Sure, if I've got enough room for a collection of 2 gigabyte raw avi movies, that's great. But if I can't get enough speed to download them quickly it will suck.
Storage aint worth crap if you dont' got enough stuff to fill it.
Remember the days when DOS games would ask questions like this
minimum install (if you're low on space)- 50MB
standard install (reccommended)- 100MB
big install (runs faster)- 250MB
CRAZY INSTALL (no cd required!) - 500MB!!!
those were the days...
The GeekNights podcast is going strong. Listen!
looked at another way, hard drive capacities have just been doubling faster than processor speeds.
If 10MB back then cost $1k then 1MB cost $100, so we just do the 60G/1M and get a 60,000 time increase in storage capacity for the same price. Doubling times would then be log(2)60k = 15.9 or so, or about once every 1.1 years over 18 years. Contrast this with moore's law which states that processor speeds double every 1.5 years.
The downside is that access times have tracked closer to a linear function.
Actually, I started a project to do just this for myself about 7 years ago. I'm 28 now and it's taken me until NOW to catch up (and I'm only about 80-85% caught up.... still have plenty of baby pictures to scan). I just spent last weekend sifting through old documents about yours truly that my parents has stored away. I now have a ton of interesting data about myself on tap, including every report card I had until college. :)
The original motivation for this project for me was the realization that my generation may never have to face death and I was terrified at the prospect of being 1000 years old and having no recollection of my life as of today. So I decided to digitally augment my memory. And it's VERY effective. Seeing scans of ticket stubs of concerts I had completely forgotten brought back all those memories in a flash. I'd hate to image what would happen if I just forgot permamently and never had a 'key' to unlock it.
Since starting the project the original motivation has been eclipsed by an even more compelling one: Going through all these documents made me realize that I've lived an amazingly full and rich life and I'm only 28. I was overwhelemed with awe and gratitude at how huge and wonderous my relatively normal life really is and how tragic it is that I've forgotten that. In other words, it was a great way to refresh the brightness of all the colors in my memory, which left me feeling uplifted and more optimistic about the future than ever.
(Still, every once in a while I imagine myself a few centuries from now, bored out of my mind on an multi-year interstellar trip to somewhere and enjoying a good browse through 'The Story Of Me')
It would be much better if we could combine this growth in the industry, into producing CF cards that can hold 2+ GB, and give us mass storage on small microchips.
Saskboy's blog is good. 9 out of 10 dentists agree.
In a few years time, when storage is cheap enough, I'm going to have a camera permanently strapped to my head (think better minaturisation + wireless tech) recording my life full-time. At moments of boredom I'll be able to relive any part of my life.
How much storage? Say, 500Mb/1hour (better compression as well, hopefully) * 24 * 365 ~= 4.4Tb/year. Doesn't seem that far away...
Record everything. Once your life is half over you need to cue up the recording and start watching what happened in the first half of your life. Then when that is over you drop dead.
The challenge is no longer whether you can store everything, it is whether you'll be able to find it later when you need it.
This sig has been temporarily disconnected or is no longer in service
Nah, not really. There is only so big an app can get in terms of code, and it's way, way less than a small HD can do. For consumers, who are using more than 25% or so for their drive, it's likely digital media that's doing it. MP3 files, and especially video. Textures and videos in games. Tutorial files for media apps. That kind of stuff.
.app, and you'll likely find less than 25% of the total file is being used for application code. The rest is multilingual help, graphics, sounds, etcetera.
Crack open your average 20 MB MacOS X
My video compression blog
If prices fell by 60000 times over 18 years, then we're looking for the solution of the equation: x^18 = 60000, which implies x = 1.84 years = about 22 months. So prices fall in half every 22 months (assuming it's exponential). Moore's law says transistor density doubles every 18 months. It's not that much different, although it does make a huge difference over time.
It will be a 3gig version of IIS, .Net, or whatever. The extra 2.9gigs are bundled data so you can buffer overrun yourself.
Can't you see that everyone is buying station wagons?
You hear people refer to the assumption that electronics will keep getting cheaper and and cheaper as "Moore's Law". Nit-pickers hate this, insisting that "Moore's Law" only refers to the number of transistors on a chip. But even casting Moore's predictions as a "Law" goes beyond what Moore actually said. So it makes just as much sense (or just as little) to speak of the whole economic trend as "Moore's Law". After all, the fact that transitor logic keeps getting cheaper and cheaper isn't obvious to most people. The resulting collapse in the cost of computing and electronics is.
Creating bigger hard disks does nothing to solve the problem of reading data from existing storage devices. As time goes on our society stores more and more information without any real plan on how to ensure that the information we're collecting will be accessible in the future. Every year we lose more and more precious data to the deterioration of media as well as the loss of the equipment to read the remaining media.
Well, they have in some ways.. Increases in capacity have come due mainly to vast increases in the areal density of the media. This, in turn, yielded massive increases in the rate the data moved under the heads..
The problem with hard disks isn't the data transfer rates they are capable of - it's their latency we need to worry about.
We need better defragmentation algorithms - I suspect that files are usually accessed in list order.. When running a program, for example, it's always going to want to read the same files in the same order. If we can arrange files that are usually accessed to be contiguous on the disk surface, and also make the filing system read the whole list of files that are situated contiguously into the disk cache when the first file in that list is read then the relatively high burst transfer rates will take more precedence over the access times, and things will seem a lot quicker.
Moore's law is largely due to manufacturing improvements in which the feature size of transistors keeps becoming smaller, such that you can get (approximately) twice as many transistors in the same amount of space. (yes, yes, I know, die sizes keep growing, but not nearly at the pace at which transistors shrink.) The tricky part here is that this shrinking has generally been coupled with ramping up frequency. Increasing the capacity of a disk has no such benefit due to the fact that mechanical parts (disk heads, spinning platters) are the overwhelming determining factors for performance. Hence, the gap between processor performance and disk performance is being exacerbated - we can only make a disk spin & heads move so fast.
It's an interesting comparitive trend to notice (between processor performance growth & disk capacity to see the effect on the overall system), but you can't really compare the way disks have improved with the way microprocessors have.
Moores Law for Microchips
(doubles every 18 mnths)
Porns Law For Storage?
Good lord! Are you serious??? You obviously never had to use debug to partition an esdi drive. You obviously have somehow missed the whole transition from 10Mb burst to 160Mb burst we've seen in the last few years. Scsi has, as well, gone from 8-bit with 5Mb transfer (scsi1) to 10Mb transfer @ 8bit or 20Mb transfer @ 16bit (scsi2), 40Mb transfer @ 16bit wide (scsi3) and now 80Mb transfer @ 16bit wide with ultra2. And while that's just what the *bus* can handle, I can promise you that the disks of today are far faster than the disks of even just a year ago.
Ask yourself...what restricts data fransfer speed? Several things, really. Density is actually a factor, as its an engineerign feat to get the disks spinning fast, so the more bits go past the heads during a given time, the more can be put on/pulled off. Also, the ability to process that data, which - guess what, has significantly increased. Then there's the length of time a head needs to spend to actually get a bit to seat at a N/S, 0/1 - materials platters are made of are constantly being improved, so that's far better. Then theres the mamangement of the data itself, algorythms for where to write what, etc. Again, substantially improving, constantly. And all I've discussed was scsi - ide has improved (has quantity on its side) far more than scsi has the last few years, too.
How in the WORLD could you say hard disks haven't gotten faster? Oh wait, I know how...because you are either being sarcastic, you're insane, or you simply have no idea what you're talking about. Did you just start using computers last week?
Except that processors don't just give up the ship randomly(well, except in VERY rare circumstanecs)- drives do it all the time; it's almost expected. I don't give a crap about another 20GB or $20 off, I want a hard drive that won't turn itself into a paperweight after a year or two. If I'm going to own the drive for 5 years, what's another $20?
SMART was an improvement, but most OS's(linux included) don't even recognize SMART info out of the box. Even if you've got the SMART utilities installed and the kernel modules etc, /var/log/messages is so noisy, I mostly ignore it- same for Win2k boxes, Event Manager is full of TONS of crap(thank god it has filtering, but still...) If SMART were to be useful, the HD would beep at you, or blink its LED, or the OS would annoy you with popup messages so you knew, "oh shit, I gotta back up my stuff to somewhere else, NOW!"
I had an ancient 4GB Digital drive I got second-hand, in the early 90's; it was already several years old when I got my hands on it, so it was probably pre-90's. It weighed a ton, took up the full space of a 3.5" drive bay, and even had its own little suspension system. I abused that thing to hell and back, carrying it in bookbags, cooking it when the fan on the external case died...the whole nine yards. I think I low-level formatted it a dozen times(something you're not supposed to do often on SCSI drives, supposedly). It only finally gave up the ship around '99, when it spent a couple months cooking itself to death hooked up "temporarily" to a machine I forgot about.
Meanwhile, I've lost two quantum drives(one laptop, one Ultra2 3.5") and my athlon's Maxtor drive is making funny noises every once in a while. None of them were more than 2, 3 years old TOPS. WTF? The excuse seems to be that consumers don't need the reliability corporate users 'demand'.
Home users users have, at the very least, equal needs as business users, because while businesses need to keep going 24x7, they often have backups, clusters, RAID units, etc. Most home users don't have any of their data backed up, RAID is practically unheard of among the jane-and-bob computer users, and of course no clustering.
Please help metamoderate.
"The point was that they haven't gotten faster in proportion to the increase in size. "
Ah I see, that's true.
Not that surprising, though. The mechanical arm inside of the drive has its limitations. I wonder what it'd take to replace it with a magnetic field sorta like what TV's use to fire energy at the phosphors on the tube. I wonder if a disc (maybe optical disc?) could be read that way. Seems like you could dramatically increase it's read speed that way.
Blah I'm sure there's a serious issue that I'm not thinking about. Oh well.
"Derp de derp."
I don't agree.
The physical drive passes by the head at a certain rate, depending on the speed of rotation of the platter and the distance the head is from the center of the platter.
Lets say 1 inch from the center, going 7200 rpm.
This means that the disk will be passing under the head at about
2 * pi * 7200 = inches per minute
/ 60 = inches per second
/ 12 = feet per second
((2 * pi * 7200) / 60) / 12 = about 62.8 feet per second 1 inch from the center of the platter.
Now, lets pretend that there is some amount of data in this 6.28 inch circle that the head travels. Lets say 1,000 bytes. This means that 62800 bytes pass under the head in one second. Now, lets make the data more dense, which is how hard drives hold more data. Lets say there is 1,000,000 bytes. This means that there are 62,800,000 bytes passing under the head in one second. So really, the data has gotten faster. That's why you don't really need a high RPM drive that has extremely high data density (like 150gig drives), cause that won't be the limiting factor.
So actually, hard drives have gotten faster. At least the data passing by has gotten faster.
Ever try and run NT4 on a 300 mhz system with ~only~ 64M ram? Runs pretty damn well.
Then add all the SP's and IE6, IT...SLOWS...TO...A...CRAWL...
fslg503-985-8686503-985-8686503-985-8686503-985-8
Drives today have 10.000 rpm or 15.000 rpm. Eight years ago the high end was 7200 rpm, 5400 before that...
;)
That's approximately a 2X performance increase per EIGHT YEARS. This is very very far from being impressive.
Disk seek time is dominated (today) by rotational latency. The fastest disks have seek times around 4ms, and that is pretty much the rotational latency on a 15000 rpm disk.
In order to improve disk performance (the seek time, not the throughput), disks need to spin faster. This does pose some interesting problems though...
A normal 3.5" drive has a platter with approximately 48mm radius, giving roughly 0.3 meter circumference. At 15000rpm the speed of the circumference is 75.4m/s.
Doing the math, this gives us a centripetal acceleration of v^2/r = 118435 m/s^2, or roughly 12085G. Sure as hell beats most drag racers out there (by more than a factor of 12000
The fun part is, that a simple doubling of the rotational speed, will do really interesting things to the acceleration (note the v^2 thing above).
A 30000rpm disk will have a centripetal acceleration of the circumference of approximately 48000G.
A mass-element at the circumference weighing one gram, will have a "pull" corresponding to (F=m*a) 118kg - which again will be approximately half a tonne on the 30000rpm disk.
You need to find a material that will weigh little, not deform under the given stress, and still have the necessary properties for use as a hard drive platter...
I'll be able to couple some hard drives to my flux capacitor and record the entire history of the universe.
Why is it no time traveller goes and says 'hi' to Jesus? Thats what I'd do.
The Kruger Dunning explains most post on
Consider the IBM XT 4.77MHz with a factory default formatted 17 sector per track MFM hard drive with a 6:1 interleave. The peak throughput of this machine was roughly 87 kilobytes per second.
... 3.06GHz, maybe a touch faster?
Now consider the new SATA machines with measured (not calculated) throughputs of 87 megabytes per second.
This is a 1,000x fold increase. For CPU processer throughput (speed) to keep up with this performance at the same rate, you would be able to buy a machine with a 4.77GHz CPU in it. Right now the fastest stock boxes are running what
CPUs have gotten faster. Hard drives have gotten faster faster.
Glonoinha the MebiByte Slayer
While it's nice that you took the time to rant about how much better of a programmer that you are then everyone else (the whole "If I didn't code it, it's crap" attitude really shines through), I think your scale is a bit off.
Lets say a library saves you a week. Now, lets say that like more people you use at least 4 libraries. Now, you've saved a month. A *month*, at which point you say you'll start to "consider" using external libraries. Well, I'm underpaid, but lets say you hired me to do this. By shaving a month off, you've saved over $3500 in my salary alone. And that's assuming that I (or anyone) could fully implement, *debug*, and "finish", a given complicated lib in 1 week. Great! Now, I quit, because I'm underpaid, and my replacement comes in. Now, I write good, well documented stuff, but it's not industry standard. So my replacement can't just sit down and pick up where I left off, but has to learn how *I* decided to implement libfoo. But it turns out that he's a lot like you, and thus 'he didn't write it, so it's crap'. And then *he* spends a month throwing away my stuff, and redoing it all. And on, and on, and on. There's a *reason* that things like Boost and Roguewave and Qt and Gtk and glib exist. And until you figure that out, you're doomed to be 1/10th as productive as you could be. Or, assuming that (as you claim) you've polished your libs to perfection and the productivity is there, I pity whomever has to take over your code. No, actually, I just pity you.
Moores is the # of transistors/processing power every 18 mos... you're looking at price per byte.
lets see what $100 gets you
$100/meg = 1985 10 meg
$50/meg = 1986.5 20 meg
$25/meg = 1988 40 meg
$12.5/meg = 1989.5 80 meg
$6.25/meg = 1991 160 meg
$3.13/meg = 1992.5 320 meg
$1.56/meg = 1994 640 meg
$0.78/meg = 1995.5 1.2 g
$0.39/meg = 1997 2.4 g
$0.19/meg = 1998.5 4.8 g
$0.09/meg = 2000 9.6 g
$0.04/meg = 2001.5 18.6g
$0.02/meg = 2003 37.2g
$0.01/meg = 2003.5 74.4g
Looks like the curve is a bit faster than every 18 mo... I think 12 months might be a better approximation of storage/cost.
meh
"Basically if programmers still gave a damn like they did when writting code for C64's we wouldn't have alot of these issues. Nowdays they would rather churn out crap so long as it's better than some of the other crap they've seen."
If you're trying to tell me we should go back to the days of non-portable assembly, I think I'm going to cry. Yes, people should write tighter code, but trying to make believe that we should write code just like in the "good old days" is ignoring years upon years of advancement in the field of computer science.
And, also, look at what they were doing back in the days of the C64, and look at what they're doing now. You really do need more code to do more. Trying to tell me that they had 6kb executables with the C64 and then telling me our 6mb ones are bloated is ludicrous.
-Erwos
Plausible conjecture should not be misrepresented as proof positive.
18 years ago a 40 Mb HD has the size of a toaster...
Yep. Generated the same amount of heat, too.
Who moderates the meta-moderators?
Nope. A fragmented disk is what you get when you chuck your hard drive against a brick wall.
SQUEAK, the Death of Rats explained.
You know everytime the subject of hdd capacity/performance comes up I wonder why no one has produced a hdd with dual servo arms with the capacity to read/write to different sections of the disk simultaneously. More or less single disk raid 0. This would result in an incredible performance increase at a small price increase. Furthermore while raid 0 decreases storage system reliability a single disk with dual arms would actually be slightly more reliable than a standard disk; i.e. one arm fails you've got another one to keep going, (and I realize an arm failing isn't the primary cause of hdd failure). Maybe it would even be possible to optimize one arm for reading and one for writing.
I believe what has made code so bloated these days, is people believing it's more important to produce more faster and fix it later than do it right the first time.
I don't believe that I said anything about shipping broken code. Maybe I did. Maybe you could point out where. Or is your point (again!), that if you didn't write it, it can't possibly be Right(tm)?
And in your example nobody has saved $3500 on your salary alone. Every 3rd party lib you add for the sole purpose of saving time at the moment costs you about 10% when it comes time debug... you run into weird problems and can't trace them down.. why? because someone else wrote the code and now your learning THEIR code.
That 10% figure still stinks from where you pulled it out of. And if you don't think that taking a month off of a twelve month project is a *serious* gain, then I don't believe you have ever had to ship a working product.
Your app is guaranteed to be slower as well, a library that is designed to be more universal has to make sacrifices to that end.
Let's take your logic to its conclusion (without exagerating anything you've said). Your first premise is that general purpose libs are bloated and prone to error. The solution is to write your own versions, and then (as you said in the last paragraph of your first comment), you don't have to keep reinventing the wheel as you go on to other projects. Can I take that to mean that you reuse that code that you have polished? Code reuse is a good thing, I think we both agree. So, you have a lib that does something. You wrote it for app A, and then reuse it in app B. Of course, when reusing it in app B, you discover something that worked well at first, but needs to be more flexible now. So you change it a bit, and now it works equally fine in apps A and B (because you need to maintain A, that's a given). Now along comes app C, you make more changes AND WHAT THE HELL IS DIFFERENT BETWEEN YOUR LIBRARY AND ONE THAT OTHER PEOPLE WROTE??!??!!?!? You just managed to (as you so aptly put it) reinvent the wheel, making a generalized lib that can be reused but has only been 1)tested by you 2)understood by you 3)given a design review by you. How, *exactly*, is this magically faster/better/strong than something that's had people spending time optimizing it, had people looking at it from completely different angles to expose design flaws, and testing it in one thousand and one different situations, exposing bugs?
I'll give you a hint: it isn't. And the *only* reason for someone to continually and habitually refuse to use other people's libs like you've described, is pure and utter arrogance.
I'm going to bed...
Moore's law causes a problem to manufacturers - how to keep up the profit margins. I suspect this is the major driving force behind many new technologies.
For example, once disk drives are cheap enough to give everyone 100+GB local storage, we get much more expensive SANs, NAS servers and network caches.
Once complete PCs began to cost under $200, we get blade servers and micro cases to keep the price (and total profit) up.
Before the flames start, I'm perfectly aware that some people's requirements will dictate the more expensive solution. But in many cases you can go for multiply redundent cheaper devices, with higher total reliability, and still get change from the price of the "enterprise" products.
Andrew Yeomans