Turing Award Winner On The Future of Storage

weileong writes "Ars Technica highlights an interview at ACM Queue with Jim Gray, a winner of the ACM Turing award *(among other things) by one of the pioneers of RAID (among other things). Many issues touched upon, including: "programmers have to start thinking of the disk as a sequential device rather than a random access device." "So disks are not random access any more?" "That's one of the things that more or less everybody is gravitating toward. The idea of a log-structured file system is much more attractive. There are many other architectural changes that we'll have to consider in disks with huge capacity and limited bandwidth." Actual interview has MUCH detail, definitely worth reading."

Comments

dupe

[Anonymous Coward]

dupe dupe dupe

Solid state is the way to go.

[caluml]

"programmers have to start thinking of the disk as a sequential device rather than a random access device."

I think we'd all be better off when solid state, non-mechanical disks become commonplace.

Is there any reason other than cost why we can't have 100Gb solid-state drives yet?

Re:Solid state is the way to go.

[stratjakt]

Is cost not a good enough reason for you?

HDD = a buck a gig, solid-state = 100 bucks a gig.

Though supposedly magical MRAM will come along and revolutionize the world. OLED screens too. And oh yeah, Duke Nuk'Em Forever.

Re:Solid state is the way to go.

[grub]

I think we'd all be better off when solid state, non-mechanical disks become commonplace.

A company named SolidData sells solid state "drives".

Re:Solid state is the way to go.

[ananiasanom]

The point is not that solid-state will not get bigger and cheaper (it will), but that disk is getting bigger and cheaper faster.

So sure, you could replace your current 80Gb disk drive with 80Gb of solid state, but where are you going to store your 50Gb 3D movies in 1000x1000x1000 resolution? They're going to be on disk, and you'll have to deal with the increasing size:bandwidth and size:access-speed ratios. After all, I can buy a smartmedia card with the capacity of my first hard drive for about what I used to pay for a box of floppies, but I still use a hard disk.

Secondly, as others have pointed out, just as the article describes future disk behaving more like tape, future solid-state memory may behave more like disk. Where is it now? chips can pump out sequential data at close to 1 gigabit, but jumping about in memory is much slower (any expert got figures?).

Next on Slashdot

[Do not eat]

This week: You can make a trade-off between latency and throughput!
Next week: Cars that can haul less can be more fuel-effiecent!
The week after: Algorithms that use more memory, but are faster to execute!

Huge disks

[heironymouscoward]

If I look at the trends of the last decades, while disk sizes increase exponentially, the actual number of top-level objects I store on my systems increases only linearly, and quite slowly. True, I still store individual documents, but I also store AVIs, ISOs, entire photo albums that take gigabytes each.

It's still random access: I can choose and access an object, even individual photos, without scanning through large amounts of unwanted data.

Bandwidth...

[Ratface]

I love his commenta about mailing disks to Europe and Asia..

The biggest problem I have mailing disks is customs. If you mail a disk to Europe or Asia, you have to pay customs, which about doubles the shipping cost and introduces delays.

Thereby adding a corrolary to the old adage "Never underestimate the bandwidth of a vanload of tapes barrelling down the highway"...

"Never underestimate the bottleneck caused by a far-Eastern customs inspector." .-D

Let me just read your mind...

[WIAKywbfatw]

...does anybody else think this sounds familar?

I must have read an article earlier about this same thing, probably by this same guy. Can anybody confirm that?

Thanks to my well-developed powers of telepathy, I can tell you that you have read a previous article on the topic by the same author. So I'm happy to confirm that for you.

I can also tell you, thanks to my equally well-honed powers of clairvoyance, that this post will soon be modded up as funny.

(Sheesh. And I thought that some recent "Ask Slashdot" questions were dumb.)

Very much a pioneer, even IF he works for MS

[Anonymous Coward]

Check out Jim Grey's info page on Microsoft Research He's done research on many diverse and interesting technologies such as distributed computing and sequential I/O performance. There are some nifty sites he has taken part in creating, such as a browsable photo of Earth, and a map of the Universe

Network speed

[CausticWindow]

they are part of Internet 2, Virtual Business Networks (VBNs), and the Next Generation Internet (NGI). Even so, it takes them a long time to copy a gigabyte. Copy a terabyte? It takes them a very, very long time across the networks they have

Is this really true? Wasn't there a recent Slashdot story where researchers transfered a gigabyte of data, in fourteen seconds or so, on Internet 2 from California to the Netherlands?

I suppose that disk access times will be limiting factor in both ends if you were to read and write the data from/to a disk.

Re:Network speed

[CausticWindow]

Couldn't find the article with the Slashdot search, but Google produced it. Here it is.

The real numbers were 8,609 Mbps, which translates roughly into a DVD transfered every five seconds. Btw., it was Switzerland, not the Netherlands.

Also, I don't understand the part where he mentions bandwidth costs of $1 per gigabyte. Maybe you have to pay that much on the Internet 2, but my DSL costs is somewhere in the region of $0.05 per gigabyte, i figure. Maybe I'm just spoilt.

Ouch...

[cybermace5]

Frankly the interview was painful every time Dave Patterson said something. How many times does he have to ask questions about the concept of mailing a computer? "We mail computers because transferring over the Internet is too slow for these massive data transfers." "Are they computers?" "Yes." "Do you mail them?" "Yes." "It's like a movie." "Uhh ok." "Is it a whole computer that you mail?" "Yes, it is a computer full of hard drives." "Why don't you just use the Internet?" "Because it is too slow."

pr0n

[leomekenkamp]

We have a dozen doing TeraServer work; we have about eight in our lab for video archives, backups, and so on.

That's a good excuse to use on my wife: "No honey, those are my ..., uhhm..., video archives."

ACM Turing Award Winner

[m1kesm1th]

Does that mean he managed to convince someone he was a computer?

Re:ACM Turing Award Winner

[jc42]

Does that mean he managed to convince someone he was a computer?

My wife likes to tell people that her first job, back in the late 70's, was with a Civil Engineering firm in New York, where her job title was "Computer". She did the calculations (and error checking ;-) for their engineering drawings. She used machines to do this, of course, but those machines were called "calculators".

They've since changed the job title.

Funny how quickly such terminology can change.

Tweaked

[Flamesplash]

My prof talked about this in my networking class. Apparantly they tweaked the hell out of the data link layer to do this, so it was not a generic data transfer at all.

Troll in the article

[panurge]

...semi-seriously. Look at all the stuff about MySQL and Linux in the middle. It's as if a Microsoft Marketoid had suddenly taken over the interview. Or someone who didn't understand the difference between many thousands of developers working on Linux and the smaller number that work on MySQL.

Apart from speculating as to whether this attempt at FUD was the real payload of the article, did it really say anything that most of us haven't already noticed? Whether Flash or fast SCSI, we could do with an intermediate layer of backing store, with faster random access than current IDE HDDs. And we are fast heading for removable IDE drives to be a better and cheaper tape replacement. And the Internet has limited bandwidth. I'm sorry, but you don't need a Turing prize to work any of that out.

LSFS

[smd4985]

For more info on (very-cool) Log-Structed File Systems, check out Mendel's original paper at:

http://citeseer.nj.nec.com/rosenblum91design.htm l

The hierarchical object file system

[master_p]

One final thing that is even more speculative is what my co-workers at Microsoft are doing. They are replacing the file system with an object store, and using schematized storage to organize information. Gordon Bell calls that project MyLifeBits. It is speculative--a shot at implementing Vannevar Bush's memex [http://www.theatlantic.com/unbound/flashbks/compu ter/bushf.htm]. If they pull it off, it will be a revolution in the way we use storage

I've talked about it before. This guy thinks what Microsoft is doing is revolutionary. Come on all you people, can't you see the problem with today's file systems ? the problem is that the type information is lost!!! we need objects, and we need type information to be stored along those objects!!! This is the only way lots of problems will go away.

MRAM saves the day

[Markus Registrada]

All the tradeoffs will change radically when MRAM hits the streets. It's potentially denser than disk and DRAM, as fast as static RAM, nonvolatile, doesn't use power when it's not used, and can be made on regular silicon process machinery. Expect it first in cell phones next year, and then everywhere.

This doesn't just affect file storage and virtual memory. It also changes the economics of cache and main memory, and makes deployment of 64-bit CPUs more urgent. It also makes system crashes much less tolerable, because turning the computer off and on doesn't involve long shutdown and boot procedures any more.

Fuzzy numbers, or can this be right?

[abulafia]

JG Twenty-megabyte disks were considered giant. I believe that the first time I asked anybody, about 1970, disk storage rented for a dollar per megabyte a month. IBM leased rather than sold storage at the time. Each disk was the size of a washing machine and cost around $20,000.

So, one could rent a $20K device for $240/year? Those must have been the days...

That can't be right.

Defending Jim Gray

[chrisd]

I didn't really read that as fud or even invalid criticism of MySQL. Maybe I'm biased because of my previous work with Queue and since I have met Jim, but if you get the impression that Jim doesn't like MySQL (which I did not) then I would actually assume it is because he felt that way, not because of Microsoft. Jim is one of those guys that will never be looking for a job, his early work on databases were pivotable to the development of transactions and his work on fault tolerant systems is legendary, he really is beyond reproach.

Chrisd

Three letters: F, U, and D

[heironymouscoward]

Take this choice quote from the article:

My buddies are being killed by supporting all the Linux variants. It is hard to build a product on top of Linux because every other user compiles his own kernel and there are many different species.

Ain't it sweet? I count five lies:

(1) people being killed by supporting (gasp) operating systems... gosh, horror and violence, not nice at all!

(2) all the Linux "variants", are in fact pretty much one standard, LSB, with several skins

(3) "hard to build a product on top of Linux", rather than, hmmm, Windows? Linux is incredibly easy to build for. I suspect the fact that it's very standard helps.

(4) "every other user compiles his kernel"... maybe at Microsoft. I suspect less than 1 in 20 Linux users ever compiled a kernel.

(5) compiling a kernel means you can't support it... WTF? The kernel is incredibly stable, since most changes are in external modules. And I can't remember a single case where a kernel change broke one of my apps.

(6) (sorry, I was not counting well), "many different species"... well, AFAICS the only difference between the Linux distributions is that they have different packaging methods, different timelines as to their versions, and different UI tools for hardware detection, configuration, etc. Nothing at all that makes life hard.

Look: I just installed Xandros, which is Debian with a nice face. On two different types of machine, and it installed without asking a single question about my hardware except whether the mouse was left or right-handed. Check my journal...

Windows never worked this nicely. Where is the support issue?

In the writing indistry we call this "to condemn with faint praise".

Yeah, Windows kinda works, I mean, it'll run Office without crashing too often, but it's just killing by buddies to have to maintain Win2K, WinXP, and even some older Win98 machines, not to mention we have a whole cupboard simply filled with driver CDs for every PC we have.

3 Terrabytes on a credit card?

[polyp2000]

Anyone know what happened to that bloke at keele who
invented a way of cramming 3 Terrabytes on a credit card. Apparently it would have cost about 35 pounds to manufacture. this was a couple of years ago, why hasnt it happened yet?

Surely something like this is the real future of storage ?

Terrabyte on a credit card

Sneaker net?

[computerlady]

"Sneaker net" was when you used your sneakers to transport data?

Oh my. How old I feel when someone has to ask what "sneaker net" was. And someone has to answer...

AMAZING!!!

[X86Daddy]

This is a *MAJOR* breakthrough! Most Turing Test contestants don't even win, but this one can eloquently discuss topics and give complex answers, rather than just turning back the question, Eliza-style.

Can we download a copy of this "Jim Gray" yet?

IDE replaces DVD

[G4from128k]

With an ever growing collection of digital photos, I've come to the same conclusion as Jim Gray. Hard disks are superior for backups.

I currently have about 100 GB of images and it takes more than 20 4.7 GB DVD-R discs to create a full backup. Although DVD media is still slightly cheaper than new large capacity IDE drives, the added time and hassle factor of burning 20 disks far out weighs any minor costs savings. Moreover a 3.5" drive in a padded anti-static bag takes up less room in the safe deposit box than 20 DVDs (especially if you have the DVDs in protective jewel cases). And if HD-based-backup lets me avoid some future artists tax on burnable media, so much the better.

A Firewire enclosure and a rotating collection of IDE drives is the way to go.

Interesting Idea...

[polyp2000]

Interesting thought popped when i read your post,
there is a current trend towards cramming as much storage into something the size of a 3in Hard drive.

I wonder why they dont make larger harddrives in the physical sense? A hard drive the size of a washing machine using todays technology would store a phenomenal amount of stuff, but whatabout something more reasonable like a hard drive merely twice the physical size of todays. how much more storage could you get just by scaling up the platters? anyone here good at math . Hard drives today must be up to 200-250gb.

He is right, but nothing to do with the kernel

[brunes69]

His basic idea is 100% correct, but the reson is all wrong. It *IS* much harder to develop an app Linux the myriad of flavours, not because of the kernel, but because every distro has its own versions of libraries. I work for a company that makes Linux software, and we only support RedHat, and even certain versions of RedHat at that. While our product would probably compile against any number of distros, and even the BSDs, we just don't have the time and manpower required to build, test, debug, package, and maintain 15 different releases for every sub-release or patchlevel we have in the product. With Windows products, at least, (unless you are doing some lower-level stuff) if you build something you can be reasonably assured it will run on Windows 2000, or Windows XP, or Windows 2003. Not the same if you build something with RedHat 9 and try to run it on Debian or Suse, etc. And before you go on about "release a source package", not all companies release everything GPL, and want to keep their IP theirs, since they like to put some money on the table at night. It's definitly not FUD to say it is much more effort to develop and release cross platform binaries in Linux than Windows.