Turing Award Winner On The Future of Storage

weileong writes "Ars Technica highlights an interview at ACM Queue with Jim Gray, a winner of the ACM Turing award *(among other things) by one of the pioneers of RAID (among other things). Many issues touched upon, including: "programmers have to start thinking of the disk as a sequential device rather than a random access device." "So disks are not random access any more?" "That's one of the things that more or less everybody is gravitating toward. The idea of a log-structured file system is much more attractive. There are many other architectural changes that we'll have to consider in disks with huge capacity and limited bandwidth." Actual interview has MUCH detail, definitely worth reading."

Comments

dupe

[Anonymous Coward]

dupe dupe dupe

Re:dupe

[fermion]

get back to work!

Or are you in fact a government agent paid to monitor the alleged seditious activities occurring on /. Or even worse, an agent of the RIAA.

If you are engaging in such activities under commission, then i can understand why ups are so distressful to you.

Re:dupe

[miruku]

thought it sounded familiar...

Re:dupe

[Anonymous Coward]

I'm suffering Alzheimer, you insensitive clod !

Re:dupe

[zulux]

dupe dupe dupe [slashdot.org]

We know it a dupe - but we're simulating stream storage. We can't use that old random access thingy to go back. That would be cheating.

Re:dupe

[K-Man]

Well, yes, but, according to Moore's law, we now have 36% more storage to store the dupe.

Re:dupe

[Anonymous Coward]

Does timothy ever manage to post an original story, or is every single one a dupe?

This sounds familar..

[grasshoppa]

...does anybody else think this sounds familar?

I must have read an article earlier about this same thing, probably by this same guy. Can anybody confirm that?

Re:This sounds familar..

[Anonymous Coward]

Probably on google. They have all the cool stories first

Solid state is the way to go.

[caluml]

"programmers have to start thinking of the disk as a sequential device rather than a random access device."

I think we'd all be better off when solid state, non-mechanical disks become commonplace.

Is there any reason other than cost why we can't have 100Gb solid-state drives yet?

Re:Solid state is the way to go.

[stratjakt]

Is cost not a good enough reason for you?

HDD = a buck a gig, solid-state = 100 bucks a gig.

Though supposedly magical MRAM will come along and revolutionize the world. OLED screens too. And oh yeah, Duke Nuk'Em Forever.

Re:Solid state is the way to go.

[grub]

I think we'd all be better off when solid state, non-mechanical disks become commonplace.

A company named SolidData sells solid state "drives".

Re:Solid state is the way to go.

[Anonymous Coward]

Yeah, you're right to be skeptical, look at the past thirty years and how many revolutionary new technologies do you see in computers? If technological breakthroughs were for real, we wouldn't still be using these damn TRS-80s.

I'm so frickin tired of this cassette drive!

Re:Solid state is the way to go.

[caluml]

Grub - have a look on my homepage at link nr. 2. It's truely horrible, and I have you to blame for it :)

Re:Solid state is the way to go.

[grub]

Woo! I'm fay-moose! ;)

Re:Solid state is the way to go.

[stratjakt]

Technological breakthroughs?

Faster CPUs, bigger HDDs, and denser RAM, but fundamentally its the same type of shit.

Optical drives (CDRoms) and LCD monitors are about the only new techs I can think of, everything else is merely improving upon the old.

Re:Solid state is the way to go.

[Serious Simon]

Is there any reason other than cost why we can't have 100Gb solid-state drives yet?

If the price is not an issue, I'll start working on it right now.

Re:Solid state is the way to go.

[Anonymous Coward]

Laser & Inkjet printing, which is almost (but not quite) entirely unlike a computer.

Lets see, thirty years. Ethernet is another one. If we include software, then the WIMP GUI is a biggie. Which of course required bitmapped displays, which have lead to hardware rendered 3D, which have lead to CAVE systems.

There are plenty of advances, you're not looking hard enough.

Re:Solid state is the way to go.

[torpor]

The costs involved are chiefly ecological.

Chip-manufacturing is bad voodoo in the chemicals and pollutants department.

Re:Solid state is the way to go.

[ikkonoishi]

So do you think the diamond chips will help with that?

I know they are fast, but can they be used as permanent solid state storage?

Re:Solid state is the way to go.

[mblase]

Is cost not a good enough reason for you?

I think someday cost will be less of an issue than convenience. Think of the state of monitors today: LCD sales are going well, and while they haven't replaced CRTs yet, they're on their way. Apple no longer sells CRTs at all. This is despite the fact that CRTs are cheaper for the same size screen, because LCDs have a significant edge in size, weight and power consumption.

Flash memormay be around $100/GB right now, but if that drops low enough (say $20/GB), it'll be enough to replace HDDs regardless of how cheap the HDDs are, because (again) the convenience of size and energy consumption outweigh a small enough difference in price.

Re:Solid state is the way to go.

[ananiasanom]

The point is not that solid-state will not get bigger and cheaper (it will), but that disk is getting bigger and cheaper faster.

So sure, you could replace your current 80Gb disk drive with 80Gb of solid state, but where are you going to store your 50Gb 3D movies in 1000x1000x1000 resolution? They're going to be on disk, and you'll have to deal with the increasing size:bandwidth and size:access-speed ratios. After all, I can buy a smartmedia card with the capacity of my first hard drive for about what I used to pay for a box of floppies, but I still use a hard disk.

Secondly, as others have pointed out, just as the article describes future disk behaving more like tape, future solid-state memory may behave more like disk. Where is it now? chips can pump out sequential data at close to 1 gigabit, but jumping about in memory is much slower (any expert got figures?).

Re:Solid state is the way to go.

[Carbonite]

Flash memormay be around $100/GB right now, but if that drops low enough (say $20/GB), it'll be enough to replace HDDs...

There's no way that people will pay $20/GB for primary storage. The cost of a HDD is around $1/GB and dropping fast. It would be exceedingly difficult to convince people to pay a 100% premium (2x the price) for solid state storage. $20/GB would be a 1900% price premium! Smaller size and lower energy consumption are all very nice and good, but $2000 for a 100GB drive seems a little steep.

Re:Solid state is the way to go.

[vidarh]

What "convenience in size"? Solid state storage is nowhere near achieving the densities of hard disks. I also doubt they provide an energy consumption advantage, especially if you use DRAM based solid state storage solutions (which, btw. are usually delivered in the form of large rack mounted boxes full of DIMMs with a tiny little corner occupied with a harddisk used to dump the data to if the box switches to UPS or for other reason may soon loose power).

Sure, they may eventuelly catch up, but the attractive part of solid state storage devices today is access speeds, not size or energy consumption.

That said, access speeds can be adressed by striping (unless latency is a big issue) over a large number of disks, as you usually have big data sets if you need high IO throughput (if not, just cache everything in RAM...).

However, IO bandwidth of typical servers is low, and you'll often find that the problem isn't pure disk bandwidth, but making sure you don't have IO bottlenecks throughout your installation (everywhere from bus, to networking infrastructure, to good enough storage controllers). Servers these days are usually optimized for processing power, not IO - and that even goes for lots of servers being sold as file servers...

Re:Solid state is the way to go.

[Anonymous Coward]

Well, for one thing, it's only because of a fundamental breakthrough in physics by IBM that those giant hard drives are possible. For a while it looked like they were going to top out at a gig or two. That's why, in 1990, everybody was looking for alternatives like 3-d optical storage...then we had IBM's breakthrough instead, and the other stuff wasn't worth the effort anymore.

Re:Solid state is the way to go.

[MagikSlinger]

That's just not fair! I'm positive OLED screens and MRAM will actually see the light of day. Comparing it to the DNF is just plain mean.

Re:Solid state is the way to go.

[Wiwi Jumbo]

But on the bright side, chances of mechanical failure would be greatly reduced, no? :)

Tho' I still don't think that would be enough to justify it.

I'm actually afraid to move to the higher Gig'ed drives, I don't backup enough now and larger drives will just let me put it off even longer.

Having 30 Gig's die would be bad. Having 100 snuffed out could kill me.

Re:Solid state is the way to go.

[ipjohnson]

Well DNF is better than DFL :)

Re:Solid state is the way to go.

[mblase]

Solid state storage is nowhere near achieving the densities of hard disks.

Perhaps not now. But I can buy a CompactFlash card in capacities from as little as 8MB to as large as 4GB today, and they're all the same physical size. It's just a matter of time (and cost) before 40GB CF cards are available to the masses.

Re:Solid state is the way to go.

[mblase]

but $2000 for a 100GB drive seems a little steep.

So you get a 25GB drive for $500. If you take out all the MP3s and MPEGs and 3D videogames from your typical PC, you don't need a third of the harddrive space any longer. And a flash drive at 25GB would be ideal for an ultra-lightweight laptop, especially since the boot and sleep time would be next to nothing and battery life would be dramatically extended compared to a HDD laptop.

Don't like it? Wait a bit longer until flash memory drops to $5/GB and you can get your 100GB drive for the same price.

DUPE

[Anonymous Coward]

you may have seen this somewhere....

Next on Slashdot

[Do not eat]

This week: You can make a trade-off between latency and throughput!
Next week: Cars that can haul less can be more fuel-effiecent!
The week after: Algorithms that use more memory, but are faster to execute!

Huge disks

[heironymouscoward]

If I look at the trends of the last decades, while disk sizes increase exponentially, the actual number of top-level objects I store on my systems increases only linearly, and quite slowly. True, I still store individual documents, but I also store AVIs, ISOs, entire photo albums that take gigabytes each.

It's still random access: I can choose and access an object, even individual photos, without scanning through large amounts of unwanted data.

Re:Huge disks

[Llurien]

Interesting point. I guess that's partly because a human collects stuff in a more or less linear fashion. Everything you collect, create or use takes time, and time is a resource that we don't get more of simply because our computers get faster. It is possible to handle one single 4 GB file such as a movie, but it would be impossible to do something meaningfull with 4000 1MB files, it would simply take too much time. Offcourse, you could think of automated tasks operating on large sets of files, but again random access would serve no benefit here. Throughput is important in the case of a program handling a sequence of small files.

Re:Huge disks

[Anonymous Coward]

Bah humbug.. they can double or quadruple the access times NOW. Here's the idea: add more drive arms per disk. With command queuing each arm could move independantly to the nearest sector.

Re:Huge disks

[Bingo Foo]

I have an easier solution! They could double or quadruple access times now by just putting some kind of wait statement in the drive firmware.

Re:Huge disks

[Dwonis]

You obviously don't use maildirs. ;-)

dwon@rivest:~/MyMail/boxes$ find -type f | wc -l
81966

Bandwidth...

[Ratface]

I love his commenta about mailing disks to Europe and Asia..

The biggest problem I have mailing disks is customs. If you mail a disk to Europe or Asia, you have to pay customs, which about doubles the shipping cost and introduces delays.

Thereby adding a corrolary to the old adage "Never underestimate the bandwidth of a vanload of tapes barrelling down the highway"...

"Never underestimate the bottleneck caused by a far-Eastern customs inspector." .-D

Re:Bandwidth...

[Anonymous Coward]

Lie about the contents and say it's a set of pictures or something. That's what I do when I send vcds full of movies off to my girlfriend in Japan. Works everytime.

Re:Bandwidth...

[Anonymous Coward]

I think he meant customs IN GENERAL, not far-eastern in particular. And a likely place that a disk would be sent to would be a first world nation - mainly Europe and Asia came to his mind. If I say "never underestimate the bottleneck caused by an Australian customs inspector," it won't be edgy anymore.

Let me just read your mind...

[WIAKywbfatw]

...does anybody else think this sounds familar?

I must have read an article earlier about this same thing, probably by this same guy. Can anybody confirm that?

Thanks to my well-developed powers of telepathy, I can tell you that you have read a previous article on the topic by the same author. So I'm happy to confirm that for you.

I can also tell you, thanks to my equally well-honed powers of clairvoyance, that this post will soon be modded up as funny.

(Sheesh. And I thought that some recent "Ask Slashdot" questions were dumb.)

Re:Let me just read your mind...

[grasshoppa]

I can also tell you, thanks to my equally well-honed powers of clairvoyance, that this post will soon be modded up as funny

Don't flatter yourself.

Re:Let me just read your mind...

[rabidcow]

Ooh, you're good. Now can you tell me where my keys are?

Re:Let me just read your mind...

[Anonymous Coward]

Look under the middle couch cushion.

I tried to use a tape drive this way :-)

[192939495969798999]

When I was really young, I tried to use a tape drive as a sequential hard-disk drive. I figured since the tape drive was sequential access it would work... let's just say it didn't go real fast. I tried to run an EXE from it, on probably a 386 mind you, and yeah. The laundry got done before the EXE ran! (SIGH) Was I *ever* that young?
Still, I am glad to see that the paradigm is now realizeable.

Re:I tried to use a tape drive this way :-)

[192939495969798999]

My creative-art abilities help in situations like thinking to use a sequential access system, but knowing jack about the subsystems would have been more beneficial in actually implementing that system :D

Re:I tried to use a tape drive this way :-)

[tigersha]

Reminds me of the time Windows NT4 persisted in creating a Swap file on my 100 MB ZIP disk which it though to be a hard disk.

I was really wondering why my !"!" machine was suddenly to totally slow. Fortunately they fixed this in some later SP.

Very much a pioneer, even IF he works for MS

[Anonymous Coward]

Check out Jim Grey's info page on Microsoft Research He's done research on many diverse and interesting technologies such as distributed computing and sequential I/O performance. There are some nifty sites he has taken part in creating, such as a browsable photo of Earth, and a map of the Universe

Re:Very much a pioneer, even IF he works for MS

[Anonymous Coward]

> such as a browsable photo of Earth

Earth is a lot bigger than the USA.

Re:Very much a pioneer, even IF he works for MS

[Tarrio]

Browsable photo of Earth? Oh, yes, there's nothing in Earth out of the USA, except for that one guy who licks Bush's shoes clean. Mr. Answer, or something like that.

Network speed

[CausticWindow]

they are part of Internet 2, Virtual Business Networks (VBNs), and the Next Generation Internet (NGI). Even so, it takes them a long time to copy a gigabyte. Copy a terabyte? It takes them a very, very long time across the networks they have

Is this really true? Wasn't there a recent Slashdot story where researchers transfered a gigabyte of data, in fourteen seconds or so, on Internet 2 from California to the Netherlands?

I suppose that disk access times will be limiting factor in both ends if you were to read and write the data from/to a disk.

Re:Network speed

[Anonymous Coward]

14 seconds? Aww, I want it now.

Even if thats true, it's still about 4 hours a terabyte. And that, of course, is when you have all the bandwidth to yourself.

I mean, if there's only two people on the network it can go pretty fast.

Re:Network speed

[CausticWindow]

Couldn't find the article with the Slashdot search, but Google produced it. Here it is.

The real numbers were 8,609 Mbps, which translates roughly into a DVD transfered every five seconds. Btw., it was Switzerland, not the Netherlands.

Also, I don't understand the part where he mentions bandwidth costs of $1 per gigabyte. Maybe you have to pay that much on the Internet 2, but my DSL costs is somewhere in the region of $0.05 per gigabyte, i figure. Maybe I'm just spoilt.

Re:Network speed

[interiot]

I don't quite understand network costs either. I work at a Fortune 100 company, and supposedly they pay 4 cents per megabyte that goes to/from the internet, or $41 a gig. Certainly there are firewall / antivirus / constantly-on-call-net-admin costs included in there, but I've always been puzzled at the difference between my cable modem costs and my workplace's costs.

Now certainly, the broadband companies don't expect you to be downloading at maximum speed constantly, and if you were you'd be in the top 0.0001% of bandwidth users so they may very well find a reason to boot you. So you can't really say your bandwidth costs $0.05 per gigabyte. If you pay $50 a month for broadband, then apparently the broadband people think the average user will use less than 50 gig of bandwidth a month, which I think is a good bet.

Re:Network speed

[russianspy]

I've seen the presentation by one of the people involved.

They basically got alpha versions of a 10Gig ether cards from intel. Got the ISP company go give them a light path directly between two points. I think they got one of the underwater cables entirely to themselves for a few hours or something like that. The cost for that was what makes it so expensive.

Yes, they did tweek the heck out of that software (by the way, they did use linux - don't remember which distro).

Re:Network speed

[vidarh]

$41 !?!?!?!?!?

You can easily buy bandwidth in the sub $1 pr gigabyte when buying bandwidth per gigabyte transferred to a colo. If you need to lease physical lines to your office, the cost may end up a bit higher. $41 per GB sounds like someone has been smoking crack.

The reason there's a gap for your DSL, though, is exactly as you mention - that most users only utilize a fraction, so you're only paying for the average amount transferred per user plus some contingency.

Re:Network speed

[interiot]

Like I said, being a rather large company, we have a large restrictive firewall (only telnet and http/s go out, and even then only with a password) and spam/virus detection and such, and we need 100% uptime connectivity to our other sites around the world. Even a single ethernet drop at a desk costs each department $30 / month. That has to be administrative overhead, so I'm pretty sure that's where the $41 / gig comes from. Though a 41 times increase seems a bit much, eh?

Ouch...

[cybermace5]

Frankly the interview was painful every time Dave Patterson said something. How many times does he have to ask questions about the concept of mailing a computer? "We mail computers because transferring over the Internet is too slow for these massive data transfers." "Are they computers?" "Yes." "Do you mail them?" "Yes." "It's like a movie." "Uhh ok." "Is it a whole computer that you mail?" "Yes, it is a computer full of hard drives." "Why don't you just use the Internet?" "Because it is too slow."

Re:Ouch...

[JonathanBoyd]

Or:

I hadn't thought about it the way you explained it. It isn't that the access times have been improving too slowly; it's that the capacity has been improving too quickly.

He seems to be suggesting that rather than try to make access quicker, we should stop making hard drives bigger. ?!?!

Re:Ouch...

[Anonymous Coward]

If anyone says "never underestimate the bandwidth of a $vehicle full of tapes/discs/hard drives (pick one)" just pretend I punched them in advance.

Re:Ouch...

[AJWM]

I wonder why they didn't go with removable harddrives. Or external drives. No reason to mail all the other crap that goes with the computer.

(That's the solution we came up with for sending updated DV files to a site in Australia: a handful of 80 GB drives in caddies that we kept swapping by FedEx.)

pr0n

[leomekenkamp]

We have a dozen doing TeraServer work; we have about eight in our lab for video archives, backups, and so on.

That's a good excuse to use on my wife: "No honey, those are my ..., uhhm..., video archives."

Jim Gray is a jerk

[Seth Finklestein]

Jim Gray? Why the fuck should I believe anything from a man who demonized future Hall of Famer Pete Rose?

And what does Jim Gray know about storage? He's a sports commentator, and a terrible one at that.

Research

[Anonymous Coward]

According to research at an English university, many people store data on magnetic disks.

ACM Turing Award Winner

[m1kesm1th]

Does that mean he managed to convince someone he was a computer?

Re:ACM Turing Award Winner

[jc42]

Does that mean he managed to convince someone he was a computer?

My wife likes to tell people that her first job, back in the late 70's, was with a Civil Engineering firm in New York, where her job title was "Computer". She did the calculations (and error checking ;-) for their engineering drawings. She used machines to do this, of course, but those machines were called "calculators".

They've since changed the job title.

Funny how quickly such terminology can change.

Re:ACM Turing Award Winner

[GypC]

Nobody has a job as a computer anymore... they've all been replaced by computers.

./ed: full text

[Anonymous Coward]

Fatal error: Call to undefined function: message_die() in db/db.php on line 88

Tweaked

[Flamesplash]

My prof talked about this in my networking class. Apparantly they tweaked the hell out of the data link layer to do this, so it was not a generic data transfer at all.

The van metaphor

[ArmenTanzarian]

I've seen this a couple times before, but Google seems to come up with nothing useful for it. It doesn't help that every crappy musician who has made a tape sells it out of their crappy van or that so many scientist have the old prussion "van der something" in their names. But perhaps it's crappy musicicans and these van der scientists who really control the highspeed data transfer.

Re:The van metaphor

[jpop32]

I've seen this a couple times before, but Google seems to come up with nothing useful for it.

It's from:

Andrew S. Tannenbaum. Computer Networks. Prentice Hall, third edition, 1996.

A de facto bible of computer networks. Had it as a textbook in college. You're bound to run into it if you ever get to formally studying networks.

Re:The van metaphor

[roguerez]

Make that Tanenbaum (one n).

My methodology

[Anonymous Coward]

I started long ago doing this. I treat my drives as sequential and my code as random access by applying random number functions to all my pointers. I find that even my most complex algorithms now finish almost instantly!

2 quotes...

[leomekenkamp]

Two quotes from the article (emphasis mine):

Gray, head of Microsoft's Bay Area Research Center, sits down with Queue and tells us (...)

JG: If it is business as usual, then a petabyte store needs 1,000 storage admins. Our chore is to figure out how to waste storage space to save administration.

MS bashers will have a field day on this one...

Re:2 quotes...

[Shimbo]

JG: If it is business as usual, then a petabyte store needs 1,000 storage admins.

Tell that to the high energy physics community; they use petabyte size stores as local caches.

Re:2 quotes...

[Anonymous Coward]

Microsoft has yet to find out about object databases...

Re:2 quotes...

[Anonymous Coward]

Only if they are complete fucking retards.
His whole point was that disk space is cheap, while administration is expensive. So it makes perfect sense to waste space to save on administration. RTFA.

Re:2 quotes...

[Anonymous Coward]

Well, they are. So that works out nicely.

Troll in the article

[panurge]

...semi-seriously. Look at all the stuff about MySQL and Linux in the middle. It's as if a Microsoft Marketoid had suddenly taken over the interview. Or someone who didn't understand the difference between many thousands of developers working on Linux and the smaller number that work on MySQL.

Apart from speculating as to whether this attempt at FUD was the real payload of the article, did it really say anything that most of us haven't already noticed? Whether Flash or fast SCSI, we could do with an intermediate layer of backing store, with faster random access than current IDE HDDs. And we are fast heading for removable IDE drives to be a better and cheaper tape replacement. And the Internet has limited bandwidth. I'm sorry, but you don't need a Turing prize to work any of that out.

Re:Troll in the article

[stratjakt]

Whats his yammering about ATA and SCSI being supplanted by TCP/IP? Gimme a break. Why would a hard drive, or any other peripheral need all the bloat of a routable network protocol?

Sorry fella, I want a fast cheap hard drive, not one with buzzwords all over it.

Re:Troll in the article

[mr_majestyk]

You're the troll. Jim Grey wrote the book on transaction processing. If there is anyone qualified to judge the capabilities of a DB system it is him.
Or someone who didn't understand the difference between many thousands of developers working on Linux and the smaller number that work on MySQL.
WTF??? His comments show a clear understanding of how many developers are working on MySQL ("Twenty-five people can do a pretty full-blown system, and ship it, and support it, and get manuals written, and test it. The Postgress and MySQL teams are on that scale and likely represent the leading open-source DBMSes out there."). It's like you saw the word "Microsoft" and automatically assumed everything following would be Market-speak.

Re:Troll in the article

[Anonymous Coward]

According to research at an English university, Linux is open-source.

Re:Troll in the article

[panurge]

I'm going to confess that I have probably misunderstood the point. The precise bit of the article I was referring to was:

The challenge is similar to the challenge we see in the OS space. My buddies are being killed by supporting all the Linux variants. It is hard to build a product on top of Linux because every other user compiles his own kernel and there are many different species. The main hope for Oracle, DB2, and SQLserver is that the open-source community will continue to fragment. Human nature being what it is, I think Oracle is safe.

DP Is MySQL.com trying to be the Red Hat of MySQL?

JG It could be that they will step forward and provide all of those things that IBM, Microsoft, and Oracle provided, and do it for a much lower price. I think the incumbent vendors will have to be innovative to make their products more attractive.

One thing that works in the incumbents' favor is fear, uncertainty, and doubt (FUD). If you base your company on a database, you are risking a lot. You want to buy the best one. People are usually pretty cautious about where they want to put their data. They want to know that it's going to have a disaster recovery plan, replication, good code quality, and in particular, lots and lots and lots of testing.

The thing that slows Oracle, IBM, and Microsoft down is the testing, and making sure they don't break anything--supporting the legacy. I don't know if the MySQL community has the same focus on that.

At some point, somebody will say, "I'm running my company on MySQL." Indeed, I wish I could hear Scott McNealy [CEO of Sun Microsystems] tell that to Larry Ellison [CEO of Oracle].

DP The whole corporation?

JG Right. Larry Ellison announced that Oracle is now running entirely on Linux. But he didn't say, "Incidentally we're going to run all of Oracle on MySQL on Linux." If you just connected the dots, that would be the next sentence in the paragraph. But he didn't say that, so I believe that Larry actually thinks Oracle will have a lot more value than MySQL has. I do not understand why he thinks the Linux problems are fixable and the MySQL problems are not.

I was concentrating on his claims that building a system on top of Linux is particularly hard, and his mentioning Microsoft in the same sentence as IBM and Oracle. Although I make extensive use of MySQL for small systems in our consultancy, I think it is a long way from being ready for the main enterprise RDBMS. In fact, I felt he was trying to tar Linux with the MySQL brush, if you see what I mean.

I now think he probably did not mean it the way I read it. If anyone cares to mod my original post down, feel free. But I do think that, for a long article, there was actually not a lot of real content.

Re:Troll in the article

[Doom Ihl' Varia]

Last time I checked you couldn't connect to your ATA hard drive from multiple sources without software. It is a similiar idea to the printers that you can just hook up a CAT5 to and any computer on the network with SAMBA can use it. Just have a cabinet that is basically a big RAID array and hook up a cable to the ethernet port and there you go.

Re:Troll in the article

[zangdesign]

Personally, I find it to be rather telling evidence that a small team is more efficient than a large mob. With a small team of people, decisions can be made much more rapidly - they may later turn out to be wrong decisions, but it takes less time to disseminate information and corrections to the other team members.

As for FUD about MySQL, I don't see it in the article. MySQL is lacking some features that keep it from competing in the same spaces as Oracle, but that's a decision on the part of the MySQL team, not an oversight, AFAIK.

Re:Troll in the article

[NineNine]

Look at all the stuff about MySQL and Linux in the middle. It's as if a Microsoft Marketoid had suddenly taken over the interview. Or someone who didn't understand the difference between many thousands of developers working on Linux and the smaller number that work on MySQL.

His point was simply that DB2 and Oracle, and, to a lesser extent, SQL Server, are mature database products, and MySQL is just a baby. "Real" databases are optimized like crazy for each OS version/CPU combination. MySQL isn't there yet. That's his point. Now go back under your bridge.

Re:Troll in the article

[leandrod]

> Look at all the stuff about MySQL and Linux in the middle. It's as if a Microsoft Marketoid had suddenly taken over the interview. Or someone who didn't understand the difference between many thousands of developers working on Linux and the smaller number that work on MySQL.

He's correct as far as he goes.

MySQL and MS SQL Server actually have the same problem, and it is called SQL; both even go downhill from there.

SQL is simply too complex to implement properly, and it only gets worse when you start with a non-standard implementation. While MySQL benefits from a better OS to run on, it has the more fundamental flaws that its developers don't really understand data in general and the relational model in particular, and it has started with something that wasn't really SQL at all and not a DBMS at all; MS began with something that was a real if weak DBMS, and almost SQL already, and since has hired some pretty good guys and improved impressively.

Eventually MySQL will reach maturity, and with more ports, a more variated and complicated legacy and less understanding, it will have a rougher time developing the future and supporting the past. See that MySQL's current idea of future is SAPdb, which is stuck with Oracle v7 feature parity and less-than-SQL 92 Entry Level compliance.

Obviously PostgreSQL is a better base to build on, it originally was even better than SQL (Ingres QUEL was based on Codd's own Alpha), since it got into the SQL cult it was never as unfaithful as MySQL is now or even SQL Server was, and PostgreSQL was always a real DBMS. Too bad the exposition it gets is so small Mr Gray can't even spell its name or see its superiority.

Re:Troll in the article

[zog karndon]

As I read the article, it's not that building a system on top of Linux is hard, it's that supporting a system on top of Linux is hard.

Different distributions have different package systems, filesystem organization, runtime libs, etc. I'm not saying that this is necessarily a bad thing, but it makes supporting J. Random LinuxUser a lot harder.

Re:Troll in the article

[JoeBuck]

Actually, I read it the other way around: it's an attack on Oracle. Larry Ellison says that Linux is great, and everyone will be running Oracle on top of Linux. Linux still has the odd problem or two, but Larry's confident that they will be fixed.

But if Linux can displace Unix and Windows, why can't MySQL displace Oracle? If you ask Oracle for why, their reasons sound pretty much exactly the same as the stuff McNealy will tell you about Solaris.

Re:Some asshat on the future of blah blah

[SoTuA]

hamburgers will eat people!

Bah, check your history books. It has happened already! (in Soviet Russia, of course ;)

LSFS

[smd4985]

For more info on (very-cool) Log-Structed File Systems, check out Mendel's original paper at:

http://citeseer.nj.nec.com/rosenblum91design.htm l

Re:LSFS

[kscguru]

Same paper, directly in PostScript format.

Really good idea. The canonical criticisms, as described by OS teachers I had (hint: one of them WAS Mendel...):

• Unnecessary - Unix FFS improved (a few years after LSFS came out) by adding clusters and cylendar clustars, reaching almost the same performance.
• CPU-intensive. Requires a background daemon to reclaim disk space (~10% of disk access was this daemon, IIRC). Being Slashdotters who hate even the CPU cycles Winmodems consume...
• Poor performance in common cases. LSFS is lightning-fast writing, comprable to other filesystems reading small files, and dog slow reading large files because of how they fragment. When the paper was written, writes and small reads (logs) were very important; lately, large reads (e.g. on-disk databases/swap, large datasets) have become more important.

Not criticizing - it's a brilliant paper. But I think it would take a few more generations before LSFS could overcome some of the inherent problems - and Mendel's too busy having fun with VMWare now.

Wait

[Lane.exe]

I guess this makes a convoluted sense -- a huge hard drive with large chunks of data written on it would be hard to access, given current speeds. But I don't see this idea catching on at the end-user desktop level. The average Joe doesn't have corporate volumes of data, and he's not going to spend half a day in sequential access looking for his pron or his Word files.

Not to say that large, sequential access hard disks aren't a good idea for archives or corporate data. It'll cut out the need for tape drives. But this will never fly with the home user.

Re:Wait

[stratjakt]

I think the deal is harddrives are getting so big, so fast, and noone has a use for all that space.

Back in the days of 20 meg drives, pretty much all of it was in use by your OS, apps, and data files. Now that you have 200 gig drives, you really "use" about 10 gigs for that stuff, and the rest is archival, storing ISOs or downloads or pron or mp3s or whatever.

I have a 120gig at home, and its full up, but I really only need about 10 gigs or so of the stuff thats there. This is already "flying" with the home user.

The stuff is staying on disk, when in the olden days it would be moved to a tape or floppy or /dev/null.

I really dont get what all this sequential access jibber-jabber is about though. I guess he's talking about backing up an important directory with TAR. I think a lot of this interview is these two simply in love with the sound of their own voices.

Re:Wait

[Anonymous Coward]

There are multiple levels of access within a file system. The sequential versus random decisions they are talking about is at a much lower level than you are thinking. Somewhat simplified:

Now, when software opens a file, it gets a handle to the storage and seeks all over it to get the data it needs and finally write it back. This is particularly true of files that consist of many records. Some software mmaps (memory maps) the file, mapping it into the memory address space and making it appear as a large, slow section of RAM in order to make this easier.

Relatively recently, you see many more programs which open a file, slurp the entire thing into memory, and close the file on disk. When they want to make changes, they open the file again and rewrite it from scratch. You see this more in text editors and word processors. Programming editors will often have some alternate behavior for very large files, although the threshhold for "very large file" is always increasing.

When you do this with record oriented files and or incremental save/autosave, etc, you get into journalling. You write all of the user's changes sequentially to a log file rather than saving the actual file (and re-writing it) repeatedly. This is sometimes what you are seeing when a program has a 'recovery file'. Having only one recovery file or journal for any number of open files means you are consistently writing appends to a single location and avoiding disk seeks.

What the article is getting at is that this sort of behavior will get more and more common, even moving into the FS and OS level. Support for this kind of journalling may move its way into FS handling, for instance. Also, instead of opening individual files, the FS may block transfer a whole directory into RAM at once. We already see this with advanced file systems which store small files directly in the directory inode. We may see the inodes get larger and the definition of 'small file' become steadily larger. When you have GBs of RAM and TB of storage, why not have a 64 MB+ inode?

From this point of view, random seeking within files slowly becomes irrelevent. Rather, the primary operations become streaming and append.

Re:Wait

[tigersha]

You might be surprised at the amount of effort people would put up with in pursuit of their Pron :)

But he's a Microsofty!

[KingDaveRa]

...Gray, head of Microsoft's Bay Area Research Center...

And here he is singing the praises of open source software, MySQL, Linux, Posgresql, Oracle, IBM etc! He'll most likely be getting a visit from Balmer in person I think. Obviously the brainwashing didn't work on this guy.

TAOCP

[Papineau]

So I guess the disk algorithms from Knuth's TAOCP are still useful after all those years?

Re:TAOCP

[falzer]

This is unrelated, but what the hell:

Microsoft buys TeX!

Don Knuth finally sells out.

MOD PARENT DOWN

[Anonymous Coward]

Last link doesn't lead to 'a map of Universe'.

The hierarchical object file system

[master_p]

One final thing that is even more speculative is what my co-workers at Microsoft are doing. They are replacing the file system with an object store, and using schematized storage to organize information. Gordon Bell calls that project MyLifeBits. It is speculative--a shot at implementing Vannevar Bush's memex [http://www.theatlantic.com/unbound/flashbks/compu ter/bushf.htm]. If they pull it off, it will be a revolution in the way we use storage

I've talked about it before. This guy thinks what Microsoft is doing is revolutionary. Come on all you people, can't you see the problem with today's file systems ? the problem is that the type information is lost!!! we need objects, and we need type information to be stored along those objects!!! This is the only way lots of problems will go away.

Re:The hierarchical object file system

[Anonymous Coward]

Jeez, you make it sound like hierarchical filesystems are slowly killing us all. They are still easy and they still work. For all their faults, we still love 'em. Don't we guys?

(General murmer of agreement.)

%s/!\+/!/g
Ah, that's better.

Re:The hierarchical object file system

[iggymanz]

Really, the only problem is that there is no standard to do this with our existing filesystems. We already have the capability to do this a myriad of ways with ANY filesystem in common use

Re:The hierarchical object file system

[Brandybuck]

Right and wrong. I love the hierachical filesystem, but we still need good metadata attached to the files. Unfortunately, all the proposed solutions are to throw the baby out with the bathwater.

I want my old HPFS file system back, preferably with the WPS!

"Rand McNally" : A Ridiculous Cartographic MYTH!

[Anonymous Coward]

There is no island of "Rand McNally"!

Duke Nukem 4-0 : Balder, Fatter!

[Anonymous Coward]

Garland, TX

Looks like middle age hasn't been kind to action hero Duke Nukem. In a prerelease press preview, presented by Joe Siegler, the studly hero is bald with a huge beer-gut. "We wanted to flesh out the character of Duke", Siegler said, "we want to make him more a character that his fans can directly relate to".

In the new title, Duke is in a custody dispute with his ex-wife. Apparently, since he lost his job, he's in arrears on his child-support payments. When his (alien) wife kidnaps their kids and leaves for her mothers on Moltar III, it's butt-kicking time!"

MRAM saves the day

[Markus Registrada]

All the tradeoffs will change radically when MRAM hits the streets. It's potentially denser than disk and DRAM, as fast as static RAM, nonvolatile, doesn't use power when it's not used, and can be made on regular silicon process machinery. Expect it first in cell phones next year, and then everywhere.

This doesn't just affect file storage and virtual memory. It also changes the economics of cache and main memory, and makes deployment of 64-bit CPUs more urgent. It also makes system crashes much less tolerable, because turning the computer off and on doesn't involve long shutdown and boot procedures any more.

Re:MRAM saves the day

[HerbieStone]

All the tradeoffs will change radically when MRAM hits the streets. It's potentially denser than disk and DRAM, as fast as static RAM, ...

Yup. And Duke-Nukem Forever will eat Half-Life 2s panties.

Downstream is cheap!

[Anonymous Coward]

You're just looking at the massive downstream capacity you have, which is really only available if nobody else at your CO is using it. Odds are your upstream capacity is significantly more expensive. I have a 128kbps upstream on my $40/mo cable modem, which comes to about $1/GB up.

If you have a dedicated T3 (45Mbps), for example, and pay $7000/mo for it (a reasonable price), you are paying about $1/GB up AND down.

aQazaQa

Incredible Interview

[Anonymous Coward]

It's a shame this guy passed away recently. This was the absolute most intelligent and insightful interview/story I've ever read.

Fuzzy numbers, or can this be right?

[abulafia]

JG Twenty-megabyte disks were considered giant. I believe that the first time I asked anybody, about 1970, disk storage rented for a dollar per megabyte a month. IBM leased rather than sold storage at the time. Each disk was the size of a washing machine and cost around $20,000.

So, one could rent a $20K device for $240/year? Those must have been the days...

That can't be right.

Re:Fuzzy numbers, or can this be right?

[Zygo]

He's close, except the space unit is probably less than a kilobyte (this is an age where programmers worried about using dozens of bytes, and used "kilo" the same way we use "giga" today), and the time unit is probably a day or at most a week. No doubt the bill arrived every month--after all, your phone company doesn't mail you a bill after every call, even if the amount you used paid for the envelope and stamp...

I recall the price was something like $0.03 per 480 bytes (remember: a kilobyte is over a *thousand* bytes) for storage on a non-IBM mainframe (the vendor shall remain nameless, but you can probably recognize it from the size of the allocation unit) in the 1980's. The billing was exact to a tenth of a penny--the filesystem kept track of how old your files were when you deleted them, so that you could be billed accurately. This was actually considered a feature, like per-second billing with cell phone and long-distance companies, because The Competition would round your usage up based on your daily peak usage or something.

If you're wondering about what happens if you shorten a file or append to it--well, you can't do either with this filesystem, so don't ask. The OS did provide a file type that could be accessed randomly, but the billing structure for those was different. There were no directories either.

Incidentally, if you wanted that data sent to your terminal during the daytime, it cost on the order of $0.80 per kilocharacter (which was almost, but not quite, a kilobyte). There were also charges for I/O, memory, CPU time, operator attention, and any other scarce resource that could be metered and sold.

To put this in perspective: the swap space used by a typical KDE desktop would cost a little under $200 per day in disk space rental fees at 1980's prices.

Defending Jim Gray

[chrisd]

I didn't really read that as fud or even invalid criticism of MySQL. Maybe I'm biased because of my previous work with Queue and since I have met Jim, but if you get the impression that Jim doesn't like MySQL (which I did not) then I would actually assume it is because he felt that way, not because of Microsoft. Jim is one of those guys that will never be looking for a job, his early work on databases were pivotable to the development of transactions and his work on fault tolerant systems is legendary, he really is beyond reproach.

Chrisd

Re:Defending Jim Gray

[Kashif Shaikh]

Actually I found the interview really informative. From his take on MySQL, he basically said that big companies would rather not depend on a piece-meal solution, but a product that has serious backing and has undergone lots and lots of testing. Something which is easier for big companies(since they got the dough) and small companies like MySQL AB don't. Kashif

Re:Defending Jim Gray

[brettper]

Jim is one of those guys that will never be looking for a job - well, given that he's apparently passed away since the article was written, you could be on to something there

Disks as sequential devices?

[CrazyTalk]

This does indeed sound familiar - like the old cassette drive hooked up to my C64. I guess everything old really *is* new again.

Three letters: F, U, and D

[heironymouscoward]

Take this choice quote from the article:

My buddies are being killed by supporting all the Linux variants. It is hard to build a product on top of Linux because every other user compiles his own kernel and there are many different species.

Ain't it sweet? I count five lies:

(1) people being killed by supporting (gasp) operating systems... gosh, horror and violence, not nice at all!

(2) all the Linux "variants", are in fact pretty much one standard, LSB, with several skins

(3) "hard to build a product on top of Linux", rather than, hmmm, Windows? Linux is incredibly easy to build for. I suspect the fact that it's very standard helps.

(4) "every other user compiles his kernel"... maybe at Microsoft. I suspect less than 1 in 20 Linux users ever compiled a kernel.

(5) compiling a kernel means you can't support it... WTF? The kernel is incredibly stable, since most changes are in external modules. And I can't remember a single case where a kernel change broke one of my apps.

(6) (sorry, I was not counting well), "many different species"... well, AFAICS the only difference between the Linux distributions is that they have different packaging methods, different timelines as to their versions, and different UI tools for hardware detection, configuration, etc. Nothing at all that makes life hard.

Look: I just installed Xandros, which is Debian with a nice face. On two different types of machine, and it installed without asking a single question about my hardware except whether the mouse was left or right-handed. Check my journal...

Windows never worked this nicely. Where is the support issue?

In the writing indistry we call this "to condemn with faint praise".

Yeah, Windows kinda works, I mean, it'll run Office without crashing too often, but it's just killing by buddies to have to maintain Win2K, WinXP, and even some older Win98 machines, not to mention we have a whole cupboard simply filled with driver CDs for every PC we have.

Re:Three letters: F, U, and D

[Junks Jerzey]

From the Devil's Dictionary:

FUD: The sound made by someone attempting to wish away inconvenient facts.

http://www.eod.com/devil/archive/fud.html

Re:Three letters: F, U, and D

[Anonymous Coward]

> because every other user compiles his own kernel and there are many different species.

That is almost exactly what a MS lackey I was with said when asked about Linux. Five years later and Linux is still going strong. How ya like them apples.

Re:Three letters: F, U, and D

[NineNine]

Well, I just responded to another troll, so I might as well respond to you too. If you knew anything about enterprise class databases, you'd know that they have many, many, many years of maturity behind them that MySQL doesn't have, and because of that, are optimized per each OS version & CPU combination that they're run on. MySQL is just a generic, primitve DB now, that hasn't been tweaked out, because nobody's going to buy a refrigerator-sized Sun array to run MySQL on. You do that with Oracle, and it's already installed, optimized, tweaked, and working when you get. Back under your bridge, troll.

Re:Three letters: F, U, and D

[pavon]

This is a real issue, not FUD, especially for drivers, which RAID type people would have to deal with. Basically every distro I've used has modified the kernal, and the with the checksum name mangling in the 2.4 kernel, this means that a kernel module (ie driver) compiled for a RedHat kernel will not work for a Mandrake kernel. In fact the Mandrake 9.0 and 9.1 kernels are incompatible in this respect! The only solution it to provide dozens of binaries, or go with some elaborate scheme like NVIDIA does and compile part of the module when installing it.

Then there's the issue of supporting your code. That means testing it on all the platforms you will support. Lets count the common distro's out there. RedHat 6.x, 7.x, 8.x, 9.x, Mandrake 8.x, 9.x, Suse 7.x, 8.x, the Debian gradient, and a suite of other ones. Now you dont' have to officially support all of them, but from looking at this list it certainly involves more testing than the 3 to 5 historic Windows platforms.

The truth of the matter is that there are differences between distro's - otherwise there would be no reason to have more than one. Furthermore libraries are finer grained and change (improvements often require incompatible changes) more often than in Windows. And while the differences are small, and it is likely you won't have too many problems, determining that you won't have problems takes time and money.

Supporting linux is often more work than supporting windows - something that ID software and others have discovered. But fortunatly it is also often worth the effort.

Ob:ugly 'merkin

[Anonymous Coward]

Only until we finish bombing off all the other parts that AREN'T the USA!

3 Terrabytes on a credit card?

[polyp2000]

Anyone know what happened to that bloke at keele who
invented a way of cramming 3 Terrabytes on a credit card. Apparently it would have cost about 35 pounds to manufacture. this was a couple of years ago, why hasnt it happened yet?

Surely something like this is the real future of storage ?

Terrabyte on a credit card

Re:3 Terrabytes on a credit card?

[jetkust]

This article claims some kind of software based 8:1 compression scheme on binary data. Am i reading this wrong or does this seem a bit like nonsense?

Re:3 Terrabytes on a credit card?

[polyp2000]

Im sorry , I appear to have put the wrong link,
trouble is i couldnt find the link it was supposed to be.
Any way i found this amongst the press releases from that site.

Terrabyte credit card

nick ...

Sneaker net?

[computerlady]

"Sneaker net" was when you used your sneakers to transport data?

Oh my. How old I feel when someone has to ask what "sneaker net" was. And someone has to answer...

Re:Sneaker net?

[evilviper]

How old I feel when someone has to ask what "sneaker net" was

Actually, it's not that it's an old term, it's that there are idiots everywhere now. Unwashed masses, invading the computer sector.

Do you feel old when someone asks what "Either Net" is? Of course not, you do the sensible thing, and hit them over the head...

Same rule applies here.

And an old one!

[siskbc]

Damn, timothy, when it says June on the article it just might be a dupe, ya know? But it's nice to know that the future of disk access hasn't changed since then.

AMAZING!!!

[X86Daddy]

This is a *MAJOR* breakthrough! Most Turing Test contestants don't even win, but this one can eloquently discuss topics and give complex answers, rather than just turning back the question, Eliza-style.

Can we download a copy of this "Jim Gray" yet?

Re:AMAZING!!!

[cybermace5]

Can we download a copy of this "Jim Gray" yet?

No, too big to transfer over the Internet at this point. You'll have to use UPS.

sequential vs direct-access

[CaptnMArk]

>programmers have to start thinking of the disk as a sequential device rather than a random access device

This is partially already true for classic UNIX userspace behavior. You pipe the data from the input file(s) trough a filter and generate the output, sequentially.

A completely different model from the FS drivers or a SQL database.

New File System

[Archangel Michael]

What current file systems need is meta data in them. That is that the File system itself stores the MetaData about the file. Think about the Mac File system, with the Meta data contained in the file itself, as the "resource fork". Now imagine a systemized, extensable meta file system, that organized files by what the Meta Data said about them.

Imagine, media files stored in such a way that both random and sequential access was optimized, where the file structure was automagically defragmented and organized behind the scenes.

Imagine a computer that watched what files were used at bootup, and organized them so that the hard drive streamed the bootup data sequentially, straight into memory.

Imagine being able to start PRELOADING applications before you even finish the second of your double clicks on the datafile.

Imagine Database files that were automagically indexed as part of the file system.

Imagine Security and encryption being built into the filesystem beyond today's capabilities, where the security and encryption does not rely upon a master controller or centralized security policies, but rather has the ability to follow the file, seemlessly.

I am sure that I haven't even begun to tap the possibilities.

Re:New File System

[MattRog]

Yes we do need a new model and yes, we have one: it's called the Relational Model.

We've discussed this before.

Re:New File System

[Archangel Michael]

That is just a start. That article mentions nothing about Meta Data, which is required to make advanced capabilities of the File System come alive.

For Meta Data to work, there has to be some sort of STANDARDS based way of describing said data.

For instance, a table. How would you describe a table? Is it Tab delimited text, Spreadsheet or a HTML based Table? Does it reference cells and or other tables? Are those available? Is the data from missing tables, available as a static value?

Is the data within the table used in other work, such as a presenation or Brochure? The value is not in the system, but in the interlocking way we use it, and that needs to be described as Meta Data.

Re:New File System

[MattRog]

In the relational model that is all encapsulated in the physical (e.g. internal to the RDBMS) data type. You, as an application developer/user doesn't give a hoot how it is physically stored - that's the whole point of abstraction! The concept of a "file" is gone - you would have a table datatype that would be defined with rows/columns/etc. and you would just select from it in your application.

The relational model also allows for definite relationships between the tuples (and indeed the different elements of the tuple).

For one that knows very, very little of the relational model you are quick to reject it.

Re:New File System

[Archangel Michael]

Relational Databases, I was probably using them and programming them before you probably knew what they were.

And Tuples (never heard of them before) look like adjustble arrayed variables. Nothing new there.

I don't know of any application developer that gives a hoot how stuff is physically stored. I do know plenty that care how it is LOGICALLY stored.

Finally, I am not careing about how stuff is difined inside an application, just how the SYSTEM defines the data.

I can see where your problem is in understanding what I am saying. You are mixing up physical storage (hard drive, tape, cd etc) with logical storage (file structure etc). In my case the system doesn't care about MEDIA, but how the data is organized across all media, and identifying the type of data contained within a file.

Your shift in terminology not withstanding, the ideas and concepts you are promoting haven't changed as much as you think they have.

Let us say that we have a snippit of data. That data can be found in a file, all by its lonesome; in a file with other snippits (related or not). Abstracting the snippet of data one level further doesn't negate that the snippet is just that, a piece of data.

However a file system that can indentify that snippet as DINAMIC data (subject to change), of what kind (table, text whatever), format (table, tab delimited) by META is much more useful than say, building a huge abstraction around it, for say, an application.

Remember, the higher the number of abstractions, the harder it is for a HUMAN to find what he is looking for. Normal humans cannot see beyond about two or three layers of abstraction without substantial training.

However, using META information as PART of the file system (which would be a RDMS of sorts by definition), would allow faster identification of the DATA by both HUMANS and computers. And Inaddition, it would be transportable between systems, not being bound to the system.

Re:New File System

[MattRog]

There is no such thing as a 'Relational Database' - it's a Relational Database Management System. How the data is stored is irrelevant to the model (it's up to the RDBMS to determine how to store the data internally).

I'm not sure if you're being sarcastic or not, but a tuple (in the relational model) is an element (e.g. row) of a relation. I don't know what you mean by 'adjustble[sic] arrayed variables'. One difference is that arrays have an order and tuples do not.

By Physical I mean physical in the Relational sense, not 'writing 1s and 0s at the disk level'. Per Codd's Rule 8, Physical Data Independence: Interactive applications and application programs should not have to be modified whenever changes in internal storage structures and access methods are made to the database.

So physical in the cases of RDBMS would be the actual row format. XML, comma delimited lists, etc. would be examples of a 'physical' row format, and the 'logical' would be the table-ized presentation we're all used to.

Consider the 'snippet' of data that you talk about. It very well may live in a file, or spread out over several files. But that, along with the 'format' (XML, comma, etc.) is an implementation detail of the model which should not be exposed to the user. Like you say - it is simply data and how it is stored is irrelevant.

So - in the case of the relational model the user is not exposed to a 'file system' because that is an implementation detail. As far as abstraction is concerned I contend that exposing a file-oriented model to the end-user is far from intuitive. We think in 'emails', 'reports to the boss', 'post-its stuck to my monitor' etc. - and given proper tools (e.g. a proper RDBMS and presentation layers) they would never need to see the underlying implementation.

The relational model already has provisions for 'metadata' - it's built-in!! And as far as transferring between systems there are about a billion different formats, the least of which is XML (which I dislike for a number of reasons but seemingly has become the standard). Given knowledge on both ends (which would be required in your scenario as well) you could seamlessly transfer documents, contacts, emails, etc. between systems and maintain their related properties.

I don't think we're too far off in the end result, but the implementations are far different (and, given the relational model has already been proved to be sound, it makes perfect sense to use it).

I2 != Research

[Flamesplash]

The thing is that while the internet2 was supposed to be a research tool, it is actually being used for high bandwidth data transfers from academic units. This happens so much that it actually hinders networking research to an extent.

next level up is more sequential (always?)

[dpilot]

To answer your question directly, cost is probably the biggest reason. But there's more...

"programmers have to start thinking of the disk as a sequential device rather than a random access device."

"I think we'd all be better off when solid state, non-mechanical disks become commonplace."

Now that you mention it, I don't think so. In reality, there's a lot time spent dissing the latency (and even bandwidth) of DRAM. (any flavor) That's why caching is being elevated to a fine art form. That's why Intel is introducing L3 cache in their new "Extreme" gaming chip.

The reality is that the next level up is never as "good" as the level below, only denser and cheaper. Caching helps get around this, and good old Single Level Store extended the concept by treating the disk as the real (bit) address space and main memory as a cache of that.

Even without Single Level Store already disk paging and caching mechanisms, head elevator algorithms, I/O schedulers and the like already do some of this 'sequentializing' for us, behind the covers.

IMHO, we're getting where he wants to go, through the back door. ...the other answer to your question, by the time solid-state disk prices are low enough, access methods will have shifted enough and memory hierarchy grown enough that the effects simply won't be what you'd expect given the technology, today.

IDE replaces DVD

[G4from128k]

With an ever growing collection of digital photos, I've come to the same conclusion as Jim Gray. Hard disks are superior for backups.

I currently have about 100 GB of images and it takes more than 20 4.7 GB DVD-R discs to create a full backup. Although DVD media is still slightly cheaper than new large capacity IDE drives, the added time and hassle factor of burning 20 disks far out weighs any minor costs savings. Moreover a 3.5" drive in a padded anti-static bag takes up less room in the safe deposit box than 20 DVDs (especially if you have the DVDs in protective jewel cases). And if HD-based-backup lets me avoid some future artists tax on burnable media, so much the better.

A Firewire enclosure and a rotating collection of IDE drives is the way to go.

Re:IDE replaces DVD

[Anonymous Coward]

100GB of images? Damn, that's a lot of pr0n!!

Re:IDE replaces DVD

[Cyno]

I came to that conclusion 3 years ago. What's taking you guys so long?

CD and DVD media is great as a replacement for the floppy disk. But harddrives have been the only affordable transportable mass-storage media for years. DVD has never been an option for me, but I suspect it will replace CDRW media within a couple years.

$1/GB or less since 2001.

Re:IDE replaces DVD

[Kris_J]

What I found particularly interesting was the comment about tape price per MB v's HDD ppMB. Our company tape drive failed on Friday and I've reworked the process such that backups now use the excess capacity of my Laptop's hard drive plus my iPod for off-site storage.

Interesting Idea...

[polyp2000]

Interesting thought popped when i read your post,
there is a current trend towards cramming as much storage into something the size of a 3in Hard drive.

I wonder why they dont make larger harddrives in the physical sense? A hard drive the size of a washing machine using todays technology would store a phenomenal amount of stuff, but whatabout something more reasonable like a hard drive merely twice the physical size of todays. how much more storage could you get just by scaling up the platters? anyone here good at math . Hard drives today must be up to 200-250gb.

Re:Interesting Idea...

[trtmrt]

I think the problem is in the speed. The bigger the platter the longer it will take you to get to where you want to read (assuming the head moves at the same speed). There are probably all sorts of mechanical and heating problems if you try to spin larger platters faster. Hard disk speeds have not been increasing as fast as their storage capacity so this is probably something you wouldn't want to sacrifice.

Re:Interesting Idea...

[generic-man]

Single point of failure. EMC will sell you an 18 terabyte storage array that will fail over if one drive stops working. It's much easier to switch one drive than one platter.

Re:Interesting Idea...

[Yeroc]

Imagine all the energy stored by huge disks turning at 10k rpm.... Imagine if one of those disks disintegrated.

Re:Interesting Idea...

[WNight]

Yeah, a drive cabinet would be a nightmare to use.

But 5.25 HDs, single height, could hold (judging by platter area) over twenty times the data. Still easy to swap, but a nicer size.

They'd be slow, but I connect to my fileserver over 100mbps, even with 10 people using it I rarely see any delay.

Re:Interesting Idea...

[evilviper]

It all depends what you want. For larger hard drives, you'd have to make them significantly slower.

Also, any change in the size of the devices you manufacture, requires a significant investment in upgrading your manufacturing facilities. There are more issues, but I think that's good enough.

At significantly larger sizes (washing-machines), disc jukeboxes have very significant advatages. In fact, you would probably do much better to have a stack of small hard drives, rather than one large one.

Can we have more articles like this, please? (nb)

[Anonymous Coward]

Like the subject said.

Turing Award?

[ConceptJunkie]

Does that mean Jim Gray proved to someone over a computer terminal that he was human?

He is right, but nothing to do with the kernel

[brunes69]

His basic idea is 100% correct, but the reson is all wrong. It *IS* much harder to develop an app Linux the myriad of flavours, not because of the kernel, but because every distro has its own versions of libraries. I work for a company that makes Linux software, and we only support RedHat, and even certain versions of RedHat at that. While our product would probably compile against any number of distros, and even the BSDs, we just don't have the time and manpower required to build, test, debug, package, and maintain 15 different releases for every sub-release or patchlevel we have in the product. With Windows products, at least, (unless you are doing some lower-level stuff) if you build something you can be reasonably assured it will run on Windows 2000, or Windows XP, or Windows 2003. Not the same if you build something with RedHat 9 and try to run it on Debian or Suse, etc. And before you go on about "release a source package", not all companies release everything GPL, and want to keep their IP theirs, since they like to put some money on the table at night. It's definitly not FUD to say it is much more effort to develop and release cross platform binaries in Linux than Windows.

Re:He is right, but nothing to do with the kernel

[battjt]

What about statically linked releases? Joe

Re:He is right, but nothing to do with the kernel

[Sanga]

It's definitly not FUD to say it is much more effort to develop and release cross platform binaries in Linux than Windows.


Am waiting for a MS sponsored "study" to validate what you said. They would show notepad working on different flavours of Windows (I doubt if it works though).

Re:He is right, but nothing to do with the kernel

[Hatta]

If they don't want to play the game and instead hoard IP, they can get left behind with the rest of the dinosaurs. No sympathy.

The Ol' Roadmaster Scenario

[codefool]

What Gray is talking (mostly) about is what we used to call the "Roadmaster Scenario." When I worked for [a major electronics company], we had a data center in Dallas and a redundant site about 30 miles away in Lewisville. Every Sunday the entire IMS database was archived to mag tape and shipped to the other data center for a second level of redundancy. This begged the question, why not just copy them over the T1 lines (this was 1980) to the other site's tape drives directly? The answer, of course, was that it takes a helluva lot of bandwidth to outrun a Roadmaster full of mag tapes.

Missing the logical boat

[leandrod]

> To some extent you can think of Codd's relational algebra as an algebra of punched cards. Every card is a record. Every machine is an operator.

Interesting how the guy literally wrote the book on transactions, yet grossly misrepresents Codd's work, which BTW wasn't simply the relational algebra, but even higher level: the relational model of database management, including the relational calculus.

While the algebra is somewhat procedural, the calculus is set-oriented, and they are fully equivalent. The idea is exactly not looking at records and operators, but describe what you want -- just leave the relational system set the procedures to get that in the most efficient way it can.

Incidentally this has a big impact on all Gray is discussing -- without a fairly simple and powerful data model, so much data is basically a waste. He's thinking too low level, including the object stuff he touts, but we will only find use for so much data the day we get proper relational implementations, and this excludes SQL in general and MySQL in particular.

Re:Missing the logical boat

[tomlord]

He isn't grossly misrepresenting Codd's work.

You said it yourself:

While the algebra is somewhat procedural, the calculus is set-oriented, and they are fully equivalent.

and, uncoincidentally, the isomorphism extends further to machines that manipulate physical punch cards. You go on to say:

The idea is exactly not looking at records and operators, but describe what you want -- just leave the relational system set the procedures to get that in the most efficient way it can.

Right. And what Gray has pointed out is that Codd's work on the math and how to implement it doesn't really require computers, as such.

In an alternate timeline, there were no computers just lots of expensive punch-card machines and racks and racks of data stored on punch-cards.
(Such was the economic value of all this data that the racks of cards were often stored with an almost military degree of jealous protection: the origin of the term "Data Base".)

Each card machine could perform a simple operation like "duplicate this card stack" or "pull out the cards that have a Q in column 3". The machines could be organized into a sort of assembly line for a particular computation, with technicians looking at a script on a clipboard and carrying trays of cards between machines, configuring each machine with the right parameters, running the cards through, then going to the next step. It was an expensive, labor-intensive process and the ad-hoc procedures used to write the scripts for the technicians were black-magic, often error prone.

Time-study super-genious, Alternate-Codd, studied the machines and the procedures used to operate them. He realized that they could be described by set math. He realized that if you let the managers define their "Card Searches" in very high-level, very mathy terms -- then there was a straightforward optimization problem to get from that "Search Specification" to set of "compiled instructions" for the technicians. The goals was produce a set of Compiled Instructions that would use the punch card machines in an optimal way -- saving time and money.

He studied the optimization problem and developed some techniques for it. Companies used his results by highering a "Compiler Pool" -- most often a group of women chosen from the secretarial pool for the accuracy of their work. When a new Card Search request came in, the search would be typed up and mimeographed, and handed to the head of the Compiler Pool. It typically took "the girls" about a day to compile a query but, every time, the scripts they wrote for the technicians produced the right answer, usually much faster than anyone thought possible.

In one office, though, in Rochester New York, there was a famous accident. The office used by the Compiler Pool had developed a problem with flies. One day, one was swatted and killed with the mimeograph master of a compiled query, leaving a mark that obscured some important numbers. Nobody noticed, the technicians dutifully followed the errant script, and by the next afternoon the company's entire collection of precious data was strune, unsorted, in a huge pile on the machine room floor. The company was bankrupt only 9 months later.

The company president demanded an explanation when the accident occured and much investigation followed, eventually revealing the fly and its consequences. This was, of course, the origin of the familiar phrase (known to every customer whose ever gotten a $500 bill for a month of telephonic service), "compiler bug".

Three answers and an observation

[heironymouscoward]

First, DLL hell on Windows, shared libraries on Linux, same headaches. Consider static linking: larger binaries but fewer headaches.

Second, it is trivial and cheap to build packages for RedHat, Debian, and SuSE as you need them, we do this automatically. See, when the OS is free, it costs you little to set-up development systems. If you're tight for hardware, use UML.

Third, there are serious arguments against delivering binary-only packages, and in favour of building from source, and these arguments are not related to the GPL. My company has always had a policy of delivering source code whenever possible, and we've not had any issues with that. In contrast, it allows our customers to get much more out of the product.

Observation: you will notice many, many packages that install and run just fine on a wide mixture of Linux systems. Someone, somewhere, is not killing themselves doing this. Perhaps Microsoft buddies just die easily. Or perhaps the problem is easily solved by the application of tools like autoconf that are unknown in the Windows world.

Best Part

[Anonymous Coward]

The best part of the Turing awards ceremony is when the camera pans to the crowd and we see a bunch of losers wearing Japanamation T-shirts stitting with ugly girlfrieds.

[OT] Your sig

[achurch]

So, if the plural or "virus" is "virii", then I guess the plural of "radius" is "radiii".

The thing is, it is, except for the number of i's (radii, not radiii).

Re:[OT] Your sig

[ConceptJunkie]

Then why do people who employ the same contruction with the word "virus", use two i's?

Re:[OT] Your sig

[achurch]

Who knows... dictionary.com doesn't list either virii or viri as a valid plural of virus (and I assumed you were commenting on virii vs. virus; apologies if that's not the case). I guess "viri" looks too short, or not imposing enough, or something.

Hell, it's all in fun anyway. I hope so, at least.

Re:[OT] Your sig

[AndrewHowe]

Well, they are wrong. In Latin, "virus" doesn't have a distinct plural form (it's already a mass noun). In English the accepted plural form (resulting from treating "virus" as "an individual virus") is "viruses".
By the way, I think you need "of" rather than "or" as the fifth word of your sig.

Re:[OT] Your sig

[BetterThanCaesar]

Viri happens to be the plural of latin vir, 'man'. Although, some (probably not men) might agree that that would be a suitable plural of virus :-)

Re:[OT] Your sig

[ConceptJunkie]

Correct. However, it is hackish tradition (cf. The Jargon File) to borrow pluralizing conventions from other languages. Thus, we get amusing words like "vaxen", "fora" or even "fen". It's all meant for fun.

However, when you apply the sample plural construction to "virus" as you do to other words like "octopus" or "cactus" or "radius", one would think the result would be "viri", not "virii". I don't know where the extra 'i' comes from, but seeing as how both constructions are not standard, I suppose it is particularly pedantic of me to complain.

Re:[OT] Your sig

[AndrewHowe]

"Octopus" is Greek, not Latin, so the plural is technically "octopodes". However, in English we often take just the root word and add our own plural suffix.
"Viri" is already a word, it's the nominative plural of "vir" (man) so it means "men". It can also be the genitive singular of "virus", but we need a plural.
"Viruses" is the accepted word in the non-hackish tradition.

near-infinite storage

[gosand]

"programmers have to start thinking of the disk as a sequential device rather than a random access device." I think we'd all be better off when solid state, non-mechanical disks become commonplace.

I believe that sometime in the future, we'll look back on our spinning disks and chuckle. I think we will eventually get to near-infinite storage, and sequential will be the way to go. There won't be any erasing necessary, you will just write to the next available space, move the pointer to it, and move on. Why did we come up with erasing data anyway? or compression? It was to save space. What if you didn't have to save space because there was no limit on it? Some technology will come along that will offer us near-infinite storage space, and we won't have to worry about random access, erasing data, space management, etc. We'll get there, I just hope it is within my lifetime.

Re:near-infinite storage

[Hatta]

30 years ago did you ever think a personal computer owner could say ".5 TB is not enough"? As long as it's physical, it's limited. And as long as it's limited some of us will reach those limits.

Re:near-infinite storage

[gosand]

30 years ago did you ever think a personal computer owner could say ".5 TB is not enough"?

30 years ago there weren't any personal computers. :-)

As long as it's physical, it's limited. And as long as it's limited some of us will reach those limits.

Hence, my use of the term "near-infinite", and by that I mean "infinite for most uses". If storage goes to the organic molecular level, you would be talking about near-infinite storage, depending on the size of the media. But if we get to the point where we don't have to worry about conserving space, and our storage capacity is near limitless, the way we manage data would change dramatically.

Re:near-infinite storage

[Anonymous Coward]

What happens when disease strikes the organic molecular material in these new organic storage devices? Then we'll have true real life viruses and worms attacking our systems!

Re:near-infinite storage

[Bingo Foo]

Erasing won't just be a slight nuisance, it will be a thermodynamic problem. Simply setting a mole of bits to zero might heat up your hard drive enough to melt it.

Re:near-infinite storage

[gosand]

Erasing won't just be a slight nuisance, it will be a thermodynamic problem. Simply setting a mole of bits to zero might heat up your hard drive enough to melt it.

The point is you wouldn't even need to erase anything. There would be so much storage capacity that you wouldn't need to reuse the bits.

MySQL vs. Oracle

[heironymouscoward]

Oh, please, keep those coming.

I just love your sense of humour. I remember when we switched an ISAM application to Oracle in the mid 1990's, on a Unix box. A single record access by primary key was 20,000 times faster with the ISAM system than under Oracle.

I retested this with later versions of Oracle and found that the performance was worse, not better.

Now, I have a nice server under a desk here, and we reloaded an Oracle 9 database on it, it took something like 8 hours to rebuild. Since we make portable software, just for fun I reloaded the same database under MySQL. Less than 15 minutes.

Oh, but perhaps you were serious. No, you were serious? Jesus! You really were serious! Oh, that's even funnier. (wipes tear from corner of eye).

More, more, more...

Re:MySQL vs. Oracle

[MattRog]

Oh, please, keep those (trolls) coming.

Because, of course, performance is the sole indicator of a product's worth. Using that reasoning, everyone should be driving an Indy car instead of a station wagon.

Sure, the Volvo ($enterprise_DBMS) may not be quickest off the line as a Ferrari (MySQL) but I'd rather be in the Volvo when something goes wrong.

It's "A station wagon full of..."

[Doug Merritt]

["Never underestimate the bandwidth of a vanload of tapes barrelling down the highway"]
I've seen this a couple times before, but Google seems to come up with nothing useful for it.

That's because the original is "station wagon" (or "stationwagon"). Another common variant is "a 747 full of...". See e.g. this story

And no, it's certainly not Tannenbaum 1996; it was (IIRC) mentioned in Bentley's "Programming Pearls" CACM column/book in the 1980s. It's unclear that anything original can be attributed to Tannenbaum (okay, that's flamebait, but Tannenbaum irritates me).

Re:It's "A station wagon full of..."

[Anonymous Coward]

okay, that's flamebait, but Tannenbaum irritates me

Why, is it because without him there would be no Linux?
If Linus didn't screw around with minix and get into his famous flame war with Tanenbaum (As everyone knows, Tanenbaum wanted minix to remain a small and simple teaching operating system, while Linus wanted to extend it and make it a production quality operating system. Tanenbaume said no, and the rest is history.)
I think Tanenbaum made the right decision. Minix makes learning about operating systems way easier.
If Minix were made into a production-ready OS, his book would probably have been five times as thick, and way too boring for anyone to complete in a one or two semester course on operating systems.
As far as Tanenbaum's writing is concerned, his books are better than most others covering the same subjects, and are comprehensive, authoritative, very readable, and even funny at times.

So what's your problem dude? :-)

Re:It's "A station wagon full of..."

[Doug Merritt]

Why, is it because without him there would be no Linux?

I don't believe this is true, having both lived/programmed through that era and also having read all the various histories...but at best you're speculating that the existence of Minix was important historically... I didn't think so at the time, although it was somewhat interesting. People might say the same thing about Xinu, Bill Jolitz' Unix efforts, Cromemco Unix, Xenix, etc etc etc.

I saw and see nothing critically important about Minix, even though, yes, Linus was interested in it. If Minix had not existed, something else would have taken its place.

(The same is true of Linux, BTW...if Linus hadn't done Linux, something similar would nonetheless have risen to similar popularity. Nature abhors a vacuum, and lots of us in the field had been KEENLY interested in a freeware version of Unix since the late 70s...BSD didn't mostly-lose due to technical inferiority, it got clobbered by *ssh*le lawyers. But I digress.)

But anyway, I said "original". I never heard anyone claim that there was something "original" about Minix, whether it was historically important or not. Similarly with Tanenbaum's books (only one of which I've read)... they may or may not be good books (let's say they're good, for the sake of the argument)... but is there anything original in them?

Lastly, you're assuming that the Torvalds-Tanenbaum flame war is why I'm down on Tanenbaum, but you'd be wrong. I met him in Real Life well before Linus ever started messing with Minix, and Tanenbaum irritated the hell out of me when I talked to him face to face...the Linus flamewar just added fuel to the fire for me.

As to the flamewar itself, I've always been interested in microkernels, as an OS designer myself, but Tanenbaum was arrogantly presumptuous on the subject even back then, and certainly in hindsight now...microkernels have both pluses and minuses, and any prof who would assign a failing grade just because of someone's choice, is merely proving that they themselves are jerks, by elevating a matter of taste up to pretending it's a matter of right and wrong. Tanenbaum does not belong in teaching, regardless of his other virtues, when he grades on the basis of his own ego rather than on the student's work.

To paraphrase, neither microkernels nor monolithic kernels are perfect, but Tanenbaum is a perfect jerk, ok? :-)

Re:It's "A station wagon full of..."

[AJWM]

Certainly 1980s, probably circa 1983 or 1984 at the latest. I came up with the phrase (which may well have been independently coined before me, at the time I was unaware of it) when we were setting up NETNORTH, the Canadian counterpart to BITNET (networks of typically college campus mainframes, not directly part of ARPANET). There was discussion about setting up the HQ at University of Guelph (where I worked at the time - west of Toronto) or Waterloo University.

The highway in question (as in station wagon travelling on) was the Highway (7? it's been a long time) between Waterloo and Guelph (at least part of which I drove every day, since I lived in Waterloo). I don't recall the numbers now, but my calculation of the bandwidth of Hwy 7 was based on a couple of boxes of 2400' reels of 6250 BPI tape (standard IBM mainframe tape size) in a car (or station wagon) travelling at the posted 90 km/h speed limit.

Back in those days, aside from dedicated leased-line networks like BITNET or commercial X.25 packet networks like Tymnet, a 2400 baud dialup modem was considered blazingly fast. (And long distance charges were not cheap, hence the popularity of multi-hop dialup networks using UUCP or like Fidonet.)

Re:It's "A station wagon full of..."

[shepd]

>The highway in question (as in station wagon travelling on) was the Highway (7? it's been a long time)

Yup, it's highway 7. I drive on that road twice daily. It's getting packed solid at rush hour at the very end of it now (Victoria St.)

>but my calculation of the bandwidth of Hwy 7 was based on a couple of boxes of 2400' reels of 6250 BPI tape (standard IBM mainframe tape size) in a car (or station wagon) travelling at the posted 90 km/h speed limit.

It's now:

50 km/h (section of Victoria St.)
60 km/h (another section of Victoria St.)
70 km/h (ANOTHER section of Victoria St.)
80 km/h (part most of Hwy. 7)

BTW: Take the shortcut through Maryhill using Shantz Station Rd. + Woodlawn Rd. Most people are driving over 120 km/h on that road without fuzz trouble. :-) Or take the secondary detour past Woodland Christian High School. It's a bit faster.

I wouldn't dare do under 100 km/h on the 80 stretch of Hwy 7. I'd get run over... Nobody does the speed limit in the country. Well, city slickers do.

Glad to see someone from the U of W here.

Re:It's "A station wagon full of..."

[Anonymous Coward]

O.k. he's a jerk. :-)
But influential nontheless.

Re:It's "A station wagon full of..."

[Doug Merritt]

O.k. he's a jerk. :-)
But influential nontheless.

Guess I can't argue with that.

And my irritations aside, he's got his good points, too.

Re:It's "A station wagon full of..."

[Doug Merritt]

Interesting, thanks for the history.

I suggest you cut/paste your post and put it on your web site, for the sake of historians who are always researching stuff like this.

(Well, I'm always trying to research stuff like this, anyway :-) Give it long enough, and posterity will too, as they say. Or will say.)

Re:It's "A station wagon full of..."

[AJWM]

Actually as often as not I'd take the 'back roads' through Breslau and past the Waterloo-Wellington Regional Airport, occasionally stopping in there to go flying if it was a nice day.

Glad to see someone from the U of W here.

Actually it was my (now ex-)wife that was the UW grad student, I was on the computer centre staff at U of Guelph at the time. Obviously I knew a number of other UW people. I'm sure, though, that there are other U of W folks here. I know I've recognized a few folks from my various past lives at GeoVision (hi Paul) and Concordia (hi Rene).

How about FibreChannel and 14 IDE drives?

[Johnny Mozzarella]

Apple came to a similar conclusion for it's server storage solutions (XServe and XRaid)

IDE drives can offer the speed & reliability that are needed even for demanding applications

bizarre error in the story...

[Jaeger-]

In the article at the bottom they say "In Memoriam -- Since this interview, Edgar "Ted" Codd, inventor of the relational data model, died of heart failure. He was much loved by his colleagues both for his warmth and for his many contributions to computer science. "

But then in the middle of the article theres this: DP Let's talk about the higher-level storage layers. How did you get started in databases? You were at IBM when the late Ted Codd formulated the relational database model. What was that like?

So which is it? Did he die AFTER the interview or BEFORE it??

I can help you with that too...

[WIAKywbfatw]

Ooh, you're good. Now can you tell me where my keys are?

Uhhh, I can't use my telepathy to tell you where they are because you don't know yourself. And if you don't know where you left them then how am I supposed to read that info from your mind?

However, I can use my clairvoyance to tell you where they will be. They'll be in the last place that you look.

Glad to be of assistance.

Re:I can help you with that too...

[AyeRoxor!]

'However, I can use my clairvoyance to tell you where they will be. They'll be in the last place that you look.'

That's not clairvoyance. It's logic. If I found them, why would I continue to look?

Re:I can help you with that too...

[Anonymous Coward]

Wow, you killed the joke. Good job

Re:I can help you with that too...

[Anonymous Coward]

"Wow, you killed the joke. Good job."

Another triumph for LOGIC!

And AWAY!

You haven't met him.

[Anonymous Coward]

Gray has some strongly-held opinions. Not many folks can convince him of much without damned good reasons. Of course, he also rags on how poor free software is.

Pay attention.

[Anonymous Coward]

First, DLL hell on Windows, shared libraries on Linux, same headaches. Consider static linking: larger binaries but fewer headaches.

Maybe they don't want to distribute larger binaries? There are some other annoying issues besides just the shared libraries. There's all sorts of little stuff that can be a pain in the ass.

the OS is free, it costs you little to set-up development systems. If you're tight for hardware, use UML

I'll quote from the original poster, "we just don't have the time and manpower required to build, test, debug, package, and maintain 15 different releases for every sub-release or patchlevel we have in the product."

Third, there are serious arguments against delivering binary-only packages, and in favour of building from source, and these arguments are not related to the GPL.

And, there are serious arguments against distributing source, in favor of distributing binaries. Primarily that you don't want to release your source code, you fucking idiot. Not everyone wants to (or realistically can) release their source. There are plenty of programs that are valuable because of the algorithmic secrets they contain. Generic web/db apps are shit, you can release the source for them. Anything that's really computationally heavy probably contains some neat tricks in the code, that you might not want to get out to the public.

you will notice many, many packages that install and run just fine on a wide mixture of Linux systems

And there are plenty that don't. What's your fucking point?

Or perhaps the problem is easily solved by the application of tools like autoconf that are unknown in the Windows world.

He didn't say he wrote for Windows. He specifically said they distribute for RH9. Now, there are other companies who write their software for Solaris, AIX, HP-UX, etc. It's easier to write for any of those proprietary than it is to do it for all flavors of Linux. Sure, writing for one version of Linux is easy, designing/packaging/distributing/installing/suppor ting for multiple flavors is very manpower intensive. If you don't know this, you aren't very experienced in software.

Re:Pay attention.

[heironymouscoward]

Man, you are polite today.

It takes a few hours to entirely automate the build process for a product under Linux. You have this CVS thing at one end, a bunch of Linux distros at the other, you press a button, and ten minutes later you get a bunch of neat binary packages back.

So painful it hurts.

Ah, insult me again, I'm not doing anything special today.

The Jim Gray we never knew.

[AceyMan]

It's nice to see Jim Gray has learned to apply himself since he was black-balled by the sporting public after his mean-spirited and inflamatory interview of Pete Rose at the World Series a few years ago.

He didn't come across as that bright at the time, but I guess he wasn't showing his true colors. And now he's won a Turing Award. Utterly amazing...

Ooooh, wait...

Nice paid link, turkey

[Seth Finklestein]

I've taken the liberty of removing the referrer tag from your amazon.com link.

If anyone were stupid enough to buy that book (an $89 book? WTF? text is free), you would have made $4.45 in commissions. This abuse of the comments system for personal gain is egregious and in violation of the Terms of Service for Slashdot.

Please delete your account.

Hey, does this sound familiar?....

[3seas]

"The algorithms are simple enough so most implementers can understand them, and they are complicated enough so most people who can't understand them will want somebody else to figure it out for them. It has this nice property of being both elegant and relevant."

Or what happens when the concept of taking complexity (made up of simpler things) and automating it such that it is easy to use and reuse, by the user, is applied here?

Performance

[heironymouscoward]

No, of course performance is not the only indicator of a product's worth.

Let me list my criteria for, e.g. a database product:

1. accuracy
2. performance
3. ease of administration
4. ease of installation
5. price

Not in any specific order. I've used Oracle databases for about 12 years, and on every single one of these counts, MySQL wins. Every single one, without exception.

Oracle wins on a number of other criteria:

1. profitability
2. complexity
3. need for expensive DBAs
4. consumption of excess time
5. image
6. marketing strength
7. market share
8. number of marketing drones

But as an independent software developer, thank goodness none of these actually help me in my business. MySQL has already killed Oracle's database, and their attempt to escape that trap and move to ERP systems and clustering technologies is just another industry troll.

Sorry, you're talking to someone who has been there, seen it, and speaks from long, painful experience.

Re:Performance

[MattRog]

Wow, you've been on Oracle since version 6? What a beast that was!

You have not used Oracle's database (unless you write low-level driver calls to the actual data itself) - you use Oracle's DBMS product.

In any rate, I find it difficult that you can say that MySQL is more 'accurate' than Oracle (and by extension PostgreSQL, or MS SQL Server, or Sybase ASE or ...).

The constraint handling is poor at best (you can only have very minimal constraints). You have no such thing a triggers or views. The datatype bounds checking is horrendous (you can issue an insert statement with character values for numbers, for example, and it won't whine; it silently truncates values which are too small/large for the datatypes; you can insert nonsense dates in datetimes, etc.).

Obviously the biggest indicator of your trollish nature is the line: "MySQL has already killed Oracle's database". What a laugh! That is, like almost all of your post, 100% unsubstantiated and undefined. If any open-source product would ever be in a position to chip away at Oracle it would be PostgreSQL. At least those guys know a little bit about the relational model (even if they insist on breaking it).

MySQL guys go 'Codd who?'

A 10 pound 1 TB storage device

[baja]

This is a great article. Jim Grey' idea of shipping boxes around really makes sense, but he talks about spending $400 to ship his 300MB a heavy computer, why not use a 10 pound terabyte box. I have to believe it would be cheap ship a 10 lb archiving appliance like the RocketVault " http://www.intradyn.com/products/specs.php" ? You could also have this thing do all the work, set it up to get the data over the network and e-mail you when it's done. Then pack it up and ship it. Since everything is self contained all you need at the other end in a network connection to restore the data.

Re:A 10 pound 1 TB storage device

[Boone4th]

It looks to be a plug and dust device too -- That's twice in one week these guys have crossed my desk. I just finished reading this week's Mike Karp article on them in Network World: "Storage management should be as easy as making toast" http://www.nwfusion.com/newsletters/stor/2003/0908 stor2.html. He really seems to like the product.

Re:A 10 pound 1 TB storage device

[f5t]

Yeah....I heard a great deal about it too!! Read it on nwfusion.com and one of my client bought it and according to him, "it changed my lifestyle". He used to babysit his archives every Saturday...now he doesn't :=) We are thinking of buying one for ourself too.

Re:A 10 pound 1 TB storage device

[towandaalso]

Thanks for the rocket vault link. After talking to them, their appliance looks promising. It looks feasible to move my data. I didn't expect to find a tapeless onsite archive that also sends encrypted copies offsite. (And/or offsite replication.) BTW, I HATE TAPE. My goal is to be DLT free in '03.

Cost calculations.

[mindstrm]

And how long would it take you to transfer a terabyte of information to the UK. Total cost here.. the other end has to pay as well.

Your cost is cheap because your ISP does cost averaging. If you pin your connection at maximum usage in/out 24/7, most broadband residential ISPs will send you a nasty letter, and shortly after, simply drop you as a customer. They aren't REALLY selling you bandwidth at that price. OR if you want to look at it, they are, but only if you use it a small amount.

If you need to transfer terabytes of data long distance, quickly, it's cheaper and faster to send computers via fedex than it is to purchase the bandwidth from some network provider.

The exception that proves the rule

[sacrilicious]

Sit down, turn off your cellphone, and prepare to be fascinated. Clear your schedule, because once you've started reading this interview, you won't be able to put it down until you've finished it.

Actually I got about halfway through and decided to skip the rest of it.

This ran in JUNE!

[Anonymous Coward]

That's why it seems so old - we all saw this months ago!

Catching dupes?

[oblom]

Maybe /. needs to implement a dupe catching feature. Whenever an article is submitted, extract URLs supplied and compare it to the list of previously submitted URLs. Then display the list to submitter and let him check the links to insure that it's not a dupe.

Potential problem is links to front pages (i.e. "Yahoo! reports ...") instead of deep links. These can be skipped.

For deep URLs that change content but don't change URL run a comparison between checksum on new link and old one.

Certanly this is not bulletproof, just an additional check that can be run by submitter and /. editors.

Re:Catching dupes?

[zor_prime]

Perhaps the code developed to refute the SCO claims can be put to good use in catching Slashdot dupes.

Just a thought. Might work better than just simple checksums.

And it's not like running this on new submissions would be that hard.

Past Tense Future Perfect

[meehawl]

This has been around a while, I mentioned it on Slashdot over a month ago. But it's still a great interview.

Here's an idea

[shadow_slicer]

Why don't you send out a mixed source/binary package:
The binary part can be the core of your program and contain all your IP.
The source part can be an interface layer to the rest of the system (aliases for library calls, or equivalent implementations for missing functions, etc...basically a wrapper layer between the system and the program).

During the installation the source part can be compiled and (statically/dynamically) linked to the binary part. The source package doesn't have to be GPL (since, if it linking it to your binary would force the binary to be GPL), but it could still use some other open source license.
That way you can mitigate the disadvantages of a binary distribution without having to use a full source distribution.

Also, if many companies were doing this, it might be a good idea to open source these compatability layers so that every company that makes something for linux isn't duplicating the effort. (though this is kind of what libraries are supposed to do....)

Another alternative is to *trust* your customers:
You could have a full source package, but under a proprietary license (not GPL). Just because the source is available doesn't mean that the customers have full reign over your IP, or even are more likely to pirate it: I have the full "source" for several books, but that doesn't cause me to violate the IP of those authors.

I really doubt that PHB's will go for the full-source approach though, as they tend to be paranoid about such things...which is why I suggested the first thing.......first....

Re:Jim Gray !is a jerk

[Artifakt]

If I recall correctly, there is a Jim Gray who writes about computing, and was involved in a recent lawsuit to keep using his own name without infringing the trademark of Jim Gray - the one who writes about sports. The case got quite a bit of press, particularly here on slashdot, as an example of how IP laws were fuxored. Despite this, people still keep confusing the two. Poor guy ought to sue us all for not knowing the difference by now, including me for not remembering the details.

What to do with a 200G Drive?

[webster]

I had a real problem when Gray indicated that there was no real use for a 200G drive. I dedicate a 120G drive to storing photographs I take, and it's nowhere big enough. I would like to have LOTS and LOTS more storage space, but I'm perfectly happy with the access times. When the six megapixels per picture I use become twelve megapixels I'll want even more lots and lots of storage space and maybe a doubling of the access times. One of these days I'll be getting into DV video, and the need for storage will go up even more, while the access time needed to do realtime editing is already here.

I guess that, no matter how smart someone is or how dedicated to keeping on top of what's going on someone will come in out of left field with a use for technology that is completely unexpected ('though I would not have thought that storage of photo or video files would be unexpected).

Re:What to do with a 200G Drive?

[Trejkaz]

Thankfully digital video *is* one big stream, so the big speed optimisations by removing random access which he was talking about would be perfectly applicable for this case.

Funny, not what I heard...

[Anonymous Coward]

Funny, not what I heard... I didn't hear:

"programmers have to start thinking of the disk as a sequential device rather than a random access device."

What I heard was:

"vendors have to get their head out and start thinking of the disk as a random access device rather than a sequential device, and they need to have their equipment quit lying about whether something has actually been committed to stable storage, or is just in a cache."

A system optimized for adding not changing...

[Paul Fernhout]

The free Pointrel Data Repository System I have been working on is optimized for adding data, not changing it. So it fits in somewhat with his model of primarily linear access to data. http://sourceforge.net/projects/pointrel/

HFS+!

[Walabio]

The Resource-Fork of a file on HFS+-Volume on a Partition of a disk does not contain metadata. The resource-fork contains resources such as the icon-image. The metadata lives in the Desktopdatabase.

If I had it all to do over again, I would implement the file-system a journaled file-system with the file-system consisting of a series of caches containing the ID# of each file as a 64-bit integer, the location of the file on a volume as a 64-bit integer, the paths to the file through the hierarchy, ownership and permissions, the MIME-Type, the file-size, the file-dates, and the name of the file as upto 255 UTF-8-Octets.

The file would would have three forks:

1. A Resource-Fork
2. A Metadata-Fork
3. A Data-Fork

The resource-fork would contain the icon. The metadata-fork would all of the metadata like ID3-tags and EXIF. The Metadata-fork would use an XML-based language. The metadata and the XML-based language would be extendable. The data-fork would contain the bit-stream of the file.

A 64-bit file-system can handle up to 16 exafiles. Each file could be upto 16 exabytes. Assuming that the minimum file-size is 1 kilobyte, a volume could have upto 16 zettabytes of data.

Unfortunately, HFS+ was not built this way.

US != Earth

[Anonymous Coward]

That browseable photo of the Earth only seems to take in the US.

MOD DOWN PARENT

[Anonymous Coward]

This is a copy and paste troll trying to whore karma.