Slashdot Mirror
Turing Award Winner On The Future of Storage
weileong writes "Ars Technica highlights an interview at ACM Queue with Jim Gray, a winner of the ACM Turing award *(among other things) by one of the pioneers of RAID (among other things). Many issues touched upon, including: "programmers have to start thinking of the disk as a sequential device rather than a random access device." "So disks are not random access any more?" "That's one of the things that more or less everybody is gravitating toward. The idea of a log-structured file system is much more attractive. There are many other architectural changes that we'll have to consider in disks with huge capacity and limited bandwidth."
Actual interview has MUCH detail, definitely worth reading."
I think we'd all be better off when solid state, non-mechanical disks become commonplace.
Is there any reason other than cost why we can't have 100Gb solid-state drives yet?
Get your own free personal location tracker
they are part of Internet 2, Virtual Business Networks (VBNs), and the Next Generation Internet (NGI). Even so, it takes them a long time to copy a gigabyte. Copy a terabyte? It takes them a very, very long time across the networks they have
Is this really true? Wasn't there a recent Slashdot story where researchers transfered a gigabyte of data, in fourteen seconds or so, on Internet 2 from California to the Netherlands?
I suppose that disk access times will be limiting factor in both ends if you were to read and write the data from/to a disk.
How small a thought it takes to fill a whole life
Two quotes from the article (emphasis mine):
Gray, head of Microsoft's Bay Area Research Center, sits down with Queue and tells us (...)
JG: If it is business as usual, then a petabyte store needs 1,000 storage admins. Our chore is to figure out how to waste storage space to save administration.
MS bashers will have a field day on this one...
Wenn ist das Nunstueck git und Slotermeyer? Ja! Beiherhund das Oder die Flipperwaldt gersput.
This doesn't just affect file storage and virtual memory. It also changes the economics of cache and main memory, and makes deployment of 64-bit CPUs more urgent. It also makes system crashes much less tolerable, because turning the computer off and on doesn't involve long shutdown and boot procedures any more.
Anyone know what happened to that bloke at keele who
invented a way of cramming 3 Terrabytes on a credit card. Apparently it would have cost about 35 pounds to manufacture. this was a couple of years ago, why hasnt it happened yet?
Surely something like this is the real future of storage ?
Terrabyte on a credit card
Electronic Music Made Using Linux http://soundcloud.com/polyp
What current file systems need is meta data in them. That is that the File system itself stores the MetaData about the file. Think about the Mac File system, with the Meta data contained in the file itself, as the "resource fork". Now imagine a systemized, extensable meta file system, that organized files by what the Meta Data said about them.
Imagine, media files stored in such a way that both random and sequential access was optimized, where the file structure was automagically defragmented and organized behind the scenes.
Imagine a computer that watched what files were used at bootup, and organized them so that the hard drive streamed the bootup data sequentially, straight into memory.
Imagine being able to start PRELOADING applications before you even finish the second of your double clicks on the datafile.
Imagine Database files that were automagically indexed as part of the file system.
Imagine Security and encryption being built into the filesystem beyond today's capabilities, where the security and encryption does not rely upon a master controller or centralized security policies, but rather has the ability to follow the file, seemlessly.
I am sure that I haven't even begun to tap the possibilities.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
With an ever growing collection of digital photos, I've come to the same conclusion as Jim Gray. Hard disks are superior for backups.
I currently have about 100 GB of images and it takes more than 20 4.7 GB DVD-R discs to create a full backup. Although DVD media is still slightly cheaper than new large capacity IDE drives, the added time and hassle factor of burning 20 disks far out weighs any minor costs savings. Moreover a 3.5" drive in a padded anti-static bag takes up less room in the safe deposit box than 20 DVDs (especially if you have the DVDs in protective jewel cases). And if HD-based-backup lets me avoid some future artists tax on burnable media, so much the better.
A Firewire enclosure and a rotating collection of IDE drives is the way to go.
Two wrongs don't make a right, but three lefts do.
Interesting thought popped when i read your post,
there is a current trend towards cramming as much storage into something the size of a 3in Hard drive.
I wonder why they dont make larger harddrives in the physical sense? A hard drive the size of a washing machine using todays technology would store a phenomenal amount of stuff, but whatabout something more reasonable like a hard drive merely twice the physical size of todays. how much more storage could you get just by scaling up the platters? anyone here good at math . Hard drives today must be up to 200-250gb.
Electronic Music Made Using Linux http://soundcloud.com/polyp
There are multiple levels of access within a file system. The sequential versus random decisions they are talking about is at a much lower level than you are thinking. Somewhat simplified:
Now, when software opens a file, it gets a handle to the storage and seeks all over it to get the data it needs and finally write it back. This is particularly true of files that consist of many records. Some software mmaps (memory maps) the file, mapping it into the memory address space and making it appear as a large, slow section of RAM in order to make this easier.
Relatively recently, you see many more programs which open a file, slurp the entire thing into memory, and close the file on disk. When they want to make changes, they open the file again and rewrite it from scratch. You see this more in text editors and word processors. Programming editors will often have some alternate behavior for very large files, although the threshhold for "very large file" is always increasing.
When you do this with record oriented files and or incremental save/autosave, etc, you get into journalling. You write all of the user's changes sequentially to a log file rather than saving the actual file (and re-writing it) repeatedly. This is sometimes what you are seeing when a program has a 'recovery file'. Having only one recovery file or journal for any number of open files means you are consistently writing appends to a single location and avoiding disk seeks.
What the article is getting at is that this sort of behavior will get more and more common, even moving into the FS and OS level. Support for this kind of journalling may move its way into FS handling, for instance. Also, instead of opening individual files, the FS may block transfer a whole directory into RAM at once. We already see this with advanced file systems which store small files directly in the directory inode. We may see the inodes get larger and the definition of 'small file' become steadily larger. When you have GBs of RAM and TB of storage, why not have a 64 MB+ inode?
From this point of view, random seeking within files slowly becomes irrelevent. Rather, the primary operations become streaming and append.
I now think he probably did not mean it the way I read it. If anyone cares to mod my original post down, feel free. But I do think that, for a long article, there was actually not a lot of real content.
Panurge has posted for the last time. Thanks for the positive moderations.
> To some extent you can think of Codd's relational algebra as an algebra of punched cards. Every card is a record. Every machine is an operator.
Interesting how the guy literally wrote the book on transactions, yet grossly misrepresents Codd's work, which BTW wasn't simply the relational algebra, but even higher level: the relational model of database management, including the relational calculus.
While the algebra is somewhat procedural, the calculus is set-oriented, and they are fully equivalent. The idea is exactly not looking at records and operators, but describe what you want -- just leave the relational system set the procedures to get that in the most efficient way it can.
Incidentally this has a big impact on all Gray is discussing -- without a fairly simple and powerful data model, so much data is basically a waste. He's thinking too low level, including the object stuff he touts, but we will only find use for so much data the day we get proper relational implementations, and this excludes SQL in general and MySQL in particular.
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
Certainly 1980s, probably circa 1983 or 1984 at the latest. I came up with the phrase (which may well have been independently coined before me, at the time I was unaware of it) when we were setting up NETNORTH, the Canadian counterpart to BITNET (networks of typically college campus mainframes, not directly part of ARPANET). There was discussion about setting up the HQ at University of Guelph (where I worked at the time - west of Toronto) or Waterloo University.
The highway in question (as in station wagon travelling on) was the Highway (7? it's been a long time) between Waterloo and Guelph (at least part of which I drove every day, since I lived in Waterloo). I don't recall the numbers now, but my calculation of the bandwidth of Hwy 7 was based on a couple of boxes of 2400' reels of 6250 BPI tape (standard IBM mainframe tape size) in a car (or station wagon) travelling at the posted 90 km/h speed limit.
Back in those days, aside from dedicated leased-line networks like BITNET or commercial X.25 packet networks like Tymnet, a 2400 baud dialup modem was considered blazingly fast. (And long distance charges were not cheap, hence the popularity of multi-hop dialup networks using UUCP or like Fidonet.)
-- Alastair
Interesting point. I guess that's partly because a human collects stuff in a more or less linear fashion. Everything you collect, create or use takes time, and time is a resource that we don't get more of simply because our computers get faster. It is possible to handle one single 4 GB file such as a movie, but it would be impossible to do something meaningfull with 4000 1MB files, it would simply take too much time. Offcourse, you could think of automated tasks operating on large sets of files, but again random access would serve no benefit here. Throughput is important in the case of a program handling a sequence of small files.