Conquest FS: "The Disk Is Dead"

andfarm writes "A few days ago, I sat in at a presentation of a what seems to be a new file system concept: Conquest. Apparently they've developed a FS that stores all the metadata and a lot of the small files in battery-backed RAM. (No, not flash-RAM. That'd be stupid.) According to benchmarks, it's almost as fast as ramfs. Impressive." The page linked above is actually more of a summary page - there's some good .ps research reports in there.

Comments

well and good

[iamweezman]

this is great. We all have seen this coming, but how is the industry going to take this and implement it. My bet is it won't. The only way that it will take hold is if you can find some small application that will take and apply it.

Re:well and good

[robslimo]

I've predicted and eagerly anticipated the demise (by replacement) of spinning media (magnetic and optical) for 10 or more years now... I've predicted it will happen, not when.

As this new filesystem implicitly admits, the price/MB is still so much dramatically lower for HDD's than solid state memory, it will still take quite a will for this replacement to happen.

I disagree that some small killer app must come along to make this happen. Yes, solid state media is coming down in cost and increasing in density, but both need to change by 2 or 3 orders of magnitude before the HDD is dead. What we're waiting for here is the classis convergence of technology and its applications... the apps won't some until the technology can support it and the tech is driven by our demand for it. Expect another 10 years at least.

Re:well and good

[CrosbieSmith]

It will happen when the price difference between solid-state devices and magnetic storage gets narrower. That's not happening.

This was also pointed on Saturday's Slashdot Story

A mere $US5,000 would be something of a price sensation by the standards of current large capacity SSDs, whose prices aren't dropping nearly as quickly as are those of magnetic media.

Re:well and good

[robslimo]

True, and that narrowing will have occurred by the time the cost/density ratio of SSM has improved by 2 or 3 orders of magnitude.

A couple of reasons I see the death of the HDD to be not-to-imminent:

(1) Those damned HDD makers keep pulling new physics out of their as^H^H hats and keep pushing the storage densities to rediculous new levels.

(2) the solid state memory of the future ainta gonna be Flash as we know it now (with slow and limit write cycles) and it also will not be battery-backed RAM (unless we go write it all back to disk for 'permanent' storage at some point). I bet on some variation on today's Flash without its limitations, but the tech has got some ground to make before this all happens.

My other long-term prediction has been that CRTs (vacuum tube, for pete's sake!) will be replaced with LCD or similar tech and we're getting really close.

Re:well and good

[iabervon]

One important thing to realize about storage is that people's storage needs for some types of files grow over time, but storage needs for other things do not grow significantly. For example, if you separate attachments and filter spam, you can now buy an SD card which will store all of the email you will get in the next few years; when that runs out, you'll be able to buy a card which will store the rest of the email you will ever receive. There are similar effects for all of the text you'll ever write.

Furthermore, there are a number of important directories on any system whose total size won't double in the next ten years, because they add one more file of about the same size for each program you install, and they already have ten years of stuff.

In the cases where you do have exponential growth of storage use, the structure of the stored data is extremely simple; you have directories with huge files which are read sequentially and have a flat structure.

I see a real opportunity for a system when you have one gig of solid-state storage for your structured data and HDDs (note that you can now add a new HDD without any trouble, because it's only data storage, not a filesystem) for the bulk data.

Re:well and good

[DoraLives]

I see a real opportunity for a system when you have one gig of solid-state storage for your structured data

It will be OS-on-a-chip (and a good OS at that), it will go for about twenty bucks a pop down at WalMart or CompUSA and Bill Gates will die of an apoplectic fit when it hits the streets. Hackers will figure out ways to diddle it, but corporations and average users will upgrade by merely dropping another sawbuck on the counter and plugging the damned thing in when they get back to their machine(s). Computers will come with these things preinstalled, so there'll be no bitching about not having an OS with any given machine. High-end weirdness will, as ever, continue to drive a niche market, but everybody else will regard it about the same as they regard their pair of pliers; just another tool. Ho hum.

Re:well and good

[GreyPoopon]

Around 40% of new computers we sell are sold with lcd's, wake up and smell the pixels.

I think you're just confirming what the parent said ... we're real close to replacing the CRT in most uses with LCD.

Re:well and good

[Milik]

If you consider this new media it will require completely new device that will be build from scratch allowing for the battery and permanent memory. Also you have to worry about reliable memory and batteries, not only disks. That will reduce safety of the data. Also such disks will not be used in RAID arrays. I see this technology making it to very small market of embedded devices and computers but not to every computer. Once we can fit 60-80 GB of RAM inside of the HDD sized box than we can talk about new media, and death of HDD. This is a major step on the way but I don't' believe that we will ever implement this technology

Re:well and good

[Daniel Phillips]

I've predicted and eagerly anticipated the demise (by replacement) of spinning media (magnetic and optical) for 10 or more years now... I've predicted it will happen, not when.

You may have to keep predicting for some time yet. So far, nobody has managed to come up with a solid-state approach that gets anywhere close to the cost of spinning media, and though solid state gets cheaper over time, spinning media does too.

For the most part, posters to this thread missed the point of this effort. The authors observed that some relatively small portion of filesystem data - the metadata - accounts for a disproportionate amount of the IO traffic. So put just that part in battery-backed ram, and get better performance. Hopefully, the increased performance will outweigh the cost of the extra RAM.

The fly in the ointment is that, in the case where there's a small amount of metadata compared to file data, the cost of transferring the metadata isn't that much. But when there's a lot of metadata, it won't all fit in NVRAM. Oops, it's not as big a gain as you'd first think.

It's surprising how well Ext2 does compared to RAMFS and ConquestFS in the author's benchmarks.

Re:well and good

[ces]

we're real close to replacing the CRT in most uses with LCD

Most high end users who are concerned about image quality are still buying CRTs. If you have to do color matching, CAD/CAM, or are gaming you probably still want a CRT.

The price differential between CRT and LCD monitors is still enough that most larger businesses are still only buying CRTs for most of their users. Sure the executives and receptionists are getting LCDs but everyone else gets cheap 17" CRT monitors.

Re:well and good

[Suidae]

you can have my CRT when you can give me a 19+ inch 2Megapixel LCD for the same price.

My code does not word wrap at 80 characters.

Re:well and good

[GreyPoopon]

Most high end users who are concerned about image quality are still buying CRTs.

Can I assume that by "image quality", you mean color-trueness within images? I don't see resolution as driver in the high-end market. There are high-resolution LCD screens available for that. But color is definitely an issue.

...CAD/CAM...

Enlighten me. What's the issue there?

gaming

I think we had an article in the last week or so about this. Apparently, some of the new LCDs with faster refresh are winning over some gamers.

The price differential between CRT and LCD monitors is still enough that most larger businesses are still only buying CRTs for most of their users.

Although I agree with you, I'm not sure WHY this is true. At risk of giving the pointy-haired boss new ideas, I'd like to observe that by replacing monitors with LCDs, you can cut down on the total space per employee and squeeze in a few extra cubes. That may offset the cost enough to make it worth it -- especially if the alternative is to rent more space. :-)

Re:well and good

[ces]

Can I assume that by "image quality", you mean color-trueness within images?

Yes color matching is the big issue.

Enlighten me. What's the issue there?

At the very high-end CRTs are still cheaper and still have better performance. Some places all of the engineers doing CAD work have something like the Sony W900 on their desk. A 24" Sony GDM-FW900 is $1,800 and does 2304x1440, a 23" Sony SDM-P232W/B is $2,600 and does 1920x1200. Also it is only fairly recently that large (19 inches and up) LCD displays have come anywhere near CRTs in cost or resolution. This is also a factor with graphic design as well.

Apparently, some of the new LCDs with faster refresh are winning over some gamers.

Still most FPS gamers buy CRTs and most that have a decent display won't be replacing it any time soon.

Although I agree with you, I'm not sure WHY this is true. At risk of giving the pointy-haired boss new ideas, I'd like to observe that by replacing monitors with LCDs, you can cut down on the total space per employee and squeeze in a few extra cubes. That may offset the cost enough to make it worth it -- especially if the alternative is to rent more space.

Don't forget reduced HVAC costs, CRTs pump out a lot of heat compared to LCDs.

In most organizations the IT people are responible for ordering computers, while facilities handles things like space allocation, squeezing in more cubes, and paying the HVAC bills. They don't necessarily talk to each other. Given the layoffs and downturns most companies I know of have a surplus of space right now so there really isn't a good reason to spend a bunch of money upgrading everyone to an LCD.

Most businesses buy the absolute cheapest monitors they can get away with for a majority of their users. Large monitors or LCDs are considered a status symbol much as laptops are. Unless you can give a valid business reason why you need something other than the standard you aren't going to get it unless you are high enough on the pecking order.

Some places may be buying LCDs for new systems but LCDs still cost slightly more and there is the status symbol problem as well.

Drawback

[ifreakshow]

One quick draw back I see in this system is on a computer where you have more small files than available RAM space. How does the system decide which small files to keep on the regular disk and which ones to keep in RAM?

Re:Drawback

[oconnorcjo]

How does the system decide which small files to keep on the regular disk and which ones to keep in RAM?

That is an algorithm issue but the worst that happens as far as I can see is that when the ram borks a little, it just means the throuput goes down while it writes more stuff to disk. I highly doubt that will be a big issue since algorithm's for determining priority [they could almost be plucked from a "VM" HOWTO] are abundent and if they are not all perfect- many are good enouph.

Re:Drawback

[AaronMB]

Since the metadata is stored in RAM, it'll have access to the atime,ctime and mtime variables quickly. Thus, it'd be pretty easy to do an LRU scheme to dump rarely-used files to disk periodically.
-Aaron

Re:Drawback

[ottffssent]

LRU eviction is somewhat costly, but highly effective. Pseudo-LRU can be much cheaper and nearly as effective. The replacement policy is not hard - it is a well-researched problem in cache design.

What I find telling is that such a system has to be implemented at all. It seems clear to me that the operating system's filesystem, in conjunction with the VM, should implement this automatically. In Linux, this is true - large portions of the filesystem get cached if you have gobs of RAM lying around. Why certain more commonly-used OSes do the exact opposite is beyond me.

From my perspective, the right way to handle this is obvious. RAM is there to be used. Just as we have multiprogramming to make more efficient use of CPU and disk resources, we should be making the best possible use of available RAM. Letting it sit idle on the odd chance the user will suddenly need hundreds of meg of RAM out of nowhere is rediculous. From the perspective of the CPU, RAM is dog slow, but from the perspective of the disk, it's blazing fast. ANYTHING that can be done to shift the burden from magnetic storage to RAM should be done. Magnetic storage excels in one area and one area only: cheap permanent storage of vast amounts of data. RAM should be used to cache oft-used data. Why is this not painfully obvious to anyone designing an operating system?

Re:Drawback

[Gromer]

I think I was at the same talk as the poster.

In point of fact, Conquest does not use LRU. Conquest uses a very simple rule- files larger than a threshold are stored on disk, and files smaller than a threshold are stored in RAM. The threshold is currently a compiled-in constant (1 MB), but plans are for it eventually to be dynamic.

The advantage of this approach is that it eliminates the many layers of indirection needed to implement LRU-type caching, which is one reason Conquest consistently outperforms FS's based on LRU cacheing.

wow, if it only could be cost effective...

[Falconpro10k]

wow... thats all i have to say, something like this could make waiting over a min or two to boot totally obselete... sort of like a "turn on" welcome to your OS of choise type of thing... i also tons of other possibilities such as high end graphics work and maybe even phasing out the disk as we know it 100%.... all solid state.. the possibilities are endless

Who are they kidding?

[asdfasdfasdfasdf]

The idea of RAM as storage is great and all, but can we work towards the elimination of STORAGE as RAM before we get to RAM as storage?

I mean, why *DO* we still have pagefiles?

A MS Gripe: I seriously don't understand why I can't turn it off completely. With multiple GB of RAM dirt cheap, writing to a disk pagefile slows my system down-- It has to!

Re:Who are they kidding?

[DigitalGlass]

what version of windows are you running? I have had no problem with turning off the pagefile in 2000 and xp, my machines have 1gb in them and they cranked when i disabled the pagefile.

should be in control panel - system - advanced - performance --- look in there for something to set the page file to 0 or to disable it.

Re:Who are they kidding?

[Neck_of_the_Woods]

Correct me if I am wrong, but the only time that it starts to write to your page file is the moment that the amount of data passes the amount of RAM you have. At which point it starts writing information from the ass end of your RAM to the pagefile. While this is not as fast as your RAM data pull it is faster than finding it on the disk, and parsing it back into usable info. It knows where it is already and drops it back to the front of your RAM storage.

This is just my limited view of what I kind of figured happened. Please correct me if I am wrong but if you turned off your page file you would be slowing yourself down a bit.

Re:Who are they kidding?

[hurtta]

Who says that you need to have pagefile? (Or is that A MS Gripe refering some certain Operating System?)

Re:Who are they kidding?

[gl4ss]

with some versions of windows it will swap to pagefile even before the ram fills up (recent too), sometimes putting there stuff that will be needed very shortly too, theres some programs that try to battle this though(i had success with one with win2k back when i still had only 128mb, supposedly it prohibited windows from swapping some constantly needed system resources to disk, and worked mostly this way, i don't have a lot of faith in 'memory managing' programs that will just do a malloc(some_big_number_of_bytes) every now and then and free that straight after supposedly resulting in most useless stuff getting thrown into the swap and leaving the ram free for what you're going to run next)

Re:Who are they kidding?

[pclminion]

I mean, why *DO* we still have pagefiles?

Well, a couple of reasons. Most important, the "pagefile" is there to protect against a hard out-of-memory condition. Modern operating systems are in the habit of overcommitting memory, which means they grant allocation requests even if the available RAM can't fulfill them. The idea is that an app will never actually be using all those pages simultaneously. If things go wrong and all that extra memory is actually needed, the system starts kicking pages to disk to satisfy the cascade of page faults. This means the system will become slow and unresponsive, but it will keep running. But say you didn't have anywhere to swap to. The system can't map a page when a process faults on it, and the process gets killed. But which process gets killed? After all, is it the process's fault if the OS decided to overcommit system memory? The swap space serves as a buffer so a real administrator with human intelligence can come in and kill off the right processes to get the system back in shape.

Swap is also important because not all data can just be reloaded from the filesystem on demand. Working data built in a process's memory is dynamic and can't just be "reloaded." If there's no swap, that means this memory must be locked in RAM, even if the process in question has been sleeping for days! We all know the benefits of disk caching on performance. Process data pages are higher priority than cache pages. Thus if old, inactive data pages are wasting space in RAM, those are pages that could have been used to provide a larger disk cache.

You basically always want swap.

Re:Who are they kidding?

[ceswiedler]

First, pagefiles (or swap partitions) are usually of fixed size. So their existance doesn't necessarily mean they're being used.

Second, RAM is used for disk cache as well as application memory space. 'Unused' RAM is really being used to cache your slow hard drive, which is a good thing.

Third, stuff is paged in and out when necessary to free up space for other stuff. It's swapped at the memory page level (4k) based on when data is used. So if you leave your system running for a while, the stuff that gets written to disk is stuff that hasn't been used.

The result is that the OS uses the physical RAM in a very efficient way. Your pagefile speeds up your system (assuming it's well written, and both NT and Linux do very well in this regard) and you really don't want to get rid of it.

Re:Who are they kidding?

[Surak]

So long as data + applications physical RAM you won't have a problem. I often have to deal with files that are several hundred megabytes in size and grow once in RAM. Your setup wouldn't work for at me.

That's why we still have swapfiles.

Re:Who are they kidding?

[Anonymous Coward]

Does not quite work like that, you can set your pagefile size to 0, but Windows 2000 wil create a small pagefile about 20 MB and still use it !

Re:Who are they kidding?

[TheRaven64]

A MS Gripe: I seriously don't understand why I can't turn it off completely. With multiple GB of RAM dirt cheap, writing to a disk pagefile slows my system down-- It has to!

This has to do with the way in which Windows NT handles disk cache and paging. Windows treats your swap file as your main RAM, and your RAM as a disk cache, so it can unify the swapping and caching alogrithms. If you disable swap, then it will act as if you have no RAM (with the exception of the unpagable RAM reserved for the kernel). This can be particularly irritating if you have a lot of disk throughput, since your RAM will be paged in preference to some of the disk cache, resulting in hundreds of page faults when you context switch to an application. To see how badly your system is affected by this enable the PD Delta column in the task manager and watch it when you switch to an app you haven't used for a while.

I believe that this is no longer the case with XP, but it is with 2000.

The next boost will be

[scorp1us]

Execute in Place (EIP)- currently, your system will copy the program to RAM. Here, you'd copy everything from volatile ram to Non-volatile ram - a rather wasteful operation don't you think?

This is not just for exe's but for datafiles as well...

Re:The next boost will be

[hey]

Doesn't Linux almost do this?
It nmap()s executables before running them.

Re:The next boost will be

[ZenShadow]

That would be mmap(). nmap is a command line network toy.

--ZS

Re:The next boost will be

[b0r1s]

The inventer of Beowulf is currently working on something called "Processor in memory", the idea being that you embed a number of smaller, slower processors within the memory to speed up the smaller, easier calculations, and send the slower, longer calculations to the main processor.

For instance, if you were searching through a huge 1000000x1000000 matrix for a single entry to hash, you don't want to have to move each and every entry to the processor to decide whether it's the right one: offload the searching work to the processor in memory, and then once the right entry is found, send that to the main processor.

It's a rather novel idea, but it seems that it'll take a few years before anyone even tries to implement it.

Dead? Hardly...

[KC7GR]

This is 'stepping-stone' technology, along the same lines as hybrid gasoline/electric vehicles. They're still depending on hard drives for mass data storage. It's just the executables, libraries, and other application-type goodies that they're sticking into RAM.

You can do exactly the same thing by sticking an operating program into any sort of non-volatile storage (EPROM, EEPROM, memory card, whatever), and including a hard drive in the same device if need be. The new filesystem they're describing simply shifts more of the load to the silicon side instead of the electromechanical realm.

In short; The Disk is far from dead. This is just a first step in that direction.

Old news.

[Nathan Ramella]

These guys have already done it..

http://www.superssd.com/products/tera-ramsan/

Up to a terabyte even.

-n

Re:Old news.

[cube_mudd]

Actually, what they've done is novel.

If you read the article, the point of Conquest is to remove the filesystem complexities that pander to disks. Giant RAM based storage arrays do nothing to simplify and streamline the filesystem code in your kernel.

Re:Old news.

[Duck_Taffy]

That RAM-SAN is remarkably inefficient, compared to HDDs. For a comparable hard drive-based SAN, you'd think the manufacturer was insane if they said it required 5,000 watts of electricity to operate. I know it's fast, but I don't want a dedicated 60-amp circuit just for a single storage device. And I can hardly imagine the heat it produces.

Yeah wutever

[Apparition-X]

God is dead. (Neitzche) Tape is dead. (Innumerable pundits) Disk is dead. (Conquest FS) Yet somehow, they all seem to be alive and kicking as of right now. I wouldnt be throwing a wake just yet for any of 'em.

Re:Yeah wutever

[gsfprez]

don't forget the most popular one of all time... Apple is dead. (everyone, starting with that stupid ass Dvorak)

unfortunately, i find any of these calls of "teachnology death" a waste of time... i'm working at a frmr TRW (now NG) location - and my cow-orker just brought in a floppy disk to my computer because they can't seem to get us network access to the printer.

nothing is dead - its all just where you're at.

Re:Yeah wutever

[Big_Monkey_Bird]

"Print is Dead" - Egon Spengler

Re:Yeah wutever

[NDSalerno]

God is dead. (Neitzche)

Actually, Hegel originally wrote that line. However, society seems to attribute this to Nietzsche because he followed up on the idea and proclaimed it louder than any of the other atheist philosophers.

Re:Yeah wutever

[moonbender]

That's tragic. Whoever this "Neitzche" fellow was, my condolences to his friends and family.

(I just searched Google for "Neitzche". 2,020 results, nearly all of them misspellings of the philosopher's name. Sad.)

Re:Yeah wutever

[Drakonian]

The already beleagured disk community was hit with a crippling bombshell today... etc etc... Truly and American icon.

Could be accomplished with a "preferential cache?"

[mcgroarty]

I wonder if a kernel could realize many of the same performance benefits with current filesystems by identifying directory inodes and small file inodes and lowering the probability of those falling out when it's time to free pages.

Who are you kidding?

[amembrane]

What do you mean you can't turn it off? I haven't had a pagefile since I hit 1GB of RAM in my desktop. The Windows XP option (System Properties, Advanced, Performance Options, Advanced, Change) is even called "No paging file".

How is this any different from ....

[binaryDigit]

a very aggressive caching algo? I mean other than the battery backed part. You should be able to attain similar performance benefits using a purely software solution assuming your app doesn't do a lot of "important" writing to "small" files (where Conquest would do it all in RAM and still be able to persist it). But things like dll's,exes and whatnot don't change.

I guess I can understand the benefits (as minor as they may be relative to price), but the thing that bothers me the most is why does it take 4 years and NSF funds to come up with something that seems so obvious?

And one major problem would be getting over the fact that if the machine craters, you can't just yank the drive and have everything there, though I assume they have some way to "flush" the ram (can't read the .ps files to check).

Re:How is this any different from ....

[platos_beard]

I'd go further than that. This is like having a caching algorithm that determines what to keep in RAM based entirely on file size and whether the data is FS metadata. That seems unlikely to be the best way to use whatever RAM you have available for caching.

Re:How is this any different from ....

[Pheersome]

I guess I can understand the benefits (as minor as they may be relative to price), but the thing that bothers me the most is why does it take 4 years and NSF funds to come up with something that seems so obvious?

First, a wee bit of background. The research is the thesis project of An-I Wang, a grad at UCLA. One of his advisors is Geoff Kuenning, who teaches at my school, Harvey Mudd College. I went to Geoff's presentation here.

If you'd seen the performance statistics, you wouldn't be calling the benefits "minor." It stomps all over XFS, ext2, and ReiserFS in pretty much every test, and as opposed to ramfs actually has the capability to store more than what fits in RAM.

Also, a big part of what they did was matching the FS behavior to typical usage patterns. All small files are in RAM because they are the most frequently used. Most large files are read mostly-sequentially: in the example of MP3s, tags are read from the last few dozen bytes, then the player application reads the rest of the file straight from the beginning to the end. Also, most large files (media being the main example) are written once and rarely or never modified.

So, my point is, Conquest is designed to perform optimally for a desktop user. That was a nontrivial task, and while it seems an obvious idea in retrospect, it seems plain to me that there was in fact a substantial amount of research that went into this design.

Obvious solution

[palad1]

Put your swap file on a Ramdisk.

Palad1, MCSE

What if the battery fails?

[GrimReality]

...battery backed RAM...

Pardon my ignorance, but what happens if the battery fails? Of course, this is highly unlikely, but just a scenario.

In a conventional disk the data would remain even if power is switched off, but a RAM would lose the data (or get corrupted or cannot be sure if the data is exactly the same).

Thank you.
GrimReality
2003-04-21 15:51:18 UTC (2003-04-21 11:51:18 EDT)

Page files considered good

[Merlin42]

Even with tons of RAM pagefiles are a GOOD thing and if used properly (Even MS uses them properly these days) they speed up the system on the whole.I have done a *little* OS development so I may not be an expert but I do have an idea what I 'm talking about.

The swapfile is where the OS puts things it hasn't used in a while. On windows this would probably include things such as the portions of IE that are now part of the OS and you are forced to have loaded even if you are not using the box for web browsing. Having placed these items in the page file frees up room for things that are currently usefull such as IO buffers/cache (disk and/or net) that can dramatically increase speed by storing things such as recently used executables, meta-information .... wait this sounds familiar ;)

That being said I think the technology discussed in this article is a bit too single minded. I think adding an extra level in the storage heirarchy between main ram and non-volitile HD is probably a good thing. My idea is to add a HUGE pile of PC100 or similar ram into a system and have this RAM accessed in a NUMA style which is becoming very popular. The nintendo GameCube uses a form of this aproach, there are two types of RAM with a smaller-faster section and a larger-slower section.

The problem with my idea is that the price difference b/w cheap-slow RAM and fast-expensive RAM is not enough to make it worth the extra complexity currently. But, I would guess that if someone took the effort to design/build cheap slow RAM they could find a niche market for a system accelerator device ... but then again I could just be not well enough informed (a little knowledge is dangerous ;) and rambling like an idiot.

Umm..

[Anonymous Coward]

My raid controller basically does this already..

It's an old IBM 3H 64 bit PCI model with 32MB of ram and battery backup.. newer 4H models support more ram.. but how is this any different?

The most used and smallest files stay in the cache.. the rest are called when needed.. and if god forbid the power fails, and the ups fails.. the card has a battery backup to write out the final changes once the drives come back online.

Re:Umm..

[Mannerism]

I believe the researchers' point was that existing filesystems were (understandably) not designed with a RAM component in mind. Therefore, caching controllers and similar solutions are inefficient compared to Conquest, which assumes the presence of the RAM and is optimized appropriately.

Reliability

[InodoroPereyra]

There are several goals in a next generation filesystem implementation. Low cost and speed are important, and conquest goes in the right direction. But how reliable is persistent RAM for storage ? Hard drives go belly up every once in a while, and we would all like to see some sort of affordable solid state storage come up and replace HDD. Persistent RAM seems to be a step in the opposit direction, or Am I wrong ?

Re:Reliability

[TClevenger]

A decent system would maintain an image of the RAM on the HDD. In case of battery failure, replace the battery, boot the system up, and it should rebuild the RAMdisk from the hard disk--just like rebuilding a drive in a RAID.

Full paper in HTML

[monk]

For those who are not PS worthy.
The paper
Looks like a great server side file system. This is finally a step away from this whole "file" madness. All storage and IO should be memory mapped, and all execution should be in place. Anything else is just silly.

Dead?

[Sandman1971]

Dead? I don't think so. Get back to me when they start using Non-volatile RAM and the price per byte is equal or less than the price for harddrives. Until then, the HD is going to be alive and kicking.

One thing I've always wondered though. Why not release an OS on an EPROM? It would make boot time and OS operations extremely fast. I'm still surprised to this day that this isn't mainstream. Ahhh, the good ol' days of Commodore when you OS was instantly on when you turned on the PC.....

Re:Dead?

[kitty tape]

Actually, the full name of the talk was "The Disk is Dead, Long Live the Disk". Obviously, the idea of eliminating the disk was meant to be taken with a grain of salt.

...goes great with 64-bit && cheap RAM

[ceswiedler]

Though I don't think it's a useful general-purpose concept to have a RAM-only FS, I'm hoping that fast RAM will catch up to magnetic disks in size. A standard FS/VM will end up caching everything if the RAM is available. I seem to recall that ext3 on Linux, if given the RAM for cache, is faster than many ramfs/tmpfs implementations. Plan9 completely removes the concept of a permanent filesystem versus temporary memory. Everything is mapped in memory, and everything is saved to disk (eventually). It's a neat concept, and it happens to go very well with 64-bit pointers and cheap RAM.

I'm hoping that hardware people will realize that we need huge amounts of fast memory...whether or not we think we need it. We're stuck in a "why would I need more RAM than the applications I run need?" kind of mindset. I think that the sudden freedom 64-bit pointers will provide to software developers will result in a paradigm shift in how memory (both permanent and temporary) is used. Though like all paradigm shifts, it's difficult to predict ahead of time exactly what the change will be like...

Same as "what if the hard drive crashes?"

[Ars-Fartsica]

Battery failure in this case is the as the case of hard drive failure in the nonvolatile model. You either have backups on another media or you are screwed.

Note that hard drive failures are still common and likely to be much more common than a battery failure, as it would be trivial to implement a scheme through which batter recharding would be automatic while the computer was plugged in. The battery would only be directly employed when the system was unplugged or the power was out. Even in that case it would be also trivial to implement a continuous/live backup system to a nonvolatile media like a hard disk, which by that point would be ridiculously cheap.

Re:Cost

[kwerle]

...cost of below $200.

For a whooping 512MB's no doubt.

Dunno where you buy your RAM, but CNET is willing to sell me Kingston memory (512MB 133 MHZ DIMM) for less than $90 (one place says $65, but I don't believe them).

Time for you to find a new RAM supplier.

Ramdrive

[DarkBlackFox]

My plan-

Once 64 bit procsesing becomes mainstream, and price per gigabyte of memory better (say, 16 gigs DDR 3200), store the OS on a small (~5 gig) hard drive partition, and transfer the entire thing to a 5 gig ramdrive on startup. Using serial ATA that shouldn't take too long, and the OS will run at dramatically increased speeds, especially if the swap is housed in the ramdrive as well. On shutdown, transfer the contents of the ramdrive back to the hard drive. With the massive RAM support 64 bit processing promises, I'll wager some incredible things are possible for those willing to experiment with technologies like this. Perhaps that's where the technology in this article is heading, although far less volatile/risky as my approach.

Not a new concept or idea at all

[Status Quo]

Way back when I was growing up, we bought an Apple IIgs. After a while, my dad went for the memory upgrade with a battery back-up option. Remeber, this is about 12 years ago. It was nice because you didn't need to wait for the system to boot anywhere near as long.

Additionally, laptops take a similar concept and save the system memory image to hard drive and just read that in order to make your boot time a little shorter when you are away from the machine and it powers down.

ram storage

[larry bagina]

Back in the 1980s, Applied Engineering produced an Apple II card called "RamKeeper", IIRC. It was a memory card backed up with a battery, so you had a permanent RAM disk, which standard software could read/write from as if it were a normal device.

Has anyone statistics on RAM/HD prices?

[mseeger]

Hi,

does anyone has a statistic on HD and RAM prices throughout the last years?

I only have a feeling, that the last years RAM prices have fallen quicker than HD prices. This will naturally lead towards such developments as mentioned.

I think it would be very interesting to study the technical developments in the light of price developments. My bet is, that most inventions are not caused by bright minds but the need for them. For most technical breakthroughs, the mind is not cause but catalysator ;-).

CU, Martin

That's what I said about Linux . . .

[Idou]

"something like this could make waiting over a min or two to [re]boot totally obselete..."

But my company's IS dept. is still very content on making me reboot whenever I try to do something "luxurious," like having two applications open at the same time.

I will post my sig as soon as I finish rebooting.

I'm the one who gave the talk

[one-egg]

Well, since I'm the person who gave the talk referenced in the original post, I suppose I ought to clear up a few misconceptions for folks. I'm not going to address every objection that's been raised, because most of them have been well addressed in our papers. I'll just highlight the most common misunderstandings.

First, the full title of the talk was "The Disk is Dead! Long Live the Disk!" We make no claim that disk manufacturers are going to go out of business tomorrow; history suggests that the technology will survive for at least a decade, and probably more than two. Talk titles are intended to generate attendance, not to summarize important research results in 8 words.

Second, the most common objection to the work boils down to "just use the cache". This point has been raised repeatedly on Slashdot over the past few years. However, if you read our papers or attend one of my colloquium talks (UCSC, May 22nd -- plug), you'll learn that LRU caching is inferior for a number of reasons. We were surprised by that result, but it's true. Putting a fake disk behind an IDE or SCSI interface is even worse, since that cripples bandwidth and flexibility.

Third, for people worried about battery failures, the only question of interest is the MTBF of the system as a whole. All systems fail, which is why we keep backups and double-check them. If your disk failed every 3 days, you couldn't get work done, but there was a time when we dealt with a failure every few months. Conquest's MTBF hasn't yet been analyzed rigorously, but I believe it to be more than 10,000 hours, which is good enough to make it usable.

Finally, I have chosen not to put my talk slides on the Web, at least not for the moment. But you're welcome to mail me with questions: geoff@cs.hmc.edu. It might take me a few days to answer, so be patient.

Unfortunately . . .

[Idou]

for a company about to face MAJOR global workforce reductions, my "upgrade" would be software only, and 2k literally CRAWLS on this harware specs (some of my coworkers were dumb enough to "upgrade").

I really wish they would get off their high-horses, realize they are encapable of meeting ALL their user's needs, and let the more capable users support themselves, using the tools of their choice. Perhaps employees will be required to support themselves when "telecommuting" becomes more wide-spread and having a central IS authority will become impractical (those unable to support themselves will either be stuck at a central office or stuck without a job).

So, basically, with the new MS licensing, our IS dept. as blown its budget on software licenses that it can't effectively use, because it doesn't have enough $ left over to upgrade hardware specs. Meanwhile, the same hardware runs far better on a Knoppix live cd which is FREE.

On the bright side, maybe they will be able to afford better hardware after they lay some of us off.

And using MS products is good for the economy, how?

Netapp filer NVRAM???

[lazardo]

Network Appliance OS 'Data OnTap' has been storing FS metadata/data in battery backed NVRAM for many years. Disclosure/SEC: former employee, individual and/or family members may hold NTAP stock.