Conquest FS: "The Disk Is Dead"

andfarm writes "A few days ago, I sat in at a presentation of a what seems to be a new file system concept: Conquest. Apparently they've developed a FS that stores all the metadata and a lot of the small files in battery-backed RAM. (No, not flash-RAM. That'd be stupid.) According to benchmarks, it's almost as fast as ramfs. Impressive." The page linked above is actually more of a summary page - there's some good .ps research reports in there.

Comments

well and good

[iamweezman]

this is great. We all have seen this coming, but how is the industry going to take this and implement it. My bet is it won't. The only way that it will take hold is if you can find some small application that will take and apply it.

Re:well and good

[robslimo]

I've predicted and eagerly anticipated the demise (by replacement) of spinning media (magnetic and optical) for 10 or more years now... I've predicted it will happen, not when.

As this new filesystem implicitly admits, the price/MB is still so much dramatically lower for HDD's than solid state memory, it will still take quite a will for this replacement to happen.

I disagree that some small killer app must come along to make this happen. Yes, solid state media is coming down in cost and increasing in density, but both need to change by 2 or 3 orders of magnitude before the HDD is dead. What we're waiting for here is the classis convergence of technology and its applications... the apps won't some until the technology can support it and the tech is driven by our demand for it. Expect another 10 years at least.

Re:well and good

[CrosbieSmith]

It will happen when the price difference between solid-state devices and magnetic storage gets narrower. That's not happening.

This was also pointed on Saturday's Slashdot Story

A mere $US5,000 would be something of a price sensation by the standards of current large capacity SSDs, whose prices aren't dropping nearly as quickly as are those of magnetic media.

Re:well and good

[robslimo]

True, and that narrowing will have occurred by the time the cost/density ratio of SSM has improved by 2 or 3 orders of magnitude.

A couple of reasons I see the death of the HDD to be not-to-imminent:

(1) Those damned HDD makers keep pulling new physics out of their as^H^H hats and keep pushing the storage densities to rediculous new levels.

(2) the solid state memory of the future ainta gonna be Flash as we know it now (with slow and limit write cycles) and it also will not be battery-backed RAM (unless we go write it all back to disk for 'permanent' storage at some point). I bet on some variation on today's Flash without its limitations, but the tech has got some ground to make before this all happens.

My other long-term prediction has been that CRTs (vacuum tube, for pete's sake!) will be replaced with LCD or similar tech and we're getting really close.

Re:well and good

[GreyPoopon]

Around 40% of new computers we sell are sold with lcd's, wake up and smell the pixels.

I think you're just confirming what the parent said ... we're real close to replacing the CRT in most uses with LCD.

Re:well and good

[Daniel Phillips]

I've predicted and eagerly anticipated the demise (by replacement) of spinning media (magnetic and optical) for 10 or more years now... I've predicted it will happen, not when.

You may have to keep predicting for some time yet. So far, nobody has managed to come up with a solid-state approach that gets anywhere close to the cost of spinning media, and though solid state gets cheaper over time, spinning media does too.

For the most part, posters to this thread missed the point of this effort. The authors observed that some relatively small portion of filesystem data - the metadata - accounts for a disproportionate amount of the IO traffic. So put just that part in battery-backed ram, and get better performance. Hopefully, the increased performance will outweigh the cost of the extra RAM.

The fly in the ointment is that, in the case where there's a small amount of metadata compared to file data, the cost of transferring the metadata isn't that much. But when there's a lot of metadata, it won't all fit in NVRAM. Oops, it's not as big a gain as you'd first think.

It's surprising how well Ext2 does compared to RAMFS and ConquestFS in the author's benchmarks.

Re:well and good

[ces]

we're real close to replacing the CRT in most uses with LCD

Most high end users who are concerned about image quality are still buying CRTs. If you have to do color matching, CAD/CAM, or are gaming you probably still want a CRT.

The price differential between CRT and LCD monitors is still enough that most larger businesses are still only buying CRTs for most of their users. Sure the executives and receptionists are getting LCDs but everyone else gets cheap 17" CRT monitors.

Re:well and good

[GreyPoopon]

Most high end users who are concerned about image quality are still buying CRTs.

Can I assume that by "image quality", you mean color-trueness within images? I don't see resolution as driver in the high-end market. There are high-resolution LCD screens available for that. But color is definitely an issue.

...CAD/CAM...

Enlighten me. What's the issue there?

gaming

I think we had an article in the last week or so about this. Apparently, some of the new LCDs with faster refresh are winning over some gamers.

The price differential between CRT and LCD monitors is still enough that most larger businesses are still only buying CRTs for most of their users.

Although I agree with you, I'm not sure WHY this is true. At risk of giving the pointy-haired boss new ideas, I'd like to observe that by replacing monitors with LCDs, you can cut down on the total space per employee and squeeze in a few extra cubes. That may offset the cost enough to make it worth it -- especially if the alternative is to rent more space. :-)

Re:well and good

[ces]

Can I assume that by "image quality", you mean color-trueness within images?

Yes color matching is the big issue.

Enlighten me. What's the issue there?

At the very high-end CRTs are still cheaper and still have better performance. Some places all of the engineers doing CAD work have something like the Sony W900 on their desk. A 24" Sony GDM-FW900 is $1,800 and does 2304x1440, a 23" Sony SDM-P232W/B is $2,600 and does 1920x1200. Also it is only fairly recently that large (19 inches and up) LCD displays have come anywhere near CRTs in cost or resolution. This is also a factor with graphic design as well.

Apparently, some of the new LCDs with faster refresh are winning over some gamers.

Still most FPS gamers buy CRTs and most that have a decent display won't be replacing it any time soon.

Although I agree with you, I'm not sure WHY this is true. At risk of giving the pointy-haired boss new ideas, I'd like to observe that by replacing monitors with LCDs, you can cut down on the total space per employee and squeeze in a few extra cubes. That may offset the cost enough to make it worth it -- especially if the alternative is to rent more space.

Don't forget reduced HVAC costs, CRTs pump out a lot of heat compared to LCDs.

In most organizations the IT people are responible for ordering computers, while facilities handles things like space allocation, squeezing in more cubes, and paying the HVAC bills. They don't necessarily talk to each other. Given the layoffs and downturns most companies I know of have a surplus of space right now so there really isn't a good reason to spend a bunch of money upgrading everyone to an LCD.

Most businesses buy the absolute cheapest monitors they can get away with for a majority of their users. Large monitors or LCDs are considered a status symbol much as laptops are. Unless you can give a valid business reason why you need something other than the standard you aren't going to get it unless you are high enough on the pecking order.

Some places may be buying LCDs for new systems but LCDs still cost slightly more and there is the status symbol problem as well.

Who are they kidding?

[asdfasdfasdfasdf]

The idea of RAM as storage is great and all, but can we work towards the elimination of STORAGE as RAM before we get to RAM as storage?

I mean, why *DO* we still have pagefiles?

A MS Gripe: I seriously don't understand why I can't turn it off completely. With multiple GB of RAM dirt cheap, writing to a disk pagefile slows my system down-- It has to!

Re:Who are they kidding?

[hurtta]

Who says that you need to have pagefile? (Or is that A MS Gripe refering some certain Operating System?)

Re:Who are they kidding?

[ceswiedler]

First, pagefiles (or swap partitions) are usually of fixed size. So their existance doesn't necessarily mean they're being used.

Second, RAM is used for disk cache as well as application memory space. 'Unused' RAM is really being used to cache your slow hard drive, which is a good thing.

Third, stuff is paged in and out when necessary to free up space for other stuff. It's swapped at the memory page level (4k) based on when data is used. So if you leave your system running for a while, the stuff that gets written to disk is stuff that hasn't been used.

The result is that the OS uses the physical RAM in a very efficient way. Your pagefile speeds up your system (assuming it's well written, and both NT and Linux do very well in this regard) and you really don't want to get rid of it.

Dead? Hardly...

[KC7GR]

This is 'stepping-stone' technology, along the same lines as hybrid gasoline/electric vehicles. They're still depending on hard drives for mass data storage. It's just the executables, libraries, and other application-type goodies that they're sticking into RAM.

You can do exactly the same thing by sticking an operating program into any sort of non-volatile storage (EPROM, EEPROM, memory card, whatever), and including a hard drive in the same device if need be. The new filesystem they're describing simply shifts more of the load to the silicon side instead of the electromechanical realm.

In short; The Disk is far from dead. This is just a first step in that direction.

Nonvolatile cache does similar things

[Anonymous Coward]

Nice to have a filesys like this, but a cache living on nonvolatile storage that is much faster than the host is probably easier to get folks to adopt. That notion can be used with cache on fast disk being used with backing store on some much slower technology too. Only changes to the metadata about what is in cache need be synched across a cluster as I recall (designed one c. 1995 but never built it). The direct execution part is hard to do in a pure cache system, but on the other hand >1 layer of such caching can be done. The cache has to do some of the jobs a filesystem will do, to manage what is on its store, but because it is nonvolatile, many boundary conditions during cluster state transitions become easy to handle, and as a practical matter it gets much easier to get people to adopt a cache system which is keeping their familiar filesystem and all the utilities which know about it.

...goes great with 64-bit && cheap RAM

[ceswiedler]

Though I don't think it's a useful general-purpose concept to have a RAM-only FS, I'm hoping that fast RAM will catch up to magnetic disks in size. A standard FS/VM will end up caching everything if the RAM is available. I seem to recall that ext3 on Linux, if given the RAM for cache, is faster than many ramfs/tmpfs implementations. Plan9 completely removes the concept of a permanent filesystem versus temporary memory. Everything is mapped in memory, and everything is saved to disk (eventually). It's a neat concept, and it happens to go very well with 64-bit pointers and cheap RAM.

I'm hoping that hardware people will realize that we need huge amounts of fast memory...whether or not we think we need it. We're stuck in a "why would I need more RAM than the applications I run need?" kind of mindset. I think that the sudden freedom 64-bit pointers will provide to software developers will result in a paradigm shift in how memory (both permanent and temporary) is used. Though like all paradigm shifts, it's difficult to predict ahead of time exactly what the change will be like...

Re:The next boost will be

[ZenShadow]

That would be mmap(). nmap is a command line network toy.

--ZS