Slashdot Mirror
Conquest FS: "The Disk Is Dead"
andfarm writes "A few days ago, I sat in at a presentation of a what seems to be a new file system concept: Conquest. Apparently they've developed a FS that stores all the metadata and a lot of the small files in battery-backed RAM. (No, not flash-RAM. That'd be stupid.) According to benchmarks, it's almost as fast as ramfs. Impressive." The page linked above is actually more of a summary page - there's some good .ps research reports in there.
One quick draw back I see in this system is on a computer where you have more small files than available RAM space. How does the system decide which small files to keep on the regular disk and which ones to keep in RAM?
wow... thats all i have to say, something like this could make waiting over a min or two to boot totally obselete... sort of like a "turn on" welcome to your OS of choise type of thing... i also tons of other possibilities such as high end graphics work and maybe even phasing out the disk as we know it 100%.... all solid state.. the possibilities are endless
Execute in Place (EIP)- currently, your system will copy the program to RAM. Here, you'd copy everything from volatile ram to Non-volatile ram - a rather wasteful operation don't you think?
This is not just for exe's but for datafiles as well...
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
I wonder if a kernel could realize many of the same performance benefits with current filesystems by identifying directory inodes and small file inodes and lowering the probability of those falling out when it's time to free pages.
Correct me if I am wrong, but the only time that it starts to write to your page file is the moment that the amount of data passes the amount of RAM you have. At which point it starts writing information from the ass end of your RAM to the pagefile. While this is not as fast as your RAM data pull it is faster than finding it on the disk, and parsing it back into usable info. It knows where it is already and drops it back to the front of your RAM storage.
This is just my limited view of what I kind of figured happened. Please correct me if I am wrong but if you turned off your page file you would be slowing yourself down a bit.
Neck_of_the_Woods
#/usr/local/surf/glassy/overhead
a very aggressive caching algo? I mean other than the battery backed part. You should be able to attain similar performance benefits using a purely software solution assuming your app doesn't do a lot of "important" writing to "small" files (where Conquest would do it all in RAM and still be able to persist it). But things like dll's,exes and whatnot don't change.
.ps files to check).
I guess I can understand the benefits (as minor as they may be relative to price), but the thing that bothers me the most is why does it take 4 years and NSF funds to come up with something that seems so obvious?
And one major problem would be getting over the fact that if the machine craters, you can't just yank the drive and have everything there, though I assume they have some way to "flush" the ram (can't read the
Pardon my ignorance, but what happens if the battery fails? Of course, this is highly unlikely, but just a scenario.
In a conventional disk the data would remain even if power is switched off, but a RAM would lose the data (or get corrupted or cannot be sure if the data is exactly the same).
Thank you.
GrimReality
2003-04-21 15:51:18 UTC (2003-04-21 11:51:18 EDT)
Well, a couple of reasons. Most important, the "pagefile" is there to protect against a hard out-of-memory condition. Modern operating systems are in the habit of overcommitting memory, which means they grant allocation requests even if the available RAM can't fulfill them. The idea is that an app will never actually be using all those pages simultaneously. If things go wrong and all that extra memory is actually needed, the system starts kicking pages to disk to satisfy the cascade of page faults. This means the system will become slow and unresponsive, but it will keep running. But say you didn't have anywhere to swap to. The system can't map a page when a process faults on it, and the process gets killed. But which process gets killed? After all, is it the process's fault if the OS decided to overcommit system memory? The swap space serves as a buffer so a real administrator with human intelligence can come in and kill off the right processes to get the system back in shape.
Swap is also important because not all data can just be reloaded from the filesystem on demand. Working data built in a process's memory is dynamic and can't just be "reloaded." If there's no swap, that means this memory must be locked in RAM, even if the process in question has been sleeping for days! We all know the benefits of disk caching on performance. Process data pages are higher priority than cache pages. Thus if old, inactive data pages are wasting space in RAM, those are pages that could have been used to provide a larger disk cache.
You basically always want swap.
There are several goals in a next generation filesystem implementation. Low cost and speed are important, and conquest goes in the right direction. But how reliable is persistent RAM for storage ? Hard drives go belly up every once in a while, and we would all like to see some sort of affordable solid state storage come up and replace HDD. Persistent RAM seems to be a step in the opposit direction, or Am I wrong ?
Dead? I don't think so. Get back to me when they start using Non-volatile RAM and the price per byte is equal or less than the price for harddrives. Until then, the HD is going to be alive and kicking.
One thing I've always wondered though. Why not release an OS on an EPROM? It would make boot time and OS operations extremely fast. I'm still surprised to this day that this isn't mainstream. Ahhh, the good ol' days of Commodore when you OS was instantly on when you turned on the PC.....
It's better to burn out than to fade away
Note that hard drive failures are still common and likely to be much more common than a battery failure, as it would be trivial to implement a scheme through which batter recharding would be automatic while the computer was plugged in. The battery would only be directly employed when the system was unplugged or the power was out. Even in that case it would be also trivial to implement a continuous/live backup system to a nonvolatile media like a hard disk, which by that point would be ridiculously cheap.
One important thing to realize about storage is that people's storage needs for some types of files grow over time, but storage needs for other things do not grow significantly. For example, if you separate attachments and filter spam, you can now buy an SD card which will store all of the email you will get in the next few years; when that runs out, you'll be able to buy a card which will store the rest of the email you will ever receive. There are similar effects for all of the text you'll ever write.
Furthermore, there are a number of important directories on any system whose total size won't double in the next ten years, because they add one more file of about the same size for each program you install, and they already have ten years of stuff.
In the cases where you do have exponential growth of storage use, the structure of the stored data is extremely simple; you have directories with huge files which are read sequentially and have a flat structure.
I see a real opportunity for a system when you have one gig of solid-state storage for your structured data and HDDs (note that you can now add a new HDD without any trouble, because it's only data storage, not a filesystem) for the bulk data.
It will be OS-on-a-chip (and a good OS at that), it will go for about twenty bucks a pop down at WalMart or CompUSA and Bill Gates will die of an apoplectic fit when it hits the streets. Hackers will figure out ways to diddle it, but corporations and average users will upgrade by merely dropping another sawbuck on the counter and plugging the damned thing in when they get back to their machine(s). Computers will come with these things preinstalled, so there'll be no bitching about not having an OS with any given machine. High-end weirdness will, as ever, continue to drive a niche market, but everybody else will regard it about the same as they regard their pair of pliers; just another tool. Ho hum.
Is it fascism yet?
This has to do with the way in which Windows NT handles disk cache and paging. Windows treats your swap file as your main RAM, and your RAM as a disk cache, so it can unify the swapping and caching alogrithms. If you disable swap, then it will act as if you have no RAM (with the exception of the unpagable RAM reserved for the kernel). This can be particularly irritating if you have a lot of disk throughput, since your RAM will be paged in preference to some of the disk cache, resulting in hundreds of page faults when you context switch to an application. To see how badly your system is affected by this enable the PD Delta column in the task manager and watch it when you switch to an app you haven't used for a while.
I believe that this is no longer the case with XP, but it is with 2000.
I am TheRaven on Soylent News
My plan-
Once 64 bit procsesing becomes mainstream, and price per gigabyte of memory better (say, 16 gigs DDR 3200), store the OS on a small (~5 gig) hard drive partition, and transfer the entire thing to a 5 gig ramdrive on startup. Using serial ATA that shouldn't take too long, and the OS will run at dramatically increased speeds, especially if the swap is housed in the ramdrive as well. On shutdown, transfer the contents of the ramdrive back to the hard drive. With the massive RAM support 64 bit processing promises, I'll wager some incredible things are possible for those willing to experiment with technologies like this. Perhaps that's where the technology in this article is heading, although far less volatile/risky as my approach.
Back in the 1980s, Applied Engineering produced an Apple II card called "RamKeeper", IIRC. It was a memory card backed up with a battery, so you had a permanent RAM disk, which standard software could read/write from as if it were a normal device.
Do you even lift?
These aren't the 'roids you're looking for.
does anyone has a statistic on HD and RAM prices throughout the last years?
I only have a feeling, that the last years RAM prices have fallen quicker than HD prices. This will naturally lead towards such developments as mentioned.
I think it would be very interesting to study the technical developments in the light of price developments. My bet is, that most inventions are not caused by bright minds but the need for them. For most technical breakthroughs, the mind is not cause but catalysator ;-).
CU, Martin
If you consider this new media it will require completely new device that will be build from scratch allowing for the battery and permanent memory. Also you have to worry about reliable memory and batteries, not only disks. That will reduce safety of the data. Also such disks will not be used in RAID arrays. I see this technology making it to very small market of embedded devices and computers but not to every computer. Once we can fit 60-80 GB of RAM inside of the HDD sized box than we can talk about new media, and death of HDD. This is a major step on the way but I don't' believe that we will ever implement this technology
God is dead. (Neitzche)
Actually, Hegel originally wrote that line. However, society seems to attribute this to Nietzsche because he followed up on the idea and proclaimed it louder than any of the other atheist philosophers.
for a company about to face MAJOR global workforce reductions, my "upgrade" would be software only, and 2k literally CRAWLS on this harware specs (some of my coworkers were dumb enough to "upgrade").
I really wish they would get off their high-horses, realize they are encapable of meeting ALL their user's needs, and let the more capable users support themselves, using the tools of their choice. Perhaps employees will be required to support themselves when "telecommuting" becomes more wide-spread and having a central IS authority will become impractical (those unable to support themselves will either be stuck at a central office or stuck without a job).
So, basically, with the new MS licensing, our IS dept. as blown its budget on software licenses that it can't effectively use, because it doesn't have enough $ left over to upgrade hardware specs. Meanwhile, the same hardware runs far better on a Knoppix live cd which is FREE.
On the bright side, maybe they will be able to afford better hardware after they lay some of us off.
And using MS products is good for the economy, how?
Sdelat' Ameriku velikoy Snova!
you can have my CRT when you can give me a 19+ inch 2Megapixel LCD for the same price.
My code does not word wrap at 80 characters.
Network Appliance OS 'Data OnTap' has been storing FS metadata/data in battery backed NVRAM for many years. Disclosure/SEC: former employee, individual and/or family members may hold NTAP stock.