Slashdot Mirror


Linux Gains Lossless File System

Anonymous Coward writes "An R&D affiliate of the world's largest telephone company has achieved a stable release of a new Linux file system said to improve reliability over conventional Linux file systems, and offer performance advantages over Solaris's UFS file system. NILFS 1.0 (new implementation of a log-structured file system) is available now from NTT Labs (Nippon Telegraph and Telephone's Cyber Space Laboratories)."

69 of 331 comments (clear)

  1. Bloat? by shadowknot · · Score: 3, Insightful

    Please correct me if I'm wrong here but wouldn't a log that is only appended to and never overwritten cause a massive ammount of bloat after a period of prolonged use?

    1. Re:Bloat? by Ayanami+Rei · · Score: 3, Interesting

      There is a "cleaner" that is on the TODO list.
      Of course, you can delete files and re-use the space. But the performance slows down greatly once you start filling in "holes" left in the log after wrapping to the end of the allocated area. (A similar situation to database where you might want to compact, vacuum, condense, etc. a table).

      --
      THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
    2. Re:Bloat? by Skye16 · · Score: 4, Funny

      Scratch my previous post, I actually read the article. My bad :)

    3. Re:Bloat? by ivan256 · · Score: 5, Informative

      I wrote a (unfortunatly, closed source) filesystem that was remarkably similar to this once. Generally these types of filesystems are used when you're constantly writing new data. You're going to be eating the space anyway, but you want the reliability of syncronous writes with the performance of asyncronous cached writes. Reading from these filesystems is incredibly slow in comparison.

      The version I wrote took advantage of the client's bursty IO pattern and used the slow periods to offload the data to an ext2 filesystem on a seperate disk. Hopefully your system memory was large enough that the offload to the secondary filesystem happened without any disk reads. Once that was done, the older sections of log could be re-used.... But only once the disk filled up and wrapped back to the beginning, because you want to keep your writes (essentially... There's other timing tricks you can play to get more speed) sequential.

      There's been lots of research done on this method of write structuring. Look for papers on the "TRAIL" project (also closed source), for example.

    4. Re:Bloat? by b100dian · · Score: 2, Informative

      Actually this is a journal filesystem, as opposed to journalized. That is, each file is a journal.

      --
      gtkaml.org
    5. Re:Bloat? by Alien+Being · · Score: 5, Funny

      /. is a journaled website. We can't scratch your previous post. Years from now you will be able to come back and show your grandchildren how silly you were.

    6. Re:Bloat? by DrSkwid · · Score: 2, Informative

      for some of us, our whole file system is append only :

      http://cm.bell-labs.com/sys/doc/venti/venti.html

      http://cm.bell-labs.com/magic/man2html/8/venti

      Sean Quinlan now works at Google, I'm not sure if Sean Dorward does, but it seems most of the other people who built plan9 at Bell Labs do.

      --
      There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
  2. New Improved? by TheRaven64 · · Score: 4, Insightful

    The article was a bit light on details. Perhaps someone could enlighten me as to exactly why this is better than existing log-structured filesystems, such as NetBSD's LFS.

    --
    I am TheRaven on Soylent News
    1. Re:New Improved? by cowens · · Score: 2, Informative

      On its project page LFS is listed as a related project.

    2. Re:New Improved? by EvilTwinSkippy · · Score: 2, Funny
      The article was a bit light on details. Perhaps someone could enlighten me as to exactly why this is better than existing log-structured filesystems, such as NetBSD's LFS.

      Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

      --
      "Learning is not compulsory... neither is survival."
      --Dr.W.Edwards Deming
    3. Re:New Improved? by Feyr · · Score: 4, Informative

      the why is dependent on your application,

      for common servers, or day-to-day use. it isn't

      but notice how this was developped by a telecom company? a log structured filesystem is perfect or even required, due to speed and integrity constraints (depending on the size of the network), when you're dealing with billing and monitoring data on a telecom network. you want something that's simple and extremely resistant to failures. a complete system crash (which never happen, short of nuking the box) should not result in any data loss, or the extreme minimum, and you should be able to recreate that data from somewhere else (eg, the other endpoint in a telephone network).

      a log structured filesystem allow this, the "head" is never over previous data in normal operation. you don't typically read the data back until the end of a cycle (whatever that cycle may be) or in a debugging condition. you simply append to the end. minimizing head movement, and thus increasing mtbf (replacing a disk in those things is costly)

      this is also extremely useful for logging to WORM media (write once, read many), for security logs mostly. you don't want a hacker to be able to remove them, no matter what they do

    4. Re:New Improved? by rpresser · · Score: 5, Funny

      Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

      Even worse, when many logs are added together, the problems multiply.

    5. Re:New Improved? by LaCosaNostradamus · · Score: 2, Funny

      Yes, yes, when I leave behind too large a log, it won't properly flush, and that does lead to overflow.

      Er ... we were talking about bathrooms, weren't we?

      --
      [You have a stable society when some nut guns down a schoolyard and the law doesn't change.]
  3. Horrible headline by Quasar1999 · · Score: 4, Insightful

    A lossless file system? Good lord... I most certainly hope all the exisiting file systems out there are not lossy. I have hundreds of gigabytes of data that I don't want to lose.

    Or is this filesystem somehow able to recover data once the hard drive crashes? That would be neat...

    --

    ---
    Programming is like sex... Make one mistake and support it the rest of your life.
    1. Re:Horrible headline by TheRaven64 · · Score: 5, Informative
      The title was written by a numpty. This is a log-structured filesystem. These systems have been around for ages. NetBSD has LFS (originally from 4.4BSD), and I believe Minix also had some form of log-structured filesystem.

      A log-structured filesystem doesn't modify existing files. Every time you write to the disk, you simply append some deltas. This gives very good write performance, but poor read performance (since almost all files will be fragmented, and the entire log for that file must be replayed to determine the current state of the file). To help alleviate this, most undergo a vacuuming process[1], whereby the log is replayed, and a set of contiguous files is written. This also frees space - something that is not normally done since deleting a file is done simply by writing something at the end of the log saying it was deleted. In addition to the good write performance, log-structured filesystems also have an intrinsic undo facility - you can always revert to an earlier disk state, up until the last time the drive was vacuumed.

      The snapshot facility is not particularly impressive. It's a feature intrinsic to log-structured filesystems, and also available in other filesystems (such as UFS2 on FreeBSD and XFS on Linux). The performance advantage claims must be taken with a grain of salt - write performance for log-structured filesystems is always close to the theoretical maximum of the disk, but this is at the expense of some disk space, and read speed (although LFS did beat UFS in several tests on NetBSD).

      [1] This is usually done in the background when there is little or no disk activity.

      --
      I am TheRaven on Soylent News
    2. Re:Horrible headline by addaon · · Score: 5, Insightful

      It should be said that "good write performance, bad read performance" is essentially the point, not a defect. It's easy these days to speed up reads a huge amount through caching; these days 100MB+ of UBC isn't rare. But when you have to write, you have to write (for reliability reasons); this can't be cached into memory, so it should be optimized for. The goal here is to make BOTH operations as fast as possible, though one is made fast at the disk layer and one is made fast above it.

      --

      I've had this sig for three days.
    3. Re:Horrible headline by TheRaven64 · · Score: 2, Informative

      I hope you deal with very large documents. If you are dealing with things under a couple of MBs then you will get better performance by overwriting the entire file than by writing small chunks (at least, on mechanical hard disks - it's different if you are using Flash). Hard disks are best at transferring large amounts of contiguous data - small reads and writes can cripple their performance.

      --
      I am TheRaven on Soylent News
    4. Re:Horrible headline by addaon · · Score: 2, Interesting

      But, on a log based file system, writing is (intended to be) "fast enough". The problem with writing to a normal file system is mechanical issues; moving the head, waiting for rotation, etc. The (unobtainable in the limit) goal of a log based file system is to be able to start writing /immediately/ with the current head position; therefore, latency is (or approaches) zero and bandwidth is limited by disk bandwidth (which is, in practice, not the limiting point for most small writes).

      --

      I've had this sig for three days.
  4. So... by Juiblex · · Score: 5, Funny

    If it is lossless, I won't be able to store MPEG, XVid, JPEG and MP3 on it anymore? :(

    1. Re:So... by valeriyk · · Score: 2, Funny

      No, but you can use the soon to be released MILF 1.0 file system for your jpg and mpg needs.

    2. Re:So... by BottleCup · · Score: 3, Funny

      No, but you can use the soon to be released MILF 1.0 file system for your jpg and mpg needs.

      Now that's one filesystem I would like to fsck upon every boot(y) ;)

  5. Old news by Anonymous Coward · · Score: 5, Funny

    Websites with MILFS have been around for years.

    Oh, wait. NILFS. My bad.

    1. Re:Old news by SatanicPuppy · · Score: 4, Funny

      Well, we know it doesn't stand for nerds...

      (Sorry...Couldn't resist)

      --
      ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
    2. Re:Old news by schon · · Score: 3, Funny

      Willow wasn't a nerd, she was a geek (geeks have social skills.)

      I can think of at least one Norwegian-ILF (Kristanna Loken.)

  6. Database Servers by mysqlrocks · · Score: 4, Insightful

    Log-structured filesystems write down all data in a continuous log-like format that is only appended to, never overwritten. The approach is said to reduce seek times, as well as minimizing the kind of data loss that occurs with conventional Linux filesystems.

    This sounds a lot like how database servers work. They keep both a log file and a database file. The log file is continuously written to and is only truncated when backups occur.

  7. The dreaded question by digitalgimpus · · Score: 3, Funny

    Will there be a Windows Driver?

    If there isn't, this has no chance on taking off. Consumers today want portability. They don't like lock-in. A linux exclusive format is lock-in.

    Create a good windows (and Mac OS) driver, and it's got massive potential.

    1. Re:The dreaded question by pesc · · Score: 3, Insightful

      Consumers today want portability. They don't like lock-in.
      That's unfortunately not true, which is proved by all the people using NTFS (or Office).

      --

      )9TSS
    2. Re:The dreaded question by reynaert · · Score: 3, Funny

      Will there be a Windows Driver? If there isn't, this has no chance on taking off.

      Yes, that's why I only use FAT filesystems on my Linux server.

    3. Re:The dreaded question by thc69 · · Score: 2, Informative
      Will there be a Windows Driver? If there isn't, this has no chance on taking off.

      Yes, that's why I only use FAT filesystems on my Linux server.
      You're probably joking, but fyi... There's at least one driver for mounting ext2 fs in windows: ext2fsd. If you don't need to mount it, explore2fs works well too.
      --
      Procrastination -- because good things come to those who wait.
  8. Stable? by theJML · · Score: 5, Informative
    I like how they say it's reached a stable release but if you look at the known bugs on the Project Home Page http://www.nilfs.org/ You'll see that:

    The system might hang under heavy load.

    The system hangs on a disk full condition.
    Aren't those kind of important to saying that something is stable?

    --
    -=JML=-
    1. Re:Stable? by Ulrich+Hobelmann · · Score: 2, Informative

      Oh, hanging when the disk is full wouldn't be that bad. The real problem is that once your disk (i.e. the log) is full, you'll never have a non-full hard disk again, because NILFS doesn't yet have a working garbage collector ;)

      News that matters.

  9. Here's an overview for lazy people like me by Work+Account · · Score: 3, Informative

    NILFS is a log-structured file system developed for the Linux kernel 2.6. NILFS is an abbreviation of the New Implementation of a Log-structured File System. A log-structured file system has the characteristic that all file system data including metadata is written in a log-like format. Data is never overwritten, only appended in this file system. This greatly improves performance because there is little overhead regarding disk seeks. NILFS also has the following specific features:

            * Slick snapshots.
            * B-tree based file and inode management.
            * Immediate recovery after system crash.
            * 64-bit data structures; support many files, large files and disks.
            * Loadable kernel module; no recompilation of the kernel is required.

    --

    If you "get" pointers add me as a friend (116)!
  10. NTFS by JustASlashDotGuy · · Score: 2, Interesting

    " When the system reboots, the journal notes that the write did not complete, and any partial data writes are lost. "

    Isn't this similar to NTFS's journaling file system?

  11. Bundling by superpulpsicle · · Score: 2, Interesting

    If they are serious about a filesystem, it has to be bundled with the linux distros every release. Take Reiser and JFS for example, some distros have it, some don't. Not every release of the same distro has it, what a mess. Only two have stayed permanently EXT2, EXT3. Everything else is trendy.

    1. Re:Bundling by kobaz · · Score: 2
      If they are serious about a filesystem, it has to be bundled with the linux distros every release. Take Reiser and JFS for example, some distros have it, some don't. Not every release of the same distro has it, what a mess. Only two have stayed permanently EXT2, EXT3. Everything else is trendy.

      Reiser and JFS have been in the mainline kernel since umm, I think early 2.4. They were put in around the same era that ext3 showed up in the mainline kernel.

      I don't know about you but I never use the included kernel of a distro (except in the install phase) because 90% of the time it doesn't have all the drivers compiled in that I need for the system I'm using.
      --

      The goal of computer science is to build something that will last at least until we've finished building it.
  12. Shutdown versus power off by Matt+Perry · · Score: 2, Funny

    More file integrity is always good. Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command. It never made sense to me that we'd have to "shut down" as opposed to just turning the thing off.

    --
    Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
    1. Re:Shutdown versus power off by pesc · · Score: 4, Interesting

      Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command.

      That's a very bad idea. Normally, journaling file systems only guarantee that the file/directory structure remains intact. It does not necessarily guarantee that the data in the files hit the disk. Also, your disk will probably have a cache that is lost when you remove power. Whatever is in the cache will also be lost.

      So your file system may be intact, but your practices will probably destroy data.

      --

      )9TSS
    2. Re:Shutdown versus power off by dlamming · · Score: 2, Informative

      Don't be ridiculous. Every drive in the last 5 years, maybe the last 10, is able to park the head safely even in the event of your pulling the plug on the drive itself. It's got some springy/inductory dealie that pulls the head to a safe landing area.

      That doesn't mean that you won't lose data that hasn't been written yet, of course. :)

      --
      Not only am I a scientist, I play one on TV
    3. Re:Shutdown versus power off by TheRaven64 · · Score: 2, Interesting
      Wow, just wow. Seriously, I really, really hope you don't have any important data.

      Here's a little (simplified) tutorial on what happens when you a program writes a file to disk:

      1. It goes into the OS filesystem cache. This is often several hundred MBs. At some point, the OS decides it is not important enough to be kept around. At this point,
      2. it is written to the hard drive. Here, it sits in the hard drive controller's on-board cache. When the cache is full,
      3. it is written to disk.
        • At any given time, you may have several hundred MBs of data that show up when you browse your filesystem, but have never made it to disk. When you hit the power button, all of this data evaporates as the current fades from your RAM chips (over about 10ns). The only thing journalling filesystems do to improve this is ensure that, when the data is being written to disk, if the system crashes in the middle of the write then the system can be restored to either the beginning or end of the transaction, rather than the middle. A journalled filesystem will (in theory) never be corrupted by the power failing, but that does not mean you will never lose data - you will, and potentially large amounts of it. In short, if you continue with this practice, then you are a numpty.
      --
      I am TheRaven on Soylent News
    4. Re:Shutdown versus power off by morpheus800e · · Score: 2, Informative

      Automatic head parking has been around for a LONG time - Here's the data sheet from the 120 MEGAbyte drive that I had years ago. It came in my 386. Note the following line, about half-way down:
      "Turning the system power off causes the WD Caviar to perform an automatic head park operation."

      It wasn't a high-end drive at the time, (just a consumer-level IDE drive), and was utterly obsolete years ago, yet it still had the technology to park the heads out of the way when power is disconnected. There's no way that turning off power to a new drive is going to physically damage it. You just might lose the data on it.

    5. Re:Shutdown versus power off by Marillion · · Score: 4, Funny
      Hard drives with voicecoil head positioning (some 99.99% of them sold in the past 10 years) will autopark on poweroff. The real risk is how much latency the os has between when an application saved it's data and when the buffers are flushed.

      Some applications keep files open for a long time: MySQL, gDBM-based apps, Squid. Most of those application implement their own mini-filesystems within a file optimised for task. These systems are supposed to preserve their integrity by journaling their modifications in case the underlying os doesn't.

      Switching off a computer because it has a journaling filesystem is like stopping a car by driving into a something because it has seat belts.

      --
      This is a boring sig
    6. Re:Shutdown versus power off by heson · · Score: 2, Informative
    7. Re:Shutdown versus power off by 0xABADC0DA · · Score: 3, Informative

      Close, but no cigar:

      1. It goes into the OS filesystem cache. After 5 seconds the modified data gets flushed to the disk (sometimes set to 30 sec).
      2. It is written to the hard drive. Here, it sits in the hard drive controller's on-board cache until the head arrives at the write point, which is a fraction of a second.
      3. It is written to disk.

      So it *can* happen that data is not written properly, but unlike the scary picture you paint it is extremely unlikely. Even if you just saved your data, just do a sync and you'll be fine turning the power off.

  13. actual info about the fs by cowens · · Score: 4, Informative
  14. Excellent information retention by totallygeek · · Score: 4, Funny
    I installed this lossless file system. rm is now chmod 444. I have not been able to lose information since.

    Note: instead of modding this +1 funny, mod it +0.1 pathetic.

  15. Data is the new currency my friend by Work+Account · · Score: 3, Interesting

    Aside from gasoline and water, data is the most valuable thing in the world.

    Walmart's most prized possesion is their billion-billion-billion transaction customer sales database. They use it to find things like, among other things, men tend to buy beer and diapers at the time.

    With disks costing $1.00/GB or less these days, many people including myself simply DON'T delete data anymore. I keep all my original digital photos (in .tiff format) along with full-quality movies and all the games I've ever played back to Duke Nukem on 80x386 on a RAID array that's grown to nearly 2 terabytes.

    So yes, for many people, disk space is just something you keep adding to, like you'd move from a coupe to a sedan when you have kids and when you have that 6th kid you move to a minivan and if you happen to have 2 more, you get a cargo van when #8 comes along :) Same for HDDs

    --

    If you "get" pointers add me as a friend (116)!
    1. Re:Data is the new currency my friend by Ahnteis · · Score: 2, Funny

      How about "Redundant Raid Array of Disks"?

    2. Re:Data is the new currency my friend by mc_barron · · Score: 2, Funny
    3. Re:Data is the new currency my friend by Old+Wolf · · Score: 4, Funny

      I keep all my original digital photos (in .tiff format) along with full-quality movies and all the games I've ever played back to Duke Nukem on 80x386 on a RAID array that's grown to nearly 2 terabytes.

      So, basically, you're going to keep Duke Nukem forever?

    4. Re:Data is the new currency my friend by pod · · Score: 2, Funny

      I prefer Redundant RAID Array myself.

      --
      "Hot lesbian witches! It's fucking genius!"
  16. There's no replacement for ext3fs yet for me... by davegaramond · · Score: 4, Insightful

    I'd looove to replace ext2/3 as my filesystem for years since it's not so fast and most distro don't include binary tree indexing for ext3 (so large dir is slow). Unfortunately I haven't been able to do so. Here are my requirements:

    1. Distro support. I don't want to have to compile my own kernel. The FS needs to be supported by the distro (Debian in this case). I want to be able to create root partition and RAID with the FS.

    2. ACL and extended attributes.

    3. extended inode attributes would be nice ("chattr +i" is handy sometimes).

    4. optionally I would like to be able to create large Bestcrypt partitions (e.g. 30GB) with that FS.

    5. fast large dir and small files performance (I have millions of small files on my desktop).

    6. no need to fsck or fast fsck (i.e. journalling or some other technique or whatever).

    7. disk quota!

    8. optionally, transparent compression and encryption will be a big plus point.

    9. Snapshots would be nice too, for consistent backups.

    10. Versioning is also very welcome.

    XFS: very close but it still has problems with #4. It also doesn't have undelete like ext2/ext3 (not that it's a requirement though).

    JFS: it just lacks many features.

    Reiser3: How's the quota support, still have to patch kernel everytime? Plus it doesn't have ACL.

    Reiser4: not ready yet.

    I might have to look at FreeBSD after all. Background fsck, hmm....

    1. Re:There's no replacement for ext3fs yet for me... by metamatic · · Score: 3, Funny

      Reiser3 works fine on Debian with no kernel patching required.

      It seems as if you're holding out for perfection, not willing to upgrade from ext3 to anything else unless you find The Perfect Filesystem. I think that's kinda silly; better to get 90% of what you need now, than to wait another 2-4 years, surely?

      --
      GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
    2. Re:There's no replacement for ext3fs yet for me... by m50d · · Score: 3, Informative
      Reiser3: How's the quota support, still have to patch kernel everytime? Plus it doesn't have ACL.

      It does have ACL, and quota support is fine at least in gentoo kernels (can't check a vanilla one atm)

      --
      I am trolling
    3. Re:There's no replacement for ext3fs yet for me... by swillden · · Score: 2, Insightful

      9 is basically a cron job

      Ummm, no.

      9 (snapshots), is a very important feature that makes it possible to create a cron job to do nice, consistent backups easily. Unless you don't mind writing a cron job that remounts the fs as read-only before doing the backup... an approach that's likely to cause one or two small problems.

      That said, snapshotting doesn't need to be implemented in the file system. LVM, for example, implements it below the file system, at the level of a block device. That approach has the benefit of making it applicable to any file system.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
  17. Re:Needs a new name by chochos · · Score: 4, Funny

    NILF: Netserver I'd Like to fsck (but I don't need to anymore, apparently)

  18. HDFS (home-dir FS)? by Ramses0 · · Score: 4, Interesting

    I've had an idea kicking around for a while now... "HDFS / Home-Dir File System" ... I want a (s)low-performance, bloated, version controlled, roll-back featured, viewcvs.cgi enabled file system for my /home/rames (or at least /home/rames/documents).

    With FUSE it might even be possible for mere mortals like me.

    Basically, I very rarely push more around more than 100-200kb at a time of "my stuff" unless it's big OGG's or tgz's, etc. Mostly source files, documents, resume's, etc. In that case, I want to be able to go historical to any saved revision *at the file-system level*, kindof like "always on cvs / svn / (git?)" for certain directories. Then when I accidently nuke files or make mistakes or whatever, I can drag a slider in a GUI and "roll-back" my filesystem to a certain point in time and bring that saved state into the present.

    Performance is not an issue (at first), as I'm OK if my files take 3 seconds to save in vim or OpenOffice instead of 0.5 seconds. Space is not an issue because I don't generally revise Large(tm) files (and it would be pretty straightforward to have a MaxLimit size for any particular file). Maintenance would also be pretty straighforward: crontab "@daily dump revisions > 1 month". Include some special logic for "if a file is changing a lot, only snapshot versions every 5-10 minutes" and you could even handle some of the larger stuff like images without too much work.

    Having done quite a bit of reading of KernelTraffic (Hi Zack) and recently about GIT, maybe it's time to dust off some python and C and see what happens...

    --Robert

    1. Re:HDFS (home-dir FS)? by RosenSama · · Score: 2, Informative
      How about mounting a WebDAV. I read that subversion can work this way when I upgraded to 1.2. Quoting from http://subversion.tigris.org/svn_1.2_releasenotes. html
      Autoversioning is a feature whereby generic WebDAV clients can write to a DeltaV server (like mod_dav_svn), and the server performs commits silently in the background. This means that if you use Apache httpd as your Subversion server, then most modern operating systems can mount the repository as a network share, and non-technical users get "transparent" versioning for free. (Of course, technical users can still use Subversion clients to examine repository history.)
    2. Re:HDFS (home-dir FS)? by Dr.Dubious+DDQ · · Score: 2, Informative

      You could possibly implement it with DavFS...

  19. Re:Needs a new name by EvilTwinSkippy · · Score: 2, Funny

    You know, F'ed is the last state I want to see my file system in.

    --
    "Learning is not compulsory... neither is survival."
    --Dr.W.Edwards Deming
  20. Nothing has been gained just yet... by Jason+Hood · · Score: 2, Interesting

    Here is a sampling of the known bugs

    The system might hang under heavy load.

    --
    Are you intolerant of intolerant people?
  21. RCS or something similar by jd · · Score: 2
    For ASCII text, the easiest way to maintain a log system is to use a revision control system such as RCS, CVS, Subversion, etc. Essentially, that's all these systems are - except they usually support forks in the log, they're not strictly linear collections of diffs.


    For binary document formats (eg: MS Office's .doc format), things get tougher. There are versions of diff that'll work on binary files (which is why you can get binary patches), but it's more common to see logging done as a series of macros where the instruction sequence is replayed to get the same result, as opposed to recording actual changes. (Most so-called program "flight recorders" operate by this method.)


    A better approach might be a hybrid of these - have a method of extracting/inserting the text as ASCII (which you can then log using a conventional source-control system) and have control sequences recorded as a macro on top of that. You then have to be absolutely certain that the correct two are applied together and in the right order, which may get complicated if there are branches, but the increase in complexity should be more than matched by a reduction in the amount you need to actually store.

    --
    It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  22. Not really +1, Insightful by hummassa · · Score: 2, Informative

    Because old stuff can be overwritten when you fill the whole disk. As I mentioned in other posting, data writes are Real Fast in log filesystems, but data reads are Real Slow.

    The biggest problem of this filesystem (link is missing from the original posting) is that it's Not Really Ready (among other important stuff, mmap() is not implemented yet).

    --
    It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
  23. Re:Faster by shmlco · · Score: 2, Insightful

    Un huh. And the extra processing time the required "cleaner" takes up shouldn't be charged to the overall speed, much like Java's GC?

    --
    Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
  24. VMS isn't entirely closed source... by Medievalist · · Score: 2, Interesting

    If you have a licensed copy of the OS you can get source. Or at least, you used to be able to - many years ago when I was considered a VMS expert (something I certainly am not now) I used to look stuff up in the fiche; most of the stuff I was referencing was written in BLISS, I think.

    As I recall, RMS is an indexed file management system. I wrote a molluscan taxonomy database system that used it in the 80s... but I usually encapsulate all OS-specific stuff in subroutines, so somebody has probably ported it to a cheaper database by now.

    1. Re:VMS isn't entirely closed source... by Minwee · · Score: 2, Funny
      "As I recall, RMS is an indexed file management system."

      I have heard Stallman referred to as a great many things on this site, but I think this is the first time anyone has used that comparison.

  25. Re:Lossless??? by Medievalist · · Score: 2, Funny
    Is this lossless as in "my gmail emails are lossless" or is this lossless as in "my Escalade is lossless (because when it gets stolen or lost I call onstar and they find it for me)"
    Your Escalade is lossless because nobody would steal that pig with gas at $3 a gallon!

  26. Re:getting rid of unwanted data by Just+Some+Guy · · Score: 2, Interesting
    Does such a thing exist for Linux?
    SCRATCH=/filesystem/to/clean/someUnlikelyFileName
    dd if=/dev/urandom of=$SCRATCH; rm $SCRATCH
    will probably work. It overwrites every available block on your drive with random data, then deallocates them. If anyone knows why that wouldn't work, I'd be interested in hearing it.

    Assuming that it does actually do the trick, it might be even better than wiping a single file. Since the whole drive would be filled with random data, there wouldn't be any conspicuous wiped blocks sitting in the middle of an otherwise normal looking filesystem.

    --
    Dewey, what part of this looks like authorities should be involved?
  27. Version control FS by lemaymd · · Score: 2, Interesting

    I've always wondered if it would be worthwhile to have a version control style filesystem. Basically, a built-in subversion or CVS-type backend that operates without your knowledge until you need to roll something back. Of course you'd need to manage space usage, but with 1TB storage capacities within reach of the home user perhaps it's less of an issue? I think losing data is quite a bit more expensive than storage these days. :-)

  28. Re:Linux files systems suck ass.. by csirac · · Score: 2, Informative

    So what are the choices? ext2/ext3 which are slow, reiserfs which sucks ass when it breaks..

    Apart from the big (production quality) alternatives like IBM's JFS (which I use myself) and SGI's XFS (and Reiser - "Reiser sucks when it breaks" is so 1999) Linux additionally supports the following filesystems (from http://www.xenotime.net/linux/linux-fs.html, also try http://www.tldp.org/HOWTO/Filesystems-HOWTO.html):


    * accessfs: permission filesystem
    * AFS FAQ
    * AutoFS
    * AVFS: A Virtual File System
    * AVFS: A Virtual Filesystem
    * BeFS for Linux2.4
    * BeOS filesystem for Linux
    * BFS UnixWare, Linux Implementation of
    * CDfs
    * CIFS
    * Cluster File Systems Home
    * Coda File System
    * Crypto HOWTO
    * convertfs: convert Linux filesystems
    * DAFS: Direct Access File System
    * Devfs (Device File System) FAQ
    * efslook: read/debug EFS filesystems (ported to Linux & x86) {new}
    * Enhanced Loopback (for XFS)
    * E2fsprogs: Ext2 Filesystem Utilities
    * E2fsprogs: Ext2 FS Utilities
    * Ext2 Compression Extension
    * FSDEXT2: ext2 filesystem for Win95
    * ext2fsnt: Linux Ext2 filesystem for Windows NT driver
    * ext2 fs resize utilities
    * ext2 online growth patch (adilger)
    * Explore2fs
    * FHS: Filesystem Hierarchy Standard
    * FHS Test Suite
    * Filesystems HOWTO
    * FiST: File System Translator
    * FreeVXFS
    * fsv: 3D File System Visualizer
    * Global File System (Sistina)
    * GNU ext2resize: ext2 filesystem resizer
    * HFS+ filesystem
    * HFS+ filesystem #2
    * InterMezzo FS
    * JFFS (axis.com)
    * Journaling Flash File System, v2
    * JFS for Linux (IBM)
    * journal_fs report (OSDL)
    * Kragen's Amazing List of Filesystems
    * Large File Support in Linux
    * Large File Summit (LFS) stuff
    * Large File / File System Support
    * Linux EXT2 salvage utility
    * Linux Ext2fs Undeletion mini-HOWTO
    * Linux FAT-32 support
    * Linux filesystem redundancy
    * Linux Links: File Systems
    * Linux NFS FAQ
    * Linux RAID Solutions
    * LUFS: Linux Userland File System
    * Lustre & Lustre Light filesystems
    * NFS project
    * NFSv4 for Linux 2.4
    * Linux NTFS file system support
    * loop-aes cryptoloop device