Linux Gains Lossless File System

Anonymous Coward writes "An R&D affiliate of the world's largest telephone company has achieved a stable release of a new Linux file system said to improve reliability over conventional Linux file systems, and offer performance advantages over Solaris's UFS file system. NILFS 1.0 (new implementation of a log-structured file system) is available now from NTT Labs (Nippon Telegraph and Telephone's Cyber Space Laboratories)."

Comments

Bloat?

[shadowknot]

Please correct me if I'm wrong here but wouldn't a log that is only appended to and never overwritten cause a massive ammount of bloat after a period of prolonged use?

Re:Bloat?

[Ayanami Rei]

There is a "cleaner" that is on the TODO list.
Of course, you can delete files and re-use the space. But the performance slows down greatly once you start filling in "holes" left in the log after wrapping to the end of the allocated area. (A similar situation to database where you might want to compact, vacuum, condense, etc. a table).

Re:Bloat?

[Skye16]

Scratch my previous post, I actually read the article. My bad :)

Re:Bloat?

[ivan256]

I wrote a (unfortunatly, closed source) filesystem that was remarkably similar to this once. Generally these types of filesystems are used when you're constantly writing new data. You're going to be eating the space anyway, but you want the reliability of syncronous writes with the performance of asyncronous cached writes. Reading from these filesystems is incredibly slow in comparison.

The version I wrote took advantage of the client's bursty IO pattern and used the slow periods to offload the data to an ext2 filesystem on a seperate disk. Hopefully your system memory was large enough that the offload to the secondary filesystem happened without any disk reads. Once that was done, the older sections of log could be re-used.... But only once the disk filled up and wrapped back to the beginning, because you want to keep your writes (essentially... There's other timing tricks you can play to get more speed) sequential.

There's been lots of research done on this method of write structuring. Look for papers on the "TRAIL" project (also closed source), for example.

Re:Bloat?

[b100dian]

Actually this is a journal filesystem, as opposed to journalized. That is, each file is a journal.

Re:Bloat?

[Qzukk]

hm, might be interesting (read exciting but not necessarially useful) to implement this in some sort of really huge circular buffer, where old files are simply overwritten when their time is up.

Re:Bloat?

[NitsujTPU]

Actually.

Log file systems are faster, safer, and just better. Period.

Re:Bloat?

[moro_666]

faster ? did i just miss something ?

if you have a regular file system, where some data is at point X and your new high-end apps just mmap it for usage, it's super easy for the kernel, just copy the data over ata/sata/scsi commands as it comes and goes, nothing is easier and for sure nothing is faster. if you have 1 journal to update in between here and then, it's rather easy to have that one in too.

now if you have a long log what should be where written by whom and why, this just wont work, awesome cool overhead will kick in and you'll be in big trouble.

they are safer, for sure. but no faster. anything besides a direct data copy is slow, period.

if i were you, i'd use the logging file systems only in places where the data isn't storaged on usual hdd-s. because hdd-s are known to have a magnetic head which writes data on a magnetic disc. running over logged data that's all over the place isnt fun for the machinery, it's really painful.

it may however be really fast on a memory card like storage, because there the performance wouldnt depend on a magnet/engine pulling the head around all the time.

Re:Bloat?

[NitsujTPU]

http://citeseer.ist.psu.edu/ousterhout88beating.ht ml
http://citeseer.ist.psu.edu/hartman93zebra.html

Whatever you say boss.

Re:Bloat?

[Philip K Dickhead]

No.

The entire file system redundancy is acheived by posting "Dupes" to Slashdot.

Re:Bloat?

[Alien Being]

/. is a journaled website. We can't scratch your previous post. Years from now you will be able to come back and show your grandchildren how silly you were.

Re:Bloat?

[Skye16]

Man down!

:(

Re:Bloat?

[DrSkwid]

for some of us, our whole file system is append only :

http://cm.bell-labs.com/sys/doc/venti/venti.html

http://cm.bell-labs.com/magic/man2html/8/venti

Sean Quinlan now works at Google, I'm not sure if Sean Dorward does, but it seems most of the other people who built plan9 at Bell Labs do.

New Improved?

[TheRaven64]

The article was a bit light on details. Perhaps someone could enlighten me as to exactly why this is better than existing log-structured filesystems, such as NetBSD's LFS.

Re:New Improved?

[cowens]

On its project page LFS is listed as a related project.

Re:New Improved?

[EvilTwinSkippy]

The article was a bit light on details. Perhaps someone could enlighten me as to exactly why this is better than existing log-structured filesystems, such as NetBSD's LFS.

Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

Re:New Improved?

[Feyr]

the why is dependent on your application,

for common servers, or day-to-day use. it isn't

but notice how this was developped by a telecom company? a log structured filesystem is perfect or even required, due to speed and integrity constraints (depending on the size of the network), when you're dealing with billing and monitoring data on a telecom network. you want something that's simple and extremely resistant to failures. a complete system crash (which never happen, short of nuking the box) should not result in any data loss, or the extreme minimum, and you should be able to recreate that data from somewhere else (eg, the other endpoint in a telephone network).

a log structured filesystem allow this, the "head" is never over previous data in normal operation. you don't typically read the data back until the end of a cycle (whatever that cycle may be) or in a debugging condition. you simply append to the end. minimizing head movement, and thus increasing mtbf (replacing a disk in those things is costly)

this is also extremely useful for logging to WORM media (write once, read many), for security logs mostly. you don't want a hacker to be able to remove them, no matter what they do

Re:New Improved?

[sbryant]

The larger log structures don't cooperate with the flush procedure; leaving things unflushed is just asking for problems - you're going to get an overflow sooner or later.

-- Steve

Re:New Improved?

[rpresser]

Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

Even worse, when many logs are added together, the problems multiply.

Re:New Improved?

[Waffle Iron]

Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

This filesystem avoids those problems with its ACQ implementation.

Re:New Improved?

[Aperculum]

not to mention fire security, fire burns easily if I'm not completely mistaken

Re:New Improved?

[LaCosaNostradamus]

Yes, yes, when I leave behind too large a log, it won't properly flush, and that does lead to overflow.

Er ... we were talking about bathrooms, weren't we?

Re:New Improved?

[phaetonic]

Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

Even worse, when many logs are added together, the problems multiply.

Naturally....

Re:New Improved?

[Ulrich Hobelmann]

Easy. While the concept of LFS and the original implementation are almost 15 years old, this isn't about a file system, it's about a technology that's being implemented on Linux, AND AS SUCH IT'S REVOLUTIONARY.

Never mind that they don't even have a garbage collector running, and when I ran LFS on NetBSD 1.6 it worked quite well, but hey, this is Slashdot ;)

Re:New Improved?

[Ulrich Hobelmann]

I tried it in NetBSD 1.6 and it worked. *Before* then it had been broken for quite some time, but there's no reason why it shouldn't work. Ok, on the enterprise thing I agree.

Re:New Improved?

[NanoGator]

"Logs structures are suceptible to termites, carpenter ants, and various forms of rot."

I always wondered what file system the Windows Registry uses. Learn something new every day!

Re:New Improved?

[dafunn]

Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

Even worse, when many logs are added together, the problems multiply.

Naturally....

Often exponentially.

Horrible headline

[Quasar1999]

A lossless file system? Good lord... I most certainly hope all the exisiting file systems out there are not lossy. I have hundreds of gigabytes of data that I don't want to lose.

Or is this filesystem somehow able to recover data once the hard drive crashes? That would be neat...

Re:Horrible headline

[TheRaven64]

The title was written by a numpty. This is a log-structured filesystem. These systems have been around for ages. NetBSD has LFS (originally from 4.4BSD), and I believe Minix also had some form of log-structured filesystem.

A log-structured filesystem doesn't modify existing files. Every time you write to the disk, you simply append some deltas. This gives very good write performance, but poor read performance (since almost all files will be fragmented, and the entire log for that file must be replayed to determine the current state of the file). To help alleviate this, most undergo a vacuuming process[1], whereby the log is replayed, and a set of contiguous files is written. This also frees space - something that is not normally done since deleting a file is done simply by writing something at the end of the log saying it was deleted. In addition to the good write performance, log-structured filesystems also have an intrinsic undo facility - you can always revert to an earlier disk state, up until the last time the drive was vacuumed.

The snapshot facility is not particularly impressive. It's a feature intrinsic to log-structured filesystems, and also available in other filesystems (such as UFS2 on FreeBSD and XFS on Linux). The performance advantage claims must be taken with a grain of salt - write performance for log-structured filesystems is always close to the theoretical maximum of the disk, but this is at the expense of some disk space, and read speed (although LFS did beat UFS in several tests on NetBSD).

[1] This is usually done in the background when there is little or no disk activity.

Re:Horrible headline

[HermanAB]

It depends on your reference. It is lossless compared to FAT and NTFS. In that context, it is a huge breakthough...

Re:Horrible headline

[Mark Gillespie]

I can understand that comment with regard to FAT. But please tell us Why NTFS is lossy? It's a journelling filesystem, as is EXT3, supports all the features that EXT3 does (and provide more). Technically you could argue that because NTFS5 supports encrpyion, you could encrypt data and forget the password, but that's the only thing I can see that makes it's any less lossy that EXT3. Or perhaps your just a *nix fanboy that likes to bash Microsoft...

Re:Horrible headline

[addaon]

It should be said that "good write performance, bad read performance" is essentially the point, not a defect. It's easy these days to speed up reads a huge amount through caching; these days 100MB+ of UBC isn't rare. But when you have to write, you have to write (for reliability reasons); this can't be cached into memory, so it should be optimized for. The goal here is to make BOTH operations as fast as possible, though one is made fast at the disk layer and one is made fast above it.

Re:Horrible headline

[HermanAB]

I had a disk corruption on NTFS. Usually it works fine, but it sure isn't bullet proof.

Re:Horrible headline

[LWATCDR]

Where I work we are looking to do the same thing within a document file formats.

Re:Horrible headline

[chrismcdirty]

NTFS is lossy because multiple times, after a clean power down, I've booted up to corrupted and missing files, essential to the operation of the OS.

Re:Horrible headline

[Mark Gillespie]

I had disk corruption on pretty much all filesystem types, Nothing is totally infallable. NTFS, EXT3, ReiserFS are pretty darn robust. My problem, was with the comment lumping FAT and NTFS in the same bag. Nothing could be further from the truth. Some people simply assume, if it's Microsoft, it's cr@p

Re:Horrible headline

[TheRaven64]

I hope you deal with very large documents. If you are dealing with things under a couple of MBs then you will get better performance by overwriting the entire file than by writing small chunks (at least, on mechanical hard disks - it's different if you are using Flash). Hard disks are best at transferring large amounts of contiguous data - small reads and writes can cripple their performance.

Re:Horrible headline

[SparkyMartin]

The title was written by a numpty

I learned a new word today!

Re:Horrible headline

[Bellum Aeternus]

I'm a filesystem layman - but isn't this a little like how Win16/32 works and it's why we have to 'defrag'? Sure sounds the same. Since switching to WinXP and NTFS I no longer get space back from a defrag, so I assume that's different.

Bleh - using M$ products pretty much guarantees that I have no idea how my PC works, doesn't it? ;-)

Re:Horrible headline

[LWATCDR]

They can be very large. They are also being constantly being written to by one source and edited by another. It is an odd application to say the least. The undo and audit trail will also be handy.

Re:Horrible headline

[Mark Gillespie]

Then you obviously have a hardware problem. Any filesytem would give you similar problems.

Re:Horrible headline

[Trepalium]

You should never have got "space back" from a defrag. The point of the defragmentation was to make sure the files are contigious on the disk, so the drive doesn't need to seek to read it all. NTFS is still prone to fragmentation, and it can be even more dangerous for NTFS than FAT. NTFS has it's file of all files (\$MFT) which can become fragmented over time (especially if you have lots of little files and sometimes get close to the disk size limit). I've seen a few instances where the MFT overflowed it's reserved area, and began becoming heavily fragmented and eventually damaged outright. Unfortunately, Microsoft's defragment tools can not defragment the MFT or other critical system files. A number of third party defragmentation tools can, however, but they generally require doing this defragmentation offline. Raxco PerfectDisk, O&O defrag and Executive Diskeeper are the three that I know have this capability.

Defragmentation is supposed to increase performance of your system on filesystems that are prone to fragmentation. Recently, defrag vendors have discovered that there are other ways of using this ability to increase performance, perhaps even by intentionally fragmenting files. They discovered that by manually placing files on disk in the order in which they would be loaded, you can often reduce load times.

Re:Horrible headline

[squoozer]

Yes, it's very different. You didn't get any space back when you defragged your drive although it might look like you did on the fancy defrag graphical display because the display doesn't depict reality very well.

FAT 16/32 (not win16/32) drives get fragmented quickly because of the way files are written to the drive in basically a splatter gun fashion with little regard to using the space sensibly. NTFS has a much more inteligent method for writting files to the disk which helps avoid fragmentation. Fragmentation still occurs it's just not as bad as when you use FAT 16/32.

This log based file system mentioned int eh article is a totally different way of laying out the file structure that isn't really comparable. The GP gave a pretty good description of how it works so there's not much point me repeating it here suffice to say that any modified file will be (possibly badly) fragmented. The fragmentation isn't the only cause of slow access times though. Playing back the log is expensive as well.

Re:Horrible headline

[Anonymous Coward]

> A lossless file system? Good lord... I most certainly hope
> all the existing file systems out there are not lossy.

I usually store my files at the "CD Quality" setting. For simple emails and stuff I'll go as low as "Radio Quality" but I have to squint a little to make out the text later due to the aliasing.

Re:Horrible headline

[LaCosaNostradamus]

How do these systems handle the alarming problem of running out of disk space? Do they progressively (file by file) degrade into a fs normal for the OS, hence allowing admins time to respond to the alert to expand disk space or manually invoke compression procedures?

Re:Horrible headline

[Lucractius]

what you describe reminds me of the RMS versioning that the ODS filesystem on OpenVMS uses. But due to its closes source nature its difficult to tell wether its actualy writing a copy of the whole file or as you said, a delta of the changes. I would be interestred to know just how it was done myself.

Re:Horrible headline

[lscotte]

A lossless file system? Good lord... I most certainly hope all the exisiting file systems out there are not lossy. I have hundreds of gigabytes of data that I don't want to lose.

It's really no big deal. I'm using mp3fs right now. Sure, everything you read is slightly different than what you originally wrote, but you learn to deal with it. It does help to pick the right psychoacoustical model so it's harder to tell exactly what data is missing. As a side effect, I can listen to all my data on an mp3 player, so that's cool.

Re:Horrible headline

[Drakonite]

Unfortunately, Microsoft's defragment tools can not defragment the MFT or other critical system files. A number of third party defragmentation tools can, however, but they generally require doing this defragmentation offline. Raxco PerfectDisk, O&O defrag and Executive Diskeeper are the three that I know have this capability.

The saddest part about that is starting with windows 2000 the defrag utility included with windows has been a cutdown version of Diskeeper... This is the only instance I can think of where I wish MS would just buy out the company (or at least all the rights to diskeeper) and distribute the full diskeeper with windows. It's not like they are strangers to the whole "buy out the company and to take their products" game.

Re:Horrible headline

[pyrotic]

If you're really serious about write performance, it's easier and much faster to throw hardware at the problem. For example, we have a RAID array with over 100M of battery backed write cache. Anywthing we want to write to disk is cached in RAM, and if there's a power outage, nothing is lost as the battery saves the RAM cache. Speed is several orders of magnitude faster than disk, at least until you run out of cache, even using a slowish filesystem like ext3.

This filesystem could be usefull for embeded high performance systems, especially if you only end up using this filesystem for rapidly changing data. The other neat thing about it is snapshot style backups (making a backup of a busy filesystem at 4am, not making a backup of a busy filesystem at 4am and waiting around till the tape finishes at 4.50, and hoping data hasn't changed on disk), but these are supported via LVM on Linux already.

Re:Horrible headline

[ErikZ]

100 Megabytes???

You can get a Gig of memory now for 80 bucks man. Tell your boss to splurge a little.

Re:Horrible headline

[addaon]

But, on a log based file system, writing is (intended to be) "fast enough". The problem with writing to a normal file system is mechanical issues; moving the head, waiting for rotation, etc. The (unobtainable in the limit) goal of a log based file system is to be able to start writing /immediately/ with the current head position; therefore, latency is (or approaches) zero and bandwidth is limited by disk bandwidth (which is, in practice, not the limiting point for most small writes).

Re:Horrible headline

[pyrotic]

The system is a 64 bit box with 2G RAM, and 192M RAID cache. It's not quite cutting edge (you can get 12G RAM and a 256M cache in a 1U box), but it's plenty fast enough if you don't need to write more than 100M per second. To give you some (slightly bogus) figures:

time dd if=/dev/zero count=64000 of=file.txt
64000+0 records in
64000+0 records out

real 0m0.339s
user 0m0.020s
sys 0m0.318s

And compared to a single 10k RPM SCSI disk

time dd if=/dev/zero count=64000 of=file.txt
64000+0 records in
64000+0 records out

real 0m6.299s
user 0m1.054s
sys 0m4.856s


That's 18 times faster using the RAID array to write an arbitary 31M file. Try that with your desktop PC. And that's with the RAID array in use by several other people. You're not going to see that kind of real-world performance boost using software.

Re:Horrible headline

[ErikZ]

That's 18 times faster using the RAID array to write an arbitary 31M file. Try that with your desktop PC. And that's with the RAID array in use by several other people. You're not going to see that kind of real-world performance boost using software.

Ok, I've got the impression that you're using system memory to do this? If so, where can I look up how to do this?

Also, my original point was that 100M isn't a lot of memory. Your file writing/reading test is interesting, but if that happens, you've blown 1/3 of your available memory on just one transaction. Of course, it's not my system and I have no idea what kind of activity you're dealing with, I'm just surprised, that's all.

Thanks for the figures, how many disks do you have in your RAID? 7500 or 10000 RPM?

Re:Horrible headline

[pyrotic]

No, this memory is built in to the RAID controller, you access it over the system bus. All the kernel sees is a device on /dev/cciss/c0d0p1 which looks just like a disk, the controller handles its own disks and cacheing. In our case, there's just a couple of 10K RPM disks in RAID 1, being used as a mail server (postfix, spamassassin, squirrelmail, dovecot IMAP) which does lots of small writes. The system spec is here, and you can try out the same controller card at hp. A few years back this kind of storage steup was considered exotic, now it's standard on most servers, and has mainline kernel support.

So...

[Juiblex]

If it is lossless, I won't be able to store MPEG, XVid, JPEG and MP3 on it anymore? :(

Re:So...

[valeriyk]

No, but you can use the soon to be released MILF 1.0 file system for your jpg and mpg needs.

Re:So...

[parasonic]

Definitely not, but look at the bright side...you'll begin to admire FLAC once you're forced to use it :)

Re:So...

[Iriel]

I wouldn't have space for that on a Linux box after my (practically monthly) regular download of $distro['foo']!

That's why I have Windows, because I can afford to lose what's on it ^_^
</toungeincheek>

Re:So...

[BottleCup]

No, but you can use the soon to be released MILF 1.0 file system for your jpg and mpg needs.

Now that's one filesystem I would like to fsck upon every boot(y) ;)

Re:So...

[shione]

Hey at least it's not in WMA. chug chug chug...

Old news

[Anonymous Coward]

Websites with MILFS have been around for years.

Oh, wait. NILFS. My bad.

Re:Old news

[schon]

Oh, wait. NILFS. My bad.

OK.. so we all know MILFs are "Mothers I'd like to"...

What would NILFs be?

Norweigians?

Re:Old news

[SatanicPuppy]

Well, we know it doesn't stand for nerds...

(Sorry...Couldn't resist)

Re:Old news

[schon]

Willow wasn't a nerd, she was a geek (geeks have social skills.)

I can think of at least one Norwegian-ILF (Kristanna Loken.)

Re:Old news

[Feyr]

you need to see more norwegians if you can only think of one

Re:Old news

[schon]

you need to see more norwegians if you can only think of one

Or I need to increase my knowledge of nationality of the *ILFs I have seen. :o)

Re:Old news

[KillShill]

http://www.imdb.com/title/tt0096446/

hmmm.

i'm pretty sure he wasn't a geek or a nerd.

but you never know.

Database Servers

[mysqlrocks]

Log-structured filesystems write down all data in a continuous log-like format that is only appended to, never overwritten. The approach is said to reduce seek times, as well as minimizing the kind of data loss that occurs with conventional Linux filesystems.

This sounds a lot like how database servers work. They keep both a log file and a database file. The log file is continuously written to and is only truncated when backups occur.

The dreaded question

[digitalgimpus]

Will there be a Windows Driver?

If there isn't, this has no chance on taking off. Consumers today want portability. They don't like lock-in. A linux exclusive format is lock-in.

Create a good windows (and Mac OS) driver, and it's got massive potential.

Re:The dreaded question

[pesc]

Consumers today want portability. They don't like lock-in.
That's unfortunately not true, which is proved by all the people using NTFS (or Office).

Re:The dreaded question

[reynaert]

Will there be a Windows Driver? If there isn't, this has no chance on taking off.

Yes, that's why I only use FAT filesystems on my Linux server.

Re:The dreaded question

[Soup50]

Will there be a Windows Driver? If there isn't, this has no chance on taking off.

Yeah. that's why there is no real market for ext2/ext3 as well. Because windows cant read it...

Re:The dreaded question

[smeager]

Actually there are many drivers out there that will let you read from ext2/ext3 filesystems on windows. I use Ext2fsd to read my linux partitions on my windows machine (dual boot) and it works great. (It would be nice though if there was a cross-platform FS (other then FAT) that would allow rw, without any hick-ups. The newest version of Ext2fsd 0.25 states it has read/write capabilities but I have yet to try it).

Re:The dreaded question

[thc69]

Will there be a Windows Driver? If there isn't, this has no chance on taking off.

Yes, that's why I only use FAT filesystems on my Linux server.

You're probably joking, but fyi... There's at least one driver for mounting ext2 fs in windows: ext2fsd. If you don't need to mount it, explore2fs works well too.

Re:The dreaded question

[WhiteWolf666]

Huh?

Say, what?

The following is left as an exercise to the reader:
1. Please list all the linux file systems avaliable.

2. Please list all the linux file systems avaliable with read/write support in both linux and windows.

3. Please add up the total amount invested by various corporations in the development of the file systems listed in #1.

Please don't forget that although you may use differently tweaked filesystems between servers and desktops, there is a great deal of overlap. Linux as a desktop system may not be popular (yet. I use nothing but Linux on my desktops). Linux is *quite* popular as a server OS, as is Unix, and the two share quite a few filesystems. IBM, Sun, SGI, SuSE, Redhat, Novell, Mandriva, Google, and a variety of smaller tech laboratories spend a ton of money on filesystem research, very little of which ever finds its way to Windows.

Re:The dreaded question

[Aranth Brainfire]

Yes, because Windows users have so many choices for file systems, and the average person knows exactly what the differences between FAT32 and NTFS are.

Re:The dreaded question

[HishamMuhammad]

In what way is an open-source implementation lock-in? The problem of lock-in is not being able to switch in the future, whatever the future looks like. The fact that you have three implementations (Linux, Win, Mac) doesn't mean a thing about your options wrt whatever may come next. The fact that the implementation is open-source guarantees that you can have the filesystem ported, and your work accessible, in whatever happens to be the flavor du jour be it, today, next year or 20 years from now.

Re:The dreaded question

[reynaert]

You're probably joking,

Yeah, you probably need a new sarcasm detector.

Needs a new name

[winkydink]

NILFS is too close to MILFs

Re:Needs a new name

[Speare]

Score:5, Insightful? We need a new meta-moderation scheme, where we can moderate the moderations as Funny.

Re:Needs a new name

[chochos]

NILF: Netserver I'd Like to fsck (but I don't need to anymore, apparently)

Re:Needs a new name

[Scaba]

Too close? Not close enough!

Re:Needs a new name

[EvilTwinSkippy]

You know, F'ed is the last state I want to see my file system in.

Stable?

[theJML]

I like how they say it's reached a stable release but if you look at the known bugs on the Project Home Page http://www.nilfs.org/ You'll see that:

The system might hang under heavy load.

The system hangs on a disk full condition.
Aren't those kind of important to saying that something is stable?

Re:Stable?

[Ulrich Hobelmann]

Oh, hanging when the disk is full wouldn't be that bad. The real problem is that once your disk (i.e. the log) is full, you'll never have a non-full hard disk again, because NILFS doesn't yet have a working garbage collector ;)

News that matters.

NILFs really turn me on!

[McGregorMortis]

Or am I thinking of something else? I'm not sure now. I better go check. No interruptions for 10 minutes, please.

Here's an overview for lazy people like me

[Work Account]

NILFS is a log-structured file system developed for the Linux kernel 2.6. NILFS is an abbreviation of the New Implementation of a Log-structured File System. A log-structured file system has the characteristic that all file system data including metadata is written in a log-like format. Data is never overwritten, only appended in this file system. This greatly improves performance because there is little overhead regarding disk seeks. NILFS also has the following specific features:

        * Slick snapshots.
        * B-tree based file and inode management.
        * Immediate recovery after system crash.
        * 64-bit data structures; support many files, large files and disks.
        * Loadable kernel module; no recompilation of the kernel is required.

Re:Here's an overview for lazy people like me

[Anonymous Coward]

Here's an overview for lazy people like me

Hold on a sec -- if lazy people like you provide overviews, does that mean they (and you) are no longer lazy? And do all of us have to provide overviews now?

Good God, man, think of the ramifications of what you say!!!! ;)

NTFS

[JustASlashDotGuy]

" When the system reboots, the journal notes that the write did not complete, and any partial data writes are lost. "

Isn't this similar to NTFS's journaling file system?

Bundling

[superpulpsicle]

If they are serious about a filesystem, it has to be bundled with the linux distros every release. Take Reiser and JFS for example, some distros have it, some don't. Not every release of the same distro has it, what a mess. Only two have stayed permanently EXT2, EXT3. Everything else is trendy.

Re:Bundling

[m50d]

I've never used anything but reiserfs. If a distro won't support it, I won't use that distro, simple as that. It's a really nice filesystem.

Re:Bundling

[kobaz]

If they are serious about a filesystem, it has to be bundled with the linux distros every release. Take Reiser and JFS for example, some distros have it, some don't. Not every release of the same distro has it, what a mess. Only two have stayed permanently EXT2, EXT3. Everything else is trendy.

Reiser and JFS have been in the mainline kernel since umm, I think early 2.4. They were put in around the same era that ext3 showed up in the mainline kernel.

I don't know about you but I never use the included kernel of a distro (except in the install phase) because 90% of the time it doesn't have all the drivers compiled in that I need for the system I'm using.

Re:Bundling

[mrsbrisby]

I've never used anything but reiserfs. If a distro won't support it, I won't use that distro, simple as that. It's a really nice filesystem.

Funny. I've used xfs, ext[23], reiserfs, jfs, and linlogfs, and reiserfs is the only thing that actually ate my data.

Re:Bundling

[sp0rk173]

Funny, i can say anything i want and you can't prove me wrong because i gave you no verifyable evidence to back up my claims. In other words: shut your pie-hole.

Re:Bundling

[mrsbrisby]

http://list.linux-vserver.org/archive/vserver/msg0 2546.html
http://groups.google.com/group/mlist.linux.kernel/ browse_frm/thread/50a838e9734da325/36f7c8badcb0675 2?lnk=st&q=unexplained+reiserfs+corruption&rnum=1& hl=en#36f7c8badcb06752
http://groups.google.com/group/alt.os.linux.slackw are/browse_frm/thread/9193d9ab458cbc91/61101e11a18 d28b5?lnk=st&q=unexplained+reiserfs+corruption&rnu m=2&hl=en#61101e11a18d28b5

Lots of people have weird problems with reiserfs that they never have anywhere else. They also never seem to get their questions answered.

Shutdown versus power off

[Matt Perry]

More file integrity is always good. Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command. It never made sense to me that we'd have to "shut down" as opposed to just turning the thing off.

Re:Shutdown versus power off

[Kiashien]

...that's mean to the hard drive. That can cause serious damage- and not because data hasn't been written yet...

When you cut the power to a HD, the head stops wherever it is- sometimes even settling on the drive itself. This causes severe wear and tear, and sometimes damage. It's one of the many reasons why people are told always leaving your computer on is less damaging than turning it off.

The difference between cutting power and a proper shut down, is that during a proper shut down, the OS ensures A) everything is written (which you may not care about) and B) that the drive heads are in a locked, safe(ish) position.

You may not care about A, but you really should care about B.

Re:Shutdown versus power off

[pesc]

Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command.

That's a very bad idea. Normally, journaling file systems only guarantee that the file/directory structure remains intact. It does not necessarily guarantee that the data in the files hit the disk. Also, your disk will probably have a cache that is lost when you remove power. Whatever is in the cache will also be lost.

So your file system may be intact, but your practices will probably destroy data.

Re:Shutdown versus power off

[dlamming]

Don't be ridiculous. Every drive in the last 5 years, maybe the last 10, is able to park the head safely even in the event of your pulling the plug on the drive itself. It's got some springy/inductory dealie that pulls the head to a safe landing area.

That doesn't mean that you won't lose data that hasn't been written yet, of course. :)

Re:Shutdown versus power off

[TheRaven64]

Wow, just wow. Seriously, I really, really hope you don't have any important data.

Here's a little (simplified) tutorial on what happens when you a program writes a file to disk:

1. It goes into the OS filesystem cache. This is often several hundred MBs. At some point, the OS decides it is not important enough to be kept around. At this point,
2. it is written to the hard drive. Here, it sits in the hard drive controller's on-board cache. When the cache is full,
3. it is written to disk.
   • At any given time, you may have several hundred MBs of data that show up when you browse your filesystem, but have never made it to disk. When you hit the power button, all of this data evaporates as the current fades from your RAM chips (over about 10ns). The only thing journalling filesystems do to improve this is ensure that, when the data is being written to disk, if the system crashes in the middle of the write then the system can be restored to either the beginning or end of the transaction, rather than the middle. A journalled filesystem will (in theory) never be corrupted by the power failing, but that does not mean you will never lose data - you will, and potentially large amounts of it. In short, if you continue with this practice, then you are a numpty.

Re:Shutdown versus power off

[Rendus]

Hah. The parent post is brought to you by the letters I, B and M, and the year 1994.

The only major issue with abruptly removing power to the HD or PC is if there were writes in cache waiting to be written to disk.

Re:Shutdown versus power off

[Kiashien]

Eh, who said he had a new hard drive? I still use a 40 gig from 1998.

Haven't had a reason to throw it out. It's much simpler to simply be careful, and be aware of what could happen, as there's no way to simply "know" that the hard drive can prevent it.

Besides, that particular technology wasn't in every drive (and still may not be). It was only included massively in laptop drives for many years, as they deal with the issue of power loss much more often. It is highly possible it is included in all drives now, but I'd be very surprised if it was in "all" hard drives in the last 5 years. I could believe the last 3. Anything that equals more manufacture costs rarely is made unless specifically needed for some reason- like a high-end drive, or a laptop in this case.

Re:Shutdown versus power off

[morpheus800e]

Automatic head parking has been around for a LONG time - Here's the data sheet from the 120 MEGAbyte drive that I had years ago. It came in my 386. Note the following line, about half-way down:
"Turning the system power off causes the WD Caviar to perform an automatic head park operation."

It wasn't a high-end drive at the time, (just a consumer-level IDE drive), and was utterly obsolete years ago, yet it still had the technology to park the heads out of the way when power is disconnected. There's no way that turning off power to a new drive is going to physically damage it. You just might lose the data on it.

Re:Shutdown versus power off

[Pecisk]

...and it is actually very bad to don't have a proper ACPI support, because, if your model is supported by your operational system's ACPI, then pressing power button will shutdown your operational system, including Linux, by itself. No loss of data, correct shutdown, no waiting for that to happen.

But if it is _not_ supported - well, it could be very bad for your laptop, for example. I know lot of laptops without support for them in Linux will heat themselves too much and it can cause big trouble. With proper ACPI support, they work very good.

And, in fact, better don't switch your computers off at all. Because it will cost more you to change motherboard - if some day switching on/off will overcharge it too much - than paying your electricity bills (computers, at least modern ones, in sleep mode consumes almost nothing).

And on topic - yeah, journaling file systems gives a false sense of security. They just a little bit faster and safer. But they are not solution.

Re:Shutdown versus power off

[Marillion]

Hard drives with voicecoil head positioning (some 99.99% of them sold in the past 10 years) will autopark on poweroff. The real risk is how much latency the os has between when an application saved it's data and when the buffers are flushed.

Some applications keep files open for a long time: MySQL, gDBM-based apps, Squid. Most of those application implement their own mini-filesystems within a file optimised for task. These systems are supposed to preserve their integrity by journaling their modifications in case the underlying os doesn't.

Switching off a computer because it has a journaling filesystem is like stopping a car by driving into a something because it has seat belts.

Re:Shutdown versus power off

[bcmm]

Even assuming that the file system returns in the same state you left it in, you've killed all running processes. Data still in RAM, etc. Unsaved changes. For example, if you're running mldonkey, it sometimes leaves it's files in stupid inconsistent states if it is kill -9'ed.

Re:Shutdown versus power off

[heson]

You might want to read this http://linuxmafia.com/faq/Filesystems/reiserfs.htm l

Re:Shutdown versus power off

[mrsbrisby]

Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command.

That's a very bad idea. Normally, journaling file systems only guarantee that the file/directory structure remains intact. It does not necessarily guarantee that the data in the files hit the disk.

Sure it does: mount -o sync,data=journal

Also, your disk will probably have a cache that is lost when you remove power. Whatever is in the cache will also be lost.

Hogwash. Disks that have such a cache have no command that tells them to actually flush their cache. They simply flush the cache whenever convenient or when power supply is lost to the disk.

So your file system may be intact, but your practices will probably destroy data.

That makes no sense. Either my files system is intact, or it isn't. I reject the notion that my file system can be intact but that data is destroyed.

What's more likely is that you use a program that does a open(...,O_RDWR|O_TRUNC); and lose power at that point. Well no amount of journalling or shutdown will help that one. So always creat() a new file and rename() it over the original.

When demonstrating solutions to clients I always give them a power-plug yank-test to show them it's perfectly possible and reasonable to expect a system that isn't a fragile as you suggest.

By the way, log-structured filesystems provide exactly the semantics that I describe (creat() followed by rename()) except it does it in such a way that your open(...,O_TRUNC) can be reversed due to automatic versioning.

Re:Shutdown versus power off

[0xABADC0DA]

Close, but no cigar:

1. It goes into the OS filesystem cache. After 5 seconds the modified data gets flushed to the disk (sometimes set to 30 sec).
2. It is written to the hard drive. Here, it sits in the hard drive controller's on-board cache until the head arrives at the write point, which is a fraction of a second.
3. It is written to disk.

So it *can* happen that data is not written properly, but unlike the scary picture you paint it is extremely unlikely. Even if you just saved your data, just do a sync and you'll be fine turning the power off.

Re:Shutdown versus power off

[oddfox]

So do any other filesystems work like this because it seems like your post is specifically speaking of Ext3. I'm just curious because I've yet to be burned by a sudden loss of power to the system as opposed to a shutdown operation, but I'm still not going to make a habit out of it. I mean really, it doesn't take much time to do a restart, and I don't see why you would need a system shut down immediately. In that much of a hurry to make it to the LAN party or something?

Re:Shutdown versus power off

[sasami]

Close, but still no cigar.

1. It goes into the OS filesystem cache. After 5 seconds the modified data gets flushed to the disk (sometimes set to 30 sec).

That's if the OS is completely unloaded. Sure, maybe after 5 seconds it becomes eligible to flush, and will wait in line right behind anyone else whose turn came up. Is the wait likely to be very long? No. Can it be very long? Bet your data.

2. It is written to the hard drive. Here, it sits in the hard drive controller's on-board cache until the head arrives at the write point, which is a fraction of a second.

That's if the disk is completely unloaded. Modern disk firmware is extremely intelligent about ordering, but they optimize for performance and do not care very much about getting your data out of cache right away. Using NCQ we have observed write latencies on the order of ten seconds. It could probably get even higher, but we'd force a flush rather than letting it get away with that.

3. It is written to disk.

Yup.

--
Dum de dum.

Re:Shutdown versus power off

[sasami]

Hogwash. Disks that have such a cache have no command that tells them to actually flush their cache. They simply flush the cache whenever convenient or when power supply is lost to the disk.

Hogwash. The [S]ATA spec allows the disk cache to be enabled or disabled by the user at any time. Disabling the cache cannot return "success" until there are no writes outstanding to the disk. A flush is a disable/enable sequence.

And no disk is able to flush a full cache of writes in the minuscule window available to it after a power failure. We're talking a few dozen milliseconds at best. It is not part of the spec, nor would any manufacturer be stupid enough to offer such a feature. Do you have any idea how long it would take to flush out even a megabyte of small random writes? You'd have to slap a battery on the disk, and a 12V battery at that. Or use NVRAM for the cache, which is what good RAID controllers do.

--
Dum de dum.

Re:Shutdown versus power off

[Matt Perry]

Why don't you just use the power switch to get out of every program?

Because I might not be done using the computer. When I'm done using the computer, I turn it off.

Re:Shutdown versus power off

[Matt Perry]

That's a very bad idea.

If you mean having to shut down as opposed to power off, then I agree, it's a very bad idea. It seems that the trend in the last 10 years has been to move to having to shut down the machine. We need to make an effort to improve operating systems to the point where they can be turned off when you are done with them, just like any other appliance.

Re:Shutdown versus power off

[Matt Perry]

if your model is supported by your operational system's ACPI, then pressing power button will shutdown your operational system, including Linux, by itself. No loss of data

That sounds great until a program hangs and the machine stay on in the shutdown procedure because of some problem. As far as I'm concerned power switches for computers should break the curcuit for power. Instant power off. No write caching. The machine should be ready to lose power at any moment. None of this ATX power supply nonsense that sends signals to the motherboard.

correct shutdown, no waiting for that to happen.

If the power is still on when I remove my finger from the power button then I'm waiting. My powerstrip method assures no waiting.

Re:Shutdown versus power off

[DarthStrydre]

Depending on the fragmentation of the data in the cache, it is not necessarily impossible to write it out. I am not sure whether it is done any longer, but at one point the rotational energy stored in the spinning platters was used to park the heads and perform final housekeeping, flywheel-energy-storage style. I do not believe this was used for any disk writes, as the platters would become a non-standard target for writes as they slow down, and angular timing would be an issue.

However, it would be conceivable and a drive-firmware only fix/hack for manufacturers to designate a track or two specifically for cache dump area. Given that controller -> platter speed is > 20MB, it would be possible upon power loss, to seek to track -1, dump contents of write cache, and park in under half a second (assuming worst case 16MB buffer, half used as write cache), long before the platters are rotating too slow for the heads to float in the ground effects. Since there would be nothing valuable on the track other than the equivalent of a cache core dump, there would be no risk of corrupting other data due to timing issues, and no compensation would have to be done upon write.

Upon re-application of power, the drive would read this track (more difficult than normal, due to the predictable but irregular angular spacing of the bits) and complete the drive internal writes before becoming 'ready'.

Then again the assumption that flywheel-storage energy is sufficient for operation for an unknown time, especially when the device was not intended to be a flywheel in the first place, can produce problems, Chornobyl case in point. (Yeah, disable the backup generators, disable the automatic SCRAM, disable everything else, cut steam to the turbines and see how long they can power everything using stored rotational energy before things go boom. Oops! I guess not long enough!)

Re:Shutdown versus power off

[Minwee]

"As far as I'm concerned power switches for computers should break the curcuit for power. Instant power off. No write caching."

I see that you do not live in a house with small children or other pets with opposable thumbs.

In my world power switches should be behind locked cabinet doors with no visual clues to show where they are. When they are eventually pressed, which is always at the least convenient time, the systems they control should very gracefully power down and then restart in exactly the same state when the switch is pressed again, usually by an adult.

Your needs may vary.

Re:Shutdown versus power off

[Matt Perry]

Even assuming that the file system returns in the same state you left it in, you've killed all running processes.

That's the point. I'm done using my computer. I want to turn it off.

Data still in RAM, etc. Unsaved changes.

I save my changes and close programs before I turn off my machine. Sometimes I don't bother to close the programs. It depends. I turn off caching in the OS and hard drives. I've never lost data (two years and counting).

Re:Shutdown versus power off

[Matt Perry]

You really should work on improving your self-esteem.

Re:Shutdown versus power off

[bcmm]

That's the point. I'm done using my computer. I want to turn it off.

I said kill, not term. I suppose it's possible that you have no daemons running which dislike being killed though.

Also, I once had a reiserfs partition become unbootable after an unrelated kernel panic forced a hard reboot.

Re:Shutdown versus power off

[mrsbrisby]

The [S]ATA spec allows the disk cache to be enabled or disabled by the user at any time. Disabling the cache cannot return "success" until there are no writes outstanding to the disk.

There exist IDE disks that ignore this setting.

... nor would any manufacturer be stupid enough to offer such a feature.

Compaq Smart Array 5300 family uses a battery to accomplish this very thing. They advertise it too.

Do you have any idea how long it would take to flush out even a megabyte of small random writes? You'd have to slap a battery on the disk, and a 12V battery at that. Or use NVRAM for the cache, which is what good RAID controllers do.

Why would you need a battery exactly? Many drives have such a small on-disk cache that a few capacitors have enough power to flush the cache.

Re:Shutdown versus power off

[mrsbrisby]

Do do any other filesystems work like this because it seems like your post is specifically speaking of Ext3

All UNIX filesystems operate in the manner that I described. It's a requirement, and is often referred to as "UNIX semantics".

The -o data=journal part IS ext3 specific. Both ext3 and jfs are theoretically safe (on Linux. BSD's FFS should also be safe due to ordering). Reiserfs is NOT safe. XFS probably is, but the manuals should be checked.

I don't see why you would need a system shut down immediately.

I don't see why you wouldn't want a system that could shut down immediately. I mean, a TiVO should just eat all my shows if lightning strikes my house, right?

I find that weird stuff happens once you give a system to a customer that runs on a computer, and looks like a computer, but isn't a computer. These people don't know how to shut things down, and they become extremely irritated when you tell them that they have to do some kind of weird magic.

Especially after they've seen my demonstration and know that they don't :)

Re:Shutdown versus power off

[00110011]

Are you using special mount options or kernel settings or something?

What you are describing sounds almost like what the "laptop-mode" program in Ubuntu does. When "laptop-mode" is started (I think it runs when a laptop is switched from AC to battery power), disk writes are deferred for up to the configured time period (defaults to 10 minutes, I think) so that the hard drive in a laptop can spin down to conserve battery power.

Re:Shutdown versus power off

[sasami]

Compaq Smart Array 5300 family uses a battery to accomplish this very thing. They advertise it too.

An AC already responded to this, but just for kicks I decided to look up the Smart Array 5300, knowing exactly what I would find.

First, these are controllers, not disks. Second, they do indeed have a battery, as they well should -- it's used for their nonvolatile cache, which keeps your data nice and safe in RAM while the power goes out. It is absolutely not used to flush the cache by running the disks on battery. This is a PCI card, fer crying out loud. There isn't even a physical power cable between it and the disks you seem to think it's powering!

Perhaps you didn't notice that I had already written:

Or use NVRAM for the cache, which is what good RAID controllers do.

I also don't understand why you say that [m]any drives have such a small on-disk cache that a few capacitors have enough power to flush the cache when I am specifically talking about those drives which do have a large cache. And which could take up to, oh, maybe a minute to completely flush out 8 megabytes of small, random writes.

But even in your case, please cite exactly how small a cache would have to be, and how large those capacitors would have to be, to guarantee a complete flush at power failure. I'm asking genuinely, since I don't know enough about such ancient beasts, except possibly that drives run around 8-12 watts when actively seeking around the platter.

--
Dum de dum.

Re:Shutdown versus power off

[mrsbrisby]

Simple - a journalling filesystem is designed to protect changes to the filesystem itself - not to it's contents. So in the case of a bad shutdown, it ensures that all the information about files and directories and which disk blocks belong to which files is consistent.

Absolutely not! Some journalling filesystems are designed to protect changes to metadata, but not all. Many consider protecting data to be prohibitively expensive, but that's not the same thing.

However, it makes no guarantees about the data contained within those disk block - if you've appended data to a file, the extra blocks will have been added to the file, but the actual data may not have been written to them.

I never append to files that I cannot live without. Rarely is it that I do it at all; my most common abuse of this is debugging logs that are easier to watch live. I also usually put them in /tmp.

Re:Shutdown versus power off

[mrsbrisby]

First, these are controllers, not disks. Second, they do indeed have a battery, as they well should -- it's used for their nonvolatile cache, which keeps your data nice and safe in RAM while the power goes out. It is absolutely not used to flush the cache by running the disks on battery. This is a PCI card, fer crying out loud. There isn't even a physical power cable between it and the disks you seem to think it's powering!

From:
http://h18004.www1.hp.com/products/servers/prolian tstorage/arraycontrollers/smartarray5300/

Battery-backed cache protects cached data in the event of a power outage, server failure or controller failure, and redundant batteries take that protection even further.

And by the way, when most of the controller hardware is physically on some disks and on an expansion card on others, blurring the distinction seems acceptable to me when we're talking about durable systems.

Re:Shutdown versus power off

[k8to]

You're right in what you say, but the scary scenario to me is not "oh my some data did not get out to the disk". In typical personal user cases, the last thing I saved was a long time ago, and I don't really care that the last thing the computer was up to gets saved. Obviously you could have higher level logic issues if stuff doesn't hit the disk, but that's rarely a problem in Linux.

The personal use case scenario disaster situation is when a program begins rewriting the content of a file, such as appending data to it, but then the system crashes or is powered off. The data inside the file is now corrupt, and potentially unusable, depending on how it is loaded/parsed etc. So a system which is arbitrarily powered off does not just stand to lose the data which has not yet been written, but the content of files undergoing modification is likely to become unusable, and thus lost. The best part of this is of course you may not realize it is gone for weeks or months, until you need it again, at which point the data on your backups is probably corrupt too.

actual info about the fs

[cowens]

http://www.nilfs.org/

Good news

[kc01]

I'm delighted to hear this. I have found ext3 to be a very fragile filesystem- Unexepected reboots or powerfails can have a devastating effect.

I've experienced roached ext3 filesystems a number of times- With no hardware failure. Recovering that data by converting it to ext2 helped once, but I'm still trying to recover the data after the last incident. I'm sure (rather, hope!) that these experiences aren't the norm.

Re:Good news

[djdanlib]

Are you a consultant or "support technician", or are these your hard drives? If they are yours, you might want to consider a few things.

1) Uninterruptible Power Supply. Is keeping your data if the power gets cut mid-write worth $80 or so?

2) Different file systems with more journaling support: I use ReiserFS. You may like JFS or something else exotic. Sure, it's a pain to convert a drive - you have to have another one with a lot of space to hold your files - but it's worth it if you're losing data due to a weakness in one file system.

3) Use only stable software in a production environment. By "production environment", I mean any environment where you produce useful data that you need to keep. The core of your primary OS - including the kernel and toolchain - should be stable release versions. If you're a developer for said core, then you should already know this and have a dedicated sandbox installation for development purposes and not be using that for other work.

If you want to be able to get into and repair your drive with a hex editor (weird), ext2 is one of the very few options for you. But if you take a few precautions, you shouldn't ever have to do that.

Excellent information retention

[totallygeek]

I installed this lossless file system. rm is now chmod 444. I have not been able to lose information since.

Note: instead of modding this +1 funny, mod it +0.1 pathetic.

Re:Excellent information retention

[Cryptnotic]

Now, modify your kernel and disable the unlink(1) system call.

Re:Excellent information retention

[alexhs]

I suggest you try chattr +a+i+u to strengthen further your filesystem :)

Data is the new currency my friend

[Work Account]

Aside from gasoline and water, data is the most valuable thing in the world.

Walmart's most prized possesion is their billion-billion-billion transaction customer sales database. They use it to find things like, among other things, men tend to buy beer and diapers at the time.

With disks costing $1.00/GB or less these days, many people including myself simply DON'T delete data anymore. I keep all my original digital photos (in .tiff format) along with full-quality movies and all the games I've ever played back to Duke Nukem on 80x386 on a RAID array that's grown to nearly 2 terabytes.

So yes, for many people, disk space is just something you keep adding to, like you'd move from a coupe to a sedan when you have kids and when you have that 6th kid you move to a minivan and if you happen to have 2 more, you get a cargo van when #8 comes along :) Same for HDDs

Re:Data is the new currency my friend

[shokk]

Even for a 320GB Western Digital RAID Edition drive this comes out to $0.62/GB. We recently purchased a bunch for a new RAID server we're putting together on the cheap (1.8TB for $4500) - RAID 5 plus hot spare. Later on we can expand to 3.6TB for another $2k, assuming drive prices don't drop, which they will by then. When you figure in the cost of the hardware that reliably supports that, the price climbs to $1.80/GB. That doesn't cover the cost of admin salary (or your own time to support it if this is a home system). That's still not even 2x your example and it's still very cheap.

Re:Data is the new currency my friend

[zippthorne]

nitpik:

" on 80x386 on a RAID array"

I move that we actually change the "prefered" usage of the term RAID to RAID array or similar to enhance the redundancy involved. This is the one circumstance where repeating terms in the acronym is actually appropriate.

--

Now, how do i indicate that this was not intended as sarcasm without the statement itself appearing sarcastic...

Re:Data is the new currency my friend

[magarity]

men tend to buy beer and diapers at the time
 
Just FYI: this is a business school urban legend. The one about pop-tarts and approaching hurricanes is not.

Re:Data is the new currency my friend

[hanshotfirst]

men tend to buy beer and diapers at the time

Just FYI: this is a business school urban legend. The one about pop-tarts and approaching hurricanes is not.

Maybe so, but almost every Walmart supercenter I've been in has the beer and diapers close to each other, off the main isle.

Re:Data is the new currency my friend

[Minwee]

"So yes, for many people, disk space is just something you keep adding to, like you'd move from a coupe to a sedan when you have kids and when you have that 6th kid you move to a minivan and if you happen to have 2 more, you get a cargo van when #8 comes along."

If I kept having kids like that I'd be more inclined to reevaluate my method of birth control than my choice of car, but that's just me.

Re:Data is the new currency my friend

[Ahnteis]

How about "Redundant Raid Array of Disks"?

Re:Data is the new currency my friend

[mc_barron]

Here's a link on the legend: http://web.onetel.net.uk/~hibou/Beer%20and%20Nappi es.html/

Re:Data is the new currency my friend

[Old Wolf]

I keep all my original digital photos (in .tiff format) along with full-quality movies and all the games I've ever played back to Duke Nukem on 80x386 on a RAID array that's grown to nearly 2 terabytes.

So, basically, you're going to keep Duke Nukem forever?

Re:Data is the new currency my friend

[pod]

I prefer Redundant RAID Array myself.

Re:Data is the new currency my friend

[lav-chan]

Walmart's most prized possesion is their billion-billion-billion transaction customer sales database. They use it to find things like, among other things, men tend to buy beer and diapers at the time.

I don't know a single thing about Wal-Mart's 'billion-billion-billion transaction customer sales database', so i'm curious about this.

How exactly does Wal-Mart know stuff like that? I don't know about men buying diapers and beer, but i, for one, have been to Wal-Mart many times in many different cities, and i don't recall any way that they could possibly have ever known my gender, unless their employees were watching me and marking it down secretly.

Re:Data is the new currency my friend

[vacuum_tuber]

off the main isle.

Wouldn't they be better putting it on the mainland?

Actually they put it off the mein isle since their database revealed that men often like Chinese takeout with their beer and diapers and the idea of an exotic tropical vacation enhances sales.

OK, shoot me now.

Re:Data is the new currency my friend

[eam]

Data is much, much, MUCH cheaper to maintain than children. If I could add kids for nothing more than a one time cost of $1/kid, I probably wouldn't spend any money on birth control.

First kid: bought a pickup truck so that I could drive him around when the wife has the car.

Second kid: traded the pickup for a explorer sport trac so I could fit more carseats.

Third kid: couldn't wedge more carseats in the wife's saturn, so moved to a minivan.

At this point, we could probably manage another vehicle change to expand capacity to handle 1 or even 2 more kids. However, the ongoing maintenance is going to kill us so we have to stop here. Which leads me to wonder about people with terabytes of RAID at home. Adding data is not a one time expense. At least, if you care about the data, it shouldn't be. Do you do any backup, or are you just willing to accept the loss of all that data? If you do backups, how do you do it?

Of course, another difference between kids and data is we can't delete kids to make room for more. Well, not for another 14 years. By then we'll be too old to be relied on for primary storage of new data...er, kids. We'll be relegated to short term scratch space after that (grandkids on the weekends).

Re:Data is the new currency my friend

[Minwee]

Another important difference is that if one of your kids becomes corrupted you can't restore from a backup. Disaster recovery tends to be a much messier, hands on process.

I suggest N = "Nobody"

[StressGuy]

Otherwise, an ACLU lawyer might start to salivate...

Privacy

[lilmouse]

Has anyone considered the privacy implications of this yet?

Not sure I like logs listing that 3 years ago, I had a file named bad_kiddie_pr0n.jpeg (or whatever) on my computer.

They'd better have a good cleanup script!

--LWM

I don't know about anyone else....

[Doctor Memory]

...but I want a business card that says I work at "Cyber Space Laboratory"!

There's no replacement for ext3fs yet for me...

[davegaramond]

I'd looove to replace ext2/3 as my filesystem for years since it's not so fast and most distro don't include binary tree indexing for ext3 (so large dir is slow). Unfortunately I haven't been able to do so. Here are my requirements:

1. Distro support. I don't want to have to compile my own kernel. The FS needs to be supported by the distro (Debian in this case). I want to be able to create root partition and RAID with the FS.

2. ACL and extended attributes.

3. extended inode attributes would be nice ("chattr +i" is handy sometimes).

4. optionally I would like to be able to create large Bestcrypt partitions (e.g. 30GB) with that FS.

5. fast large dir and small files performance (I have millions of small files on my desktop).

6. no need to fsck or fast fsck (i.e. journalling or some other technique or whatever).

7. disk quota!

8. optionally, transparent compression and encryption will be a big plus point.

9. Snapshots would be nice too, for consistent backups.

10. Versioning is also very welcome.

XFS: very close but it still has problems with #4. It also doesn't have undelete like ext2/ext3 (not that it's a requirement though).

JFS: it just lacks many features.

Reiser3: How's the quota support, still have to patch kernel everytime? Plus it doesn't have ACL.

Reiser4: not ready yet.

I might have to look at FreeBSD after all. Background fsck, hmm....

Re:There's no replacement for ext3fs yet for me...

[metamatic]

Reiser3 works fine on Debian with no kernel patching required.

It seems as if you're holding out for perfection, not willing to upgrade from ext3 to anything else unless you find The Perfect Filesystem. I think that's kinda silly; better to get 90% of what you need now, than to wait another 2-4 years, surely?

Re:There's no replacement for ext3fs yet for me...

[m50d]

Reiser3: How's the quota support, still have to patch kernel everytime? Plus it doesn't have ACL.

It does have ACL, and quota support is fine at least in gentoo kernels (can't check a vanilla one atm)

Re:There's no replacement for ext3fs yet for me...

[fireboy1919]

8. optionally, transparent compression and encryption will be a big plus point.

9. Snapshots would be nice too, for consistent backups.

10. Versioning is also very welcome.

I sure hope that none of these things are ever part of the filesystem itself. I want my filesystems 100% portable, and fast. You know why NTFS isn't so much, right? All the extra, nearly useless features that should be handled by the OS, but that are done by the file system instead.

These should be layers on top of the file system that are implemented by something else. 9 is basically a cron job, and 10 can be done more ways than I have fingers (and I have a full decimal complement) without using the file system, and there are a few solutions on freshmeat that will do 8 today.

Hopefully there aren't any file system designers who actually think that putting these into their system is actually a good idea.

As far as no "fsck"...I would never use a filesystem without it. Disks and memory go bad. Anyone who tells you that their filesystem never needs checking is deluding themselves. If anything, I want a filesystem with even more error recovery capability. I'd like the file metadata to have lots of redundancy for the sake of reliability, and some forensic tools that are at least as good as the ones already available for ext2.

Re:There's no replacement for ext3fs yet for me...

[TheRaven64]

I might have to look at FreeBSD after all. Background fsck, hmm....

It might be worth it:

1. Not really an issue, since there aren't really FreeBSD distros. UFS2 is supported by NetBSD, however.
2. Yup. Both available.
3. Yup. Been around since 4.4BSD (1993). It's called chflags though, not chattr.
4. No idea.
5. It's not bad, but I don't have any real performance figures so try it and see.
6. Softupdates should ensure consistency after a fsck, and the fsck can run in the background. There was a Summer of Code project to add journaling to UFS2/3, but the wiki page was never updated so I have no idea how much progress was made.
7. Yup, since 4.2BSD.
8. Yup, since FreeBSD 5.0.
9. Sadly not. I've not seen a filesystem that did versioning outside VMS, but it would be a nice addition.

Re:There's no replacement for ext3fs yet for me...

[TheRaven64]

Ooops. Missed one. Your 8 should have been:

You can transparently encrypt an entire volume via GEOM, but if you want per-file encryption or compression, then you're out of luck. The only FS I've seen do this nicely was NTFS.

My 8 and 9 should then be your 9 and 10.

Re:There's no replacement for ext3fs yet for me...

[swillden]

9 is basically a cron job

Ummm, no.

9 (snapshots), is a very important feature that makes it possible to create a cron job to do nice, consistent backups easily. Unless you don't mind writing a cron job that remounts the fs as read-only before doing the backup... an approach that's likely to cause one or two small problems.

That said, snapshotting doesn't need to be implemented in the file system. LVM, for example, implements it below the file system, at the level of a block device. That approach has the benefit of making it applicable to any file system.

Re:There's no replacement for ext3fs yet for me...

[lintux]

I'm not sure, but does ext3 actually have all the features you mention?

Anyway, some other things... AFAIK Ext3 undeletion doesn't work anymore (at least not using debuge2fs, lsdel always gives me an empty list these days). Also, snapshots are possible with any filesystem in Linux, as long as the fs is on LVM.

I'm sticking with Ext3 too. Heard too many scary stories about XFS, JFS and even Reiser, so I'll stick with what's known to work.

Re:There's no replacement for ext3fs yet for me...

[davegaramond]

I'm not sure, but does ext3 actually have all the features you mention?

Sorry, should've stated more clearly that some of the points are optional. The most important features needed is quota, ACL, fast recovery, large dir and small files performance (think Maildir and squid), and being able to work nicely with Bestcrypt. Oh and yeah, it has to be stable. Reiser3 burns me once several years ago, have been reluctant to try it again. It's really not fun moving many terabytes of files only to find later that some of them rot or got corrupted because of filesystem bugs.

Re:There's no replacement for ext3fs yet for me...

[mystik]

...And also important (and why i stick w/ ext2/3): Tools for the inevitable do-it-yourself "oh sh_t" recoveries.

Re:There's no replacement for ext3fs yet for me...

[Fnord]

It also has a tendency to corrupt your data under heavy use (talking personal experience here).

Re:There's no replacement for ext3fs yet for me...

[jabuzz]

Perhaps, but snapshots work best when you can do filesystem freezes, such as XFS supports.

Re:There's no replacement for ext3fs yet for me...

[swillden]

Can you explain how freezing improves snapshots? I've never found a good explanation of why it matters, at least with journaling file systems.

Re:There's no replacement for ext3fs yet for me...

[farnz]

My personal experience is that ext3 has a tendency to lose data silently, then "suddenly" supply a huge stack of lost+found entries on fsck. Reiser3 doesn't do that, nor have I ever lost data to Reiser3 without an intervening hardware failure.

YMMV, but I switched from ext3 to Reiser3 around 3 years ago, and I've only lost data since to a disk failure (backups were my friend). Ext3 would lose documents for me from time to time, then spit them back out into lost+found when I fsck'd.

Re:There's no replacement for ext3fs yet for me...

[Fnord]

At the place I work for which shall remain nameles, we set up several Reiser3 file servers to store a few terabytes of data. We chose reiser3 because the data is in lots of small files and Reiser4 wasn't stable yet. Within a year all six of the storage servers (all mirrors of each other) had shown severe corruption, 3 of them catastrophic (filesystem wasn't useable at all). One of the guys on the team looked at the corrupted data and found kernel stacks dumped into filesystem structure blocks. The only thing we could do was to reformat the downed machines and copy data from an intact one. This kept us alive until we got funding for a real storage system (NetApp Filers).

I won't trust ReiserFS again.

Re:There's no replacement for ext3fs yet for me...

[m50d]

Now, see, I've had the opposite experience. ReiserFS has never lost anything for me, and I managed to get all the data off it after a pretty nasty partition messup, while I've had occasional problems with other filesystems I've tried. I think personal experience can go either way on things like this, just goes to show anecdotal evidence is no evidence at all. I would be interested to see some reliability benchmarks, though I'm not sure how one would produce such a thing.

Re:There's no replacement for ext3fs yet for me...

[peter]

As you say, when you can replay journals when you mount the snapshot device, but e.g. for ext3, data=ordered (the default) doesn't guarantee which version of the file's data you'll have after a journal replay, just that the metadata will be consistent.

  So it's a Good Thing to have your filesystem in a consistent state when you make a snapshot. You avoid all the bad ju-ju of dirty filesystems...

Re:There's no replacement for ext3fs yet for me...

[josh82]

"It seems as if you're holding out for perfection, not willing to upgrade from ext3 to anything else unless you find The Perfect Filesystem. I think that's kinda silly; better to get 90% of what you need now, than to wait another 2-4 years, surely?"...

Especially considering that in about 1-2 years, he'll once again complain that no filesystem has every single new feature that he wants, and use that fact to justify keeping his ext3 filesystem yet again. And so on, and so on, until the white suits from the nuthouse finally take him away.

Ext3 Forever!

Re:There's no replacement for ext3fs yet for me...

[swillden]

As you say, when you can replay journals when you mount the snapshot device, but e.g. for ext3, data=ordered (the default) doesn't guarantee which version of the file's data you'll have after a journal replay, just that the metadata will be consistent.

So what? Does it really matter if you get a snapshot of the data as it was a few seconds ago or as it is "now"?

I'm sure the guys at SGI aren't stupid, and I'm sure there is a reason for freezing, but danged if I can figure out what it is, and Google can't seem to find me any answers, either.

You avoid all the bad ju-ju of dirty filesystems...

"Bad juju", "problems"... that's the limit of the detail I've been able to find so far. Hmph.

Re:There's no replacement for ext3fs yet for me...

[hab136]

Imagine you open a file to back it up. Halfway through this backup, someone rewrites the file. With snapshots, you get the contents of the file at the time you started. Without, you get half of the original contents, and half of the new contents.

Re:There's no replacement for ext3fs yet for me...

[swillden]

Imagine you open a file to back it up. Halfway through this backup, someone rewrites the file. With snapshots, you get the contents of the file at the time you started. Without, you get half of the original contents, and half of the new contents.

Right. That's why snapshots are good. But that wasn't my question. What I was asking is why it's a good idea to use xfs_freeze before taking a snapshot. What "freezing" an XFS file system does is block all new update requests to that file system, and to flush all outstanding updates. Anyway, the claim is that it's somehow better to do:

xfs_freeze -f /home
lvcreate --snapshot --name home_snap /dev/vg/home
xfs_freeze -u /home
mount /dev/vg/home_snap /mnt
# Back up /mnt here
# ...
umount /dev/vg/home_snap
lvremove /dev/vg/home_snap

rather than doing the same thing without the freezing (-f) and thawing (-u). I suppose the freezing and thawing probably saves a small amount of time replaying the journal when you mount the snapshot, but other than that I can't figure out how the freeze/thaw makes a difference.

Re:There's no replacement for ext3fs yet for me...

[swillden]

lvcreate --snapshot --name home_snap /dev/vg/home

Oops, just noticed I forgot to specify the snapshot volume size. Typing too fast.

Pity not what I thought it was

[SmallFurryCreature]

RAID obviously protects against complete harddisk failure. This protects against dataloss during other hardware failure like powerloss BUT is there anything that protects against data loss due to slight defects on the hard disk?

I am probably not the only one to come back to an old file saved years ago only to find a glitch in it. I noticed it with a couple of movies. Movies I know were perfect as I watched them without copying them. So the only explanation is that part of the disk got corrupted.

The solution is availbale manually in the form of PAR or other recovery software but wouldn't it be nice if this was part of the OS or hardware itself? I would happily add another disk to my raid this one loaded with data to help check that the data is 100% correct.

Does such a solution exist?

Re:Pity not what I thought it was

[TheRaven64]

Sounds like you want ZFS - it stores per-block checksums, so you can detect single-block faults on a disk (not sure if it includes correction codes as well). As the other poster mentioned, if you are willing to add more hardware, RAID 5 does this.

Re:Pity not what I thought it was

[ChrisA90278]

Sun's "ZFS" applies error detection and correction to all data. ZFS is a file system that also includes a disk manager that provide RAID and volume concatination. It does it all in a single layer. I don't know if standard PC's are fast enough yet that checksumming all the data as it goes to and from the disks would work. But on Sun server hardware the extra checksume is triveal ZFS puts 64 bit checksums on all the data and the metadata ZFS is also Open Source so someday we might, maybe see a port to some other OS

Re:Pity not what I thought it was

[mrsbrisby]

I am probably not the only one to come back to an old file saved years ago only to find a glitch in it. I noticed it with a couple of movies. Movies I know were perfect as I watched them without copying them. So the only explanation is that part of the disk got corrupted.

That doesn't follow. Perhaps your filesystem isn't safe, or configured to be safe.

I've had a UFS system eat entire (old) directories due to the directory being updated and an untimely crash.

Re:Pity not what I thought it was

[MechaStreisand]

It was my understanding that the hard drive itself is supposed to have its own ecc codes on every block, and they're checked by the firmware and transparently marked as bad and the data moved to another block if an error is detected. So in most cases, an extra software layer of checksums or codes would not really be necessary, and that's probably why there's only that one by Sun that does that all those others mentioned. Your hard drives must really be in rough shape if data is actually getting corrupted on them...

That, or because you hadn't accessed the file in years, the drive never had a chance to detect the errors before it's too late. Hmm. But it was also my understanding that if a block goes bad unrecoverably, it's not allowed to be accessed at all, and the application that's trying to access it will usually hang. I've had that on an IBM Deathstar once - it's how I knew it was time to dump everything off of it and put it on the shelf as an antique.

Re:Pity not what I thought it was

[49152]

>I am probably not the only one to come back to an old file saved years ago only to find a
>glitch in it. I noticed it with a couple of movies. Movies I know were perfect as I watched
>them without copying them. So the only explanation is that part of the disk got corrupted.

Make sure you test them with an equally old version of the video codec and/or player. I have seen this happen with old divx and xvid coded movies, playing with old codec fixed the problem.

HDFS (home-dir FS)?

[Ramses0]

I've had an idea kicking around for a while now... "HDFS / Home-Dir File System" ... I want a (s)low-performance, bloated, version controlled, roll-back featured, viewcvs.cgi enabled file system for my /home/rames (or at least /home/rames/documents).

With FUSE it might even be possible for mere mortals like me.

Basically, I very rarely push more around more than 100-200kb at a time of "my stuff" unless it's big OGG's or tgz's, etc. Mostly source files, documents, resume's, etc. In that case, I want to be able to go historical to any saved revision *at the file-system level*, kindof like "always on cvs / svn / (git?)" for certain directories. Then when I accidently nuke files or make mistakes or whatever, I can drag a slider in a GUI and "roll-back" my filesystem to a certain point in time and bring that saved state into the present.

Performance is not an issue (at first), as I'm OK if my files take 3 seconds to save in vim or OpenOffice instead of 0.5 seconds. Space is not an issue because I don't generally revise Large(tm) files (and it would be pretty straightforward to have a MaxLimit size for any particular file). Maintenance would also be pretty straighforward: crontab "@daily dump revisions > 1 month". Include some special logic for "if a file is changing a lot, only snapshot versions every 5-10 minutes" and you could even handle some of the larger stuff like images without too much work.

Having done quite a bit of reading of KernelTraffic (Hi Zack) and recently about GIT, maybe it's time to dust off some python and C and see what happens...

--Robert

Re:HDFS (home-dir FS)?

[cowens]

What you want is something like the katie fs. it is a versioned filesystem. You can access the current version by saying vi /home/user/foo or an older version by saying vi /home/user/foo@@main/5 where main is the branch and 5 is the version number. I don't know if katie is still under active development anymore though.

Re:HDFS (home-dir FS)?

[onpaws]

Wait! Windows XP has:
a Previous Versions feature that can be tuned for a particular directory and a frequency (not CVS based though). Works very well. Also previous versions of files are available across network shares as well.

Plus, w00t you get the extra bonus of (S)low Performance that you mention...

Re:HDFS (home-dir FS)?

[Anonymous Coward]

http://www.kitenet.net/~joey/svnhome.html

Re:HDFS (home-dir FS)?

[mrsbrisby]

VMS has done this for years, and it IS great. But really, you should consider using one of these log structured filesystems (like what the article is about) because it already does exactly this.

Re:HDFS (home-dir FS)?

[SumDog]

You beat me to it. I was going to mention how OpenVMS uses versioning allowing you to keep several version of the same file. The syntax is a big odd, myfile.txt;1 and myfile.txt;2 are two different versions. Good luck trying to use OpenVMS though. If you do, I highly suggest installing the GNU file-utils. Thinks like ls and cd are wonderful commands compared to down and dir.

Re:HDFS (home-dir FS)?

[RosenSama]

How about mounting a WebDAV. I read that subversion can work this way when I upgraded to 1.2. Quoting from http://subversion.tigris.org/svn_1.2_releasenotes. html

Autoversioning is a feature whereby generic WebDAV clients can write to a DeltaV server (like mod_dav_svn), and the server performs commits silently in the background. This means that if you use Apache httpd as your Subversion server, then most modern operating systems can mount the repository as a network share, and non-technical users get "transparent" versioning for free. (Of course, technical users can still use Subversion clients to examine repository history.)

Re:HDFS (home-dir FS)?

[raynet]

It sounds like you could use Venti filesystem from Plan 9 OS. This FS never deletes files and all revisions are stored and you can rollback any file you like. And it is being ported to Linux.

Re:HDFS (home-dir FS)?

[Strolls]

The FUSE page you link to already lists (on its "Filesystems based on FUSE" page) Wayback which sounds exactly as you describe.

The project page doesn't give any details of how you access previous versions of files, but:

Wayback is an implementation of a versioning file system for Linux. This means that when you use a Wayback file system, old versions of files are never lost. No matter how much you change a file or directory, everything is always kept in a versioning file so that you never lose important data. Wayback provides the ability to remount any already mounted file system with versioning support under a different directory. Because of this, you can use Wayback on any block device with any base file system (ext3, ReiserFS, FAT, etc).

Re:HDFS (home-dir FS)?

[OnanTheBarbarian]

You should at least look at the papers and documentation on the Plan 9 file system, if you haven't already.

This will be no direct help for you, as you're talking about doing something that sounds fairly user-level. Still, I think you'd probably find the design of the system thought-provoking.

The Plan 9 WORM-based file system was doing something like what you describe (on a daily basis - for pretty much the entire filesystem, large files and all) back in the early 90s. This was implemented on top of a WORM jukebox (with hard drives serving as a cache) and done on a separate file server. The capacity of the systems and cache in question are, in todays terms, not spectacularly huge, but the technology is still very impressive. And remember, this is from the days where a 9Gb hard drive was a pretty l33t thing to have...

Re:HDFS (home-dir FS)?

[Dr.Dubious DDQ]

You could possibly implement it with DavFS...

Re:HDFS (home-dir FS)?

[po8]

The home page for the ext3cow filesystem also does a good job of referencing other versioning filesystems and the like. I think you'll find what you want linked from there. I recommend Wayback, although it doesn't do everything you asked for.

Re:HDFS (home-dir FS)?

[adolf]

A lot of what you ask has been available in Linux LVM[1/2] for a Long Time, in FreeBSD for years and years, and in NetApps filers since the Dawn of Time. In userspace, rsnapshots does much of what you mention, along with a variety of related scripts/utilities.

I assume that you know this, but I'm filling in a few holes for anyone else who might feel inspired to code...

Snapshots rock. Making them smarter and more efficient would be very nice.

Re:HDFS (home-dir FS)?

[Ramses0]

Thanks for the links! ... Gotta love the Time-Travelling File Manager!

--Robert

V0.1 vs V1.0

[HermanAB]

I suppose the released version is lossless, compared to their first development version, which was a wee little bit buggy?

Lossless vs. Lossy Filesystems

[xilmaril]

nobody seems to know the difference between lossy and lossless filesystems. neither do I, and neither does whoever wrote the article, it seems.

but hey, that's never slowed me down.

This new filesystem is like old ones, with a big difference and a few small ones.

It has something called 'snapshots', which seems to mean that you can work off of a partition, but seperately load up the version of that partition you had before you last had a power failure, or whatever went wrong.

it also claims to:
        * Fast write and recovery times
        * Minimal damage to file data and system consistency on hardware failure
        * Correctly ordered data and meta-data writes
        * File and inode blocks are managed by a B-tree structure
                    o Can create and store huge files
        * Internal data are processed in 64 bit wide word size

that no doubt means something, to somebody. lucky them!

and yes, this is actually more informative than the article.

remember kids, never, NEVER RTFA. it will only confuse and scare you.

Re:Lossless vs. Lossy Filesystems

[mrsbrisby]

It has something called 'snapshots', which seems to mean that you can work off of a partition, but seperately load up the version of that partition you had before you last had a power failure, or whatever went wrong.

Snapshots are easy to get without filesystem support (LVM).

* Minimal damage to file data and system consistency on hardware failure

Log-structured filesystems do NOT do this. For this you need RAID.

* Correctly ordered data and meta-data writes

Again, not inherent to a log-structured filesystem. In fact, every filesystem should have a correct way to perform ordered writes, and as long as recovery is done by a knowlegable system, most filesystems can be trivially converted to journalled filesystems.

* File and inode blocks are managed by a B-tree structure

Again, not inherent to a log-structured filesystem. Many filesystems use this method because it's easy, but note that B-trees (or B+Trees) aren't necessarily faster for real-world activity, and a poorly balanced tree can even provide much worse performance than a naive approach.

Yes this happens in reality.

Isn't this just like a tape drive on hard disk?

[GecKo213]

Sounds to me an aweful lot like a tape drive. Start at one end and start writting until you're done. I can see the point of wanting to keep all parts of each single file together in one block so that it's not broken up. That way there is no need to defrag, but I thought ext2 and ext3 did that type of thing already. Correct me if I'm wrong, but I was told that ext2/ext3 would keep a file whole at just about every cost pending a really really full drive and absolutley no contiguous room to put it, then it'd break it up. Am I wrong?

I'm also not sure how seek time is going to be affected by this. I'm not an expert on seek times for hard drives or anything, but if the File Allocation Table says it's out at point X and the PC then gives instructions to retrient that file by saying "the file starts at X address" the seek time is the same no matter how the file is sitting on the drive. Correct me if I'm wrong, but lets look at a quick scenario to make sure we're all on the same page. Then if I have fault in my thinking, please tell me. Lets say that a file starts at the farthest point away from the reader head on the drive that it can start and still fit the file there. (The very last section on the drive that the file can fit) In both instances the head would have to move the same distance from it's starting point wouldn't it? I though that seek time was dependant mostly on the hard drive hardware speed itself as the time that it took the head to get to the starting point of the file. If the file is broken up into tiny pieces all over the drive it's going to take longer for the file to be read in completely, but the seek time isn't changed to find the head of the file is it? Maybe I misunderstand "seek time" someone, if I'm way off please enlighten me. I'm always good for an education. Thanks.

Re:Isn't this just like a tape drive on hard disk?

[mrsbrisby]

Sounds to me an aweful lot like a tape drive

Exactly like a tape drive. Because of this, you get all the benefits of a type drive: straightforward implementation, guaranteed write times, and inherently transactional.

This is a good thing.

Correct me if I'm wrong, but I was told that ext2/ext3 would keep a file whole

You're wrong. Firstly, fragmentation isn't necessarily a bad thing. Generally collecting data as long as you're going the same direction (continuous sectors, or alternating) is very fast. Fragmentation is a very good thing in specialized systems where you need a guaranteed read time (as opposed to fast if possible, slow if not).

... The seek time is the same no matter how the file is sitting on the drive.

No it's not. On modern disks, switching heads is instantaneous because it's an electronic (not a mechanical) operation. Moving the head to the next physical sector is fast, but moving it "backwards" is slow (you have to do a full revolution). Extending to a different track is also slow.

Certainly, there is some aspect of the geometry that can make seeking faster.

Nothing has been gained just yet...

[Jason Hood]

Here is a sampling of the known bugs

The system might hang under heavy load.

That is what raid can do?

[nietsch]

Raid 5 allows you to keep 1 or more parity checksums of the volume. In principle you could use partitions on the same disk if you cannot afford a multi disk setup.

N is for...

[JemalCole]

Nannies.

And yes, they've been around for years.

Different FS

[phorm]

I usually put /var/log in a seperate partition anyhow. The easy solution would be to put this FS just on the areas that you want to be "lossless" and leave the rest with standard filesystems.

Re:erm.. lossless file system?

[WhiteWolf666]

I only use LOSSY filesystems. I figure entropy is the natural way of things, and as such, my filesystem should loose bits left and right (not really loose, just reset them randomly left and right).

These 'holes' in the data allow new data to 'evolve' through a darwinian selection process.

Yes, I use FAT16 for everything. MS clearly understood how to generate mutations in the 'ecosystem' of the software world, and MS clearly understands how mutations drive us towards the inevitable evolution of Skynet!

I, for one, welcome our new, homicidal robotic overlords!

Better than some, anyway.

[jd]

As mentioned on the LFS website, there were a number of attempts to implement a log-based filesystem for Linux. Some did work for the kernels they were written for, others do not appear to have worked at all. Virtually all were abandoned - and, to judge from my own research and that of the people who wrote the LFS website, many were abandoned around the Linux 2.2 era.

I guess you can argue that if a project is actively maintained, any problems are potentially fixable. Even with Open Source, an abandoned project is going to be a lot tougher to work on - especially if it was broken at the time it was abandoned, as then you've got to spend time figuring out what was intended, not just what was coded.

Personally, I'd be more in favour of improving the file system layering - eg: having a layer that supports different forms of journalling and logging. Then you could remove explicit logging from the filesystems themselves, you'd only need to provide hooks to the journal/log layer. This would surely reduce bloat, allow people to experiment with different combinations of journal/log and filesystem, etc. If you add a metadata layer, you could even resolve many of the politics of Reiser4 by moving the directories-as-files and other metadata code into such a layer, making it universally available to all filesystems.

It is certainly true that abstracting out and encapsulating in this way usually produces slower code, as you can't have the calls as tightly written or the layer as specialized. On the other hand, I'm going to argue that loose bindings make for easier maintenance and will also increase the number of people who would have an interest in maintaining the code, which will eventually lead to better code and therefore faster code. Only it won't just be faster for one filesystem, it'll be faster for ALL filesystems the user has linked that log/journal component to.

Why compared with Sun and not BSD?

[Alejo]

The UFS filesystem used by Solaris provides a data "snapshot" feature that prevents such data loss, NTT Labs says, but filesystem operation must be suspended to use the feature, reducing performance. NILFS, in contrast, can "continuously and automatically [save] instantaneous states of the file system without interrupting service," NTT Labs says.

The BSD original Soft Updates and snapshot implementation has a *minimal* impact on operation. See McKusik's paper Running Fsck in the Background.

Re:Why compared with Sun and not BSD?

[bani]

soft updates sounds exactly like what reiserfs does.

Faster

[hummassa]

Actually, it's faster-to-write, slower-to-read.

Re:Faster

[NitsujTPU]

Ideally a cleaner will go in, after which it will be nice and clippy to read as well.

Re:Faster

[shmlco]

Un huh. And the extra processing time the required "cleaner" takes up shouldn't be charged to the overall speed, much like Java's GC?

Re:Faster

[NitsujTPU]

I don't believe that garbage collection is the major performance issue with Java. I could be wrong, but I'm pretty sure that I'm not.

Please refer to this post for a couple papers on log file systems.

Re:Faster

[NitsujTPU]

Oh yeah, fsck runs much more quickly on log structured filesystems than on conventional filesystems. Journaling has its own overheads, if you're going to jump onto that track.

Re:Faster

[controlguy]

Perhaps adding the "cleaner" time as your suggesting should depend upon the context:

(1) Applying it to a machine that performs read and write operations all day: yes, it definitely should be added because it impacts perceived performance

(2) Applying to a machine that performs read and write operations all day except during the period 2AM-5AM: then, no, it should not be added because the "cleaner" should be executed during this time period, giving no percieved slow-down.

For those of us who leave our machines idle for long periods at a time, this could be useful -- if it performs as advertised.

RCS or something similar

[jd]

For ASCII text, the easiest way to maintain a log system is to use a revision control system such as RCS, CVS, Subversion, etc. Essentially, that's all these systems are - except they usually support forks in the log, they're not strictly linear collections of diffs.

For binary document formats (eg: MS Office's .doc format), things get tougher. There are versions of diff that'll work on binary files (which is why you can get binary patches), but it's more common to see logging done as a series of macros where the instruction sequence is replayed to get the same result, as opposed to recording actual changes. (Most so-called program "flight recorders" operate by this method.)

A better approach might be a hybrid of these - have a method of extracting/inserting the text as ASCII (which you can then log using a conventional source-control system) and have control sequences recorded as a macro on top of that. You then have to be absolutely certain that the correct two are applied together and in the right order, which may get complicated if there are branches, but the increase in complexity should be more than matched by a reduction in the amount you need to actually store.

Re:RCS or something similar

[Trinn]

Strangely enough, this sounds like one of the few places XML actually -makes sense-. Generally I am against the bloat it introduces, but in this case it sounds perfect as it would contain the document structure as well as information all in ASCII/UTF text.

Re:RCS or something similar

[pediddle]

FYI, CVS is pretty much the only revision control system that can't efficiently handle binary files. Subversion, for example, will use binary diff automatically (even on ASCII files), so changes to Word documents or other binary files will always be stored efficiently.

Not really +1, Insightful

[hummassa]

Because old stuff can be overwritten when you fill the whole disk. As I mentioned in other posting, data writes are Real Fast in log filesystems, but data reads are Real Slow.

The biggest problem of this filesystem (link is missing from the original posting) is that it's Not Really Ready (among other important stuff, mmap() is not implemented yet).

getting rid of unwanted data

[pe1chl]

Doesn't such a filesystem store all data semi-permanently, even the data you no longer want?
When I delete a file, maybe I want it to be nonrecoverable.

It would be nice to have a feature in the system that automatically wipes deleted files.
While I have found utilities that wipe files by overwriting them with different passes of varying data, this may not work at all in a journalling filesystem where a rewrite of a file not necessarily stores the datablocks in the same location. It also does not work when you forget to wipe the file before deleting, or for temporary files that are deleted without user action.

What I would like to see is some daemon process or kernel thread that securely erases all files that have been deleted (or truncated) through the normal system calls. Probably you would want to have it configurable to write only zeroes or write that DOD-certified series of patters for the really paranoid. And maybe some configurable priority for the overwrites (only do it when the disk is really idle, or perform the overwrites with some urgency).
Blocks from files being deleted by the filesystem would be handed over to this daemon to be securely erased at a convenient time.

Same for swapspace. Blocks from processes that have exited or that have been swapped back in and subsequently modified in memory would be erased.

Does such a thing exist for Linux?

Re:getting rid of unwanted data

[SumDog]

The trouble with what you want is that it is horrible inefficient and slow, and a Journaling file system is no less secure than a regular file system. The data still gets written, however before a critical part of the table is updated, the write is stored in a journal first. That way if the computer dies during a write to the allocation table, you don't loose all your file and directory information.

If you want to a specific file permanently, there are utilities that do that for you and remove all traces of the data from the hard drive, but doing this for each and every request would require a lot of overhead.

Re:getting rid of unwanted data

[vranash]

alias 'rm'='wipe' :P

Re:getting rid of unwanted data

[mrsbrisby]

When I delete a file, maybe I want it to be nonrecoverable.

Log structured filesystems normally come with a garbage collector that cleans up things like this.

[secure deletion] Does such a thing exist for Linux?

http://intgat.tigress.co.uk/rmy/uml/sparsify.html

Re:getting rid of unwanted data

[pe1chl]

That is why I prefer some background service in the system that does the erasing when the disk is idle, instead of for each remove request. The freed blocks would be put in a destruction queue to be overwritten when convenient, and the system can continue operation.
When a diskblock has to be allocated and will be overwritten anyway, the system could use a block from that destruction queue and it would not need to be overwritten.

Re:getting rid of unwanted data

[pe1chl]

This is somewhat like what I mean, but:
- I use Reiserfs, not ext2fs
- I'm not that paranoid that I want immediate zeroing at remove time. for me it would be "good enough" when the overwriting is done in the background and at low priority.

Also I do not like the idea of a user process directly accessing the FS. Probably good enough for a virtual machine environment where you can stop the machine and run the program, but this is for a live filesystem.

Up to now, I have not found anything that really does what I want...

For swap, it would be possible to zero it on boot and shutdown, but it takes a bit too long on my 2GB swap, and also it would not help against forensic analyzers who are careful not to boot the system.

Re:getting rid of unwanted data

[Just Some Guy]

Does such a thing exist for Linux?

SCRATCH=/filesystem/to/clean/someUnlikelyFileName
dd if=/dev/urandom of=$SCRATCH; rm $SCRATCH

will probably work. It overwrites every available block on your drive with random data, then deallocates them. If anyone knows why that wouldn't work, I'd be interested in hearing it.

Assuming that it does actually do the trick, it might be even better than wiping a single file. Since the whole drive would be filled with random data, there wouldn't be any conspicuous wiped blocks sitting in the middle of an otherwise normal looking filesystem.

Re:getting rid of unwanted data

[mrsbrisby]

I use Reiserfs, not ext2fs

Well, since Reiserfs doesn't do what you want, why not switch? Or consider adding the support you DO want to Reiserfs? Or pay someone to do it!

Also I do not like the idea of a user process directly accessing the FS. Probably good enough for a virtual machine environment where you can stop the machine and run the program, but this is for a live filesystem.

You're kidding, right?

What do you think fsck does?

For swap, it would be possible to zero it on boot and shutdown, but it takes a bit too long on my 2GB swap, and also it would not help against forensic analyzers who are careful not to boot the system.

What you really seem to want is an encrypted block device.

Re:getting rid of unwanted data

[lemonjelo]

An encrypted filesystem will get you most of what you want. It'll require overhead for every file, not just deleted ones, can't push that overhead to the background, and could make recovery of the files you didn't delete more difficult (although this is a feature =)

If you use things like PGP, look into encrypted swap anyway, and set /proc/sys/vm/swappiness to 5 to make it less painful as linux loves to swap

Re:getting rid of unwanted data

[MSG]

That process isn't really better or worse than using 'dd if=/dev/zero'. One advantage of using /dev/zero is that if the filesystem is corrupted, you're better able to locate actual data on the disk.

Re:getting rid of unwanted data

[pe1chl]

It would work, but it has two problems:

- it would take a very long time to write over all the hundreds of free gigabytes on this 1TB system, while it would be sufficient to write only over the blocks that once belonged to files and are now free
- at the moment the dd hits a full filesystem, random things may happen to other processes running on the system (disk full conditions are typically badly handled in Linux programs)

Also, as remarked, it would be better to use zero blocks. I have tried the above on my swapspace once, and it was apparent that the rate of output from /dev/urandom was only 10-20% of the sustained write rate of the disks. I.e. using /dev/urandom made it 5-10 times slower than using /dev/zero.

Re:getting rid of unwanted data

[pe1chl]

Well, since Reiserfs doesn't do what you want, why not switch?

Because Reiserfs is a very good filesystem.

Or consider adding the support you DO want to Reiserfs?

Before I consider adding something, I always investigate if that same thing already exists.
When googling does not yield anything obvious, discussing on some forum may help.

What do you think fsck does?

fsAUUCck is not supposed to run on a live filesystem. when you do that, you are considered insane.

What you really seem to want is an encrypted block device.

The block device is already encrypted for the filesystem. As you may know, this offers little protection when the system is investigated before being shut down, or when you are forced to provide the key.

SuSE does not support encrypted swap by default. I could add this by fiddling a bit with their startup scripts, but I have not done that yet. Indeed, it would be possible to swap to a device with a random key generated at boot.
However, this still leaves the analysis possibility as long as the system has not been shut down.

You should try to run "cat (swapdevice) | strings | less" for fun.

(Useless Use of Cat Award alert: this is not a useless use of cat)

Re:getting rid of unwanted data

[pe1chl]

An encrypted filesystem does nothing whatsoever w.r.t. erasing deleted data.
My filesystem is already encrypted. Do you think that makes any difference when analyzing the loop device on which it resides at the block level?

Encrypted swap would be similarly affected.
Encryption serves a different purpose than destruction.

Re:getting rid of unwanted data

[lemonjelo]

Of course encryption serves a different purpose, note I didn't say this is exactly what you want. But if you're worried someone will recover data from the disk after it's deleted, what about the same data before it's deleted?

Apparently I misunderstood why you want this daemon, and wrongly assumed the threat of recovery you're considering.

Re:getting rid of unwanted data

[pe1chl]

Well, it seems the world and its idea of freedom is changing...
Before, when you were a normal citizen in a "free" country you had nothing to worry about.
Now, laws are proposed to keep all Internet traffic data for each citizen in the country, keep the location data for his mobile phone, photograph license plates at strategic points and keep that data.
Each buyer of recordable media is pre-assumed to be using it to record copyrighted material and charged with a levy.

From an "innocent until proven guilty" we are quickly moving into a "suspect until proven innocent" society.
In this environment it is best not to keep data on your disks that is no longer relevant to you.
It might be used as evidence in whatever case. Even browser cache files that happen to contain illegal material can be used as additional evidence indicating "he is looking at illegal material for some reason".

So it is better to erase it. You never know.

Flash based filesystems

[Mike McTernan]

I think a log based file system would be really good for FLASH device based file systems, where erasing can typically only be achieved for a block of the device at a time. Essentially most flash file systems end up copying data out of a block to some new block before erasing the old block and starting again, much like how a log based file system works.

Given that a lot of FLASH based devices are embedded and maybe portable using battery power (think PDA/Mobile etc...), this technique could add extra safety to data stored on the device if the battery were to drain during an erase or write cycle. Nice.

HOWTO-shut-down-fast

[achurch]

It never made sense to me that we'd have to "shut down" as opposed to just turning the thing off.

As everyone else is saying, you lose your disk cache that way. The solution to long shutdown sequences is rm /etc/rc.d/rc?.d/K* (or the equivalent for your distribution). 99% of the daemons out there will shut down just fine on SIGTERM and won't suffer any harm from a SIGKILL; let the standard shutdown script unmount everything and you're done. Shutdown time on my desktop is around 3-5 seconds after the global SIGKILL, which is definitely within my acceptable range.

(Caveat: make sure you check what you're deleting before you delete it! Some weird distribution might have the shutdown script as a K* script, who knows)

Re:HOWTO-shut-down-fast

[Matt Perry]

As everyone else is saying, you lose your disk cache that way.

No write cache. It is disabled both in OS and the hard drive. When I save it's written to disk.

The solution to long shutdown sequences is rm /etc/rc.d/rc?.d/K* (or the equivalent for your distribution).

I've already trimmed those, years ago before I started my current practice of just removing power. In my opinion a "long shutdown sequence" is if there is still power to the machine when I remove my finger from the power button. It used to be that power supplies cut the curcuit of power when you flipped the switch. Now it's "smart" power supplies that signal the machine to shut down. Sorry, I prefer it how it used to be. Done using the computer? Turn it off. No waiting.

My next step is to figure out how to get Linux, or some other OS of choice, into a ROM for instant booting. Waiting for booting sucks more than waiting to shut down.

Re:Lossless???

[Krach42]

Gmail lossless. As in, I have so much space here that I can store every copy of every email that I've sent out.

Also a bit in the Escalade lossless, as in, if it crashes, you can put it back together again, and have most of the pieces... or is that Volvo lossless?

what nitwit modded this troll?

[bgbarcus]

People who sling mod points around without engaging their sense of humour first should be banned.

a nice start, can't wait for it to mature

[Khopesh]

1. Distro support. I don't want to have to compile my own kernel. The FS needs to be supported by the distro (Debian in this case). I want to be able to create root partition and RAID with the FS.

You got it; the FS has to be supported in the default build of the kernel for the major distributions, especially the rescue-disk distros. They're actually almost there - nilfs.com notes it is supported by a "loadable kernel module; no recompilation of the kernel is required."

If I can't mount it with Knoppix, no way in hell am I going to use it for anything important.

There are also several bugs that need squishing - ls, df, and du all have problems representing disk space, and support of the GNU Tools is listed in their "planned" section (read: distant future, outside the "todo" list). (ACL, GRUB support, atime, and other newly deemed "essential" aspects are also "planned.")

Re:a nice start, can't wait for it to mature

[joel48]

My problem with Knoppix recently is that lvm* tools have been removed, so I can't activate my logical volumes.

Acronym

[tommyServ0]

For the lazy among you, I looked up what NILFS stands for:

New Implementation I'd Like to FSck

VMS isn't entirely closed source...

[Medievalist]

If you have a licensed copy of the OS you can get source. Or at least, you used to be able to - many years ago when I was considered a VMS expert (something I certainly am not now) I used to look stuff up in the fiche; most of the stuff I was referencing was written in BLISS, I think.

As I recall, RMS is an indexed file management system. I wrote a molluscan taxonomy database system that used it in the 80s... but I usually encapsulate all OS-specific stuff in subroutines, so somebody has probably ported it to a cheaper database by now.

Re:VMS isn't entirely closed source...

[Lucractius]

mmm what i would do to get a copy of the OpenVMS source... These days all such chance has vanished, HP are tight lipped, and not half the engineers that DEC were. I belive the Cost for access to the Codebase has gone from reasonable to 6 figure or so... not entirely sure... they dont exactly make it public the price figure, so thats an estimation peiced together from various sources.

Its good to run into another VMS person though. as an undergrad presently, im hoping to break out of the MS/Linux admin BA COmp Sec gets me desk job managing 50 computers kinda job and get myself a place in that 20 thousand new mainframe techs IBM want. I love the big iron. :-) Alphaservers and VMS are like little slices of the big iron cake that i can enjoy in my spare time. Soon as i get a spot for my Alphaserver 2100 Ill be popping 8.2 on it and firing it up. When was the last time you used VMS?

Re:VMS isn't entirely closed source...

[Minwee]

"As I recall, RMS is an indexed file management system."

I have heard Stallman referred to as a great many things on this site, but I think this is the first time anyone has used that comparison.

Re:VMS isn't entirely closed source...

[Medievalist]

HP are ... not half the engineers that DEC were.

Word, brother. Testify. I think Microsoft scooped up all the DEC geniuses that haven't already retired.

I belive the Cost for access to the Codebase has gone from reasonable to 6 figure or so... not entirely sure... they dont exactly make it public the price figure, so thats an estimation peiced together from various sources.

That sucks, but at least there is a FOSS VMS project running. Still seems to be in its infancy unfortunately.

im hoping to break out of the MS/Linux admin BA COmp Sec gets me desk job managing 50 computers kinda job and get myself a place in that 20 thousand new mainframe techs IBM want. I love the big iron. :-)

Don't get sucked into the horrible old IBM mainframe OS390/MVS mindset, though - linux on mainframes is the cutting edge of computer science, if you ask me.

Alphaservers and VMS are like little slices of the big iron cake that i can enjoy in my spare time. Soon as i get a spot for my Alphaserver 2100 Ill be popping 8.2 on it and firing it up.

The Alpha architecture is getting a little long in the tooth, but it's still a great design; IMO better than anything IBM ships. On the other hand, IBM rules when it comes to solidly built chassis... they just pay more attention to the physical characteristics of their machines, using thicker metal, stronger materials, etc.

When was the last time you used VMS?

I stopped adminning VMS machines in 1994, after more than a decade of concentration on DEC hardware and OSes. I still occasionally use them because I mostly do systems integration work these days, making dissimilar machines interoperate transparently to end-users. I think the last time I hacked on a VAX/VMS box was about two years ago (I have one in my basement, but I haven't fired it up in at least that long).

Re:VMS isn't entirely closed source...

[Lucractius]

The a team of software engineers quit DEC before it had even started going to compaq, Dave Cutler and a chunk of his infamous PRISM project team were pulled to MS for their initial NT os, and cutler subsequently left after (before NT 4.0), not sure about the others. The Hardware engineers have stuck it through though mostly. There are people that have worked For DEC, Compaq, and HP without even moving their office in over a decade lol. But with the axing of the alpha line, the hardware engineers are now mostly redundant, and the remaining ones are slowly leaking out of HP. There was a large chunk "sold" to intel, and a large chunk contracted to, then bought by AMD (Ever realise the AMD K7 EV6 bus and the EV6 bus for alpha machines were one and the same. The Athlon owes a lot to the Alpha.) And in the dying days of the final development of the last generation of the Alpha cpus, those engineers that had stayed the course began to look at their options, and many Chose AMD. It seems that AMD not MS was the one that got most of the engineers.

But while " The Alpha architecture is getting a little long in the tooth," is true. Theres some personal satisfaction to knowing that the CPUs were designed manualy, All the way to the end the Alpha cpus were designed by the most talented engineers manualy without the kind of automated structure building tools that have speed up the development of electronic devices, and integrated circuts. And besides, Alphas were built to last, My rack mount alphaserver has the most industrial case ive ever seen it looks like a steel impact protection cage and with how much it weighs id swear it was one. But that was A DIGITAL AlphaServer, not a Compaq one... so again it seems like DEC truly did know the best way to do things, if not always the most profitable.

Re:Lossless???

[Medievalist]

Is this lossless as in "my gmail emails are lossless" or is this lossless as in "my Escalade is lossless (because when it gets stolen or lost I call onstar and they find it for me)"

Your Escalade is lossless because nobody would steal that pig with gas at $3 a gallon!

So you prefer vaporware?

[Medievalist]

I don't get it - you are bragging that Sun has a filesystem that is too broken for release?

JFS, XFS

[SumDog]

I've been a long time user of JFS and XFS. I've found JFS more stable in the past, and it is nice that JFS's fsck tool actually does something as opposed to XFS's fsck which exists with 0 and whos manpage tells you to use xfs_recover.

However both of them seem to have grown to be fairly stable, however there are still many instances where I've suffered dataloss, sometimes entire directories which I though was what a journaling filesystem was supposed to prevent...journaling directory meta data before writing it.

There was one instance where I had a hard drive continually being corrupted because of a bad hard drive controller, in which case no file system could have prevented data loss. Still in the pass I hate to think about how much media and data I've lost because of a power outage and a corrupt filesystem. Good think in Gentoo you can always to an "emerge world --emptytree" to get all your binary executables back, so long as didn't loose your compiler in the crash.

Quibble

[abulafia]

Snapshots are easy to get without filesystem support (LVM).

Not atomic snapshots. Not that everyone needs this, but if you do, it does require FS support. Netapp's file system, for instance, supports atomic snapshots. Very nice for hot backups of DB data files, for example.

Re:Quibble

[mrsbrisby]

Not atomic snapshots. Not that everyone needs this, but if you do, it does require FS support. Netapp's file system, for instance, supports atomic snapshots. Very nice for hot backups of DB data files, for example

You don't need the filesystem itself to be designed for it. If the filesystem supports an unmount operation, LVM could eventually do it. XFS's freeze operation is a good example of this.

Useful for audit trail, no other unique value

[Medievalist]

What is cool about it, for auditors obsessed with SOX, GLB & HIPAA compliance, is that it maintains a near-perfect audit trail at the cost of disk space (which is cheap compared the other costs of regulatory compliance).

You can mount the disk at any checkpoint, essentially providing a snapshot of each of the filesystem's unique states - unlike conventional backup (the usual 24-hour backup cycle gets a very coarse picture of the changes, obviously).

Other than that, it's yawn-yawn SSDD. The whole "lossy/lossless" thing is overblown hype, because all disk hardware fails to complete a write operation if you remove power before the operation is complete. They are trying to make their system seem better than it really is by making others seem worse than they really are. The fact is, plenty of people have run totally lossless data storage systems using ext2/ext3 and ISO9660, and no filesystem (including NILFS) can prevent your data from being butchered if your power supply is crap.

Re:MILFS1.0

[beanluc]

NetBSD's LFS is totally unexciting compared to MILFS 1.0, which everyone knows will go down on you, play rusty trombone with you, wrap her boobs around you, and otherwise get you off while her hubby and kids aren't home.

It sounds like you want NTFS

[Anonymous Coward]

NTFS has had criteria 1-6 for over 12 years, 7 and 8 for 5 years, and 9 has been around since the most recent versions. About the only thing it doesn't support is versioning, although snapshots are often as good.

Since it was designed for POSIX compliance, it has file change times, case sensitivity, and hard links. About the only issue is that you may have to write your own plug-in for proper soft link support.

As bonus features, you also get a change journal (persistent log of all directory changes -- useful for indexing applications), programmatic sparse files (you use ioctls to tell it where the holes are), Unicode filenames, multiple data streams per file, and a plug-in architecture (reparse points).

Additionally, there are already plug-ins (reparse point filters) for things like copy-on-write files, hierarchical storage (an old file may be archived to CD, but still has a directory entry, and when opened is brought back onto disk), and persistent mount points.

And in upcoming versions there will be more goodies, such as directory quotas, filename restrictions (so you could prevent somebody from creating a filename that ends in '.exe' in your web root), and fully distributed ACID transactions.

Now some may argue that some of these things, such as compression and encryption, don't belong in the filesystem. However, I can't think of any way to do it cleaner. For one thing, in order to encrypt a file you must first compress it (reduce entropy). Do you have compression on top of encryption on top of the filesystem, or do you have two different compression implementations?

Your compression layer would need to intercept not only reads and writes, it would have to intercept calls to get or set the file length, otherwise an unsuspecting program would get the compressed size. And now your compression layer needs some metadata about which files are compressed and what their expanded sizes would be. Where do you store that? Do you put it all in one file somewhere that an unwitting sysadmin might be tempted to delete it to save space, or where a single corrupt sector could screw up all the data, causing you to lose every one of your compressed/encrypted files? Remember, most filesystems log metadata changes, not data changes. Since compression is sitting on top of the filesystem, its metadata looks like regular data to the FS, so it is (along with all of your files) vulnerable in the event of a crash.

What happens when somebody decides to optimize the compression layer by storing some of the compressed data (perhaps a Huffman coding table) in the metadata file? Since the metadata file probably isn't encrypted, an attacker now has some information to help him decrypt your files!

dom

Re:It sounds like you want NTFS

[davegaramond]

Yes, I might be secretly in love with NTFS. If only it had solid support on Linux/Unices.

But I read that NTFS is slow with large dir and lots of small files?

ECC

[RAMMS EIN]

Huh? I was under the impression that hard drives have had Error Correction Codes for ages. Wouldn't that make the kind of errors you describe extremely unlikely, and even less likely that they would go undetected?

What the hell is a numpty?!

[sp0rk173]

Anyone? Seriously that's an awesome word. What's it mean?! Is it the same as moron?!

Re:What the hell is a numpty?!

[otterboy]

Look it up, it's faster than asking:
UrbanDictionary
Scottish usage: a) Someone who (sometimes unwittingly) by speech or action demonstrates a lack of knowledge or misconception of a particular subject or situation to the amusement of others.

Based off of Plan9 Venti filesystem

[Anthony Liguori]

This filesystem most likely stems from the ideas in Venti which is a filesystem that came out a few years ago. Link is included at the bottom. The (quite compelling) argument made originally is if you graph unique data growth (unique is important here) and compare it to the growth of commidity storage, as long as the storage growth is greater than unique data growth, there's no reason to ever delete anything since you essentially have infinite storage.

This is actually compelling for a large number of use cases. Many types of servers (think any sort of application server that's not a file server or something) fit this model. The disaster recover story is just fantastic, you can rollback to virtually any instance of the previous state of the filesystem.

FWIW, most new distributed SCMs are based on this same idea (like git or mercurial) so this is a pretty well-understood paradigm.

http://www.cs.bell-labs.com/sys/doc/venti/venti.ht ml

Server cases [was Re:Data is the new currency...]

[dzarn]

Totally offtopic, but do you have a good source for cases to hold a lot of drives? Right now I'm just using a mid-tower, but I'm out of bays. I'd like something with a crapload of 3.5" bays and at least some thought as to air circulation, without paying for an all out server case. Any tips?

Faster than UFS?

[glwtta]

Is there anything out there that's actually slower than UFS? Achieving that seem like it would be a real technological feat.

Re:Geez, talk about confusion!

[jcuervo]

Damn it, now every time I see a MILF video, I'll be thinking about Linux.

Version control FS

[lemaymd]

I've always wondered if it would be worthwhile to have a version control style filesystem. Basically, a built-in subversion or CVS-type backend that operates without your knowledge until you need to roll something back. Of course you'd need to manage space usage, but with 1TB storage capacities within reach of the home user perhaps it's less of an issue? I think losing data is quite a bit more expensive than storage these days. :-)

file versioning on OpenVMS

[John_Sauter]

what you describe reminds me of the RMS versioning that the ODS filesystem on OpenVMS uses. But due to its closes source nature its difficult to tell wether its actualy writing a copy of the whole file or as you said, a delta of the changes. I would be interestred to know just how it was done myself.

The file versioning was actually in ODS-2, not in RMS. Creating a new file by the same name as an existing file would increment the version number, which was part of the file name. For example, if you edit file.txt.1 the result would be file.txt.2. The new file contained the complete contents; you could delete file.txt.1 without disturbing file.txt.2.
        John Sauter, former DEC software engineer, (J_Sauter@Empire.Net)

Re:file versioning on OpenVMS

[siglercm]

Wow, surely I'm not the only one who remembers the syntax/nomenclature.... Existing file is file.txt;1. Edit it -->

$ edit/edt file.txt

Save and exit and, voila, you have file.txt;2. Original is version 1, new is version 2. Gets really annoying when you've futzed around with a PROG.FOR source code file a lot because you're brain-dead. I wrote a DCL script called something like CLEANUP.COM to get rid of all those hundreds of versions of files I had lying around.

Re:file versioning on OpenVMS

[John_Sauter]

I hope your CLEANUP.COM used the PURGE DCL command.
        John Sauter, former EDT project leader, (J_Sauter@Empire.Net)

But how long ago?

[Sterling Christensen]

Maybe you tried it back when it used to be buggy?

Re:But how long ago?

[Fnord]

Beginning of this year?

Old News, New Robot

[LRSDscout]

The sniper's bullet makes sound AS IT TRAVELS because it is traveling at supersonic speed. That is how the location of the sniper is found. If there is more than one robot, they can triangulate the location within a few feet. We used this technology in Bosnia mounted on Bradley's almost 10 years ago and it worked well.

Re:Linux files systems suck ass..

[csirac]

So what are the choices? ext2/ext3 which are slow, reiserfs which sucks ass when it breaks..

Apart from the big (production quality) alternatives like IBM's JFS (which I use myself) and SGI's XFS (and Reiser - "Reiser sucks when it breaks" is so 1999) Linux additionally supports the following filesystems (from http://www.xenotime.net/linux/linux-fs.html, also try http://www.tldp.org/HOWTO/Filesystems-HOWTO.html):


* accessfs: permission filesystem
* AFS FAQ
* AutoFS
* AVFS: A Virtual File System
* AVFS: A Virtual Filesystem
* BeFS for Linux2.4
* BeOS filesystem for Linux
* BFS UnixWare, Linux Implementation of
* CDfs
* CIFS
* Cluster File Systems Home
* Coda File System
* Crypto HOWTO
* convertfs: convert Linux filesystems
* DAFS: Direct Access File System
* Devfs (Device File System) FAQ
* efslook: read/debug EFS filesystems (ported to Linux & x86) {new}
* Enhanced Loopback (for XFS)
* E2fsprogs: Ext2 Filesystem Utilities
* E2fsprogs: Ext2 FS Utilities
* Ext2 Compression Extension
* FSDEXT2: ext2 filesystem for Win95
* ext2fsnt: Linux Ext2 filesystem for Windows NT driver
* ext2 fs resize utilities
* ext2 online growth patch (adilger)
* Explore2fs
* FHS: Filesystem Hierarchy Standard
* FHS Test Suite
* Filesystems HOWTO
* FiST: File System Translator
* FreeVXFS
* fsv: 3D File System Visualizer
* Global File System (Sistina)
* GNU ext2resize: ext2 filesystem resizer
* HFS+ filesystem
* HFS+ filesystem #2
* InterMezzo FS
* JFFS (axis.com)
* Journaling Flash File System, v2
* JFS for Linux (IBM)
* journal_fs report (OSDL)
* Kragen's Amazing List of Filesystems
* Large File Support in Linux
* Large File Summit (LFS) stuff
* Large File / File System Support
* Linux EXT2 salvage utility
* Linux Ext2fs Undeletion mini-HOWTO
* Linux FAT-32 support
* Linux filesystem redundancy
* Linux Links: File Systems
* Linux NFS FAQ
* Linux RAID Solutions
* LUFS: Linux Userland File System
* Lustre & Lustre Light filesystems
* NFS project
* NFSv4 for Linux 2.4
* Linux NTFS file system support
* loop-aes cryptoloop device

I don't know, but I don't believe it

[hummassa]

Even with a "cleaner", I bet any block is more blocks of access away in a log-structured fs to read than in more conventional types of fs.

Re:I don't know, but I don't believe it

[nick8325]

Well, the original paper about it is here (PDF here).

Quite an interesting read if you're into that sort of thing...they got much better performance than FFS - the paper says it could use 70% of the disk's bandwidth for writing, compared to 5-10% in FFS, and was only worse than FFS in sequentially reading a file that had been written randomly.

I think there was a follow-up paper improving that, too, but I can't find it.

Re:I don't know, but I don't believe it

[NitsujTPU]

There are tons of follow-up papers. I think that I've read 4 this semester where someone used some form of log file system.

Re:I don't know, but I don't believe it

[nick8325]

I was thinking of one specifically improving the performance of reads on Sprite LFS. I bumped into it when I originally read that paper, but can't find it any more.

By the way, the link below your username leads to a 404.

Re:I don't know, but I don't believe it

[NitsujTPU]

Yeah, we just changed servers. I should updated it.

Wow, you know what you're talking about. Thank goodness someone in this discussion does.

I should have figured when you started pointing to links to papers (I mention the same ones in another post actually), and knew your way around citeseer.

I get a bit frazzled whenever someone posts anything of significance to Slashdot. A bunch of people with little knowledge on the topic tend to chime in. Kudos on chiming in and actually being educated about the material at hand :-D I wish more folks like you would post here.

Re:I don't know, but I don't believe it

[NitsujTPU]

Oh, rather, we changed servers, and when we changed servers, I changed my user name. And s/updated/update/g.

Re:I don't know, but I don't believe it

[nick8325]

Wow, that's one of the nicest compliments I've had! You made my day :-)

Actually, I'm only a second year undergraduate studying CS, but I've been interested in operating systems for a long time, so I tend to know (and read) more than I should about them...

Re:I don't know, but I don't believe it

[NitsujTPU]

Not a problem. It's impressive that a second year UG would even know what a log filesystem is. I didn't learn what one was until grad school.