Linux Gains Lossless File System

Anonymous Coward writes "An R&D affiliate of the world's largest telephone company has achieved a stable release of a new Linux file system said to improve reliability over conventional Linux file systems, and offer performance advantages over Solaris's UFS file system. NILFS 1.0 (new implementation of a log-structured file system) is available now from NTT Labs (Nippon Telegraph and Telephone's Cyber Space Laboratories)."

Comments

New Improved?

[TheRaven64]

The article was a bit light on details. Perhaps someone could enlighten me as to exactly why this is better than existing log-structured filesystems, such as NetBSD's LFS.

Re:New Improved?

[Feyr]

the why is dependent on your application,

for common servers, or day-to-day use. it isn't

but notice how this was developped by a telecom company? a log structured filesystem is perfect or even required, due to speed and integrity constraints (depending on the size of the network), when you're dealing with billing and monitoring data on a telecom network. you want something that's simple and extremely resistant to failures. a complete system crash (which never happen, short of nuking the box) should not result in any data loss, or the extreme minimum, and you should be able to recreate that data from somewhere else (eg, the other endpoint in a telephone network).

a log structured filesystem allow this, the "head" is never over previous data in normal operation. you don't typically read the data back until the end of a cycle (whatever that cycle may be) or in a debugging condition. you simply append to the end. minimizing head movement, and thus increasing mtbf (replacing a disk in those things is costly)

this is also extremely useful for logging to WORM media (write once, read many), for security logs mostly. you don't want a hacker to be able to remove them, no matter what they do

Re:New Improved?

[rpresser]

Logs structures are suceptible to termites, carpenter ants, and various forms of rot.

Even worse, when many logs are added together, the problems multiply.

Horrible headline

[Quasar1999]

A lossless file system? Good lord... I most certainly hope all the exisiting file systems out there are not lossy. I have hundreds of gigabytes of data that I don't want to lose.

Or is this filesystem somehow able to recover data once the hard drive crashes? That would be neat...

Re:Horrible headline

[TheRaven64]

The title was written by a numpty. This is a log-structured filesystem. These systems have been around for ages. NetBSD has LFS (originally from 4.4BSD), and I believe Minix also had some form of log-structured filesystem.

A log-structured filesystem doesn't modify existing files. Every time you write to the disk, you simply append some deltas. This gives very good write performance, but poor read performance (since almost all files will be fragmented, and the entire log for that file must be replayed to determine the current state of the file). To help alleviate this, most undergo a vacuuming process[1], whereby the log is replayed, and a set of contiguous files is written. This also frees space - something that is not normally done since deleting a file is done simply by writing something at the end of the log saying it was deleted. In addition to the good write performance, log-structured filesystems also have an intrinsic undo facility - you can always revert to an earlier disk state, up until the last time the drive was vacuumed.

The snapshot facility is not particularly impressive. It's a feature intrinsic to log-structured filesystems, and also available in other filesystems (such as UFS2 on FreeBSD and XFS on Linux). The performance advantage claims must be taken with a grain of salt - write performance for log-structured filesystems is always close to the theoretical maximum of the disk, but this is at the expense of some disk space, and read speed (although LFS did beat UFS in several tests on NetBSD).

[1] This is usually done in the background when there is little or no disk activity.

Re:Horrible headline

[addaon]

It should be said that "good write performance, bad read performance" is essentially the point, not a defect. It's easy these days to speed up reads a huge amount through caching; these days 100MB+ of UBC isn't rare. But when you have to write, you have to write (for reliability reasons); this can't be cached into memory, so it should be optimized for. The goal here is to make BOTH operations as fast as possible, though one is made fast at the disk layer and one is made fast above it.

So...

[Juiblex]

If it is lossless, I won't be able to store MPEG, XVid, JPEG and MP3 on it anymore? :(

Old news

[Anonymous Coward]

Websites with MILFS have been around for years.

Oh, wait. NILFS. My bad.

Re:Old news

[SatanicPuppy]

Well, we know it doesn't stand for nerds...

(Sorry...Couldn't resist)

Database Servers

[mysqlrocks]

Log-structured filesystems write down all data in a continuous log-like format that is only appended to, never overwritten. The approach is said to reduce seek times, as well as minimizing the kind of data loss that occurs with conventional Linux filesystems.

This sounds a lot like how database servers work. They keep both a log file and a database file. The log file is continuously written to and is only truncated when backups occur.

Stable?

[theJML]

I like how they say it's reached a stable release but if you look at the known bugs on the Project Home Page http://www.nilfs.org/ You'll see that:

The system might hang under heavy load.

The system hangs on a disk full condition.
Aren't those kind of important to saying that something is stable?

Re:Bloat?

[Skye16]

Scratch my previous post, I actually read the article. My bad :)

actual info about the fs

[cowens]

http://www.nilfs.org/

Excellent information retention

[totallygeek]

I installed this lossless file system. rm is now chmod 444. I have not been able to lose information since.

Note: instead of modding this +1 funny, mod it +0.1 pathetic.

There's no replacement for ext3fs yet for me...

[davegaramond]

I'd looove to replace ext2/3 as my filesystem for years since it's not so fast and most distro don't include binary tree indexing for ext3 (so large dir is slow). Unfortunately I haven't been able to do so. Here are my requirements:

1. Distro support. I don't want to have to compile my own kernel. The FS needs to be supported by the distro (Debian in this case). I want to be able to create root partition and RAID with the FS.

2. ACL and extended attributes.

3. extended inode attributes would be nice ("chattr +i" is handy sometimes).

4. optionally I would like to be able to create large Bestcrypt partitions (e.g. 30GB) with that FS.

5. fast large dir and small files performance (I have millions of small files on my desktop).

6. no need to fsck or fast fsck (i.e. journalling or some other technique or whatever).

7. disk quota!

8. optionally, transparent compression and encryption will be a big plus point.

9. Snapshots would be nice too, for consistent backups.

10. Versioning is also very welcome.

XFS: very close but it still has problems with #4. It also doesn't have undelete like ext2/ext3 (not that it's a requirement though).

JFS: it just lacks many features.

Reiser3: How's the quota support, still have to patch kernel everytime? Plus it doesn't have ACL.

Reiser4: not ready yet.

I might have to look at FreeBSD after all. Background fsck, hmm....

Re:Shutdown versus power off

[pesc]

Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command.

That's a very bad idea. Normally, journaling file systems only guarantee that the file/directory structure remains intact. It does not necessarily guarantee that the data in the files hit the disk. Also, your disk will probably have a cache that is lost when you remove power. Whatever is in the cache will also be lost.

So your file system may be intact, but your practices will probably destroy data.

Re:Bloat?

[ivan256]

I wrote a (unfortunatly, closed source) filesystem that was remarkably similar to this once. Generally these types of filesystems are used when you're constantly writing new data. You're going to be eating the space anyway, but you want the reliability of syncronous writes with the performance of asyncronous cached writes. Reading from these filesystems is incredibly slow in comparison.

The version I wrote took advantage of the client's bursty IO pattern and used the slow periods to offload the data to an ext2 filesystem on a seperate disk. Hopefully your system memory was large enough that the offload to the secondary filesystem happened without any disk reads. Once that was done, the older sections of log could be re-used.... But only once the disk filled up and wrapped back to the beginning, because you want to keep your writes (essentially... There's other timing tricks you can play to get more speed) sequential.

There's been lots of research done on this method of write structuring. Look for papers on the "TRAIL" project (also closed source), for example.

Re:Needs a new name

[chochos]

NILF: Netserver I'd Like to fsck (but I don't need to anymore, apparently)

HDFS (home-dir FS)?

[Ramses0]

I've had an idea kicking around for a while now... "HDFS / Home-Dir File System" ... I want a (s)low-performance, bloated, version controlled, roll-back featured, viewcvs.cgi enabled file system for my /home/rames (or at least /home/rames/documents).

With FUSE it might even be possible for mere mortals like me.

Basically, I very rarely push more around more than 100-200kb at a time of "my stuff" unless it's big OGG's or tgz's, etc. Mostly source files, documents, resume's, etc. In that case, I want to be able to go historical to any saved revision *at the file-system level*, kindof like "always on cvs / svn / (git?)" for certain directories. Then when I accidently nuke files or make mistakes or whatever, I can drag a slider in a GUI and "roll-back" my filesystem to a certain point in time and bring that saved state into the present.

Performance is not an issue (at first), as I'm OK if my files take 3 seconds to save in vim or OpenOffice instead of 0.5 seconds. Space is not an issue because I don't generally revise Large(tm) files (and it would be pretty straightforward to have a MaxLimit size for any particular file). Maintenance would also be pretty straighforward: crontab "@daily dump revisions > 1 month". Include some special logic for "if a file is changing a lot, only snapshot versions every 5-10 minutes" and you could even handle some of the larger stuff like images without too much work.

Having done quite a bit of reading of KernelTraffic (Hi Zack) and recently about GIT, maybe it's time to dust off some python and C and see what happens...

--Robert

Re:Shutdown versus power off

[Marillion]

Hard drives with voicecoil head positioning (some 99.99% of them sold in the past 10 years) will autopark on poweroff. The real risk is how much latency the os has between when an application saved it's data and when the buffers are flushed.

Some applications keep files open for a long time: MySQL, gDBM-based apps, Squid. Most of those application implement their own mini-filesystems within a file optimised for task. These systems are supposed to preserve their integrity by journaling their modifications in case the underlying os doesn't.

Switching off a computer because it has a journaling filesystem is like stopping a car by driving into a something because it has seat belts.

Re:Bloat?

[Alien Being]

/. is a journaled website. We can't scratch your previous post. Years from now you will be able to come back and show your grandchildren how silly you were.

Re:Data is the new currency my friend

[Old Wolf]

I keep all my original digital photos (in .tiff format) along with full-quality movies and all the games I've ever played back to Duke Nukem on 80x386 on a RAID array that's grown to nearly 2 terabytes.

So, basically, you're going to keep Duke Nukem forever?