Linux Gains Lossless File System
Anonymous Coward writes "An R&D affiliate of the world's largest telephone company has achieved a stable release of a new Linux file system said to improve reliability over conventional Linux file systems, and offer performance advantages over Solaris's UFS file system. NILFS 1.0 (new implementation of a log-structured file system) is available now from NTT Labs (Nippon Telegraph and Telephone's Cyber Space Laboratories)."
Please correct me if I'm wrong here but wouldn't a log that is only appended to and never overwritten cause a massive ammount of bloat after a period of prolonged use?
The article was a bit light on details. Perhaps someone could enlighten me as to exactly why this is better than existing log-structured filesystems, such as NetBSD's LFS.
I am TheRaven on Soylent News
A lossless file system? Good lord... I most certainly hope all the exisiting file systems out there are not lossy. I have hundreds of gigabytes of data that I don't want to lose.
Or is this filesystem somehow able to recover data once the hard drive crashes? That would be neat...
---
Programming is like sex... Make one mistake and support it the rest of your life.
If it is lossless, I won't be able to store MPEG, XVid, JPEG and MP3 on it anymore? :(
Websites with MILFS have been around for years.
Oh, wait. NILFS. My bad.
Log-structured filesystems write down all data in a continuous log-like format that is only appended to, never overwritten. The approach is said to reduce seek times, as well as minimizing the kind of data loss that occurs with conventional Linux filesystems.
This sounds a lot like how database servers work. They keep both a log file and a database file. The log file is continuously written to and is only truncated when backups occur.
Bradley Holt
Will there be a Windows Driver?
If there isn't, this has no chance on taking off. Consumers today want portability. They don't like lock-in. A linux exclusive format is lock-in.
Create a good windows (and Mac OS) driver, and it's got massive potential.
The system might hang under heavy load.
The system hangs on a disk full condition.
Aren't those kind of important to saying that something is stable?
-=JML=-
NILFS is a log-structured file system developed for the Linux kernel 2.6. NILFS is an abbreviation of the New Implementation of a Log-structured File System. A log-structured file system has the characteristic that all file system data including metadata is written in a log-like format. Data is never overwritten, only appended in this file system. This greatly improves performance because there is little overhead regarding disk seeks. NILFS also has the following specific features:
* Slick snapshots.
* B-tree based file and inode management.
* Immediate recovery after system crash.
* 64-bit data structures; support many files, large files and disks.
* Loadable kernel module; no recompilation of the kernel is required.
If you "get" pointers add me as a friend (116)!
" When the system reboots, the journal notes that the write did not complete, and any partial data writes are lost. "
Isn't this similar to NTFS's journaling file system?
If they are serious about a filesystem, it has to be bundled with the linux distros every release. Take Reiser and JFS for example, some distros have it, some don't. Not every release of the same distro has it, what a mess. Only two have stayed permanently EXT2, EXT3. Everything else is trendy.
More file integrity is always good. Ever since journaling file systems became available I just started turning the power off to my computers (via a power strip) rather than going through the shutdown command. It never made sense to me that we'd have to "shut down" as opposed to just turning the thing off.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
http://www.nilfs.org/
Note: instead of modding this +1 funny, mod it +0.1 pathetic.
Click here or here.
Aside from gasoline and water, data is the most valuable thing in the world.
.tiff format) along with full-quality movies and all the games I've ever played back to Duke Nukem on 80x386 on a RAID array that's grown to nearly 2 terabytes.
:) Same for HDDs
Walmart's most prized possesion is their billion-billion-billion transaction customer sales database. They use it to find things like, among other things, men tend to buy beer and diapers at the time.
With disks costing $1.00/GB or less these days, many people including myself simply DON'T delete data anymore. I keep all my original digital photos (in
So yes, for many people, disk space is just something you keep adding to, like you'd move from a coupe to a sedan when you have kids and when you have that 6th kid you move to a minivan and if you happen to have 2 more, you get a cargo van when #8 comes along
If you "get" pointers add me as a friend (116)!
I'd looove to replace ext2/3 as my filesystem for years since it's not so fast and most distro don't include binary tree indexing for ext3 (so large dir is slow). Unfortunately I haven't been able to do so. Here are my requirements:
1. Distro support. I don't want to have to compile my own kernel. The FS needs to be supported by the distro (Debian in this case). I want to be able to create root partition and RAID with the FS.
2. ACL and extended attributes.
3. extended inode attributes would be nice ("chattr +i" is handy sometimes).
4. optionally I would like to be able to create large Bestcrypt partitions (e.g. 30GB) with that FS.
5. fast large dir and small files performance (I have millions of small files on my desktop).
6. no need to fsck or fast fsck (i.e. journalling or some other technique or whatever).
7. disk quota!
8. optionally, transparent compression and encryption will be a big plus point.
9. Snapshots would be nice too, for consistent backups.
10. Versioning is also very welcome.
XFS: very close but it still has problems with #4. It also doesn't have undelete like ext2/ext3 (not that it's a requirement though).
JFS: it just lacks many features.
Reiser3: How's the quota support, still have to patch kernel everytime? Plus it doesn't have ACL.
Reiser4: not ready yet.
I might have to look at FreeBSD after all. Background fsck, hmm....
NILF: Netserver I'd Like to fsck (but I don't need to anymore, apparently)
Go hug some trees.
I've had an idea kicking around for a while now... "HDFS / Home-Dir File System" ... I want a (s)low-performance, bloated, version controlled, roll-back featured, viewcvs.cgi enabled file system for my /home/rames (or at least /home/rames/documents).
With FUSE it might even be possible for mere mortals like me.
Basically, I very rarely push more around more than 100-200kb at a time of "my stuff" unless it's big OGG's or tgz's, etc. Mostly source files, documents, resume's, etc. In that case, I want to be able to go historical to any saved revision *at the file-system level*, kindof like "always on cvs / svn / (git?)" for certain directories. Then when I accidently nuke files or make mistakes or whatever, I can drag a slider in a GUI and "roll-back" my filesystem to a certain point in time and bring that saved state into the present.
Performance is not an issue (at first), as I'm OK if my files take 3 seconds to save in vim or OpenOffice instead of 0.5 seconds. Space is not an issue because I don't generally revise Large(tm) files (and it would be pretty straightforward to have a MaxLimit size for any particular file). Maintenance would also be pretty straighforward: crontab "@daily dump revisions > 1 month". Include some special logic for "if a file is changing a lot, only snapshot versions every 5-10 minutes" and you could even handle some of the larger stuff like images without too much work.
Having done quite a bit of reading of KernelTraffic (Hi Zack) and recently about GIT, maybe it's time to dust off some python and C and see what happens...
--Robert
You know, F'ed is the last state I want to see my file system in.
"Learning is not compulsory... neither is survival."
--Dr.W.Edwards Deming
Here is a sampling of the known bugs
The system might hang under heavy load.
Are you intolerant of intolerant people?
For binary document formats (eg: MS Office's
A better approach might be a hybrid of these - have a method of extracting/inserting the text as ASCII (which you can then log using a conventional source-control system) and have control sequences recorded as a macro on top of that. You then have to be absolutely certain that the correct two are applied together and in the right order, which may get complicated if there are branches, but the increase in complexity should be more than matched by a reduction in the amount you need to actually store.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Because old stuff can be overwritten when you fill the whole disk. As I mentioned in other posting, data writes are Real Fast in log filesystems, but data reads are Real Slow.
The biggest problem of this filesystem (link is missing from the original posting) is that it's Not Really Ready (among other important stuff, mmap() is not implemented yet).
It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
Un huh. And the extra processing time the required "cleaner" takes up shouldn't be charged to the overall speed, much like Java's GC?
Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
If you have a licensed copy of the OS you can get source. Or at least, you used to be able to - many years ago when I was considered a VMS expert (something I certainly am not now) I used to look stuff up in the fiche; most of the stuff I was referencing was written in BLISS, I think.
As I recall, RMS is an indexed file management system. I wrote a molluscan taxonomy database system that used it in the 80s... but I usually encapsulate all OS-specific stuff in subroutines, so somebody has probably ported it to a cheaper database by now.
Assuming that it does actually do the trick, it might be even better than wiping a single file. Since the whole drive would be filled with random data, there wouldn't be any conspicuous wiped blocks sitting in the middle of an otherwise normal looking filesystem.
Dewey, what part of this looks like authorities should be involved?
I've always wondered if it would be worthwhile to have a version control style filesystem. Basically, a built-in subversion or CVS-type backend that operates without your knowledge until you need to roll something back. Of course you'd need to manage space usage, but with 1TB storage capacities within reach of the home user perhaps it's less of an issue? I think losing data is quite a bit more expensive than storage these days. :-)
So what are the choices? ext2/ext3 which are slow, reiserfs which sucks ass when it breaks..
Apart from the big (production quality) alternatives like IBM's JFS (which I use myself) and SGI's XFS (and Reiser - "Reiser sucks when it breaks" is so 1999) Linux additionally supports the following filesystems (from http://www.xenotime.net/linux/linux-fs.html, also try http://www.tldp.org/HOWTO/Filesystems-HOWTO.html):
* accessfs: permission filesystem
* AFS FAQ
* AutoFS
* AVFS: A Virtual File System
* AVFS: A Virtual Filesystem
* BeFS for Linux2.4
* BeOS filesystem for Linux
* BFS UnixWare, Linux Implementation of
* CDfs
* CIFS
* Cluster File Systems Home
* Coda File System
* Crypto HOWTO
* convertfs: convert Linux filesystems
* DAFS: Direct Access File System
* Devfs (Device File System) FAQ
* efslook: read/debug EFS filesystems (ported to Linux & x86) {new}
* Enhanced Loopback (for XFS)
* E2fsprogs: Ext2 Filesystem Utilities
* E2fsprogs: Ext2 FS Utilities
* Ext2 Compression Extension
* FSDEXT2: ext2 filesystem for Win95
* ext2fsnt: Linux Ext2 filesystem for Windows NT driver
* ext2 fs resize utilities
* ext2 online growth patch (adilger)
* Explore2fs
* FHS: Filesystem Hierarchy Standard
* FHS Test Suite
* Filesystems HOWTO
* FiST: File System Translator
* FreeVXFS
* fsv: 3D File System Visualizer
* Global File System (Sistina)
* GNU ext2resize: ext2 filesystem resizer
* HFS+ filesystem
* HFS+ filesystem #2
* InterMezzo FS
* JFFS (axis.com)
* Journaling Flash File System, v2
* JFS for Linux (IBM)
* journal_fs report (OSDL)
* Kragen's Amazing List of Filesystems
* Large File Support in Linux
* Large File Summit (LFS) stuff
* Large File / File System Support
* Linux EXT2 salvage utility
* Linux Ext2fs Undeletion mini-HOWTO
* Linux FAT-32 support
* Linux filesystem redundancy
* Linux Links: File Systems
* Linux NFS FAQ
* Linux RAID Solutions
* LUFS: Linux Userland File System
* Lustre & Lustre Light filesystems
* NFS project
* NFSv4 for Linux 2.4
* Linux NTFS file system support
* loop-aes cryptoloop device