Slashdot Mirror
Ext4 Filesystem Enters Experimental Kernel Tree
An anonymous reader writes "Looks like the next version of the venerable Linux 'ext' filesystem is just around the corner. Andrew Morton has added an early version of ext4 to his 2.6.19-rc1-mm1 tree, enabling Linux to support storage volumes up to 1020 petabytes in size, and to write files in 'extents,' or contiguous, reserved areas. According to an article at Linux-Watch, ext4 will be ready for production use within six to nine months, if all goes well. On the downside, the new ext4 filesystem will offer only limited backward compatibility with ext3-aware Linux kernels."
Unfortunately, this will just murder Reiser4.
1 Exabyte!
Not to be confused with Excitebike, which is something entirely different.
The Slashdot Paradox: "100% Overrated"
EXT4-fs warning (device sdb1): ext4_journal_start_sb: Detected tasteless ReiserFS jokes - hahahaha!
biopowered.co.uk - catalytically cracking triglycerides for home automotive use since 2008. Just say no to big oil!
By now you don't even now what to do with 1024PB. Just as we couldn't imagine filling a 250GB harddrive 15 years ago when 500MB were considered huge.
What will happen? We store our digital photos in raw format, not JPEG. We store our songs in raw format, not artificially crippled. We will store high-definition video, possibly even in raw format, not MPEG4 or the likes.
And, woosh, 1024PB will be nothing leaving us wondering how we could ever survive with a measly 250GB drive -- just as we ask ourselves today how life was with nothing but 170kB disk drives.
>> 1020 petabytes
My porn collection will now be complete.
Lindsay Blanton
RadioReference.com
How does ext4 perform when compared to, say, reiserfs 3.6 or 4? What new features there are?
Who is John Galt?
My question is why they don't mention why it is better to use ext4 then XFS.
XFS can do 9 exabytes (exabyte = 1024 petabytes).
They mention that ext4 is not faster than other filesystems.
Ofcourse people can do whatever they want, but why not spend their time making XFS easily resizable for example?
If you mod this up, your slashdot background will turn into a beautiful sunset!
enabling Linux to support storage volumes up to 1020 petabytes in size
Now, is there anybody who still believes that porn does not drive innovation?
From lwn current issue(you have to subscribe for the full article ;):
Also merged is the developmental ext4 filesystem, which includes a number of enhancements, including support for extents and 48-bit block numbers. See the ext4 documentation file if you are interested in playing with ext4 (and have good backups).
Will we back all this data up?
I'm honestly more interested in someone coming up with cheap, long term archival storage. Hard disks have gone so far past our ability to archive information it's beyond comprehension.
Clear, Dark Skies
This is exactly how I'm filling up my space. I got a new computer with a 160 GB drive, thinking it would be "enough". Started storing all my CDs in FLAC, and I'm currently transfering all hte movies I downloaded in AVI to DVD so I can watch them easily on my home theatre. Once you start working with video and sound that isn't compressed to nothing, you start to realize just how fast you can use up all that space. If my camera did RAW i'd probably use that to store my photos. I usually save any edits I do in PNG or TIFF so that I don't have to worry about the lossy encoding. Granted I still have space to spare, but I could see very easly using up a Terabyte drive if I had it, and a faster internet connection.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
Well, that depends on what your expectations for the future are. I don't think it is impossible that demands on multimedia will reach high enough sometime. Let us as an example consider a movie file from the Future (tm). Given better and bigger screens (perhaps covering whole walls) a frame dimension of 3000 x 2000 pixels is not inconceivable. Each pixel might consist of three RGB values of 16 bits each. Such a movie, if two hours long and running with 25 frames / second, would require about 6.5 TB in raw format.
framedimensions = 3000 x 2000
framebytes = framedimensions x 6
moviebytes = framebytes x 25 x 60 x 120
moviebytes / 10^12 ~= 6.5
to no longer use ReiserFS as its default FS (orig. reported on OSNews.com...don't think I've seen it here yet). I think this came out before the whole Hans Reiser affair, BTW.
SuSE contrasted the ease of upgrading ReiserFS and ExtFS versions:
Carousel is a lie!
What I don't onderstand is that this is merged into the 2.6 kernel tree today. What has happened to the concept of -stable (2.6) and -experimental (2.7) trees? This would be aperfect opportunity to open the next experimental branche..
To Terminate, or not to Terminate, that's the question - SCSIROB
Exactly. How well will EXT4 handle CI/CA splits? What userspace tools will exist to tune VSAM, I mean, EXT4 extents?
With EXT4 having extents we'll finally have the joy of defragmenting a hard drive like Windows people. Yea, progress!
-- @rjamestaylor on Ello
You forgot the 16 channels of 192KHz 32-bit audio you need to. That's 84GiB on it's own!!! :-)
Tom
Someday, I'll have a real sig.
Other Reiser issues aside, the SuSE folks at Novell are looking to leave the nearly unsupported reiserfs3 (in maintenance support, which isn't enough for them) and move to ext3 as their default FS. Why? They feel ext3 is a lot more mature & better/wider supported then reiserfs4, is an easier migration, and appreciate that there is a solid roadmap from ext3 to ext4.
Of course this would also be the week that (coincidentally) Andrew Morton gives reiserfs4 the green light for eventual mainline kernel inclusion.
I don't read ACs: If a post isn't worth so much as a nom de plume to its author then I wont bother either.
The limits are set by our senses, more concretely, our ears and our eyes.
Our ears are only capable of hearing up to about 20Khz (less than that for most people) and 16-bit samplings are enough that most people cannot hear the difference with anything more. Thus CD-quality is, if not perfect, then good enough that further improvements are ignorable for most people. CD-quality losslessly-compressed music is around 300MB/hour.
In a year, there's 8760 hours, so you'd need on the order of 2.5 TB to store a year worth of around-the-clock never-repeateing losslessly-compressed music. If computers keep getting replaced at the current rate, this means you'll never need more than about 10TB to store sound. This assumes you don't store more than you listen to, if you choose to for example store all music ever produced for convenience, despite never listening to more than a tiny fraction of it, then this requirement goes up by a couple of orders of magnitude. Still, there's good reason to suppose that 10TB will suffice for most peoples sound-storage needs. (even if you wanted to store all the sound you've *ever* heard in your life, including traffic at nigth, that'd still only be 200TB or so)
The real killer is video. We can take in a *lot* more data with our eyes. 10GB/hour is in the ballpark of what you'd need for the sort of quality a modern cinema can deliver. (and there's no particular reason we couldn't go higher.) That works out to 100TB/year, more or less. A lifetime of high-quality video is thus on the order of 10PB.
In short, it is unlikely that an individual (or family) will be able to fill a 1000PB disc with sound and video-recordings. Infact it's unlikely they'll be able to fill it with anything, if that anything is to be consumed only trough their 2 eyes and 2 ears.
That doesn't mean it won't happen. Only that it'll be filled with something more. Once we fire up the holodecks all bets are off. I don't even want to try to estimate the bandwith needed for that kind of immersive experience.
it's unlikely they'll be able to fill it with anything
:-)
People always used to ask me if I had the internet at home. Maybe when I can get my hands on a 1020 PB hard drive, I will be able to download it all for local access...
Any fool can talk, but it takes a wise man to listen.
Insightful? Yes. Informative? Certainly not. Finally the Funny mod hits (what took so long?) This is the funniest thing I've read on Slashdot in ages, on so many levels.
On second thought, maybe it is Informative, since I was not previously aware you could cram that many puns into so few words.
Karma: Incomprehensible (Mostly affected by posting at +5, reading at -1, and metamoderating everything unfair.)
Honestly it's a right pig.
Most of the databases I work on are 10-50Tb.
Initially we built 3Gb filesystems - we couldn't back then up, the sequential file pointer in HPUX can only address 2Tb, which meant we couldn't copy the whole filesystem to tape. I had to rebuild with max 2Tb filesystem.
Then, Veritas Netbackup can only parallelise backups in different directory trees so I was taking ages to perform a full back up - 18 - 19 hours (bit of a bugger in a 24hr backup window).
We don't do incremental backups because restoring takes an age with incremental. Even with fulls, if the backup takes 10 hours, the restore takes 15 - its to do with how different simultaneous backups are stiped across the tape. Therefore we always do fulls.
Eventually (and fortunately during test phase) we rebuild the filesystems with 500Gb limits.
Which makes growing the database really sticky because we have to allocate new filesystems, rather than grow the ones we've got.
There's a major move towards disk-based archival storage, but tape's reliable and extremely portable so its generally what we end up using.
My question is why they don't mention why it is better to use ext4 then XFS.
Simple: ext4 is a backwards compatible, evolutionary change from ext3, while XFS is a different file system and codebase. XFS doesn't offer sufficient advantages to overcome that built-in advantage of ext4 (after all, neither XFS nor ReiserFS managed to succeed even against ext3).
Yes, there is a problem.
First of all, disk drives are advancing faster than tapes.
But the problem is worse than that. Different aspects of disk drives are advancing at different rates. Capacity is increasing faster than interface speed is increasing faster than access speed is increasing faster than block reliability.
Consider an old 500MB drive from the mid-90s; it takes maybe a couple hours to read every block on the drive, and odds are that you won't have bad blocks before the disk dies entirely.
The new 1000GB disks we'll have soon, though... you're pretty much guarenteed some of the blocks will go bad before the disk fails; there are just so many of them. And if you can read at 10MB/second, sustained (pretty respectable, I think?), it literally takes 24 hours just to dump all the data from the drive, never mind back it up to something else.
The whole model of 'back up your fixed media to removable media' is not going to work anymore. It does not scale up to modern hardware. I think the answer will be some wacky combo of raid and file-system version control, but YMMV.
Regards,
Exabyte Corporation
- Excitebike sounds like exabyte
- 1020 petabytes != 1 exabyte
- 1 exbibyte != 1 exabyte
- In terms of scale, 1 exbibyte and 1 exabyte are completely different, the difference in this kind of mistake could equal several hundred thousand copies of Wikipedia, as opposed to confusing KiB/KB or MiB/MB, which are different, but not earth-vs-sun different.
- The post was moderated Insightful and then Informative before it was modded Funny
- I'm counting the excellent sig too
That's 6, and I'm sure there are more I'm missing... Sorry if you think I'm stretching, but I really think this is one finely crafted layer-joke, and I just enjoy that kind of thing.Karma: Incomprehensible (Mostly affected by posting at +5, reading at -1, and metamoderating everything unfair.)
The article says "On the downside, the new ext4 filesystem will offer only limited backward compatibility with ext3-aware Linux kernels."
Ext4 is going to be the MOST compatible with Ext3, relative to ANY other option out there.
Upgrading to Ext4 is NOT going to involve a dump and restore from Ext3, likely a tunefs -j or similar command, just as the ext2 -> ext3 migration worked. Ext4 will be able to mount ext3.
If older versions of software could use the new format, you wouldn't need the new format. Yes, upgrading to Ext4 means your 120 petabyte raid array will not be compatible with your old "ext3 aware kernel". But it is PRECISELY because such an array is not possible under ext3 that ext4 is going to be introduced.
And does this submitter think other fancy new filesystems magically work on old kernels? Of course not. Does the submitter know if ext4 will be backported and made available to older releases? It doesn't look like they gave that much thought either.
Please read this for a more detailed description of what is happening.
Slashdot's always good for a smile.
Super computers? Once, maybe - not today, and not for the last decade or so. There are a bunch of companies (I'm working for one of them, now) that will quite cheerfully sell you a storage system that spans hundreds of disks. Assuming your OS won't flake out when it sees a 500+ TB volume, you could mount it on your desktop, if you want. There's absolutely no need to conflate processing power (super computers) with storage capability (NAS, SAN, etc.)
"Great men are not always wise: neither do the aged understand judgement." Job 32:9
who wanted to delete some of the preference files in his home directory, so he typed
.*
rm -rf
It never occurred to him (or me) that ".." matched that pattern. He worked his way right up the directory tree and back down again...
Clear, Dark Skies
I think by that point we'll be digitizing our pron *people*, not pictures and storing them..... think holodeck.