Reaching Beyond Two-Terabyte Filesystems

Jeremy Andrews writes: "Peter Chubb posted a patch to the lkml, with which he's now managed to mount a 15 terabyte file (using JFS and the loopback device). Without the patch, Peter explains, "Linux is limited to 2TB filesystems even on 64-bit systems, because there are various places where the block offset on disc are assigned to unsigned or int 32-bit variables." Peter works on the Gelato project in Australia. His efforts include cleaning up Linux's large filesystem support, removing 32-bit filesystem limitations. When I asked him about the new 64-bit filesystem limits, he offered a comprehensive answer and this interesting link. The full thread can be found here on KernelTrap. Reaching beyond terabytes, beyond pentabytes, on into exabytes. I feel this sudden discontent with my meager 60 gigabyte hard drive..."

Comments

Pentabytes?

[mbrubeck]

Petabytes, please!

pentabytes?

[circletimessquare]

what is that, 5 bytes? ;-P

Wow!

[gazbo]

Any other patches been submitted to the kernel? Perhaps an off-by-one error has been found; maybe an unchecked buffer has been fixed?

Keep it up guys - until they create some sort of 'Linux kernel mailing list' the Slashdot front page is my only source for this information.

fsck times

[danny]

Because fsck would take so long, it's unlikely that a non-journalled filesystem would be used on a large partition/logical volume.

You can say that again! Fscking even 60 gig takes a painfully long time - with 10 terabytes it wouldn't be "go away and take a long coffee break", it would be more like "go away and read a book". And the with the 9EB limit he mentions, maybe "go away and write a book"!

Danny.

Re:fsck times

[orcwog]

Fscking even 60 gig takes a painfully long time

Wow, fsck used to mean fsck, and not....uh...ahem

You know you read /. (and Penny Arcade) too much when you read that and think about Gabe putting a harddrive down his pants.

Re:mod this up... what the hell is a peNtabyte??

[Anonymous Coward]

5 bytes.

No problem

[Beliskner]

Multi-terrabyte files. Hmmmm.

Problem solved: Use lzip

MBA Managers won't notice ;-)

For the hardcore, we can build lzip into the FS. So we'll have Reiserfs, ext2, ext3, JFS, and lzipFS. Heck lzipFS might be faster than RAM!

Re:No problem

[Beliskner]

Only if you can "tune" your lzipFS and trade compression for speed. Something like:

tunelzipfs -c [compression %] /dev/sda1

Uhhh, dude, I was kidding, *someone please mod parent as funny before other innocent people get confused*. Lzip is lossy compression. With a MySQL database or similar this would REALLY test the recovery features. Since MySQL doesn't attach a CRC to each field to ensure field data integrity, you might as well set lzip to 100% compression.

In other words when you try to save a file to lzipFS it might as well return, "yeah" immediately. You tell lzipFS to fsync() and it'll return "yeah" immediately

class lzipFS {
.....
long int fsync() {
// cache->doflush(); /* what we save will be lossy so, what's the point? */
return YEEEEAH_FSYNC_SUCCESFUL;
.....
}}

Re:OS X does this for some time now.

[Wakko Warner]

I'm a proud owner of a Mensa membership card.

Really? Who did you take it from?

- A.P.

Filesystem on tape

[Jah-Wren Ryel]

Reaching beyond terabytes, beyond pentabytes, on into exabytes

Woohoo! A filesytem on a tape drive, that's what I need.