Slashdot Mirror


Reaching Beyond Two-Terabyte Filesystems

Jeremy Andrews writes: "Peter Chubb posted a patch to the lkml, with which he's now managed to mount a 15 terabyte file (using JFS and the loopback device). Without the patch, Peter explains, "Linux is limited to 2TB filesystems even on 64-bit systems, because there are various places where the block offset on disc are assigned to unsigned or int 32-bit variables." Peter works on the Gelato project in Australia. His efforts include cleaning up Linux's large filesystem support, removing 32-bit filesystem limitations. When I asked him about the new 64-bit filesystem limits, he offered a comprehensive answer and this interesting link. The full thread can be found here on KernelTrap. Reaching beyond terabytes, beyond pentabytes, on into exabytes. I feel this sudden discontent with my meager 60 gigabyte hard drive..."

12 of 173 comments (clear)

  1. Testing by Rick+the+Red · · Score: 3, Interesting
    The thought of generating the test files is mind-boggling. Unless you work at CERN, where they probably have 16T files just lying around...

    --
    If all this should have a reason, we would be the last to know.
  2. Brain Contents by dscottj · · Score: 3, Interesting
    I seem to recall reading, probably in a science fiction book, that the human brain is thought to store somewhere in the neighborhood of ~2-4 terabytes of information.


    Aside from all sorts of quantum fiddly bit problems, I wonder just how long it will be before we can store the state of every neuron in a brain (doesn't have to be human, at least not at first) on a hard drive.


    Of course, then what would you do with it?

    --
    AMCGLTD.COM. Where cats, science fictio
  3. Pentabytes? by mbrubeck · · Score: 4, Funny

    Petabytes, please!

  4. pentabytes? by circletimessquare · · Score: 5, Funny

    what is that, 5 bytes? ;-P

    --
    intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
  5. Wow! by gazbo · · Score: 5, Funny
    Any other patches been submitted to the kernel? Perhaps an off-by-one error has been found; maybe an unchecked buffer has been fixed?

    Keep it up guys - until they create some sort of 'Linux kernel mailing list' the Slashdot front page is my only source for this information.

  6. xfs for linux by mysticbob · · Score: 5, Informative
    xfs for linux has provided significantly larger than 2Tb filesystems for a while. the official size supported is:

    26^3 = 9 x 10^18 = 9 exabytes

    check out the feature list.

  7. Files that big by Alien54 · · Score: 3, Interesting
    I can see certain high resolution videos getting this large.

    but I worry about other data types.

    For example, I grumple at the MS stupidity of putting all datafiles into one large container file in a database base under Access in Windows. Which is why I never use it. I prefer discrete files. If one gets hosed, then it is easier to fix.

    obviously a database that is that big would run into other performance issues as well. Some of which is handled by moore's law, and some of which isn't.

    for similar reasons I tend to divided my drive into various partitions, regardless of which OS I use.

    --
    "It is a greater offense to steal men's labor, than their clothes"
  8. fsck times by danny · · Score: 4, Funny
    Because fsck would take so long, it's unlikely that a non-journalled filesystem would be used on a large partition/logical volume.
    You can say that again! Fscking even 60 gig takes a painfully long time - with 10 terabytes it wouldn't be "go away and take a long coffee break", it would be more like "go away and read a book". And the with the 9EB limit he mentions, maybe "go away and write a book"!

    Danny.

    --
    I have written over 900 book reviews
  9. arithmetic? by Anonymous Coward · · Score: 3, Informative

    For those who wish to communicate with the rest of the world, the following calculations actually make sense:

    • 10^18 bytes = 1 000 000 000 000 000 bytes = 1 decimal terabyte = 1 terabyte = 1 TB
    • 2^50 bytes = 1 125 899 906 842 624 bytes = 1 binary terabyte = 1 tebibyte = 1 TiB

    For the uninitiated, these terms are described here

    Even accounting for your typographical error, 2^63 != 9 * 10^18 (9223372036854775808 != 9000000000000000000)

  10. Filesystem on tape by Jah-Wren+Ryel · · Score: 3, Funny

    Reaching beyond terabytes, beyond pentabytes, on into exabytes

    Woohoo! A filesytem on a tape drive, that's what I need.

    --
    When information is power, privacy is freedom.
  11. Trademark infringement by Webmoth · · Score: 3, Interesting

    Looks like we'll have to come up with a different naming scheme. Someone's already trademarked the exabyte.

    Couldn't it weaken the trademark to have Western Digital or Seagate making a '9 exabyte' hard drive? Or HP or Sony making an 'exabyte-class' tape drive? Wouldn't a judge find (in favor of Exabyte) that the consumer would easily be confused?

    *The USPTO are idiots.*

    --
    Give me my freedom, and I'll take care of my own security, thank you.
  12. Re:No problem by Beliskner · · Score: 3, Funny
    Only if you can "tune" your lzipFS and trade compression for speed. Something like:

    tunelzipfs -c [compression %] /dev/sda1
    Uhhh, dude, I was kidding, *someone please mod parent as funny before other innocent people get confused*. Lzip is lossy compression. With a MySQL database or similar this would REALLY test the recovery features. Since MySQL doesn't attach a CRC to each field to ensure field data integrity, you might as well set lzip to 100% compression.

    In other words when you try to save a file to lzipFS it might as well return, "yeah" immediately. You tell lzipFS to fsync() and it'll return "yeah" immediately

    class lzipFS {
    .....
    long int fsync() {
    // cache->doflush(); /* what we save will be lossy so, what's the point? */
    return YEEEEAH_FSYNC_SUCCESFUL;
    .....
    }}

    --
    A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?