Reaching Beyond Two-Terabyte Filesystems

Jeremy Andrews writes: "Peter Chubb posted a patch to the lkml, with which he's now managed to mount a 15 terabyte file (using JFS and the loopback device). Without the patch, Peter explains, "Linux is limited to 2TB filesystems even on 64-bit systems, because there are various places where the block offset on disc are assigned to unsigned or int 32-bit variables." Peter works on the Gelato project in Australia. His efforts include cleaning up Linux's large filesystem support, removing 32-bit filesystem limitations. When I asked him about the new 64-bit filesystem limits, he offered a comprehensive answer and this interesting link. The full thread can be found here on KernelTrap. Reaching beyond terabytes, beyond pentabytes, on into exabytes. I feel this sudden discontent with my meager 60 gigabyte hard drive..."

Comments

Testing

[Rick the Red]

The thought of generating the test files is mind-boggling. Unless you work at CERN, where they probably have 16T files just lying around...

Brain Contents

[dscottj]

I seem to recall reading, probably in a science fiction book, that the human brain is thought to store somewhere in the neighborhood of ~2-4 terabytes of information.

Aside from all sorts of quantum fiddly bit problems, I wonder just how long it will be before we can store the state of every neuron in a brain (doesn't have to be human, at least not at first) on a hard drive.

Of course, then what would you do with it?

Re:OS X does this for some time now.

[say]

On the other hand, approximately 0 % of Linux' intended uses does need 60 TB at this time. As the world and kernel evolves, this will be fixed if the linux community needs it.

Yes, it's hobbyist based. Yes, it's great that FreeBSD supports it. Honourful! But Linux has had more important features to implement before this - because only a very few people have had access to these kind of disks.

However, 2 TB is not that much - and it's about time Linux supports it.

Do you really need it?

[Thor Ablestar]

Once upon a time, I saw a big company producing some classified devices for the Soviet military-industrial complex. Of course, the company had an accounting department. And there was a company accounting database. It was a single file about 80 MBytes long (The typical drive size these days was 20-40 MB). To simplify the access tasks, the programmers that created the database software decided that all the data from time immemorial are to be kept in this file. The file grows with every operation, and since the data are thought to be needed forever there is no method to remove irrelevant entries.

The programmers didn't imagine that in pair of years the base will be so big that it will not fit into any available HDD.

Maybe it will be the lesson for some people who are going to misuse the file system features?

Files that big

[Alien54]

I can see certain high resolution videos getting this large.

but I worry about other data types.

For example, I grumple at the MS stupidity of putting all datafiles into one large container file in a database base under Access in Windows. Which is why I never use it. I prefer discrete files. If one gets hosed, then it is easier to fix.

obviously a database that is that big would run into other performance issues as well. Some of which is handled by moore's law, and some of which isn't.

for similar reasons I tend to divided my drive into various partitions, regardless of which OS I use.

Trademark infringement

[Webmoth]

Looks like we'll have to come up with a different naming scheme. Someone's already trademarked the exabyte.

Couldn't it weaken the trademark to have Western Digital or Seagate making a '9 exabyte' hard drive? Or HP or Sony making an 'exabyte-class' tape drive? Wouldn't a judge find (in favor of Exabyte) that the consumer would easily be confused?

*The USPTO are idiots.*