Reaching Beyond Two-Terabyte Filesystems

Jeremy Andrews writes: "Peter Chubb posted a patch to the lkml, with which he's now managed to mount a 15 terabyte file (using JFS and the loopback device). Without the patch, Peter explains, "Linux is limited to 2TB filesystems even on 64-bit systems, because there are various places where the block offset on disc are assigned to unsigned or int 32-bit variables." Peter works on the Gelato project in Australia. His efforts include cleaning up Linux's large filesystem support, removing 32-bit filesystem limitations. When I asked him about the new 64-bit filesystem limits, he offered a comprehensive answer and this interesting link. The full thread can be found here on KernelTrap. Reaching beyond terabytes, beyond pentabytes, on into exabytes. I feel this sudden discontent with my meager 60 gigabyte hard drive..."

Comments

xfs for linux

[mysticbob]

xfs for linux has provided significantly larger than 2Tb filesystems for a while. the official size supported is:

26^3 = 9 x 10^18 = 9 exabytes

check out the feature list.

when a terabyte is not a terabyte

[Anonymous Coward]

As you may know if you've been following recent IEC and IEEE standards (or if you've ever bothered to figure out exactly how large a terabyte is), what disk manufacturers call a terabyte and what this article calls a terabyte differ slightly.

When used in the standard way, the "tera" prefix means 1 * 10^12, so a terabyte would be 1 000 000 000 000 bytes. Unfortunately, computer systems don't use base 10 ("decimal"), they use base 2 ("binary"). When trying to express computer storage capacities, somebody noticed that the SI prefixes kilo, mega, giga, tera, and so on (meaning 10^3, 10^6, 10^9, 10^12, ...) were about the same as 2^10, 2^20, 2^30, 2^40, and so on, so used the terms as multiples of 1024 rather than the usual 1000. On the other hand, many hardware manufacturers (especially hard disk manufacturers) use these prefixes in the standard way to mean exactly multiples of 1000.

This discrepancy causes some confusion. For instance, if you could afford to purchase such a 2 terabyte hard disk, you might well be annoyed when your system tells you your disk is almost 200 gigabytes (2 * (2^40 - 10^12)) smaller than you thought it would be (most systems would report a 2 terabyte disk as a 1.8 terabyte disk).

The moral of the story is one of:

1. don't buy 2 terabyte hard disks (blame the hard disk manufacturers)
2. complain about it then continue the current ambiguity
3. use the standard terminology for binary units
   

Interestingly the Slashdot community seems to think it should be a combination of 1 and 2.

arithmetic?

[Anonymous Coward]

For those who wish to communicate with the rest of the world, the following calculations actually make sense:

• 10^18 bytes = 1 000 000 000 000 000 bytes = 1 decimal terabyte = 1 terabyte = 1 TB
• 2^50 bytes = 1 125 899 906 842 624 bytes = 1 binary terabyte = 1 tebibyte = 1 TiB

For the uninitiated, these terms are described here

Even accounting for your typographical error, 2^63 != 9 * 10^18 (9223372036854775808 != 9000000000000000000)

Maybe Peter Chubb is wrong.

[nr]

Peter explains, "Linux is limited to 2TB filesystems even on 64-bit systems, because there are various places where the block offset on disc are assigned to unsigned or int 32-bit variables."

From the Linux Kernel mailinglist on the status of XFS merge into 2.5:

I know it's been discussed to death, but I am making a formal request to you to include XFS in the main kernel. We (The Sloan Digital Sky Survey) and many, many other groups here at Fermilab would be very happy to have this in the main tree. Currently the SDSS has ~20TB of XFS filesystems, most of which is in our 14 fileservers and database machines. The D-Zero experiment has ~140 desktops running XFS and several XFS fileservers. We've been using it since it was released, and have found it to be very reliable. Uh, so Peter Chubb says there is a 2 TB limit, but these science guys on Fermilab are using Linux with 20 TB filesystems on the SGI XFS port.

Re:Some Disk Array

[Scott Laird]

The point is that you can build a 2+TB system for well under $10k, using 160GB IDE drives and 3ware cards. I have 5 of them, and I've actually had problems -- my first partitioning attempt gave me a 2.06 TB RAID, which mke2fs decided was only 60 GB :-(.

The next round of storage servers that I buy will probably be even bigger, and it'd be nice to be able to use them as one big partition. Pity that I'll have to wait for 2.6 for that.

Not an impossible amount of data

[wirelessbuzzers]

Here is a (somewhet incomplete) answer to the two questions everyone seems to have about 2TB of data:

1) Where would you store it?
Well, you could store it in a holographic Tapestry drive. The prototype, just unveiled a few months ago, stores 100GB in a removable disk, and that is nowhere near the max density of the technology. In their section on projects for the tech, they say that a floppy-sized disk should hold about 1TB in a couple years. Impressive.

2) What would you do with it?
Well, other than high-definition video or scientific experiments, nothing on your own PC, unless you are making a database of all the MP3s ever made or backing up the Library of Congress. But on a file server, you could easily use this much space. The 2TB limit will probably never affect most home users (realizes he will be quoted as an idiot in 10 years when 50TB HDs are standard). On the other hand, Tapestry will probably be useful in portable devices, esp video cameras.