Slashdot Mirror
Reaching Beyond Two-Terabyte Filesystems
Jeremy Andrews writes: "Peter Chubb posted a patch to the lkml, with which he's now managed to mount a 15 terabyte file (using JFS and the loopback device). Without the patch, Peter explains, "Linux is limited to 2TB filesystems even on 64-bit systems, because there are various places where the block offset on disc are assigned to unsigned or int 32-bit variables."
Peter works on the Gelato project in Australia. His efforts include cleaning up Linux's large filesystem support, removing 32-bit filesystem limitations. When I asked him about the new 64-bit filesystem limits, he offered a comprehensive answer and this interesting link. The full thread can be found here on KernelTrap.
Reaching beyond terabytes, beyond pentabytes, on into exabytes. I feel this sudden discontent with my meager 60 gigabyte hard drive..."
If all this should have a reason, we would be the last to know.
Aside from all sorts of quantum fiddly bit problems, I wonder just how long it will be before we can store the state of every neuron in a brain (doesn't have to be human, at least not at first) on a hard drive.
Of course, then what would you do with it?
AMCGLTD.COM. Where cats, science fictio
Petabytes, please!
Well, we have here and RAIDED 60 TB array which runs well und Mac OS X. This is mainly because Darwin is based on FreeBSD. The BSD series comes from the professional/academic unix world and has automatical 64 bit support at all level for 9 years or so.
It's not very suprising that Linux is lacking these features. It's more hobbyist style and still contains some serious design failures like the missing microkernel Mac OS X has for some time now.
Many people here at slashdot bitch at the academic/professional world but at examples like this you see that professional, thoughful design always pays off in some time.
Owner of a Mensa membership card.
what is that, 5 bytes? ;-P
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Yup. But divide that number by two every two years, and you'll see that it'll be within small business' price range within a decade.
Keep it up guys - until they create some sort of 'Linux kernel mailing list' the Slashdot front page is my only source for this information.
26^3 = 9 x 10^18 = 9 exabytes
check out the feature list.
Once upon a time, I saw a big company producing some classified devices for the Soviet military-industrial complex. Of course, the company had an accounting department. And there was a company accounting database. It was a single file about 80 MBytes long (The typical drive size these days was 20-40 MB). To simplify the access tasks, the programmers that created the database software decided that all the data from time immemorial are to be kept in this file. The file grows with every operation, and since the data are thought to be needed forever there is no method to remove irrelevant entries.
The programmers didn't imagine that in pair of years the base will be so big that it will not fit into any available HDD.
Maybe it will be the lesson for some people who are going to misuse the file system features?
but I worry about other data types.
For example, I grumple at the MS stupidity of putting all datafiles into one large container file in a database base under Access in Windows. Which is why I never use it. I prefer discrete files. If one gets hosed, then it is easier to fix.
obviously a database that is that big would run into other performance issues as well. Some of which is handled by moore's law, and some of which isn't.
for similar reasons I tend to divided my drive into various partitions, regardless of which OS I use.
"It is a greater offense to steal men's labor, than their clothes"
Danny.
I have written over 900 book reviews
Imagine if the Trueman Show (like in the movie) was recorded as one huge MPEG video - you could store it one of of these! :-)
You could fit movies of everything anyone's ever seen on a Beowulf cluster of these filesystems!
Follow me
For those who wish to communicate with the rest of the world, the following calculations actually make sense:
For the uninitiated, these terms are described here
Even accounting for your typographical error, 2^63 != 9 * 10^18 (9223372036854775808 != 9000000000000000000)
Problem solved: Use lzip
MBA Managers won't notice ;-)
For the hardcore, we can build lzip into the FS. So we'll have Reiserfs, ext2, ext3, JFS, and lzipFS. Heck lzipFS might be faster than RAM!
A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?
I work with Seismic data, usually for finding natural resources like oil. Since the data is aquired from surveying the under-ground with sound waves, the amount of data possible is insane. The software I work with, has to limit the files to 2 gig because of the 32 bit file offset limit.
I'm sure nuclear simulations, or any natural simulation (like weather) will create massive datasets too.
metric
From the Linux Kernel mailinglist on the status of XFS merge into 2.5:
I know it's been discussed to death, but I am making a formal request to you to include XFS in the main kernel. We (The Sloan Digital Sky Survey) and many, many other groups here at Fermilab would be very happy to have this in the main tree. Currently the SDSS has ~20TB of XFS filesystems, most of which is in our 14 fileservers and database machines. The D-Zero experiment has ~140 desktops running XFS and several XFS fileservers. We've been using it since it was released, and have found it to be very reliable. Uh, so Peter Chubb says there is a 2 TB limit, but these science guys on Fermilab are using Linux with 20 TB filesystems on the SGI XFS port.
Or is that _still_ at a meager 2GB limit?
The point is that you can build a 2+TB system for well under $10k, using 160GB IDE drives and 3ware cards. I have 5 of them, and I've actually had problems -- my first partitioning attempt gave me a 2.06 TB RAID, which mke2fs decided was only 60 GB :-(.
The next round of storage servers that I buy will probably be even bigger, and it'd be nice to be able to use them as one big partition. Pity that I'll have to wait for 2.6 for that.
Here is a (somewhet incomplete) answer to the two questions everyone seems to have about 2TB of data:
1) Where would you store it?
Well, you could store it in a holographic Tapestry drive. The prototype, just unveiled a few months ago, stores 100GB in a removable disk, and that is nowhere near the max density of the technology. In their section on projects for the tech, they say that a floppy-sized disk should hold about 1TB in a couple years. Impressive.
2) What would you do with it?
Well, other than high-definition video or scientific experiments, nothing on your own PC, unless you are making a database of all the MP3s ever made or backing up the Library of Congress. But on a file server, you could easily use this much space. The 2TB limit will probably never affect most home users (realizes he will be quoted as an idiot in 10 years when 50TB HDs are standard). On the other hand, Tapestry will probably be useful in portable devices, esp video cameras.
I hereby place the above post in the public domain.
Reaching beyond terabytes, beyond pentabytes, on into exabytes
Woohoo! A filesytem on a tape drive, that's what I need.
When information is power, privacy is freedom.
It seems to me that it would be more practical to make the file storage and management system be *independent* of the OS. This would allow storage companies to get economies of scale by not having to worry about OS-specific issues.
The "native" disk storage could be used as a kind of cache. The "big fat" storage would be like a *service* that could be local or remote. The OS would not care. It simply makes an API call to the "storage service".
Table-ized A.I.
Looks like we'll have to come up with a different naming scheme. Someone's already trademarked the exabyte.
Couldn't it weaken the trademark to have Western Digital or Seagate making a '9 exabyte' hard drive? Or HP or Sony making an 'exabyte-class' tape drive? Wouldn't a judge find (in favor of Exabyte) that the consumer would easily be confused?
*The USPTO are idiots.*
Give me my freedom, and I'll take care of my own security, thank you.
You want at least a 20KM grid resolution (actually, you rather better than that, but we have real-world constraints in my business :-) -- that means something like a
200x200x25 grid. So that (at 4 bytes/number), one 3D state variable occupies 4 MB. For air
pollution, you will need about 60 such variables
(12 meteorology state variables, and another 48
or so chemistry variables): 240 MB per time step.
The summer air pollution season is about 100 days long, and you'll want to use a time step of half an hour or better (by Courant's theorem, wind speed gives the translation factor between spatial resolution and the required temporal), so that's 4800 time steps.
240 MB/step times 4800 steps -- about a terabyte.
Go to a (better) 10KM resolution, and the compute time and data set size go up by 2^4. fwiw.
"My opinions are my own, and I've got *lots* of them!"
since 64 bit ints are sufficient for any imaginable program.
That's just like saying, no one would ever need more than 640K.
Just because it CAN be done, doesn't mean it should!
Intelligent people have no problem with the idea that a kilobyte has 1,024 characters. Hard drive manufacturers always have, but they are hardly paragons worthy of emulation.
Stop out the kibibyte nonsense now, before it gets any further.
The solution is to label hard drives in accordance with the rest of computer technology. A kilobyte is 1,024 bytes, not 1,000. The kibibyte does not exist!
When things go larger and larger, I get confused.
Okay, I know what's Exabyte, but what are the-still-larger ones ?
Muchas Gracias, Señor Edward Snowden !