Slashdot Mirror
Linux Breaks 100 Petabyte Ceiling
*no comment* writes: "Linux has broken the barrier with the 100 petabyte ceiling, and
doing it at 144 petabytes." And this is even more impressive in pebibytes, too.
← Back to Stories (view on slashdot.org)
Hm, that can't be right, I swear I heard it was supposed to be two raised to the power of 50, multiplied by 128.. hm.
when the rain comes, they run and hide their heads. they might as well be dead.
1e3 terabytes, or 1e6 gigabytes.
N4st0r, trixx0r h0bb1tz0rz! Th3y st0l3 0ur pr3c10uzz!
This would be handy for over 8200 years of DVD video.
Portable versions of Firefox, GIMP, LibreOffice, etc
This is what we've all been waiting for!
Now Linux can really own as a legitimate desktop OS!
Seriously though...Isn't there a better place for someone who has the time to contribute? I'd rather see a better desktop environment, a better E-mail package, etc...
(Flame away, all of you running on 200Mhz machines with a four gig drive who will post about how awesome this new support is!)
The IDE driver supports such rediculously large files, but no filesystem that I know of currently does, not to mention the buffer management code in the kernel.
So does linux support 18pb files? kind of -- pieces of it do. But the system as a whole does not.
2.197265625 trillion Commodore 64's.
98.7881981 billion 1.44 meg floppy disks.
1.44 million 100 gig hard drives
or
3.5 trillion 4K ram chips (remember those?)
If tits were wings it'd be flying around.
remember when 640k was enough for everybody?
well i for one am scared by the fact that oneday soon 144pentabyte files will seem small
- Lord of the Rings is boring. There is a distinct lack of giant robots in it. Good movies have giant robots, bad movies don't. -james
-------
Drink Coffee - Do Stupid Things Faster And With More Energy!
http://oss.sgi.com/projects/xfs/features.html
XFS is a full 64-bit filesystem, and thus, as a filesystem, is capable of handling files as
large as a million terabytes.
263 = 9 x 1018 = 9 exabytes
In future, as the filesystem size limitations of Linux are eliminated XFS will scale to the
largest filesystems
There are about 10^10 solar masses of mass in a large galaxy like our own. At ~10^33 g/ solar maxx, and 10^23 atoms per gram, That's 10^66~2^219 particles in our galaxy. Beleive me, scientists will make use of as much computing power, RAM, and storage space as they can get their hands on. If only the limiting factor were operating system limitations rather than the more practicalities realities of funding and costs of hardware.
"144 PB should be enough for anybody."
- Bowie J. Poag, November 7, 2001
Bowie J. Poag
No, not really. But you must remember, that if you have a need and you wait until you have that need to develop a solution for it, you have developed the solution too late. Remember "640K sould be enough for anyone?" That's an example of not planning for the future adequately. So, no it you're looking for a here and now reason for it, you're not going to find it. But remember, that that's not the point.
Touch everywhere, even when inappropriate.
Just a side note: BeOS has support for files up to 18 exabytes, not 18 petabytes, as stated in the article. This is roughly 18,000 petabytes, or 2^64 bytes.
Just wanted to set the record straight.
- Mike
Now, I can really imagine someone that buys a 144Pb drive (array) and will use IDE?? I would personally go for SCSI there ;-)
What I am really wondering is: is there at the current moment ANY company/application/whatever that required this amount of storage? I thought that even a large bank could manage with a few TB's
Not intended as a flame, just interested
but still, this is a Good Thing (r)
Fighting for peace is like fucking for virginity
So you whip out your two hundred million cd recordables, and start inserting them. Let's say you get 1 frisbee for each 25 700Mb CDs.
This leaves you with eight million frisbees.
That's a stack 13 kilometres high.
So who needs this on a desktop OS again?
Free Java games for your phone: Tontie, Sokoban
Sounds like all they are saying is that the new
IDE driver can support 48 bit addressing. With 2^48 seconds of 512 bytes, you get 144 PB. But there are a LOT of other barriers to huge filesystems or files.
For instance, the Linux SCSI driver has always support 32 bit addressing, good enough for 2 terabytes on a single drive. But until recently, you couldn't have a file larger than 2 gigabytes (1024x smaller) in Linux. I think that the ext2 filesystem still has a limit of 4 TB for a single partition.
So while the IDE driver may be able to deal with a hard drive 144 PB in size, you would still have to chop it into 4 TB partition.
Well, it's good to see that Linux has caught up, but the article is not correct that Linux is the first OS to support 48-bit ATA; FreeBSD has had this support for over a month now.
See for example: this file which is one of the files containing the ATA-6r2 code, committed to FreeBSD on October 6.
http://www.cacr.caltech.edu/~roy/dataquan/
George W. Bush
President, United States of America
I'm looking at the Linux XFS feature page, which states:
My understanding is that the 2TB limit per block device (including logical devices) is firm (regardless of the word size of your architecture), and unrelated to what Mr. Hedrick did. Am I wrong? Does this limit disappear if you build the kernel on a 64-bit architecture?And, on 32-bit architectures, there's no way to get the buffer cache to address more than 16TB.
Is 1 petabyte 1000^5 or 1024^5? (i.e. is it 10^15 or 2^50)?)
:-)
If 1kB = 1024 Bytes, then I've always assumed that 1MB = 1024kB (instead of 1000kB), 1GB = 1024MB, and so on.
Normally this doesn't make that much difference, but when you consider the cost of a 16 (144-128) petabyte hard drive, then the difference is more important
... but a couple of years ago, I was investigating OODBMSs. The sales bloke for (I think it was) Objectivity claimed that CERN were using their database for holding all the information from the particle detector things - which I can see being a shedload of data (3d position + time + energy). He was suggesting figures of 10 petabytes a year for database growth (so it must be frigging huge by now).
Of course, this was probably salescrap. Does anyone know the truth on this?
This sig made only from recycled ASCII
From my perspective, while obscenely large limits on file system sizes are no bad thing, I'm more interested by the prospect for scalability in the context of realistic problems. I see much larger challenges in establishing systems to maximally exploit locality of reference. I'd also like to see memory mapped IO extended to allow direct use to be made of entire large scale disks in a single address space using a VM-like strategy ... but I guess this will only be deemed practicable once we're all using 64 bit processors. Are there any projects to approximate this on 32 bit architectures?
Before you start thumping your chest about how superior or cutting edge *Linux is, go look at these two links
A slashdot story pointing out how without the FreeBSD ATA code, the Linux kernel would be 'lacking'
The FreeBSD press release announcing the code is stable
If The Reg actually researched the story, Andy would have notice it is not a 'first' but more a 'dead heat' between the 2 leading software libre OSes. Instead, The Reg does more hyping of *Linux.
If it was said on slashdot, it MUST be true!
Well, according to the IEC standard, one petabyte is 10^15 (or 1e+15) bytes, while one pebibyte is 2^50 (or 1.125899e+15) bytes.
So 144 petabytes is 1.44e+17 bytes or 127.89769 pebibytes. Can't say that's more impressive tho. :P
SSH has done quite a bit of work to support +2GB files. As always, the following will and always has worked:
:-)
cat file | ssh user@host "cat > file"
More recent builds of SCP will also support +2GB, so:
scp file user@host:/path
or
scp file user@host:/path/file
will both work.
In fact, probably the best way for syncing two directories is rsync. Rsync's major weakness is that it's *tremendously* slow for large numbers of files, and I believe it has to read every byte of a large file before it can incrementally transfer it(so you're looking at 2GB+ of reading before transfering). The following will do rsync over ssh:
rsync -e ssh file user@host:/path/file
rsync -e ssh -r path user@host:/path
For incremental log transfers, I actually had a system built that would ssh into the remote side, determine the filesize of the remote file, and then tail from the total file size minus the size of the remote file. It was a bit messy, but it was incredibly reliable. Did have problems when the remote logs got cycled, but it wasn't too ugly to detect that remote filesize was smaller than localfilesize. Just a shell script, after all.
SFTP should, as far as I know, handle 2GB+ without a hitch.
Both SCP and SSH of course have compression support in the -C tag; alternatively you can pipe SSH through gzip.
Email me for further info; there's some SSH docs onto my home page as well. Good luck
--Dan
www.doxpara.com
BTW, it may also re-open the debate:
Trolling using another account since 2005.
I'm already fed up of the time it takes to back up large disks to tape. Drive transfer rate has not improved at the rate of disk capacity in the last few years and is becoming a bottleneck. It was unimportant when the backup time of a single disk was well below one hour (our Ultrium tapes give about 40Gb/hour).
Just figure that if you want to transfer 144PB in about one day, you need a transfer rate of the order of 1TB/s. Electronics is far from there since it means about 10 terabits/second. Even fiber is not yet there. Barring a major revolution, magnetic media and heads can't be pushed that far. At least it is way further than the foreseeable future.
Don't get me wrong, it is much better to have more address bits than needed to avoid the painful limitations of 528 Mb, 1024 cylinders etc... But, as somebody who used disks over 1Gb on mainfranmes around 1984-1985, I easily saw all the limitations of the early IDE interfaces (with the hell of CHS addresses and its ridiculously low bit numbers once you mixed the BIOS and interface limitations) and insisted on SCSI on my first computer (now CHS is history thanks to LBA, but the transition has been sometimes painful).
However, right now big data centers don't always use the biggest drives because they can get more bandwidth by spread the load on more drives (they are also slightly wary of the greatest and latest because reliability is very important). Backing up starts to take too much time,
In short, the 48 bit block number is not a limit for the next 20 years or so. I may be wrong, but I'd bet it'll take at least 15 years, perhaps much more because it is too dependent on radically new technologies and the fact that the demand for bandwidth to match the increase in capacity will become more prevalent. Increasing the bandwidth is much harder since you'll likely run into noise problems, which are fundamental physical limitations.
<Insert Poster's Name Here>
<Insert Sig Here>
The real advance here is that the disk drive weenies have at last realised that they need to come out with a real fix for the 'big drive' problem and not yet another temporary measure.
Despite the fact that hard drives have increased from 5 Mb storage to 100Gb over the past 20 years the disk drive manufacturers have time after time proposed new interface standards that have been obsolete within a couple of years of their introduction.
Remember the 2Gb barrier? Today we are rapidly approaching the 128Gb barrier.
What annoys me is that the disk drive manufaturers seem to be unable to comprehend the idea of 'automatic configuration'. Why should I have to spend time telling my BIOS how many cylinders and tracks my drive has? I have a couple of older machines with somewhat wonky battery backup for the settings, every so often the damn things forget what size their boot disk is. Like just how many days would it take to define an interface that allowed the BIOS to query the drive about its own geometry?
Of course in many cases the figures you have to enter into the drive config are fiddled because the O/S has some constraint on the size of drives it handles.
We probably need a true 64 bit Linux before people start attaching Petabyte drives for real. For some reason file systems tend to be rife with silly limitations on file sizes etc.
Bit saving made a lot of sense when we had 5Mb hard drives and 100kb floppy drives. It does not make a lot of sense to worry about a 32bit or 64 bit file size field when we are storing 100kb files.
If folk go about modifying Linux, please don't let them just deal with the drives of today. Insist on at least 64 bits for all file size and location pointers.
We are already at the point where Terrabyte storage systems are not unsusual. Petabyte stores are not exactly commonplace but there are several in existence. At any given time there are going to be applications that take 1000 odd of the largest disk available in their day. Today that means people are using 100Tb stores, it won't be very long before 100Pb is reached.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
I figure that at ATA-100 speeds, it would take 49 years to read the entire file.
144 * 2^50 # n bytes
/ 100 * 2^20 # bytes/sec ATA-100
= 1.44 * 2^30 # n I/O seconds
/ 60*60*24*365 # ~ secs/year
= 49.03 # n I/O years
isn't that called google? ;-)
Fighting for peace is like fucking for virginity
144 Petabytes doesn't sound like a lot. When putting it into writing:
144,000,000,000,000,000 or 144*10^15
it's impossible to comprehend.
Here's a way to visualise it - although it's also mindboggeling:
Take a sheet of paper with the squares on it. If you put a single byte in each 5mm by 5mm (1/5" by 1/5") square and use both sides, you'd need:
3,600,000 km^2 of paper to have room for those 144 PB. That's roughly 1,325,525 square miles for you people who don't use the metric system.
So when people say "it doesn't sound like a lot", you know how to get them to understand that it really IS a lot.
We do not live in the 21st century. We live in the 20 second century.
Keeping an archive of Slashdot. As the solar system's population grows and grows, it won't be long before every little news story gets a thousand comments per minute. There will be so many moderators that law of averages suggests that every comment will be modded up to 5, and in an ironic twist Slashdot will be flooded. Still, it's Slashdot, and no self-respecting high-bandwidth nerd will be without an up-to-date archive of Slashdot.
Leeching Aminet. By the time we actually have these monster size drives, processors will finally be fast enough to properly emulate an Amiga, WinUAE will have been perfected and bandwidth will be so plentiful that we can all enjoy the latest Amiga software, whether we want it or not.
Freaking out newbies. Remember your scriptkiddie days when you would h4x0r some dude's Windows machine and pop up something resembling the Matrix? Simply add a little matter-to-energy technology, and you can download the newbie onto his computer, FTP him along (resumable downloading, now, we don't want him to materialise with missing parts!) and rematerialise him in your fridge. He'll think he's been transported to some crazy ice planet. Just like in sci-fi, eh folks!
Somewhere to keep all your Pokémon hentai! Don't try and hide it, man. I've seen your sick pictures of Misty and Bulbasaur.
You'll finally have enough diskspace to install Windows 2024. Naturally, you'll be using Linux instead, but it's nice to brag that you could, if you wanted.
Think *Cinema*.
Current codecs already do a pretty decent job of compression of smaller(resolution) streams. However, what if I want my linux box feeding my HDTV projector at high resolution? This might be one more step in my vision of the ultimate entertainment center.
You can't legislate goodness. Let each to his own destiny, by will of his freely made choices.
This obviously mattered to the people who implemented it. If you'd rather see development move in a different direction, by all means, write some code that you feel is useful.
See, the people who implemented this probably don't give a damn what you feel is important, they care about what they feel is important.
It's really very simple, put up or shut up.
This limit is for a SINGLE IDE disk. Now, if you use Logical Volume Management (which is in the standard 2.4 kernel, no patches required) you can combine multiple disks into one.
Since my machine has 2 IDE controllers, with 2 buses each, and 2 drives per bus, you could make a system with 8 144 pB drives, put an XFS partition on it, and have 1152.92 pB of storage.
And for meaningless statistics sake: I make my MP3s (from CDs that I own, thankyouverymuch) at an average of 160 kb/sec. At that rate, the specified drive array would store 1826693 YEARS of MP3s. None of which would be Brittany Spears.
www.eFax.com are spammers
http://www.freebsd.org/news/newsflash.html#2001Nov ember3:1
... look at that 48 bit addressing ATA drivers are now working? Wow... maybe FreeBSD people should run around making bogus claims too. FreeBSD invented the Question Mark! Wooo hoo.
Hey
I also think you can use Vinum to mount such a petabyte sized file system fairily easilly.
Really FreeBSD doesn't get enough credit for work that's been done. I know linux has a lot of good marketing for technical features but you also have to believe everything you read to fall for it.
I was just doing an order of magnitude estimate to illustrate the point that scientists will use as much computing power, memory, storage, etc. as they can afford. I wasn't paying attention to factors of 6 here or there.
Obviously, there are more relevant issues. For example, how are you going to store X bits of information using Y particles? At least for classical computing, you have a problem if Y is orders of magnitude less than X. Hence storing 8 numbers for each atom in the galaxy would be impossible if you were confined to using only the atoms on the earth, at least in classical computing. (I beleive with quantum computing in principle you can be clever and get around this, but I don't know enough to say for sure.)
But to answer your question since over 70% of the baryonic matter is hydrogen, nearly all the rest is helium, and less than 2% is heavier, the average molar mass of baryonic matter in the universe is less than 2.
However, there might one day be information processing systems to which 1.44 petabytes is a small amount of information. In a sense, these systems will have a richer experience of the world than human beings. I wonder if human consciousness would seem marvellous or valuable to such a machine.
That's only about 177 years' worth of 640x480, 24-color, 30fps uncompressed video.
Sheesh. I at least want to be able to chronicle the entire history of mankind in uncompressed video on my Linux box. Right now I'll have to settle for the history of the Industrial Age, or split my documentary into several smaller files.
dinner: it's what's for beer
The point is, if I have the choice, I will choose to develop on a system where I have access to the source, for a number of reasons, only partially technical. There is no "collective mind". Developers are highly independent and like to work on what interests them. If you're interested in reaping the rewards from something, you sometimes need to earn them. Whether this is actually contributing code, funding development, etc.
Open source platforms were created by hackers, for hackers. And typically we don't give a damn about widespread acceptance or overthrowing microsoft's dominance of the desktop. We just want something that works well for what we need. Try to understand it from that perspective and you'll do better.
People want a lot of things, but the only people who really matter here are the people implementing this system. See, thats the great part, if you want it to be something it's not, make it that way. And personally, I do think they're possible. If it weren't for legacy applications, Linux would likely be on a lot more desktops than it is. I know any clueful sysadmin would much rather maintain a bunch of linux boxen than windows boxen. From a management perspective, Linux is lightyears beyond windows. Especially considering if something doesn't work right, instead of looking for a kludge or trying to get a vendor to include the needed functionality (usually a combonation of the two), you can locate the problem, isolate it, and correct it. I know of at least one place I've worked where this ability would have saved the company literally millions of dollars. Theres nothing wrong with cheerleaders to keep the team motivated. As a matter of fact, if you really think about it, recognition is the sole form of payment quite a few oss developers receive. Its all about the right tool for the job. If you want to play the latest and greatest games, linux isn't a good desktop choice for you. For the people maintaining 5000 corporate PCs with custom apps, it becomes a very sensible desktop OS.Personally, I run FreeBSD and a mix of NT/2000. Windows is still a requirement for me (a couple of addictive games, and some apps that my job requires). And the majority of the time I'm in windows, I have emacs/tcsh/python windows up (Exceed is a dream here). I personally would LOVE to get windows off my desktop, but it's the applications that keep me there, applications are key.
Thanks for the thoughtful reply :)
Patrick Doyle
I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
This is just geek fodder....Hey, we're using 48bit addressing. Which means you can have 10 petabytes of pr0n now!!! It just sounds cool is all, it doesn't mean anything practical.
But, impracticality is much more interesting, isn't it?
Why do you care what the BIOS thinks? Set it to NONE and be done with it.
Any "modern" OS doesn't use what the BIOS thinks anyway. Try it with Linux sometime. Stick a 60 gig disk into an old 486 that can't handle it, set it to none, boot up Linux and watch it tell you that there's a 60 gig disk there, and more importantly, watch it WORK in all respects. Watch it have full access to the whole thing. Be careful, if the BIOS is set to something other than NONE, it *can* lie to Linux when the kernel asks for the size of the drive. But I've done this with systems that have a 32 gig limit on disk size, and have it work just fine.
Windows can do this too, sometimes. Not exactly certain on the details there, but having done this myself with Linux, I know that much of it does work.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
18 petabytes *is* enormous, as was more enormous back then. Linux is not the first, but rather the current overwhelming champion. I hate grammar.
That single drive could hold... everything!
Let's all pitch in and buy a big fat bandwidth pipe and fancy hardware interface and an array of these drives and we can store everything we want.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
It's impossible to conceive of Linux _needing_ that big a hard drive. But think of how fast Microsoft code bloats. Every few years M$ has to invent a new file system to properly handle the larger drives needed to hold Windows & Office. And so who knows how big common disk drives will be in 10 years? But Linux is ready NOW... ;-)
This would be a much better article if the headline read "IBM breaks 100 petabyte barrier." Or Maxtor. Or Western Digital. Or perhaps Quantum. See what I mean?
The Register updated their article. It now acknowledges FreeBSD as being the first Unix to support multi-petabyte filesizes.
However, NTFS 5.0 (the filesystem that is used by Windows 2000) has had 64-bit addressing since Windows 2000 was released. This yields a maximum capacity of 16 exabytes, which is 8388608 Petabytes. That's right, Windows has supported files eighty thousans times larger than Linux with an experimental patch for the past few years. Still, by the time people actually start needing this kind of storage, I don't think it'll actually matter much...
Check out the corrected register article. :-)
FreeBSD had 48bit IDE addressing support
in the CVS repository on Oct 6! A full month
before these patches to linux were released. So
far no released kernel supports this.
An entire day of a human life could be recorded in perfect detail (with no compression) on a 120 GB disk.
Let me guess - you're using roughly the bitrate of DVD, extrapolating over 24 hours, and fudging the numbers. Well, either you or whoever came up with this figure.
Let's look at this from a cocktail napkin perspective. At the CURRENT resolution and audio sampling rate etc for DVD, 24 hours is about 50GB of storage. Only problem is, this assumes that DVD catches every single bit of visual/audio information that is out there. Well, just ask your dog how well 44Khz records high pitched noises. And then remember that not everyone has as poor eyesight/hearing as the masses. So even fudging this number by a factor of 2 or 3 starts to hit and overtake 120GB.
Oh wait, this assumes that all we care about is what the eyes see and the ears hear. Too bad that things are happening all around you. Also too bad that you have 3 other external senses, plus several other internal ones (balance comes to mind) that are continually inputting data into your brain.
Estimates like this really make me shake my head, as they assume artificial limitations that just aren't there in the real, ANALOG world.
Endless arguments over trivial contradictions in books written by ignorant savages to explain thunder in the dark.