Ask Slashdot: How Do You Store a Half-Petabyte of Data? (And Back It Up?)

An anonymous reader writes: My workplace has recently had two internal groups step forward with a request for almost a half-petabyte of disk to store data. The first is a research project that will computationally analyze a quarter petabyte of data in 100-200MB blobs. The second is looking to archive an ever increasing amount of mixed media. Buying a SAN large enough for these tasks is easy, but how do you present it back to the clients? And how do you back it up? Both projects have expressed a preference for a single human-navigable directory tree. The solution should involve clustered servers providing the connectivity between storage and client so that there is no system downtime. Many SAN solutions have a maximum volume limit of only 16TB, which means some sort of volume concatenation or spanning would be required, but is that recommended? Is anyone out there managing gigantic storage needs like this? How did you do it? What worked, what failed, and what would you do differently?

Comments

use slashdotFS

[goombah99]

I use slashdotFS which is a markovian random comment generator which effectively embeds data in a stegenographic comment. The FS handles the details of creating and saving these so it's all transparent and mounts on your desktop like a regular drive. It's slow but it's capacity seems unlimited and frequently gets modded insightful

Easy

[ArcadeMan]

How Do You Store a Half-Petabyte of Data? (And Back It Up?)

That's the easiest question I've ever seen.

1. Wait about a decade or so.
2. Buy two half-petabyte flash drives.
3. Alternate your copies on the two flash drives, the previous one becomes your backup.

NEXT!