Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: How Do You Store a Half-Petabyte of Data? (And Back It Up?)

Posted by timothy on from the thumb-drives-my-friend-thumb-drives dept.

An anonymous reader writes: My workplace has recently had two internal groups step forward with a request for almost a half-petabyte of disk to store data. The first is a research project that will computationally analyze a quarter petabyte of data in 100-200MB blobs. The second is looking to archive an ever increasing amount of mixed media. Buying a SAN large enough for these tasks is easy, but how do you present it back to the clients? And how do you back it up? Both projects have expressed a preference for a single human-navigable directory tree. The solution should involve clustered servers providing the connectivity between storage and client so that there is no system downtime. Many SAN solutions have a maximum volume limit of only 16TB, which means some sort of volume concatenation or spanning would be required, but is that recommended? Is anyone out there managing gigantic storage needs like this? How did you do it? What worked, what failed, and what would you do differently?

3 of 219 comments (clear)

  1. Ambiguous by smittyoneeach · · Score: 4, Insightful

    Do you mean:
    (a) "Don't store it. Employ Amazon (or some other cloud) storage."? or
    (b) "Do not use Amazon."
    Clarity: it's like that one thing that is not the other thing, except for when it is.

    --
    Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
  2. You are not knowledgeable enough to be in charge by Anonymous Coward · · Score: 0, Insightful

    of this project. You probably can supply enough information to vendors to get proposals (which will be all over the map because you can't be very specific) but I fear you're not in a strong position to evaluate them. You need to have your solution developers talk to a experienced consultant you hire to make recommendations and provide evaluations of vendor bids. You might be able to get it done that way.

  3. Re: Don't by Anonymous Coward · · Score: 3, Insightful

    I think that the intention was to stimulate a discussion amongst a community of geeks who have a genuine interest in this type of technology and enjoy discussing solutions that they have built. Sure, you could just outsource the service and pay consultants to do it for you but I don't think that is the general ethos of the traditional Slashdot reader. Also, if you feel that you should be paid for commenting here then this is probably not the forum for you. Twat.