Slashdot Mirror
Building a Massive Single Volume Storage Solution?
An anonymous reader asks: "I've been asked to build a massive storage solution to scale from an initial threshold of 25TB to 1PB, primarily on commodity hardware and software. Based on my past experience and research, the commercial offerings for such a solution becomes cost prohibitive, and the budget for the solution is fairly small. Some the technologies that I've been scoping out are iSCSI, AoE and plain clustered/grid computers with JBOD (just a bunch of disks). Personally I'm more inclined on a grid cluster with 1GB interface where each node will have about 1-2TB of disk space and each node is based on a 'low' power consumption architecture. Next issue to tackle is finding a file system that could span across all the nodes and yet appear as a single volume to the application servers. At this point data redundancy is not a priority, however it will have to be addressed. My research has not yielded any viable open source alternative (unless Google releases GoogleFS) and I've researched into Lustre, xFS and PVFS. There some interesting commercial products such as the File Director from NeoPath Networks and a few others; however the cost is astronomical.
I would like to know if any Slashdot readers have any experience in build out such a solution? Any help/idea(s) would be greatly appreciated!"
Check out the Internet Archive's Petabox. They have a 100 TB rack running in Europe right now.
I have a feeling you haven't seen http://code.google.com yet. This site just so happens to release code, written by Google employees, available for free. 100% open-source free.
Wny not just a bunch of PC's, each with 6x400GB drives? That's 2.4 TB per PC. 25TB is only 10 PC's -- 60 drives. What's the big deal just using NFS? Seems like not a very difficult target to hit.
Now, going to 1 PB -- that would be 400 PC's. At this point you've left the domain of something which is all that simple. Even so, if you use wake-on-LAN, you could no doubt get away with having 400 PCs, without special power, heating/cooling, etc -- as long as you were able to control the flow of information, so that no more than tiny fraction of the PCs would be on at a given time. And, of course, you'd have significant latency -- waiting for a node to wake. Since I don't know your requirements, I can't say how significant a barrier this would be.
You would get into a reliability issue with 400 PC's. Again, it would be useful to understand something about your problem space. It's possible that a butt-simple method would work for you.
Really, the difficulty/cost of the implementation is a direct function of data flow. If your data flow requirements are tiny, then the solution could be quite cheap.
Of course, even with low outgoing data flow, you'll still have your hands full just filling up the disks in the first place! Consider this: a typical cheap hard drive on a cheap PC can sustain, say, 5MB per second. If one PC is handling 2.4 TB, that would take 5-6 days to fill up. I suppose it would be reasonable to fill up, say, 10-20 PCs at a time. For the 25 TB system, you could fill it up in a week, easy. But for the 1 PB system, filling 20 PCs at a time, it would take you 3 months! Still, that might be OK, depending on your requirements...