Slashdot Mirror
Building a Massive Single Volume Storage Solution?
An anonymous reader asks: "I've been asked to build a massive storage solution to scale from an initial threshold of 25TB to 1PB, primarily on commodity hardware and software. Based on my past experience and research, the commercial offerings for such a solution becomes cost prohibitive, and the budget for the solution is fairly small. Some the technologies that I've been scoping out are iSCSI, AoE and plain clustered/grid computers with JBOD (just a bunch of disks). Personally I'm more inclined on a grid cluster with 1GB interface where each node will have about 1-2TB of disk space and each node is based on a 'low' power consumption architecture. Next issue to tackle is finding a file system that could span across all the nodes and yet appear as a single volume to the application servers. At this point data redundancy is not a priority, however it will have to be addressed. My research has not yielded any viable open source alternative (unless Google releases GoogleFS) and I've researched into Lustre, xFS and PVFS. There some interesting commercial products such as the File Director from NeoPath Networks and a few others; however the cost is astronomical.
I would like to know if any Slashdot readers have any experience in build out such a solution? Any help/idea(s) would be greatly appreciated!"
register a few thousand gmail accounts and write the interface that will make writing of data to gmail inboxes invisible to the app.
I know a certain recent Zombie network that was discovered which collectively had quite a few Pbs of storage... Of course I wouldn't recommend going down that road as it leads to you know ... jail.
News Reporters Make Tasty Polar Bear Treats!
I never thought I'd see the day when sites were boasting a petabyte of porn. .avis -- if you sat down and watched them end-to-end, you'd have 348 years of "backdoor sliders", "dribblers to short", "pop flies", and "long balls". We live in an enlightened age.
That's over 3 million hours of
With a project this large, they may be able to do it in-house and still take advantage of economies of scale. They can buy HDDs, motherboards, rackmount cases, etc. by the pallet or container load and temporarily up-hire some of their part-timers to do the assembly.
With a network bootable bios, the nodes could just be plugged in and install an image off a server, then customize it based on their MAC.
Just don't call it PetaFile.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
I concur. Everything I have researched matches what you have stated. It is not likely this will be a very easy task to perform on a budget (depending on what he is calling a "budget"). I would guess that GoogleFS is the only viable solution other than Lustre, depending on what he is attempting to use this storage for. If large file storage is what he desires, this may be the answer once it is released to the public.
Then I suggest using *nothing*. It's free, and will work with the appropriate hardware and software add-ons.
Ask Slashdot answers have somehow failed to put EMC, IBM, HP, etc. out of business.
! !!!!!!!!!!
Yes, that's right... You can't do it yourself... There's never an open source solution... You must buy an expensive solution.... From a big company.... You're getting very....very...sleepy...
MMMMWWWUUUUUUHAHAHAHAHAHAHAHAHAHAHAHAHAHA!!!!!!!!
- large porn archives.
I have about 2.5TB of it here; and I'm an amateur. If I were a porn business, I could easily desire acquiring 20X what I have. And in those quantities, 90+% of it isn't worth backing up."I can't imagine having that much data I didn't care about."
Hollywood script archive.
The global economy is a great thing until you feel it locally.
Dear Slashdot,
I have been tasked with (insert very difficult, very important job). This is very important to my company. I have (insert number much lower than it should be) dollars to do this. I do not want to use (insert company name specializing in this exact thing) because management thinks they are too expensive. I think I can do this (insert better/faster/cheaper/...) than said company, even though they have vastly more experience and have invested much more time and research than I have. My continued and future employment probably rests on this project. Please advise.
Maybe not
And you'll probably get at least one nice lunch out of the sales deal. I recommend saving your lunch money and asking for sales visits from all of the major players.
Why not store the data randomly in a dilithium matrix with asynchronous data transfer and AJAX? Maybe some RUBY on RAILS too - I hear that's hot right now. Of course, you'd have to make use of a couple of Heisenberg compensators configured in parallel to keep track account for any memory addressing issues, but no need to state the obvious there.
"Nokia is not a country, it's the capital of Finland!" -Moderated "Informative". Yeesh.
We at Vap-o-tech 2003 Inc. (not associated with Vap-o-tech 2001 Inc. which has closed its doors due to allegations of investor fraud) have developed ToastFS 2003. Using patented CRUMB technology and high capacity BUTTER read/write caching, we are able to turn your average loaf of Wunderbread into a 200gb storage media. Simply buy a loaf of our own specially tested Wunderbread ($250 USD) along with a USB-to-Popup Toaster interface (don't worry, USB 2.0 is more than capable of handling 120amp wall sockets without a problem, except in California). Then take our Vap-o-bake ToastFS drive and pop two pieces in. For doubled capacity, buy our Vap-o-bake ToastFSx2 drive, which takes four pieces. From a command prompt, simply type FORMAT C: and answer yes. Your new ToastFS drive will be formatted in minutes. Please note that we have 24 hour technical support via 1-900-842-8524 ext 241. Please don't hang up. Our operators in the Dutch Antilles are very busy and could take upwards of an hour to get to you.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Maybe he needs it for a swap file? I heard Microsoft upped the memory requirements in the next version of windows.
Dear Poster.
This is Slashdot India and Slashdot China. With over a Billion people combined we can do it by using PeopleRAID, which is hot-swappable once every generation, and redundant once we eliminate our "one child per a couple" restrictions, and increase our Viagra imports.
what on Earth kind of data do they anticipate will take a petabyte of contiguous storage?
I know. They don't know I know, but I do. It's data gathered by the black helicopters, by Echelon, by Carnivore, by our very own printers, by RFID, about every movement of every single one of us... *They* do it. They.
Stupidity is an equal opportunity striker.
Fellow slashdotter Bill Dog
Wow! This is a real conspiracy theory. All I could think of was Porn. :)
exit();
Ah, but the grandparent post said no redundancy.
Yeah, but apart from that, did it work out for you? Don't hold back, I can take the truth.
Money for nothing, pix for free